Treasury Board of Canada Secretariat
Symbol of the Government of Canada

Outcome Management Guide and Tools


Appendix O:  References

Agence pour le Développement de l’Administration Électronique, Guide méthodologique MAREVA : Analyse de la valeur des projets d’ADELE, Ministere du Budget et de la reforme de l’Etat, 2005, http://www.adae.gouv.fr/IMG/pdf/050405_MAREVA_GuideMethodologique_vf.pdf

This document covers the e-government program ADELE from France, which sets out the government’s online strategy for the period of 2004-2007.  To evaluate gains and savings for each e-government project, the French government developed the evaluation methodology called MAREVA, which enables a precise evaluation of financial gains of e-government services for the State and the public sector, as well as of gains and benefits for end users.

Australian Government, Demand and Value Assessment Methodology, Information Management Office, 2004.

The methodology presented in this document represents the culmination of over a year’s effort designing and refining a standardized system to forecast and articulate demand and value in any proposed e-government service.

Arveson, P. What is the Balanced Scorecard?, Balanced Scorecard Institute, 1998, http://www.balancedscorecard.org/basics/bsc1.html

This webpage provides an overview of Robert S. Kaplan and David Norton’s balanced scorecard management system, a strategic management approach that was developed in the early 1990s.  The system suggests viewing an organization from four perspectives (learning and growth, business process, customer, and financial) and developing metrics and collecting data based on these perspectives.

Binnendijk, A., Results Based Management in the Development Co-operation Agencies: A Review of Experience, DAC Working Party on Aid Evaluation, 2000, http://www.oecd.org/dataoecd/secure/14/29/31950852.pdf

This paper is based on a document review of the experiences and practices of selected OECD Member development co-operation agencies with establishing performance or results based management systems. Covered in the review are the experiences of seven donor agencies establishing and implementing their results based management systems, comparing similarities and contrasting differences in approach.

Canadian Transportation Agency, Performance Measurement Framework, 2004, http://www.cta-otc.gc.ca/about-nous/excellence/performance/performance-eng.pdf

This framework’s purpose is to provide a consistent approach for systematically collecting, analyzing, utilizing, and reporting on the performance of the Canadian Transportation Agency’s programs and activities. This document presents an overview of the framework, as well as performance measurement principles, the program management process, and key steps for measuring performance.

Covey, S., The Seven Habits of Highly Effective People: Restoring the Character Ethic, New York, NY: Simon and Schuster, 1989.

This book contains an integrated approach to solving personal and professional problems by learning principles rather than merely practices. The Seven Habits are a step-by-step pathway to the principles of fairness, integrity, honesty, and human dignity that give security to adapt to change in family and business lives.

Government of Canada, Privy Council Office, Regulatory Affairs and Orders in Council Secretariat: Glossary, 2001, http://www.pco-bcp.gc.ca/raoics-srdc/default.asp?Language=EandPage=glossary

This glossary provides unofficial definitions of Privy Council Office (PCO) terms in order to facilitate users’ understanding of PCO documents and information.

Government of South Australia, Triple Bottom Line, Department for Environment and Heritage, 2005, http://www.environment.sa.gov.au/sustainability/triple_bottom_line.html

This webpage provides an overview of Triple Bottom Line (TBL) reporting and its benefits, as well as measures taken by the Government of South Australia and departmental agencies in implementing the practice.

Kaplan, R. and D. Norton, The Balanced Scorecard: Translating Strategy into Action, Boston, MA: Harvard Business School Press, 1996.

The Balanced Scorecard is a management system designed to channel abilities, energy, and knowledge toward achieving long-term strategic goals. Encompassing current and future performance, Kaplan and Norton’s method can be used in four categories to meet organizational objectives: financial performance, customer knowledge, internal business processes, and learning and growth.

Lau, E. E-Government and the Drive for Growth and Equity, Organisation for Economic Cooperation and Development, E-Government Project, 2005, http://www.dsg.ae/en/ds_images/conf/05_lau.pdf

This paper presents three broad types of benefits related to e-government: financial, public, and economic, with emphasis on the latter two, arguing that they are the least well understood. The proposed outline of benefits seeks to allow governments to support investment decisions and evaluate results.

Office of the Auditor General of Canada, December 2000 Report of the Auditor General of Canada, 2000, http://www.oag-bvg.gc.ca/domino/reports.nsf/html/0019xe01.html

The December 2000 volume of the Office of the Auditor General (OAG) report contains 18 chapters of audits of governmental departments and progress of reporting performance to Parliament. The chapter on reporting performance contains a results chain diagram that was adapted into section 3.1 of this report.

Office of the Auditor General of Canada, Implementing Results-Based Management: Lessons from the Literature, 2000, http://www.oag-bvg.gc.ca/domino/other.nsf/html/00rbm-eng.html

A follow-up to a review prepared by the Office of the Auditor General in 1996, this report is a concise synthesis of lessons learned from implementing results-based management in a variety of Canadian and international jurisdictions. The first review, summarized in Annex A, focused on implementation, while this update also includes lessons learned on more operational issues such as development of indicators, data collection, analysis, monitoring, and reporting.

Organisation for Economic Co-operation and Development, Evaluation and Aid Effectiveness: Glossary of Key Terms in Evaluation and Results Based Management, DAC Working Party on Aid Evaluation, Development Assistance Committee, 2002, http://www.oecd.org/dataoecd/29/21/2754804.pdf

This document contains a glossary of terms relating to quality assurance, stakeholders, logical framework, results-based management, evaluation tools, and types of evaluations.

Organisation for Economic Co-operation and Development, OECD E-Government Project – Costs and Benefits of E-Government: Identifying Public Benefits, Public Governance and Territorial Development Directorate, Public Governance Committee, 2005.

This report is designed as a scoping paper that looks at different dimensions of the public benefits of e-government, and makes tentative suggestions as to how these might be measured.  It provides answers relating to the non-financial public benefits governments can expect from e-government, how these softer indicators can be identified, categorized, and measured, as well as the proxy measures that exist for providing an estimate of public benefits.

Plantz, M. et al., Outcome Measurement: Showing Results in the Nonprofit Sector, United Way of America, 1997, http://national.unitedway.org/outcomes/resources/What/ndpaper.cfm

This article describes the activities of non-profit agencies in relation to Outcome Management initiatives, discussing 30 lessons learned and seven key challenges to be overcome. It also summarizes the history of performance measurement in the non-profit health and human services sector and defines the key concepts of outcome measurement.

Schacter, M., Results-based Management at the Water Cooler: Perspectives from the working level on RBM, Mark Schacter Consulting, 2004.

Based on opinion data gathered from approximately 100 public servants at a series of results-based management (RBM) workshops, this paper provides a window into perceptions of working-level officials about the implementation of RBM in Canada’s public service. Analysis of the data suggests that public-sector staff have six types of concerns about RBM implementation, two of which are predominant: high-level leadership for RBM and technical capacity to implement RBM.

Treasury Board of Canada Secretariat, Business Transformation Enablement Program – Strategic Design and Planning Methodology, 2004, /btep-pto/documents/2004/method/method-eng.pdf

This document describes the first release of the BTEP Design and Planning Methodology, which is the overall process methodology for business transformation. It is intended to be used by business transformation teams responsible for producing transformation project deliverables. The goal of BTEP is to enable coherent business design across the government with a formal, standards-based approach that will guide and expedite business transformation to meet the government’s high-level business objectives.

Treasury Board of Canada Secretariat, Changing Management Culture: Models and Strategies to Make It Happen, 2003, http://www.tbs-sct.gc.ca/cmo_mfc/Toolkit2/GCC/cmc-eng.pdf

This guide presents a step-by-step approach to managing change, one that deputy ministers, heads of agencies, and their executive teams can follow when undertaking management reforms.  For illustration purposes, the guide focuses on Modern Comptrollership, but it is generic in nature and its approach can be applied to any effort to change management culture.

Treasury Board of Canada Secretariat, Companion Guide: The Development of Results-based Management and Accountability Frameworks for Horizontal Initiatives, 2002, http://www.tbs-sct.gc.ca/cee/tools-outils/comp-acc00-eng.asp

This guide was developed to complement the Guide for the Development of Results-based Management and Accountability Frameworks and provide federal managers with practical advice on how to develop effective RMAFs for horizontal initiatives. It addresses the challenges of building an effective team that will draft the RMAF, covers the five main components of an RMAF, and provides a list of additional lessons learned and reference documents.

Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects, Project Management Office, Financial and Information Management Branch, 1996, http://www.tbs-sct.gc.ca/emf-cag/abu-ans/itp-pti/itp-pti00-eng.asp

This paper describes a proposed enhanced framework for the management of information technology projects in the federal government. This enhanced framework is designed to ensure that government information technology projects fully meet the needs of the business functions they are intended to support, deliver all expected benefits, and are completed within their approved time, cost, and functionality.

Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects: Creating and Using a Business Case for Information Technology Projects, Project Management Office, Chief Information Officer Branch, 1998, http://www.tbs-sct.gc.ca/emf-cag/business-rentabilisation/business-rentabilisation-eng.asp

This document was designed to ensure that federal government IT projects fully meet the needs of the business functions they are intended to support and deliver all expected benefits, and are completed on time and within budget. Moreover, it identifies the need for a business case analysis before a government IT investment can be approved. The guide can be used as a planning tool for users to mark and monitor the factors that are crucial to implementing IT successfully.

Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects Part II – Solutions: Putting the Principles to Work, Chief Information Officer Branch, 1998, http://www.tbs-sct.gc.ca/emf-cag/abu-ans/ppw-slp/ppw-slp00-eng.asp

This document is a companion to Part I, which was approved and published in May 1996.  The purpose of the document is to facilitate implementation of the Enhanced Framework within federal government departments by providing an overview of the Enhanced Framework, identifying where and how to begin the process of implementation, outlining solutions to assist departments in applying the Framework, describing the roles and responsibilities of the key departmental players in project delivery, and providing guidance on how to get started.

Treasury Board of Canada Secretariat, Enterprise Value Management Outcome Management Practice Implementation Strategy, CIO Branch, Alignment and Stewardship, 2005.

This document outlines the recommended strategy for the establishment of an Outcome Management (Outcome Management) Practice within the Alignment and Stewardship Division of the CIO Branch, Treasury Board Secretariat. The key objective of the Outcome Management Practice is to help the Alignment and Stewardship Division further evolve in its role and to support improved decision-making to ensure that the best value for the government enterprise is paramount in all choices made for Canada’s portfolio of initiatives.

Treasury Board of Canada Secretariat, Guide to Realizing Outcomes from Government of Canada Programs (Draft), 2004.

This guide presents the outcomes realization process, which is described as the set of activities for planning, managing, and realizing desired outcomes from initiatives. Through value management, the guide provides a framework involving tools and techniques to proactively plan, manage, and monitor the realization of the outcomes of a change initiative.

Treasury Board of Canada Secretariat, Integrated Measurement Framework: Concept Paper (Draft), Chief Information Officer Branch, 2005.

This document presents ideas towards the development of an Integrated Measurement Framework (IMF) for the Chief Information Officer Branch (CIOB). It presents the IMF vision, initial strategies, the initial IMF design, and the next steps in the IMF development.

Treasury Board of Canada Secretariat, Management Accountability Framework, President of the Treasury Board, 2003, www.tbs-sct.gc.ca/maf-crg/index-eng.asp

This document was developed to provide deputy heads and all public service managers with a list of management expectations that reflect the different elements of current management responsibilities. It is intended to translate the vision of modern public service management into a set of management expectations. The Framework focuses on management results rather than required capabilities, provides a basis of engagement with departments, and suggests ways for departments both to move forward and to measure progress.

Treasury Board of Canada Secretariat, Outcomes Management Realization: Service Canada Policy – Outcomes Sought, prepared for Service Canada, 2005.

This overview consists of an outline of the outcomes sought by Service Canada policy, as well as a summary of the four stages involved in outcomes management realization.

Treasury Board of Canada Secretariat, Privacy Impact Assessment (PIA) E-Learning Tool, 2003, http://www.tbs-sct.gc.ca/pgol-pged/piatp-pfefvp/index-eng.asp

The Privacy Impact Assessment (PIA) e-learning tool is an introductory course that reviews the basic principles of privacy in Canada discusses the fundamentals of the PIA process. The tool covers key privacy definitions, Canadian privacy legislation and policy, the main features and benefits of PIAs, and the key stakeholders involved in PIAs.

Treasury Board of Canada Secretariat, Privacy Impact Assessment Guidelines: A Framework to Manage Privacy Risks, 2002, http://www.tbs-sct.gc.ca/pubs_pol/ciopubs/pia-pefr/paipg-pefrld-eng.asp

This document contains guidelines that are intended to provide a comprehensive framework for the completion of a Privacy Impact Assessment (PIA). The PIA ensures that privacy principles and legislation are considered and adhered to throughout the lifecycle of a new program, service or initiative and where appropriate, for existing initiatives undergoing service transformation or redesign.

Treasury Board of Canada Secretariat, Results-based Management and Accountability Framework of the Modern Comptrollership Initiative, 2003.

This Results-based Management and Accountability Framework (RMAF) for the Modern Comptrollership Initiative (MCI) has been developed to provide managers in departments, agencies, and at the Treasury Board Secretariat (TBS) with a single, comprehensive, and reliable instrument to evaluate and report on the performance of this major learning and culture-changing initiative for the Government of Canada. This document contains a profile of the MCI, guidance for ongoing performance measurement, as well as evaluation and reporting strategies.

Treasury Board of Canada Secretariat, Results-Based Management and Accountability Framework of the Modern Comptrollership Initiative, 2003, http://www.tbs-sct.gc.ca/cmo_mfc/resources2/RMAF/RMAF.pdf

This document contains a profile of the Modern Comptrollership Initiative (MCI), guidance for ongoing performance measurement, in addition to evaluation and reporting strategies. This Results-Based Management and Accountability Framework (RMAF) was developed as a tool for managers in departments, agencies, and at the Treasury Board Secretariat (TBS) to help in measuring and reporting on results being achieved through the MCI.

Treasury Board of Canada Secretariat, Results-Based Management in Canada: Country Report Prepared for the OECD Outcome-Focused Management Project, Planning, Performance and Reporting Sector, Comptrollership Branch, 2000, http://www.ppx.ca/NewsArchives/PDF/Result_Based_Management.pdf

The objective of this report is to describe how outcome goals are defined and used, and how progress towards them is measured in the Government of Canada. In the report, there sections relating to how departments and agencies integrate results-based management in policy formulation and implementation, concrete examples to illustrate aspects of results-based management, a comparison of the working terminology of the OECD with that of Canada, and questions on results-based management for the annual OECD Survey of Budgeting Development.

Trochim, W. Qualitative Validity, The Research Methods Knowledge Base, 2005, http://www.socialresearchmethods.net/kb/qualval.htm

This webpage provides an overview of four proposed criteria for judging qualitative validity, which are: credibility, transferability, dependability, and confirmability. These four criteria, proposed by Guba and Lincoln, are intended to be analogous to the traditional quantitative criteria, which are: internal validity, external validity, reliability, and objectivity.

U.S. Government, Balancing Measures: Best Practices in Performance Management, National Partnership for Reinventing Government, 1999, http://govinfo.library.unt.edu/npr/library/papers/bkgrd/balmeasure.html

This report represents an extensive undertaking to survey and interview agencies and companies for practices that contribute to improving service as well as business results. The findings show that the process followed has not been exactly the same in every instance. Balancing business results with customer, stakeholder, and employee information generally produces marked improvement in performance, service, and overall satisfaction. This study partners report gains in efficiency, data tied to strategic goals and measurement systems, and improved relationships with employees and customers.

U.S. Government, Rating the Performance of Federal Programs, Office of Management and Budget, 2004, http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf

This document gives background information on the Program Assessment Rating Tool (PART) of the U.S. federal government. The PART is a systematic method of assessing the performance of program activities across the U.S. government. As a diagnostic tool, the main of objective of the PART review is to improve program performance. The PART assessments help link performance to budget decisions and provide a basis for making recommendations to improve results.

U.S. Government, Serving the American Public: Best Practices in Performance Measurement, National Performance Review, 1997, http://govinfo.library.unt.edu/npr/library/papers/benchmrk/nprbook.html

This report documents the Performance Measurement Study Team’s findings, which are to be used as a tool for public and private leaders and managers in identifying and applying best-in-class performance measurement and performance management practices. This intergovernmental benchmarking study identifies the processes, skills, technologies, and best practices that can be used by government to link strategic planning with performance planning and measurement by establishing and updating performance measures, establishing accountability for performance, gathering and analyzing performance data, and reporting and using performance information.


Return to footnote reference [1] “Outcome,” Merriam-Webster Online Dictionary, 2005, http://m-w.com/dictionary/outcome

Return to footnote reference [2]  Adapted from the Office of the Auditor General of Canada, December 2000 Report of the Auditor General of Canada, 2000, http://www.oag-bvg.gc.ca/domino/reports.nsf/html/0019xe01.html 

Return to footnote reference [3] Arveson, P. What is the Balanced Scorecard?, Balanced Scorecard Institute, 1998, http://www.balancedscorecard.org/basics/bsc1.html 

Return to footnote reference [4] Trochim, W. Qualitative Validity, The Research Methods Knowledge Base, 2005, http://www.socialresearchmethods.net/kb/qualval.htm 

Return to footnote reference [5] Treasury Board of Canada Secretariat, Guide to Realizing Outcomes from Government of Canada Programs (Draft), 2004.

Return to footnote reference [6] OECD, Evaluation and Aid Effectiveness: Glossary of Key Terms in Evaluation and Results Based Management, DAC Working Party on Aid Evaluation, Development Assistance Committee, 2002, http://www.oecd.org/dataoecd/29/21/2754804.pdf 

Return to footnote reference [7]Government of Canada Privy Council Office, Regulatory Affairs and Orders in Council Secretariat: Glossary, 2001, http://www.pco-bcp.gc.ca/raoics-srdc/default.asp?Language=EandPage=glossary