Treasury Board of Canada Secretariat
Symbol of the Government of Canada

Outcome Management: Lessons Learned and Best Practices


Appendix D:  Annotated Bibliography

* indicates a Key reference

Agence pour le Développement de l’Administration Électronique, Guide méthodologique MAREVA : Analyse de la valeur des projets d’ADELE, Ministere du Budget et de la reforme de l’Etat, 2005, http://www.adae.gouv.fr/IMG/pdf/050405_MAREVA_GuideMethodologique_vf.pdf

This document covers the e-government program ADELE from France, which sets out the government’s online strategy for the period of 2004-2007.  To evaluate gains and savings for each e-government project, the French government developed the evaluation methodology called MAREVA, which enables a precise evaluation of financial gains of e-government services for the State and the public sector, as well as of gains and benefits for end users.

Australian Government, Demand and Value Assessment Methodology, Information Management Office, 2004.

The methodology presented in this document represents the culmination of over a year’s effort designing and refining a standardized system to forecast and articulate demand and value in any proposed e-government service.

Benko, C. and F.W. McFarlan, Connecting the Dots: Aligning Projects with Objectives in Unpredictable Times, Boston, MA: Harvard Business School Press, 2003.

In this book the authors argue that an organization’s project portfolio is its single most important asset for delivering on strategic and operational objectives.  To harness portfolios into an efficient, coherent whole, the authors propose a goal of alignment: better matching a company’s portfolios with its objectives, since better aligned companies achieve greater investment returns.

* Binnendijk, A., Results Based Management in the Development Co-operation Agencies: A Review of Experience, DAC Working Party on Aid Evaluation, 2000, http://www.oecd.org/dataoecd/secure/14/29/31950852.pdf 

This paper is based on a document review of the experiences and practices of selected OECD Member development co-operation agencies with establishing performance or results based management systems.  Covered in the review are the experiences of seven donor agencies establishing and implementing their results based management systems, comparing similarities and contrasting differences in approach.

Bryson, J. et al., Visible Thinking: Unlocking causal mapping for practical business results, Chichester, West Sussex: John Wiley & Sons Ltd, 2004.

This book shows how to create causal maps for individuals and groups to capture the power and broad applicability of causal mapping in a business, as well as a personal, context.  The process is illustrated through a series of real cases – from tackling personal problems to strategy-change issues in business, public, and not-for-profit organizations.  The cases are used to present a comprehensive set of process guidelines designed to help you create your own action-oriented causal maps.

Canadian International Development Agency, RBM Handbook on Developing Results Chains: The Basics of RBM as Applied to 100 Project Examples, Results-Based Management Division, 2000, http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Performancereview6/$file/Full_report.pdf

This handbook aims to provide the basic concepts behind Results-Based Management (RBM) supported with 100 examples to better articulate what projects, programs or organizations hope to achieve.  The purpose of the guide is to promote a better understanding of the key concepts of RBM illuminated with examples, graphics, and tools that help readers build their confidence with regard to RBM.

Canadian Transportation Agency, Performance Measurement Framework, 2004, http://www.cta-otc.gc.ca/about-nous/excellence/performance/performance-eng.pdf

This framework’s purpose is to provide a consistent approach for systematically collecting, analyzing, utilizing, and reporting on the performance of the Canadian Transportation Agency’s programs and activities.  This document presents an overview of the framework, as well as performance measurement principles, the program management process, and key steps for measuring performance.

Government of Canada, Canada’s Performance: Annual Report to Parliament, Treasury Board of Canada Secretariat, 2004.

The purpose of this report is to provide parliamentarians and Canadians with a whole of government perspective from which to view the plans, results, and resources reported by individual federal departments and agencies in their spring planning and fall performance reports.  The report is divided into six themes: (1) Canada’s Place in the World; (2) Canada’s Economy; (3) Society, Culture and Democracy; (4) Aboriginal Peoples; (5) The Health of Canadians; and (6) The Canadian Environment.

Government of Canada Privy Council Office, Regulatory Affairs & Orders in Council Secretariat: Glossary, 2001, http://www.tbs-sct.gc.ca/ri-qr/index-eng.asp 

This glossary provides unofficial definitions of Privacy Council Office (PCO) terms in order to facilitate users’ understanding of PCO documents and information.

Government of South Australia, Triple Bottom Line, Department for Environment and Heritage, 2005, http://www.environment.sa.gov.au/sustainability/triple_bottom_line.html

This webpage provides an overview of Triple Bottom Line (TBL) reporting and its benefits, as well as measures taken by the Government of South Australia and departmental agencies in implementing the practice.

Industry, Science and Technology Canada, The Competitive Enterprise: An Executive’s Guide to Investing in Advanced Manufacturing and Processing Technology, 1991.

This guide focuses on Advanced Manufacturing and processing Technology (AMT), a key means by which Canadian manufacturers and processors can improve their performance.  This guide provides an inexpensive, comprehensive, step-by-step method to assist in identifying, evaluating, and implementing AMT.

Kaplan, R. and D. Norton, The Balanced Scorecard: Translating Strategy into Action, Boston, MA: Harvard Business School Press, 1996.

The Balanced Scorecard is a management system designed to channel abilities, energy, and knowledge toward achieving long-term strategic goals.  Encompassing current and future performance, Kaplan and Norton’s method can be used in four categories to meet organizational objectives: financial performance, customer knowledge, internal business processes, and learning and growth.

Kaplan, R. and D. Norton, Strategy Maps: Converting Intangible Assets into Tangible Outcomes, Boston, MA: Harvard Business School Press, 2004.

This book is a blueprint for describing, measuring, and aligning intangible assets for superior performance, providing the missing link between strategy formulation and implementation.  Based on their work with more than 300 organizations spanning over a dozen years, the authors have created a new tool, strategy maps, that allow organizations to clarify and communicate their strategies to all employees, identify the key internal processes that drive strategic success, align investments in people and organizational capital for the greatest impact, and expose gaps in the strategies and take early corrective action.

Lebow, R. and R. Spitzer, Accountability: Freedom and Responsibility without Control, San Francisco, CA: Berrett-Koehler Publishers, Inc., 2002.

This book looks at the past fifty years of control-based initiatives that underachieved and over-promised.  It challenges conventional beliefs about the true value TQM, MBO, incentive programs, personal improvement plans, quotas, systems thinking, performance reviews, and job descriptions have contributed to the organizational landscape.  It provides practical guidelines for transforming control-based operations into freedom-based work environments where managers take on the new role of Wise Counsels and employees design and fully own their jobs.

Maizlish, B. and R. Handler, IT Portfolio Management Step-By-Step: Unlocking the Business Value of Technology, Hoboken, NJ: John Wiley & Sons, Inc., 2005.

This book presents an approach to simplifying the process of achieving a rationalized and business-aligned IT portfolio.  It includes extensive coverage of the five-level IT portfolio maturity model, policies, principles, organizational roles, the IT life cycle, detailed stages of building the IT portfolio, advanced IT portfolio management topics, and the impact of future trends and technologies on IT portfolio management.

Millard, J., Best eEurope Practices: Final Project Report, European Communities, 2003, http://www.beepknowledgesystem.org/File/Beep_Final_Report.pdf

Beep (Best eEurope Practices) is a technical tool for identifying, capturing, evaluating, and learning from good practice in any field of interest.  This report provides an overview of the Beep objectives, methodology, and results.  Special focus is given to the main issues and policy and research recommendations arising out of an in-depth analysis of many of the over 270 cases in the Beep knowledge system.

* Office of the Auditor General of Canada, Implementing Results-Based Management: Lessons from the Literature, 2000, http://www.oag-bvg.gc.ca/internet/English/meth_gde_e_10225.html 

A follow-up to a review prepared by the Office of the Auditor General in 1996, this report is a concise synthesis of lessons learned from implementing results-based management in a variety of Canadian and international jurisdictions.  The first review, summarized in Annex A, focused on implementation, while this update also includes lessons learned on more operational issues such as development of indicators, data collection, analysis, monitoring, and reporting.

Organisation for Economic Co-operation and Development, Evaluation and Aid Effectiveness: Glossary of Key Terms in Evaluation and Results Based Management, DAC Working Party on Aid Evaluation, Development Assistance Committee, 2002, http://www.oecd.org/dataoecd/29/21/2754804.pdf

This document contains a glossary of terms relating to quality assurance, stakeholders, logical framework, results-based management, evaluation tools, and types of evaluations.

Plantz, M. et al., Outcome Measurement: Showing Results in the Nonprofit Sector, United Way of America, 1997, http://national.unitedway.org/outcomes/resources/What/ndpaper.cfm

This article describes the activities of non-profit agencies in relation to outcome management initiatives, discussing 30 lessons learned and seven key challenges to be overcome.  It also summarizes the history of performance measurement in the non-profit health and human services sector and defines the key concepts of outcome measurement.

Schacter, M., Results-based Management at the Water Cooler: Perspectives from the working level on RBM, Mark Schacter Consulting, 2004.

Based on opinion data gathered from approximately 100 public servants at a series of results-based management (RBM) workshops, this paper provides a window into perceptions of working-level officials about the implementation of RBM in Canada’s public service.  Analysis of the data suggests that public-sector staff have six types of concerns about RBM implementation, two of which are predominant: high-level leadership for RBM and technical capacity to implement RBM.

Slevin, D. et al., Critical success factor analysis for information systems performance measurement and enhancement: A case study in the university environment, Elsevier Science Publishers, 1991.

Described in this study is the application of the Critical Success Factor (CSF) process as a performance measurement and enhancement device.  Measurement issues are discussed, along with the establishment of performance standards.  Case examples of actual behaviour change (performance improvement) due to the use of CSFs are given.  Finally, guidelines for the use of the CSF procedure in other Information Systems (IS) organizational contexts are discussed and prescriptive recommendations are presented.

Treasury Board of Canada Secretariat, Business Transformation Enablement Program – Strategic Design & Planning Methodology, 2004, http://www.tbs-sct.gc.ca/btep-pto/index_e.asp

This document describes the first release of the BTEP Design and Planning Methodology, which is the overall process methodology for business transformation.  It is intended to be used by business transformation teams responsible for producing transformation project deliverables.  The goal of BTEP is to enable coherent business design across the government with a formal, standards-based approach that will guide and expedite business transformation to meet the government’s high-level business objectives.

Treasury Board of Canada Secretariat, Changing Management Culture: Models and Strategies to Make It Happen, 2003, www.tbs-sct.gc.ca/cmo_mfc/Toolkit2/GCC/cmc-eng.pdf

This guide presents a step-by-step approach to managing change, one that deputy ministers, heads of agencies, and their executive teams can follow when undertaking management reforms.  For illustration purposes, the guide focuses on Modern Comptrollership, but it is generic in nature and its approach can be applied to any effort to change management culture.

Treasury Board of Canada Secretariat, Companion Guide: The Development of Results-based Management and Accountability Frameworks for Horizontal Initiatives, 2002, www.tbs-sct.gc.ca/eval/tools_outils/comp-guide-eng.pdf 

This guide was developed to complement the Guide for the Development of Results-based Management and Accountability Frameworks and provide federal managers with practical advice on how to develop effective RMAFs for horizontal initiatives.  It addresses the challenges of building an effective team that will draft the RMAF, covers the five main components of an RMAF, and provides a list of additional lessons learned and reference documents.

* Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects, Project Management Office, Financial and Information Management Branch, 1996, www.tbs-sct.gc.ca/emf-cag/abu-ans/ppw-slp/ppw-slp00-eng.asp 

This paper describes a proposed enhanced framework for the management of information technology projects in the federal government.  This enhanced framework is designed to ensure that government information technology projects fully meet the needs of the business functions they are intended to support, deliver all expected benefits, and are completed within their approved time, cost, and functionality.

* Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects: Creating and Using a Business Case for Information Technology Projects, Project Management Office, Chief Information Officer Branch, 1998, www.tbs-sct.gc.ca/emf-cag/business-rentabilisation/business-rentabilisation-eng.asp

This document was designed to ensure that federal government IT projects fully meet the needs of the business functions they are intended to support and deliver all expected benefits, and are completed on time and within budget.  Moreover, it identifies the need for a business case analysis before a government IT investment can be approved.  The guide can be used as a planning tool for users to mark and monitor the factors that are crucial to implementing IT successfully.

* Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects Part II – Solutions: Putting the Principles to Work, Chief Information Officer Branch, 1998, www.tbs-sct.gc.ca/emf-cag/abu-ans/ppw-slp/ppw-slp00-eng.asp

This document is a companion to Part I, which was approved and published in May 1996.  the purpose of the document is to facilitate implementation of the Enhanced Framework within federal government departments by providing an overview of the Enhanced Framework, identifying where and how to begin the process of implementation, outlining solutions to assist departments in applying the Framework, describing the roles and responsibilities of the key departmental players in project delivery, and providing guidance on how to get started.

Treasury Board of Canada Secretariat, Enterprise Value Management Outcome Management Practice Implementation Strategy, CIO Branch, Alignment and Stewardship, 2005.

This document outlines the recommended strategy for the establishment of an Outcome Management (Outcome Management) Practice within the Alignment and Stewardship Division of the CIO Branch, Treasury Board Secretariat.  The key objective of the Outcome Management Practice is to help the Alignment and Stewardship Division further evolve in its role and to support improved decision-making to ensure that the best value for the government enterprise is paramount in all choices made for Canada’s portfolio of initiatives.

* Treasury Board of Canada Secretariat, Guide to Realizing Outcomes from Government of Canada Programs (Draft), 2004.

This guide presents the outcomes realization process, which is described as the set of activities for planning, managing, and realizing desired outcomes from initiatives.  Through value management, the guide provides a framework involving tools and techniques to proactively plan, manage, and monitor the realization of the outcomes of a change initiative.

Treasury Board of Canada Secretariat, Integrated Measurement Framework: Concept Paper (Draft), Chief Information Officer Branch, 2005.

This document presents ideas towards the development of an Integrated Measurement Framework (IMF) for the Chief Information Officer Branch (CIOB).  It presents the IMF vision, initial strategies, the initial IMF design, and the next steps in the IMF development.

Treasury Board of Canada Secretariat, Management Accountability Framework, President of the Treasury Board, 2003, www.tbs-sct.gc.ca/maf-crg/index-eng.asp 

This document was developed to provide deputy heads and all public service managers with a list of management expectations that reflect the different elements of current management responsibilities.  It is intended to translate the vision of modern public service management into a set of management expectations.  The Framework focuses on management results rather than required capabilities, provides a basis of engagement with departments, and suggests ways for departments both to move forward and to measure progress.

Treasury Board of Canada Secretariat, Outcomes Management Realization: Service Canada Policy – Outcomes Sought, prepared for Service Canada, 2005.

This overview consists of an outline of the outcomes sought by Service Canada policy, as well as a summary of the four stages involved in outcomes management realization.

Treasury Board of Canada Secretariat, Results-Based Management and Accountability Framework of the Modern Comptrollership Initiative, 2003, www.tbs-sct.gc.ca/cmo_mfc/resources2/RMAF/RMAF.pdf 

This document contains a profile of the Modern Comptrollership Initiative (MCI), guidance for ongoing performance measurement, in addition to evaluation and reporting strategies.  This Results-Based Management and Accountability Framework (RMAF) was developed as a tool for managers in departments, agencies, and at the Treasury Board Secretariat (TBS) to help in measuring and reporting on results being achieved through the MCI.

Treasury Board of Canada Secretariat, Results-Based Management in Canada: Country Report Prepared for the OECD Outcome-Focused Management Project, Planning, Performance and Reporting Sector, Comptrollership Branch, 2000, http://www.ppx.ca/NewsArchives/PDF/Result_Based_Management.pdf

The objective of this report is to describe how outcome goals are defined and used, and how progress towards them is measured in the Government of Canada.  In the report, there sections relating to how departments and agencies integrate results-based management in policy formulation and implementation, concrete examples to illustrate aspects of results-based management, a comparison of the working terminology of the OECD with that of Canada, and questions on results-based management for the annual OECD Survey of Budgeting Development.

* U.S. Government, Balancing Measures: Best Practices in Performance Management, National Partnership for Reinventing Government, 1999, http://govinfo.library.unt.edu/npr/library/papers/bkgrd/balmeasure.html

This report represents an extensive undertaking to survey and interview agencies and companies for practices that contribute to improving service as well as business results.  The findings show that the process followed has not been exactly the same in every instance.  Balancing business results with customer, stakeholder, and employee information generally produces marked improvement in performance, service, and overall satisfaction.  This study partners report gains in efficiency, data tied to strategic goals and measurement systems, and improved relationships with employees and customers.

U.S. Government, Rating the Performance of Federal Programs, Office of Management and Budget, 2004, http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf

This document gives background information on the Program Assessment Rating Tool (PART) of the U.S. federal government.  The PART is a systematic method of assessing the performance of program activities across the U.S. government.  As a diagnostic tool, the main of objective of the PART review is to improve program performance.  The PART assessments help link performance to budget decisions and provide a basis for making recommendations to improve results.

* U.S. Government, Serving the American Public: Best Practices in Performance Measurement, National Performance Review, 1997, http://govinfo.library.unt.edu/npr/library/papers/benchmrk/nprbook.html 

This report documents the Performance Measurement Study Team’s findings, which are to be used as a tool for public and private leaders and managers in identifying and applying best-in-class performance measurement and performance management practices.  This intergovernmental benchmarking study identifies the processes, skills, technologies, and best practices that can be used by government to link strategic planning with performance planning and measurement by establishing and updating performance measures, establishing accountability for performance, gathering and analyzing performance data, and reporting and using performance information.