Five-Year Evaluation of the Management Accountability Framework

• Previous page
• Table of contents
• Complete text
• Next page

Final Report

Executive Summary

Performance management frameworks are integral to the success of organizations within the public and private sectors.  This is especially true for complex organizations such as the Federal Government of Canada. 

In November 2008, the Treasury Board Secretariat (TBS) commissioned PricewaterhouseCoopers LLP and Interis Consulting Inc. to conduct a five-year evaluation of the Management Accountability Framework (MAF), TBS’ performance management framework.  As part of this evaluation, we considered how TBS should continue its evolution of the MAF. 

Our evaluation approach involved interviews, consultations, literature review, international comparison, a costing survey and cost analysis.  We have been greatly assisted by feedback provided by various advisory groups, a DM Steering Committee and discussions with the MAF Directorate within TBS.  We note that there were some limitations with respect to the evidence – primarily around the lack of robust costing data as departments and agencies are not tracking costs related to the MAF assessment and empirical evidence with respect the improvement of management practices due to the evolving nature of MAF.

Our evaluation compared MAF against three comparable frameworks in the following jurisdictions:  the United Kingdom, the European Union and the United States.  The frameworks in place in these jurisdictions are similar in intent (performance improvement) and methodology (regular diagnosis of organizational capability) to the MAF.  Key elements of these frameworks that may be leveraged to further enhance the usefulness and sustainability of MAF are as follows.

• United Kingdom (Capability Review)
  • This framework includes additional information gathering techniques such as interviews and workshops, as a means to gather evidence to support the lines of evidence.
  • The framework involves increased engagement between the central organization and departments, at the senior levels.
• European Union (Capability Assessment Framework)
  • The framework includes a tailored/flexible assessment approach for each organization or category of organization being assessed.
• United States (President’s Management Agenda)
  • This approach includes an assessment of an organization’s progress towards improvement plans, in addition to the current state assessment.
  • An assembly of a deputy head oversight committee is in place to provide input and support to the development of assessment criteria and horizontal issues

A summary of our findings from the multiple lines of evidence is as follows:

1. Through formalizing expectations of management, MAF has led to increased focus on management practices, i.e., management matters.  MAF is becoming a catalyst for integrating best practices into departments and agencies.
2. There continues to be a place for ongoing dialogue between TBS and deputy heads during the MAF process.
3. TBS has in place a structured and rigorous process for reviewing the MAF assessment results.
4. There has been increasing stability of the MAF indicators in recent rounds
5. The current approach of assessing each AoM annually for large departments and agencies does not consider the unique risks and priorities of organizations.
6. While the reporting burden associated with MAF has been reduced, there are further opportunities to reduce the impact of the MAF assessment on departments and agencies.
7. The subjectivity of the MAF assessments is a result of a significant number of qualitative indicators built into the assessment approach.
8. Many of the lines of evidence from Round VI are process-based, which do not measure effectiveness of the outcomes resulting from the process.
9. Our consultation indicated that most departments and agencies wee unsure of the cost effectiveness of MAF.
10. TBS is the appropriate entity to measure managerial performance with in the Federal Government.

Based on our analysis of the evidence arising from the multiple lines of enquiry, we have concluded that MAF is successful and relevant.  MAF is meeting its objectives and it should continue to be maintained and supported.  MAF provides a comprehensive view to both deputy heads and TBS on the state of managerial performance within a department or agency.  In support of prioritization and focus on management practices across the Federal government, MAF is a valuable tool.

Due to the limitations of the costing data, as noted above, we are unable to conclude on the cost effectiveness of MAF.  Going forward, TBS may want to consider providing guidance to departments and agencies to allow the costs of the MAF process to be tracked.

Based on our analysis of the inputs and comparison to comparable jurisdictions, our recommendations for the continued evolution of MAF as a performance enabler are as follows.

1. Implement a risk/priority based approach to the MAF assessment process; possible options include:
   • Assessment based on the risks/priorities unique to each organization;
   • Clustering of the MAF indicators subject to assessment into categories:  mandatory, optional, and cyclical;
   • Assessment based on the results of previous rounds; and
   • Assessment based on department/agency size.
2. Develop guiding principles (“golden rules”) for assessing managerial performance and incorporate into the existing MAF assessment methodology, including:
   • Maintain an appropriate balance between quantitative and qualitative indicators within each Area of Management (AoM);
   • Develop outcome-based indicators of managerial performance;
   • Leverage information available through existing oversight activities;
   • Maintain the stability of indicators, which is critical to the assessment of progress over time;
   • Ensure clarity and transparency of indicators, measurement criteria and guidance documentation;
   • Engage functional communities in open dialogue in the ongoing development of AoM methodology and assessment measures;
   • Include assessment and identification of both policy compliance and result-based managerial performance; and
   • Seek ways to recognize and provide incentives to encourage innovation.
3. Introduce a governance body with senior representatives from departments and agencies to guide MAF.  This deputy head or departmental senior executive committee could be used to advise on changes to the MAF process and the MAF system-wide results to guide government management priorities.
4. Develop a stakeholder communication/engagement strategy and plan, including early engagement when changes are made to the MAF assessment process.  TBS could leverage existing communities and forums across the Federal government to ensure timely communication of changes, which will allow stakeholders sufficient time to respond and accept any changes.
5. Assign formal responsibilities within TBS to oversee the MAF assessment methodology and framework and management of horizontal issues.  This could be achieved by expanding the role of the MAF Directorate within TBS to provide horizontal oversight and integration across AoMs for both methodology and results.

• Previous page
• Table of contents
• Complete text
• Next page