Five-Year Evaluation of the Management Accountability Framework

Text Versions

Figure 1:  Evaluation Framework

Multiple lines of evidence were used:

• Document/literature review
• Interviews with key stakeholders
• International benchmarking
• Costing data

Twenty–three questions from Deputy Heads informed four strategic questions: (See Annex E)

1. Is MAF meeting its objectives?
2. Are MAF Assessments Robust?
3. Is MAF Cost–Effective?
4. Is MAF Governance Effective?

Main evaluation issues:

1. Relevance
2. Success
3. Cost–Effectiveness

Back to figure 1

Figure 2: MAF 10 elements

Public Service Values

Through their actions, departmental leaders continually reinforce the importance of public service values and ethics in the delivery of results to Canadians (e.g. democratic, professional, ethical and people values).

Governance and Strategic Directions

The essential conditions - internal coherence, corporate discipline and alignment to outcomes - are in place for providing effective strategic direction, support to the minister and Parliament, and the delivery of results.

Policy and Programs

Departmental research and analytic capacity is developed and sustained to assure high quality policy options, program design and advice to ministers.

People

The department has the people, work environment and focus on building capacity and leadership to assure its success and a confident future for the Public Service of Canada.

Citizen-focused Service

Services are citizen-centred, policies and programs are developed from the "outside in", and partnerships are encouraged and effectively managed.

Risk Management

The executive team clearly defines the corporate context and practices for managing organizational and strategic risks proactively.

Stewardship

The departmental control regime (assets, money, people, services, etc.) is integrated and effective, and its underlying principles are clear to all staff.

Accountability

Accountabilities for results are clearly assigned and consistent with resources, and delegations are appropriate to capabilities.

Results And Performance

Relevant information on results (internal, service and program) is gathered and used to make departmental decisions, and public reporting is balanced, transparent, and easy to understand.

Learning, Innovation and Change Management

The department manages through continuous innovation and transformation, promotes organizational learning, values corporate knowledge, and learns from its performance.

Support from TBS
Advisory services can be obtained through the Comptrollership Modernization Directorate, TBS web site or contact (613) 957-2437. (http://www.tbs-sct.gc.ca/CMO_MFC/)

Back to figure 2

Figure 3:  MAF Assessment Process

Step 1 — Priority Setting

• The context page outlines the key priorities to be addressed in the coming year

Step 2 — Preparation and Document Submission

• TBS introduces the assessment criteria and organizations submit documentation

Step 3 — Assessment

• Draft assessment results are released

Step 4 — Finalization and Reporting

• Assessment results are finalized

Step 5 — Communication

• Deputy Heads are informed of the results

Step 6 — Publication

• MAF Assessment results are publicly released

Step 7 — Post–Mortem

• Areas for improvement are identified

Back to figure 3

Figure 4: 2008/09 MAF Assessment Estimated Total Cost per Organization

(A sample of fourteen organizations; estimated total cost is rounded to nearest the thousandth)

1. $45,000
2. $50,000
3. $30,000
4. $45,000
5. $150,000
6. $55,000
7. $320,000
8. $95,000
9. $90,000
10. $80,000
11. $5,000
12. $10,000
13. $40,000
14. $50,000

Back to figure 4

Figure 5:08/09 MAF Assessment Estimated Cost for Three Sample Organizations

(Estimated cost is rounded to nearest the thousandth)

Small Organization – $40,000

Medium Organization – $100,000

Large Organization – $350,000

Back to figure 5

Figure 6: Estimates of MAF Assessment Costs within TBS by Sector

Policy Centres (Non HR) – $3,008,000 (53% of total costs within TBS)

Policy Centres (HR) – $838,000 (15% of total costs within TBS)

Program Sectors – $647, 000 (11% of total costs within TBS)

MAF Directorate – $883,000 (16% of total costs within TBS)

Corporate Services – $262,000 (5% of total costs within TBS)

Back to figure 6

Figure 7: Mapping of Evaluation Recommendations to Principal Findings and Lines of Evidence

Recommendation #1: Implement a risk/priority based approach to the MAF assessment process.  

This recommendation was based on the following findings:

Finding #2: There continues to be a place for ongoing dialogue between TBS and Deputy Heads during the MAF process (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #5: A tailored assessment approach would enable consideration of the unique risks and priorities of organizations (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #6: There are further opportunities to reduce the impact of the MAF assessment on departments and agencies (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #9: Our consultation indicated that most departments and agencies were unsure of the cost–effectiveness of MAF (source of evidence: interviews, consultations, costing analysis)

Recommendation #2: Develop guiding principles for assessing managerial performance

This recommendation was based on the following findings:

Finding #1: MAF has led to increased focus on management practices (source of evidence: interviews, consultations, literature review)

Finding #4: There has been increasing stability of the MAF indicators in recent rounds (source of evidence: interviews, consultations, literature review)

Finding #6: There are further opportunities to reduce the impact of the MAF assessment on departments and agencies (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #7: Qualitative measures require increased subjectivity in analysis (source of evidence: interviews, consultations, literature review)

Finding #8: Management practices are best measured by outcome–based indicators (source of evidence: interviews, consultations, literature review)

Recommendation #3: Introduce or leverage an existing governance body from client departments

This recommendation was based on the following findings:

Finding #1: MAF has led to increased focus on management practices (source of evidence: interviews, consultations, literature review)

Finding #2: There continues to be a place for ongoing dialogue between TBS and Deputy Heads during the MAF process (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #10: TBS is the appropriate entity to measure managerial performance within the federal government (source of evidence: interviews, consultations, literature review, international benchmarking)

Recommendation #4: Develop a stakeholder engagement and communication strategy and plan

This recommendation was based on the following findings:

Finding #3: TBS has in place a structured and rigorous process for reviewing the MAF assessment results (source of evidence: interviews, consultations, literature review)

Finding #5: A tailored assessment approach would enable consideration of the unique risks and priorities of organizations (source of evidence: interviews, consultations, literature review, international benchmarking)

Finding #7: Qualitative measures require increased subjectivity in analysis (source of evidence: interviews, consultations, literature review)

Recommendation #5: Assign formal responsibilities within TBS to oversee the MAF assessment methodology/framework and management of horizontal issues/action plans

This recommendation was based on the following findings:

Finding #3: TBS has in place a structured and rigorous process for reviewing the MAF assessment results (source of evidence: interviews, consultations, literature review)

Finding #10: TBS is the appropriate entity to measure managerial performance within the federal government (source of evidence: interviews, consultations, literature review, international benchmarking)

Back to figure 7

Figure 8: MAF Logic Model

Ultimate Outcome 1 (Continuous improvement in quality of public management in federal public service) is a product of the following intermediate outcomes:

• Intermediate Outcome 1 (Departments/Agencies take action to improve management performance) AND
• Intermediate Outcome 2 (TBS has greater capacity to meet needs of Departments/Agencies for advice on management)

Intermediate Outcome 1(Departments/Agencies take action to improve management performance) is a product of the following immediate outcomes:

• Immediate Outcome 1 (Shared understanding in Government of Canada of standards of good management)
• Immediate Outcome 2 (Departments/Agencies recognize management issues and prepare action plans)
• Immediate Outcome 4 (Availability to Parliament and public of information on state of public management)

Intermediate Outcome 2(TBS has greater capacity to meet needs of Departments/Agencies for advice on management) is a product of the following immediate outcome:

• Immediate Outcome 3 (Better understanding in TBS on state of public management, including key risks)

Immediate Outcome 1 (Shared understanding in Government of Canada of standards of good management) is a product of the following outputs:

• Output 1 (MAF assessments for Departments/Agencies)
• Output 3 (Ongoing advice, assistance and outreach to Departments/Agencies)
• Output 4 (Analysis of Government of Canada and global trends in public management)

Immediate Outcome 2 (Departments/Agencies recognize management issues and prepare action plans) is a product of the following outputs:

• Output 1 (MAF assessments for Departments/Agencies)
• Output 2 (Recommended management priorities for Departments/Agencies)
• Output 3 (Ongoing advice, assistance and outreach to Departments/Agencies)

Immediate Outcome 3 (Better understanding in TBS on state of public management, including key risks) is a product of the following outputs:

• Output 1 (MAF assessments for Departments/Agencies)
• Output 3 (Ongoing advice, assistance and outreach to Departments/Agencies)
• Output 4 (Analysis of Government of Canada and global trends in public management)

Immediate Outcome 4 (Availability to Parliament and public of information on state of public management) is a product of the following outputs:

• Output 5 (Public communications products related to MAF)

Back to figure 8

Figure 9: MAF Alignment with Government Priorities

The Management Accountability Framework (MAF) informs TBS oversight role as the Management Board and Budget Office; understanding of departmental performance, capacity and risks, whether and how to reduce restrictions limits on authorities and departmental compliance with TB policies.

The MAF supports core TBS oversight functions:

Expenditure Management in Departments:

• Understanding of departmental capacity for spending for results
• Strategic Reviews
• Assessment of new spending proposals/TBS submissions

Financial Management in Departments:

• Ensuring adequate financial management controls

People Management in Departments

• Implementation of PSMA Act and HR Modernization

Ultimately, MAF and TBS support federal priorities:

• Managing spending to ensure programs and services are efficient, effective,aligned with the priorities of Canadians, and affordable over the long term

Back to figure 9