Evaluation Guidebook for Small Agencies

• Previous page
• Table of contents
• Complete text
• Next page

Section Three: Developing an Evaluation Function

3.1  Challenges in Developing an Evaluation Function

As mentioned previously, many small agencies are currently facing or have faced considerable challenges in developing evaluation and performance measurement capacity in their organizations. Despite these challenges, some small agencies have made considerable progress with respect to implementing evaluation and performance activities in their organizations in a meaningful way. In these agencies, information and findings from performance measurement and evaluations is valued and is actively used to support decision making and planning within the organization.

Some factors that have been found to be associated with enhanced evaluation capacity in smaller organizations include the following:

• Agencies that have a regulatory or research mandate may be inherently more evaluative in nature because they are used to the analytic and evaluative process required for performance measurement and evaluation.
• There is a commitment to evaluation and performance measurement activities by the political and senior management levels of the organization.
• Managers have a good understanding of the role of evaluation in the management cycle.
• The existing culture promotes the use of information for decision making.
• There is an internal group that has the capacity to market performance measurement and evaluation services within the organization and to make the effort to communicate evaluation results to managers and external stakeholders.
• There is an identifiable "champion" in the agency who understands and works consistently at explaining the benefits of performance measurement and evaluation to other members of the organization.
• There is senior management support to produce the overall cultural shifts required in an organization as it integrates the concepts and process of performance measurement and evaluation within the day-to-day activities of the organization. [6]

Small agencies face many challenges with respect to enhancing evaluation capacity within their organization. Examples include the following:

• The political appointee (Head of Agency) often does not have extensive experience within the Public Service. One possible outcome of this is that there is a lack of support or understanding from the Head of the Agency with regard to issues of performance reporting and evaluation within a public service context.
• Resource limitations of small agencies are notable. Consequently, there is often little flexibility in the allocation of resources for the development of new internal processes that are not directly part of the agency's mandate.
• With regard to human resources considerations, there is difficulty in attracting internal capacity, even where positions exist.
• The workload may not justify the need for a full-time evaluation function.

As illustrated, the small agency environment creates unique circumstances and challenges for building capacity in evaluation and performance measurement. Suggestions for building capacity are outlined in later subsections.

3.2  Checklists for Implementing the Evaluation Policy

In this section, there are a number of checklists available that will assist the reader in becoming familiar with various sections of the TB Evaluation Policy . For readers already familiar with the policy, it may be a good exercise to go through the various lists to identify potential gaps or areas covered by your agency in implementing the policy.

*  For small agencies a departmental evaluation committee or a combined audit and evaluation committee may serve the same role. In the smaller agencies (say less than 50 FTEs) the Evaluation Committee, Senior Departmental or Management Committee is often comprised of the same people.

3.2.1  Establishing an Agency Evaluation Plan

The Evaluation Plan is strategically focussed and designed to balance the evaluation workload between meeting internal agency information needs and serving external reporting against federal government priorities. It should be developed or reviewed annually, although it may span several years. Many small agencies may conduct one or possibly two evaluations every five years. As a result, the annual evaluation plan is more likely to be reviewed rather than developed each year. During the review, the plan should be updated and modified to address any changes or shifts that have occurred within the previous twelve months.

The evaluation plan incorporates measures against the MRRS strategic outcomes, covers the program inventory, utilizes the ongoing performance measures, and feeds the products into the governance structure to support the MRRS decision-making needs.

See the TBS Web site at http://www.tbs-sct.gc.ca/cee/pubs/guide/dep-epe-eng.asp for a detailed guide to good practices for developing and tabling annual evaluation plans. The following checklist outlines an abbreviated version of these emerging or best practices. Please note that not all of the points are required elements. (See previous checklist for required elements of the Evaluation Policy .)

*  Evaluation Policy requires evaluation reports to be made public.

3.3  Building Internal Evaluation Capacity

The concept of capacity building is similar to the concepts of organizational development, organizational effectiveness and/or organizational performance.

Capacity building involves a variety of activities such as the following:

• addressing gaps in infrastructure by identifying and providing tools and training;
• providing incentives to recognize and reinforce new behaviours;
• focussing on people and ensuring that strategies are in place during the transition period;
• supporting collaboration with other agencies and partners; and
• other methods of organizational performance management including the balanced scorecard approach, principles of organizational change, cultural change, and organizational learning.

3.3.1  Steps for Building Evaluation Capacity

Building capacity within an organization usually follows three basic steps.

Step One – Identify Gaps

Identify gaps and issues with respect to evaluation capacity.

a.  Conduct inventory of existing data sources and information (i.e., operational, financial, administrative, strategic management, and accountability data).

b.  Undertake assessment with managers to identify current information needs.

c.  Identify information gaps and needs.

Step Two – Develop Strategies

Identify the changes or strategies needed to address these gaps.

d.  Consider strategies related to organizational change, human resources, infrastructure, networking. (See examples and suggested strategies below.)

Step Three – Action Plan for Change

e.  Develop an action plan for change and assign roles and responsibilities.

3.3.2  Strategies for Building Evaluation Capacity

Some examples of how small agencies have developed performance measurement and evaluation capacity within their agency are presented below.

Some approaches to enhancing capacity that were suggested during the exercise to develop models for performance measurement and evaluation in small agencies include the following:

• building commitment and supporting cultural change within an organization;
• identifying an evaluation "champion;"
• providing relevant training and staff development (e.g., mentoring);
• working with a central agency or larger department to identify lessons learned that can be adapted to a small agency setting;
• identifying the importance of evaluation planning, clarifying objectives of the evaluation function, and linking it to other functions within the agency;
• seeking external advice and support (e.g., TBS, Centre of Excellence, Small Agency Administrator's Network, Canadian Evaluation Society); [7]
• identifying and/or leveraging additional resources from external and internal sources; and
• developing or enhancing leadership training within the agency so there is a clear understanding of the role of evaluation within a result-based management environment.
  

General project management skills are required for the effective planning, implementation, and management of evaluations. In this manner, evaluation projects do not differ substantially from many other projects that public service managers will plan and implement in a results-based environment. The checklist below is a generic guide to be used for those who wish to develop and monitor capacity in the area of project management.

Key References

Community of Small Organizations Web site: .

London School of Economics and Political Science. Juggling On a Unicycle: A Short Guide to Organizing A Small Agency , 1999.

Performance and Planning Exchange. Making Change Personal: Why we don't know what we know , November 2003.

Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies. Summary Report , 2003.

Treasury Board of Canada Secretariat. Changing Management Culture: Models and Strategies to Make It Happen , 2003.

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Treasury Board of Canada Secretariat. Modern Comptrollership Practices: Toward Management Excellence , 2003.

Treasury Board of Canada Secretariat. Review of Approaches in Other Jurisdictions. Lessons Learned from International Experience with Performance Measurement/Evaluation in Small Agencies , 2003.

• Previous page
• Table of contents
• Complete text
• Next page