Handbook for Regulatory Proposals: Performance Measurement and Evaluation Plan

Appendix A: Performance Measurement and Evaluation Template

Download

If the following document is not accessible to you, please contact publicenquiries-demandesderenseignement@tbs-sct.gc.ca for assistance.

Performance Measurement and Evaluation Template (RTF version, 550 Kb)

To view the RTF version, use the document conversion features available in most word processing software, or use a file viewer capable of reading RTF.

Appendix B: Developing a logic model for regulatory activities

A logic model (also known as results logic and theory of action or intervention) is a graphic (usually accompanied by text) that tells the story of a regulatory initiative. It connects the inputs (resources) and activities (internal processes) to the outputs (products or services generated from the activities), the groups reached (partners, intermediaries, and target groups), and the expected outcomes of that initiative (the sequence of changes in groups outside the control of the regulator).

As illustrated in the following example, a logic model is commonly depicted as a graphic.

Figure 1: Logic Model Example

Figure 1 Logic Model Example
Source: Health Canada LRAD, 2009.

Appendix B Figure 1 - Text version

To depict a regulatory performance story, it is useful to start with a structured understanding of the problem, need, risk, or harm before developing the results logic. The advantages of defining the problems, needs, risks, or harms before defining results include the following:

1. It supports the first step in the regulatory process as described in the Cabinet Directive on Streamlining Regulation: "identify the problem or policy issue." If we have not properly identified the problem, then we have likely not identified an appropriate solution.
2. Identifying problems, risks, or harms serves to set the vital context. In fact, they can directly set the terms for and define the inputs, activities, outputs, and outcomes stated in the logic model. Figure 2 shows how this works.

Figure 2: Example of How Problems and Needs
Inform the Results Logic of an Environmental Initiative

Figure 2b Example of how problems and needs inform the results logic of an environmental initiative

Source: Participants Manual, Canada School of Public Service course on Performance Measurement and Evaluation (R003 – 2008)

Appendix B Figure 2 - Text version

Hint: The key to developing a problems/needs or risk-based results logic is to define the problems or needs in terms of a hierarchy, starting with outside factors and moving inward (downward), and then to translate these into desired results.

The following chart contains a comprehensive list of questions with examples of problems or risks that serve to set up appropriate results.

Logic model component	Ask yourself about the problem	Examples	Ask yourself about the result you want	Examples
Ultimate/End outcomes Social or cultural situation	What is the social or cultural problem? What is the risk to public health, the environment, public safety, or the economy? What is the severity of the risk?	Rate of health incidents, hectares of land contaminated, rate of injuries or deaths caused by airplanes	When will the problem or risk no longer be an issue? What will the problem or risk look like when it is no longer an issue? How does the program line up with the department's Strategic Outcome(s)?	Reduced rate of health incidents, hectares of land remediated, reduced rate of injuries or deaths caused by airplanes
Intermediate outcomes Stakeholder behaviour	How do stakeholder behaviours influence the social or cultural situation? What behaviours are having a negative impact on the social or cultural situation?	Industry is not adhering to voluntary guidelines Canadians are putting toxic cleaning products down the drain	What behaviours would you like to see change? How do the behaviours need to change?	Industry compliance with new regulations Reduction in percentage of Canadians putting toxic cleaning products down the drain
Immediate outcomes Stakeholder knowledge	What gaps exist in your target population's knowledge, abilities, skills, or aspirations?	Poor industry awareness and acceptance of standards Poor Canadian knowledge of risks	How do we expect our target audience(s) to react immediately to the deliverables? How will the target audience's knowledge, abilities, skills, or aspirations change?	Industry awareness and understanding of new regulations Canadian awareness and acceptance of risks and need for new rules
Reach Stakeholder participation	Are there gaps, which are critical to achieving your objective, in the participation or engagement of groups?	Low industry participation Low engagement from Canadians	Who is this initiative intended to reach or who will be affected? Whose behaviour needs to change and among which group or population? How will the participation or engagement of important groups change as a result of your program?	Increase in participation and engagement from Canadians, industry, stakeholders, and other governments and jurisdictions
Outputs Programs or supports in place for addressing the problem	Are there gaps in the suite of supports, programs, or services that are currently in place to address the problem or risk?	Industry is under no legal obligation to comply with the voluntary standards currently in place Information bulletins to advise Canadians about certain risks are not resulting in a significant enough change in the number of incidents	What product or service will we deliver in order to fill the gaps?	Regulations, inspection reports, information campaigns
Activities Internal practices	Are there problems with the current delivery practices? Are there programs or services being offered in other jurisdictions that are demonstrating better results than our programs or services?	Inefficient delivery practices, incomplete assessment of criteria, gaps in communication	What will we do? What actions or work will be done? What services will be delivered?	Conduct research, publish documents, provide advice, draft regulations, conduct inspections, provide information to the public (possibly with a focus on particular target groups)
Inputs Internal resources	Are there gaps in the financial, human, or technical resources available? Is data availability a problem?	Lack of resources, lack of data, or lack of information	What resources do we have for this regulatory initiative? What additional information is needed to improve delivery?	Staff, funding, better access to information

[1]. A program is defined as a group of related resource inputs and activities that are managed as a budget unit to address one or more specific needs and to achieve certain expected results.

[2]. Regulatory organizations will have to use their judgment to determine how many indicators are needed.

[3]. The expression of indicators in numerical form does not preclude the use of qualitative information, as shown in the examples found later in this section.

[4]. Note that in practice, it is not possible to maximize all criteria simultaneously; it is often necessary to trade off performance on some criteria against improvements on other criteria.

Menu

Handbook for Regulatory Proposals: Performance Measurement and Evaluation Plan

Appendix A: Performance Measurement and Evaluation Template

Appendix B: Developing a logic model for regulatory activities

Figure 1: Logic Model Example

Figure 2: Example of How Problems and Needs Inform the Results Logic of an Environmental Initiative

Figure 2: Example of How Problems and Needs
Inform the Results Logic of an Environmental Initiative