ARCHIVED - Societal Indicators and Government-wide Reporting in the Government of Canada

This page has been archived.

Louise Bellefeuille-Pregent, Senior Director, and Tim Wilson, Senior Analyst, Horizontal Results Management, Treasury Board of Canada Secretariat

Abstract

From the perspective of the Government of Canada's management board (the Treasury Board and its Secretariat), key societal indicators can be useful for government-wide analysis. They can be used to achieve a deeper understanding of broad societal trends in order to guide policy and planning, and to provide a context within which government performance can be assessed. The Government's initial explorations of this possibility have led to the production of an annual report, Canada's Performance (http://www.tbs-sct.gc.ca/report/govrev/03/cp-rc-eng.asp). The annual report certainly fulfils the latter of these purposes; that is, it provides a context for assessing government performance. However, the explicit link to the planning process is not there yet. This paper outlines the background of the approach to reporting on societal indicators used in the Canada's Performance report as well as possible future directions for this type of reporting in the Government of Canada - namely, the use of societal indicators in conjunction with a government-wide planning process.

---

Louise Bellefeuille-Pregent, Senior Director, and Tim Wilson, Senior Analyst, Horizontal Results Management, Treasury Board of Canada Secretariat

1. The Management Board Perspective

The Treasury Board of Canada is a Committee of Cabinet and manages the Government's financial, personnel, and administrative responsibilities. Considered the general manager and employer of the public service, it sets policy in these areas, examines and approves the proposed spending plans of government departments, and reviews the development of approved programs. The administrative arm of the Treasury Board, the Treasury Board Secretariat (TBS), supports Treasury Board Ministers in fulfilling these roles.

The perspective of the Treasury Board and its Secretariat, then, on key societal indicators is not that of the pure technician or statistician. As the Government's "management board," the Treasury Board of Canada and its Secretariat see reporting on key societal indicators as a primary means of providing a government-wide context for priority planning and for assessing performance. In addition, such reporting is a way in which the Government is accountable to Parliament and to Canadians for the results achieved with the resources allotted.

This paper outlines the experience of the Government of Canada with respect to the use of societal indicators from this management board perspective, that is, for the purposes of government-wide reporting. More particularly, the paper outlines:

• the background of the concept - the roots of the idea of using societal indicators for government-wide reporting in Canada and the objectives this type of reporting is to serve;
• the approach - the steps taken from 1996 to 2001 by the Government of Canada to develop a core set of societal indicators as well as a framework for reporting on those indicators from a government-wide perspective;
• where we are now - the Government of Canada's current vehicle for societal indicator reporting from a government-wide perspective, Canada's Performance;¹ and
• possible future directions - the use of societal indicators in conjunction with a government-wide planning process and corresponding report.

2. Background

Beginning in the mid-1990's, a number of inter-related forces coalesced to form a foundation for societal indicator reporting from a government-wide perspective in the Government of Canada, namely: projects undertaken to "improve reporting to Parliament," the emergence of the Government's commitment to "results-based management," the increasing promotion of and reporting on collaborative or "horizontal" arrangements, and the resurgence of research in "societal or quality of life indicators."

Improved Reporting to Parliament: The Improved Reporting to Parliament Project began in 1994 in collaboration with a parliamentary working group. The objectives of the Project were to improve the Expenditure Management documents supplied to Parliament and to produce and distribute departmental planning and performance information to Parliament and the Canadian public more efficiently and economically, using information technology.

Strengthening accountability to Parliament and to Canadians is a fundamental and on-going commitment of the Government of Canada. Providing Parliament and Canadians with high-quality and timely information about the plans and achievements of the Government "is key to implementing a citizen-focused agenda, since it allows Canadians to engage more effectively in understanding and shaping public policy."²

The 2003 and 2004 Budget Speeches re-iterated the Government's commitment to improve its reporting to Parliament. Then, in March 2004, the Government released details of its comprehensive plan to modernize public sector management, entitled Strengthening Public Sector Management: An Overview of the Government Action Plan and Key Initiatives (http://www.tbs-sct.gc.ca/spsm-rgsp/index-eng.asp). The plan includes a commitment to improve reporting to Parliament and the public by making it "more timely, clear and useful, based on a 'whole of government' perspective" (p. 19).

These ongoing commitments to improve reporting and to provide information from a government-wide perspective have led to calls for the use of key societal indicators in government reporting. Parliamentarians, for instance, have pointed out that because the outcomes of government efforts are often "borderless," the performance information from individual departments and agencies can be better interpreted if objective context information is also available. For example, the Thirty-Seventh Report of the Standing Committee on Procedure and House Affairs - Improved Reporting to Parliament Project - Phase 2: Moving Forward (June, 2000) describes them as "higher-level performance indicators." According to the report, "societal indicators essentially will provide a bridge linking specific government program and policy objectives to broader societal considerations."³ In that vein, in 2001 a series of seminars entitled "Measuring Quality of Life: The Use of Societal Outcomes by Parliamentarians" took place, bringing together parliamentarians, senior public servants and members of the policy community. The seminars concluded that societal outcome reporting could more effectively plug parliamentarians and citizens into the policy process, lay the foundation of a better working relationship between parliamentarians and the Public Service and provide a "whole-of-government perspective".⁴

In order for this "higher-level" reporting to be linked to government program and policy objectives, however, public service managers must plan for, monitor and report on the results of their policies and programs. For the Government of Canada, "results-based management" provides the foundation of an improved, comprehensive reporting structure.

Results-based Management: The idea that managers in the public service should manage for results has been around for a long time; it could be said that the Government of Canada started contemplating results-based management as far back as the 1960's with the Glassco Commission study on government organization and its theme of "letting the managers manage." However, it was during the mid-1990's that the concept started to gain real momentum in Canada - an era in which Canadian governments were faced with budgetary deficits and the public service was called upon to do more with less.

Results-based management is enshrined in the Government of Canada's modern management framework, Results for Canadians. At the heart of this management framework are commitments to:

• focus on citizens, for example, by using the internet and information technology to provide 'one-stop access' for government services;
• adopt a clear set of values in four areas - democratic, ethical, professional and people values;
• manage for results, for example, provide accurate and timely information on the results achieved by government programs and services, learn from experience, and build public confidence; and
• ensure responsible spending, both in individual departments and agencies and from a whole-of-government perspective.

While governments have historically focused on inputs, activities and outputs, the Government of Canada committed itself to a modern management agenda that focuses on actual results. In the public service, "results" "are positive changes in the conditions of Canadian society that occur as a consequence - direct or indirect - of public programs. They are aligned with the societal objectives established by the government of the day."⁵ Managing for results involves rethinking the "life-cycle" of a program or policy. "It means clearly defining the results to be achieved, delivering the program or service, measuring and evaluating performance and making adjustments to improve both efficiency and effectiveness. It also means reporting on performance in ways that make sense to Canadians."⁶

Reporting in ways that make sense to Canadians means taking a "citizen focus" to reporting. Just as a "citizen focus" in service delivery means moving beyond the traditional, "inside out" approach - beyond the traditional approach of reflecting government organizations more than the needs and priorities of citizens - so too does citizen focused reporting mean providing information on the outcomes of government efforts at a higher level than that of the department or program. This is another reason, then, that reporting on key societal indicators is important. This is also one of the reasons that the Government of Canada has explored ways of planning for, monitoring and reporting on "horizontal" results.

Horizontality: The social and economic outcomes that are measured by key societal indicators and that form the goals of government activity go beyond individual organizations and involve more than one department or jurisdiction, as well as other partners.⁷ Since the mid-1990's, there has been an increased concern with the business of getting this horizontal dimension of government right. It is an open question as to whether or not "horizontality" in the public sector is a new phenomenon or not. It could be said that "Canadian governments have been preoccupied since Confederation with the age-old quest for 'coordinated government.'" However, the case can also be made that over the last decade the complexity of the issue, as well as the awareness of the issue, has increased markedly.⁸

The above mentioned focus on results, as part of the "new public management" more generally, has certainly been one of the drivers for this concern with horizontality. Among the other drivers, one would certainly have to cite the commitment to provide seamless, single-window service delivery to Canadians as well as the development of communications technologies, which facilitate the spontaneous generation of policy and program delivery networks.

An important development in the area of intergovernmental collaboration and reporting occurred in February 1999 when the Social Union Framework Agreement (SUFA) (http://socialunion.gc.ca/news/020499_e.html) was signed by the federal government and all provinces and territories except Quebec. The Agreement builds on earlier arrangements and makes the accountability dimension more visible. SUFA accountability provisions commit governments to:

• be accountable directly to Canadians - that is, to monitor, measure and report publicly on social policy outcomes; and
• develop joint accountability frameworks for new Canada-wide social initiatives supported by transfers to the provinces and territories.

This definition of intergovernmental accountability relies on citizen engagement in the agenda setting and policy making process. Reporting on societal outcomes, or "quality of life," "can help inform processes for involving citizens in policy making."⁹

An example of a joint framework that involves reporting on societal outcomes is the Multilateral Framework for Labour Market Agreements for Persons with Disabilities, signed by federal, provincial and territorial Ministers of Social Services¹⁰ on December 5, 2003. The goal of the Multilateral Framework is to improve the employment situation of people with disabilities, by enhancing their employability, increasing the employment opportunities available to them, and building on the existing knowledge base.

Public reporting forms the basis of accountability under this framework, which reflects the commitments made under SUFA to monitor, measure and report regularly to citizens on the outcomes of social programs. In addition to reporting on indicators of program performance, Ministers agreed to report on societal indicators of labour market participation - such as, employment rate of working age persons with disabilities, employment income and education attainment - for their jurisdiction or at the national level, subject to data availability.

Societal Indicator Research: Over the past decade or so, the standard measures of our progress as a society - such as Gross Domestic Product (GDP) per capita - have been challenged as insufficient for fully capturing our "quality of life."¹¹ The resurgence of societal indicator research is fundamentally tied to this effort to reconceptualize our "quality of life."¹² Within this context, "quality of life" is distinguished from "standard of living" - the latter being generally understood as a quantitative assessment of economic well being solely. "For example, someone may have a high standard of living but be working odd hours, have no job security and suffer from life-threateningly high levels of stress. These will not be reflected in his or her standard of living."¹³

In Canada, the renewed interest in societal indicator research has manifested itself in efforts to develop quality of life frameworks for research and reporting at the community level, at the provincial level as well as the national level.¹⁴ National level research and reporting is not the exclusive domain of the federal government, however. For instance, the Quality of Life Indicator Project led by the Canadian Policy Research Networks, a private non-profit corporation, undertook the task of creating a prototype set of national quality of life indicators, to reflect the range of issues that truly matter to citizens. After engaging citizens in order to determine the appropriate indicators, in 2002 they released the report Quality of Life in Canada: A Citizen's Report Card.¹⁵ The difference between efforts such as this by NGOs and those of the Government of Canada can be summed up basically as follows: the former are designed to inform broad policy processes,¹⁶ whereas the efforts of the federal government are designed to link such reporting to priority setting and the assessment of government performance.

3. Approach

All of these factors - improved reporting to Parliament, results-based management, horizontality and the resurgence of societal indicator research - seemed to align in the mid-1990's and point to a common goal: government-wide reporting on social outcomes and indicators - the "objective being to achieve a deeper and shared understanding of broad societal trends to guide policy and planning, and to provide a context within which government performance can be assessed" (Accounting for Results 1997, p. 17). This section of the paper outlines the concrete steps taken by the Government of Canada to reach that destination.

Initial Commitment: In 1996, the President of the Treasury Board's annual report to Parliament on the Government's management control and review functions was tabled.¹⁷ That report recognized the confluence of forces described above and highlighted that other jurisdictions were beginning to explore the concept of government-wide reporting on societal outcomes and indicators: "Different approaches can be used to identify and report on broad social and economic objectives. Establishing a few core indicators of government performance and regularly measuring and reporting on them is becoming a popular approach in other jurisdictions." The report also highlighted some of the benefits this approach to reporting seems to provide, namely:

• A single, comprehensive perspective on the most important information that shapes the government's priorities and decisions;
• A stable and more strategic vision of the government's objectives;
• A persuasive context to show the links between programs, which encourages greater cooperation within and among departments and agencies; and
• A more open and disciplined approach to showing the value of policies and programs, which also provides for public involvement beyond consultation on specific issues. (p. 2)

The 1996 President's report committed TBS to "work with Statistics Canada and other departments and agencies to begin to bring together available information and establish a set of core performance indicators from existing and perhaps new data." This commitment involved "examining the experiences and lessons learned in other jurisdictions on items such as: which indicators are most useful; how they should be selected, measured and reported; how to assure the public that the measurement is reliable; how to link the indicators back to the actual programs and initiatives; and how to involve the public" (p. 2-3).

This commitment was reconfirmed in the President's reports of 1997 and 1998.¹⁸ The "Government-wide Performance Indicators Study" that followed from this commitment involved developing a set of cross-cutting indicators that would be useful for policy direction, fostering horizontal approaches and communicating more clearly with Canadians.¹⁹

Over these years, in addition to the consultations and indicator research described above, the Government of Canada explored the development of "Horizontal Results Frameworks." "Results" frameworks can take a variety of forms and follow a variety of methodologies - i.e., "performance frameworks," "accountability frameworks," "evaluation" frameworks, "Results-based Management and Accountability" frameworks²⁰ - but all are basically tools to plan for, monitor and report on the results of government activity. What makes these "horizontal" is that the frameworks account for results, which go beyond the limits of a single department - and so reflect the contributions of other federal departments, other jurisdictions and/or other sectors of the economy.²¹

For instance, in 1997-98, the Government of Canada worked on a particular horizontal results framework as part of the children's initiative announced in the 1997 Speech from the Throne (Accounting for Results 1997, 19). In 1998-99, TBS compiled and analyzed information from departments and agencies on "sustainable development" and on "persons with disabilities." The work showed that the information could be clustered around a few key themes and that a government-wide view on performance on the issue in question could be achieved (Managing for Results 1999, p. 16). The Treasury Board Secretariat at this time also worked with departments and other jurisdictions to promote the development and use of performance information on horizontal results more generally (Managing for Results 1998, p. 10-15).

The horizontal framework for children mentioned above is an example of a broad, horizontal "issue" or policy area - ensuring a good quality of life for children is definitely a shared goal for different organizations within and outside government, but there is not necessarily a particular children's initiative or program on which they work together. With this in mind, the Government of Canada has also tried to isolate more specific management and reporting tools for horizontal "initiatives."²² A horizontal initiative can be defined as "...an initiative in which partners from two or more organizations have established a formal funding agreement (e.g. Memorandum to Cabinet, Treasury Board submission, federal-provincial agreement) to work toward the achievement of shared outcomes."²³

Starting in 2000, TBS developed a database wherein information on these "horizontal initiatives" could be collected and analyzed.²⁴ Also, from 2000 to 2003, TBS worked with departments and agencies to promote the use of the accountability provisions under the Social Union Framework Agreement (SUFA) (see above). TBS developed reporting guidelines and a comprehensive SUFA template to report on results achieved through federal funding commitments made to provinces and territories.²⁵ At the sunset of this pilot project (2003), TBS developed guidelines integrating SUFA reporting into the "horizontal initiative" reporting and corresponding database.

A Comprehensive Reporting Framework: In 1999, this work on societal indicators and on horizontal results bore fruit in the form of the "Comprehensive Reporting Framework." Basically speaking, this framework calls for a more holistic approach to reporting, consisting of three elements:

• "departmental reporting" on their priorities, plans and achievements - as typically found in their Departmental Reports on Plans and Priorities and in their Departmental Performance Reports;²⁶
• "horizontal reporting" on outcomes that go beyond the efforts of any one department, jurisdiction or sector; and
• reporting on "societal indicators."

"The framework suggests that (quality of life) reporting be considered as part of an integrated and comprehensive performance measurement report that would offer Canadians a comprehensive synthesis of performance in areas of interest to citizens - improvement in our quality of life, the achievement of shared societal goals, and the specific results achieved by national programs and services."²⁷

The Comprehensive Reporting Framework recognizes that the purpose of each type of reporting is different. That is, reporting on quality of life or societal indicators "is intended to provide information to citizens in a way that can inform broad policy direction and agenda setting. ... Reporting on the outcomes achieved on shared societal goals is intended to provide information to citizens in order to engage them, as well as other players such as governments and non-governmental organizations, in the identification and achievement of shared goals. Reporting on departmental program results and service delivery is intended to allow citizens to hold governments accountable, ideally from the perspective of transparency and learning rather than simply to blame or criticize."²⁸

Also, the concept of producing a report, which reflected this framework, was considered. It was suggested at this time that the report be modelled on the President's Annual Reports to Parliament: "a (quality of life) report should be a public report rather than an internal report, which could be released by the federal government and perhaps tabled in Parliament, building on the model of the Managing for Results report which is tabled annually in the fall. The objective of the report would have to be clearly articulated and the linkages between a new initiative on (quality of life) reporting, the existing Departmental Performance Reports, and other reporting commitments of the federal government would have to be clarified."²⁹

As a first step to flesh out this framework, the set of societal indicators in the table below were identified by a task force of senior officials and included in Managing for Results 1999 and Managing for Results 2000. Information for each indicator was available on the Web. This information included a definition of the indicator, presented trend analysis in a graphical form, provided some interpretation of the data, and outlined relevant international comparisons, in particular comparable data from the United States where available.

Health, Environment and Public Safety	Economic Opportunity and Participation	Social Participation and Inclusion
Air/water quality Life expectancy Infant mortality Health status Crime rates Violent crime	Educational attainment Literacy rates Employment rates Per capita Gross Domestic Product Discretionary income Research and development/innovation	Measures of racism and discrimination Voter turnout Voluntarism Cultural activity and outputs

The Question of Accountability: The idea broached with the Comprehensive Reporting Framework that the federal government should report on broad social and economic outcomes in relation to its own department-level plans and performance raises the "question of accountability": can a government be held accountable for societal-level performance?

It was, and is, recognized that single governments could not be held uniquely accountable (in the strictest sense) for the performance of an economy or society: "Given the difficulty in attributing changes in these kinds of indicators to specific government actions, this type of reporting cannot be considered an instrument for holding governments accountable, although some may seek to use it in this way."³⁰ Despite these issues of "attribution," such reporting is relevant for an analysis of the "contributions" a government makes.³¹ Furthermore, while the societal-level goals that are tracked and reported on in the Comprehensive Reporting Framework are beyond what government is solely responsible for (either in terms of what one can attribute as the effects of government action or in terms of the narrowly-defined constitutional obligations of a government) they are, nevertheless, areas where government has a significant role (Managing for Results 1998, p. 22).

We could say, for instance, that while the Government of Canada is not solely responsible or accountable for Canada's environmental situation - with other players and natural processes having significant influence - the government, nonetheless, does have a role to play and invests considerable resources into the area; also, the government is "answerable" to Canadians and Parliamentarians for the situation and what actions it has taken to remediate the situation.

It should also be noted that the Province of Alberta, Canada, a jurisdiction that is often held up as a leader in the use of societal indicators for government-wide reporting, issues its annual report, Measuring Up, as a means of being "accountable" to its citizens. The report, in fact, is produced in accordance with the province's Government Accountability Act.³² Surely, given what has been said above concerning attribution, a different sense of accountability must be at stake here.

It has been suggested that the notion of accountability needs to be rethought in the context of modern government. That is, accountability should not be conceptualized in terms of compliance to rules and assigning of blame. Rather, governments should hold themselves accountable (in the sense of rendering an account), along with their partners, for "results." This is particularly the case when we try to get a grip on "horizontal accountability." That is, in the case of departmental programs, accountability is clear enough in terms of the administrative operations, rules and reporting relationships of the department in question. However, in the case of a horizontal initiative or issue, the partners must be held accountable to each other and to their stakeholders for results: for the achievement of the desired policy outcomes. In the horizontal world of "accountability for results," accountability is a matter of taking into account the complexity of the situation, monitoring and describing contributions made to the outcome and learning from past experience what works and what doesn't.³³

4. Where We Are Now: Canada's Performance Report

Beginning in 2001, TBS took up the suggestion first broached during the construction of the Comprehensive Reporting Framework: to produce a public report, preferably tabled in Parliament, using societal indicators. The first such report is entitled Canada's Performance 2001.³⁴ The President of the Treasury Board tabled it in Parliament on December 6, 2001.

What is it?: Canada's Performance is the President of the Treasury Board's annual report to Parliament on government performance - taking the place of the former Managing for Results reports - the 2002 and 2003 instalments of Canada's Performance have been tabled and we plan to table the 2004 version in late November or early December.

The reports provide information on a core set of societal indicators grouped into four themes: economy, health, the environment and communities. Trend information,³⁵ international comparisons and disaggregations are provided, when applicable, for all of the indicators. This core set of indicators was built on the original sixteen indicators mentioned above. Consultations with departmental and external experts indicated that we needed to make the themes more "user-friendly" than the original three and that we needed to add three indicators: toxic contaminants, biodiversity and physical activity.³⁶

The reports also provide information on certain key governmental programs that contribute to improving the quality of life of Canadians. In doing so, the Canada's Performance reports contribute to several of the modern management objectives described earlier:

• supporting parliamentarians who require a context for reviewing the results achieved by individual departments and agencies;
• enhancing the government's citizen focus by serving as a vehicle to engage Canadians in discussion of future policy developments;
• advancing results-based management in the federal government and improving the quality of program performance information available to Canadians and parliamentarians over time;
• supporting horizontal management and policy development by providing an overview of the connections between various issues and between the responses to these issues by different departments and agencies; and
• contributing to the transparency of the federal government's plans and achievements, as well as its accountability to Canadians and parliamentarians.³⁷

Consultations: After tabling Canada's Performance 2001, TBS consulted with Canadians, think tanks, parliamentarians, governments and other partners on the approach adopted in the report.³⁸ The consultations and engagement strategies focused on such issues as: selecting indicators that give a more comprehensive view of the economy, health, the environment and communities; presenting information in a manner that best helps Canadians to contribute to the shaping of government policy; using Canada's Performance to promote a growing culture of learning about how to manage for and by results, and engaging Canadians in the identification of themes and indicators that reflect their values and the range of issues that matter to them.

Basically speaking, there are three lenses through which to view the indicators in a government-wide report such as this: in terms of government priorities, in terms of the concerns of Canadians and in terms of the accuracy and relevance of the data.³⁹ More particularly, the consultations described above confirmed the following set of criteria with respect to the indicators and measures selected for inclusion in the report:

• Information must be relevant; indicators must reflect Canadian values.⁴⁰
• Information must be temporal; data must highlight trends over time and show progress toward goals.
• Information must be available; data must be easily accessible.
• Information must be comparable; it must be possible to compare with data from other countries.
• Information must be understandable; data must be easily grasped by various audiences.

The indicators and approach used in Canada's Performance were also informed by the work of other organizations, such as National Roundtable of the Environment and the Economy (NRTEE) - which was mandated in 2000 to develop a set of national indicators that decision makers could use to track the impact of current economic practices on natural and human assets⁴¹ - as well as by the work on comparable health indicators undertaken by the federal, provincial and territorial governments of Canada.⁴² The Government is still committed to ensuring that we are tracking these types of societal-level outcomes. For instance, for certain priority areas, such as "Cities" and "Aboriginal peoples," the Government is working on developing, tracking and reporting on key indicators of progress.

There are a few other important reporting principles which the Canada's Performance reports hold as sacred: balance (presenting both good and bad news), disaggregations to sub-groups or regional data (primarily through the electronic version),⁴³ and the inclusion of both subjective and objective indicators (for e.g., self-rated health status in addition to life expectancy).⁴⁴

Evolutions: The coverage of the report has expanded over the last couple of years. In 2003, climate change was added, bringing the core set of indicators up to twenty; for the 2004 version we plan to greatly expand the coverage by adding two whole thematic areas (Aboriginal Peoples and Canada in the World) and bringing the number of core indicators up to thirty-two.

A key development, starting with the 2002 report, has been the construction of a whole-of-government framework to support the societal-level information (see Figure 1). The framework provides a "logic model" for the Government of Canada - mapping the contributions of government programs and departments to "horizontal" (or, "Government of Canada") outcomes and ultimately to the societal-level theme in question, e.g., "Health." If the framework is the logic model, the report itself provides the "performance story." The meat of the framework, so to speak, is provided in the electronic version of the report, which allows the reader to "drill down" to the more specific department- and program-level information provided in the Departmental Reports on Plans and Priorities as well as the Departmental Performance Reports.

Figure 1: The Whole of Government Framework

Display full size graphic

Another key development being planned for this year will be the move to a more rigorous linking of government plans and performance to the societal outcomes and indicators that are tracked - rather than just grouping departments into social issue areas, the 2004 report plans to formulate these areas as "Government of Canada Outcomes" or goals - down the road, these Government Outcomes and indicators could conceivably have targets or benchmarks.

Lessons-learned: Reception on the whole has been good. Public policy experts have been enthusiastic.⁴⁵ So too have senior managers in the federal government - who believe that the "whole-of-government" perspective that the report provides allows them to situate their program or policy in the larger picture. However, there is some room for improvement when it comes to making the report more useful for Parliamentarians; similarly, we need to make the report more well known among Canadians. It could be that the objectives of the report are not always clear to Parliamentarians - since it is purporting to serve many causes. Along these lines, a lesson learned is the importance of targeting a primary audience to ensure a collective perception of the report's purpose. Clearly identifying the primary audience allows us to tailor the information in the report to explicitly meet the objectives of the reader, making the information more meaningful and useful.

5. Possible Future Directions: Government-wide Planning

During the construction of the Comprehensive Reporting Framework, two general purposes for this type of reporting were discerned: "In particular, the government would need to distinguish between using such reporting to "inform" broad policy processes and making an explicit linkage between QOL reporting and priority setting."⁴⁶ As it stands, it is not entirely clear whether Canada's Performance is seen as something to "inform" policy or if it is to be rigidly linked to a government-wide planning process. As a result, a possible future direction for the Government of Canada would be to clarify this by making the report more closely aligned with government planning and priority setting.⁴⁷

This move would take place in the context of the current Government's commitment to continually reallocate from low to high priority issues. The links made would have to show, then, how this type of reporting could drive not only priority setting but planning in the context of reallocation - perhaps via its use by Treasury Board and Cabinet Ministers when reviewing allocation and reallocation proposals. A short-term step that could be taken would be to include financials in the report: how much is Canada spending on the environment, or more particularly, on reducing the effects of climate change? This step is also in keeping with the example provided by Alberta's Measuring Up, which includes the Alberta Government's Consolidated Financial Statements.⁴⁸

Also, this notion of providing government-wide information on results and spending is enshrined in Results for Canadians: in order to "ensure rational priority setting and investment decisions, the government needs integrated, cross-departmental information on expenditures and results." The government-wide view of expenditures and results could serve as a context both for the assessment of the performance of government programming and for the setting of future priorities and plans: "First, it allows decision-makers to assess the integrity of the existing program base. ... Second, the knowledge gained from the broad-based analysis of expenditure supports rational priority setting."⁴⁹

This would also require a formalization of the government-wide planning process in accordance with the thematic areas (or "Government of Canada Outcome" areas) included in the Canada's Performance framework. One way of formalizing this process would be to pilot interdepartmental planning and budget consultation (at the Deputy Minister level) on one or two of the themes in Canada's Performance.

If formalized, this planning process could result in a corresponding government-wide report on plans (Canada Plans) - to act as the bookend to the Canada's Performance report. In its recently tabled Report on Plans and Priorities, the TBS made public its plan to explore the "development of a whole-of-government planning report that would express the government's targets and provide a clearer basis for reporting on Canada's performance over the medium term."⁵⁰

6. Conclusion

From the perspective of the Government of Canada's management board, key societal indicators can be useful for government-wide analysis. They can be used to achieve a deeper understanding of broad societal trends in order to guide policy and planning, and to provide a context within which government performance can be assessed. The Government's initial explorations of this possibility have led to the production of an annual report, Canada's Performance. The annual report certainly fulfils the latter of these purposes: namely, it provides a context for assessing government performance. However, the explicit link to the planning process is not there yet.

TBS is planning to explore the development of a whole-of-government planning report; that report could complement Canada's Performance and could make the link between societal indicator reporting and government-wide planning more explicit. If pursued to its logical end, this report on Canada's Plans (along with the existing report on Canada's Performance) would be a realization of the comprehensive reporting framework: with departmental planning and performance information (in the individual departmental reports, accessible through the electronic versions of the government-wide reports), societal-level outcome information (in the annual, government-wide planning and performance reports themselves), and planning and performance reporting around horizontal or "Government of Canada Outcomes." This latter element is the element that still needs the most work.⁵¹ The next step for the Government of Canada in this regard, then, could be to develop a government-wide planning process (that would support the corresponding planning report) based on these Government of Canada Outcomes.

 

---

^1. Canada. Treasury Board of Canada, Canada's Performance 2003, Ottawa: Queen's Printer, 2003; http://www.tbs-sct.gc.ca/report/govrev/03/cp-rc-eng.asp.

^2. Canada. Treasury Board of Canada, Results for Canadians: A Management Framework for the Government of Canada, Ottawa: Queen's Printer, 2000, p. 31; http://www.tbs-sct.gc.ca/report/res_can/rc-eng.asp.

^3. See http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?COM=8988&SourceId=116771&SwitchLanguage=1.

^4. Bennett, Carolyn, Lenihan, Donald G., Williams, John and Young, William, "Measuring Quality of Life: The Use of Societal Outcomes by Parliamentarians," Ottawa: Centre for Collaborative Government, 2001; http://kta.on.ca/pdf/cg3.pdf.

^5. Canada. Treasury Board of Canada Secretariat, "Rising to the Challenge: A Statement on Results and Results Reporting for Public Service Executives," unpublished paper, November 2004, p. ii.

^6. Canada. Treasury Board of Canada, Results for Canadians: A Management Framework for the Government of Canada, Ottawa: Queen's Printer, 2000, p. 11. Canada's approach to Results-based Management from the mid-1990's to 2000 is summarized in "Results-based Management: Country Report Prepared for the OECD Outcome-focused Management Project." December, 2000. See http://www.ppx.ca/NewsArchives/PDF/Result_Based_Management.pdf. See also, IDEA International Institute, "Results-based Management: Lessons Learnt from Case Studies," September, 2003, http://www.idea-international.org/docs/IDEA_RBM_final_report_19_09_03_Revised.pdf; and Bob Segsworth, "Accountability, Evaluation and Performance Monitoring: A Comparative Perspective," Investing in People: Creating a Human Capital Society for Ontario, RP (15), June 2003; http://www.law-lib.utoronto.ca/investing/reports/rp15.pdf.

^7. Canada, Treasury Board Secretariat, Getting Government Right: Improving Results Measurement and Accountability: Annual Report to Parliament by the President of the Treasury Board, Ottawa: Queen's Printer, 1996, p. 2; see also, Accounting for Results 1997, Ottawa: Queen's Printer, 1997, p. 13-19, (http://dsp-psd.pwgsc.gc.ca/Collection/BT1-10-1997E.pdf); Managing for Results 1998, Ottawa: Queen's Printer, 1998, p. 9-15; Managing for Results 1999, Ottawa: Queen's Printer, 1999, p. 15-28; Managing for Results 2000, Ottawa: Queen's Printer, 2000, p. 5-19 ( http://dsp-psd.pwgsc.gc.ca/Collection/BT1-10-2000E.pdf ).

^8. Bakvis, Herman and Luc Juillet, Horizontal Challenge: Line Departments, Central Agencies and Leadership, Canada School of Public Service, 2004, p. 9-16.

^9. Canada. Treasury Board of Canada Secretariat. "Quality of Life - A Concept Paper: Defining, Measuring and Reporting Quality of Life for Canadians," 2000, p. 1; http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/pubsdisc/qol-eng.asp.

^10. While the Quebec government subscribes to the general principles of this document, it did not participate in its elaboration. However, it does contribute by sharing information and best practices. The Quebec government intends to continue treating this question with the federal government in a bilateral way. All references to joint positions of the federal, provincial and territorial governments in this document do not include the Quebec government.

^11. That does not mean, however, that these traditional measures should be completely abandoned. See Joseph H. Michalski, "Quality of Life in Canada: A Citizens' Report Card - Background Report," July 2002, p. 7; http://www.cprn.com/doc.cfm?doc=47&l=en: "While the traditional measures of economic growth continue to be important benchmarks for economists to model certain types of economic behaviour, several analysts have attempted to utilize the tools of the social sciences to develop alternative accounting systems. For example, the HDI measures 'progress' through a composite index with three main components: the standard of living (GDP per capita and income above the low income cut-offs), educational attainment (adult literacy and years of schooling), and longevity or life expectancy." For a good summary of the background of societal or "quality of life" research, see also Pierre André and Dieudonné Bitondo, Development of a Conceptual and Methodological Framework for the Integrated Assessment of the Impacts of Linear Infrastructure Projects on Quality of Life, Canadian Environmental Assessment Agency's Research and Development Program, 2001, section 3.1. http://www.ceaa-acee.gc.ca/051/index_e.htm.

^12. This resurgence in interest in societal indicator research was noted in the President of the Treasury Board's report to Parliament, Managing for Results 1998: "There appears to be a renewed interest in recent years in developing additional indicators of economic and social well-being to better reflect what people want to know about how their society is doing. Within some think-tanks and research organizations, the term 'societal indicator' is also beginning to include ideas such as public engagement and building a broader social consensus on what should be measured and how" (1998, 22).

^13. Bennett, Carolyn, Lenihan, Donald G., Williams, John and Young, William, "Measuring Quality of Life: The Use of Societal Outcomes by Parliamentarians," Ottawa: Centre for Collaborative Government, 2001, p. 13; http://kta.on.ca/pdf/cg3.pdf. See also, "Quality of Life - A Concept Paper," p. 3.

^14. See Legowsk, Barbara, A Sampling of Community- and Citizen-driven Quality of Life/Societal Indicator Projects, Ottawa, 2000; [Return]

^15. Canadian Policy Research Networks, [Return]

^16. Legowsk, Barbara, A Sampling of Community- and Citizen-driven Quality of Life/Societal Indicator Projects, Ottawa, Canadian Policy Research Networks, 2000, p. iv: "Projects launched and maintained outside formal institutions where policies are made and resources assigned tend to serve as monitoring, information and advocacy tools. Depending on the credibility of the processes of development and the values framework underpinning the initiative, the project outputs may gain the recognition of institutions that control policy and resources. In contrast, projects undertaken within institutions that control policy and resources, as well as being information tools, can actively direct policies and programs and monitor results for feedback purposes."

^17. Canada, Treasury Board Secretariat, Getting Government Right: Improving Results Measurement and Accountability: Annual Report to Parliament by the President of the Treasury Board, Ottawa: Queen's Printer, 1996.

^18. Canada, Treasury Board Secretariat, Accounting for Results 1997, Ottawa: Queen's Printer, 1997, (http://dsp-psd.pwgsc.gc.ca/Collection/BT1-10-1997E.pdf); Managing for Results 1998, Ottawa: Queen's Printer, 1998.

^19. See Ekos Research Associates, "The Use of Social Indicators as Evaluation Instruments," August 1998, p. 19; (http://www.hrsdc.gc.ca/en/cs/sp/sdc/evaluation/sp-ah064e/page09.shtml).

^20. See Treasury Board Secretariat, "Guide for the Development of Results-based Management and Accountability Frameworks," (http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rmafcgrr-eng.asp).

^21. In Canada, the concept of "horizontal results" has developed over the years; being called alternately "sectoral results" (Accounting for Results 1997, p. 19), "collective results" (Managing for Results 1998-2000), and "shared outcomes."

^22. Treasury Board Secretariat, "Companion Guide: The Development of Results-based Management and Accountability Frameworks for Horizontal Initiatives," June 2002; (http://www.tbs-sct.gc.ca/eval/tools_outils/comp-acc-eng.asp); also Treasury Board Secretariat, "Managing Collaborative Arrangements: A Guide for Regional Managers," (http://www.tbs-sct.gc.ca/fcer-cfre/documents/mca-gec/mca-gec-eng.asp).

^23. Treasury Board Secretariat, Preparation Guide Departmental Performance Reports 2003-2004, (http://www.tbs-sct.gc.ca/rma/dpr/03-04/guidance/guide-eng.asp#Toc70476510).

^24. Treasury Board Secretariat, Horizontal Results Database, (http://www.tbs-sct.gc.ca/rma/eppi-ibdrp/hrdb-rhbd/profil-eng.asp).

^25. Treasury Board Secretariat, Social Union Framework Agreement: Accountability, (http://www.tbs-sct.gc.ca/rma/dpr/01-02/Guidance/templates/soc-ent-eng.asp).

^26. Every year, as part of the documentation produced to support the appropriation of funds from Parliament, the government tables two sets of departmental reports in Parliament. In the spring, departments and agencies produce their Reports on Plans and Priorities for the coming fiscal year. In the fall, they provide Parliamentarians with their Departmental Performance Reports indicating achievements attained over the previous fiscal year.

^27. "Quality of Life - A Concept Paper," p. 9. The framework was tabled in Parliament in the President's Annual Report, Managing for Results 1999.

^28. "Quality of Life - A Concept Paper," p. 2. As an example of the relations of the three levels of information: "...the results achieved by a specific federal program, such as pre-natal care, contribute to the achievement of a shared goal articulated in the National Children's Agenda, and are intended to reduce infant mortality rates which is an indicator of QOL" (p. 10).

^29. "Quality of Life - A Concept Paper," p. 7.

^30. "Quality of Life - A Concept Paper," p. 2; see also p. 8: "this kind of transparency and accountability is the objective of the movement towards a values and results based, learning approach to modern public management which is increasingly reflected in the Departmental Performance Reports. QOL reporting could also be a component of modern public management, but in a way that clearly reflects governments' often limited ability as one player to directly affect selected QOL indicators. Thus, QOL reporting should not be seen as an instrument for government accountability (although some will choose to use it this way) but rather for informing the policy agenda. For a comprehensive approach to reporting, however, it could be complemented by program results and service delivery reporting by governments that does provide citizens with information on the specific actions of government which are ultimately intended to affect Canada's quality of life." On the "problem of attribution" with respect to social indicators, see also Ekos Research Associates, "The Use of Social Indicators as Evaluation Instruments," August 1998, p. 23; (http://www1.servicecanada.gc.ca/en/cs/sp/sdc/evaluation/sp-ah064e/page08.shtml); on the distinction between "performance" indicators which measure the performance of government programs and "social" indicators which measure societal-level outcomes, see Anoma Armstrong et al, "Difficulties of Developing and Using Social Indicators to Evaluate Government Programs: A Critical Review," (http://www.evaluationcanada.ca/distribution/ 20021030_armstrong_anona_francis_ronald_bourne_michael_dussuyer_inez.pdf).

^31. See Mayne, John, "Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly," Office of the Auditor General, June, 1999; (http://www.oag-bvg.gc.ca/domino/other.nsf/html/99dp1_e.html).

^32. See Speers, Kimberley, "Performance Measurement in the Government of Alberta: Propaganda or Truth," June 2004; (http://www.cpsa-acsp.ca/papers-2004/Speers.pdf).

^33. See Mayne, John, "Modernizing Accountability Practices in the Public Sector," January, 1998; (http://www.tbs-sct.gc.ca/rma/account/oagtbs-eng.asp); also Donald G. Lenihan, John Godfrey, Tony Valeri, and John Williams, "What is Shared Accountability," Policy, Politics and Governance vol 5, November, 2003; (http://kta.on.ca/pdf/ppg5.pdf); Donald G. Lenihan, John Godfrey, Tony Valeri, and John Williams, "Accountability for Learning," Policy, Politics and Governance vol 4, June, 2003; (http://kta.on.ca/pdf/ppg4.pdf); and Donald G. Lenihan, John Godfrey, Tony Valeri, and John Williams, "Horizontal Government: The Next Step," Policy, Politics and Governance vol 2, February 2003; (http://kta.on.ca/pdf/ppg2.pdf);.

^34. Canada, Treasury Board Secretariat, Canada's Performance 2001, Ottawa: Queen's Printer, 2001, (http://www.tbs-sct.gc.ca/report/govrev/01/cp-rc-eng.asp); Canada's Performance 2002, Ottawa: Queen's Printer, 2002, (http://www.tbs-sct.gc.ca/report/govrev/02/cp-rc-eng.asp); Canada's Performance 2003, Ottawa: Queen's Printer, 2003, (http://www.tbs-sct.gc.ca/report/govrev/03/cp-rc-eng.asp).

^35. On the importance of trend information, see "Quality of Life - A Concept Paper," p. 5. "Trend analysis for each indicator would have to be undertaken, since the most useful QOL measurement is not a one-time snapshot but rather an analysis of the change in the data over relevant time frames."

^36. This core set of 19 indicators formed the basis of the 2001 and 2002 reports. In 2003, an indicator of climate change (Greenhouse Gas Emissions) was added. For the 2004 report, we plan to greatly expand the coverage, adding two whole thematic areas and increasing the number of indicators tracked to thirty-two.

^37. As David Zussman points out, another objective of this type of reporting could be "building trust in government": "Greater information should pave the way for increased engagement in public policy debates, whether through government-led consultations or through active personal participation in policy formulation." ("Evidence-based Policy Making: Some Observations on Recent Canadian Experience," April 2003; http://www.ppforum.ca/fr/).

^38. In addition to focus groups with external experts, Canadians, and Members of Parliament, as an initial gauge of the usefulness and quality of the report, a short questionnaire was included in the print and electronic versions. These questionnaires were in keeping with the government's commitment to taking the views of Canadians into account in subsequent reports. Feedback on the report is also garnered through two of the Secretary of the Treasury Board's advisory boards: the External Advisory Panel consists of experts from other jurisdictions as well as public policy experts such as David Zussman and Judith Maxwell; senior public servants are also consulted via the Secretary's Advisory Committee of Deputy Ministers.

^39. "Quality of Life - A Concept Paper," p. 6.

^40. On the need for citizen engagement, see "Quality of Life - A Concept Paper," p. 8: "... QOL reporting is the further evolution beyond transparency and accountability to actually engaging citizens in the agenda setting and policy development process. Some have described this as an evolution in thinking about citizens not just as voters or consumers of public services but as partners in governance. Public opinion polling has clearly demonstrated that citizens want to participate in discussions about public policy issues of importance to them, particularly those involving fundamental values and tradeoffs among values, and that they are not satisfied with the traditional vehicles of information sharing and consultation that governments have used to involve citizens in the policy process." This type of reporting on quality of life and citizen engagement on indicators is also a way of forging shared understanding on our goals: "The final QOL indicators agreed upon must be relevant to citizens and reflect their values about what contributes to QOL. Citizens would have to be involved in the identification of the indicators and the actual measures used to assess performance. Experts would have to be engaged to ensure that the methodology behind the reporting is credible." ("Quality of Life - A Concept Paper," p. 2.). The Quality of Life Indicator Project (described on page 6 of this paper) is a good example of the value that citizen engagement has in indicator development.

^41. See the report released in May 2003, State of the Debate: Environment and Sustainable Development Indicators for Canada, available at http://www.nrtee-trnee.ca/eng/programs/Current_Programs/SDIndicators/index.html.

^42. See the federal report that ensued from this work: Healthy Canadians: the Latest Indicators from the National Perspective, available at http://www.hc-sc.gc.ca/iacb-dgiac/arad-draa/english/accountability/indicators.html.

^43. "Quality of Life - A Concept Paper," p. 6: "For example, the story on Canada's life expectancy data is a positive one at a national level, but life expectancy data for aboriginal persons tells a very different story. A related issue is whether data should be provided on a regional basis in order to be more relevant to citizens. This is technically feasible through designing a web-site on the broad indicators allowing users to drill down to get more detailed information." On the report's capacity to "drill down" to more detailed levels of information, see David Zussman, "New tool helps track how our money is spent," April 14, 2003 - Ottawa Citizen; (http://www.ppforum.ca/fr/).

^44. "Quality of Life - A Concept Paper," p. 3: "The literature acknowledges that such a concept has both a subjective and objective dimension. "Subjective quality of life is about feeling good and being satisfied with things in general. Objective quality of life is about fulfilling the societal and cultural demands for material wealth, social status, and physical well-being." (QOL Research Centre, Denmark). This presents a challenge to the measurement and reporting of QOL. For example, a subjective indicator, such as a satisfaction survey, might demonstrate that a group of people reports a high level of QOL, while objective indicators of health, housing, income, and education for the same population might suggest a lower level of QOL, perhaps as compared to other people. Which is the most relevant measure of QOL and which should guide agenda setting and policy making?".

^45. See David Zussman, "New tool helps track how our money is spent," April 14, 2003 - Ottawa Citizen; (http://www.ppforum.ca/fr/; and CCAF Update: Canada's Performance 2001; ( http://www.ccaf-fcvi.com/english/index.html).

^46. "Quality of Life - A Concept Paper," p. 4.

^47. The idea of aligning Canada's Performance more closely to the Government's planning and accounting cycle was presented in the context of societal indicator reporting in other jurisdictions to the External Advisory Panel, where it was well received. The other jurisdictions presented at this meeting were: Australia's Measuring Australia's Progress; New Zealand's The Social Report, the UK's Achieving a Better Quality of Life, Minnesota State's Minnesota Milestones, and Alberta's Measuring Up. Reporting in this way seems to be inline with accepted best practices in other jurisdictions and may facilitate results-based appropriations in the future.

^48. Assigning financial information to the current reporting framework of Canada's Performance may prove to be a complex task when dealing with programs whose activities contribute to reaching more than one Government of Canada Outcome. For example, one social program may lead to a positive outcome in the areas of health and community, but trying to allocate an exact monetary value to these two outcomes is difficult.

^49. Canada. Treasury Board of Canada, Results for Canadians: A Management Framework for the Government of Canada, Ottawa: Queen's Printer, 2000, p. 13.

^50. Canada. Treasury Board of Canada, Secretariat. 2004-05, A Report on Plans and Priorities, p. 29; (http://www.tbs-sct.gc.ca/est-pre/20042005/TBS-SCT/pdf/TBS-SCT-eng.pdf).

^51. We are beginning to gain ground, however, in this area as well. For instance, over the last year, TBS has worked with departments to build a comprehensive framework to account for contributions made by the government to the societal-level goal of reducing the effects of climate change - (i.e., by reducing GHG emissions). Similarly, the government has committed to providing a report card on Aboriginal peoples; as a result, TBS is working with departments to build a horizontal framework capturing the contributions made by all federal departments and agencies to the quality of life of Aboriginal peoples. These developments build on the earlier work on "horizontal frameworks" (e.g., children) described earlier.