Treasury Board of Canada Secretariat
www.tbs-sct.gc.ca
Common menu bar links
Breadcrumb Trail
ARCHIVED - Evaluation Framework for Early Departure Programs - Number 11
This page has been archived.
Archived Content
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
Executive Summary
Introduction
Consistent with recent federal guidelines to better define new initiatives at an early stage, particularly in terms of measurable objectives, indicators of performance, and requirements for information collection to measure performance, the Treasury Board Secretariat commissioned the development of an evaluation framework for Early Departure Incentive (EDI) and Early Retirement Incentive (ERI). The framework is intended to support operational and corporate management as well as provide the groundwork for an effective and efficient future evaluation of the initiatives. The evaluation framework for EDI and ERI is described in this report. It has been developed in accordance with current best practices for evaluation, consistent with Treasury Board policies and practices.
Methodology
The preparation of the evaluation framework involved a series of consultations with Treasury Board Secretariat representatives as well as a series of workshops with representatives of "most affected" and "less affected" departments. We also reviewed a series of documents related to the programs, the Work Force Adjustment Directive and other related initiatives. The study was conducted in February and March 1996 in preparation for an evaluation of the EDI and ERI programs during the 1998-99 fiscal year.
Issues
A total of twenty evaluation issues, categorized according to Treasury Board guidelines of relevance, success and cost-effectiveness, were developed.
Evaluation Options
Three evaluation options were developed and put forward for consideration. The table that follows summarizes those options.
|
Option |
Methodology |
Strengths |
Limitations |
Cost |
|
1 |
15 telephone interviews with personnel representatives in departments - 15 telephone interviews with finance representatives in departments - no interviews with other representatives in departments - 5 telephone interviews with representatives of TBS - TBS file review limited to program statistics - no departmental file review - no industry consultations |
- inexpensive - entire study can be done fairly quickly - limited burden on departments - all issues covered to some extent |
- not all departments would be consulted - limited coverage of issues - less reliable |
$20,000 to $30,000 |
|
Option |
Methodology |
Strengths |
Limitations |
Cost |
|
2 |
- up to 30 telephone interviews with personnel representatives from all departments - only in headquarters - up to 30 telephone interviews with finance representatives from all departments -only in headquarters; - no interviews with other representatives in departments - up to 10 telephone interviews with representatives of TBS - TBS file review to include a review of program statistics and of documents supporting advice and support - departmental file review to include statistics on grievances, as well as extensive costing information - no industry consultations |
- relatively inexpensive - entire study can be done fairly quickly - improved coverage of issues - improved reliability |
- increased burden on departments |
$50,000 to $60,000 |
|
Option |
Methodology |
Strengths |
Limitations |
Cost |
|
3 |
- up to 50 telephone and/or interviews with personnel representatives from all departments - in headquarters and in the regions - up to 50 telephone interviews with finance representatives from all departments -in headquarters and in the regions - up to 15 interviews with other representatives in departments (DMs and their delegates) - survey of 500 affected employees; - up to 15 telephone interviews with representatives of TBS - TBS file review to include a review of program statistics and of documents supporting advice and support - departmental file review to include statistics on grievances, as well as extensive costing information (i.e., costing study) - up to 15 industry consultations |
- excellent coverage of issues - improved reliability |
- expensive - would take several months to complete - increased burden on departments |
$100,000 to $150,000 |
The evaluation could also be broadened to include an evaluation of the relevance, success and cost-effectiveness of the program from a federal government perspective. While this would meet with resistance from several of the departments consulted during the course of this study, two options were developed to provide an overview of what would be required in such a case. However, issues were not identified for these options as they were outside the scope of this study.
Management's Response
The February 1995 Budget announced government-wide plans to eliminate programs and services identified in the 1994-95 Program Review. This elimination of programs will result in a reduction of the federal government workforce by an estimated 45,000 positions by the third year (1997-98). In conjunction with the 1995 budget announcement, measures were introduced to help departments meet their reduction targets. Two programs, the Early Retirement Incentive (ERI) and the Early Departure Incentive (EDI), were designed to assist departments in managing downsizing. Additionally, the Government has designated certain departments as "most affected" in that they are deemed unable to manage job reductions through normal attrition, existing workforce adjustment measures, and other management tools. "Most affected" departments have access to both programs, while all others have access only to the ERI.
These two programs were not intended to replace the existing Work force Adjustment Directive (WFAD), but rather to complement it. The WFAD remains in force and is still the primary work force adjustment tool. Two additional programs, Leave With Income Averaging and Pre-retirement Transition Leave, were also announced to provide non-surplus employees with more flexible working arrangements while assisting managers in achieving salary budget reductions.
Several actions have been taken by Treasury Board Secretariat (TBS), in light of observations made following the Audit of the Work Force Adjustment Directive by the Auditor General in 1992. The Secretariat was extremely sensitive to the need to ensure that proper policy, audit and monitoring mechanisms would be in place not only for the WFAD but also for the two new programs. To this end, the Secretariat has provided departments with several documents, for use in managing and monitoring downsizing activities:
- An Employment Adjustment Accountability Framework was created by Treasury Board Secretariat, focusing on the role and accountability of Deputy Ministers. A letter to all Deputy Ministers was prepared to bring to their attention human resources and financial management and accountability issues and a pay-back indicator which will ensure better management of taxpayer dollars provided out of the Central Reserve. In this letter, deputies were called upon to perform an audit of the departure incentives no later than half way through the three year program
- A Manager's Guide to employment adjustment was prepared in collaboration with departments, the audit and review community and TB Secretariat.
- An Audit Guide was developed and has been distributed to departments for their use.
- A Work Force Adjustment Departmental Monitoring Framework was issued.
Consistent with recent federal guidelines to better define new initiatives at an early stage, particularly in terms of measurable objectives, indicators of performance, and requirements for information collection to measure performance, the Treasury Board Secretariat commissioned management consultants to develop an evaluation framework for EDI and ERI. The framework is intended to support operational and corporate management as well as provide the groundwork for an effective and efficient future evaluation of the initiatives. The study was conducted in preparation for an evaluation of the EDI and ERI programs during the 1998-99 fiscal year.
Management's Response
The Secretariat accepts the evaluation framework which was developed by the management consultants, in accordance with current best practices for evaluation and TBS policies and practices. The framework, developed in consultation with representatives from both "most affected" and "least affected" departments, presents potential issue areas, information requirements and possible approaches. As a basis for evaluating TBS policies and guidelines it serves as one element of an overall accountability and evaluation structure. Elaboration of this evaluation framework is part of an ongoing comprehensive monitoring and evaluation of the operational and corporate management of, and performance of, the EDI and ERI programs.
In addition to initiating the development of this evaluation framework, TBS has provided departments with an audit guide and it has communicated with Deputy Heads regarding a departmental monitoring framework for Human Resources and Financial management, including a 12-month pay-back indicator. These financial management and accounting components will provide a key indicator for program effectiveness - the degree to which public funds expended on the early departure programs achieved the results intended. Departments' own internal audits of their performance and measurement against the Monitoring Frameworks will also assess whether downsizing initiatives are being managed strategically and in compliance with policy parameters at the departmental level.
The Secretariat will be considering the evaluation options presented, and will make a decision regarding the most appropriate evaluation methodology early in 1998. A final decision will be based on the most cost-effective approach, given the extent to which there appear to be significant areas of concern following an assessment of departmental audits and reports.
1. Introduction*
* This study was conducted by an external firm.
Consistent with recent federal guidelines to better define new initiatives at an early stage, particularly in terms of measurable objectives, indicators of performance, and requirements for information collection to measure performance, the Treasury Board Secretariat commissioned the development of an evaluation framework for Early Departure Incentive (EDI) and Early Retirement Incentive (ERI). The framework is intended to support operational and corporate management as well as provide the groundwork for an effective and efficient future evaluation of the initiatives. The evaluation framework for EDI and ERI is described in this report. It has been developed in accordance with current best practices for evaluation, consistent with Treasury Board policies and practices.
The February 27, 1995 Budget announced government-wide plans to eliminate programs and services identified in the 1994-95 Program Review. This elimination of programs will result in a decrease of the federal government's workforce by an estimated 45,000 positions and, by the third year (1997-98), would result in annual savings of $7.2 billion. In the 1996 Budget, it was announced that through Program Review II, annual savings would be increased to $9.2 billion by 1998-99. At the time of this report, the impact of Program Review II on the number of employees had not been determined.
The Government established a threshold leading to the Public Service being "most affected" into two types of departments. Departments facing expenditure reductions beyond this threshold and having to manage significant employment adjustment are being designated as "most affected" departments in that they are deemed unable to manage job reductions through normal attrition, existing workforce adjustment measures, and other management tools. To date (March 15, 1996), fourteen departments and agencies in the Public Service have been designated as most affected. All other departments are considered less affected.
In conjunction with the 1995 Budget announcement, measures were introduced which would help departments meet their reduction targets. Two programs, ERI and EDI, were designed to assist departments and other organizations in managing downsizing as a result of the Program Review. "Most affected" departments have access to both programs, while all others only have access to ERI.
These programs were not intended to replace the existing Work Force Adjustment Directive (WFAD), but rather to complement it. In fact, the WFAD remains in force and is still the primary work force adjustment tool. Two additional programs, Leave With Income Averaging and Pre-retirement Transition Leave, were also announced to provide non-surplus employees with more flexible working arrangements while assisting managers to achieve salary budget reductions.
In April 1995, all federal government departments were able to offer ERI to eligible surplus employees which, until July 1995, included an additional 15-week separation payment. A policy for the Management of Alternates was put into place in May to allow employees who would be declared surplus but who wanted to continue working for the government to exchange jobs with another employee who would be willing to leave the Public Service. In July, "most affected" departments were able to offer EDI, and employees were given 60 days following surplus notification to opt for this incentive.
The objective of the current federal government review policy, as amended July 1994, is to ensure that the government:
- "has timely, relevant, and evidence-based information on the performance of its policies, programs, and operations, including the results they achieve; and,
- uses this information to improve the management and cost-effectiveness of policies, programs, and operations, and to account for results."
It is government policy that:
- "managers monitor key aspects of the performance of their programs and operations;
- departments and agencies conduct evaluations, audits, and other reviews to assess the performance of their policies, programs, and operations; and ...
- departments, agencies, and central agencies use the results of reviews and other performance information in decision making to provide cost-effective programs ... and foster good stewardship and accountability in the use of public funds."
The purpose of this evaluation framework is to establish a basis for the management and evaluation of EDI/ERI and to assist Treasury Board managers to meet the objectives of government review policy through the collection, analysis, and reporting of key performance information. The framework is designed to contribute to operational and corporate management and central agencies requirements for performance information as described in the previous section.
The framework identifies key program activities/outputs, intended client groups, and impacts. The framework is intended to present potential issues, data requirements, and analytical approaches for a credible evaluation and to support the ongoing operational and corporate management of EDI/ERI through the systematic identification and analysis of key performance characteristics of EDI/ERI. The indicators of performance are based on practical, credible information sources.
During the development of the framework, workshops were held with representatives of both "most affected" and "less affected" departments. These workshops provided valuable insights into many practical aspects of the programs from those responsible for implementing the programs in their own departments.
This Evaluation Framework addresses solely (and is solely limited to) the relevance, success and cost-effectiveness of the TBS policies and guidelines as they are affected by the operating environments of the departments. That is, the study is limited to what was done by TBS or what was done by the departments because of TBS, and not how the departments managed the programs.
Naturally, it is recognized that this is but one part of an overall accountability and evaluation structure.
The prime evaluation parameter and prime indicator of program effectiveness in the global evaluation of these departure programs is whether the public monies expended on these new programs achieved the results promised in the timeframe predicted. This is being assessed by financial management and accounting components.
The secondary evaluation parameter is whether the departments who severed employees under the provisions of the new EDI and ERI programs did so wisely, judiciously, appropriately, and in accordance with policy parameters. This will be measured by departments' own internal audits of their performance, as well as by their self-measurement against the Monitoring Framework. (Guides to both Audit and Monitoring have been supplied to departments by the Treasury Board Secretariat.)
Therefore, the true criterion of Treasury Board Secretariat involvement in the Evaluation Framework should reflect the following realities:
- That the key and crucial TBS role in this area is in creating the environment for program delivery and management, that is, clarity of policies, communication with departments, booklets, letters of instruction, leadership in committees, guidance, control guides such as the Audit and Monitoring frameworks.
- That departments have the responsibility for actual management at the micro level of the programs, with which the TBS cannot and should not become involved.
An important constraint in the development of an evaluation framework of EDI/ERI at this early stage is the evolving nature of the initiative and the ongoing reductions in federal government expenditures. For example, since the start of this study, the 1996 Federal Budget, which has an impact on the programs, was announced. Also, new departments have been periodically added to the list of "most affected" departments. This framework is therefore based on the federal government environment as of March 15, 1996.
2. Profile of the Early Departure Programs
The information in the sections which follow was taken verbatim from various Treasury Board documents. However, as per the identified the constraints, these documents may be updated, revised, adjusted over the course of the programs and may therefore not be accurate at a later date.
2.1.1 Early Retirement Incentive
Source: Treasury Board of Canada Secretariat, Managers Guide to Employment Adjustment, March 1995.
The ERI program was designed to lessen the financial impact of early retirement on the employees who have been declared surplus and are between the ages of 50 and 54, inclusive. Without this program, employees between 50 and 54 years of age taking early retirement from the Public Service would face reductions in pension benefits of 5% per year below 55, or by up to 25%. Eligible employees who choose the ERI option will not have their pensions reduced. All normal terms and conditions applicable to pensions paid under the Public Service Superannuation Act (PSSA), including indexing, will apply to the unreduced pensions payable under ERI. However, ERI benefits permanently cease if a recipient of ERI is rehired and again becomes a contributor under the PSSA.
The incentive is available to surplus indeterminate employees over a three-year period from April 1, 1995 to March 31, 1998 in all departments and organizations for whom the Treasury Board is the employer. The ERI is also available to indeterminate surplus executives in these departments and organizations. To qualify, an employee must meet ALL of the following conditions:
- The employee must be declared surplus during this three-year period. Any employee who was already surplus on April 1, 1995 must have been issued a new notice of surplus to become eligible for the program;
- The employee must have at least five years of pensionable service;
- The employee must have at least 10 years of Public Service employment. Part-time employment may count for this purpose;
- The employee must be at least 50 years old but not yet 60 by the time he or she leaves the Public Service.
The employee must also:
- choose the ERI option within 60 calendar days of the date of issue of the surplus notice and must leave the Public Service within six months of that notice or by the end of a longer period that the deputy head has decided is required for operational reasons;
- opt for an annual allowance under the Public Service Superannuation Act (PSSA) before leaving the Public Service;
- not have chosen the Early Departure Incentive (EDI), a payment under the Department of National Defence (DND) Civilian Reduction Program (CRP) or, if the employee is an executive, a negotiated settlement under the Executive Employment Transition Policy (EETP);
- if subject to the Workforce Adjustment Directive (WFAD), not have received a reasonable job offer before leaving the Public Service;
- if subject to the EETP, not have received an offer of alternative Public Service employment; and
- not have received an offer of continuing employment with the new employer in the case of work that is transferred outside the Public Service, such as through devolution to other levels of government, commercialization or privatization.
Employees entitled to ERI are also eligible for:
- supplementary death benefit plan coverage at regular employee rates (must elect to continue coverage);
- pay in lieu of unfulfilled surplus period (maximum six months' pay);
- separation benefit (maximum 15 weeks' pay) until WFAD was amended through legislation (summer of 1995);
- severance pay at lay-off rate (two weeks' pay for first complete year, and one week for each additional complete year of continuous employment [minus any amount received in severance from previous lay-off] and for executives, one week's pay for each completed year of service to a maximum of 28 weeks' pay, reduced by the number of weeks of severance pay previously granted).
Based on a survey of departments in the Spring of 1995, where the projected 45,000 FTE impact was reaffirmed, the actuarial value of the waiver of pension reductions under ERI was projected at $800 million.
2.1.2 Early Departure Incentive
Source: Treasury Board of Canada Secretariat, Managers Guide to Employment Adjustment, March 1995.
In "most affected" departments, the Government suspended the guarantee of a reasonable job offer before any possibility of lay-off, but to make up for that introduced a cash-based EDI.
During the three-year period after the 1995 Budget legislation was proclaimed, the EDI will only be available in the "most affected" departments which, as of March 15, 1996, are:
- Agriculture and Agri-Food;
-
- Canadian Grain Commission
- National Farm Products Marketing Council
- Prairie Farm Rehabilitation Administration
- Canadian Transportation Accident Investigation and Safety Board;
- Citizenship and Immigration;
- Environment;
- Finance;
- Fisheries and Oceans;
- Grain Transportation Agency;
- Human Resources Development;
- Industry;
- National Defence;
- Natural Resources;
- Public Works and Government Services;
- Transport; and,
- Treasury Board Secretariat.
Only indeterminate employees who have been declared surplus are eligible. For executives, the Executive Employment Transition policy parallels EDI and already provides them with comparable benefits. In addition, employees who choose the ERI are not eligible for the EDI.
Under the EDI, eligible employees receive a cash payment if they resign from the Public Service. The cash payment includes severance pay at the lay-off rate, as well as access to an education or training allowance if the employee is eligible. The payment is smaller for employees who have less than five years of service and for employees who are entitled to an unreduced pension. Employees who take the EDI are not eligible for pay-in-lieu of an unfulfilled surplus period or any lump-sum payment under the WFAD.
Surplus employees in "most affected" departments who choose not to take the EDI are, under the terms of the modified WFAD, on paid surplus status for a minimum of six months. If they do not receive a reasonable job offer within six months of being declared surplus, they may be placed on unpaid surplus status for twelve months. If they do not receive a reasonable job offer at the end of this period they are laid off.
Each employee's EDI package includes up to three of the following components (the total payment of the first two components cannot exceed a defined maximum).
1. Severance pay: Employees receive two weeks' pay for the first complete year of continuous employment in the Public Service, and one week for each additional complete year.
2. Lump-sum payment:
a) Base payment: Employees receive 39 weeks of regular pay if they have less than five years of continuous Public Service employment; 52 weeks of regular pay if they have more than five years of continuous Public Service employment and are not eligible for an immediate unreduced pension; and up to 52 weeks if they have more than five years of continuous Public Service employment and are eligible for an unreduced pension.
b) Service allowance: This component does not apply to employees who have less than five years of continuous Public Service employment or to employees who are entitled to an unreduced pension. It consists of additional weeks of regular pay based on an employee's age and continuous years of employment. If the total of an employee's age and his or her continuous years of employment is between 50 and 54, the employee is entitled to one additional week of regular pay; if this total is between 55 and 59, the employee gets two additional weeks; if it is between 60 and 64, the employee get three weeks; if it is between 65 and 69, the employee gets four weeks; if it is between 70 and 74, the employee gets five weeks; and if it equals 75 or more, the employee gets six weeks of regular pay.
3. In addition, employees may qualify for an education and training allowance: This component does not apply to employees receiving pension benefits of any type under the Public Service Superannuation Act (PSSA). It consists of reimbursing the employee for up to $7,000 incurred by him/her in training programs in preparation for new employment outside the Federal Public Service (upon presentation of receipts).
Based on a survey of departments in the Spring of 1995, the cash cost associated principally with EDI was estimated at $1.5 billion.
2.1.3 Work Force Adjustment Directive
For the past 25 years, federal public servants declared surplus to departmental needs have had some form of protection to mitigate the effects of potential lay-offs. The initial measures to reduce the impacts of adjustments in the federal government work force were called Manpower Adjustment Procedures. These procedures were ratified in 1969 and laid the foundation for the current Work Force Adjustment Directive.
The changes leading to the present framework of this Directive can generally be traced back to 1986 when provisions for payment in lieu of unfulfilled surplus were sanctioned and became part of the revised Work Force Adjustment Policy following the May, 1985 Budget calling for a reduction of the Public Service by 15,000 person-years.
Since that time, approximately 30,000 employees have been directly affected by Work Force Adjustment. In December 1991, the Work Force Adjustment Policy was enhanced, upon recommendation of the National Joint Council and became known as the Work Force Adjustment Directive (WFAD). It strengthened the basic provisions of the former policy. It also complemented the federal government's initiatives in contracting out, privatization and devolution of services, by providing affected employees with additional alternatives such as lump-sum payments. As well, the Directive guaranteed a reasonable job offer to every indeterminate employee whose services would no longer be required because of a work force adjustment, and in the case of privatization or contracting out, the job offer had to be in the same geographic area. The Directive also provided for open-ended salary protection and the opportunity for non-affected employees to volunteer to become surplus in place of affected employees.
WFAD not only recognized the importance of keeping employees in the Public Service, but also the importance of providing employees with an opportunity to leave, thereby recognizing the necessity for on-going reductions in the Public Service.
The Improved Reporting to Parliament project emphasizes a focus on results and performance measurement. One aspect of this emphasis involves the development of a performance framework, which is a widely used tool to help set the "big picture" for a department, key activity or line of business.
Treasury Board Secretariat, Reform of the Estimates: Guidelines for Improving Expenditure Management Information to Parliament, Draft, January 25, 1996.
The performance framework for the early departure programs (EDI and ERI combined) is provided on the following page. Keeping in mind that the framework sets the big picture, it is the intent of the framework to help the reader understand that all activities and outputs should be undertaken with common targets for reach and outcomes or results. Therefore, while the activities and outputs are linked and can therefore be read in linear fashion, the reach and impacts apply to most or all activities and outputs and are therefore not linearly related.
Further details on this framework are outlined in the following paragraphs. It is important to note that the purpose of an evaluation framework is not to analyze and describe the items in the framework in detail but rather to provide an overview of how things are done (activities and outputs) by the Treasury Board Secretariat, to or for whom (reach), what we want to do (immediate impacts or outcomes) and why (intermediate/long-term impacts).
|
Departure Incentive Programs |
|||||||||||
|
Mission: To assist departments in their downsizing efforts through departure incentives that fairly and cost-effectively deal with surplus employees leaving the Public Service. |
|||||||||||
|
Activities |
Outputs |
Reach |
Immediate Impacts |
Int. / L.T. Impacts |
|||||||
|
- develop and promulgate policies and guidelines
- coordinate and/or participate in committees/
-on-going monitoring - development of a tracking and reporting system - manage EDI / ERI funding reserves - risk management
- advice and support |
- policies - guidelines - reporting formats
- MAD Personnel Committee and Sub-Committee - WFAD Network Coordinators - Senior Financial Officers Forums - Heads of Audit Meetings - Joint Adjustment Committees (JACs) - reports - quarterly report - pay report - alternate report
- funding
- risk management framework - audit guides - monitoring guides - delegated authorities (DMs)
- interpretation and arbitration (adjudication) of policies / guidelines - information bulletins - informal communications - hot line / help line |
Clients - Deputy Ministers and their designates - Most Affected Departments Co-delivery Agents - other government departments - Human Resources - Finance (Pay) - Audit/ - PWGSC Stakeholders/ - employees - general public / taxpayers - press - Parliament - unions - private sector businesses |
- satisfaction with various aspects of the services provided by Treasury Board Secretariat: - clarity of information - timeliness - knowledge of policies and guidelines - clear understanding of policies and guidelines - adherence to policies - sufficient and appropriate communications initiated and maintained with affected departments - policies and guidelines to assist departments in: - achieving reduction targets set out in their business plans - eliminating positions, functions and salary equivalents - being cost-effective - ensuring probity - ensuring equitable treatment - minimizing impacts on affected employees |
- strategic operational objectives of departments are met, as set out in the program review - EDI / ERI anticipated take-ups are met |
|||||||
Activities are those things done by TBS in order to carry out its desired mandate. Activities typically generate costs of some kind. Outputs are those things that one can count and that occur as a result of activities.
For the early departure programs, activities and outputs include:
- development and promulgation of policies and guidelines including reporting formats;
- coordination of and/or participation in committees and meetings including the MAD Personnel Committee and sub-committee meetings (attendance only), WFAD Network Coordinators, Senior Financial Officers Forums, Heads of Audits Meetings, and Joint Adjustment Committee (JAC) meetings;
- on-going monitoring through the development of a tracking and reporting system and resulting in quarterly, pay, and alternate reports;
- management of EDI/ERI funding reserves;
- risk management through the development of risk management frameworks such as audit guides and monitoring guides, and through delegated authorities;
- advice and support including the interpretation and arbitration of policies and guidelines, information bulletins, informal correspondence and a hot-line/help- line.
A list of the letters, additional instructional and guidance material generated by Treasury Board Secretariat, as of March 1996, is provided in Appendix A.
TBS conducts activities related to other workforce adjustment, however they are outside the scope of this study. An example includes the management of the federal government's relationship with the unions.
Reach relates to the group, or groups, which are affected or have been impacted by the programs. This may include clients, as well as, co-delivery agents (those that help TBS in the delivery of the programs), and other stakeholders (those that have a stake or interest in the programs but who are not directly involved in their design and delivery).
For the early departure programs, clients are the Deputy Ministers, and their designates, of all departments for the ERI, and those of most affected departments for the EDI.
Co-delivery agents include Human Resources, Finance, and Audit/Evaluation/
Review groups in the departments. Public Works and Government Services Canada
must also issue the cheques for the employees taking advantage of one of the two
programs, therefore, it is also identified as a co-delivery agent.
Stakeholders/interested parties include the employees (who may or may not be eligible for one of the programs), the general public or taxpayers (who, as for most federal government programs and services, may have an interest in the fairness and cost-effectiveness of the programs), the press (who receives the programs' press releases), Parliament, unions, and private sector businesses (because they may be affected by the influx of government employees, and/or because they may be pressured in offering similar incentives).
Immediate impacts or outcomes occur in the group(s) reached by the program outputs. Typically, these outcomes are perceptual, attitudinal and/or behavioral responses on the part of the group(s) reached. For the early departure programs, these include satisfaction with various aspects of the services provided by TBS (perceptual), a clear understanding of the policies and guidelines, the belief that sufficient and appropriate communications were initiated and maintained with affected departments (attitudinal) and adherence to policies (behavioral).
The policies and guidelines should also assist departments in:
- achieving the reduction targets set out in their business plans;
- eliminating positions, functions and salary equivalents;
- being cost-effective;
- ensuring probity;
- ensuring equity; and,
- minimizing impacts on affected employees.
It is important to note that the policies and guidelines can only assist departments. How the department operates given those policies and guidelines cannot necessarily be attributed to TBS. Also, other initiatives are in place in the departments to help them in achieving their individual program review goals. It would therefore be rather difficult, if not impossible, to determine the extent to which the two incentives alone contributed to the individual department's success.
However, under TBS's management and accountability framework for the programs, departments, agencies and Crown corporations will be held accountable for the prudent and cost-effective management of EDI and ERI. The test of cost-effectiveness will be a twelve-month payback on the total expenditures associated with the special expenditures associated with the special departure programs. Payback refers to the period of time necessary to recoup the expenditure, through savings in the organization's salaries and benefits. It is calculated as the ratio of the total departure costs incurred over three years to the change in expenditures on wages, salaries and benefits between 1994-95 and 1997-98.
As outlined in the framework, departments must also make available accurate, timely and reliable information on:
- the number of surplus employees (including executives), by province of work, occupational group, gender, age, first official language and indicating whether they are members of employment equity groups;
- the number of employees in unpaid surplus status (USS);
- the number of placements, acceptance of EDI and ERI, and lay-offs following USS or refusal of a reasonable job offer; and,
- new indeterminate staffing, attrition, leave without pay, terms, part-time and, where feasible, services contracts.
2.2.4 Intermediate and Long-Term Impacts
For the early departure programs, intermediate and long-term impacts include the achievement of the strategic operational objectives of the departments, as set out in the program review. Also, the EDI and ERI anticipated take-up should be met.
3. Issues, Indicators and Data Sources
Issues were classified according to Treasury Board categories of relevance, success and cost-effectiveness. With respect to EDI and ERI, the following issues have been identified during the course of interviews and workshops with both TBS and departmental representatives. It is important to note that the issues raised focus more on best practices rather than on changes required since EDI/ERI are three year programs that will sunset prior to the conduct of an evaluation.
Relevance issues are meant to respond to the all encompassing question: does the program make sense? Issues identified are:
1. Did the programs, as designed, meet the needs of the departments?
2. Were there other more effective ways of helping departments meet their workforce reduction targets, which can be kept in mind in designing future programs of a similar nature?
Success issues relate to the effectiveness of programs in meeting their objectives, within budget and without significant unwanted outcomes. The questions raised focus on what happened as a result of the programs and whether they have achieved their intended results. They include:
3. How satisfied are clients with the important aspects of the services provided by TBS?
4. Did departments obtain the services they expected and needed from TBS? If not, what more was expected and/or needed?
5. Was there a clear understanding of the policies and guidelines provided by TBS? What was unclear? What happened as a result of the lack of clarity in the policies and guidelines?
6. Were sufficient communications initiated and maintained with affected departments? Were these communications appropriate? If not, what were the negative impacts?
7. Was the advice and support provided appropriate? consistent? If not, what were the impacts?
8. Were the downsizing targets met? If yes, to what extent did the policies and guidelines related to the EDI/ERI assist departments in achieving the targets? If no, to what extent did the policies and guidelines hinder departments in achieving the targets?
9. Did the policies help departments ensure probity? If yes, in what way? If no, why not?
10. Did the policies help departments ensure equitable treatment? If yes, in what way? If no, why not?
11. To what extent did the policies and guidelines assist departments in minimizing the impacts on affected employees?
12. What were the major benefits of the programs?
13. What were the negative impacts of the programs?
14. How could the programs have been improved?
In general, cost effectiveness issues deal with how well government programs and operations perform and whether they are the most appropriate and effective means of achieving the objectives relative to alternative design and delivery approaches. In essence, are there better ways of achieving the intended results? As noted, cost-effectiveness issues surrounding EDI/ERI relate to the calculation of pay-back. Specific issues raised include:
15. Overall, was the twelve month payback period target achieved? If no, why not?
16. Was the twelve month payback period target achieved in all departments? If no, why did some departments achieve it, and why did some others not achieve it?
17. Were the cost elements included in the twelve month payback period calculation complete? If not, what others should/could have been included?
18. Was the twelve month payback period a reasonable measure? Would others have been more relevant, useful, reliable, meaningful, etc.?
19. Did implementation of the programs require the development of new systems for monitoring? What were the costs of these systems? Could these costs have been avoided? How?
20. Was there another more cost-effective way of achieving the results achieved through these programs while still ensuring fairness? What were they?
3.2 Issues, Indicators and Data Sources Matrix
The table over the next several pages delineates the issues, their indicators and appropriate data sources to address each indicator. The table also indicates the level of reliability of the source in addressing the issue.
To summarize, the data sources include:
- Interviews with representatives of Personnel/Human Resources in "most affected" departments to deal with issues related to EDI and ERI and in "least affected" departments to deal with issues related to ERI and to a very limited extent, EDI.
- Interviews with representatives of the Finance departments in all departments.
- Interviews with others in departments. Depending on the issues, these could include Deputy Ministers and/or their delegates, managers, or affected employees.
- Interviews with representatives of Treasury Board. These relate to representatives involved in the on-going activities related to the policies and guidelines, not those involved in implementing the policies and guidelines within TBS as a "most affected" department. These TBS representatives are included in the "Departments" group of interviewees.
- Review of files and other documents maintained by TBS. These include statistics, correspondence with departments, on-going monitoring reports, etc.
- Review of files and other documents maintained by departments. These include personnel (grievances, etc.) and finance statistics (cost of systems, etc.), and other relevant documents (audit reports, etc.).
- Industry refers to interviews with representatives of the private sector to obtain information on other downsizing programs.
|
Evaluation Issue |
Indicators |
Data Sources |
||||||
|
Departments |
TBS |
Indus- |
||||||
|
Person- |
Finance | Others |
TBS |
Depart- |
||||
| Relevance | ||||||||
| 1. Did the programs, as designed, meet the needs of the departments? | What are the needs of departments? |
High |
Medium | Medium | ||||
| Which needs were met through the programs? | High | Medium | Medium | |||||
| Were other needs met some other way, or were there gaps? | High | Medium | Medium | |||||
| 2. Were there other more effective ways of helping departments meet their workforce reduction targets, which can be kept in mind in designing future programs of a similar nature? | Perceptions/ opinions of clients and stakeholders |
Medium | Medium | Medium | Low | Low | ||
| Success | ||||||||
| 3. How satisfied are clients with the important aspects of the services provided by TBS? | Identification of TBS service features which are important to clients | High | High | High | ||||
| Satisfaction of clients with the TBS on the service features which are important | High | High | High | |||||
| 4. Did departments obtain the services they expected and needed from TBS? If not, what more was expected and/or needed? | Services expected and/or needed from TBS | High | High | High | ||||
| Services obtained from TBS |
High |
High | ||||||
| Gap | High | High | ||||||
| Impact of not obtaining certain expected and/or needed services | High | High | ||||||
| 5. Was there a clear understanding of the policies and guidelines provided by TBS? What was unclear? What happened as a result of the lack of clarity in the policies and guidelines? | Perceptions/opinions of departments | Medium | ||||||
| Cases of litigation/union grievances related to EDI/ERI | Low | High | ||||||
| Other reported impacts | High | |||||||
| 6. Were sufficient communications initiated and maintained with affected departments? Were these communications appropriate? If not, what were the negative impacts? | Perceptions/opinions of departments regarding the amount and appropriateness of the communications | Medium | Medium | |||||
| Reported impacts | Medium | Medium | ||||||
| 7. Was the advice and support provided appropriate? consistent? If not, what were the impacts? | Perceptions/opinions of department regarding the appropriateness and consistency of advice and support | Medium | Medium | Medium | ||||
| Reported impacts | Medium | Medium | Medium | |||||
| Advice and support provided | Medium | High | ||||||
| 8. Were the downsizing targets met? If yes, to what extent did the policies and guidelines related to the EDI/ERI assist departments in achieving the targets? If no, to what extent did the policies and guidelines hinder departments in achieving the targets? | Were downsizing targets met? | High | ||||||
| #s who took EDI and ERI | High | |||||||
| Perceptions of departments regarding how policies and guidelines helped/hindered | Medium | Medium | Medium | |||||
| 9. Did the policies help departments ensure probity? If yes, in what way? If no, why not? | Perceptions/opinions of departments | Medium | Medium | Medium | ||||
| Reported probity as per departmental audits of EDI/ERI | High | |||||||
| 10. Did the policies help departments ensure equitable treatment? If yes, in what way? If no, why not? | Perceptions/opinions of departments | Medium | ||||||
| Cases of litigation/union grievances related to EDI/ERI | Medium | |||||||
| 11. To what extent did the policies and guidelines assist departments in minimizing the impacts on affected employees? | Perceptions/opinions of affected employees | Medium | ||||||
| Perceptions/opinions of departments | Low | |||||||
| 12. What were the major benefits of the programs? | Perceptions/opinions | Medium | Low | Medium | Medium | |||
| 13. What were the negative impacts of the programs? | Perceptions/opinions | Medium | Low | Medium | Medium | |||
| 14. How could the programs have been improved? | Perceptions/opinions | Medium | Low | Medium | Medium | |||
| Cost effectivess | ||||||||
| 15. Overall, was the twelve month payback period target achieved? If no, why not? | Payback period achieved | High | ||||||
| If applicable, reported reasons why not achieved | Medium | Medium | Medium | |||||
| 16. Was the twelve month payback period target achieved in all departments? If no, why did some departments achieve it, and why did some others not achieve it? | Payback period achieved | High | ||||||
| If applicable, reported reasonswhy not achieved | Medium | Medium | Medium | |||||
| 17. Were the cost elements included in the twelve month payback period calculation complete? If not, what others should/could have been included? | Opinions | Medium | Medium | Medium | ||||
| Costing study | High | High | ||||||
| 18. Was the twelve month payback period a reasonable measure? Would others have been more relevant, useful, reliable, meaningful, etc.? | Opinions | Medium | Medium | Medium | ||||
| Costing study | High | High | ||||||
| 19. Did implementation of the programs require the development of new systems for monitoring? What were the costs of these systems? Could these costs have been avoided? How? | Reported incidence of new systems | High | ||||||
| Departmental statistics on costs of systems | High/ medium | |||||||
| Opinions regarding possible cost avoidance | Medium | |||||||
| 20. Was there another more cost-effective way of achieving the results achieved through these programs while still ensuring fairness? What were they? | Opinions | Medium | Medium | Medium | Medium | Medium | ||
4. Evaluation Options
The following section presents three evaluation options as follows:
- a description of the option;
- its strengths;
- its limitations;
- the issues addressed through this option;
- the level of effort;
- timing; and,
- cost.
4.1 Option 1 - Limited Methodology
Description:
- 15 telephone interviews with personnel representatives in a sample of departments;
- 15 telephone interviews with finance representatives in a sample of departments;
- no interviews with other representatives in departments
- 5 telephone interviews with representatives of TBS;
- TBS file review limited to program statistics;
- no departmental file review; and,
- no industry consultations.
Strengths:
- inexpensive;
- entire study can be done fairly quickly;
- limited burden on departments;
- all issues covered to some extent.
Limitations:
- not all departments would be consulted;
- limited coverage of issues;
- less reliable.
Issues addressed:
- relevance - adequate coverage
well covered: 1
average: 2
poorly: none
not covered: none
- success - adequate coverage
well: 3, 4, 8
average: 5, 6, 7, 12, 13, 14
poor: 9, 10, 11
not covered: none
- cost-effectiveness - poor coverage
well: 15, 16
average: 20
poor: 17, 18, 19
not covered: none
Level of effort:
20 to 30 consulting days
experienced consulting staff
Timing:
4 to 6 weeks
Cost:
$20,000 to $30,000
4.2 Option 2 - Broader Methodology
Description:
- up to 30 telephone interviews with personnel representatives from all departments - only in headquarters;
- up to 30 telephone interviews with finance representatives from all departments - only in headquarters;
- no interviews with other representatives in departments
- up to 10 telephone interviews with representatives of TBS;
- TBS file review to include a review of program statistics and of documents supporting advice and support;
- departmental file review to include statistics on grievances, as well as extensive costing information (i.e., costing study); and,
- no industry consultations.
Strengths:
- relatively inexpensive;
- entire study can be done fairly quickly;
- improved coverage of issues;
- improved reliability.
Limitations:
- increased burden on departments.
Issues addressed:
- relevance - adequate coverage
well covered: 1
average: 2
poorly: none
not covered: none
- success - adequate coverage
well: 3, 4, 8
average: 5, 6, 7, 9, 10, 12, 13, 14
poor: 11
not covered: none
- cost-effectiveness - good coverage
well: 15, 16, 17, 18, 19
average: 20
poor: none
not covered: none
Level of effort:
70 to 80 consulting days
mixed levels of consulting staff
Timing:
2 to 3 months
Cost:
$50,000 to $60,000
4.3 Option 3 - Optimal Methodology
Description:
- up to 50 telephone and/or interviews with personnel representatives from all departments - in headquarters and in the regions;
- up to 50 telephone interviews with finance representatives from all departments -in headquarters and in the regions;
- up to 15 interviews with other representatives in departments (DMs and their delegates);
- survey of 500 affected employees;
- up to 15 telephone interviews with representatives of TBS;
- TBS file review to include a review of program statistics and of documents supporting advice and support;
- departmental file review to include statistics on grievances, as well as extensive costing information (i.e., costing study); and,
- up to 15 industry consultations.
Strengths:
- excellent coverage of issues;
- improved reliability.
Limitations:
- expensive;
- would take several months to complete;
- increased burden on departments.
Issues addressed:
- relevance - adequate coverage
well covered: 1
average: 2
poorly: none
not covered: none
- success - adequate coverage
well: 3, 4, 8
average: 5, 6, 7, 9, 10, 11, 12, 13, 14
poor: none
not covered: none
- cost-effectiveness - good coverage
well: 15, 16, 17, 18, 19
average: 20
poor: none
not covered: none
Level of effort:
100 to 125 consulting days
100 days of clerical and survey staff
mixed levels of consulting and clerical/
survey staff
Timing:
4 to 6 months
Cost:
$100,000 to $150,000
5. Additional Evaluation Possibilities
The previous section presented three options for the evaluation of the EDI and ERI from a TBS perspective. The evaluation could also be broadened to include an evaluation of the relevance, success and cost-effectiveness of the program from a federal government perspective. While this would meet with resistance from several of the departments consulted during the course of this study, two options are presented to provide the reader with an overview of what would be required in such a case.
5.1 Option A - Limited Methodology
Description:
- 15 telephone interviews with personnel representatives in a sample of departments;
- 15 telephone interviews with finance representatives in a sample of departments;
- no interviews with other representatives in departments
- 5 telephone interviews with representatives of TBS;
- TBS file review limited to program statistics;
- no departmental file review; and,
- no industry consultations.
Strengths:
- government-wide review of the programs;
- inexpensive;
- entire study can be done fairly quickly;
- limited burden on departments.
Limitations:
- not all departments would be consulted;
- resistance/lack of buy-in from departments;
- unreliable.
Issues addressed:
The issues addressed would be different from those identified in this study. Within the scope of this evaluation framework study, it is not feasible to identify these. Because of this, an evaluation assessment would be required at the start of the study to determine the issues to be addressed which can also have an impact on the timing and costs as identified. Therefore the reader is cautioned that the level of effort and costs are best guesses only and could easily be higher.
Level of effort:
40 to 50 consulting days
experienced consulting staff
Timing:
1 to 2 months
Cost:
$40,000 to $50,000
5.2 Option B - Optimal Methodology
Description:
- up to 50 telephone and/or interviews with personnel representatives from all departments - in headquarters and in the regions;
- up to 50 telephone interviews with finance representatives from all departments -in headquarters and in the regions;
- up to 50 interviews with other representatives in departments (DMs and their delegates, managers, etc.);
- survey of 500 affected employees;
- up to 15 telephone interviews with representatives of TBS;
- TBS file review to include a review of program statistics and of documents supporting advice and support;
- departmental file review to include statistics on grievances, extensive costing information (i.e., costing study), as well as documents relevant to the implementation of the programs in the departments; and,
- up to 15 industry consultations.
Strengths:
- government-wide review of the programs;
- improved reliability (from option 4).
Limitations:
- expensive;
- would take several months to complete;
- increased burden on departments.
Issues addressed:
The issues addressed would be different from those identified in this study. Within the scope of this evaluation framework study, it is not feasible to identify these. Because of this, an evaluation assessment would be required at the start of the study to determine the issues to be addressed. As noted in Option 4, the results of the evaluation assessment may have an impact on the timing and costs identified below and therefore the reader should be cautioned that these figures are only reasonable guesses.
Level of effort:
150 to 175 consulting days
100 days of clerical and survey staff
mixed levels of consulting and clerical/
survey staff
Timing:
6 to 8 months
Cost:
$150,000 to $175,000
Appendix A - List of Letters and Booklets
|
A.1 |
Letters from Treasury Board Secretariat to Deputy Heads, Directors of Personnel, or Heads of Collective Bargaining Directly or Indirectly Impacting on EDI/ERI Administration |
|
|
Alternations for "incumbent-based" occupational groups (March 18, 1996) |
