Executive Group Position Evaluation Plan - September 2005

MEASURING PROBLEM SOLVING / THINKING

Problem Solving / Thinking is the opportunity, need or requirement on the part of the position to put Know-How to use in original, self-starting thinking in order to deal with issues and solve problems on the job.

Measuring Problem Solving / Thinking involves evaluating the intensity of the mental processes required by the position. Activities include employing Know-How to analyze, identify, define, evaluate, draw conclusions about and resolve issues. To the extent that thinking is circumscribed by standards, covered by precedents or referred to others, the Problem Solving / Thinking requirement of the job is diminished.

The raw material of any Problem Solving / Thinking is knowledge of facts, principles and means. Ideas are put together from something already there: "You think with what you know." This is true of even the most creative work.

However, this mental manipulation of Know-How is different from the straight application of skill measured by the Know-How factor. For this reason, not all of the Know-How required in a job will necessarily be utilized in the Problem Solving / Thinking elements of that job. Problem Solving / Thinking is therefore treated and measured as a percentage of Know-How, and the numbering pattern on the chart is comprised of a series of percentages rather than point values.

Problem Solving / Thinking has two dimensions:

• Thinking Environment (vertical axis) - how much assistance is available to help the incumbent do the thinking required.
• Thinking Challenge (horizontal axis) - the complexity and novelty of the thinking required and the time pressures within which the thinking must be done.

THINKING ENVIRONMENT

The first step in evaluating a job's Problem Solving / Thinking element involves considering the amount of help available to the job holder. That help can come from precedents, people, and service-wide, department-wide or functional goals, policies, objectives, procedures, instructions or practices. In general:

• Goals, policies and objectives provide help by describing the "what" of a subject matter.
• Procedures detail the steps necessary to follow through on a policy (how, where, when, and by whom).
• Instructions and practices outline the specific how-to's.

The degree to which help is available to job holders varies. For example: help from functional specialists and superiors may be less readily available to managers in geographically remote or organizationally isolated areas. The degree to which help is available is evaluated along the vertical axis of the Chart. There are no hard-and-fast rules. However, here are some guidelines:

• At the D level, what has to be done is often defined. How things have to be done is less defined.
• At the E and F levels, thinking is more about what has to be done. Naturally, how things are to be done is also not clearly defined.
• At the G and H levels, thinking is more about why things should be done. The what is generally less defined, and how things are done is not defined at all.

The key concept to remember when evaluating the Thinking Environment is as follows:

Concept

The relationship between the Know-How level and the Problem Solving level

Application

Logically, jobs do not require the incumbent to think beyond the limit of the Know-How required for the job. Therefore, the Thinking Environment level (as designated by its letter) should generally be no deeper than the depth of the Practical/Technical/Specialized Know-How level/letter previously assigned. (Example: When Practical/Technical/Specialized Know-How is at the F level, the Thinking Environment will probably be E or F - but not G).

THINKING CHALLENGE

Thinking Challenge, the second dimension of Problem Solving / Thinking, measures the complexity of the thought processes required of the job holder. It addresses the questions, "How tough are the things that come the job holder's way in terms of the thinking to be done?" and, "How quickly must the thinking be done?" The various levels of "Thinking Challenge" appear across the top of the Problem Solving / Thinking chart.

There are three key concepts to bear in mind when evaluating Thinking Challenge:

Concept	The definition of "problems"
Application	"Problems" in this context refer to the wide range of challenges confronting job holders. The concept is not restricted to things that have gone wrong, although such things must certainly be considered.

 

Concept	Equivalency of technical and management issues in measuring Thinking Challenge
Application	It is a common mistake to pay too much attention to technical issues when considering Thinking Challenge and too little attention to management issues. Both are equally valid when considering this sub-factor.

 

Concept	The difference between Thinking Environment and Thinking Challenge
Application	Thinking Environment measures the context in which problem solving takes place, and its main constraint is the amount of help available in that context. Thinking Challenge measures the inherent difficulty of the thinking required, and its main constraint is the novelty of the issues being considered.

Problem solving points are at the intersection of the column for the Know-How score and the row for the problem solving percentage

PS %	<----- Know-How Points ----->																									PS %
	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	920	1056	1216	1400
87%	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	920	1056	1216	87%
76%	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	920	1056	76%
66%	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	920	66%
57%	29	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	57%
50%	25	29	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	50%
43%	22	25	29	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	43%
38%	19	22	25	29	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	38%
33%	16	19	22	25	29	33	38	43	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	33%
	50	57	66	76	87	100	115	132	152	175	200	230	264	304	350	400	460	528	608	700	800	920	1056	1216	1400

Most Likely       Less Likely       Unlikely

COMBINING THE PROBLEM SOLVING / THINKING SUB-FACTORS

The result of making independent judgements for each of the two Problem Solving / Thinking sub-factors is that the evaluation falls within a cell which contains two percentage step values.

50 57

Your choice of which specific Problem Solving / Thinking percentage to use to represent the job's total Problem Solving / Thinking requirements will be a judgment, based on your "feel" for the strength or weakness of the job's fit in relation to the Chart definitions of the two sub-factors. Generally, a "solid" fit in relation to the definitions should result in your choosing the lower number in the cell. A "pull" to a higher Thinking Environment or Thinking Challenge would change the choice to the higher percentage. For example:

Problem Solving / Thinking Evaluation = F4	50%	solid fit choice
	57%	"pull" to G and/or 5

To determine Problem Solving / Thinking points, you can use the chart on the facing page. Simply locate the Problem Solving / Thinking percentage in the left column and the Know-How points along the top or bottom. The resulting Problem Solving / Thinking points are found at the intersection.

CHECKING THE PROBLEM SOLVING / THINKING EVALUATION

Evaluators should take the time to review their Problem Solving / Thinking evaluations. Since Problem Solving / Thinking is the application of Know-How, experienced evaluators have found that the relationship between the two factors tends to fall into patterns. These patterns are shown by the shadings on the chart on the facing page. They will serve as a general guide for checking the Problem Solving / Thinking evaluation:

• Normally, an evaluation should fall in the Most Likely areas.
• An evaluation can fall in the Less Likely areas as long as it can be supported by sound reasons.
• If an evaluation falls in the Unlikely shaded areas, the evaluation of both the Know-How and the Problem Solving / Thinking factors should be re-checked. It is possible that the body of knowledge the incumbent is expected to have is insufficient for thinking at the level indicated by the Problem Solving / Thinking evaluation, or that too much knowledge is expected of the position given the degree to which it will be put to use, as indicated by the Problem Solving / Thinking evaluation.

VALIDATING AGAINST THE BENCHMARKS

The most important test of the validity of the evaluation is finding comparable reference evaluations in the standardized continuum of the Benchmarks, as outlined in Appendix C3.

Government of Canada Executive Group—Guide Chart for Evaluating Accountability/Decision Making

Accountability/Decision Making

General
Accountability/Decision Making is the measurement of the degree to which a job is responsible for achieving results and the importance of those results to the organization. There are three components in the following order of importance:

• Freedom to Act
  The degree to which a job, through delegation or empowerment, acts independently to achieve end results before seeking advice or direction as defined in the left‑hand column of the next page.
• Impact
  As defined.
• Magnitude
  The size, relative to the whole Public Service, of the unit or function most clearly affected by the decisions AND/OR recommendations of the job.  The process for determining Magnitude is described in Measuring Accountability / Decision Making.

N.B.
Magnitude and Impact must fit together; neither can be final or meaningful without being related to the other.

Impact
The degree to which the job affects or brings about the results expected of the unit or function being considered.

Indirect (I)
Supportive and ancillary services, where activities are noticeably removed from final decisions and assistance is modified or merged with other support before the end result stage.

Contributory (C)
Interpretative, advisory or facilitating services, for use by others in taking action, which are influential and closely related to action or decisions by others OR measurable contribution, as a member of a team, in achieving end results.

Shared (S)
Equal, joint, and significant control, with (usually only one) another position(s) (except own subordinates and superior), over the activities and resources which produce the results, OR control of what are clearly many (but not all) of the significant variables in determining results.

Primary (P)
Controlling Impact—The position has effective control over the significant activities and resources which produce the results, and is the sole position (at its level of Freedom to Act) which must answer for the results.

		••• Magnitude (Constant Dollars)	1 Very Small (under $100K)				2 Small ($100K to $1 Million)				3 Medium ($1 to $10 Million)				4 Medium–Large ($10 to $100 Million)				5 Large ($100 Million to $1 Billion)				6 Very Large ($1 to $10 Billion)				7 Largest  (over $10 Billion)
		•• Impact	I	C	S	P	I	C	S	P	I	C	S	P	I	C	S	P	I	C	S	P	I	C	S	P	I	C	S	P
• Freedom to Act	D	These jobs are subject, wholly or in part, to practices and procedures covered by precedents or well-defined policies; supervisory review, usually after the fact.	38 43 50	50 57 66	66 76 87	87 100 115	50 57 66	66 76 87	87 100 115	115 132 152	66 76 87	87 100 115	115 132 152	152 175 200	87 100 115	115 132 152	152 175 200	200 230 264	115 132 152	152 175 200	200 230 264	264 304 350	152 175 200	200 230 264	264 304 350	350 400 460	200 230 264	264 304 350	350 400 460	460 528 608
	E	These jobs, by their nature or size, are subject to broad practices and procedures covered by functional precedents and policies; achievement of a circumscribed operational activity; direction from well-defined objectives.	57 66 76	76 87 100	100 115 132	132 152 175	76 87 100	100 115 132	132 152 175	175 200 230	100 115 132	132 152 175	175 200 230	230 264 304	132 152 175	175 200 230	230 264 304	304 350 400	175 200 230	230 264 304	304 350 400	400 460 528	230 264 304	304 350 400	400 460 528	528 608 700	304 350 400	400 460 528	528 608 700	700 800 920
	F	These jobs, by their nature or size, are broadly subject to functional policies and goals; managerial direction of a general nature.	87 100 115	115 132 152	152 175 200	200 230 264	115 132 152	152 175 200	200 230 264	264 304 350	152 175 200	200 230 264	264 304 350	350 400 460	200 230 264	264 304 350	350 400 460	460 528 608	264 304 350	350 400 460	460 528 608	608 700 800	350 400 460	460 528 608	608 700 800	800 920 1056	460 528 608	608 700 800	800 920 1056	1056 1216 1400
	G	These jobs, by reason of their size, independent complexity and high degree of effect on department results, are subject only to general guidance from top-most management.	132 152 175	175 200 230	230 264 304	304 350 400	175 200 230	230 264 304	304 350 400	400 460 528	230 264 304	304 350 400	400 460 528	528 608 700	304 350 400	400 460 528	528 608 700	700 800 920	400 460 528	528 608 700	700 800 920	920 1056 1216	528 608 700	700 800 920	920 1056 1216	1216 1400 1600	700 800 920	920 1056 1216	1216 1400 1600	1600 1840 2112

MEASURING ACCOUNTABILITY / DECISION MAKING

Accountability / Decision Making measures the degree to which a job is responsible for action and the consequences of that action. It is the measured effect of the job on end results.

Up to this point, judgments have been made about the total Know-How required for fully competent job performance, and the degree of mental intensity employed in Problem Solving / Thinking. Now the task is to consider the job's ability to bring about, or assist in bringing about, some specific end results. This includes considering the Magnitude of, and Impact on, those results.

The Accountability / Decision Making Guide Chart shows three sub-factors:

• Freedom to act: The freedom the incumbent has to make decisions and carry them out. This is the most important sub-factor.
• Impact on end results: How direct the job's influence is on the end results of a unit, function or program.
• Magnitude (or result area impacted): The general size of the unit, function or program affected. This is the least important sub-factor.

FREEDOM TO ACT

Freedom to Act is the most important of the three Accountability / Decision Making sub-factors. By examining the nature and extent of the controls - or the lack of the controls - that surround the job, it directly addresses the question of the job's freedom to take action or implement decisions. Because of its importance, this sub-factor carries the most quantitative weight in the evaluation of a position's total Accountability / Decision Making.

The controls placed on the position's Freedom to Act can be supervisory or procedural, or both. A key concept to keep in mind when considering Freedom to Act is:

Concept	The difference between Freedom to Act and Thinking Environment
Application	It is a common mistake to confuse the restraints placed on Freedom to Act with the help available in the Thinking Environment: Freedom to Act is concerned with action or decisions about action. Thinking Environment is concerned with mental manipulation.

Since controls tend to diminish as you rise in the organization, Freedom to Act increases with organizational rank. However, while it is true that no job can have as much Freedom to Act as its superior, the evaluator should be wary of automatic slotting according to organization level alone.

Here are some broad guidelines that can help in assessing Freedom to Act:

• At the D level, positions are relatively free to decide how to achieve predetermined results under some direction from superior management. Positions at this level are subject to managerial approval of tactical objectives and periodic evaluation of results, generally on a quarterly or annual basis.
• At the E and F level, positions are relatively free to determine what the general results are to be. Managerial direction will be general in nature. Assessment of end results must be viewed over longer time spans (e.g., six months to a year or longer).
• At the G level, the what is communicated only in very general terms. Positions become subject to guidance rather than direction or control. Any job evaluated here is subject only to broad policy.

IMPACT

While the explanation of how to evaluate Impact and Magnitude is presented sequentially, these two sub-factors should always be considered together.

The Impact sub-factor measures the directness of the position's effect on end results. The Impact levels are as follows:

I - Indirect: The position provides information, recording or other supportive services for use by others. Activities are noticeably removed from final decisions / end results. The position's contribution is modified by or merged with other support before the end result stage.

C - Contributory: The position provides interpretative, advisory or facilitating services for use by others or by a team in taking action. The position's advice and counsel are influential and closely related to actions or decisions made by others. Such an Impact is commonly found in staff or support functions which significantly influence decisions related to various units or programs. For example:

S - Shared: The position is jointly accountable with others (usually one other) for taking action and exercising a controlling Impact on end results. Positions with this type of Impact have noticeably more direct control over action than positions evaluated at the Contributory level, but do not have total control over all the variables in determining the end result. In addition, Shared Accountability can be used to indicate that a position makes an extremely strong contribution to end results (stronger than its peers) but does not have a Primary Impact.

A basic rule is that Shared Impact does not exist vertically in an organization (i.e., between superior and subordinate). However, Shared Impact can exist between peer jobs within the same organization or with a position from outside the organizational unit. Shared Impact suggests a degree of partnership in, or joint Accountability for, the total result. In this way it differs from Contributory Impact, where the position is only accountable for a portion of the end result.

Note: There are few situations in the Public Service where true shared accountability exists.

P - Primary: The position has controlling Impact on end results, and the accountability of others is subordinate. Such an Impact is commonly found in managerial positions which have line accountability for key end result areas, be they large or small. For example:

Concept	The relation between control and Primary Impact
Application	The relative size of the unit is not an issue in deciding whether or not the position has Primary Impact on its results. The key is that: The position controls the end results of the unit That control is not shared with others ("the buck stops here")

MAGNITUDE

Magnitude measures the size of the area affected by a position. While it does give an indication of the "weight" to be assigned to the position, it is the least important of the three sub-factors used to determine the overall Accountability / Decision Making evaluation.

For measuring Executive Group positions, a common, quantifiable means or "proxy" for representing the diverse units, functions and programs that could be affected by the position must be identified. Dollars have proven to be the most widely applicable "proxy" for measuring the Magnitude to be assigned to a given position.

However, to make a logical, rational determination of Magnitude, the evaluator must remember that dollars are simply a proxy, not an absolute measure. The key concept behind this is:

Concept	Dollars are only a proxy to represent Magnitude
Application	Dollars are the most convenient measure of the size of the unit or function affected by a job. However, this does not mean that jobs impact on dollars. Jobs impact on functions, programs or operations of organizational units.

Dollar ranges on the Guide Chart correspond to varying alignments of Magnitude/Impact/Freedom to Act. The Benchmarks provide evaluators with a wide variety of examples of different levels of Magnitude to assist them in the determination of this sub-factor.

The Magnitude continuum has seven degrees, from "Very Small" to "Largest." These headings provide a rough idea of the appropriate Magnitude for the subject position. References to the appropriate Benchmarks will help refine this initial determination. In this way, evaluators can arrive at a reasonable determination of Magnitude and avoid jumping immediately to a premature consideration of budget dollars.

Evaluators should use the following process for applying the proxy to establish the appropriate Magnitude:

• Determine and describe (in words) what part(s) and/or function(s) of the organization the job affects, and the nature of the job's effect on each of them. For example: the position controls a Branch.
• Once the part(s) and/or function(s) most appropriate to the job have been identified, think about the relative size of the part(s) or function(s) under consideration and describe these in words. For example: the Branch is very small, small, large, very large, etc.
• Once these relationships have been articulated, verify them and the "size" selected for the job against the dimensions of the Benchmark positions.

Use of the Accountability Magnitude Index to Adjust for Inflation

To maintain consistency over time, the Magnitude proxies of the Benchmarks use constant dollars. To make comparisons between a subject job's proxy dollars (which are expressed in current dollars) and the constant dollars in the Benchmarks, it is necessary to convert the current dollars into constant dollars. The annual Accountability/Magnitude Index (AMI) provides the factor used for this purpose.

The AMI is based on the implicit price deflator used by Statistics Canada to produce GNP data in constant dollars. To convert to constant dollars, divide current dollars by the current AMI. For example: if an operating and maintenance budget of $4 million were selected to represent the Magnitude of a position, you would divide this amount by the current AMI to arrive at constant dollars [$4 million / 6.50 = $615,385 (constant)].

The current AMI is updated periodically by the Public Service Human Resources Management Agency. The AMIs from financial year 1980/81 are as follows:

Accountability Magnitude Index
1980/81	2.45	1992-1993	4.60
1981/82	2.77	1993-1994	4.70
1982/83	3.06	1994-1995	4.80
1983/84	3.41	1995-1996	5.00
1984/85	3.61	1996-1997	5.00
1985/86	3.72	1997-1998	5.00
1986/87	3.83	1998-1999	5.20
1987/88	3.91	1999/2000	5.40
1988/89	4.03	September 2000	6.00
1989/90	4.17	September 2002	6.50
1990/91	4.37	April 2006	7.00
1991-1992	4.50

"Pass-Through Dollars"

Many positions may appear to have a very large Magnitude, but the dollars being used to measure the unit are "Pass-Through Dollars." (Transfer payments to individuals or other jurisdictions under social programs which are controlled largely by legislation, regulation or formula fall into this category. An example would be Canada Pension Plan payments.) The key to handling Pass-Through Dollars is as follows:

Concept	Pass-Through Dollars are not an appropriate Magnitude proxy
Application	In cases of Pass-Through Dollars, the position deals with the process of payment but has practically no impact on determining whether payments should be made or what payments should be made. These dollars do not properly represent the Magnitude of the position. A more appropriate proxy should be found.

CHOOSING THE CORRECT IMPACT/MAGNITUDE COMBINATION

An evaluation score may differ depending on the combination of Impact and Magnitude used. For instance:

• A function head (e.g., a Director General of Human Resources) may be seen to have a Contributory Impact on the operations of the Department or a Primary Impact on the operations of the Human Resources Branch.
• Very often the point totals available in these two slots will be the same. Where they are not the same, it is advisable to use the higher score to properly reflect the full job size as long as you are confident that your reasoning is correct.

The key is to find the combination of Impact and Magnitude that results in the highest legitimate evaluation. This is because it is vital to get the fullest, most complete measure of the position for these two sub-factors so as to properly reflect the job size. Table 1 on the following pages provides some guidelines for evaluating certain types of expenditures when these are used as the proxy.

TABLE 1 EXAMPLES OF IMPACT FOR VARIOUS DIMENSIONS
Dimensions	Impact of Operating Management	Impact of Staff or Support Function	Comments
1. Salary, operating and maintenance budget (used to represent an organizational unit)	Primary	Contributory or Indirect	The Impact of operating management is Primary because the main accountability for unit or program end results rests with operating management. The Impact of staff depends on the significance of the advisory and facilitating role (i.e., Policy Advisor could be C or I).
2. Capital budget (used to represent a capital program)	Primary or Shared	Contributory, Indirect or None	The Impact of operating management is Primary when feasibility, design, construction, installation and utilization are controlled by a single manager (which is rarely the case). It is less (e.g., Shared) when a department project manager acts as a "knowledgeable" client for a specific project. The Impact of functional staff depends on the significance of the advisory and facilitating role.
3. Human resources costs (used to represent the human resources function)	Contributory or Indirect	Contributory or Indirect	The number of positions which play a role in designing and/or implementing this function (e.g., central agencies, departmental managers, departmental Human Resources jobs) means that no one or two positions control the significant activities necessary for Primary or Shared.

COMBINING THE ACCOUNTABILITY / DECISION MAKING SUB-FACTORS

The result of evaluating the three sub-factors is that the evaluation falls within a cell with three possible point values, each representing one step up in size.

350
	400
		460

The number chosen from the cell will depend upon your assessment of the relative strength of the job's fit to sub-factor definitions. However, unlike the sub-factors of Know-How and Problem Solving/Thinking (which have a more or less equal weight in determining the factor's score), in the case of Accountability / Decision Making, the fit of the Freedom to Act sub-factor is the most important one to consider.

There is a propensity for evaluators to forget this hierarchy of values and use Magnitude to drive the selection of the number in the cell. This is inappropriate. Considerations of Impact and Magnitude (the least important sub-factor) should be used to confirm the direction of the overall evaluation, as determined by considerations of Freedom to Act.

TABLE 1 (Continued) EXAMPLES OF IMPACT FOR VARIOUS DIMENSIONS
Dimensions	Impact of Operating Management	Impact of Staff or Support Function	Comments
4. Purchased materials and equipment (used to represent the purchasing function)	Shared, Contributory or Indirect	Shared, Contributory or Indirect	The Impact of Department of Public Works and Government Services people would be Contributory for the normal supply & service role, or Shared where their role is one of heavy involvement in determining specifications, in addition to the normal supply & service role. The extent to which departmental functional staff (e.g., head of informatics) act as the department's purchasing agents will affect the Impact recognized in the evaluation.
5. Grants and contributions (used to represent a program)	Contributory or Indirect	Indirect	The Impact could range from Contributory to Indirect depending on the degree of: 1. Discretion in grant or contribution amounts. 2. Control over the end results expected from the grant or contribution.
6. Transfer payments (used to represent a program)	Indirect or None	None	When transfer payments are determined by a formula with no discretion, the Impact would likely be none. Where there is some discretion in determining amount and/or use, the Impact would likely be Indirect because the position has some effect on the program.

CHECKING THE ACCOUNTABILITY / DECISION MAKING EVALUATION

Up, Down and Level Profiles

The evaluation score of a position gives an indication of its size, relative to other jobs. It answers the question, "How big is this job?" However, the relationship between the scores for the Problem Solving / Thinking and the Accountability / Decision Making factors is indicative of the shape of the job and answers the questions, "What sort of job is this?" "Is it characterized by thinking (Problem Solving) or action (Accountability), or is the balance about equal?"

• An action-oriented job is primarily oriented toward generating end results. Problem Solving takes a secondary position in this position. Therefore, the points given to Accountability / Decision Making will be higher than those for Problem Solving / Thinking. This relationship is known as an Up, or "A," Profile.
• A thinking job exists to apply Know-How in the analysis, investigation and identification of situations. The Problem Solving / Thinking points will be greater than those for Accountability / Decision Making. This is known as a Down, or "P," Profile.
• A balanced or level job is one in which the Accountability / Decision Making and Problem Solving / Thinking points are the same. The position will be staff-oriented and have responsibility for managerial or supervisory functions. This is known as a Level, or "0," Profile.

While there are no hard-and-fast rules, particular types of jobs do tend to have predictable profiles:

Down Profiles

P4 P3  Problem Solving / Thinking points exceed Accountability / Decision Making points by four or three steps, respectively. Jobs with these profiles will tend to be concerned with basic or pure research, with little orientation to, or regard for, development aspects. P4 Jobs will rarely be found outside a university. P2 P1  Problem Solving / Thinking points exceed Accountability / Decision Making points by two steps or one step, respectively. Applied research or policy development jobs will tend to have these profiles.

TABLE 1 (Continued) EXAMPLES OF IMPACT FOR VARIOUS DIMENSIONS
Dimensions	Impact of Operating Management	Impact of Staff or Support Function	Comments
7. Revolving funds (represents payment received from clients for services rendered)	None	None	Individual positions do not have sufficient impact on what is to be measured. That is, the impact is less than Indirect. Payments received should not be double-counted against corresponding expenditures, nor should they be used to reduce operating expenditures to a net figure.
8. Dimensions lying outside the Public Service such as value of the GNP	None or Contributory or Indirect	None or Indirect	The relationship of Public Service positions to these dimensions is, in most cases, too remote for the measurement of any Impact. Where influence can be clearly identified, the Impact of operating management positions is normally Indirect and is typically exerted through legislative, regulatory or enforcement authorities. Contributory Impact could apply to operating management positions in which the degree of control over end results is considerable. Indirect Impact could apply to staff or support functions when the degree of control or influence over end results is considerable. In all these instances, the incumbent must be identified in the job description and the performance review process as answerable for results.

Level Profiles

L   Problem Solving / Thinking points equal Accountability / Decision Making points. Jobs with these profiles will tend to involve providing support services in staff functions or supervisory positions such as financial analysts or heads of functional specialties.

Up Profiles

A1 Accountability / Decision Making points exceed Problem Solving / Thinking points by one step. Jobs with A1 profiles are often hybrid jobs with significant people management responsibilities (such as human resources managers), line management positions, or jobs which receive a significant degree of direction from functional units, such as project managers or regional directors of administrative services. A2 A3  These profiles are found in line management jobs which have a clear and well defined responsibility for achieving results, such as regional director for operations. A4  Examples of this profile are unusual but can occur where the Accountability for results is high but the Problem Solving or Know-How content of the job is relatively low.

Up, Down and Level profiling allows the validity of evaluations to be checked against typical job profiles. Discrepancies, if found, may indicate an incorrect evaluation. However, they might also indicate an inappropriately structured job. Therefore, it is important to avoid letting profiles drive the evaluation process.

VALIDATING AGAINST THE BENCHMARKS

The most important test of the validity of the evaluation is finding comparable reference evaluations in the standardized continuum of the Benchmarks, as outlined in Appendix C3.

USING THE BENCHMARKS

The Benchmark reference positions have two critical roles to play in the job evaluation process:

• They provide the necessary discipline of a constant set of reference points.

• They allow for flexibility within a disciplined framework .

Copies of the Benchmark reference positions are found in Appendix C3.

PROCESS FOR SELECTING SUITABLE BENCHMARKS

1) How Benchmarks are arranged

The Benchmarks are sorted in two different ways:

• The Benchmark Job Descriptions are arranged into groups based on job function (Appendix C1).
• There is a second listing of the positions by level, in descending order of total points (Appendix C2).

2) Information needed about the subject position

In order to select suitable Benchmarks, the evaluator first needs to know:

• Where the subject position fits in the organization (e.g., number of levels from the Deputy Head)
• The job function (e.g., financial, operational, human resources)
• The basic nature of the job (e.g., to think deeply, as in research positions, to think broadly, as in policy development, to direct activities, as in field operations, or to administer policies and practices, as in staff positions)
• Whether the position is line or staff, regional or located at headquarters

All this information should be in the Job Description.

3) Selecting Suitable Benchmarks

Two or three Benchmark comparisons should be sufficient for testing the validity of an evaluation. A strong comparator is one in which the organizational context, the overall evaluation and the evaluations of the three factors are all similar to that of the subject position (i.e., fewer than three steps away on any one factor).

However, on occasion, it may be difficult to find a Benchmark that fits the subject position on all three factors. For example, one Benchmark might provide a close fit on the Know-How factor but not work well on the Accountability / Decision Making factor. The best thing to do would be to try to find other, more suitable, Benchmarks since the imbalance affects the profile fit of the two positions. However, if a good, overall fit cannot be found with any of the Benchmarks, the evaluator should look for an additional Benchmark position which provides a good fit for the missing factor (in the above example, Accountability / Decision Making).

BENCHMARK COMPARISONS EXAMPLE A
GENERAL ACCOUNTABILITY
Subject Position	Is accountable for the proper and effective management and administration of all operations programs in the Region to ensure the provision of economic support, pension, medical examinations, social well-being and health care to qualified veterans and their dependants. Is accountable for the operation of the Saskatoon Veterans Home.
Benchmark 7-A-2	Is accountable for directing and managing regional operational functions essential to conducting the Agency's statistical programs, and for developing the overall strategic framework and business plans for the regional operations to enhance the Agency's capacity to provide Canadians with statistical information.
Benchmark 6-A-2	Is accountable for the efficient and effective direction of regional operations and programs designed to assist inmates and parolees with their reintegration into civil society.
Benchmark 5-A-1	Is accountable for providing a regional lens in the conception, development and delivery of national programs to promote good nutrition and informed use of drugs, food and natural health products, and to maximize the safety and efficacy of pharmaceutical drugs, food, natural health products, medical devices, biologics and related biotechnology products in the Canadian marketplace and health system to promote and protect the health of Canadians.

SAMPLE VALIDATIONS USING BENCHMARKS

Below are two examples of the process for using Benchmarks. Descriptive statements from the job description of the subject position and the sample Benchmarks are shown on this and the following facing pages. These statements are not the only relevant differentiating considerations. They are presented simply to illustrate the thinking process involved in using Benchmarks.

Sample Validation: Example A

EXAMPLE A

Subject

Regional Director General

Position

Prairie Region

Evaluation

Know-How Problem Solving / Thinking Accountability / Decision Making

F III 3 F 4 (50%) F 3 P

608 304 400 1312

Benchmarks Selected

7-A-2

6-A-2

5-A-1

Director General, Regional Operations

Assistant Deputy  Commissioner, Operations

Regional Director, Health Products and Food

G III 3 F 4 (57%) F 4 P

700 400 528 1628

F III 3 F 4 (50%) F 2 P

608 304 350 1262

F III 3 F4(50%) F 2 P

528 264 304 1096

BENCHMARK COMPARISONS EXAMPLE A
MAJOR CHALLENGE
Subject Position	The position is expected to negotiate with provincial authorities to ensure an equitable distribution of contract beds for the priority use of veterans, and achieve an optimum care/cost ratio within the framework of provincial standards in Manitoba, Saskatchewan and Alberta.
Benchmark 7-A-2	The position must build national consensus for the implementation of policies and procedures that respond to the changing needs of clients. In meeting this challenge, the incumbent must balance increasing demands for quality information with public and political concern regarding invasion of privacy, response burden, voluntary participation and respondent refusal or mistrust.
Benchmark 6-A-2	The position leads the cooperative development of innovative programs with community support organizations or spiritual and cultural leaders to help offenders benefit from a broad range of programs and services, and assist them in successfully reintegrating into the community.
Benchmark 5-A-1	The position fosters a spirit of cooperation, collaboration, teamwork and partnership between the Department and the communities, through contacts with senior officials such as the Deputy Minister or Minister, Assistant Deputy Ministers, Presidents or Directors in provincial health, social services and education departments or agencies.

Example A Explanation: The Know-How Factor

Subject Job		7-A-2		6-A-2		5-A-1
		Director General, Regional Operations		Assistant Deputy Commissioner, Operations		Regional Director, Health Products and Food
F III 3	608	G III 3	700	F III 3	608	F III 3	528

1) One consideration for relating the Know-How of the subject position to the Benchmarks is the management structure above each position. There are the same number of layers between the Deputy Minister and the position holders in all four cases.

2) Another consideration is the diversity of the programs managed. While the program mixes of the subject position and the three Benchmark positions are relatively homogeneous in end results, the national role of Benchmark 7-A-2, the variety of its clientele and the ad-hoc nature of the demands placed upon it warrant a higher degree of specialized and practical skills and knowledge than the subject position and the other two Benchmark positions.

3) A number of variables must be considered when evaluating Know-How: cultural diversity of the publics served, geographic dispersion of the territory managed, and size and type of staff managed. Benchmark 7-A-2 is accountable for the department-wide provision of products and services to a paying clientele with varied interests in and purposes for the products requested, and it manages a larger staff than the subject position and the other two Benchmark positions.

4) The subject position and Benchmarks 6-A-2 and 5-A-1 are all responsible for managing the delivery of direct client services within a specific geographic area, whereas Benchmark 7-A-2 leads the management of service delivery nationally in all regions, which adds to the depth of procedures, techniques and disciplines required and the variety of environments and clienteles.

5) The subject position is considered comparable to Benchmark 6-A-2 as both positions require a similar breadth and depth of knowledge to direct the delivery of the full range of departmental services at the regional level, whereas Benchmark 5-A-1 reports to a corporate ADM at National Headquarters and requires knowledge and skill related to one departmental business line.

BENCHMARK COMPARISONS EXAMPLE A
DIMENSIONS (Constant Dollars)
Subject Position	Employees:	317
	Operating budget:	$3.3 million
	Grants and Contributions:	$15.0 million
Benchmark 7-A-2	Employees (both Public Service and Statistics Act):	3,012 (Non-census year) 49,645 (Census year)
	Operating budget: Annual Sales:	$12.6 million (Non-census Year) $30.3 million (Census Year) $ 1.2 million
Benchmark 6-A-2	Employees:	52
	Operating budget:	$438,500
Benchmark 5-A-1	Employees:	32
	Operating budget:	$310,000

Example A Explanation: The Problem Solving / Thinking Factor

Subject Job		7-A-2		6-A-2		5-A-1
		Director General, Regional Operations		Assistant Deputy Commissioner, Operations		Regional Director, Health Products and Food
F4 (50%)	304	F 4 (57%)	400	F 4 (50%)	304	F 4 (50%)	264

1) The geographic dispersion, the nation-wide leadership, the specific requirements of the clientele and the revenue-generating function create discernible differences between the complexity of programs managed by Benchmark 7-A-2 and those delivered by the subject position and the other two Benchmark positions.

2) The difference in geographic areas serviced by the subject position and Benchmarks 6-A-2 and 5-A-1 is not considered to require significantly different innovative thinking.

Example A Explanation: The Accountability / Decision Making Factor

Subject Job		7-A-2		6-A-2		5-A-1
		Director General, Regional Operations		Assistant Deputy Commissioner, Operations		Regional Director, Health Products and Food
F 3 P	400	F 4 P	528	F 2 P	350	F 2 P	304

1) The subject position and the Benchmark positions appear to have a similar degree of latitude and authority to act on behalf of the department within the scope of their delivery responsibilities in their respective areas.

2) Benchmark 7-A-2 is viewed as making a noticeably more significant contribution to the achievement of the department's end results than the subject position, hence a push to the middle number of the magnitude range of the Benchmark position. The other two Benchmark positions and the subject position are viewed as having a similar degree of contribution within their respective magnitude ranges.

BENCHMARK COMPARISONS EXAMPLE B
GENERAL ACCOUNTABILITY
Subject Position	Is accountable for managing the development, implementation and maintenance of departmental financial and accounting policies, systems and procedures to meet the operational needs of management and to conform with the requirements of government acts, statutes and regulations, and for providing non-transfer payment accounting services and advice to responsibility centre managers across the Department.
Benchmark 6-O-2	Is accountable for establishing the Agency's strategic and corporate guidelines in the sectors of modern comptrollership, human resources, financial resources and administrative services management, in order to achieve maximum effectiveness in the use of the Department's human, financial and material resources.
Benchmark 5-M-1	As the senior full-time financial officer and comptroller, is accountable for the financial policies, systems, procedures, operations and accounting activities to ensure effective control and stewardship of the financial resources appropriated, and for ensuring the introduction and acceptance of modern accounting and financial management standards and practices.
Benchmark 4-O-3	Is responsible for the proper and effective management of the finance, materiel management, information systems, contract administration, and administration functions in the Department's Atlantic Region.

Sample Validation: Example B

EXAMPLE B
Subject		Director
Position		Financial Policies, Systems and Accounting
Evaluation		Know-How					F III 3	460
		Problem Solving / Thinking					E 4 (50%)	230
		Accountability / Decision Making					E 5 C	264
								954
Benchmarks Selected
6-O-2				5-M-1			4-O-3
Director General, Resource Management				Director, Financial Management			Regional Manager, Finance & Administration
G III 3 F 4 (57%) F 2 P	608 350 350 1308			F III 3 E 4 (50%) E 5 C	528 264 264 1056		F II 3 E 4 (43%) E 2 P	460 200 230 890

 

BENCHMARK COMPARISONS EXAMPLE B
MAJOR CHALLENGE
Subject Position	A major challenge for the Director is to provide management with an integrated accounting and financial systems operation that has effective accounting mechanisms and control, and at the same time meets the requirements of Parliament and Central Agencies with respect to departmental initiatives in the areas of Economic Development and Trust and Loan Fund Management.
Benchmark 6-O-2	The position is expected to provide corporate leadership in the development, dissemination and implementation of policies, systems, processes, services and control mechanisms for the seamless implementation of the management reform underpinning the modern comptrollership and modern management functions affecting the management of the Agency's financial, material and human resources.
Benchmark 5-M-1	The position's challenge is to ensure the continued integrity of the financial management data, systems and procedures while ensuring the successful implementation of significant changes in financial management practices, information systems and accounting procedures arising out of the Modern Comptrollership Initiative and other financial modernization initiatives.
Benchmark 4-O-3	The position guarantees the integrity and reliability of budgetary control and reporting systems in the Region to ensure that probity and prudence are duly exercised in the handling of regional resources.

Example B Explanation: The Know-How Factor

Subject Job		6-O-2		5-M-1		4-O-3
		Director General, Resource Management		Director, Financial Management		Regional Manager, Fin. & Admin.
F III3	460	G III 3	608	F III3	528	F II3	460

1) While the subject position and Benchmark 5-M-1 are both rated F III 3, the Benchmark is a senior full-time financial officer and is thus seen as requiring higher degrees of professional and operational knowledge. Benchmark 6-O-2 reflects a degree of mastery required to provide executive direction at the first managerial level for the management of all the infrastructure support functions across the department. The subject position and Benchmark 4-O-3 reflect the depth of professional expertise required to provide executive level support in the delivery of the department's programs.

2) The managerial expertise required for the subject position is less than that required for Benchmarks 6-O-2 and 5-M-1, which have department-wide functional responsibilities. The department-wide role of the subject position and the focus of its end results, although rated higher, are weighed as comparable with the regional focus of Benchmark 4-O-3 and receive the same numerical value.

Example B Explanation: The Problem Solving / Thinking Factor

Subject Job		6-O-2		5-M-1		4-O-3
		Director General, Resource Management		Director, Financial Management		Regional Manager, Fin. & Admin.
E 4 (50%)	230	F 4 (57%)	350	E 4 (50%)	230	E 4 (43%)	200

1) As the senior functional position in the department, Benchmark 6-O-2 is clearly involved in more demanding, extensive and varied issues than are the subject position or the other two Benchmark positions.

2) The departmental functional roles of the subject position and of Benchmark 5-M-1 are viewed as requiring a higher degree of innovation, creativity and integration than Benchmark 4-O-3.

BENCHMARK COMPARISONS EXAMPLE B
DIMENSIONS (Constant Dollars)
Subject Position	Division Employees: Division Operating Budget: Departmental Operating Budget	50 $500,000 $885 million
Benchmark 6-O-2	Branch Employees: Department Employees: Branch Operating Budget: Agency Operating Budget:	30 380 $800,000 $6.7 million
Benchmark 5-M-1	Division Employees: Division Operating Budget: Department Operating Budget:	48 $484,000 $177 million
Benchmark 4-O-3	Region Employees: Division Employees: Region Operating Budget: Division Salary Budget:	1,370 133 $49 million $866,000

Example B Explanation: The Accountability / Decision Making Factor

Subject Job		6-O-2		5-M-1		4-O-3
		Director General, Resource Management		Director, Financial Management		Regional Manager, Fin. & Admin.
E 5 C	264	F 2 P	350	E 5 C	264	E2 P	230

1) The ratings reflect the higher level of empowerment and contribution of Benchmark 6-O-2 in the achievement of expected end results and departmental objectives.

2) Note that while the magnitude levels of the subject position and Benchmark 5-M-1 could warrant a difference in their respective numerical ratings, their contribution to the achievement of the department's goals is viewed overall as being of equal value.

			Likely Step Differences
Designation	Unit Diagram	Common Characteristics	Know How	% Problem Solving	Account-ability
"Normal" Superior/Subordinate Relationship		reasonable span of control Clear channels of communication Balance between subordinates Management delegation	2	1	3
Lean Staff		Broad span of control Communication processes fuzzy Imbalance between subordinates positions Management and decision-making centralized	3	1 or 2	3 to 5
Missing Level		Large span of control Communication of tasks only Large differentials in subordinates position Management and decision-making highly centralized	4	2	5 to 7
One Over One		One subordinate Superior and  subordinate function as a team Usually a temporary "grooming" position for  subordinate  prior to taking over the top position Appropriate where critically of top job dictates (i.e. CEO) or where there is a need for a split between external focus and internal focus	1	1	2

ORGANIZATION CHECK

A crucial test of the validity of the evaluation is whether it fits with the evaluations for other positions in the unit. This means that when you isolate each factor, the step differences between the subject job and the supervisor, peer and subordinate positions for each factor all make sense.

A common evaluation error is over-emphasizing the differences between peer positions and under-emphasizing the differences between superior and subordinate. Note, however, that there are no rules for determining the proper relationship between levels in an organization. Each case must be assessed on its own.

• a variety of superior/subordinate relationships that can make sense
• the importance of looking at the reality of the actual departmental structure when testing the validity of a new evaluation

In the final analysis, as throughout the evaluation process, informed common sense should be the tool for making and checking all judgments.

APPENDIX A - EXECUTIVE (EX) GROUP DEFINITION

The Executive Group comprises positions located no more than three hierarchical levels below the Deputy or Associate Deputy level and that have significant executive managerial or executive policy roles and responsibilities or other significant influence on the direction of a department or agency. Positions in the Executive Group are responsible and accountable for exercising executive managerial authority or providing recommendations and advice on the exercise of that authority.

Inclusions

Notwithstanding the generality of the foregoing, it includes positions that have, as their primary purpose, responsibility for one or more of the following activities:

1. Managing programs authorized by an Act of Parliament, or an Order-in-Council, or major or significant functions or elements of such programs;

2. Managing substantial scientific or professional activities;

3. Providing recommendations on the development of significant policies, programs or scientific, professional or technical activities; and

4. Exercising a primary influence over the development of policies or programs for the use of human, financial or material resources in one or more major organizational units or program activities in the Public Service.

Exclusions

Positions excluded from the Executive Group are those whose primary purpose is included in the definition of any other group.

CLASSIFICATION LEVELS: EXECUTIVE GROUP

The total points assigned through the evaluation process will determine the classification level for a newly evaluated position. Positions within the assigned point bands fall into compensation levels from EX-1 to EX-5. The bands are as follows:

EXECUTIVE GROUP CLASSIFICATION LEVELS
Classification Level	Minimum Points	Maximum Points
EX-5	2448	N/A
EX-4	1868	2447
EX-3	1560	1867
EX-2	1262	1559
EX-1	920	1261