Measuring Client Satisfaction

October 1991

Office of the Comptroller General
Evaluation and Audit Branch

                                                                                DRAFT

                            MEASURING CLIENT SATISFACTION


                             Developing and Implementing
            Good Client Satisfaction Measurement and Monitoring Practices
                                       Content


Introduction

1.  What is "client satisfaction" ?

2.  Defining the clientele

3.  Defining the objectives of the client satisfaction assessment program

4.  Assessing clients' expectations

5.  Assessing the actual performance:  Choosing the appropriate measurement strategy 

    5.1  Types of measures
    5.2  Methodological approaches

6.  Analysis and interpretation of results

7.  Communicating and utilizing client satisfaction information

8.  Monitoring client satisfaction


Annex I  : Key steps in developing and implementing a client satisfaction assessment                        program

Annex II : Examples of satisfaction indicators
           for different types of public service clients

Annex III: References

Introduction

Improved service to the public and increased client consultation are key objectives of PS 2000 renewal exercise. 
Accordingly, federal departments and agencies, and above all their line managers, are being more and more
pressured to survey clients and to reconsider the way they are delivering services and communicating with the
external world in the light of survey results.  However, most line managers don't have the time nor the expertise
to conduct such studies.  Tight budget conditions also limit significantly the amount of resources an organization
can spend on client studies.  Therefore, line managers should be able to rely on the assistance of specialists in
surveys and performance measurement to design and realize effective and efficient client satisfaction studies.  

This paper is aimed at people generally asked to carry out or manage client satisfaction studies, as well as at
people involved in the development of TQM projects.  Its purpose is to discuss key issues related to effective and
efficient measurement and monitoring of client satisfaction.  It has no pretension whatsoever to be revolutionary. 
It basically summarizes the research work and experience gained in the last ten to fifteen years in the areas of
performance measurement, program evaluation and client satisfaction measurement both in private and public
organizations.  A shorter version, intended for line managers, will be also available shortly. 

In the past, client satisfaction studies have tended to result in a gathering of information that is probably not as
meaningful or useful to program managers as it could or should have been.  Conclusions like: "in general clients
are satisfied with the program" may be contributing to reinforce the status quo more than they are helping
managers to pinpoint areas of client satisfaction and discontent, or to find innovative solutions to improve
program delivery.  Good client satisfaction studies are not ends in themselves; they are a means to improve
service to the public.  Knowledge of clients' expectations and of the extent to which these are met may prove
really beneficial indeed to managers.  Broadly speaking, this knowledge serves two purposes: (1) identifying areas
of improvement in the quality of the services offered; and (2) highlighting the need for corrective actions when
clients' expectations exceed what the organization can afford to offer or what a particular program is meant to
provide.  

Most of the time, it takes more than a simple survey or a monitoring of level of complaints to measure effectively
the satisfaction of clients with a particular service or program and to get the type of information needed for
service improvement.  However, design and implementation of satisfaction assessment programs should not be
regarded as an extremely complex enterprise with limited chances of true success.  A client satisfaction
assessment program is a comprehensive and strategic approach of measuring and monitoring client satisfaction
and of utilizing satisfaction measures in order to improve service to the public.  Carefully designed and
implemented, such programs can bring considerable benefits to any public organization, and they can contribute
significantly to the culture change needed within an organization to establish a total quality orientation.




1. What is "client satisfaction" ?

Generally speaking, client satisfaction measures the extent to which a client's expectations for a good or service
are met.  Thus, it is essential to distinguish clearly the two components of satisfaction - client expectations and
the actual or perceived quality of the service offered - and not to consider satisfaction as a global entity.  A
proper measure of satisfaction include an separate assessment of both client expectations and the quality of
service provided.

Client expectations

Even before entering in contact with the organization, most clients already have more or less precise expectations
of the level and quality of service that it should be providing.  These expectations may not be in line with what
the program is or should be delivering, as stated in its objectives, or with what it can actually provide, given the
resources available.  Communications efforts are usually needed to inform clients of the goals, limits and
possibilities of the program, in order for the clients to have more realistic expectations.

Research has shown that different factors contribute to determining client expectations.  Client cultural
background, level of aspiration and worldview do exert some influence on the setting of expectations.  Client
expectations are usually higher if there is a direct out-of-pocket expense associated with the use of a public good
or service.  Regularity of use is another key element in shaping client expectations; people tend to have much
more accurate expectations for a service that they use regularly, than for one that they seldom use.  Other factors
contributing to shape clients' expectations include, of course, the personal needs of clients, their past experience
with the service or with similar services provided by other organizations, and what they hear from other users
of the service.

As opposed to the often compartmentalized clientele of competitive private sector enterprises, the clientele of
many public services is a lot more diverse.  This means that the expectations of clients for a particular public
service are often not homogeneous.  For example, the income level of clients may impact on their acceptance
of user's fees, on the importance they give to family allowances; education level of parents may impact on their
expectations toward the public education system that their children attend; etc.  

It is important to remember also that the organization that delivers the service can exert by itself a significant
influence on the level of its clients' expectations through external communications.  For instance, by informing
clients of the service standards of its program, an organization is contributing to the definition of the expectations
of the clientele, as it is telling to the clients what level and quality of service they can legitimately expect to
receive.  Thus, not meeting publicized standards, will result surely in client dissatisfaction.  A strategic use of
external communications should be made to prevent the organization from creating 
too high expectations, as well as to help reduce client dissatisfaction resulting from unmeetable expectations.

Quality of service

Quality of service refers to the quality of both the transaction and the outcome of the service.  It is a multi-
dimensional concept.  An honest portrait of client satisfaction implies that variation for each of the major
dimensions has been measured.  According to Zeithaml & al. (1990), the ten most common dimensions cited
by clients in judging quality are:

-      Tangibles:  Appearance of physical facilities, equipment, personnel, and communication
       materials.

-      Reliability:  Ability to perform as promised, dependably and accurately.

-      Responsiveness:  Willingness to help clients and provide prompt service.

-      Competence:  Possession of the required skills and knowledge to perform the service.

-      Courtesy:  Politeness, respect, consideration and friendliness of contact personnel.

-      Credibility:  Trustworthiness, believability, honesty of the provider.

-      Security:  Freedom from danger, risk or doubt.

-      Access:  Approachability and ease of contact.

-      Communication:  Keeping customers informed in language they can understand and listening
       to them.

-      Understanding the Client:  Making the effort to know clients and their needs.

Needless to remind that in the specific context of federal institutions, quality of service also
imply the provision of the service in both official languages, which means having bilingual
personnel and communication material, and ensuring that rooms and offices to which clients
have access present bilingual characteristics (e.g., having bilingual signs, brochures in both
languages in display, magazines in both languages in waiting room, etc.). 

Of course, the different dimensions of a quality service are not necessarily independent of
one another, and they overlap somewhat.  Moreover, their respective importance can vary
significantly depending on the type of clientele or on the nature of the service delivered by 
the organization.  The assessment of the expectations of clients should normally highlight
their key areas of interest with respect to quality of service. 


2. Defining the clientele

Defining who the clients are and what type of services these clients receive from the
program is a necessary first step in designing a satisfaction measurement and monitoring
program.  This may not be as obvious as one may instinctively think.  It often involves long
discussions and brainstorming sessions where participation of employees at all levels of the
organization is required.  The idea is to reach a fairly large consensus throughout the
organization on who the clients are exactly and how they are affected by the different
outputs of the program.

The concept of "client" cannot be applied indistinctively in the private and public sectors. 
According to Politt (1988), public service consumer/provider transactions sometimes bear
little resemblance indeed with typical "over-the-counter" customer relations in the private
enterprise.  For example, the public service consumer does not necessarily buy the service;
he may have the statutory right or he may be compelled to receive the service.  On the
other hand, in the public sector, wanting a service or even willing to pay for it does not
necessarily give an access to service, since eligibility criteria for need of the service can be
determined by a mixture of political and professional judgment.  Finally, often the
disappointed public service consumer, or the one who believes the service received has
actually done harm, has little or no opportunity to sue the provider, but can complain and
show his/her discontent through voting.

Overemphasizing the importance of these differences can be as harmful as not considering
them enough.  Too often the argument of the distinctiveness of the public sector is used as
a pretext for doing nothing.  The specific characteristics of the public sector do not suggest
that reference to the concept of client is inappropriate so much as they suggest that a broad
definition of the notion of client - including both internal and external clients -is needed. 
Thus, the clients of a government program should include anyone for whom the program -
or more specifically the organization delivering the program - provides a service.

Also of importance to remember is that government programs are not monolithic; the
services that each provide are peculiar and of different natures.  For instance, the PS 2000
task force on service to the public refers to three broad categories of services: legislation
and policy development; regulation and enforcement; and conventional services.  In other
words, it means that there are different types of clients and client/provider relationships for
government programs:

A)POINT-OF-SERVICE OR "DIRECT" CLIENTS

       1-  Voluntary users/customers.  Such clients are able to choose whether or not they
       want to use a particular public service (e.g. CBC listeners or viewers, National Parks
       visitors, Via Rail traveller, etc.).  This type of client has a lot in common with private
       sector customers.  It is possible to distinguish two sub-groups of voluntary clients:
       users/customers of monopolistic program -i.e., there are no private or public program
       that can serve as an alternative; and users/customers of a program in a competitive
       environment -i.e., program alternatives can be found either elsewhere in the public
       sector or in the private sector.

       2-  Entitled users/recipients.  Factors not directly controlled by the client give him/her
       an automatic right to benefit from a particular government program (e.g. U.I. or
       Family Allowances recipients, legal aid users, etc.).  Entitled clients can be more or
       less captive, depending on the ease of "exiting" from the program.   

       3-  Compelled "users".  Many government programs are prescriptive or have a
       prescriptive component (e.g., regulation programs).  Compelled users are people or
       organizations that fall under the jurisdiction of such programs (e.g. inmates, tax
       payers, regulated industries, etc.).  Punitive action will normally be taken against
       these users if they "exit" or don't comply with the prescriptions of the program they
       are part of.


B)INDIRECT CLIENTS

       1-  Beneficiaries.  These clients directly benefit from the use of a public sector service
       or good by other point-of-service clients. (e.g. Canadian businesses from customs
       inspection of travellers; meat buyers from meat inspection; families of hospital
       patients, of inmates, etc.)

       2-  Citizens.  Most public services generate a large amount of externalities, thus
       benefiting the public in general.  For example, the society as a whole benefits from
       having literate, healthy, well-educated and law-observing people, and everybody
       benefits from a clean environment.  Therefore, citizens in general are clients, at least
       indirectly, of many public programs.

Obviously, the interests and expectations of indirect clients are not the same as those of
point-of-service clients.  Furthermore, they can often be contradictory.  The objectives of a
program, when properly defined, should be a particularly helpful reference to determine who
the indirect clients of are, and to what extent their views need to be taken into account
when measuring client satisfaction. 

A client satisfaction assessment program should be fully tailored to the needs, environment,
and clientele of the program.  Satisfying clients does not necessarily mean the same thing
for the manager of a program with a voluntary clientele and for a manager of a program
with a compelled clientele.  Whereas the former may be aiming at maximizing satisfaction
with the program or service in general, the latter will likely be looking instead at minimizing
dissatisfaction and be focusing on very specific service attributes such equity and fairness in
delivering the service.  Also important to remember is that satisfaction measures should
reflect appropriately the views of both direct and indirect clients when it is relevant to do
so.  Annex II, on page 29, provides a list of examples of possible satisfaction indicators for
different types of clienteles.


3. Defining the objectives of the client satisfaction assessment program

Objectives of the satisfaction assessment program should also be explicitly and precisely
defined before any attempt is made at measuring clients' attitudes.  A lot of problems
associated with the lack of utility of satisfaction data can be attributed to not having defined
the objectives of the exercise well enough.  If program managers have only a vague idea of
how they will use the information gathered from their clients, there is a good chance that
the right questions or the questions that would have given the most practical information
will not be asked.  The objectives of the client satisfaction measurement and monitoring
program should of course relate to the information needs of program managers.  Due to the
limited resources an organization can spend on client inquiries, it is essential to look for
client consultation to be as cost-effective as possible.  This can be achieved through careful
thought on what program managers want to achieve with their client satisfaction assessment
program.  Defining such objectives normally involves finding answers to the following
questions:

-      Do we already know, or have indicators of, the actual performance of the
       organization in delivering a quality service?

-      Why do we want to measure and monitor client satisfaction?

-      What do we want to measure exactly?  Are we looking for an overall rating of
       satisfaction or for an assessment of satisfaction with the diverse aspects of service
       delivery?  Are we aiming at minimizing dissatisfaction (e.g., reducing level of
       complaints) or at maximizing satisfaction (e.g., increasing level of support from
       clients) ?

-      Whose satisfaction do we want to measure?  The satisfaction of the entire clientele
       or of specific segments?

-      What do we intend to do with the gathered information?  How can it be applied to
       improve service to the public and program performance more generally?

As it is the case for the definition of the clientele, defining the objectives of a client
satisfaction measurement and monitoring program should normally allow the participation
of employees at all levels of the organization.  Management as well as front-line employees
are concerned.  Managers are the primary users of client satisfaction information; they are
responsible for implementing the appropriate changes that are highlighted by a satisfaction
study.  Front-line employees are in close contact with clients and they constantly receive
instant feedback from them.  Therefore, they may have a good idea of what is right and
wrong with the service your are providing.  The perspective of front-line employees is
invaluable to the design of most satisfaction assessment programs.


4. Assessing clients' expectations

As it was mentioned above, measuring client satisfaction supposes an assessment of both
factors contributing to client satisfaction or dissatisfaction:  client expectations and their
assessment of the quality of service provided. 

Assessing expectations implies a concern for the nature of these expectations as well as for
their origins.  If realistic, clients' expectations toward the levels and quality of service can
provide helpful information and reference for the development of standards of service for
the program.  On the other hand, information on the origins of the expectations can be of
great value when corrective actions need to be taken as a result of unrealistic expectations -
 i.e. largely exceeding what a public authority has decided or can afford to offer.

Without a doubt, a good knowledge of clients' expectations is vital to the success of the
exercise.  Focus groups and surveys are particularly useful to this end.  Client focus groups
are usually used at first to inventory expectations and issues of interest or concern to clients. 
Using this information, surveys can be designed to help weigh the relative importance of the
different expectations and issues, as indicators of the key components of - and factors
influencing - client satisfaction.[1]  This knowledge of the key "drivers" of satisfaction helps
management to determine which aspects of the service are a priority for monitoring and
should be addressed by service standards.

Knowing clients' expectations is of interest also to identify what can be called "artificial
dissatisfaction" with the service provided.  For example, if expectations for a service are
unrealistic, there will probably be a lot of dissatisfaction with it.  This dissatisfaction cannot
necessarily be attributed to bad service; it can mean instead that there is a lack of
understanding from the public of what the government program is meant to do.  Thus it
might be more a problem of communication or a symptom of non-acceptance of a particular
government policy, than a problem with the quality of the service offered.

Another example may be the services or programs for which clients' expectations and/or
needs are heterogeneous.  Heterogeneity of expectations for a given public service can often
result in client dissatisfaction.  It does not mean that the service provided is of bad quality,
but rather that it is not tailored to the demand curve for the service.  Public education may
be a good example of this situation.  A lot of parents may find that the education offered
in public schools is not up to their standards, while at the same time, others may think that
it is too demanding.  The actual level of service provided probably falls somewhere in the
middle of these divergent expectations, and furthermore it may even correspond to the true
expectations of very few clients.  The compromise reached on service level may be the result
of public policy or societal objectives, such as universality of access or equity.  Nevertheless,
it can be expected that it will create dissatisfaction among clients since the "average" service
provided, as a result of compromise, is not well suited to the wants of a clientele whose
expectations' profile follows a "camel bumps" curve.  Identifying variations in the
expectations of the different segments of the clientele can be a key issue in the assessment
of clients' expectations.  This is particularly true if the clientele presents heterogeneous
characteristics. 

If the program has both direct and indirect clients, as it is very often the case in the public
sector, chances are that it will be confronted by the problem of conflicting expectations.  Of
course, it should not stop program managers from measuring and monitoring client
satisfaction.  One important thing to remember though, is that the objectives of the program
delivered should not be distorted or displaced by an undue or unbalanced consideration of
either point-of-service or indirect clients concerns.  In practical terms it means that trade-
offs may be needed and/or that different parts of the organization (i.e. senior management,
middle management, front-line employees) will have their own priorities in terms of clients
to serve.


5. Assessing the actual performance:  Choosing the appropriate measurement strategy 

Knowing how the clients rate the organization in providing a quality service is at least as
important as knowing what their expectations are.  It allows program managers to identify
aspects of service where major gaps exist between what clients think they are entitled to
receive and what they are actually receiving.  Low performance programs are likely to
produce client dissatisfaction indeed.  Measuring the actual performance requires that the
different aspects of a quality service responsible for client satisfaction or dissatisfaction are
assessed; some examples of these aspects include: satisfaction with the access and facilities,
with communication material, with the personnel, with the nature of the service per se, with
service result/outcome, and with the program more generally.

In order for the client consultation to be as cost-effective as possible, decisions on how to
measure need to be thoughtfully made, and normally require the assistance of specialists in
the area, such as program evaluators and/or external consultants.  Choosing a measurement
strategy means formally deciding on (1) the type(s) of measures and (2) methodological
approach(es) that will be used.  There are a number of valid alternatives - and rarely a
single "miracle solution" - when it comes to choosing a methodology for assessing clients
satisfaction, each of which has its own strengths and weaknesses.  In fact, the choices to be
made depend largely on the objectives of the client satisfaction program - i.e., on what you
want to measure and on what you intend to do with the information gathered - and on the
resources available.  Trade-offs may be needed between the quality of the information and
the cost of collecting, processing and reporting it in a timely manner.


5.1 Types of measures

Initial or Final Satisfaction:

       -  Initial satisfaction:  these measures are used to assess (point-of-service) clients'
       attitudes right after receiving a service.  They are particularly effective in assessing
       the perceived quality of the transaction.  Transaction-based surveys are a common
       tool for measuring initial satisfaction.  Initial measures are particularly adapted to
       "compelled" clients as well as to clients of grants and contributions programs that
       didn't receive financial support; this type of measures focus on the quality of the
       transaction, thus allowing to control the impact of the program's outcome on client's
       assessment.

       -  Final satisfaction:  studies have demonstrated that satisfaction does not rest only
       in a good quality transaction, but also in the final result or outcome for the client. 
       Depending on the service or good offered, satisfaction regarding outcome can occur
       more or less rapidly.  Normally, the more time elapsed between transaction and
       surveying of clients, the more satisfaction rating will be based on the client
       assessment of outcome.  

Subjective or Objective Measures of Satisfaction:

       -  Subjective measures: they focus on the impressions or perceptions of clients.  They
       are very helpful in telling what clients think, but they do not tell if their thinking is
       right.  However, in the area of client satisfaction, perceptions are often as important
       and meaningful as the reality.

       -  Objective measures: there are different objective indicators that can help assess
       client satisfaction.  For example, level of activities and "market share" (for a
       voluntary clientele), time required for processing clients' files, volume of general
       complaints, volume of errors disadvantaging clients in the treatment of their files that
       were identified or reported by clients, etc.  The major strength of objective measures
       is their reliance on facts.  However, they often have the inconvenience of resting on
       indicators that do not assess satisfaction directly or solely.  Therefore, in order for
       the objective satisfaction measure to be valid and reliable, special attention should
       be given to the choice of its indicators.
 
       Dissatisfaction measures: 

       The aim of the measurement program can be either to minimize dissatisfaction or
       to maximize satisfaction.  Dissatisfaction measures focus on the segments of the
       clientele that are likely to be dissatisfied, such as people who complain, people who
       are entitled to receive a particular service but who decide not to receive it, users of
       alternatives or competing goods or services, people who stop using a service or
       participating in a program before completion, etc.  According to Larsen & al. (1979),
       focusing on dissatisfaction data is a good way to enhance the usefulness of your
       measure.  Dissatisfaction measures may be particularly helpful to providers of
       services who have to deal with heterogeneous clients' expectations that cannot be
       totally addressed.  Moreover, this type of measure can also be useful in assessing the
       attitudes of "compelled clients".  Most of the time indeed, constraint will not result
       in a high level of overall satisfaction with a program; satisfaction assessment
       programs for compelled clienteles could therefore benefit from focusing on issues
       that create the most dissatisfaction or on overall levels of dissatisfaction with the
       program, rather than focusing on probably less meaningful measures of overall
       satisfaction.  

Acceptability measures:

       Whereas satisfaction includes only the client's reaction during a transaction and after
       receiving goods or services, acceptability includes both the a priori willingness to
       participate in a program and satisfaction during and after the delivery of goods and
       services.  Assessing the attitudes of "compelled clientele" or of a "captive entitled
       clientele", is often in fact measuring acceptability rather than satisfaction.  Because
       of the very limited capacity of "exiting" of such clienteles, their assessment of the
       program reflects more or less indistinctively their a priori willingness to participate
       as much as their reaction during a transaction and after receiving goods or services. 


5.2 Methodological approaches

There are two broad categories of approaches. Direct approaches are the ones in which
clients are directly asked or solicited for feedback (e.g., surveys, clients focus groups, clients
panels, clients visits, comments/suggestions box, etc.).  These approaches allow to get in
touch with the clients.  Using either qualitative or quantitative methods, or both, a direct
approach can help a program manager to get precisely the type of information that he/she
is looking for.  On the other hand, indirect approaches allow to assess client satisfaction
without having to ask clients directly for information (monitoring the volume and nature of
complaints, front-line employees surveys or focus groups).  According to Neese (1991), this
type of approaches might be best suited for situations where a relatively high level of
dissatisfaction is expected.  Research has shown indeed that dissatisfied clients are less
inclined to participate in satisfaction enquiries; many people simply do not want to discuss
things that cause them stress (Neese, 1991; Lebow, 1987; Larsen & al., 1979).  Therefore,
indirect approaches can help reduce the bias in favour of satisfied clients likely to be
introduced by direct approaches. 

Whatever the choice is, a "pilot test" of the  methodology is always recommended. 
Conducting such pre-tests on a relatively small sample of the clientele allow to assess the
effectiveness of methodological instruments (e.g. questionnaires) and to make the
appropriate refinements to these instruments before engaging in the expensive full-scale
research.  Testing methodology is part of most cost-effective and successful researches.   

The following overview of principal measurement instruments summarizes the major pros
and cons of each instrument when assessing client satisfaction.  Few of the methods
discussed here could be used without consulting additional reference material - such as the
Program Evaluation Methods (OCG, Evaluation and Audit Branch, 1991) -  and experienced
practitioners.

a)  Direct approaches

-  Surveys  

       When conducted on a representative sample of clients, surveys have the main
       advantages of presenting quantitative data and results that can be generalized to the
       entire clientele.[2]

       Wulfsberg and Pulaski (1990) mention that good satisfaction survey questionnaires
       include three types of questions:  (1) assessing the client satisfaction with the
       organization overall; (2) satisfaction with various aspects or dimensions that
       contribute to the definition of a quality service or good; (3) "events" questions which
       make your survey more action-oriented (example of an event question might be: "To
       what extent do you agree or disagree to the following sentence: the last time I spoke
       with an employee of ______, I had the feeling that he or she was not genuinely
       concerned with my problem").  To further emphasize the dynamic component of the
       satisfaction measure, survey questionnaires may also include improvement questions,
       whereby clients are asked to assess the change in the quality of service over a given
       period.

       However, survey questionnaires have to be adapted to the objectives of the
       satisfaction assessment program and to the specific profile of the clientele it is aimed
       at.  Measures of overall satisfaction with the program may be well suited to measure
       the attitudes of indirect clients (Landon, 1977; Roth & al., 1991), but may not be as
       useful when assessing the attitudes of "point-of-service compelled clients".  Most
       people are not really happy to pay taxes, to spend time in prison or to follow specific
       constraining regulations.  Therefore, different indicators - focusing on the quality of
       the transaction rather than the outcome of the program or type of service provided
       for the client - may be needed to assess the satisfaction of these clienteles.  For
       example, it might be more appropriate to measure compelled clients satisfaction with
       specific issues or elements of the program they participate in (for example, was
       information from Revenue Canada Taxation easy to get and helpful?  Is Department
       X doing its best to facilitate compliance with regulations?  Is the organization willing
       to discuss your grievances?  Is the personnel respectful of your rights?, etc.)

       Satisfaction surveys can be conducted through mailed questionnaires, by telephone
       or by personal interviews. They can also be transaction-based (i.e. "on-site") or home-
       based.  Phone surveys usually have a much higher percentage of respondents than
       mailed surveys.  On the other hand, both types are confronted with a sampling bias
       problem.  Research has shown that satisfied clients are generally more inclined to
       participate than dissatisfied clients (Neese, 1991; Lebow, 1987; Larsen & al., 1979). 
       Satisfaction ratings based on data gathered through mailed, phone or home-based
       personal interviews surveys have therefore a structural tendency to be inflated.  The
       problem can be largely overcome by conducting transaction-based personal
       interviews, although this technique has also its weaknesses.  Transaction-based
       surveys emphasize satisfaction with transaction as opposed to satisfaction with
       outcome, and obviously, they cannot take into account the views of indirect clients. 
       The problem of inflated ratings can also be addressed by probing clients for negative
       as well as positive comments.  For example, instead of always asking clients: "were
       you satisfied with...?", one can ask also: "did you have any problem with...?".  This
       approach should make room for negative comments and, simultaneously contribute
       to make the study more objective.

       When relevant, satisfaction surveys should reach both point-of-service and indirect
       clients.  Furthermore, for organizations with a voluntary type or non-captive type of
       point-of-service clientele, it is often useful to survey eligible non-users, former users
       and users of alternative programs.  This will help evaluate the "market share" of the
       program and the organization delivering it, and to determine why people prefer
       "competitors" or simply not to use the service/good.  There is a possibility of hidden
       dissatisfaction with the service provided in non-users or former users.  

       Another problem with the survey technique lies in the assessment of attitudes and
       behaviour of minority groups.  Because of the relatively low number of members of
       such groups in a normal representative sample, statistical analysis and inferences of
       their behaviour and attitudes is often difficult or misleading.  In the private sector,
       this may not constitute a real problem since marketers will identify a specific and
       very homogeneous clientele they are aiming at, and are looking most of the time for
       the opinion of the "average customer" within this strictly defined clientele.  This can
       not always be the case in Government where the principle of equity is fundamental. 
       Particular attention must therefore be paid to the fair treatment of minority groups'
       views when conducting client satisfaction surveys.  This is further emphasized by the
       facts that, as it was discussed above, (1) cultural background contributes to the
       definitions of expectations, and (2) clients' expectations are often not distributed
       along a bell's curve where the expectations of average clients are more or less in tune
       with the expectations of the majority of clients. 

-  Focus groups:

       The qualitative information gathered through client focus groups is habitually richer
       than information gathered through surveys.  The inconvenience is that the opinion
       expressed by the groups cannot be generalized to the entire clientele.  There is no
       scientifically reliable or cost-effective way - except the conduct of a survey - to know
       if the opinions expressed by focus groups members are really in tune with what the
       broader clientele thinks.

       Thus, focus groups are particularly effective when used as an adjunct to surveys. 
       They can further scrutinize areas of concerns or dissatisfaction that have been
       highlighted by surveys or explain results that were surprising or otherwise unexpected,
       and they can also help to find appropriate solutions.  Moreover, focus groups are a
       particularly useful tool to assess client expectations through the  identification of
       specific problems and satisfaction issues of clients.

-  Client panels and visits:

       Organizations or programs with a few but very important clients may prefer to
       develop a more personal approach to measure client satisfaction.  It can be
       accomplished with clients panels and visits. 

       Panels consist in a periodical reunion of selected clients or of their representatives
       to discuss problems they encounter in dealing with the services provided by a specific
       program, their possible solutions, and the progress (or lack of it) accomplished since
       last meeting.  The synergy developed through panel members' exchanges helps pin-
       point areas of dissatisfaction and define innovative approaches to overcome them. 
       Getting sustained participation from clients may be a problem though.  The use of
       incentives to increase clients' participation in panels, a technique utilized by many
       private organizations, has to be considered with some care in the public sector, since
       the incentives may be seen as favouritism, thus breaching the principle of equity.
       Client visits may be the best technique to get an in-depth knowledge of individual
       clients.  Not only do visits allow service providers to get in close contact with their
       clients, they also allow them to get in touch with their clients' environment, which
       can be particularly helpful in understanding their constraints and objective needs.

       As it is the case with the other approaches using non-probabilistic samples, the
       information gathered through panels and visits of selected clients cannot however be
       generalized to your entire clientele.  The two techniques are principally suitable for
       assessing the attitudes of very specific segments of the clientele, or, as mentioned
       above, for organizations with a small number of important clients. 

-  Comments/suggestions box:

       This is certainly the simplest and least expensive way to assess client satisfaction.  But
       people who are using it may be getting exactly what they are paying for.  Research
       has shown that people who write comments or suggestions are probably not
       representative of either the satisfied or dissatisfied components of the clientele
       (Andreasen, 1977; Lebow, 1987; Rogers, 1989; Zeithaml & al. 1990; Roth & al.,
       1991).  Comments are often disparate or even contradictory; and therefore they are
       of little help.  On some occasions however, it is possible to find in the suggestion box
       recurring comments clearly indicating that the organization is doing something right
       or wrong.  Once in a while program managers may find also an innovative and
       interesting suggestion that should be seriously considered for implementation.  

       Information found in comments/suggestions boxes tends to focus on initial
       satisfaction (i.e., satisfaction with transaction) and almost exclusively reflect the views
       of point-of-service clients.  Considering their low cost, comments/suggestions boxes
       can be used as a supplementary source of information on clients' attitudes, but the
       satisfaction measurement strategy should not rest solely on this instrument.  Adding
       a comments/suggestions box to a client satisfaction program has also a highly
       symbolic - "public relation" - impact; it is indicating to the clients and employees that
       the views of the clients are important and taken into account.

b)  Indirect approaches

-  Monitoring the volume and nature of complaints and related data:

       Indirect approaches, such as this one, allow to assess client satisfaction without
       having to ask clients directly for information, thus without engaging in often
       expensive client research.  Variations in the level and nature of complaints give
       program managers an early warning of problems and a broad idea of the satisfaction/
       dissatisfaction trends of the clientele.  Moreover, when many clients express the same
       type of complaint, management is getting a clear indication that something is going
       wrong.  On the other hand, this type of measurement approach cannot produce a
       very precise assessment of the extent to which people are really satisfied with the
       products or services.  It is indeed much better suited to satisfaction assessment
       programs aimed at minimizing voiced dissatisfaction than to programs aimed at
       maximizing satisfaction.  Here again we are confronted with the problem of
       representativeness.  People who complain are probably not representative of
       dissatisfied clients.  In fact, most dissatisfied people prefer either doing nothing or
       "exiting" from a program, if it is possible (Hirschman, 1970; Roth & al., 1991).  

       Monitoring of complaints may be a more appropriate technique for organizations
       with point-of-service clients that have virtually no possibility to exit (e.g., captive
       entitled clients, compelled clients).  Since they cannot exit, people have no other
       choice than to "voice" their dissatisfaction.  Therefore, complaints should be a more
       revealing indicator of true levels of satisfaction and dissatisfaction for such clienteles. 
       This instrument of measurement is less appropriate, however, to assess the
       satisfaction of clienteles with a medium to high capacity of exiting, or to assess the
       satisfaction of indirect clienteles.

-  Internal research:

       Front-line employees are constantly in contact with clients either through the delivery
       of service per se  or through the reception of clients coming to receive service.  In
       any organization, they are the people who know point-of-service clients best.  By
       surveying or interviewing these employees or organizing focus groups with them,
       program managers can get a lot of good information on their clients, on what causes
       their satisfaction or dissatisfaction.  Brown (1991) is a strong advocate of the
       involvement of front-line employees; according to him, organizations must recognize
       that the perceptions of those who deliver quality services are as important as those
       who receive them, since the goal of client satisfaction research is not to measure past
       performance, but to drive future behaviour.   The behaviour and attitudes of front-
       line employees are a key component of client satisfaction.  Accordingly, most
       satisfaction assessment programs should make room for the views and knowledge of
       these employees.  However, it should be stressed that employees' surveys need to be
       considered - by both management and employees - as a management tool and not
       as report cards on individuals' performance if their real intent is to develop a client
       focus within the organization.

       On the other hand, this approach can lead sometimes to the confusion of front-line
       employees and clients interests.  Front-line employees' opinions of clients are not
       necessarily accurate of what clients are or of what they think.  For instance,
       employees may have preconceptions toward some categories of clients.  And there
       is also a potential danger that employees will associate the well-being of clients to
       their own well-being.  Opinions of employees might be, in fact, a better assessment
       of their level of satisfaction with their work environment than an assessment of
       clients' satisfaction.  In order to avoid such a confusion,  the employee research
       component of client satisfaction assessment programs should not be primarily aimed
       at measuring work climate or employees' morale; it should focus instead on the
       employee's understanding of his or her clients and on his or her ability to deliver
       quality service to them (Wulfsberg & Pulaski, 1990).  Balancing the employees'
       opinions with direct measurement instruments is also fundamental.




                               Key Points to Remember



-      Tailoring.  A satisfaction measurement strategy should be tailored to the information needs of managers,
       the resources it can afford to spend on client satisfaction research, the objectives of the clients
       satisfaction program, and to the profile of the clientele.

-      Simplicity should be as highly-valued as reliability and validity.  Complex approaches are expensive, they
       need a high level of expertise, there is no guarantee that they will give you the most practical and useful
       information, and moreover, they can be hard to maintain over time.

-      Diversity.  Due to the limitations of each approach taken individually, the best thing to do is to include
       different types of instruments in the measurement strategy: direct, indirect, quantitative and qualitative,
       etc.  Using different instruments also contribute to produce multiple lines of evidence, thus making study
       results more convincing. 

-      Selectivity.  The information gathered should be directly linked to the objectives of the client satisfaction
       program, and address the key components of client satisfaction that have been identified.  Otherwise
       a lot of time and money will be spent on compiling data that is not really relevant and useful.

-      Unbiased.  When conducting surveys, special attention should be paid to the control of the possible
       sampling biases highlighted earlier.

-      Direct and indirect clients.  Satisfaction measures should reflect adequately the views of both point-of-
       service and indirect clients, especially if the program has compelled point-of-service clients and/or clearly
       identifiable indirect beneficiaries.

-      Hidden dissatisfaction.  If the organization has a voluntary clientele or an entitled non-captive clientele,
       the satisfaction assessment should also make room for the views of non-users, former users or users of
       alternatives.

-      Front-line employees.  They are in close contact with point-of-service clients.  A proportionate
       consideration of the opinions of these employees can be advantageous.

-      Specific measures for specific clients.  Measures of overall satisfaction with the program are best suited
       for indirect clients, while dissatisfaction measures, acceptability measures, and measures of specific
       attributes of the good or service offered are particularly effective for compelled clients, and to a lesser
       extent to captive entitled clients.




6. Analysis and interpretation of results

A thorough analysis of the gathered data is essential to reach meaningful conclusions. 
Reporting levels of satisfaction in absolute terms and in isolation from other data is almost
always meaningless.  Statements like: "in general, clients are satisfied with the program", are
not very useful and can be misleading (Larsen & al., 1979).  Maybe too often though this
is the kind of conclusion satisfaction studies reach.  Things can be done to improve
interpretation of data.

For instance, satisfaction ratings can be compared in time and across sites.  The importance
of such comparisons lies primarily in the fact that most satisfaction studies reveal a high
level of overall satisfaction (Larsen & al., 1979; Lebow, 1987).  For example, if 75% of
clients are satisfied with a program in a particular province, this doesn't mean necessarily
that the organization is doing a great job, especially if the average score in other parts of
the country or if the score of the clients of the same province six months ago was (is)
reaching 80%.  A drop of 5% can be meaningful.  It is also possible to compare satisfaction
ratings for different segments of the clientele; segments can be defined according to age
groups, gender, socio-economic groups, cultural or language groups, as well as to the type
and/or level of service received.  Since satisfaction ratings have a propensity to be inflated,
importance should not be given so much to absolute ratings as to difference in ratings in
time, geographically or within segments of the clientele. 
 
Different factors can contribute to the inflation of overall satisfaction ratings.  We talked
earlier about the difficulty for client research to reach the dissatisfied segments of the
clientele.  On the other hand, point-of-service clients may tend to overrate their satisfaction
with some governments programs in order to keep on benefiting from these programs.  This
is particularly the case for programs whose purpose is to give money to the point-of-service
clients (e.g. grant and contribution programs) and for advantageous programs whose future
is somewhat linked to evaluation or client study results.  Clients are likely to show overall
satisfaction with such programs not only because of the quality of the service - and probably
despite some flaws in the quality of the service provided - but also because they want the
programs to be maintained, or because they would like to see similar programs being
developed in the future.  By reporting a high level of overall satisfaction, these clients may
in fact intend to put pressure on decision-makers for forthcoming policy or program
improvements.  The analysis of satisfaction data should attempt to uncover the presence of
this type of behaviour from surveyed clients.  A regression analysis, that isolates the
variables most closely linked with overall satisfaction, could prove very useful at that point.

Analyzing client satisfaction ratings also means situating them in their broader context. 
Changes in satisfaction ratings can sometimes be attributed indeed to environmental changes
rather than to an increase or decrease in agency's performance.  For example, a sudden rise
in the number of people asking for service or in the complexity of cases treated, following
a government policy change or changes in economic conditions, can result in a temporary
overload that is likely to produce dissatisfaction.  This dissatisfaction can probably be
explained by the tougher conditions program staff is working in, more than by a decrease
in its actual performance.

For most client satisfaction studies, a central part of the analysis rests in the identification
of the major gaps between clients expectations and major satisfaction components and the
actual ratings of service deliverers.  Once these areas of dissatisfaction have been clearly
identified and explained,  possible strategies to fill the gaps should be developed.  This
should be done in consultation with front-line employees; their knowledge of clients and
service constraints can be really helpful in finding practical and innovative solutions.  Finally,
study analysts should carefully evaluate the cost, cost-effectiveness, timeliness and feasibility
of introducing those changes to increase client satisfaction, and draw their recommendations
for change from this exercise.

Data analysis may find that in some occasions point-of-service client satisfaction is
contradictory to indirect client satisfaction, or vice-versa.  This is particularly the case for
organizations or programs with compelled point-of-service clients and/or with clearly
identifiable indirect beneficiaries.  If such a dilemma implies a compromise, fidelity to the
stated objectives of the organization or program should be the main focus in reaching for
this compromise.

Rigorous analysis and interpretation of data can not make up for badly compiled or
unreliable data.  But at the same time, data that were gathered with great care and with
sophisticated methods may be rendered almost useless, or at least not as profitable as they
should have been, by a weak analysis. 


7. Communicating and utilizing client satisfaction information 

Results of satisfaction studies as well as initiatives undertaken to improve client satisfaction
should be largely and openly diffused throughout your organization and your clientele. 
Some client survey specialists even suggest that a data communications plan is part of every
good client satisfaction assessment program.

In order to increase the chances for study results to be utilized, senior management and
program managers, as clients of the satisfaction assessment study, need to be kept informed
regularly of the unfolding of events during the conduct of the study.  They should also be
the first informed of the results, of course.  Wulfsberg & Pulaski (1990) stress the
importance of the quality of results presentation to management.  If client feedback reports
are to command all the attention they deserve, their design and appearance must be clean
and clear.  High-quality graphics, easily read charts, and the elimination of extraneous
information are critical in communicating results so that managers will read and act on
them.

Communicating results to front-line employees is also particularly important, since they are
the one who are responsible in the end for the success of the satisfaction improvement
program.  When informing employees of study results and of the changes that will be
implemented to increase satisfaction, management should also set clear targets and service
standards for future improvements.  If they are realistic and defined in consultation with
employees, these targets and standards should have a motivational effect on employees. 
Motivation could be further increased by linking employees' reward schemes to
improvements in clients satisfaction ratings.  Involving all employees in the satisfaction
assessment program is probably the best way to make sure that everybody is going in the
same direction: towards greater client satisfaction.
Clients also need to be informed of study results and of the changes that will be made to
improve service.  By doing so, organizations are clearly indicating to their clients that they
are responsive to their wants and needs and that they are giving due consideration to their
views.  Furthermore, by informing clients of latest initiatives and service quality targets and
standards, an organization is significantly reducing the chances of being confronted with
unrealistic clients expectations, thus reducing the chances of "artificial" dissatisfaction. 
Particular attention should be given to clients who participated in the satisfaction study. 
They are probably the clients most interested in the study results.  Giving them relevant
information on the outcome of the study demonstrates appreciation for their participation
and ensures future participation. 

Client satisfaction studies that rest on a shelf are not useful.  As Band (1989) mentions, in
measuring satisfaction, collecting data is only half the story; what you do with this information
is critical.  Management must show commitment to the utilization of satisfaction information. 
A wide diffusion of results is not enough.  Management should also closely monitor the
implementation of the program adjustments required to better meet clients' expectations. 
These adjustments will be easier to monitor if their objectives and expected impacts have
been clearly defined from the outset.  All the efforts spent on determining clients'
expectations, on measuring the corresponding satisfaction, and analyzing the data may be
in vain if the information gathered is not utilized advantageously.

Client satisfaction information can be a very helpful reference to program managers who
are developing service standards and performance targets.  Service standards identify service
level and quality that program deliverers commit to deliver to clients, and consequently that
clients can expect to receive; whereas performance targets refer to the service level and
quality that an organization wishes to achieve in a more or less near future.  Standards are
mainly used as a communication and accountability tool for clients; effectively
communicated, they could impact significantly on clients' expectations and related
satisfaction.  On the other hand, performance targets' major goal is to motivate personnel
to improve service to the public; by defining precisely the areas and scope for improvement,
the targets help to set the priorities of the organization in terms of service to the public. 
Valid standards and targets are challenging though budget realistic, and focus at the same
time on key clients' concerns.  Client satisfaction measurement and monitoring will highlight
these concerns, provide a client assessment of the actual performance of the organization
in providing a quality service, and indicate areas for improvement that could lead to greater
client satisfaction.  Therefore, information on clients' expectations and on the capacity of
the organization to meet these expectations is of great value to program managers when
developing service standards and performance targets.

The ultimate value of collecting, analyzing and reporting client satisfaction information will
depend on how much it is used.  Program managers should always keep in mind that almost
any expenditure will not be warranted if the information is not used to improve service to
the public and/or, to a somewhat lesser extent, report on performance.


8. Monitoring client satisfaction 

Experts in the area agree that it is essential to measure client satisfaction regularly. 
Periodical measures will help the satisfaction assessment program in keeping its motivational
effect on the organization.  They will also allow management to assess the impact of the
changes previously implemented to improve your client satisfaction.

Good monitoring practices share three common characteristics: usefulness, validity and
efficiency.  A useful monitoring system is simple and provides management with
understandable and relevant information in a timely manner.  A valid system provides
accurate and reliable information.  Finally, an efficient monitoring system is streamlined (i.e.,
avoids duplications, reports only the information that is essential) and makes a cost-effective
use of data (i.e., the benefits derived from the system surpasses its costs; readily available
data and sampling techniques are used to the greatest extent possible). 

A possible option for monitoring client satisfaction, regularly used in the private sector, is
the creation of a Client Satisfaction Index (CSI).  A client satisfaction index is a standardized
overall measure of service delivery performance against selected service indicators which
clients judged to be critical to their satisfaction (Band, 1990; Huot, 1989).  This type of index
can be a useful and cost-effective tool to monitor clients satisfaction, even though it cannot
be seen as a substitute to full-scale client satisfaction studies.  By focusing on the key
indicators of client satisfaction that were previously identified, the index will produce a good
estimation of the mood of the clientele at a much lesser cost than an exhaustive type of
measure.  Also, a standardized overall measure facilitates comparisons in time and across
sites, thereby providing a practical source of information to managers.  
A CSI could be developed to measure ten to fifteen key variables of client satisfaction on
a regular basis, but it could also cover a much larger number of variables.  It is important
to remember however that a detailed monitoring of the top twenty client satisfaction
variables will require much more effort than a regular measure of the top five drivers.  Each
of the variables selected for the CSI must be broken down into specific client satisfaction
indicators.  Some satisfaction indicators may be measured internally through the
establishment of appropriate monitoring mechanisms.  For others, a client survey may be
needed.  To be most effective, the survey should be short, conducted at pre-determined,
regular intervals, it should focus on key CSI variables and, of course, cover a representative
sample of the clientele.  Once the information is gathered, ratings achieved for each
satisfaction indicator need to be translated into relative scores based on each indicator's
weighted contribution to the overall CSI.  Adding up each score provides the CSI
performance for that period.  A CSI may not be the best tool to inform a manager of what
causes client satisfaction or dissatisfaction, or how to change the course of events.  In return,
it can tell him/her whether or not he/she is moving in the right direction.

Clients' expectations and requirements, and program environment as well as clientele are
changing.  The monitoring practices should be adapted to this reality.  One way to do that
is to re-assess clients' expectations periodically.  The re-assessment of clients' expectations
should normally lead to a review and adjustment of the indicators used to measure
satisfaction, which in turn may impact on the CSI, if there is one.  A review of clients'
expectations should be accomplished every year or two, depending on the characteristics of
the organization, the products or services it delivers, and on the specific profile of the
clientele (Rothman, 1991).

More detailed information on how to establish good monitoring practices can be found in
the following OCG Evaluation and Audit Branch Publications: Measuring and Monitoring
Program Performance and Service to the Public - Developing Useful Performance Indicators
(March 1991); and Line Managers and Assessing Service to the Public (April 1991).

                                       Annex I


  Key Steps in Developing and Implementing a Client Satisfaction Assessment Program


1-     Obtain senior management and program managers commitment

       Useful client satisfaction measurement and monitoring programs require commitment
       to develop and implement.  The process can create organizational stresses, so it
       requires time and patience and involves a certain amount of financial and human
       resources.  

       Although the lead should come clearly from the top, an authoritarian, top down,
       confrontative approach will likely be too threatening to achieve the best results from
       client satisfaction measurement.  Management has to set the right climate for the
       implementation of the program.  In most circumstances a participatory approach
       offers best chances of success.

2-     Involve employees

       Organize workshops or create working groups to determine precisely (1) who are
       your clients (i.e., the different types of clients, "direct" and "indirect", the cultural and
       socio-economic profile of the clientele, etc.); (2) what is the program doing
       (producing) to serve these clients; and (3) why does the organization want to
       measure client satisfaction and what it intends to do with the information (i.e.,
       definition of the objectives of the client satisfaction measurement program).

3-     Determine information needs

       As owner of the information, program managers should decide what type of
       information and reporting frequencies will be needed.

       The ultimate value of collecting information on the level of satisfaction of clients will
       depend on how much it is used.  Almost any expenditure will not be warranted if the
       information is not used, mainly to improve programs or service delivery.  Ownership
       of the process by managers, which allows the gathering of an information tailored to
       their needs, is the best guarantee of utilization.

4-     Determine how the information can and will be gathered

       Professional advice and assistance is fundamental at this point.  The skills of program
       evaluators, external consultants or any other experts in performance measurement
       are essential when it comes to deciding (1) what instruments and methods will be
       used to gather information, and (2) how client satisfaction will be monitored.
       There is no "universal recipe".  The choice of a measurement strategy and
       methodological tools will depend of the type of clientele surveyed, the type of
       information to be gathered, and the amount of resources that the organization can
       spend on client consultation.

       The benefit derived from the use of the information should not be surpassed of
       course by the cost of collecting data, processing it and distributing it to managers
       within the time frame required for decisions.  There are often trade-offs to be made
       between the type and quality of the information obtainable from alternative data
       collection methods and the cost and administrative feasibility of these methods. 
       Readily available data (as long as they are valid and useful) and sampling techniques
       should be used to the greatest extent possible.

5-     Conduct the study

       The conduct of studies - i.e. data collection, analysis and results reporting - should
       be accomplished also with the help of professionals.

       Good client satisfaction studies give a fair assessment of both the expectations of
       clients and their rating of the actual performance of your organization in providing
       quality services.  Assessing clients' expectations is helpful to identify artificial
       dissatisfaction, caused by unrealistic expectations, pin-point key areas of concern for
       clients, and develop meaningful and cost-effective service standards.  Comparing
       client expectations with their rating of the actual performance of the organization will
       also allow management to identify the major gaps creating dissatisfaction and
       accordingly the specific aspects of service that need improvement.

6-     Develop strategies to improve service to public

       In the light of study findings, management should develop strategies, in consultation
       with front-line employees, to better meet client expectations or to meet these
       expectations in a more cost-effective manner.

       These strategies could include: a review of external communications practices, if
       clients' expectations are unrealistic; the development of challenging, feasible and
       legitimate (i.e., largely accepted throughout the organization) service standards/
       performance targets; a review of service delivery procedures and practices that were 
       judged less favourably by clients, etc.

7-     Communicate study results, service standards and proposed service delivery improvements

       Study results, service standards and changes in service delivery resulting from client
       consultation should be widely and openly diffused throughout the organization and
       most importantly throughout the clientele.  By communicating effectively this
       information to clients, the organization is clearly indicating that it is responsive to the
             wants and needs of clients, that it is giving due consideration to their views,
             and that it is committed to a quality service.

       Informed clients know what to expect from the program; they are not "artificially"
       dissatisfied by unrealistic expectations.

       Developing an effective communication strategy needs expertise.  Departmental
       communications services and external consultants can provide a most useful
       assistance in that area.

8-     Ensure effective implementation of service delivery improvements

       As mentioned above, the ultimate value of collecting information on client
       satisfaction will depend on how much it is used, mainly to improve service to the
       public.  Moreover, not delivering the promised changes and levels of service to
       clients is the best way to create dissatisfaction. 

       Program managers should monitor closely the implementation of the selected strategy
       to better meet clients' expectations.  They should pay particular attention to the
       timely achievement of performance targets and the respect of service standards.

9-     Monitor satisfaction regularly

       A regular monitoring of client satisfaction can contribute to motivate program
       personnel.  It allows management to see if it is moving in the right direction, if the
       changes previously implemented to improve client satisfaction have been effective. 
       
       Moreover, this monitoring can underline the need to review service standards/
       performance targets.  If the organization meets these standards or targets regularly
       and cannot at the same time achieve any increase in the level of client satisfaction
       or maintain the same level of satisfaction, chances are that the standards/targets are
       not challenging enough or that they are not focusing on key areas of concern of
       clients.

       The need for a periodic assessment of client satisfaction is reinforced by the
       continuously evolving program environment.  Changes in the size and profile of the
       clientele, in clients' expectations, in the complexity of cases treated and in the
       resource levels will all impact on client satisfaction.  A regular monitoring allows the
       organization to better adapt to environmental changes.

                                      Annex II

                     Examples of Client Satisfaction Indicators

The following examples represent a preliminary list of client satisfaction indicators for
different types of clienteles.  This list has no pretension whatsoever to be exhaustive. 
Neither should the listed examples of indicators be seen as universal; the examples given
for a specific type of clientele are not all adequate for every government program that has
such a clientele.  Client satisfaction indicators for a particular program should be limited
in number, and selected in accordance with the nature and objectives of the program and
with key satisfaction issues of clients, as identified in the assessment of their expectations.
 

1.Voluntary Point-of-Service Clients


Satisfaction with Access & Facilities :

   -   appropriateness of location
   -   appearance of building, offices and waiting areas
   -   adequateness of parking facilities
   -   appropriateness of hours of operation
   -   ease of obtaining appointment(s) or receiving service
   -   ease of obtaining desired appointment time(s)
   -   amount of waiting time
   -   comfort of offices and waiting areas


Satisfaction with Communications :

   -   appearance and clarity of signs, posted instructions/announcements
   -   service actively offered in the official language of choice
   -   ease of getting information
   -   ease of understanding of information, instructions, forms or documents provided to
       clients 
   -   ease of filling out forms
   -   extent to which service procedures and levels are understood 


Satisfaction with the Personnel :

   -   courtesy and helpfulness of support staff
   -   skilfullness, competence of officers, professionals
   -   clarity and adequateness of information/advices provided by personnel
   -   personnel efforts to keep clients informed
   -   thoroughness of personnel
   -   humaneness of personnel
   -   sameness of service provider for repeated transactions
   -   amount of time spent with client
   -   personnel more concerned by procedures or clients
   -   interest of personnel in helping clients
   -   efforts of personnel to understand/listen to clients
   -   extent to which clients are treated fairly by personnel


Satisfaction with the Service per se :

   -   extent of satisfaction with the overall quality of the service offered
   -   extent of satisfaction with the kind of services offered
   -   extent to which the service offered met client needs
   -   extent to which the service offered met client wants/expectations
   -   extent to which the program or service addresses key concerns, problems of clients
   -   appropriateness of service levels
   -   acceptance/appropriateness of user fees
   -   simplicity of procedures 
   -   timeliness of service provided


Satisfaction with Service Results/Outcome :

   -   usefulness/helpfulness of the service
   -   success of the service
   -   speed of results achieved because of the service
   -   quality of results achieved because of the service
   -   extent to which results achieved after service meet results a priori expected
   -   value of the service
   -   amount of improvement in situation after receiving service


Overall Satisfaction :

   -   likeliness of recommending service to a friend
   -   likeliness of come back (i.e., future utilization of service)
   -   extent to which the service represents good value-for-money
   -   overall assessment of satisfaction with the program or the organization



2.Entitled Point-of-Service Clients


Satisfaction with Access & Facilities :

   -   appropriateness of location
   -   appearance of building, offices and waiting areas
   -   adequateness of parking facilities
   -   appropriateness of hours of operation
   -   ease of obtaining appointment(s) or receiving service
   -   ease of obtaining desired appointment time(s)
   -   amount of waiting time
   -   comfort of offices and waiting areas


Satisfaction with Communications :

   -   appearance and clarity of signs, posted instructions/announcements
   -   service actively offered in the official language of choice
   -   ease of getting information
   -   ease of understanding of information, instructions, forms or documents provided to
       clients 
   -   ease of filling out forms
   -   extent to which service procedures and levels are understood 


Satisfaction with the Personnel :

   -   courtesy and helpfulness of support staff
   -   skilfullness, competence of officers, professionals
   -   clarity and adequateness of information/advices provided by personnel
   -   personnel efforts to keep clients informed
   -   thoroughness of personnel
   -   humaneness of personnel
   -   sameness of service provider for repeated transactions
   -   amount of time spent with client
   -   personnel more concerned by procedures or clients
   -   interest of personnel in helping clients
   -   efforts of personnel to understand/listen to clients
   -   personnel respectful of client's rights 
   -   extent to which clients are treated fairly by personnel

Satisfaction with the Service per se :

   -   extent of satisfaction with the overall quality of the service offered
   -   extent of satisfaction with the kind of services offered
   -   extent to which the service offered met client needs
   -   extent to which the service offered met client expectations
   -   extent to which the service offered met what the client thinks he/she deserves
   -   extent to which the program or service addresses key concerns, problems of clients
   -   acceptance of service levels
   -   acceptance of user fees
   -   acceptance of service/program procedures
   -   simplicity of procedures 
   -   simplicity of requirements for eligibility
   -   timeliness of service provided


Satisfaction with Service Results/Outcome :

   -   usefulness/helpfulness of the service
   -   success of the service
   -   speed of results achieved because of the service
   -   quality of results achieved because of the service
   -   extent to which results achieved after service meet results a priori expected
   -   value of the service
   -   amount of improvement in situation after receiving service


Overall Satisfaction :

   -   likeliness of recommending service to an eligible friend
   -   extent to which the service provided or program is judged to be fair
   -   extent to which the program or service is judged to be relevant/important
   -   overall assessment of satisfaction with the program or the organization


3.Compelled Point-of-Service Clients


Satisfaction with Access & Facilities :

   -   appropriateness of location
   -   appearance of facilities
   -   comfort of facilities


Satisfaction with Communications :

   -   appearance and clarity of signs, posted instructions/announcements
   -   service actively offered in the official language of choice
   -   ease of getting information
   -   ease of understanding of information, instructions, forms or documents provided to
       clients 
   -   ease of filling out forms
   -   ease of understanding of regulations
   -   extent to which service procedures and levels are understood 


Satisfaction with the Personnel :

   -   courtesy and helpfulness of support staff
   -   skilfullness, competence of officers
   -   clarity and adequateness of information/advices provided by personnel
   -   personnel efforts to keep clients informed
   -   thoroughness of personnel
   -   humaneness of personnel
   -   sameness of service provider for repeated transactions
   -   personnel more concerned by procedures or clients
   -   efforts of personnel to understand clients
   -   personnel respectful of client's rights as an individual
   -   extent to which clients are treated fairly by personnel


Satisfaction with the Service per se :

   -   extent of satisfaction with the overall quality of the service offered
   -   extent to which the service offered met what the client thinks he/she deserves
   -   acceptance of service levels, prescriptions or regulations
   -   acceptance of user fees
   -   acceptance of service/program procedures and control mechanisms
   -   simplicity of procedures 
   -   simplicity of requirements for compliance


Satisfaction with Service Result/Outcome :

   -   usefulness/helpfulness of the service
   -   success of the service
   -   extent to which results achieved after service meet results a priori expected
   -   extent to which the program represents good value-for-money
   -   amount of improvement in situation after receiving service


Overall Satisfaction :

   -   extent to which the service provided or program is judged to be fair
   -   extent to which the service provided or program is judged to be relevant/important

4.Indirect Clients


Satisfaction with Access & Facilities :

   -   appropriateness of location
   -   appearance of facilities


Satisfaction with Communications :

   -   extent to which service procedures and levels are understood 
   -   extent to which clients are aware of the benefits they receive from the program
   -   ease of getting information


Satisfaction with the Personnel :

   -   courtesy and helpfulness of personnel
   -   skilfullness, competence of personnel
   -   extent to which clients are treated fairly by personnel


Satisfaction with the Service per se :

   -   extent of satisfaction with the overall quality of the service offered
   -   extent of satisfaction with the kind of services offered
   -   extent to which the service offered met client needs
   -   extent to which the service offered met client wants/expectations
   -   extent to which the program or service addresses key concerns, problems of clients
   -   acceptance/appropriateness of service levels
   -   acceptance/appropriateness of user fees
   -   acceptance/appropriateness of service or program procedures


Satisfaction with Service Result/Outcome :

   -   usefulness/helpfulness of the service
   -   speed of results achieved because of the service
   -   quality of results achieved because of the service
   -   extent to which results achieved because of the service meet results a priori expected
   -   value of the service
   -   amount of improvement in client's situation because of the provision of the service

Overall Satisfaction :

   -   extent to which the service represent good value-for-money
   -   extent to which the service provided or program is judged to be fair
   -   extent to which the service provided is judged to be relevant/important
   -   overall assessment of satisfaction with the program or the organization


                                      Annex III

                                     References


ANDREASEN, Alan R. (1977), "A Taxonomy of Consumer Satisfaction/Dissatisfaction
Measures", in HUNT, K.H. (ed.), Conceptualization and Measurement of Consumer
Satisfaction and Dissatisfaction, Cambridge, Mass., Marketing Science Institute, pp. 11-36.

BAND, William (1990), "Create a Customer Satisfaction Index (CSI) to Improve your
Performance", Sales and Marketing Management in Canada, vol. 31 no. 7, Jul. 1990, pp.58-
59.

BAND, William (1989), "How to Measure Customer Satisfaction", Sales and Marketing
Management in Canada, vol. 30 no. 1, Jan. 1989, pp. 24-26.

BROWN, Mark G. (1990), "Defining Customer Requirements - Service Quality
Deployment", The Journal for Quality and Participation, March 1990, pp. 98-104.

BROWN, Timothy P. (1991), "Internal Research Helps to Define Service Quality",
Marketing News, vol. 25 no. 3, Feb. 4 1991, p.11.

ELLMER, R. & OLBRISCH. M.E. (1983), "The Contribution of a Cultural Perspective in
Understanding and Evaluating Client Satisfaction", Evaluation and Program Planning, vol.
6, pp. 275-281.

GALE, Bradley T. (1990), "New Frontiers", in BAILEY, E.L. (ed), Creating Customer
Satisfaction, New York, N.Y., The Conference Board, pp. 55-59.

Government of Canada (1990), Public Service 2000 - The Renewal of the Public Service of
Canada, Ottawa, Supply and Services Canada, December 1990.

HIRSHMAN, Albert O. (1970), Exit, Voice and Loyalty, Cambridge Mass., Harvard
University Press.

HUOT, Richard (1989), "Using a Customer Satisfaction Index", Customer Focus, vol. 1 no.
2, Fall 1989, pp. 4-6.

LANDON, E. Laird (1977), "Consumer Satisfaction Research Orientation Differences
between Industry and Government", in HUNT, K.H. (ed.), Conceptualization and
Measurement of Consumer Satisfaction and Dissatisfaction, Cambridge, Mass., Marketing
Science Institute, pp. 355-362.

LARSEN, D., ATTKISSON, C., HARGREAVES, W. & NGUYEN, T. (1979), "Assessment
of client/patient satisfaction: Development of a General Scale", in Evaluation and Program
Planning, vol.2, pp. 197-207.

LEBOW, J.L. (1983), "Methodological Considerations in the Assessment of Consumer
Satisfaction with Mental Health Treatment", in Evaluation Review, vol.7, pp. 729-752.

LEBOW, J.L. (1987), "Acceptability as a Simple Measure in Mental Health Program
Evaluation", in Evaluation and Program Planning, vol. 10, pp. 191-195.

MATEJA, Pete (1987), "You Can Improve Feedback from your Customers", Sales and
Marketing Management in Canada, vol. 28 no. 9, Oct. 1987, pp. 8-9.

MISON, Kenneth (1989), "How to Implement a Customer Satisfaction Improvement
Program", Customer Focus, vol. 1 no. 1, pp. 1-3.

NEESE, William T. (1991), "Don't be so Direct when Assessing Customer Satisfaction",
Marketing News, vol. 25 no. 3, Feb. 4 1991, p. 8.

Office of the Comptroller General (1991), Line Managers and Assessing Service to the
Public, Ottawa, Evaluation and Audit Branch, April 1991.

Office of the Comptroller General (1991a), Measuring and Monitoring Program
Performance and Service to the Public - Developing Useful Performance Indicators (Draft
version), Ottawa, Evaluation and Audit Branch, March 1991.

Office of the Comptroller General (1991b), Program Evaluation Methods - Measurement
and Attribution of Program Results, Ottawa, Supply and Services Canada, January 1991.

POLITT, Christopher (1988), "Bringing Consumers into Performance Measurement:
Concepts, Consequences and Constraints", Policy and Politics, vol. 16 no. 2, pp. 77-87.

POTTER, Jenny (1988), "Consumerism and the Public Sector: How Well Does the Coat
Fit?", Public Administration, vol. 66, pp. 149-164.

PS 2000 Service to the Public Task Force (1990), Report for Discussion, Ottawa, PS 2000
Secretariat, October 12, 1990.

ROGERS, Greg (1989), "Measuring your Company's Real Level of Customer Satisfaction",
Customer Focus, vol, 1 no. 2, pp. 1-3.

ROTH, V.J., BOZINOFF, L. & MacINTOSH, P. (1991), "Public Opinion and the
Measurement of Consumer Satisfaction with Government Services", Canadian Public
Administration, vol. 33 no. 4, pp. 571-583.

ROTHMAN, Frank (1991), "Service Marketers Need Sequential Design to Measure
Satisfaction", Marketing News, vol. 25 no.3, Feb. 4 1991, pp. 5-11.

SALTER, James M. (1991), "Take Systematic Approach to Measuring Satisfaction",
Marketing News, vol. 25 no. 3, Feb. 4 1991, p.9.

WARE, J.E. Jr., SNYDER, M.K., WRIGHT, W.R. & DAVIES, A.R. (1983), "Defining and
Measuring Patient Satisfaction with Medical Care", in Evaluation and Program Planning, vol.
6, pp. 247-263.

WULFSBERG, R. & PULASKI, D. (1990), "It Takes more than a Simple Survey to
Measure Customer Satisfaction", Marketing News, vol. 24 no. 11, May 28 1990, pp. 14-19.

ZIETHAML, V.A., PARASURAMAN, A. & BERRY, L.L. (1990), Delivering Quality
Service: Balancing Customer Perceptions and Expectations, New York, N.Y., Free Press,
226p.



1.     Survey and focus group techniques are further discussed in the next section. 
2.     People interested in conducting a client survey should be reminded of the federal requirements: (1) for
       ensuring that no other department has carried out or is carrying out a similar survey; (2) for approval
       of surveys by the Public Opinion Research Group at Supply and Services Canada; and (3) for having
       a registration number - to get from Statistics Canada after SSC's approval - which will be shown on the
       questionnaire.