|
[Note: For full text of Evaluation Plan, please
download the complete file
in either MS Word or PDF format with the link provided on the top left above.]
Introduction
Creating a Medical Home for Asthma (CMHA) involved active
collaboration among several public and private organizations,
including the New York City Department of Health's Bureau of
Child Health (BCH), Columbia University College of Physicians
and Surgeons, and the Medical and Health Research Association
of New York City, Inc. With funding from the National Heart,
Lung, and Blood Institute (NHLBI, Grant No. HL56348), the CMHA
project staff developed, pilot-tested and evaluated the original
version of this program in a controlled study from 1990 to 1995.
Creating a Medical Home for Asthma is an asthma management program that encourages public
health clinics to implement a team-based approach to pediatric
asthma management and care. The team-based approach encourages
all clinic personnel to work together as a team to effectively
deliver patient-centered asthma management and care. Therefore,
CMHA is designed to offer a training program that teaches clinic
staff-including physicians, nurses, laboratory technicians, clerical
staff, and receptionists-strategies to increase communication
between the patient and health care provider, as well as to deliver
effective asthma treatment using the latest treatment protocols.
Any clinic wishing to implement the CMHA program will most
likely want to evaluate the program as well. Program evaluation
allows you to determine if the program is indeed useful to your
constituents, whether they are the patients, clinic staff, or
other clients. It will help you answer questions such as:
 Did
clinic staff find the CMHA training sessions helpful in adding
to their knowledge and skills in asthma management?
Is
the CMHA program being successfully implemented in the clinics?
How
has the CMHA program affected asthma care services since it was
implemented? For example, are asthma services more efficient
and more cost-effective?
Have
asthma patients or their caregivers perceived improvements in
the clinic's delivery of asthma care services?
The following sections offer a guide for the assessment of
the CMHA program. You can use this evaluation plan as a framework
for implementing your evaluation according to your objectives.
The plan provides suggestions on evaluation design considerations,
levels of measurement, implementation of the plan, and sample
evaluation instruments to be used by patients, clinic staff,
and clinic management.
| |
Design Considerations
Initial Key Considerations
Many types of evaluation plan designs are available for assessing
the success of intervention programs. However, keep in mind that
it is not a matter of merely selecting one of these approaches.
Rather, you should design your evaluation approach by carefully
addressing the following key considerations:
For
what purpose is the evaluation being done? That is, what do you
want to be able to decide as a result of the evaluation?
Who
are the audiences for the information from the evaluation? Will
these be your patients, clinicians, managers, or other clients?
What
kinds of information are needed to support the decision you need
to make and/or to inform your intended audiences? For example,
is the information to help understand how the program operates,
the clients (i.e., patients and/or staff) who experience the
program, the strengths and weaknesses of the program, the benefits
to the clients (outcomes), or how the program is progressing?
From
what sources should the information be collected? Do these include
the patients, clinicians, managers, other clinic personnel, or
other clients?
How
can the information be collected in a reasonable fashion? By
what means (e.g., questionnaires, interviews, examining documentation,
observation, etc.) will the information be collected?
When
is the information needed?
What
resources (such as cost, time, labor effort) are available to
collect the information?
How
might the organizational environment (e.g., healthcare delivery
setting) present potential obstacles (e.g., impediments to data
collection and analysis, potential disruption to patient care,
political/organizational issues) to the evaluation effort?
The PRECEDE/PROCEED Model
A useful framework for planning and implementing health education
and health promotion programs is the PRECEDE/PROCEED model designed
by Lawrence Green and Marshall Kreuter Underlying this model
is the principle that the most enduring health behavior change
is voluntary in nature. This principle is particularly important
when a systematic planning approach is used that involves people
who actively participate in decisions about the behavior change.
As shown in Figure 1 below, the first five phases of the PRECEDE-PROCEED
model are assessments, while the four remaining ones are implementative
and evaluative. This model provides the foundation upon which
implementation and evaluation activities can be conducted. It
is also useful for examining both short- and long-term results
of the program.
Figure 1. PRECEDE-PROCEED Model [*]
[* Source: Green, L., & Kreuter, M. (1999). Health
Promotion Planning: An Educational and Environmental Approach.
3rd edition. Mountain View, CA: Mayfield Publishing Co.]
Concepts from the five diagnostic phases refer to clinic and
community needs assessments, which have been presented in the
document, "Introduction: Creating a Medical Home for Asthma
in Your Clinic." For the purpose of evaluation design, the
last four phases will be briefly described below, along with
some guiding considerations for their adoption in evaluating
the CMHA program.
Implementation (of Intervention Program)
This is the program development and implementation step. Some
key considerations are that the implementation be:
Based
upon some theoretical model (such as the PRECEDE-PROCEED model)
for addressing the problem(s) identified in the assessment phase
of the planning process;
Have
specific measurable objectives are established on the basis of
the assessments;
Have
specific activities identified to achieve the objectives;
Have
a specific timetable established when each activity is to be
done.
Using the PRECEDE-PROCEED model, you can plan your evaluation
of the CMHA program in your clinic by engaging the last three
phases of the model: process evaluation, impact evaluation, and
outcome evaluation. Each step may entail assessment of the opinions
and perceptions of three clinic groups: patients, staff, and
clinic management.
Process Evaluation
The emphasis on process evaluation and the next two phases
serves to improve the implementation process. Process evaluation
is ongoing, starting as soon as implementation takes place. Process
objectives focus on understanding how a program works-that is,
how does it produce the results it does. These objectives are
evaluated according to the timeline created.
Is
the program reaching the target group (clients, staff)?
Is
the program being delivered as intended?
Is
the program achieving the learning objectives (new knowledge,
changed beliefs and attitudes)?
Impact Evaluation
The impact evaluation process addresses several questions:
Has
the program achieved intended changes in the behavior of staff
and patients?
- For example, has the percentage of children with persistent
asthma receiving long- term controller therapy increased?
- Has the program changed the way staff interact with patients?
- Has the program changed the willingness of families to make
and keep scheduled follow up visits?
The program's immediate effects are evaluated to determine
the extent to which it needs modification.
Outcome Evaluation
Finally, after enough time has passed, as specified by the
program objectives, the ultimate intended effects or benefits
to the community (or program clients) are assessed. A key question
that could be investigated is:
Were
there improvements in the health status and quality of life of
the target population?
Attention can now be turned to the target population(s) for
whom the program was intended to serve. The next section provides
a guide for the development of evaluation instruments that can
be used for collecting data from patients, staff, and the organization.
| |
Evaluation Measures: Patient-, Staff-, and Organizational-Level Domains
Because the CMHA program involves a patient-staff-organizational
approach, it is helpful for any evaluation plan to examine evaluation
measures for each of these three groups. The following are some
key domains and indicators for consideration within each group.
Based on the specific focus of the clinic, particular domains
and their associated items can be selected for inclusion in the
development of evaluation instruments.
Patient Level
Shortly after the CMHA program has been fully implemented
at your clinic, it is be useful to survey patients to assess
their perception of and satisfaction with the health care they
received during visits. This allows clinic staff and management
to evaluate the impact of their learning experiences from the
training sessions on patient care and satisfaction. A pre- and
post-training survey design (i.e., conduct survey before and
after the CMHA training sessions took place) can be used to assess
any changes that might be due to the training.
The following are several key topical domains and suggested items that could be used for each domain:
Perception
of delivery of care
- Scheduling a visit (previsit)
- Getting to the clinic; waiting area/time (preconsultation/treatment)
- Seeing/interacting with healthcare provider(s) (consultation/treatment)
- Being referred to an asthma specialist (special consultation/treatment)
Satisfaction
with care
- Attentiveness/helpfulness of clinic administrative staff
- Attentiveness/helpfulness of healthcare provider(s)
- Helpfulness of visit to consultation/treatment needed
- Satisfaction with overall visit.
Staff Level
Process evaluation can be conducted with clinic staff
at the end of each training session and/or upon completion of
the entire session series. Clinic staff can also be surveyed
to assess the process and impact of the CMHA program on
their patient management practices. A pre- and post-program evaluation
design can also be used.
The following are some key domains and their associated question
items that can be used:
Perception
of training aspects of program
- Rating of the trainer/speaker: subject knowledge; preparedness;
communication skills; attentiveness; overall; comments/suggestions
- Rating of the program: content; length; materials; helpfulness;
overall; comments/suggestions
Expectations
versus what was experienced/gained from the program
- Program structure
- Program content
- Delivery of information/materials
- Knowledge/skills gained
Perception of change in relationships with patients
- Comparison with previous visit(s)
- Adequacy of time spent with patient
- Quality and effectiveness of patient interactions
Impact
on work flow
- Impact on routine patient care
- Impact on other responsibilities
Impact
on role/responsibilities
- Change in attitude
- Change in practice
- Application of knowledge/skills.
Organizational Level
Clinic management and perhaps higher-level administrators,
may want to measure change (due to implementation of the CMHA
program) in terms of overall program objectives/goals. Here,
the impact and outcome evaluation phase of the PRECEDE-PROCEED
model can be applied to address the focal question: was the overall
goal of the CMHA program reached? Three important areas might
be of particular interest:
How
has the program affected any component of the clinic's asthma
care services? (impact)
How
can the organization continue to monitor the program's effect?
(process)
Has
the program improved the health and quality of life of patients?
(outcome)
These three evaluation areas include:
Indicators
of quality and continuity of care
- Assessment of severity of asthma at each clinic visit
- Prescription of long-term controller therapy for persistent
asthma
- Providing each patient with a written treatment plan
- Identifying the medical and psychosocial needs of the patient
- Providing services by the same provider(s) or team of providers
for each individual or family
Clinic
management: monitoring of program impact
- How much of program objectives/activities implemented
- How often program impact should be assessed.
Indicators
of health and quality of life outcomes for clinic patients
- Hospitalization rate for asthma
- Urgent visits to the clinic or the ED for asthma
- Measures of quality of life or activity limitation
Note that although these indicators obviously vary for each level, and will be assessed separately for each, for the overall evaluation they will feed into a logical whole picture of the program's impacts and shortcomings. This type of analysis and reporting of the findings are included in the sample evaluation plan in the next section.
| |
Implementation of Evaluation Plan
Prior to the implementation of any evaluation plan, it is
essential to have a clear idea of how to proceed with the entire
evaluation process. This begins with a well-defined set of objectives
for the evaluation, followed by preparatory activities that lead
up to the completion of an evaluation plan. Often, instrument
development is a major obstacle in the evaluation process. However,
this can be overcome by examining and modifying existing evaluation
instruments to meet the predetermined evaluation objectives.
Finally, the last three sets of activities-data collection, data
processing and analysis, and reporting-will require careful consideration
of clinic resources, time and schedule demands, and other operational
prerequisites for their completion.
As previously mentioned in the description of the PRECEDE-PROCEED
model, the focus on each of the three types of evaluation-process,
impact, and outcome-serves to improve the program implementation
process. Evaluation is an ongoing activity that begins when program
implementation proceeds and culminates with examining both end
results and long-term results through impact and outcome evaluations.
The list below offers clinic management an outline of activities
for the implementation of its evaluation plan.
Objectives
- Decide what you want to evaluate
- Decide on which of the patient-staff-organization levels
you wish to focus
Preevaluation
Activities
- Gather background information
- Decide on objective(s)
- Compile potential evaluation instruments
- Complete evaluation plan
Instrument
Design
- Consider the following:
- Instrument title
- Introductory statement
- Demographics
- Directions
- Questions
- Format
- Conduct pilot testing to ensure instruments work
- Prepare final draft
- Conduct pilot testing to ensure instruments work
Data
Collection
- Identify and train data collectors
- Set a schedule
- Pilot test data collection processes
- Implement data collection (collect data anonymously)
Data
Processing and Analysis
- Develop analysis plan
- Collect and clean your data
- Develop a coding scheme (if applicable)
- Key/tabulate data
- Obtain and check analysis outputs
- Check analysis plan
Reporting
- Summarize results from data analysis
- Compile a meaningful report
If your timeline for each set of actions is driven largely by internal information needs, the schedule and types of activities included in the evaluation plan can be altered to best suit the clinic's interests. You may find, however, that establishing a set period to begin with the design and end with the reporting step will help ensure that the results more successfully capture a useful "snapshot" of the program.
| |
|
