Step 8: Make Improvements through Evaluation
Coalition evaluations can provide the assigned staff person, lead agency, and coalition
members with important feedback. Components of coalitions that should be evaluated
include objectives, activities, processes, and unanticipated events. By assessing
the processes, outcomes, and impacts associated with coalition activities, staff
can improve their outreach and coordination skills, and members can determine which
strategies help the coalition achieve its ultimate goals most effectively. The results,
if positive, can also help the coalition improve its reputation within the community
and can be included in future resource development proposals. Furthermore, when a
coalition modifies its efforts to eliminate problems pinpointed by an evaluation,
the coalition's credibility can improve significantly.
Coalitions can employ two basic types of evaluations. These are named formative and
summative evaluations. Formative evaluations focus specifically on the coalition's
process objectives. For example, a coalition may want to encourage the media to promote
bicycle safety. A formative evaluation would analyze the process by which the coalition
attempted to achieve this goal. Questions in the formative evaluation might include:
How many members actively monitored the local media on a regular basis? How many
times did staff and members meet with local media representatives to encourage safe
bicycling pictorials? How many times did the coalition submit press releases or letters
to the editor? The results of formative evaluations help staff and members improve
the functioning of the coalition.
Summative evaluations help coalition members to determine whether or not the coalition's
strategies resulted in the desired consequences. Summative evaluations help assess
both outcome and impact objectives. To evaluate outcome objectives in the example
described above, a summative evaluation would include questions like the following:
Did the local media organizations that the coalition contacted change their practices
to include photos of safe bicycling? How many coalition sponsored activities received
coverage in the local press? To assess impact objectives, the summative evaluation
might include a component that analyzed changes in parents' and children's attitudes
and behaviors after reading coverage of coalition activities. Were parents, for example,
influenced to purchase bicycle helmets after reading the coalition's articles? The
answers to summative evaluation questions help coalition members make strategic decisions
about strengthening promising interventions and discontinuing ineffective ones.
Evaluating coalition efforts is not simply a matter of evaluating the effect of the
coalition's planned activities on injury prevention. What can be overlooked are the
myriad effects a coalition can have, whether it achieves its stated goal or not.
Because coalition building stimulates a variety of interventions and activities,
evaluation results must be interpreted thoughtfully. Critical to any evaluation planning
is the documentation of unintended successes. For example, as a result of the VPC's
trainings on school-based violence prevention, the state's health education curriculum
was revised to include violence prevention. This was an unanticipated result of the
coalition's efforts that was nevertheless quite significant.
Furthermore, a coalition's visibility may increase public awareness and the community's
perception of the problem. For example, a toy store manager who read about IPC's
pedestrian safety campaign in the newspaper encouraged the toy store's corporate
headquarters to sponsor the campaign in its stores. For a limited time, all customers
received free bumper stickers encouraging safe driving practices. These examples
represent the types of side-effects that often occur as a result of coalition efforts.
Other spin-offs might include: liaisons between agencies that previously had not
worked together, increased rates of cross referral, and improvements in the skills
and morale of coalition participants. These effects can augment more formal evaluation
results, thus enhancing the coalition's sense of effectiveness and legitimacy. Sometimes
these results are difficult to judge. A new coalition may experience "textbook
success" or "textbook failure." But usually not all of the outcomes
could have been predicted at the initiation of the effort. Therefore, all facets
of coalition life must be taken into account in a summation of efficacy.
Evaluation is an on-going process throughout the life of a coalition. Every major
coalition event should be evaluated. Surveys of coalition members will give lead
agencies an idea of the level of involvement of each member. It is ideal to evaluate
whether or not further collaboration between members may occur in addition to their
participation in the coalition. This information may be especially useful for formative
evaluations. Likewise, simple pre/post tests and satisfaction surveys work well for
trainings, courses, and conferences. Content analysis of meeting agendas, minutes,
and attendance lists will help determine if process, outcome, and impact objectives
were met and will help identify unintended successes. Taking the time to evaluate
the effectiveness of coalition efforts is a way of acknowledging that the skills
and contributions of coalition members are important. Honest reflection also assures
that the coalition grows from its experiences, regardless of the programmatic outcome.
Evaluating a coalition can lead to changes in a coalition's approach. In addition,
evaluation can increase a coalition's effectiveness and can assure that the community
and participants benefit from the coalition's activities. Coalition evaluation is
a newly emerging field, and more work needs to be done. However, the availability
of evaluation tools is increasing, and current evaluation efforts are strengthening
the on-going work of coalitions.
Coalitions do not last forever. Sometimes a coalition can be repaired, and sometimes,
the effort to do so is not justified. Be ready to dissolve a coalition if it does
not achieve satisfactory goals or if it is no longer effective. Sometimes it is best
to walk away with a handshake and a smile. At other times a celebration at the conclusion
of a successful campaign is a great way to acknowledge the relationships forged during
the life of the coalition.
Remember, virtually every carefully-crafted coalition will have an impact. "An
effort may fail, then partially succeed, then falter, and so on. Since mutual trust
is built up over a period of time, coalition organizers should avoid getting so caught
up in any one effort as to view it as 'make or break'. Every effort (at cooperation
among groups) prepares the way for greater and more sustained efforts in the future."8
Coalitions consist of people. Therefore, shared efforts leave us with surprises,
memories, and mutual respect.
Black, T. R. 1983. Coalition Building: Some Suggestions. Child Welfare. Vol. LXII,
Brown, C. R. 1984. The Art of Coalition Building: A Guide for Community Leaders.
American Jewish Community.
Coalition Etiquette: Ground Rules for Building Unity. Social Policy. 1983. Vol. 14,
No. 4 :49.
de Tocqueville, A. 1840. Vol. 2. Democracy in America.
Feighery, E., and Rogers, T. 1989. How-to Guide on Building and Maintaining Effective
Coalitions. Health Promotion Resource Center, Stanford Center for Research in Disease
Fisher, R., and Ury, W. 1981. Getting to Yes: Negotiating Agreement Without Giving
Goleman, D. 1988. Why Meetings Sometimes Don't Work. New York Times. B1.
Jones, V., and Hutchins, E. 1993. Finding Common Ground: A Call for Collaboration.
Maternal and Child Health Bureau.
Massachusetts Adolescent Violence Prevention Project. 1992. Incremental Approach
to Developing a Community Based Coalition. Phase One: Basic Definitions.
Massachusetts Adolescent Violence Prevention Project. 1992. Incremental Approach
to Developing a Community Based Coalition. Phase Two: Coalition Building with a Coalition
McKnight, L., and Kretzmann, J. 1990. Mapping Community Capacity. Center for Urban
Affairs and Policy Research Northwestern University.
McKnight, L., and Kretzmann, J. 1984. Community Organizing in the 80s: Toward a Post-Alinsky
Agenda. Social Policy.
Mulford, C., and Klonglan, G. 1982. Creating Coordination Among Organizations: An
Orientation and Planning Guide. North Central Region Extension Sociology Committee.
National Assembly of National Voluntary Health and Social Welfare Organizations.
1991. The Community Collaboration Manual.
Prevention Program. 1993. The Spectrum of Prevention.
Reisch, M., Wenocur, S., and Russel-Erlich, J. 1986. Symposium of Community Organization.
Journal of Sociology and Social Welfare. Vol. XIII, No. 3.
Riker, W. 1962. The Theory of Political Coalitions. Yale University Press Notes on
Policy and Practice.
Sofaer, Shoshanna, S. 1992. Coalitions and Public Health: A Program Manager's Guide
to the Issues. National AIDS Information and Education Program Center for Disease
Straus, D., and Doyle, M. 1976. How to Make Meetings Work.
Wiesner, S. 1983. Notes on Policy and Practice, Fighting Back: A Critical Analysis
of Coalition Building in Human Services. University of Chicago's Social Service Review.
Wolff, T., and Foster, D. Monitoring and Evaluation of Coalitions. Paper presented
at Community Partners, October 1993.
1 The Violence Prevention and Injury Prevention coalitions (VPC and IPC)
do not actually exist. They do, however, consist of composites of actual groups which
have been selected to elucidate the discussions.
2 Alexis de Tocqueville, "Democracy in America," Vol. 2 (1840).
3 Developed by the Prevention Program from the work of Marshall Swift,
PhD, Hahnemann College, Philadelphia, PA. A paper, "The Spectrum of Prevention"
and a video demonstrating this methodology, "Beyond Brochures: New Approaches
to Prevention," are available through the Prevention Program.
4 Lynn Oppenheim, Wharton Center for Applied Research, as described in
New York Times article by Daniel Goleman, "Why Meetings Sometimes Don't Work"
(June 7, 1988):B1.
5 William Riker, "The Theory of Political Coalitions" in Weisner,
"Notes on Policy and Practice," Yale University Press, New Haven, CT (1962).
6 Terry R. Black, "Coalition Building -- Some Suggestions,"
Child Welfare, Vol. LXII, #3, (May, 1983):266.
7 "Coalition Etiquette: Ground Rules for Building Unity," Social
Policy, Vol. 14, #2 (Fall, 1983): 49.
8 Cherie R. Brown, "The Art of Coalition Building - A Guide for Community
Leaders," published by the American Jewish Committee (1984).
For additional copies of this paper, please
Children's Safety Network Adolescent Violence Prevention Resource Center
Education Development Center, Inc.
Western Region, 200 Webster Street, Suite 310
Oakland California, 94607
(510) 836 - 4002
Next: Appendix C. Case Study
by Bob Parker