Presenters: George H Olson and Amy Cheney, Leadership and Educational Studies, Appalachian State University, Boone, USA

The purpose of the session proposed here is to engage participants in a discussion about one way to go about the task of evaluating learning in virtual environments. We believe that theory-based evaluation models provide a promising approach for assessing the educational benefits of these learning environments. Our session begins with an overview of theory-based evaluation models, including the differences and relative advantages of the American (Weiss, 1999; Fitz-Gibbons & Morris, 1996; Brickmayer & Weiss, 2000) and the Swedish approaches (Stromdahl and Zetterman, 2002). Following the overview, participants are asked to engage in a thoughtful discussion of what constitutes theory in virtual learning environments.

In 1972, Weiss introduced the notion of theory-based evaluation as a way to evaluate program effects by comparing outcomes to what would be expected given the theory underlying the program. According to Fitz-Gibbons and Morris (1996), in theory-based evaluation the evaluator selects program features to evaluate in accordance with an explicit conceptualization of a program theory that attempts to explain how the program should produce its desired effects. In its simplest form, theory-based evaluation is aimed at discovering how an why a program works (Weiss, 1999)

A program theory reflects the program developers’ understanding about how the program’s activities were developed, how they should be carried out, and how they should work together to achieve desired results (Cole, 1999). Although desirable, it is not absolutely necessary to have a program theory in place before a program begins; rather the theory can be developed after the program has begun and may alter the practice of the program once the evaluation begins (Rogers, Petrosino, Huebner, & Hacsi, 2000). Once developed, however, the articulated theory as guides the selection of pertinent questions, method(s) and measures use in the evaluation process.

References

Birckmayer, J. D. & Weiss, C. H. (2000) Theory-based evaluation in practice: What do we learn? Evaluation Review, 24(4), 407-431.

Cole, G. E. (1999). Advancing the development and application of theory-based evaluation in the practice of public health. American Journal of Evaluation, 20(3).

Fitz-Gibbon, C. T. & Morris, L. L. (1996). Theory-based evaluation: What do we mean by theory-based evaluation? Evaluation Practice, 17(2), 177-184.

Rogers, P. J., Peterosino, A., Huebner, T. A., & Hacsi, T. A. (2000). Program theory evaluation: practice, promise, and problems. New Directions for Evaluation, 87, 5 – 13.

Stromdahl, H. & Zetterman, M. L. (in process). Theory-anchored evaluation: An attempt to increase ownership in evaluation in higher education. KTH Learning Lab, Sweden. Retrieved November 19, 2002: http://www.skeptron.ilu.uu.se/MonicaLZ/arkiv/theory-anchored-submit.pdf

Weiss, C. H. (1972). Evaluation Research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall.

Weiss, C. H. (1999). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 1. Washington, DC; Aspen Institute. Retrieved November 19, 2002: