Evaluation Methods

Within the context of formative and summative evaluation, data may be collected through quantitative and qualitative methods.

Quantitative evaluation:

• Involves asking questions which can be statistically tabulated and analyzed, frequently using a scale, check list, or yes/no responses.

• Limits students to responding to the categories made available to them.

• Needs a large student sample for relevant statistical analyses.

Quantitative methods may be most useful for gathering information on large numbers of respondents for whom more in-depth, personalized approaches are not feasible. However, they do have some significant drawbacks:

• Many distance education courses have relatively small class sizes with students from various backgrounds. These small, stratified populations typically defy relevant statistical analysis.

• Quantitative surveys typically result in a rate of return of under 50 percent. A low rate of return often suggests that only those feeling very positively or negatively about the course responded to the evaluation.

• By definition and design, forced choice surveys offer respondents a limited number of possible response options. Therefore, fresh insights and unique perspectives falling outside the provided response categories go unreported.

• The cumbersome and often tedious nature of quantitative data collection can discourage formative evaluation, and often results in an over-reliance on summative evaluation.

• Statistical analysis often results in an illusion of precision that may be far from reality.

Qualitative evaluation:

• Is typically more subjective.
• Involves gathering a wider range and depth of information.
• Is more difficult to tabulate into neat categories.
• Will be less affected by typical small class size.
• Is a more flexible and dynamic method.
• Is not limited to pre-conceived topic of inquiry.
• Allows for student output of topics.

Can use:

• Open ended questioning -- with respondents asked to identify course strengths and weaknesses, suggest changes, explore attitudes towards distance delivery methods, etc..

• Participant observation -- with the distance educator observing group dynamics and behavior while participating in the class as an observer, asking occasional questions, and seeking insights regarding the process of distance education.

• Non-participant observation -- with the distance educator observing a course (e.g., an audioconference, interactive television class, etc..) without actually participating or asking questions.

• Content analysis -- with the evaluator using predetermined criteria to review course documents including the syllabus and instructional materials as well as student assignments and course-related planning documents.

• Interviews -- with a facilitator or specially trained individual collecting evaluative data through one-on-one and small-group interviews with students.

What to Evaluate

Consider the following areas:

• Use of technology - familiarity, concerns, problems, positive aspects, attitude toward technology.

• Class formats - effectiveness of lecture, discussion, question and answer; quality of questions or problems raised in class; encouragement given students to express themselves.

• Class atmosphere - conduciveness to student learning.

• Quantity and quality of interaction with other students and with instructor.

• Course content - relevancy, adequate body of knowledge, organization.

• Assignments - usefulness, degree of difficulty and time required, timeliness of feedback, readability level of print materials.

• Tests - frequency, relevancy, sufficient review, difficulty, feedback.

• Support services - facilitator, technology, library services, instructor availability.

• Student achievement - adequacy, appropriateness, timeliness, student involvement.

• Student attitude - attendance, assignments submitted, class participation.

• Instructor - contribution as discussion leader, effectiveness, organization, preparation,enthusiasm, openness to student views.

Evaluation Tips

• Check out and adapt already published questionnaires; there’s no need to re-invent the wheel.

• Draft and revise questions; change if necessary.

• Make use of follow-up probes:

• Alternate between instruction and interaction.

• Sequence your questions for best effect - go ahead and ask for suggestions for improvement before asking for what is good. This will help convey sincerity for seeking improvements.

• Place open ended questions after quick answer questions. This gives students built-in thinking time.

• On summative evaluation, assure anonymity. This can be accomplished by having all questionnaires sent to a neutral site where they would be removed from their envelopes and forwarded to the instructor without a postmark.

• Establish rapport by being interested and supportive. Withhold judgmental responses.

• Adapt to the student in degree of formality and pace of communication.<

• Use evaluation as a method for understanding teaching and learning.

• Try to get both positive and negative feedback. It is important not only to know what is not working, but also what is working.

References

Angelo, T. & Cross, P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco: Josey-Bass Publishers.

Willis, B. (1993). Distance education: A practical guide. Englewood Cliffs, NJ: Educational Technology Publications.

For Further Information

This guide is one in a series developed by Barry Willis and the University of Idaho Engineering Outreach staff highlighting information detailed in Dr. Willis' books, Distance Education–Strategies and Tools and Distance Education–A Practical Guide. Other guides in this series include: