Journal of Science Education and Technology,Vol.10,No.3,2001
Do No Harm—A Comparison of the Effects of On-Line Vs.Traditional Delivery Media on a Science Course
Regina Schoenfeld-Tacher,1,2Sherry McConnell,1and Michele Graham1
This paper presents the results of a study designed to examine the effects of distance delivery on
student performance and classroom interactions in an upper level science(Histology)course.
Outcomes were assessed by comparing performance on content pre-and posttests for students
enrolled in on-campus and on-line sections of the same course.Interactions were classi?ed
according to initiator,topic,and Bloom’s taxonomy level for content interactions.The resulting
patterns were analyzed to compare behaviors in different settings.It was found that although
the groups were indistinguishable in content knowledge at the outset of the study,by the end
of the semester,students in the on-line group signi?cantly out-performed their peers in the
on-campus section.The on-line settings had a greater proportion of high-level interactions
(according to Bloom’s taxonomy)than the on-campus setting.
KEY WORDS:Distance learning;on-line instruction;interactions;Bloom’s taxonomy;histology.
INTRODUCTION
When a new teaching technique is employed,it is essential to examine the potential risks associated with teaching in a different manner.If the new technique is to be adopted on a permanent basis,an in-depth assessment must be conducted to verify that “no harm is done”and students are not suffering any negative effects as a result of the innovation.On-line delivery is an example of an innovation whose effects have been severely criticized in recent times. College curriculum committees routinely question the academic rigor of distance-delivered courses, and there is a prevalent fear that the academic rigor of courses is being compromised in order to facilitate use of distance delivery.This concern is especially directed toward science courses,where the lack of hands-on laboratory sessions is fre-quently cited as a potential de?cit for learners(Carr, 2000).
1Anatomy W102,College of Veterinary Medicine and Biomedical Sciences,Colorado State University,Fort Collins,Colorado80523-1601.
2To whom correspondence should be addressed;e-mail:reginast@ https://www.wendangku.net/doc/0e14362624.html,
The question of whether or not electronic me-dia can in?uence learning has been a subject of debate in the literature for over15years.Clark (1983,1994a,b)argues that“media are mere vehicles that deliver instruction but do not in?uence student achievement...”(1983,p.445).In contrast,Kozma (1994b)states that“both media and methods in?u-ence learning and they frequently do it by in?uencing each other”(p.11).Based on all the arguments pre-sented by both parties(Clark,1983,1994a,b;Kozma, 1991,1994a,b),it can be concluded that the media it-self does not cause differences in learning,but does facilitate teaching methods that may affect learning.
In accordance with this theoretical perspective,the current study was designed not to compare the ef-fects of two different media on learning,but rather to examine how a new delivery medium affects teaching methods,as represented by classroom interactions.
A total of44students participated in the study.
All were enrolled in an upper level histology course at a large,land grant university in the Western US.Of these44students,11were enrolled in the on-line(dis-tance delivered)section of the course.These students completed the entire course,including all examina-tions at a distance,and did not participate in any type of on-campus instructional activity.In contrast,the 257
1059-0145/01/0900-0257$19.50/0C 2001Plenum Publishing Corporation
258Schoenfeld-Tacher,McConnell,and Graham
33students enrolled in the on-campus(traditional) section of the course met face-to-face in a classroom for three,50-min lecture periods per week and partici-pated in a3-hour,on-campus laboratory session each week.Students in the on-campus section only used the Internet to complete exams(administered in the same format as for the on-line section).
The speci?c objectives of this study were to
1.Determine if there was a difference in con-
tent achievement between students enrolled
in on-campus and on-line sections of Histol-
ogy(learner–content interaction).
2.Investigate the effect of computer-mediated
communication(CMC)on classroom interac-
tions(learner–learner and learner–instructor
interactions).
a)Examine the proportion of time devoted to
content interactions versus other types of
interactions in each setting.
b)Evaluate the quality of interactions based
on depth of thought(Bloom’s taxonomy)
3.Investigate how the presence or absence of
an instructor affected the number and type of
questions that occur in on-line group interac-
tions.
Bloom and Krathwohl(1956)developed a scheme to classify the levels and types of intellectual behavior important in learning.The resulting classi-?cation scheme is a taxonomy that consists of three domains:cognitive,psychomotor,and affective.The cognitive taxonomy classi?es questions based on their level of abstraction.Higher levels of abstraction are assumed to demonstrate increased depth of learning. It was therefore important to examine levels of ques-tioning taking place in the on-line Histology course in order to ensure the delivery medium did not harm students’learning by promoting lower level thinking at the expense of higher-order reasoning skills such as synthesis and analysis.
Another concern frequently associated with dis-tance delivery is the lack of classroom interactions. Moore and Kearsley(1996)de?ne three essential types of interaction within distance education envi-ronments.These are learner–content,learner–learner and learner–instructor interaction.Learner–content interaction describes the communication that occurs between the learner and the subject matter.These interactions enable the learners to construct their own knowledge by integrating new information into their preexisting mental structures.Due to its asyn-chronous design,on-line Histology was expected to provide students with a greater opportunity than a lec-ture class to engage in higher levels of thought,such as re?ection and deep processing,which Vygotsky (1978)deems as critical to learning and retention.
According to Moore and Kearsley(1996), learner–instructor interaction is necessary once the content has been presented in order to facilitate learner–content interactions.The instructor needs to provide opportunities for students to practice the con-cepts they have acquired and give students formative feedback on their progress,before assessing if instruc-tional objectives were met.This type of interaction was supported in the on-line Histology course through the use of frequent formative quizzes(referred to as bonehead quizzes)and on-line chat sessions moder-ated by the instructor(second author).These interac-tions helped the students assess their own progress in the course and encouraged further learner–content in-teractions.Another function of these activities was to facilitate learner–content interactions by elaborating on material with which students were having dif?culty.
Learner–learner interactions play an important pedagogical role in distance education,(Slavin,1996), as they provide opportunities for students to discuss the content with others,resulting in improved cog-nitive processing.During these interactions,student misconceptions are exposed and remedied during the negotiation of meaning that takes place while inter-acting with peers.The unique attributes of on-line en-vironments lead students to get to know each other more quickly than they would in a traditional envi-ronment(Kimbrough et al.,1998),a tendency that can be harnessed by instructors to promote creation of support networks and study groups.Students in on-line Histology demonstrated this tendency when they unexpectedly formed on-line study groups and orga-nized their own review sessions without the instruc-tor’s https://www.wendangku.net/doc/0e14362624.html,ually,a student spontaneously assumed the role of moderator(typically ful?lled by the instructor)by leading with questions and remind-ing her classmates to stay focused on the content.Sim-ilar to the instructor-led sessions,students also called on each other to ask questions,pulling in students who were watching but not actively participating. METHODS
This study employed a combination of qualita-tive and quantitative methods to examine the ef-fects of computer-mediated communication on stu-dent learning.Observations were conducted in two sections(on-line and on-campus)of the same course,
Do No Harm:Effects of On-Line and Traditional Delivery Media on a Science Course259
taught by the same instructor during a single semester. Since the participating instructor had sole responsi-bility for the on-line section but was part of a teach-ing team for the on-campus course,this study exam-ined student achievement only for the instruction she designed and presented in both modalities.This spe-ci?c content was selected to avoid confounding the results by comparing instruction developed by differ-ent faculty,as noted by Clark(1983).Clark explains that a large portion of the favorable results attributed to media use may be caused by“systematic but un-controlled differences in content and/or method,con-tributed unintentionally by different teachers or de-signers”(p.448–49).
Participating students in both sections completed the same pretest,consisting of25multiple choice questions.The instructor presented the same mate-rial,using the same clinical examples in both situa-tions.Learning outcomes were assessed by student achievement on course exams.Identical exam ques-tions were asked of students in both sections,but at different times during the semester.A total of32mul-tiple choice questions were used to calculate a posttest score for each participant.Student scores on pre-and posttests were compared by t-tests and an analysis of covariance in order to determine if there were any dif-ferences in performance between students enrolled in each of the sections.
Classroom interactions were observed in three settings:on-campus lecture,on-line chat(instruc-tor present),and on-line review(instructor absent, student-organized study sessions).Throughout this paper,the term“chat”will be used to denote instructor-led synchronous sessions,while“review”will indicate the student-run synchronous sessions (instructor absent).Although student-content inter-action undoubtedly takes place in more situations, these settings were selected for their accessibility to the researchers.As a group,these settings represent a continuum of environments,from a very familiar structured setting(traditional lecture),to a novel un-structured setting(on-line review session organized and facilitated by students).
The on-line Histology course is structured us-ing the common concepts of lectures and laborato-ries as an organizational metaphor.WebCT was se-lected as the delivery software.At the end of each lecture,students are led to a quick self-assessment quiz about content learning.In the laboratories,stu-dents view microscope images captured at various powers(low,medium,high and oil immersion)to sim-ulate the process of moving a microscope objective.Four multiple-choice and laboratory identi?cation ex-aminations,each comprehensive,determine student grades in the course.During the spring semester2000, on-line Histology was offered concurrently with the on-campus course,with the same instructor in both courses(entirely for the on-line version,and for about a third of the on-campus course).
Transcripts of interactions in each setting were independently coded and analyzed by two of the au-thors who then cross-checked for reliability.An inter-action was de?ned as any utterance from a participant in the form of a question.Each question was coded into one of four topic categories:
?Content–any question directly pertaining to
course material
?Administrative–questions regarding adminis-
trative details of the course,such as due dates
for assignments
?Management–questions used to manage the
?ow of a class,such as prompting to move on
to the next topic.
?Social–all questions of a nonacademic nature.
Content questions were further classi?ed accord-ing to the demonstrated level of abstraction or depth of thought,in accordance with the de?nitions estab-lished in Bloom’s taxonomy(Bloom and Krathwohl, 1956):
?Knowledge–Simple recall or recognition of a
concept.
?Comprehension–Interpretation,translation.
At this level,the student must be able to
demonstrate use of abstraction when asked to.
?Application–Requires application of an ab-
straction to new problem without being shown
how to do it in a new situation.
?Analysis–Breaking down material into con-
stituent parts and detecting relationship of
these parts in the whole.
?Synthesis–Putting together parts and elements
to elucidate a previously poorly de?ned pattern
or structure.
?Evaluation–Judging the extent to which
ideas,solutions,methods,and materials satisfy
criteria.
Data were statistically analyzed via t-tests, ANCOVA and ANOVA.When appropriate,posthoc comparisons were conducted using the LSD test. An alpha value of.05was selected as the signi?-cance level for all tests.The data were analyzed us-ing each session as an independent observation.For
260Schoenfeld-Tacher,McConnell,and Graham
each session observed,the percentage of interactions in each category was computed.These percentages were then weighted according to the total number of interactions taking place in each session.
DATA SOURCES
Subjects for this study were students,enrolled in two sections(on-campus and on-line)of a Histol-ogy course at a4-year university,and their instructor. Eleven of the students agreeing to participate were enrolled in the on-line section and33in the on-campus section.
Data were collected from a combination of course exams,chat transcripts,and direct observa-tions(lecture).The classroom management software used to run the course generated chat transcripts and tracked student performance on both pre-and posttests.An observer manually recorded data on on-campus interactions.
RESULTS
Academic Outcomes
At the outset of the study,there was no statisti-cally signi?cant difference in academic performance between students in either section,as measured by a content pretest.However,posttest results were signif-icantly different(t=?2.032,p<.05),with students in the on-line section outperforming their counterparts in the on-campus session by an average of seven per-centage points(Table I).Since the on-line group had a slightly lower pretest mean,the observed cross-over effect was further examined through an analysis of co-variance(Table II).When the effects of pretest per-formance are controlled for(by using pretest scores as a covariate),the effect of instruction type(on-line vs.on-campus)on posttest scores is statistically sig-ni?cant(F=5.95,p<.05),with a small to medium effect size(η2=0.192)according to Cohen(1988). Table I.Academic Performance on Content Tests(Percentage Scores)for Students in Each Course Section
On-line On-campus
Test N M SD N M SD df t
Pretest610.679.352215.0911.51260.863 Posttest1180.1110.673172.7810.1540?2.03?
?p<.05.Table II.Analysis of Covariance of Posttest Scores as a Function of Course Section,With Pretest Scores as Covariate
Source df MS Fη2 Pretest Score(covariate)1 1.080.120.000 Course section1542.17 5.95?0.192 Error2591.15
?p<.05.
Rate of Interactions
The average rate of interactions was computed by dividing the total number of interactions in each setting by the total amount of time elapsed over all sessions.A one-way ANOVA(Table III)proved sig-ni?cant(F=6.07,p<.01),so a posthoc LSD test was conducted.As expected,the rates of interaction were signi?cantly higher in both on-line settings(review and chat)than for the on-campus lectures(Table IV). However,since the on-campus lectures were intended to present material,whereas the on-line sessions were intended to elaborate upon previously covered mate-rial,the signi?cance of these differences has limited practical implications.
Originator of Interactions
A one-way ANOVA(Table V)revealed a signif-icant difference(F=6.49,p<.001)in the percentage of interactions initiated by students and instructor in each setting.Posthoc tests showed students initiated a larger percentage of interactions(weighted aver-age,calculated using each session as an independent observation)during on-line class sessions than in on-campus sessions(Table VI).Within a setting,there was no signi?cant difference in the percentage of in-teractions initiated by the instructor or the students. Topic of Interactions
When the topic of interactions was examined, the greatest amount of content interactions oc-curred during on-campus sessions,whereas the largest Table III.One-Way Analysis of Variance Summary for Rate of
Interactions by Setting
Source df SS MS F Between groups26328.513164.26 6.07??Within group2412521.03521.71
Total2618849.54
??p<.01.
Do No Harm:Effects of On-Line and Traditional Delivery Media on a Science Course261
Table IV.Mean Rates of Interactions and Standard Deviations
for Each Setting
Interactions per hour a Setting N M SD
On-line chat954.2929.36
On-line review841.2920.31
On-campus lecture1018.3117.45
a Posthoc tests determined that the average rates of interactions
per hour for on-line chat and on-line review sessions were not signi?cantly different from each other.However,both rates were found to be signi?cantly higher than the rate of interac-tions per hour for on-campus lecture sessions. percentage of social interactions took place during on-line review sessions(Table VII).A two-way ANOVA (Table VIII)was then conducted to examine the ef-fects of initiator(students or instructor)and setting (review,chat,or lecture)on the percentage of inter-actions devoted to each topic.Since signi?cant results were found for all topics at the p<.001level,posthoc tests(Table IX)were conducted to examine the pat-terns for each topic.Students were most likely to initi-ate content interactions in a lecture setting,followed by a chat,followed by a review.The instructor was more likely to initiate a content interaction in a lec-ture setting than in a chat.In both lecture and chat environments,students were more likely to initiate content interactions than the instructor.
It is interesting to note that no social interactions occurred in a lecture setting.Students and instructor were equally likely to initiate social interactions dur-ing chat sessions.When the frequency of social inter-actions generated by students was examined,it was found that a signi?cantly larger proportion was ini-tiated in review sessions than in chat sessions.The pattern of administrative interactions is unremark-able.When management interactions were examined, an interesting trend emerged.Students initiated more management interactions in review sessions than they did in chat sessions,and both amounts were greater than for lecture sessions.The instructor started more management interactions in chat sessions than in on-campus lectures.Whenever the instructor and Table V.One-Way Analysis of Variance Summary for Initiator of Interactions by Setting,Using Weighted Data
Source df SS MS F
Between groups33973.061324.35 6.49???Within group346938.75204.08
Total3710991.81
???p<.001.Table VI.Average Rates of Interactions and Standard Deviations by Session,Using Weighted Data
Interactions per hour
Setting and initiator M SD LSD Posthoc On-line chat
a)students20.7913.83a>c;a=b
b)instructor33.5020.29b>d;b=a On-campus lecture