Computer-mediated instruction_a comparison of online and face-to-face collaboration

Education Tech Research Dev(2008)56:101–124

DOI10.1007/s11423-007-9050-9

R E S E A R C H A R T I C L E

Computer-mediated instruction:a comparison of online

and face-to-face collaboration

Jeremy I.Tutty?James D.Klein

Published online:21June2007

óAssociation for Educational Communications and Technology2007

Abstract This study investigated the effects of collaboration mode and group compo-sition during a computer-mediated collaborative(CMC)program.Six intact sections of a computer literacy course were assigned to either a face-to-face or a virtual,online col-laboration treatment condition.Groups consisted of homogeneous lower-ability,homo-geneous higher-ability,or heterogeneous-ability pairs.The study examined the effects of collaboration mode and group composition on individual posttest performance,group project performance,collaborative interaction behavior,and attitudes towards the instruction.Results indicated that virtual dyads exhibited signi?cantly more questioning behaviors and signi?cantly better project performance than those who collaborated face-to-face.By comparison,students in the face-to-face condition performed signi?cantly better on the individual posttest than those in the virtual online condition.Findings suggest that both virtual and face-to-face collaboration can be effective in achieving learning goals. However,consideration should be given to the collaborative structure of the lesson and the type of task in the design of CMC environments.

Keywords Virtual collaborationáComputer-mediated instructionáInteractionsáAbility-groupingáE-learning

The?ndings reported in this paper were part of a dissertation study the?rst author conducted while a doctoral student at Arizona State University–Tempe.The authors wish to thank Howard Sullivan and Willi Savenye for their assistance in planning the study.We also acknowledge the contribution of Steve Ross and the anonymous ETR&D reviewers for their useful comments on an earlier draft of the paper.

J.I.Tutty(&)

Boise State University,1910University Drive,Boise,ID83725-1747,USA

e-mail:jtutty@https://www.wendangku.net/doc/0810273262.html,

J.D.Klein

Division of Psychology in Education,Arizona State University–Tempe,

Tempe,AZ85287-0611,USA

e-mail:james.klein@https://www.wendangku.net/doc/0810273262.html,

123

102J.I.Tutty,J.D.Klein Introduction

The use of the Internet and telecommunication technologies in education has increased in recent years with the proliferation of online learning.Over two million college students were enrolled in some form of web-based education in2002and more than90%of U.S. colleges and universities offered online options in2005(The Chronicle of Higher Education2005).

This movement towards online learning has prompted educational technologists to debate the most appropriate role for new technology(Golas2002)and motivated an educational paradigm shift from single-classrooms to knowledge-building communities of learners(Chou2001;Ravits1997).Theorists suggest that learning is more effective when students are able to discuss their ideas,experiences,and perceptions with peers(Jonassen and Kwon2001;Kanuka and Anderson1998).Some researchers have indicated that the ?exibility and technological support for such interactions available in computer-mediated environments point to collaborative learning strategies as a promising means to implement new technology(Laffey et al.1998;Oliver and Omari2001;Pena-Shaff and Nicholls 2004;Ravits1997;Strijbos et al.2004).

Yet,there is little empirical evidence to indicate if the positive effects of collaborative learning on achievement transfer to environments where communication is mediated by https://www.wendangku.net/doc/0810273262.html,parative studies should be done to investigate whether the positive char-acteristics of collaborative learning in the face-to-face settings transfer to synchronous CMC settings and to determine the effects of computer-mediated collaboration(CMC)in virtual environments.

Collaborative and cooperative learning

A common view is that small-group learning strategies reside along a continuum from loosely structured(collaborative)to highly structured(cooperative)(Bruffee1995;East-mond1995;Webb and Palincsar1996).Smith and McGregor(1992)espoused the position that collaborative learning refers to a variety of educational approaches that encourage students to work together,including:cooperative learning;problem-based instruction; guided design;writing groups;peer teaching;workshops;discussion groups;and learning communities.Because this study included strategies associated with collaborative and cooperative learning,the literature examined included studies based on both approaches. The term collaboration was selected for use in this study because it is inclusive of several small-group approaches(Smith and McGregor1992)and because it is re?ective of the literature in which it is more commonly associated with virtual learning environments (Joung2003).

Collaborative learning in computer-mediated environments

Collaborative learning strategies in the classroom setting can positively affect learning outcomes,social skill development,and self-esteem(Johnson and Johnson1996;Slavin 1990).Research also provides support for the use of collaborative learning strategies when students use computer-based instruction(CBI)(Cavalier and Klein1998;Dalton et al. 1989;Hooper1992;Hooper and Hanna?n1991;Klein and Doran1999;Kulik and Kulik 1991;Sherman and Klein1995).A meta-analysis of36studies by Susman(1998)indicated 123

Computer-mediated instruction103 that participants in collaborative,computer-based conditions demonstrate greater increases in elaboration,higher-order thinking,metacognitive processes,and divergent thinking than participants in individual CBI.

The characteristics of a computer-mediated environment appear to provide enhanced opportunities for dialogue,debate,and the potential for a sense of community(Collins and Collins1996;Naidu1997;Oliver and Omari2001).Theorists have extended this argument by suggesting that CMC could positively enhance learning,problem-solving,and other higher-order thinking(Adelskold et al.1999;Johnston1996;Jonassen et al.1999).

However,empirical evidence to support the impact of these theories on achievement in CMC learning lacks critical mass.According to Hara(2002)and Murphy and Collins(as cited in Uribe et al.2003),research on synchronous CMC has been limited to surveys of students,investigations of the recreational use of online chat systems,and evaluative case studies.For example,a case study by Scardamalia and Bereiter(1996)showed that5th and 6th grade students using CMC in all content areas demonstrated a higher level of knowledge building in a cumulative database activity compared to non-collaborative students.

Recent research has focused on CMC in the context of web-based education(Brewer and Klein2006;Gunawardena and McIsaac2004;Savenye2004).Yet many of these studies only examine factors such as attitude,motivation,participation,or learner inter-action,and do not consider student performance(Elvers et al.2003;Hoskins and van Hoof 2005;Macdonald2003;Pena-Shaff and Nichols2004;Sapp and Simon2005).Further-more,among the studies that have examined achievement in the online environment,many consider only one collaborative condition rather than compare online to face-to-face applications(Berge1999;Brewer2004;Davies and Graff2005;Hoskins and van Hooff 2005;Laffey et al.1998).Unless such comparisons are made,it is dif?cult to determine the effect of CMC in virtual environments.

Collaborative group composition

Regardless of the environment in which collaborative learning is implemented,there is agreement that simply grouping students does not promote higher achievement or more positive relationships among students(Johnson and Johnson1996).While many advocates of collaborative learning have recommended heterogeneous groupings,there is consider-able disagreement regarding the effects of such grouping on performance and attitudes of students with differing abilities(Klein and Doran1999;Sherman and Klein1995;Slavin 1993;Swing and Peterson1982;Uribe et al.2003;Webb1989).

Some of these studies suggest that heterogeneous ability-groupings assist students of all ability levels with the acquisition of knowledge and the cognitive processing of this knowledge(Johnson et al.1996;Slavin1993).But other studies suggest that for the optimal development of thinking and the maintenance of self-esteem,group members should have similar cognitive abilities(Saleh et al.2005).

Advocates of collaborative learning believe that heterogeneous groupings support the academic achievement of high-ability students by providing opportunities for deeper cognitive processing through the explanation of their own understanding to their partners (Johnson and Johnson1996;Sharan and Sharan1992;Slavin1990).It is also suggested that high-ability students acquire increased motivation and improved self-con?dence. However,other studies have shown negative or no impact on high-ability students when paired in heterogeneous dyads(Hooper1992;Hooper and Hanna?n1991;Sherman and

123

104J.I.Tutty,J.D.Klein Klein1995).Furthermore,heterogeneous groupings may bene?t the average or lower-ability students in a collaborative learning setting.Heterogeneous groupings compensate for the procedural-knowledge gaps that hinder the achievement of lower-ability students working alone(Hativa1988).

The literature is also inconclusive on the bene?ts of homogeneous ability groupings when students use collaborative learning strategies(Cohen1994).Swing and Peterson (1982)found that students of average ability perform better in homogeneous groupings than in heterogeneous.Hooper and Hanna?n(1988,1991)found that homogeneous groupings have no signi?cant effects on achievement of high-ability students.Slavin (1993)authored a meta-analysis of27studies that examine the effect of ability grouping on middle school students’achievement.He found that lower-ability students indicated more favorable attitudes toward learning when grouped with students of higher ability,but the comparison of achievement gains between ability-grouped and non grouped students was not signi?cant.

Student interactions

In addition to group composition,a prominent issue thought to impact the ef?cacy of collaborative groupings is related to the interactions that students exhibit when working in small groups.According to Sherman and Klein(1995),‘‘Studies in which group member interactions have been recorded and analyzed indicate that achievement and attitude dif-ferences are related to the type and amount of verbal interactions between students’’(p.6). Webb(1989)reported that students in small groups who give or receive explanations during a lesson learn more from the lesson than those who do not.King(1989)found that small groups in which members asked task-related questions,discussed strategy,and elaborated solutions were more successful at problem solving than groups that did not exhibit these interaction behaviors.

More recent research shows mode of collaboration to impact the frequency and quality of student interactions in collaborative environments.While some studies have found that participation and interaction is reduced in computer-mediated environments when com-pared to a face-to-face condition(Fahy et al.2001;Vrasidas and McIsaac1999),much of the research has indicated that the quality of learner–learner interactions in a computer-mediated environment may actually be better than interaction in a face-to-face environ-ment(Collins and Collins1996;Naidu1997;Oliver and Omari2001;Uribe et al.2003). Jonassen et al.(1995)indicated that the nature of CMC allows for‘‘collective thinking’’and time for re?ection not found in the face-to-face environment,leading to higher quality interactions.The process of writing and re?ecting may encourage higher-level learning such as analysis,synthesis,and evaluation,and promote clearer and more precise com-munication(Garrison1997;Sapp and Simon2005).Gunawardena and Zittle(1997)noted that social interaction between learners could contribute to their satisfaction and to the frequency of interaction in online or web-based instruction.

Purpose of the current study

The purpose of the current study was to investigate the effects of two levels of collabo-ration mode(virtual or face-to-face)and the composition of groups(homogeneous higher-123

Computer-mediated instruction105 ability,homogeneous lower-ability,or heterogeneous ability dyads)within the context of computer-mediated instruction.

Currently,there is little empirical evidence to indicate if the positive effects of col-laborative learning on achievement in face-to-face settings transfer to environments where communication is mediated by computers(Hara2002;Joung2003).Furthermore,there are few?ndings from research studies that can be used to guide the design and development of online collaborative learning environments especially when learner characteristics such as ability are considered.The current study was designed to build upon our understanding of collaborative learning and grouping strategies by extending the investigation of such strategies on performance within a synchronous CMC environment(Brewer and Klein 2006;Johnson and Johnson1996;Jonassen and Kwon2001;Reeves et al.2004;Slavin 1993;Strijbos et al.2004).Unless comparisons are made,it is dif?cult to determine the impact of CMC in virtual environments and identify appropriate considerations for maximizing increasingly complex tools for virtual collaboration in such settings.

Based on the literature review and identi?ed rationale for additional CMC research,this study was designed to address the following research questions:

1.What is the effect of collaboration mode(virtual or face-to-face)on achievement,

attitude and group member interaction in a computer-mediated,collaborative setting?

2.What is the effect of group composition(homogeneous higher-ability,homogeneous

lower-ability or heterogeneous dyads)on achievement,attitude and group member interaction in a computer-mediated,collaborative setting?

3.What is the interaction effect of collaboration mode and group composition on

achievement,attitude,and group member interaction in a computer-mediated, collaborative setting?

Method

Participants

Participants for this study were120undergraduate preservice teachers enrolled in a computer literacy course at a state university in the northwestern United States.All par-ticipants were completing prerequisite requirements for entry into the upper-division teacher certi?cation program.The sample included students enrolled in six sections of the course.Participants were predominantly Caucasian female(69%),with a mean age of22, specializing in diverse content areas.Upon entering the course,most participants rated their knowledge about spreadsheets(the instructional content used in the study)as a1on a scale of0(no knowledge)to4(a lot of knowledge).

The computer literacy course met twice a week for75min,and introduced students to basic technology skills in word processing,spreadsheets,databases,and presentation software.The Blackboard TM course management system was used as a supplement to face-to-face instruction in the course.Assignments were related to the basic function of each application,and face-to-face collaborative groups were often used during activities related to integration of technology into the classroom.In addition,this course prepared partici-pants to take a state-mandated technology competency exam,which must be passed to receive credit for the course.

123

106J.I.Tutty,J.D.Klein Materials

CBI lesson

A CBIal module on the basic functions of Microsoft Excel TM and the application of spreadsheets in the classroom was developed for this study by the?rst author using Macromedia Director TM.The CBI was a self-executable?le for use on either PC or Macintosh computer.It consisted of three parts:an introduction,a practice problem,and a group application project.

The program was speci?cally designed for use by collaborative dyads.Two scores contributed to the grade each student earned from the CBI lesson—one score was from a group application project each dyad completed as part of the program and the other score was from a posttest each individual completed at the end of instruction.Students were informed in the introduction that each score would contribute equally to their?nal score for the spreadsheet assignment.

The introduction contained16screens,which provided information for students’suc-cessful use of the program.It included instructions for using the program,a description of the collaborative nature of the lesson,the goals and objectives of the program,an overview of spreadsheets,and encouragement for students to help each other learn.A collaborative skills review reminded students to:give explanations when their partner asked for help,ask about their partner’s perceptions of the concepts,and wait to proceed through the instruction until their partner is ready.In addition,the review emphasized the importance of summarizing and listening when working collaboratively.The introduction concluded by informing students that they would be learning about the basics of Excel TM,and would be required to develop a spreadsheet to solve some problems.

The practice problem component of the instruction consisted of70screens containing a classroom-based scenario.Students assumed the role of a classroom teacher with the task of developing a class gradebook.The gradebook activity required students to determine several elements to include:class title,student names,assignments(homework,projects, quizzes,and exams),and the contribution of each assignment to the?nal grade.The gradebook would be formatted to calculate:student performance in each assignment cat-egory,student performance for the grading period,students’?nal grades as a percentage and letter,the class average for each assignment,and the highest and lowest grades for each assignment.The gradebook would also include a https://www.wendangku.net/doc/0810273262.html,pleted gradebooks were then saved to each student’s network folder and the?le path submitted to the researcher through the CBI,permitting the group to proceed to the group application project.

To encourage collaboration throughout the instruction,the practice problem was structured in two parallel tracks following a modi?ed Jigsaw procedure(Aronson et al. 1978).A coder and designer track contained different skills required during the practice exercise so that one student could not receive all of the information necessary to complete the practice exercise independently.

The coder track,consisting of32instructional screens,covered the use of formulas and functions in Microsoft Excel TM.The designer track,consisting of23instructional screens, covered the formatting of a spreadsheet for ef?cient use.The remaining15screens were common to both tracks.These screens concluded the practice activity by providing instruction and practice on the Excel TM chart wizard.

Both tracks presented the student with a list of concepts and terms that should be de?ned and mastered.The tracks contained brief explanations and screen captures to illustrate the de?nition and function of each element in the instruction.Students were 123

Computer-mediated instruction107 informed of the objectives and content within both tracks to ensure that all participants were aware of the skills required of the individual posttest.The program also contained a reminder that all of the skills included in both content tracks were required of the group application project.Once a student selected a track,he or she was unable to access the other track.This required each student to learn skills from,and teach skills to,his or her partner.Tracks were accessed by clicking the appropriate button.

To facilitate collaboration,10checkpoints were installed in the program at task-related points,encouraging the participants to check-in with their partner,explain what they had learned,or discuss how they might use each new feature in the practice project(see Fig.1). Each checkpoint was accompanied by a collaboration suggestion and installed where division of responsibilities or sharing of knowledge was critical to completing the task. There were three collaboration checkpoints in the introduction,?ve in the practice prob-lem,and two in the group application project.A?eld test of the program indicated that when each track was completed by students of similar ability,the coder track(M=31min) and designer track(M=28min)required approximately the same amount of time to complete.

The group application project consisted of a second spreadsheet activity,which required the same skills covered in the gradebook practice section.The group application project comprised the?nal seven screens of the CBI.No additional instruction was provided for this activity,but students were able to access their previous content track for review.One application project was submitted per dyad.The following is the problem scenario that was provided to students:

Now let’s put all of the design and coding skills you learned building your gradebook to use.Everyone is familiar with the monthly power bill,but have you ever wondered

how Array Fig.1CBI roles

123

108J.I.Tutty,J.D.Klein much it actually cost to run your clock radio for a month?Through the power of spread-sheets you will be able to answer this question.On the next screen,you are given a list of appliances,the kilowatt use per hour of each appliance,the number of hours each appliance is used each day,the number of days an appliance is used each month,and the cost per kilowatt.With this information you can design a spreadsheet to illustrate the cost of operating common appliances.Your spreadsheet should contain the following components:?Merged cells containing the title of your spreadsheet and a graphic.

?Column headings with appropriate labels.

?Formulas to determine the monthly total cost of electricity for each appliance based on the number of hours used each day and days used each month.

?Formulas to show which appliance is the most and least expensive to run each month.?A formula for the total monthly cost for all appliances.

?Once you have completed the spreadsheet,change the number of hours each appliance is used each day,or the number of days an appliance is used each month to re?ect how much your group uses each appliance.

Design

This study used a quasi-experimental,posttest-only control group design.It was a three (group composition:higher-ability dyads,lower-ability dyads,and heterogeneous dyads)by two(collaboration mode:face-to-face and virtual)factorial design.This study included six different treatment groups.Each of the six intact sections of the computer literacy course was randomly assigned to either a face-to-face or virtual collaboration treatment condition. Participants were then blocked by general computer ability and assigned to dyads in one of three ability compositions(homogeneous low,homogeneous high or heterogeneous). Ability blocking was based upon performance on a25-item,multiple-choice pretest.A one-way analysis of variance(ANOVA)conducted on the pretest scores showed no signi?cant differences between class sections prior to the study,F(5,136)=.08,P>.99.

Participants in each course section were sorted by ability into two groups(higher or lower ability)using a median split of the pretest scores based on the average of all participants in the sample(M=16out of a possible25correct).The average mean scores for the lower and higher ability treatment groups were14.2and18.6,respectively.Het-erogeneous(mixed-ability)dyads were formed by randomly selecting one participant from the higher-ability pool and one from the lower-ability pool.Homogeneous dyads were formed by pairing participants from the same ability pool.Participants scoring at the median were evenly distributed between both pools.Due to the composition of each course section,38students were assigned to higher-ability dyads,40to lower-ability dyads,and 42to mixed-ability dyads.

Data collection instruments

Pretest

Several weeks prior to the implementation of the study,a25-item,computer-based,four-choice,selected response pretest measuring general computer ability was administered to the participants.This pretest was used to determine student ability for assignment to dyads. 123

Computer-mediated instruction109 The items on the pretest were taken from retired versions of the Educator Technology Assessment(ETA).The pretest contained?ve items from?ve of the test categories:the computing environment,word processing,presentation software,spreadsheets,and data-bases.The following is an example of an item from the computing environment category: Which of the following is a common interface for peripheral devices?

(a)ABI

(b)CPU

(c)USB

(d)RAM

Each pretest item was worth one point.The primary researcher scored all pretests.The ETA has been administered over20,000times with an alpha reliability of.91(VanDehey and Thorsen2002).The alpha reliability for the administration in this study was.84.

Posttest

A25-item,computer-based,four-choice,selected response posttest measuring knowledge and skills covered in the instructional program was administered during the week following the spreadsheet CBI.Participants were required to identify various spreadsheet functions, terms,and the output of a given formula.Content validity was established by aligning the test items to the objectives of the CBI program.For example,the objective,‘‘Identify the correct syntax of common Excel TM functions’’was measured by items such as: Which formula will calculate the maximum value stored in cells A1through A20?

(a)MAX A1-A20

(b)MAX(A1:A20)

(c)MAX(A1-A20)

(d)MAX A(1:20)

Which formula does NOT calculate the sum of the values stored in cells B2,C2,and D2?

(a)=(B2+C2+D2)

(b)=SUM(B2+D2)

(c)=B2+C2+D2

(d)=SUM(B2:D2)

Each posttest item was worth one point,and all posttests were scored by the primary researcher.The split-half reliability coef?cient of the posttest was.88.

Attitude survey

An attitude survey was developed by the researchers to measure students’reactions to the instruction.The survey was administered with the posttest during the week following instruction.It contained18,?ve-choice Likert-type items(4—strongly agree,0—strongly disagree)and three open-ended questions.The survey was initially designed to include three sections(delivery system,topic,and collaborative work)with six items per section. The delivery system section included items related to the characteristics of the instructional program such as,‘‘The computer program was easy to navigate.’’The topic section included items related to the opinions about spreadsheets such as,‘‘The spreadsheet is a useful tool to know.’’The collaborative work section included items related to working

123

110J.I.Tutty,J.D.Klein with a partner such as,‘‘I learned the material better working with a partner than I would have on my own.’’All survey items are presented in the results section of this paper.

The dimensionality of the18items was analyzed using a maximum likelihood factor analysis.Based on the original design and a scree test,three factors were rotated using a Varimax rotation.The rotated solution yielded three interpretable factors.The original category titles were retained,but the item distribution was modi?ed based on factor loading.Seven items loaded on the factor of delivery system and seven items on the factor of collaborative work.Four items loaded on the topic factor.The delivery system factor accounted for38.4%of the item variance and had an alpha reliability of.86.Collaborative work accounted for11.5%of the item variance and had an alpha reliability of.86.The topic factor accounted for8.5%of the item variance and had an alpha reliability of.72.

Attitude interview

A six-item interview protocol was developed by the researchers to follow the attitude survey.Interviews were conducted within1week of completing the instruction.Five participants from each treatment condition were asked both forced-response and open-ended questions related to their opinion of the program,the helpfulness of features of the program,and the perception of collaborative learning in each treatment condition. Group project performance

The?rst author developed a rubric to evaluate participant performance on the group project.The rubric constructs were based upon the essential skills required to achieve the instructional objectives of the program.To strengthen the validity of the instrument,three experienced instructional designers reviewed the three constructs and levels of perfor-mance in relation to alignment with the task and objectives.

As a result,students were evaluated on the inclusion of content,the accuracy of their calculations,and the format of their output.For inclusion of content,participants were evaluated on their ef?ciency in using the data provided.For accuracy of calculations, participants were evaluated on conducting the correct calculations and if all equations were accurate.For output,participants were evaluated on their clarity in formatting the spreadsheet.Each category of the rubric was divided into four levels of performance.The participants received0–2,3–5,6–7or8points for each category.For example,under calculations,participants received no points if more than two formulas were incorrect,four points if two formulas were incorrect,six points if one formula was incorrect,and8points if all four formulas were correct.The group project spreadsheets were blind scored by one of two evaluators to prevent bias.Inter-rater reliability for a random sample of15student projects was.90.

Student interactions

Two trained research assistants collected student interactions for each dyad during the study and categorized the interactions as questioning,answering,encouraging,discussing, or off-task(Sherman and Klein1995).A pilot test of the interaction observation procedure was conducted during two class periods prior to the study.In between the two observation periods,the assistants discussed discrepancies in their observations with the primary researcher to establish consistency in categorizations.The consensus estimate for the second set of observations was86%.

123

Computer-mediated instruction111 In the face-to-face condition,each instance of collaboration was indicated on an observation form.Raters identi?ed each dyad by number,but were unaware of the dyad’s ability composition to avoid any bias.Each dyad was systematically observed in2-min intervals throughout the program comprising a minimum of20min of observation for each dyad during the instruction.Interactions in the virtual condition were captured using the virtual classroom session log feature of Blackboard TM.The log?les for each dyad were exported to a database for analysis by the?rst author,and examined for the same col-laborative behaviors as the face-to-face dyads.

Because the chat transcripts captured all interactions in the virtual condition(which was not possible in the face-to-face condition)a procedure was devised to ensure an equivalent 20-min comparison sample of interactions between the face-to-face and virtual dyads.The interval between observations of the same dyad in the face-to-face condition was deter-mined by the number of dyads in each course section.Time stamps for each interaction in the virtual condition were used to reconstruct the class session on a timeline.Two-minute intervals were then systematically identi?ed based on the number of dyads in each section to simulate the observation periods that occurred in the face-to-face condition. Procedure

Dyads were given three,75-min class periods to complete the program and the assess-ments.Student interactions were collected for each dyad during the?rst2days of the treatment by two trained research assistants.The frequency of these interactions were classi?ed and recorded on an observation form.

On the?rst day of the treatment period,students in the face-to-face condition were verbally informed of their dyad assignments and asked to sit at a workstation next to their partner.Students were then asked to execute the installed CBI.Students in the virtual condition were informed they were participating in a simulation of a virtual environment, so all communication would take place using the synchronous chat feature of Black-board TM.Partners in the virtual condition were not seated in proximity of one another or aware of each others’identity until logging in to Blackboard TM.Participants in both conditions were instructed to run Microsoft Excel TM along with the CBI.

The second day of instruction was nearly identical to the?rst.Students in both con-ditions were asked to return to the workstation they used the previous class period and resume the lesson.Students in the virtual condition were again reminded they had to login to Blackboard TM to communicate with their partner.At the conclusion of the second class period,groups were required to submit their?nal projects.Students?nishing before the end of the period were excused.

The individual,computer-based,multiple-choice posttest covering the content of instructional material was administered to each student on the third day on instruction. Students also completed a Likert-type attitude survey,and?ve students from each treat-ment condition were randomly selected to participate in a short interview.

Data analysis

This study used a quasi-experimental,posttest-only control group design.It was a three (group composition:higher-ability dyads,lower-ability dyads,and heterogeneous dyads) by two(collaboration mode:face-to-face and virtual)factorial design.A3·2ANOVA

123

112J.I.Tutty,J.D.Klein was conducted to evaluate the effect of group composition and collaboration mode on individual posttest performance.Two separate2·2ANOVAs were also conducted on the posttest data to(a)compare lower-ability students in the homogeneous dyads to lower-ability students in the heterogeneous dyads and(b)compare higher-ability students in the homogeneous dyads to higher-ability students in the heterogeneous dyads.Group project performance data were also analyzed using a3·2ANOVA to determine the effect of group composition and collaboration mode.Three separate3·2multivariate analyses of variances(MANOVA)were performed on the items comprising each of the three attitude factors(delivery system.topic of the lesson,collaboration).Follow-up univariate and Tukey HSD analyses were used where appropriate.Alpha was set at.01for each follow-up test.Interaction frequency data were analyzed using chi-square analyses.Interview responses were categorized and reported by theme.

Results

Posttest performance

Means and standard deviations for individual posttest performance are reported in Table1. The mean posttest score was18.69(SD=2.70)for students in the face-to-face collabo-rative condition and17.57(SD=2.90)for students in the virtual collaborative condition. Table1also shows that the mean posttest score was16.70(SD=2.30)for students in homogeneous lower-ability dyads,18.40(SD=2.80)for students in heterogeneous-ability dyads,and19.39(SD=2.78)for students in homogeneous higher-ability dyads.

A3·2ANOVA revealed a signi?cant main effect for group composition [F(2,114)=10.79,P<.001,partial g2=.16]and collaboration mode[F(1,114)=6.43, P<.01,partial g2=.05].ANOVA did not reveal a signi?cant interaction between group composition and collaboration mode.

A follow-up2·2ANOVA conducted to compare the posttest scores of lower-ability students in the four treatment groups indicated a signi?cant disordinal interaction between Table1Means and standard deviations for individual posttest

Collaboration mode Group composition Total

LL HH H/L

Face-to-face

Mean17.0020.2519.0018.69 (SD)(2.64)(2.05)(2.34)(2.70) n22202062 Virtual

Mean16.3318.4417.8617.57 (SD)(1.82)(3.20)(3.12)(2.90) n18182258 Total

Mean16.7019.3918.4018.15 (SD)(2.30)(2.78)(2.80)(2.84) n403842120

123

Computer-mediated instruction113 group composition and collaboration mode,F(1,57)=3.90,P=.05,partial g2=.06. Lower-ability students who worked with other lower-ability students in the face-to-face condition(M=17.00,68%correct)performed about the same on the individual posttest as those in the virtual condition(M=16.33,65%correct).However,lower-ability students who worked with higher ability students scored better on the posttest when they collab-orated face-to-face(M=18.80,75%correct)rather than virtually(M=15.64,63%correct) (see Fig.2).

A second follow-up2·2ANOVA conducted to compare the posttest scores of higher-ability students in the four treatment groups also indicated a signi?cant disordinal inter-action between group composition and collaboration mode,F(1,55)=3.94,P=.05,partial g2=.07.This interaction revealed that higher-ability students who worked with other higher-ability students had better posttest scores when they worked in the face-to-face collaborative mode(M=20.25,81%correct)rather than the virtual mode(M=18.44,74% correct).In contrast,higher-ability students who worked with lower-ability students had better posttest scores when they collaborated virtually(M=20.09,80%correct)rather than face-to-face(M=19.20,77%correct)(see Fig.3).

Group project performance

Means and standard deviations for group project performance are reported for dyads in Table2.The mean project score for all dyads was21.77(SD=1.91)out of a possible score of24.The mean project score was21.23(SD=2.01)for dyads in the face-to-face collaborative condition and22.34(SD=1.59)for dyads in the virtual collaborative con-dition.Table2also shows that the mean project score was20.65(SD=1.79)for students in homogeneous lower-ability dyads,22.38(SD=1.66)for students in heterogeneous-ability dyads,and22.26(SD=1.85)for students in homogeneous higher-ability dyads.

A3·2ANOVA conducted to evaluate the effect of group composition and collabo-ration mode on group project performance indicated a signi?cant main effect for group composition,F(2,54)=5.85,P<.01,partial g2=.18.Tukey HSD pairwise comparisons conducted as a follow-up to the main effect revealed that group project scores for

123

114J.I.Tutty,J.D.Klein

Table2Means and standard deviations for group project performance

Collaboration mode Group composition All groups

LL HH H/L

Face-to-face

Mean19.6422.0022.2021.23 (SD)(1.36)(1.93)(1.96)(2.01)

n11101031 Virtual

Mean21.8922.5622.5522.34 (SD)(1.45)(1.74)(1.44)(1.59)

n991129

Total

Mean20.6522.2622.3821.77 (SD)(1.79)(1.85)(1.66)(1.91)

N20192160 homogeneous lower-ability dyads were signi?cantly lower(P<.01)than scores for participants in the heterogeneous-ability dyads and those in homogenous higher-ability dyads.There was no signi?cant difference between the other group compositions.

ANOVA also indicated a signi?cant main effect for collaboration mode,F(1,54)=5.93, P<.05,partial g2=.10.Participants in the virtual collaborative condition signi?cantly outperformed those in the face-to-face condition.ANOVA did not indicate a signi?cant interaction between group composition and collaboration mode.

Student attitudes

Means and standard deviations for the individual attitude survey are reported in Table3. Attitude scores are based on a5-point,Likert-type scale(4—strongly agree,0—strongly 123

Table3Means and standard deviations for attitude

Item Group composition Mode

LL HH H/L F2F V Total

Delivery system

1.The instruction in the

computer program was

clear

3.23(.62) 3.03(1.01) 2.97(.87) 3.21(.82) 2.95(.90) 3.08(.84)

2.I am better prepared to use

spreadsheets after

completing this program*

2.92(.88) 2.97(.85)

3.12(.57) 3.23(.81) 2.79(.70) 3.01(.76)

3.The computer program was

easy to navigate*

3.28(.72) 3.17(.85) 3.10(.85) 3.44(.69) 2.93(.84) 3.18(.79)

4.The computer program

included enough practice

2.46(1.07) 2.83(1.03) 2.41(1.21) 2.84(1.12) 2.28(1.05) 2.56(1.10)

5.I was able to adequately

communicate with my

partner*

2.72(1.03) 2.92(.97) 2.69(1.06)

3.18(1.00) 2.37(.88) 2.77(.99)

6.The checkpoints in the

computer program were

useful*

2.90(.83) 2.72(.88) 2.95(.79)

3.12(.82) 2.60(.78) 2.86(.82)

7.The computer program was

a good way to learn

spreadsheets*

3.03(.83) 2.89(1.01) 2.97(.87) 3.28(.82) 2.65(.88) 2.96(.88)

Topic

8.I would like to learn more

about spreadsheets

2.92(.86)

3.08(.65) 3.12(.57) 3.04(.79) 3.05(.61) 3.04(.70)

9.The spreadsheet skills I

leaned will help me in my

career

3.36(.63) 3.28(.70) 3.41(.55) 3.49(.61) 3.21(.62) 3.35(.62)

10.The spreadsheet is a

useful tool to know*

3.62(.50) 3.58(.60) 3.62(.54) 3.75(.44) 3.46(.60) 3.61(.54)

11.I can think of other ways

to use spreadsheets*

3.00(.83) 3.31(.71) 3.23(.58) 3.35(.70) 3.00(.71) 3.18(.71) Collaborative work

12.The checkpoints in the

program helped my partner

and I communicate*

2.54(.93) 2.56(1.05) 2.67(.84)

3.04(.90) 2.14(.77) 2.59(.90)

13.I learned the material

better working with a

partner than I would have

on my own*

2.28(1.26) 2.61(1.29) 1.90(1.37) 2.75(1.26) 1.75(1.21) 2.25(1.28)

14.My partner and I used the

information we learned

from the computer

program to do the practice

and group projects*

3.03(.88) 3.17(.81) 3.00(.86) 3.27(.89) 2.86(.77) 3.06(.84)

15.My partner taught me

what I needed to learn*

2.90(1.01) 2.94(.96) 2.52(1.10)

3.12(1.03) 2.44(.93) 2.78(1.01)

16.I preferred working with a partner to working alone during the spreadsheet lesson 2.08(1.22) 1.81(1.45) 1.64(1.18) 2.07(1.28) 1.61(1.26) 1.84(1.28)

Computer-mediated instruction115

123

disagree).These data indicated that most students felt the computer program was easy to navigate (M =3.18,SD =.79)and that they are better prepared to use spreadsheets after completing the program (M = 3.01,SD =.76).Most students thought the spreadsheet is a useful tool to know (M =3.61,SD =.54),and the spreadsheet skills learned will help in my career (M =3.35,SD =.62).Students also generally liked their assigned partner (M =3.25,SD =.81).However,students did not respond as positively to working with a partner over working alone (M =1.84,SD =1.28)and did not feel that they learned the material better working with a partner than they would have on their own (M =2.25,SD =1.28).

Three separate 3·2MANOVA were conducted to determine the effect of group composition and collaboration mode on each of the three attitude factors.These analyses revealed a signi?cant main effect for collaboration mode on the attitude factors of delivery system [Wilks’s K =.74,F (7,101)=5.17,P <.001],the topic of the lesson [Wilks’s K =.89,F (4,105)= 3.14,P <.05]and collaborative work [Wilks’s K =.60,F (7,101)=9.48,P <.001].These analyses did not show a signi?cant main effect for group composition or an interaction between group composition and collaboration mode for any of the attitude factors.

Follow-up univariate analyses indicated signi?cant differences (P <.01)between face-to-face and virtual collaboration modes on ?ve of the seven survey items related to delivery system,on two of the four items related to the topic of the lesson,and on ?ve of the seven items related to collaborative work.In all cases,students in the face-to-face collaboration mode responded more positively than those in the virtual collaboration mode.

Approximately 91%of the study participants who completed the Likert portion of the attitude survey also responded to three open-ended questions.When asked what they liked best about the program,24of 96respondents (25%)mentioned that the CBI was inter-active.For example,a respondent indicated that they liked the program because,‘‘It was interactive;we could move at our own pace.I could go back if I didn’t understand something.’’Further,22participants (23%)mentioned the helpful practice,16(17%)mentioned the step-by-step presentation,and 16(17%)mentioned the access to different tracks.When asked what they liked least about the program,30respondents (31%)indi-cated the inability to copy from the program into Excel,24(25%)mentioned using chat,and 16(17%)responded working with a partner.Finally,when asked about how to improve the program,24of 96respondents (25%)indicated that the program could be improved by not using partners,20students (21%)wanted more practice in the program,16(17%)mentioned not using chat,and 12(13%)mentioned increasing the discussion of functions in the program.

Table 3continued

Item Group composition

Mode LL

HH H/L F2F V Total 17.I like my assigned partner 3.21(.94)

3.28(.66) 3.26(.85) 3.40(.89) 3.09(.71) 3.25(.81)18.I liked the system for

communicating with my

partner*

2.59(1.15) 2.56(1.32) 2.31(1.45)

3.25(.82) 1.72(1.26) 2.48(1.20)Scores are based on a 5-point scale (4—strongly agree,0—strongly disagree)

*P <.01for mode 116

J.I.Tutty,J.D.Klein 123

Computer-mediated instruction117 Student interviews

Five participants from each treatment condition were interviewed to determine their opinions of the program(n=30).Participants were?rst asked about their opinion of the program.Fourteen of the15(93%)students in the face-to-face collaborative condition and 12of15(80%)students in the virtual condition indicated they liked the program.Many responses were similar to the following:

I felt that the computer program was a good way to learn about spreadsheets.It not

only taught the basics,but had us apply them at the same time.I liked it because it was informative,creative,and taught the basics of spreadsheets.

Other students said they liked the program because of the step-by-step presentation of information,or because students were required to teach what they had learned.

When asked about which parts of the program were the most helpful,9of15(60%) students in the face-to-face condition and8of15(53%)students in the virtual condition indicated the individual instruction component was the most helpful.Six students in the virtual condition said the practice project was the most helpful,while?ve students in the face-to-face condition mentioned the group project.When asked about which parts of the program were least helpful,seven students in the face-to-face condition and four students in the virtual condition indicated the inability to copy from the program into Excel was the least helpful component.Furthermore,four students in the virtual condition identi?ed the Internet chat requirement as the least helpful.

Responses to questions about working with a partner revealed that10of15(67%) students in both the face-to-face collaborative condition and the virtual collaborative condition liked working with a partner.One of the students in the face-to-face condition made the following representative statement:

I did prefer working with a partner;it was helpful to make sure I was doing things

correctly,and for getting new ideas on our second project.I think the collaboration made things better for both of us because we were able to bounce ideas off each other.

A student in the virtual condition said,‘‘It was fun to work with a partner in the spreadsheet lesson.I think it helped me better understand what I needed to know by having to explain it to someone else.’’Conversely,5of15(33%)students in both the face-to-face and the virtual collaborative conditions indicated they did not like working with a partner.Students responding less positively towards working with a partner found it dif?cult to,‘‘keep going back to explain what I just learned’’,and indicated that working with a partner,‘‘left a few gaps in my knowledge.’’

When asked about how the lesson was used to learn about spreadsheets,17of30(57%) respondents said they simply followed the instructions.Five of the15(33%)students in the virtual collaborative condition mentioned that they relied on learning from their partner, while only1of15(7%)students in the face-to-face condition mentioned learning from their partner.

Finally,students were asked about their opinion of using a similar program in the future. Twelve of15(75%)students in the face-to-face condition and11of15(73%)students in the virtual condition indicated they thought it would be a good idea to use a similar lesson in the future.Three students in each condition thought that it should depend on the subject. For example,one student said,‘‘I don’t think it would be useful for every application.It is

123

118J.I.Tutty,J.D.Klein really only good for basic procedural information.’’Only one student indicated that they thought it would be a bad idea to use a similar program in the future.

Student interactions

Interaction behaviors were grouped into the?ve categories of questioning,answering, encouraging,discussing,and off-task.Separate chi-square analyses were conducted to determine the effect of group composition and collaboration modes on the number of interactions for each category.No signi?cant differences were found within any of the interaction categories between ability groups.However,chi-square analyses indicated that students in the virtual collaborative condition asked more questions of their partners, (v2=(1,n=29)=5.13,P=<.05,and exhibited more off-task interactions v2=(1, n=29)=9.92,P=<.01,than those in the face-to-face collaborative condition.No other signi?cant differences were found for interaction behaviors.The observed instances of student interactions that occurred during the instructional program are reported in Table4. Discussion

Results indicated that participants in the face-to-face collaborative condition performed signi?cantly better on the individual posttest than those in the virtual online condition.This ?nding likely occurred because face-to-face students found it easier to share information throughout the lesson than virtual students.This explanation is partially supported by results from the attitude survey which revealed signi?cant differences in favor of students in the face-to-face condition for the following items:(1)I was able to adequately com-municate with my partner;(2)My partner taught me what I needed to learn;(3)The checkpoints in the program helped my partner and me communicate;and(4)I liked the system for communicating with my partner.

Table4Instances of student interactions

Interactions Group composition Mode

HH LL H/L F2F V

Questioning*120145161188238 Answering9098130145173 Encouraging5139586385 Discussing185132175258234 Off-task*36617164104 Total interactions482475595718834

Note:Total number of each interaction behavior for60dyads in20min of elapsed time observed in2-min intervals

LL=Homogeneous lower-ability dyads

H/L=Heterogeneous(mixed-ability)dyads

HH=Homogeneous higher-ability dyads

*P<.05for mode

123

Computer-mediated instruction119 While observing student interactions,it was noted that several face-to-face dyads used visual cues such as pointing to their screen to provide an explanation to their partner or to acknowledge understanding.These visual cues were not available to students in the virtual dyads.Hara(2002)indicated that students are required to make assumptions about meaning when they collaborate in virtual environments because of a lack of visual cues obvious in face-to-face communication.Others suggest that student misconceptions may occur due to a lack of nonverbal interventions to signal misunderstandings and that stu-dents can become disoriented without visual anchors in a virtual environment(Ruberg et al.1996;Sapp and Simon2005).

As participants were observed in the current study,it was also noted that a few students in face-to-face dyads traded seats with their partner so that both could work individually through each track,avoiding collaboration altogether.This strategy for taking individual control during the lesson was not available to students in virtual dyads.Slavin(1995) suggested that individual learning strategies are better than cooperative strategies when students are required to learn facts and procedures.The multiple-choice posttest in the current study required individual students to identify terms,spreadsheet functions,and the output of a given formula.

In contrast to?ndings for the individual posttest,dyads that collaborated virtually performed signi?cantly better on the group project than those who collaborated face-to-face.The collaborative interactions of students in the virtual dyads likely in?uenced their scores on the group project.Observations conducted during the study revealed that dyads in the virtual condition exhibited signi?cantly more questioning behaviors than dyads in the face-to-face condition.As the lesson progressed,the frequency of interaction increased for virtual dyads while it decreased for face-to-face dyads.Several students in the face-to-face condition were observed working independently on the group project.

Other researchers have found that student interactions in?uence learning and perfor-mance in collaborative settings.Hooper and Hanna?n(1991)demonstrated that ques-tioning contributes to learning in collaborative groups.King(1989)found small groups that asked task-related questions were more successful at problem solving than groups that did not exhibit such interaction behaviors.In a comparison of computer-mediated groups, Sherman and Klein(1995)reported that dyads exhibiting more helping behaviors such as asking and answering questions performed better than dyads exhibiting signi?cantly fewer helping behaviors.

The interaction requirements for an inquiry type project,such as the group project in this study,may be better met by virtual collaboration than face-to-face.Theorists have discussed advantages of virtual over face-to-face collaboration for group problem-solving tasks.Jonassen et al.(1999)asserted that CMC environments are better suited for problem-solving activities.The process of writing and re?ecting may encourage higher level learning such as analysis,synthesis,and evaluation,and promote clearer and more precise communication(Garrison1997;Jonassen and Kwon2001).A study by Uribe et al.(2003) found that computer-mediated groups experienced performance bene?ts from the medium when performing an ill-structured problem solving task.

In addition to the?ndings for collaboration mode,group composition had a signi?cant impact on individual posttest scores and group project performance in the current study.As expected,students assigned to higher-ability dyads performed signi?cantly better on both performance measures than students assigned to lower-ability dyads.Furthermore,mixed-ability dyads performed signi?cantly better on both performance measures than students assigned to lower-ability dyads.These results are not surprising.

123

120J.I.Tutty,J.D.Klein It is interesting that when data were analyzed separately for lower-ability and higher-ability students,signi?cant interactions were found.Results indicated that lower-ability students paired with higher-ability students scored better on the posttest when placed in the face-to-face condition(M=18.80,75%correct)but performed worse when placed in the virtual condition(M=15.64,63%correct).This?nding may be due to additional demands placed on students in the virtual,online environment.The synchronous chat feature used in the virtual condition likely increased cognitive load,especially for lower-ability students who had little to no prerequisite knowledge about spreadsheets.These increased demands were not present in the face-to-face condition.Thus,the bene?ts of forming mixed-ability collaborative groups may be greater for lower-ability students in face-to-face settings.

In contrast,higher-ability students paired with lower-ability students scored somewhat better on the posttest when placed in the virtual condition(M=20.09,80%correct)when compared to higher-ability students paired with lower-ability students in the face-to-face condition(M=19.20,77%correct).When asked to work collaboratively,high-ability students have a tendency to take control in group settings especially when their grade depends in part on the achievement of their partner.Yet the instructional program used in the current study was structured in such a way that it did not provide any chance for students in the virtual condition to view both tracks of information.It is likely that higher-ability students working with lower-ability students assumed a mentoring role in the virtual collaborative condition.This explanation is supported by the?nding that students in the virtual condition asked more questions of their partners than those in the face-to-face condition.

Turning to attitudes,participants were generally positive about the delivery system and the topic of the computer-based program used in this study.However,attitudes toward collaboration were less positive.Most of the Likert-type survey items asking about col-laborative work were rated lower than items about the delivery system and the topic. Furthermore,open-ended survey items revealed that many students disliked working with a partner and thought the program could be improved by eliminating collaboration.

These results likely occurred because the collaborative structure used in the study placed too many limitations on students’ability to interact naturally,especially for those in the virtual collaborative condition.Results indicated that students in the virtual condition were signi?cantly less positive than students in the face-to-face condition toward the delivery system,the topic,and collaborative work,with signi?cant differences occurring on12of the18Likert-type items.Open-ended items revealed that half of the respondents in the virtual condition identi?ed communicating via synchronous chat as the thing they liked least about the program.In addition,almost one-third of the virtual students who participated in follow-up interviews cited the chat system as the least helpful part of the program.

These?ndings are consistent with results from other studies.Uribe et al.(2003)found that participants did not like virtual collaboration due to the dif?culties of communicating via computer.Others have reported that students in virtual groups were less satis?ed than those in face-to-face groups with instruction received from their partner(Olaniran et al. 1996;Warkentin et al.1997).

As discussed above,virtual dyads exhibited signi?cantly more questioning behaviors than face-to-face dyads.In addition,virtual dyads exhibited signi?cantly more off-task interactions than face-to-face dyads.Many studies have detailed the importance of social interaction in computer-mediated communication(Chen2005;Jung et al.2002;Savenye 2005).Anderson and Harris(1997)identi?ed that socially oriented factors contribute to the prediction of performance in computer-mediated settings.

123