Comparing Online to Face-to-Face Delivery of Undergraduate Digital Circuits Content Authors: Brock J. LaMeres & Carolyn Plumb (c) 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works. The final version can be found at https://dx.doi.org/10.1109/TE.2013.2277031. LaMeres, Brock J. , and Carolyn Plumb. "Comparing Online to Face-to-Face Delivery of Undergraduate Digital Circuits Content." IEEE Transactions on Education 57, no. 2 (August 2013): 99-106. DOI: 10.1109/TE.2013.2277031. Made available through Montana State University’s ScholarWorks scholarworks.montana.edu http://scholarworks.montana.edu/ http://scholarworks.montana.edu/ https://dx.doi.org/10.1109/TE.2013.2277031 Comparing Online to Face-to-Face Delivery of Undergraduate Digital Circuits Content Brock J. LaMeres, Senior Member, IEEE, and Carolyn Plumb Abstract—This paper presents a comparison of online to tradi- tional face-to-face delivery of undergraduate digital systems mate- rial. Two specific components of digital content were compared and evaluated: a sophomore logic circuits course with no laboratory, and a microprocessor laboratory component of a junior-level com- puter systems course. For each of these, a baseline level of student understanding was evaluated when they were being taught using traditional, face-to-face delivery. The course and lab component were then converted to being fully online, and the level of student understanding was again measured. In both cases, the same pur- pose-developed assessment tools were used to carry out the mea- surement of understanding. This paper presents the details of how the course components were converted to online delivery, including a discussion of the technology used to accomplish remote access of the electronic test equipment used in the laboratory. A compar- ison is then presented between the control and the experimental groups, including a statistical analysis of whether the delivery ap- proach impacted student learning. Finally, student satisfaction is discussed, and instructor observations are given for the successful remote delivery of this type of class and laboratory. Index Terms—Digital circuits, education, e-learning, micropro- cessors. I. INTRODUCTION R EMOTE delivery of course content using Internet-based technology has been around for the past two decades. Recently, there have been great advances in the capability and number of delivery tools available to instructors. While the flexibility and cost advantages of remote instruction are attrac- tive, engineering programs have been resistant to broad-scale adoption of online instruction due to a variety of concerns. First, some engineering topics that require equation manipulation and iterative discussion can require significant instructional design effort to deliver online [1]. In addition, some engineering courses require specialized laboratory equipment that is only accessible on campus, which makes creating a fully remote learning experience difficult. Despite these concerns, the at- tractiveness of online delivery of engineering course material Manuscript received June 27, 2012; revised December 20, 2012 and July 16, 2013; accepted July 24, 2013. Date of publication August 16, 2013; date of current version May 01, 2014. This work was supported in part by the National Science Foundation under Grant 0836961. B. J. LaMeres is with the Electrical and Computer Engineering Department, Montana State University, Bozeman, MT 59717 USA (e-mail: lameres@ece. montana.edu). C. Plumb is with the College of Engineering at Montana State University, Bozeman, MT 59717 USA (e-mail: cplumb@coe.montana.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TE.2013.2277031 makes this topic of great interest to the educational research community. This paper presents a comparison of delivery approaches for undergraduate digital circuits material that includes methods for both the lecture and laboratory components. First, direct mea- sure assessment tools were developed and used to collect out- come data across a set of learning objectives. The tools were used to collect baseline outcome scores for students being taught with the traditional face-to-face delivery approach. The course and labmaterial was then converted to fully online and delivered to subsequent cohorts of students. The same assessment tools were used to collect outcome scores for the learning objectives. These scores represented the experiment set and were compared to the original baseline scores. This paper presents the results of this comparison including the assessment tool development, grading, and delivery approach for both face-to-face and on- line versions. This paper is of interest to instructors teaching undergraduate courses in digital systems wishing to understand the impact of converting a live course into a fully online ver- sion. This paper is also of interest to instructors teaching digital courses who wish to deliver the laboratory component fully on- line using a remote lab approach in order to teach measurement techniques. A description of the remote lab development and enabling technology is provided in detail. This paper is organized as follows. Section II gives an overview of the current body of work in the area of online delivery of engineering content and the motivation for the work presented in this paper. Section III gives an overview of the course and lab material that was studied in this paper. Section IV discusses the delivery approaches used. Section V presents the assessment tool development. Section VI presents the results of the comparison, and Section VII discusses the outcomes. Finally, Section VIII concludes the paper. II. MOTIVATION A. Online Delivery of Lecture Material There has been a considerable amount of research into the effectiveness of online delivery of engineering coursework over the past two decades. The authors of [1]–[3] present an analysis of the potential benefits of online engineering education that include access, diversity, collaboration, and lifelong learning. A report to Congress by the Web-based Education Commis- sion [4] highlighted the ability of Web-based education to center learning round the student, instead of the classroom, and to provide continuous, relevant training. This has the potential of providing a customized learning environment that improves learning and reduces withdrawal and failure rates. Web-based education also broadens participation of nontraditional stu- dents. A recent US Department of Education meta-analysis of 176 online learning studies drew several important conclusions relating to past evidence-based reports of online learning, including that nontraditional students actually performed better on average than those learning the same material through traditional face-to-face lectures [5]. The potential advantages of online learning are widely ac- cepted, but are also predicated on the notion that online learning needs to achieve the same level of quality as traditional live in- struction. The authors of [1] and [6] present metrics for eval- uating the quality of online education that include learning ef- fectiveness, student satisfaction, faculty satisfaction, access and cost effectiveness. While metrics such as access and cost can be quantitatively measured, metrics such as satisfaction and ef- fectiveness are more difficult to gauge and are most commonly studied through individual experiments. A 2010 report to the Department of Education described a variety of studies com- paring online to traditional delivery and found that the content matter of the course played an important role in the effectiveness of learning [7]. There have been individual studies comparing online to live delivery that cover specific courses. The authors of [8]–[10] compare the results of different delivery techniques for introductory courses in economics, psychology, and busi- ness. The authors of [11] and [12] compare the results of dif- ferent delivery techniques for engineering-specific courses in computer graphics and circuit analysis. While the authors of [13] and [14] attempt to form a general consensus of student perception of online delivery by averaging data from many dif- ferent types of courses, the majority of literature in this area concedes that not all topics are equal with respect to their ef- fective online delivery. Topics that require equation manipula- tion and iterative discussion, for example, are most difficult to deliver [1], despite advances in computer-based tools. An in- structor wishing to convert a course to online delivery should follow the set of accepted best practices, based on sound ped- agogy, as presented by the authors of [15]–[18]. However the instructor should also review a case study of online versus live delivery for the specific content matter being taught. B. Online Delivery of Laboratory Experiments Delivering the laboratory component of an engineering course online has been identified by a number of research groups [1], [3], [12], [19] as the largest barrier to widespread penetration of online courses across the curricula in higher education. The importance of a laboratory experience is often inherently accepted by instructors as a way to reinforce class- room concepts. However, a considerable amount of research has looked into why a laboratory experience is important and what students gain from it. The authors of [19] provide a comprehensive and historical account of the purpose of a lab experience in engineering education. The authors show that a lab gives students exposure to learning across multiple knowledge domains including cognitive (instrumentation, modeling, experimentation, data analysis, and design), psy- chomotor (manipulation of an apparatus), sensor awareness, and affective/cognitive (learning from failure, creativity, safety, communication, teamwork and ethics) [19]. The quality of a remote laboratory experience should meet the standards and criteria of a traditional hands-on experience if learning effectiveness is to be achieved [20]. A variety of mechanisms exist to provide a remote labora- tory experience in distance education. The author of [21] and [22] provides a survey of remote lab technologies. These in- clude virtual laboratories (e.g., model-based Web simulations), remote laboratories (e.g., real control of equipment and devices located elsewhere), and a hybrid of the two. Some engineering labs are amenable to portable kits that students can take with them to conduct the experiments completely outside of the uni- versity laboratory facility [23]. The authors of [24] conducted a literature review on the state of the art of virtual labs and con- cluded that while virtual labs provide students with the ability to run the experiments on their own computers at any time, they also require considerable development time and often en- counter integration issues. The authors of [25] and [26] present technologies for remote laboratories for electrical engineering topics that demonstrate custom systems for performing labora- tory experiments at a distance; both report an increase in student access while being able to teach real measurement techniques. For instructors who administer laboratories and wish to teach measurement techniques using industrial-grade test equipment, a remote laboratory approach is the optimal solution. C. Motivation for This Paper This paper presents a comparison between student learning of undergraduate digital circuits content via a traditional live de- livery approach and via a fully online approach. The comparison includes both the lecture and laboratory components of this con- tent. The online delivery of the lecture uses a widely available lecture capture tool and course management system. The on- line laboratory component is achieved through remote lab tech- nology, which enables the teaching of measurement techniques as in a traditional hands-on experience. The remote laboratory approach does not use any custom technology, thus reducing cost. As mentioned earlier, while much research has been done on a variety of online delivery methods, it is also crucial for instructors to study the impact of the delivery methods for the specific content matter they teach. This paper presents the first comprehensive analysis of the delivery methods for an under- graduate digital circuits course that includes both the lecture and laboratory component. This paper is of interest to any instructor teaching undergraduate digital circuits content considering the impact on student learning if the material is delivered online. III. DIGITAL SYSTEMS CONTENT STUDIED A. Introduction to Logic Circuits The lecture-only course studied in this work, EELE261—In- troduction to Logic Circuits, is a three-credit course required of all students pursuing four-year degrees in electrical (EE) and computer (CpE) engineering at Montana State Univer- sity (MSU), Bozeman, MT, USA. MSU-Bozeman is on the semester system. Each semester consists of 16 weeks of in- struction. A live-taught, three-credit course in this system corresponds to three 50-min lectures each week and an antic- ipated effort of 6 h per week outside of class for reading and homework assignments. This is a sophomore-level course, but as it only requires knowledge of college algebra, it is often TABLE I EELE261 COURSE FLOW, GRADING, AND OUTCOMES MEASURED (HW = HOMEWORK; DISC = DISCUSSION; QZ = QUIZ) taken by freshman students. This course covers the introductory material associated with classic digital logic design (75%) and then introduces hardware description languages (HDLs) as a launching point to the process of modern digital design (25%). The VHDL language is used for the modern digital design portion of this course due to its rigid syntax and strict data type requirements. This language is advantageous for maintaining a firm program structure in a lower-level course with high enrollment. Functional logic simulations were used to verify proper VHDL design. Logic synthesis is not covered in this course. The learning objectives measured in this course are the following: A.1) accomplish number system conversions; A.2) design, analyze, and minimize basic combination circuits; A.3) design and analyze basic sequential logic circuits; A.4) understand the concepts of hardware description languages; A.5) describe and simulate simple logic circuits in an HDL. The course is broken into eight learning modules. For each module, students are graded on weekly homework assignments, weekly discussion participation, and a module quiz. At the end of the semester, the students are given a comprehensive final exam. Table I shows the course flow, the graded components of the course, and the learning objectives associated with each module. Assessment data are collected using direct measure via the homework problems, module quizzes, and the final exam. Survey data were also collected on the group of online students completing the course. B. Introduction to Microprocessors Laboratory The laboratory component studied in this work was for the course EELE 371—Introduction to Microprocessor Systems. This is a four-credit course that is required of all students pur- suing four-year EE and CpE degrees at MSU-Bozeman. The course meets for three 50-min lectures each week and has a weekly 2-h laboratory component where students get hands-on experience programming a microcomputer using assembly lan- guage to illustrate the principles of a computer system. The lab- oratory component of the course is worth 30% of the students’ overall grade. The learning objectives measured for in this lab- oratory component are the following: TABLE II EELE371 LAB FLOW, GRADING, AND OUTCOMES MEASURED (DEMO = LAB DEMONSTRATION) B.1) describe the basic architecture of a stored-program computer; B.2) describe the addressing modes of a microprocessor; B.3) describe a typical I/O interface and understand its timing; B.4) analyze a timing diagram of the interaction between the microprocessor and memory; B.5) synthesize a timing diagram of a READ/WRITE cycle between the microprocessor and memory. The laboratory component consists of 11 weekly exercises. For each lab exercise, the students are graded on a pre-lab quiz, the demonstration of a functional program, a post-lab quiz, and a screenshot of a measurement taken with a logic analyzer. Outcome data were collected via the post-lab quizzes, program demonstration, and program structure. Table II shows the lab flow, the graded components of each lab, and the learning objectives associated with each lab. IV. DELIVERY APPROACH A. Lecture-Only Course The Desire2Learn course management system was used for the EELE261 lecture-only course. Desire2Learn is a widely accepted, Web-based content management system [27]. This system provides a variety of features suitable for online teaching, including discussion groups, quizzes, content listing, a calendar, and progress monitoring. This system contained all of the course files including the syllabus, tutorials, and lecture notes. The module quizzes and final exam were administered as multiple-choice auto-graded questions within the Desire2Learn system. The homework assignments were a combination of auto-graded multiple-choice questions and instructor-graded VHDL files that were uploaded to the system. Discussion topics were also handled by the Desire2Learn system with the grading performed manually by the instructor. For the traditional face-to-face version of this course, an in- structor gave three 50-min lectures each week, delivering ma- terial by using a white board in a traditional classroom setting. For the online version, the lecture was replaced with Camtasia screen capture videos [28]. Screen capture videos were chosen for their effectiveness in subject areas where students benefit from repeated viewing of complex content [30]. While screen capture videos suffer from the lack of spontaneous student in- teraction as in a live lecture, they are becoming widely accepted as the most common method for delivering lecture content on- line [30], which makes them suitable for this comparison. In these videos, the instructor records the desktop of his/her com- puter with voice audio. This allows the instructor to talk over his/her lecture notes in addition towalking through sample prob- lems in a similar fashion to a live taught course. Each 50-min lecture was replaced with one to three videos, each 10 to 20 min long. The homework, quiz, and final exam problems adminis- tered by theDesire2Learn systemwere identical in both delivery approaches for the course. Questions from students were an- swered via instructor e-mail. Instructor answers were posted to the Desire2LearnSystem so that all students had access to the instructor response, as they would in class. The primary vari- able altered in this experiment was replacing the live lecture with online videos. For the online version of the course, stu- dents were expected to login aminimumof three times per week. Students were given a set of weekly videos to watch, a set of reading assignments, a homework assignment, and a discussion topic to comment on. At the end of eachmodule, a final quiz was given. Students were able to view and work on their homework throughout the week, saving answers along the way. Quizzes given at the end of eachmodule imposed a 60-min time limit. B. Laboratory The laboratory exercises were performed on a FreeScale HCS12 development board using the CodeWarrior development environment [30]. This platform allows students to develop programs, compile, download to the hardware, and debug all within one software environment. The hardware platform contains a variety of basic I/O such as LEDs, buzzers, switches, and buttons so that student can interact with the program and observe its execution. Each lab station also contains a Tektronix TLA5210 logic analyzer that allows students to measure digital signals on the FreeScale platform across 32 channels [31]. For the hands-on version of the lab, students were physically located at the lab station containing the FreeScale hardware and the logic analyzer. The students developed their programs on a Windows XP workstation running the CodeWarrior software and downloaded their program to the hardware with a USB cable for verification. The logic analyzer was controlled via the Microsoft Windows XP Remote Desktop Connection (RDC). Since the logic analyzer is a Windows XP platform with ad- ditional measurement hardware, it allows remote control via RDC. The reason for running the logic analyzer using RDC for the live cohort was to mimic the use-model for the online co- hort. The primary difference for the live cohort was that they could physically see and touch the microprocessor hardware. For the online version of the lab, students conducted the ex- periments at a remotely located Windows-based workstation. The students used remote desktop connection to access the logic analyzer. FreeScale Codewarrior was installed on the logic ana- lyzer, and a USB cable was connected to the FreeScale platform. This setup allowed a student to log into the logic analyzer from any other workstation, develop programs, compile and down- load to the FreeScale hardware, and then take measurements using the logic analyzer application. This setup was very sim- ilar to that used by the live cohort, with the exception that the FreeScale platform was not physically in front of them. A we- bcam was used to view and hear the basic I/O on the FreeScale platform. The CodeWarrior environment also allowed the user to create buttons on their desktop, which facilitates interaction with the program. In this way, a student could use any remote workstation and complete the lab exercises without being phys- ically located in front of the hardware. A set of four remote sta- tions were created that students used to conduct the lab exercises online. Each station was configured with predetermined cable connections identical to those of the live lab setup. Students con- ducting the labs remotely used a common computer lab at a dedi- cated time in order to have access to a teaching assistant (via cell phone) for questions, just as in the traditional setup. V. ASSESSMENT TOOL DEVELOPMENT Avarietyofassessment toolswasdevelopedtocollectoutcome data on the learning objectives studied in this work. The assess- ment included measuring cognitive skills (what the students should know), psychomotor skills (manipulating the apparatus), and affective skills (what they should be able to do). Cognitive skillswere evaluated using a series of directmeasures in the form of post-lab quizzes. These included multiple-choice questions with traditional right/wrong solutions, weighted-answer mul- tiple-choice questions, and short-answer topics. Psychomotor skills were measured by having the students take measurements on the microprocessor using the logic analyzer and uploading a screenshot of their waveforms. Affective skills were evaluated using program demonstrations and post-lab grading of program structure. All tools were implemented in the Desire2Learn system.Themultiple-choicequestionswere auto-graded, and the short-answerquestionsweremanuallygraded. A. Multiple-Choice Questions (Traditional) A set of traditional multiple-choice questions was created for both the course and lab component studied. These questions had only one correct solution. The grading for these questions was done automatically by the Desire2Learn system with no partial credit given. A total of 182 traditional multiple-choice questions was developed to assess outcomes A.1–A.5 and B.1–B.5. B. Multiple-Choice Questions (Weighted) A different style of multiple-choice questions was created to improve the course grading resolution of the right/wrong ques- tions described above. In these questions, the choices provided to the student were of varying degrees of correctness. One an- swer was the best choice and yielded full credit for the problem. Other choices had a lesser degree of correctness and yielded only partial or no credit for the problem. By posing the problem in this way, the student was forced to read through all of the potential solutions and more fully internalize the scope of the question. The answers provided were randomized by the on- line system so that students could not share their solution easily. Table III gives a sample weighted multiple-choice question used to assess outcome B.1. A total of 22 weighted multiple-choice questions was developed to assess outcomes B.1–B.5. TABLE III SAMPLE WEIGHTED MULTIPLE-CHOICE QUESTION TABLE IV SAMPLE SHORT-ANSWER QUESTION AND GRADING RUBRIC C. Short Answer Short-answer questions were also developed in order to as- sess student understanding by allowing the students to articulate their own knowledge of the subject. A limit of 150 words was imposed to force the students to formulate a concise answer. A scoring rubric was developed for each question to facilitate con- sistent grading. Table IV gives an example of a short-answer question and the associated rubric used to assess outcome B.2. A total of 11 short answers was developed questions to assess outcomes B.1–B.5. Due to the time-consuming nature of grading short-answer questions, the only feasible way to collect significant amounts of data is to have teaching assistants help with the scoring. In order to achieve consistent scoring across multiple graders, a cali- bration session was held to address variability in grading. For TABLE V COMPARISON OF OUTCOME SCORES each short-answer question, the instructor and three graduate as- sistants (with sufficient application knowledge) were provided with all of the student responses for a particular question, with the student names removed. The instructor and the three grad- uate assistants then graded the first 10 answers independently. The group then compared their scores. If these differed, they group-discussed why they gave their score and then collectively agreed what the appropriate score should be. After the calibra- tion, the group then graded the rest of the answers; the vari- ability between graders in these post-calibration scores was less than 5%. VI. COMPARISON OF RESULTS (LIVE VERSUS ONLINE) The live version of the EELE261 lecture-only course was of- fered during the Fall semester of 2010 to 26 students (19 EE, three CpE, and four other); the online version was offered during the Spring semester of 2011 to 35 students (17 EE, seven CpE, and 11 other). The live version of the EELE371 laboratory was offered during the Fall semester of 2009 to 48 students (31 EE, 10 CpE, and seven other); the online version was offered during the Fall semester of 2010 to 48 students (28 EE, 14 CpE, and six other). The same assessment tools were used to collect data when comparing the live versus online groups. Table V shows the comparison of results for the outcomes measured in this study. Table VI gives a comparison of the overall course averages for various graded components of the class including homework, quizzes, exams, and final grades. In both tables, the statistical significance, or “p-value,” between the two cohorts is provided. This value indicates whether there is a statistically significant difference between the two groups. A p-value less than 0.05 is used to indicate a statistically significant difference between the cohorts’ performance. Statistical significance alone does not guarantee that the difference is substantive or important. Similar comparisons of face-to-face versus online delivery in courses outside of engineering have shown that large sample sizes can produce statistically significant results even though the magni- tude of the difference may be inconsequential [32], [33]. Con- versely, differences present in small samples (such as in this ex- periment) may not reach statistical significance, but still reflect TABLE VI COMPARISON OF HOMEWORK, EXAM, AND OVERALL COURSE GRADES large differences [32]. Average scores alone are random vari- ables, and differences can sometimes be attributed to random fluctuations. In order to investigate the magnitude of the ef- fect of the differences between the cohorts while taking small sample sizes into consideration, the effect size (ES) is also in- cluded in Tables V and VI. The effect size is an indicator of strength of the significance (or “practical significance”) and has been used in similar comparisons to provide further insight into the statistical meaning of cohort performance differences [33]. The effect size is found by dividing the mean difference by the pooled standard deviation for both groups. An ES of is con- sidered small, moderate, and large. In this analysis, the face-to-face cohort is considered the control group, so a pos- itive ES signifies the experiment (e.g., online cohort) performed worse than the control. VII. DISCUSSION OF RESULTS A. Learning Effectiveness The cognitive skill development assessed using the direct measures indicated there was not a statistically significant dif- ference between live and online delivery except in the cases of outcome B.2 (Table V) and the homework portion of the lec- ture course (Table VI). For outcome B.2, the p-value indicated no statistically significant difference. However, the effect size indicated that the performance of the online cohort was below that of the live cohort. This may be due to the graphical nature of describing the theory behind this outcome. The live cohort may have benefited from the lab assistant’s real-time answering of questions using a white board, whereas the online cohort was not exposed to this type of graphical explanation. Another pos- sible explanation for the difference may have been the timing of themeasurements. Outcomes B.1 and B.2 represent overarching learning objectives that are reinforced continually throughout the course. B.1 had a p-value and ES value of 0.46, which is bordering on statistical significance. For both outcomes B.1 and B.2, measuring them early in the course may have not been suf- ficient to measure the students’ ultimate understanding of these outcomes. Since the online cohort performed as well or better in their graded components of the course, it is plausible that their knowledge of the B.1 and B.2 outcomes was better than reported by the outcome measures. A better approach to measuring these overarching outcomes would be to also measure them at the end of the course, when students have more laboratory experience with the entire computer system. The only graded measure that indicated a significant statis- tical difference was the homework portion of the lecture course. The p-value indicated a statistically significant difference, with the effect size showing that the online cohort performed better. A possible explanation for this was that students in the online cohort had the ability to watch the lecture videos again and again in order to grasp the concepts. Since teaching content in the area of undergraduate digital systems does not rely on in-depth mathematical derivation and equation manipulation to teach the material, it appears from this study that a recorded video is an acceptable delivery method. The student’s ability to watch the videos repeatedly at a time of his or her choosing has the poten- tial to maintain, or even improve, understanding compared to a traditional delivery method. There was also no significant difference in the psychomotor skill development between the two cohorts as measured by the waveform screenshots turned in by the students (outcomes B.4 and B.5). Despite the online cohort not being physically in front of the measurement equipment and the FreeScale platform, they were still able to set up the logic analyzer, interact with the mi- croprocessor, and take measurements on the system. Due to the setup of the laboratory system, both the live and online cohort had the same essential interface (e.g., everything was accom- plished using a single monitor). This could have contributed to the same level of understanding being achieved. This type of measurement setup is also somewhat unique to digital sys- tems. A logic analyzer is used to take measurements across a large number of digital signals. As such, the probing connec- tion is typically set up at the beginning of the experiment and not moved. This is a different use-model to that of an analog circuits lab that requires the students to move an oscilloscope around the circuit in order to take measurements at a variety of access points. The use-model of this type of measurement makes delivering the lab component of a digital circuits course more amenable to the remote laboratory approach that might be the case for other engineering courses. Affective skill development was also similar for the two co- horts as measured by laboratory demonstrations and program structure grading. Again, this is most likely due to the similar development environments experienced by the online and live groups. Since program development and debugging is accom- plished using the CodeWarrior application that runs on a work- station, there is no noticeable difference whether the student is running CodeWarrior on the workstation directly in front of them or whether it is running on a remote workstation with the display being exported back to a client workstation. Other variables that influence group-to-group performance include demographics (age, gender, ethnicity), course self-se- lection, and student preparedness. In this study, demographic in- formation was not collected due to student confidentiality poli- cies at MSU. Course self-selection has been shown to heavily influence the performance of cohorts in similar face-to versus online delivery comparisons. The authors of [34] showed in a study comparing face-to-face versus online delivery of a fire science course that while there was no difference between the Fig. 1. Incoming grade point average versus course grade. performance of two cohorts that were randomly assigned to ei- ther the online or face-to-face sections of the course, there was a significant improvement in performance of a third cohort who chose to take the course online. In this study, the self-selection was not a factor since the courses evaluated were offered either fully online or fully face-to-face and had to be taken at specific points during the curriculum. This means students were not able to choose whether they were in a face-to-face or online version of the course, so self-selection did not influence the findings. Student preparedness can be gauged by looking at whether students have met the necessary prerequisites for the course and the impact of the students’ incoming grade point average (GPA). All students in this study had met the prerequisites, so student preparedness in this regard was not a factor. Student GPA was studied between the two cohorts to see if there was a correlation between the students’ incoming GPA and their performance in the course. This in turn was used to see if students’ incoming GPA had an impact on whether they did better in an online environment. Fig. 1 shows the relationship between the students’ incoming GPA and their performance in the courses across both cohorts. As suspected, students with higher incoming GPAs tended to receive higher course grades than students with lower incoming GPAs. A linear regression is shown for both cohorts confirming that this trend is present in both online and face-to-face delivery. The linear regression for the face-to-face cohort ( , ) and the online cohort ( , ) were compared using a Chow test with an F-distribution to see if this correlation was statistically different across different delivery styles. This test found that the variable estimates from the two separate regression tests were not statistically different from a regression test on the combined samples. This implies that the delivery style did not have an impact when considering incoming GPA as a covariant. This matches a similar study that compared the impact of replacing live lectures with lecture capture in an introductory computer science course [35]. The results in this study found a similar pattern as [35], but in an undergraduate digital circuits course with a laboratory component. B. Student Satisfaction An anonymous survey of the online cohort was given near the end of the course, asking whether they preferred the online delivery approach or the traditional live-lecture approach they were receiving in their other courses. There were 60% of stu- dents who responded that they preferred the online delivery ap- proach, and 40% responded they preferred the live approach. The poll allowed the students to provide rationale for their re- sponses, which was done by 35% of the students. Of the students who preferred the online delivery approach, the most common reason given for their preference was schedule flexibility. The second most common reason was that they were able to watch the videos over and over while doing the homework assign- ment. This reinforces the findings in Table VI that the online cohort performed better on homework assignments compared to the live cohort. Of the students responding that they preferred live delivery, the main reason given for their preference was the ability to ask questions immediately when learning the material. C. Faculty Satisfaction The overall satisfaction of the instructor was primarily based on how the students perform. Since both cohorts of students were able to demonstrate understanding of the learning objec- tives across multiple knowledge domains, the instructor was satisfied with the learning effectiveness with the online delivery approach. One of the downsides of the online delivery approach was the significant course development time. Each lecture video was estimated to take two to three times longer to create than giving a traditional 50-min white board lecture. Additionally, developing the auto-graded questions was estimated to take two to three times longer than writing up custom homework problems or selecting questions from a textbook. While the additional development time appears from an administrative perspective to be a one-time cost, another instructor teaching a subsequent offering of the online course reported that ongoing effort was required to maintain the course. This effort, which consisted of altering questions, creating new questions, and responding to student e-mail, took approximately the same amount of time as teaching the course in a live delivery mode. It is anticipated that if a large pool of questions were developed and randomly selected for each assignment/quiz/exam, the instructor effort would go down. VIII. CONCLUSION In this paper, a comparison of traditional face-to-face delivery of undergraduate digital systems content compared to online de- livery was presented. Both a lecture-only course in digital logic and the laboratory component of a microprocessors course were studied. Assessment tools were developed to gauge the level of understanding across a set of learning objectives that covered cognitive, psychomotor, and affective skills. The same assess- ment tools were used to collect data from the groups being com- pared. In general, the data showed there was no noticeable dif- ference between the two delivery approaches. This case study indicates that undergraduate digital circuits content is a candi- date for effective delivery online. This study also indicates that for an undergraduate digital systems laboratory, a remote lab approach is as effective in meeting laboratory objectives as a traditional hands-on approach while still preserving the ability to teach measurement techniques. REFERENCES [1] J. Bourne, D. Harris, and F. Mayadas, “Online engineering education: Learning anywhere, anytime,” J. Eng. Educ., vol. 94, pp. 131–146, Jan. 2005. [2] R. E. Gomery, “Internet learning: Is it real and what does it mean for universities?,” J. Asynchronous Learning Netw., vol. 5, no. 1, pp. 139–146, 2001. [3] T. K. Grose, “Can distance education be unlocked?,” Pridm, vol. 12, no. 8, Apr. 19–23, 2003. [4] “The power of the Internet for learning—Moving from promise to prac- tice,” Report of theWeb-Based Education Commission to the President and the Congress of the United States, Dec. 2000. [5] B. Means et al., “Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies,” Center for Technology in Learning, Office of Planning, Evaluation, and Policy Development, US Dept. of Education, Washington, DC, USA, Sep. 2010. [6] A. F. Mayadas, “Testimony to the Kerrey commission on Web-based education,” J. Asynchronous Learning Netw., vol. 5, no. 1, pp. 134–138, 2001. [7] A. Jaggars and T. Bailey, “Effectiveness of fully online courses for college students: Response to a Department of Education meta-anal- ysis,” Community College Research Center, Teachers College, Co- lumbia University, New York, NY, USA, 2010. [8] D. Figlio, M. Rush, and L. Yin, “Is it live or is it Internet? Experimental estimates of the effects of online instruction on student learning,” Na- tional Bureau of Economic Research, Cambridge, MA, USA, Working Paper 16089, Jun. 2010. [9] W. S. Maki and R. H. Maki, “Learning without lectures: A case study,” Computer, vol. 30, no. 5, pp. 107–111, May 1997. [10] C. Scherrer, R. Butler, and S. Burns, “Student perceptions of on-line education,” Adv. Eng. Educ., vol. 2, no. 2, pp. 1–23, 2010. [11] M. Holdhusen, “A comparison of engineering graphics courses deliv- ered face to face, on line, via synchronous distance education, and in hybrid formats,” in Proc. Annu. Conf. ASEE, Jun. 2009. [12] A. Enriquez, “Assessing the effectiveness of dual delivery mode in an online introductory circuits analysis course,” in Proc. Annu. Conf. ASEE, Jun. 2010. [13] J. Carpinelli, R. Calluori, V. Briller, E. Deess, and K. Joshi, “Fac- tors affecting student performance and satisfaction in distance learning courses,” in Proc. Annu. Conf. ASEE, Jun. 2006. [14] D. R. Wallace and P. Mutooni, “A comparative evaluation of world wide Web-based and classroom teaching,” J. Eng. Educ., vol. 86, pp. 211–220, Jul. 1997. [15] K. S. Cheung, J. Lam, T. Im, R. Szeto, and J. Yau, “Exploring a ped- agogy-driven approach to E-courses development,” in Proc. ETT & GRS, Dec. 21–22, 2008, vol. 1, pp. 22–25. [16] “ADEC Guiding Principles for Distance Learning,” American Dis- tance Education Consortium, 2002. [17] SREB, Atlanta, GA, USA, “Southern Regional Education Board’s (SREB) electronic campus principles of good practice checklist,” 2002. [18] Middle States Commission on Higher Education, Philadelphia, PA, USA, “Distance learning programs: Interregional guidelines for elec- tronically offered degree and certificate programs,” 2002 [Online]. Available: http://www.msche.org [19] L. D. Feisel and A. J. Rosa, “The role of the laboratory in undergraduate engineering education,” J. Eng. Educ., vol. 94, no. 1, pp. 121–130, 2001. [20] L. Feisel and G. D. Peterson, “A colloquy of learning objectives for en- gineering educational laboratories,” in Proc. ASEE Annu. Conf. Expo., Montreal, QC, Canada, Jun. 16–19, 2002. [21] M. Cooper and J. M. M. Ferreira, “Remote laboratories extending access to science and engineering curricular,” IEEE Trans. Learning Technol., vol. 2, no. 4, pp. 342–353, Oct.–Dec. 2009. [22] M. E. Auer and C. Gravier, “Guest editorial: The many facets of remote laboratories in online engineering education,” IEEE Trans. Learning Technol., vol. 2, no. 4, pp. 260–262, Oct.–Dec. 2009. [23] J. Ventura, R. Drake, and J. McGrory, “NI ELVIS has entered the lab [educational laboratory virtual instrumentation suite],” in Proc. IEEE SoutheastCon, Apr. 8–10, 2005, pp. 670–679. [24] E. Sancristobal, S. Martin, R. Gil, P. Orduna, M. Tawfik, A. Pesquera, G. Diaz, A. Colmenar, J. Garcia-Zubia, and M. Castro, “State of art, initiatives and new challenges for virtual and remote labs,” in Proc. 12th IEEE ICALT, Jul. 4–6, 2012, pp. 714–715. [25] N. Lewis, M. Billaud, D. Geoffroy, P. Cazenave, and T. Zimmer, “A distance measurement platform dedicated to electrical engineering,” IEEE Trans. Learning Technol., vol. 2, no. 4, pp. 312–319, Oct.–Dec. 2009. [26] B. Pradarelli, L. Latorre, M. L. Flottes, Y. Bertrand, and P. Nouet, “Re- mote labs for industrial IC testing,” IEEE Trans. Learning Technol., vol. 2, no. 4, pp. 304–311, Oct.–Dec. 2009. [27] Desire2Learn, Inc., Kitchener, ON, Canada, “Desire2Learn learning management system,” [Online]. Available: http://www.desire2learn.com/ [28] TechSmith, Okemos, MI, USA, “Camtasia screen recording and video editing software,” [Online]. Available: http://www.techsmith.com/ camtasia.html [29] EDUCAUSE, “7 things you should know about lecture capture,” 2008 [Online]. Available: http://www.educause.edu [30] Freescale Semiconductor, Inc., Austin, TX, USA, “Getting started with CodeWarrior development tools,” [Online]. Available: http://www.freescale.com/mcu [31] Tektronix, Inc., Beaverton, OR, USA, “TLA5000B logic analyzer se- ries,” [Online]. Available: http://www.tek.com/logic-analyzer/tla5000 [32] M. Stassen, M. Blaustein, R. Rogers, andM. Shih, “Undergraduate stu- dent outcomes: A comparison of online and face-to-face courses,” Re- port from the Comparative Outcomes Subcommittee for the University of Massachusetts Amherst Faculty Senate Ad Hoc Committee on On- line Learning, Jul. 2007 [Online]. Available: http://www.umass.edu/ oapa/oapa/publications/misc_docs/undergrad-student-outcomes.pdf [33] R. N. Gerlich and M. Sollosy, “Comparing outcomes between a tradi- tional F2F course and a blended ITV course,” J. Case Studies Educ., vol. 1, pp. 1–9, Jan. 2011. [34] J. Collins and E. T. Pascarella, “Learning on campus and learning at a distance: A randomized instructional experiment,” Res. Higher Educ., vol. 44, no. 3, pp. 315–326, Jun. 2003. [35] M. D. Stroup, M. M. Pickard, and K. E. Kahler, “Testing the effective- ness of lecture capture technology using prior GPA as a performance indicator,” Teacher-Scholar, J. State Comprehensive Univ., vol. 4, pt. 1, Fall, 2012. Brock J. LaMeres (M’98–SM’09) received the B.S. degree fromMontana State University (MSU), Bozeman, MT, USA, in 1998, the M.S. degree from the Uni- versity of Colorado, Colorado Springs, CO, USA, in 2001, and the Ph.D. degree from the University of Colorado, Boulder, CO, USA, in 2005, all in electrical engineering. He is currently an Associate Professor with the Department of Electrical and Computer Engineering, MSU-Bozeman, where he conducts research on recon- figurable computing and e-Learning environments. Prior to joining the faculty at MSU-Bozeman in 2006, he worked as a Hardware Design Engineer with Ag- ilent Technologies, Colorado Springs, CO, USA, from 1999 to 2006. He has published over 60 papers and one textbook in the area of digital systems and engineering education and has been granted 13 US patents in the area of digital signal propagation. Dr. LaMeres is a member of the IEEE Aerospace and Electronic Systems Society and the ASEE and is the faculty advisor of the IEEE Student Branch at MSU-Bozeman. He is a Registered Professional Engineer in the states of Montana and Colorado. He has received two Best Paper Awards in international conferences on digital systems. Carolyn Plumb received the B.A. degree in English from Colorado State University, Ft. Collins, CO, USA, in 1972, the M.A. degree in English from Marquette University, Milwaukee, WI, USA, in 1974, and the M.S. degree in scientific and technical communications and Ph.D. degree in educational psychology from the University of Washington, Seattle, WA, USA, in 1988 and 1991, respectively. She is the Director of Educational Innovation and Strategic Projects with the College of Engineering, Montana State University, Bozeman, MT, USA. She works on various curriculum projects including instructional development for faculty and graduate students. She also serves as the college’s assessment and evaluation expert, currently evaluating the success of various programs and projects, including projects funded by the NSF and the Montana Department of Transportation. She also serves as the external evaluator for an NIH-funded project at the University of Washington.