Determining the Effectiveness of Various Delivery Methods in an Information Technology / Information Systems Curriculum Gary Alan Davis, davis@rmu.edu Paul Kovacs, kovacs@rmu.edu John Scarpino, scarpino@rmu.edu John Turchek, turchek@rmu.edu Computer and Information Systems Department Robert Morris University Moon Township, Pennsylvania 15108-1189 USA Abstract The emergence of increasingly sophisticated communication technologies and the media-rich extensions of the World Wide Web have prompted universities to use alternatives to the traditional classroom teaching and learning methods. This demand for alternative delivery methods has led to the development of a wide range of eLearning techniques. Nonetheless, skepticism towards delivery methods as a "means of communication" is still common. This conflict has influenced research on the subject. Many studies have examined the effectiveness of eLearning (also called “technology-enabled learning” and “online learning”) but very few of the results can be generalized to Information Technology/Information Systems curricula. The current study is an extension of a previous study conducted by the authors that examined the perceived effectiveness of online learning courses in a Computer and Information Systems curricula. The current study probes deeper into various online learning formats and into the students who enroll in the courses. Specifically, this new study looks at which non-traditional method of course content delivery (i.e., online, partially online, on-ground with online supplements) is most effective for Information Technology/Information Systems courses, as well as which instructional method of course content delivery provides the best learning for each grouping of Information Technology/Information Systems subject areas. Finally, this study attempts to profile groups of students who prefer an online format to a traditional on-ground format Keywords: Online Learning, eLearning, Web-Based Learning, Technology-Enabled Learning, IT/IS curricula 1. INTRODUCTION The emergence of increasingly sophisticated communication technologies and the media-rich extensions of the Internet have prompted universities to use alternatives to the traditional classroom teaching and learning methods. This demand for alternative delivery methods has allowed new developments in the way instructors transfer course content to their students. These new developments have resulted in the growth of a new paradigm in pedagogy: technology-enabled learning environments. Numerous terms have been used to describe this phenomenon, including Computer-Based Learning, Web-Based Learning, Technology-Supported Learning, eLearning, Distance Learning, and Online Learning. For purposes of this research, the term online learning will be used to describe any higher educational course offering that uses technology (i.e., software, electronic mail, and/or the Internet) to deliver all or part of course content. The proliferation of online learning within higher education has stimulated the ongoing debate as to which delivery method is more effective: online learning or traditional classroom learning (Noble, 2002, O'Malley, 1999). The present research looks at which non-traditional method of course content delivery (i.e., online, partially online, on-ground with online supplements) is most effective for Information Technology/ Information Systems (IT/IS) courses as well as which instructional method of course content delivery provides the best learning for each grouping of IT/IS subject areas. Also, this study attempts to profile groups of students who prefer an online format to a traditional on-ground format. 2. BACKGROUND Online learning has been cited as having many distinct advantages over traditional classroom learning. Those advantages include 1) reduced time to proficiency, 2) lower logistical costs, 3) added convenience, 4) personalized curricula, and 5) improved measurability of results. Business organizations that utilize online learning often assert a reduction in training expenses, an increase in productivity, and a decrease in employee attrition (Rich, 2001). Business organizations use online learning in a variety of ways. For example, Siemens uses online learning to deliver a performance simulation course to 10,000 employees world-wide (Rich, 2001). IBM employs various online learning methods in its world-wide training program involving 300,000 employees (Johnson, 2004). Ford Motor Company frequently uses online learning as an easy way to brief its salespeople and mechanics on the latest automotive products (Sessa, 2001). Few organizations (academic or industry-related), however, have taken the time and effort to assess the perceived effectiveness of online learning, as reported by learning participants. Variations of Online Learning As popular as online learning has become in both industry and academia, one prevailing question remains: what is the most effective utilization of online learning? (i.e., content delivered completely online, a hybrid model of content that is partially delivered online and partially delivered in the classroom, or on-ground delivery with an online supplement). Industry has typically favored the hybrid approach, opting for a combination of online learning and “on-ground” instruction. Sometimes labeled a “blended approach,” the combination of classroom education and online learning has been used by industry to maintain educational quality and foster “ . . . greater experiences in interpersonal relationships” (Trierweiler & Rivera, 2005, p. 3). Academia has also taken comfort in the hybrid approach to online learning. In their study involving undergraduate management courses, Drennan, Kennedy, and Pisarski found that research subjects “ . . . considered it important to find a balance between the use of emerging technologies and traditional face-to-face lectures” (2005, p. 332). In a subsequent study involving undergraduate accounting students, researchers found that a hybrid or blended approach to online learning was superior to course content delivered entirely online. Specifically, the researchers determined that “The use of technology as an exclusive course delivery method was considered insufficient. Face-to-face tutorials and group learning were central to both student confidence and the learning process” (Flynn, Concannon, & Bheachain, 2005, p. 433). 3. RESEARCH QUESTIONS Information Technology/ Information Systems (IT/IS) curricula can present unique challenges to the delivery of course content in an online format that are not found in traditional business and management programs. Many courses in an IT/IS curricula involve instruction in computer programming languages, requiring hands-on development and extensive drill and practice while other courses entail theoretical concepts; both of which can require an increased interaction with IT/IS faculty. It is not yet clear if online learning methods are conducive to the delivery of such course content. Furthermore, it is not clear as to what degree (if any) of online learning is effective in delivering IT/IS-specific course content. Although research has been conducted with business and management students in higher education to assess the perceived effectiveness of online learning, additional research needs to be conducted with IT/IS students. The present research examined the effectiveness of online learning, as reported by undergraduate, graduate, and post-graduate IT/IS students attending a private, medium-sized university to determine if online learning is an effective means of delivery for IT/IS-related content. Additionally, the study sought to determine which delivery method (i.e., classroom, partially online, or completely online, or on-ground with online supplement) is perceived to be more effective. The study also attempted to determine which delivery method is most appropriate for specific IT/IS-related subject areas. The IT/IS-related subject areas examined in this research included: 1) Software Development, 2) Network Administration and Security, 3) Web Development / Web Programming, 4) Multimedia / Graphics, 5) Office / Productivity Software, 6) Project Management, 7) Systems Analysis and Design, 8) Certification Courses, 9) Operating Systems, 10) Database, and 11) Business Intelligence. In order to develop the 11 subject areas, existing classifications from ABET (Accreditation Board of Engineering and Technology), ACM (Association for Computing Machinery), and AIS (Association for Information Systems) job categories were used as guidelines. Finally, the study attempted to determine the typical demographic profile of students who prefer an online format to a traditional on-ground format. Specifically, the study was conducted to answer the following research questions: 1. Which non-tradition instructional method of course content delivery (i.e., online, partially online, on-ground with online supplements) is most effective for IT/IS courses? 2. For each grouping of IT/IS subject areas presented to students, which instructional method of course content delivery do students perceive as providing the best learning? 3. What is the typical demographic profile of students who prefer an online format to a traditional on-ground format? 4. METHODS AND PROCEDURES The research involved the administration of a survey instrument that consisted of 22 closed-ended questions. The results from all 22 questions were not used in this study. Question 1 of the survey asked the participants if (given a choice for the same course) they would prefer an online format over an on-ground format. Those responding “Yes” to Question 1 were asked about the reasons for choosing an online course. Question 3 asked the participants if they had taken an online course (partially or completely online) in the past. Those respondents who answered “Yes” to Question 3 were then asked why they chose online and what training they received before enrolling in the course. The results from the above questions were not used in this study but may be used in future research. If the respondent answered “No” to Question 3, they were then branched to Question 10. Questions 10, 11, and 12 of the survey instrument asked participants to select the best delivery method for each of the eleven IT/IS subject areas. Valid responses for each subject area included “On-ground Instruction,” “Completely Online,” “Hybrid/Partially Online”, and “On-ground with Online Supplement.” Question 13 of the survey instrument asked participants to report whether the University should “Offer More Courses,” “Keep Course Offerings the Same,” or “Offer Less Courses” for each of the four delivery methods. The results from Question 13 were not included in this research study. Finally, Questions 14 through 22 solicited demographic information from the participants, such as age range, gender, and degree program, employment status. The population for this survey consisted of 498 computing majors enrolled in the Spring 2009 semester. Of these 498 students, 248 were at the undergraduate levels and 250 were at the graduate and post-graduate level. The undergraduate majors included a Bachelor of Science degree in Computer and Information Systems and a Bachelor of Science degree in Information Sciences; both of which are accredited by the Accreditation Board of Engineering Technology - Computing Accreditation Commission (ABET-CAC). At the graduate and post-graduate levels, a total of six Master's degrees and one doctoral degree are represented in the data population. Additionally, only 39 students (38 undergraduate and 1 graduate) were resident students. A total of 155 students responded to the survey. The students completed the online survey on their own time and submitted their anonymous results directly into an electronic database for analysis. The survey results were analyzed using SPSS statistical software. Statistical frequencies were used to answer the research questions posed in section 3. 5. RESULTS In order to answer the first research question, (i.e., Which non-traditional instructional method of course content delivery is most effective for IT/IS courses?) the survey instrument asked participants to select the “most effective” non-traditional delivery method for IT/IS-related courses. The available delivery methods were: Completely Online, Partially Online, or On-Ground with Online Supplement. The results are summarized in APPENDIX A, Table 1. Of the three non-traditional delivery methods, 75.5% of the research participants selected Onground with Online Supplement as Very effective or Effective as compared to 36.8% for Completely Online and 59.2% for the hybrid/ partially online formats. Furthermore, no respondents indicated Very Ineffective for either On-ground with Online Supplement or Hybrid/Partially Online. The difference in effectiveness among the delivery methods is further illustrated by a comparison of mean scores. The responses were designed using a Likert-like scale, with valid responses as “Very Effective” (value = 6), “Effective” (value = 5), Somewhat Effective” (value = 4), “Somewhat Ineffective” (value = 3), Ineffective” (value = 2), and “Very Ineffective” (value = 1).The mean score for Completely Online (? = 4.07) indicates that the respondents, in general, considered this delivery method to be “Somewhat effective.” The mean score for Hybrid /Partially Online (? = 4.65) indicates that the respondents, in general, considered the hybrid delivery method to fall between “Somewhat effective” and “Effective” (leaning more toward “Effective”). Finally, the mean score for On-ground with Online Supplement (? = 5.04) indicates that the respondents considered this delivery method to be “Effective.” In order to determine any statistical significance among the three instructional methods, a series of paired-samples T-Tests was conducted (at the 95% confidence level). The first paired-samples T-Test compared Completely Online to Hybrid/Partially Online. The results indicated that the difference between Completely Online and Hybrid/Partially Online was statistically significant (p = .000). The second paired-samples T-Test compared Hybrid/Partially Online to On-ground with Online Supplement. The results from the second T-Test indicated that the difference between Hybrid/Partially Online and On-ground with Online Supplement was statistically significant (p = .001). Finally, the third paired-samples T-Test compared Completely Online to On-ground with Online Supplement. The results from this third T-Test indicated that this difference was statistically significant (p = .000). The results of all three paired-samples T-Tests are summarized in APPENDIX B, Table 2. In order to answer the second research question (i.e., For each grouping of IT/IS subject areas presented to students, which instructional method of course content delivery do students perceive as providing the best learning?) the survey instrument asked participants to select the instructional method of course content delivery that provides the “best learning” for each grouping of IT/IS-related topics. The available delivery methods were: On-ground, Completely Online, Partially Online, and On-Ground with Online Supplement. The results from these survey questions are summarized in APPENDIX C, Table 3. More research participants selected Onground or Onground with Online Supplement as their primary course delivery method to provide the best learning (i.e., 9 of the 11 Subject Areas). Furthermore, Completely Online was the least selected delivery method for all Subject Areas (with the exception of the Office / Productivity Software category). The third research question (i.e., What is the typical demographic profile of students who prefer an online format to a traditional on-ground format?) sought to determine the typical demographic profile of students who prefer an on-line format to a traditional on-ground format? The results from these survey questions are summarized in APPENDIX D, Table 4. As the results in Table 4 indicate, it is difficult to use demographic factors to differentiate a student who prefers any online format over a traditional (i.e., on-ground) format. A few demographic statistics, however, from Table 4 are notable. For example, students who classified themselves as Graduate and ACE (Adult and Continuing Education) seemed to prefer the online format to the on-ground format. In addition, those students who reside 6 – 10 miles from campus also prefer the online format. 6. CONCLUSIONS The present research surveyed undergraduate, graduate, and post-graduate students as to which non-tradition instructional method of course content delivery (i.e., online, partially online, on-ground with online supplements) is most effective for IT/IS courses, as well as, which instructional method of course content delivery provides the best learning for each grouping of IT/IS subject areas. Also, this study profiled groups of students who prefer an online format to a traditional on-ground format In relation to the non-traditional delivery method, overall, the research participants rated the perceived effectiveness of On-Ground with an Online Supplement or Hybrid/Partially Online courses as higher than that of completely online courses. These findings are consistent with the findings of past studies in which the researchers determined that students prefer a combination of on-ground and online instruction, rather than instruction delivered exclusively online (Drennan et al. and Flynn et al.). Therefore, the present study suggests that online instruction should be coupled with traditional (i.e., on-ground) methods in order to provide the most effective delivery of content. Further supporting results from the first research question, respondents perceived a combination of online and on-ground methods as providing the best learning. Specifically, respondents reported On-Ground, and On-Ground with an Online Supplement, and Hybrid/Partially Online, as formats which provide better learning than the completely online format. The only caveat to the above findings involved the Office/Productivity Software Subject Area. For Office/Productivity Software topics, the respondents felt that Completely Online provided the best learning. The Office/Productivity exception may be explained by the fact that many of today’s students enter college with a prior working knowledge of spreadsheet, word-processing, and presentation software. The analysis of demographic factors in relation to a student’s decision to take (or not take) an online course was much more difficult. The current research found that very few demographic factors emerged as influencing a student’s choice of delivery method. Student level and distance from campus, however, did surface as possible influences to the course delivery decision. Specifically, the current study found that Graduate students and other “non-traditional” students may prefer the online format. Further, the present study found that students who live between 6 – 10 miles from campus may also prefer online delivery to on-ground. The results of this study may seem surprising when one considers the original target audience of distance learning programs: non-traditional students, who cannot attend traditional courses due to employment, distance from campus, and family responsibilities. Many past studies supporting distance learning programs, however, did not differentiate between technical and non-technical degree programs. In contrast, the current study surveyed students enrolled in IT/IS programs. Since IT/IS programs involve content that is quantitative and technically-oriented, some students may have difficulty comprehending such content in an online format. The technical nature of such course content may explain why IT/IS students in the current study preferred an on-ground delivery format or an online format that involved an on-ground component. Both the On-Ground and the On-Ground with Online Supplement formats would give IT/IS students more interaction with faculty. This increased interaction, as discussed previously, may be critical in conveying technical content. Although the results cannot be generalized to all IT/IS students in all IT/IS programs, it may be inferred that the IT/IS students in this study preferred classroom delivery of course content for IT/IS-related topics. However, if online delivery is to be incorporated into an IT/IS curriculum, the current study suggests that online delivery should be provided as a hybrid (i.e., blended learning, or as a supplement to traditional, on-ground delivery. As online learning delivery method continues to gain popularity, faculty and advisors must be equipped to help students achieve success. By recognizing variables that contribute to the success of online learners, educators can better prepare students to choose between different learning formats and across various subject matter areas. 7. REFERENCES Barker, B., & Brooks, D. (2005). An evaluation of short-term distributed online learning events. International Journal on ELearning, 4(2), 209- 228. Drennan, J., Kennedy, J., & Pisarski, A. (2005). Factors affecting student attitudes toward flexible online learning in management education. Journal of Educational Research, 98(6), 331-339. Flynn, A., Concannon, F., & Bheachain, C. N. (2005). Undergraduate students' perceptions of technology-supported learning: The case of an accounting class. ELearning, 4(4), 427-444. Huang, Z., & Cappel, J. J. (2005). Assessment of a web-based learning game in an information systems course. The Journal of Computer Information Systems, 45(4), 42-49. Johnson, G. (2004). Conversion anxiety. Training, 41(2), 34-38. Noble, D. F. (Ed.). (2002). Diploma mills: The automation of higher education. New York: Monthly Review Press. O'Malley, John (1999) Online Journal of Distance Learning Administration, Volume II, Number IV, Winter 1999 Rich, D. (2001). eLearning: A new way to develop employees. Electronic Business, 27(8), 20. Sessa, D. (2001, March 12). Business plan: A look at all the ways companies hope to make money from online education. Wall Street Journal, p. R8. Sitzmann, T. (2005). Is e-learning as effective as classroom learning? T+D, 59(8), 18. Trierweiler, C., & Rivera, R. (2005). Is online education right for corporate learning? T+D, 59(9), 44-48. APPENDIX A TABLE 1: PERCEIVED EFFECTIVENESS BY DELIVERY METHOD Effectiveness Completely Online Hybrid / Partially Online On-ground with Online Supplement Count Percent Count Percent Count Percent Very Effective 14 9.2% 30 19.4% 57 36.7% Effective 43 27.6% 62 39.8% 60 38.8% Somewhat Effective 57 36.7% 47 30.6% 27 17.3% Somewhat Ineffective 27 17.3% 11 7.1% 9 6.1% Ineffective 9 6.1% 5 3.1% 2 1.0% Very Ineffective 5 3.1% 0 0.0% 0 0.0% Total 155 100.0% 155 100.0% 155 100.0% Mean 4.07 4.65 5.04 SD 1.160 .975 .941 APPENDIX B TABLE 2: PAIRED-SAMPLES T-TEST RESULTS Paired Differences t Ddf Sig. (2-tailed) 95% Confidence Interval of the Difference Mean Std. Deviation Std. Error Mean Lower Upper Pair 1 Completely Online - Hybrid/Partially Online -.582 1.121 .113 -.806 -.357 -5.137 97 .000 Pair 2 Hybrid/Partially Online – On-Ground with Online Supplement -.388 1.172 .118 -.623 -.153 -3.275 97 .001 Pair 3 Completely Online - On-Ground with Online Supplement -.969 1.615 .163 -1.293 -.646 -5.943 97 .000 APPENDIX C TABLE 3: BEST LEARNING BY DELIVERY METHOD AND SUBJECT AREA Subject Area On-ground Completely Online Hybrid/Partially Online On-ground with Online Supplement Count % Count % Count % Count % Software Development / Programming 53 34.2% 14 9.0% 40 25.8% 48 31.0% Network Administration / Security 49 31.6% 18 11.6% 36 23.2% 52 33.5% Web Development / Web Programming 37 23.9% 25 16.1% 50 32.3% 43 27.5% Multimedia / Graphics 37 23.9% 25 16.1% 43 27.7% 50 32.3% Office / Productivity Software 33 21.3% 49 31.6% 38 24.5% 35 22.6% Project Management 44 28.4% 28 18.1% 34 21.9% 49 31.6% System Analysis & Design 48 31.0% 20 12.9% 37 23.9% 50 32.3% Certification Courses (e.g., A+,N+) 50 32.3% 24 15.5% 38 24.5% 43 27.7% Operating Systems 44 28.4% 27 17.4% 41 26.5% 43 27.7% Database 58 37.4% 17 11.0% 36 23.2% 44 28.4% Business Intelligence (e.g. Data Warehousing, Data Mining) 45 29.0% 19 12.3% 44 28.4% 47 30.3% APPENDIX D TABLE 4: DEMOGRAPHIC PROFILE OF STUDENTS WHO PREFER/DO NOT PREFER ONLINE FORMAT Demographic Prefer Online Format Do NOT Prefer Online Format Total Count Percentage Count Percentage Count Percentage Student Level Undergraduate Graduate Doctorate ACE Integrated 14 26 10 11 2 31.1% 55.3% 23.8% 64.7% 50.0% 31 21 32 6 2 68.9% 44.7% 76.2% 35.3% 50.0% 45 47 42 17 4 29.0% 30.0% 27.1% 11.0% 2.9% Sex Male Female 46 17 43.0% 35.4% 61 31 57.0% 64.6% 107 48 69.0% 31.0% Age range 18-20 22-30 31-40 41-50 51-60 61 or older 5 29 18 10 1 0 26.3% 49.2% 42.9% 41.7% 14.3% 0.00% 14 30 24 14 6 4 73.7% 50.8% 51.1% 58.3% 85.7% 100.0% 19 59 42 24 7 4 12.2% 38.1% 27.1% 15.5% 4.5% 2.6% Employment Status Part-Time Job Full-Time Job Not Employed 10 50 3 33.3% 45.0% 21.4% 20 61 11 66.7% 55.0% 78.6% 30 111 14 19.4% 71.6% 9.0% Enrollment Status Full-Time Part-Time 44 19 37.9% 48.7% 72 20 62.1% 51.3% 116 39 75.0% 25.0% Living Status Resident Commuter 2 61 22.2% 41.8% 7 85 77.8% 58.2% 9 146 5.8% 94.2% Previous Online Course Yes No 47 16 48.0% 28.1% 51 41 52.0% 71.9% 98 57 63.0% 37.0% Distance from Campus 0 - 5 Miles 6 - 10 Miles 10 – 15 Miles 15+ Miles 8 15 7 33 26.7% 60.0% 38.9% 40.2% 22 10 11 49 73.3% 40.0% 61.1% 59.8% 30 25 18 82 19.4% 16.1% 11.6% 52.9%