Implementation of a Basic Computer Skills Assessment Mechanism for Incoming Freshmen Debbie Tesch tesch@xavier.edu Marianne Murphy Elaine Crable Management Information Systems Department Xavier University Cincinnati, OH 45207, USA Abstract   Basic computer skills assessment for incoming freshmen provides an opportunity for placement appropriate to the students’ present skill level. At this university, assessment for incoming freshmen was begun in the College of Business in order to ensure that students entering the information systems core course have a set of requisite skills in file management, word processing, spreadsheet, and presentation graphics processing before entering the MIS core course in the College of Business. This paper describes the implementation and placement results of this process for incoming freshmen at a private liberal arts university in the Midwest. Keywords: computer skills assessment, basic computer skills, IS curriculum 1. INTRODUCTION Faculty in universities today continue to be challenged as they attempt to teach the basic MIS course in their undergraduate business curriculum. Typically the business curriculum includes one basic MIS course which provides a basic overview of MIS concepts plus a software component designed to develop problem solving skills using spreadsheet and database software. Instruction in word processing, presentation graphics, and basic spreadsheet skills is either relegated to the university core or basic skill proficiency is “assumed” often requiring instructors in specific disciplines, including the MIS course, to provide the necessary instruction. Since the university core at this Midwestern private university does not currently require the demonstration of basic computer skill proficiency, teaching the MIS course is increasingly challenging as technical skills and computer experience of today’s incoming freshmen continue to vary. This paper introduces the implementation of a skills' assessment mechanism designed to offer a computer-based training opportunity for students identified as skills deficient prior to enrollment in the MIS course and reports results of the initial skills assessment. 2. BACKGROUND Computer literacy courses over the last three decades have been shaped by the available computer technology and the integration of that technology into society. Today, the classroom model of computer literacy established in the mid 1980s still resembles the current design including requirements for computer awareness, computer literacy, computing fluency, and computer expertise. Emergent new technologies change the nature of computing as well as societies’ perception of technology (Hoffman and Blake, 2003) and expand requirements for computer expertise. With computer literacy becoming the third most non-negotiable employer requirement (“Overcoming IT Illiteracy,” 2003), word processing, spreadsheet, presentation graphics, and file management skills have a major role as components of computer expertise. In spite of varying degrees of agreement over appropriate content for satisfying computer literacy requirements, all definitions support a necessary computer skills base typically including file management operations, word processing, spreadsheets, and presentation graphics. Discrepancy continues however over the status of incoming freshmen in terms of basic computer skill proficiency and in terms of where this need is best met. One undergraduate program (Baugh and Morris, 2004) felt that a basic computer course was no longer necessary and that incoming freshmen were “computer literate.” Using a basic assessment of spreadsheet and database skills, faculty and administrators were convinced that the basic computer course remained necessary, but that through assessments such as these, curriculum adjustments could better meet the needs of students (Baugh and Morris, 2004). Yet, Reffell and Whitworth (2002) consider the need to embed IT skills into the more general social and academic skills taught at universities. They suggest that these integrated basic computer skills should be expanded to include library and information services skills, the information literacy component. Hindi, Miller, and Wenger (2002) investigated student perceptions of computer literacy skills via a pre-course questionnaire. The Midwestern students in the Hindi, et al. (2002) study indicated that they were comfortable with word processing, but needed training in spreadsheet and database applications. What effect does this requirement of computer self-efficacy have on performance? Self-efficacy, an individual’s personal judgment of their ability to succeed in reaching a specific goal, is believed to play a critical role in self-motivation when dealing with unfamiliar tasks. In learning computer skills, computer self-efficacy has been discovered to be positively related to learning level. Thus greater attention needs to be paid to the technology learning environment in order to avoid resistance to computer use and instead introduce training programs designed to positively impact self-efficacy (Torkzadeh, Koufteros, and Pflughoeft, 2003). Although students' perceptions of their skill levels are frequently used and fairly easy to obtain, significant differences have been discovered between students’ perceived and actual computer competency. Larres, Ballantine, and Whittington, (2003) discovered that the large majority of students’ over-estimate their computer knowledge. Only students with greater computer skill ability were more accurate in their self-assessment. Therefore, the faculty at this university felt it necessary to implement a better way to assess these skills and to implement a means for students to improve their skills. To this end, this university developed a one credit basic computer skills course to focus on word processing, basic spreadsheet processing, presentation graphics and basic file management in a self-paced computer-based training environment. Edmiston and McClelland (2001) suggest that a first literacy course would best be taught with some lectures and demonstration while the rest of the time is used for hands-on activities. However, a recent comparison of using computer-based training versus the standard lecture demonstration approach via an online assessment indicated that there was no statistical difference between implementations (Simis and Hoong, 2001). Bretz and Johnson (2000) discovered that a web-based, self-paced, competency-based computer literacy course was found to be more effective than the traditional “monkey-see, monkey-do” method. Different approaches for delivering a computer literacy course remain in question. The basic computer skills course in this study is described in the university catalog as INFO100 Basic Computer Applications, “a computer-based training course for developing core competencies using productivity tools relevant for business applications.” Students who enroll in INFO100 will use a computer-based training tool to develop competencies in only areas where deficiencies were identified during a pre-assessment. These students will have the opportunity to go beyond the core competency level in all other areas. Implementation of INFO100 intends to move skill training out of the MIS core course which will allow faculty to concentrate on problem solving and decision making skills using technology. The core MIS course will then be able to evolve from what has become a computer literacy course back to its original intent of focusing on MIS concepts, decision making and how to use technology for a competitive advantage. The basic skills set acquired in Excel will be used to solve business problems. Presentation graphics training in INFO100 will be employed as students explore the use of technology and share these investigations with the class. Business reports will be created using honed word processing skills. Ives, Valacich, Watson and Zmud, (2002) discussed the importance of being able to move to an MIS core course where key concepts and principles can be delivered in an integrated and comprehensive course. They discussed how it is necessary in today’s world to teach such concepts as technology infrastructures, economics of IS, ethical and security issues, and critical thinking skills using technology. Removal of skills competency requirements makes room for inclusion of more “value skills”, i.e. teamwork, communication, oral and personal presentation, project management, leadership, and problem-solving (Miertschin and Willis, 2003). It is the goal of these authors that the computer literacy course will allow us to move these value skills into the re-designed MIS core course. 3. METHODOLOGY A basic computer skills assessment mechanism aimed at accurately assessing student skill proficiency in file management, word processing, spreadsheets, and presentation graphics was implemented using SAM 2003 version 1.5. SAM 2003 is a skills assessment manager from Course Technology that offers skills assessment of the Microsoft Office Suite in a simulated environment. Faculty in the MIS department having had previous experience with the SAM XP product were offered the opportunity to provide input towards the improved SAM 2003 product introduced spring 2004. Improvements to SAM 2003 include true open simulations that allow students to complete most tasks using any method and improved stability and reliability in the server environment. SAM 2003 simulations let students move more freely through the application, even opening up random dialog boxes without penalty. Skills assessment tasks were chosen using SAM 2003 tasks available from the New Perspectives text series orientation. This orientation offers tasks categorized as brief, introductory, or comprehensive skills. The introductory skill level was used to select questions for each proficiency category. Once selected, task lists were generated and made available to members of the MIS faculty and to university representatives from the library and information support services interested in participating. The initial task list was modified based on participants’ suggestions and a suggested time limit of forty-five minutes for proficiency testing. The final assessment test contained 72 questions. The SAM 2003 development experts at Course Technology estimated that this test would require approximately 46 minutes for completion. The tasks and assessment results are reported on tables 2 through 5. The results are further discussed in Section 4 of this paper. Skills assessment tests were scheduled for each of five student orientation days scheduled in May and June of 2004. The computer skills assessment was given at 1:45 in the afternoon after a morning filled with math and foreign language placement exercises and other orientation activities. Students were instructed to consider the assessment strictly a placement mechanism and were offered the opportunity at any point to end the exam and accept the opportunity to take the basic computer applications one-credit course. The skills assessment was the last activity prior to completing the students’ fall schedule. SAM assessment scores were available immediately upon completion of the assessment and were used as part of the advising process. Students receiving scores below 80 were placed in the INFO 100, Basic Computer Applications course. 4. RESULTS One hundred and sixty six freshmen participated in the orientation and early registration process. Two students’ exam results were compromised by SAM 2003 and therefore were discarded. The remaining 164 participants’ scores are reported here. Additionally, 128 or 77% of the students completed a requested informational survey. Demographics for incoming freshmen that took the computer skills assessment in May and June 2004 are in Table 1. Sixty-eight percent (111) of the students are male. Sixty-eight percent (87 of 128) of those students who completed the information survey attended private schools. Students reported an average high school GPA of 3.43. Of the 33 students taking the ACT, the average ACT score was 24.5. One hundred twenty-five students reported SAT scores. The average SAT verbal score was 552.6. The average SAT math score was 565.1. Almost half (49.4%) of participants were undecided business majors. Table 1 Summary Statistics for Incoming Freshmen Entering Major: Accounting 23 Males 111 Economics 1 Females 53 Entrepreneurship 4 N 164 Finance 7 Information Systems 1 High School Type: International Business 18 Private 87 Management 6 Public 38 Marketing 14 Home School 3 Business Undecided 81 Other 9 High School Average GPA: 3.43 (n = 155) Average ACT: 24.5 (n = 33) Average SAT-verbal: 552.6 (n = 125) Average SAT-math: 565.1 (n = 125) Forty (24%) students received an assessment of 80 or better and were placed in the information systems core course (INFO 200). During advising, an additional 10 who scored 79 were offered the opportunity to enroll directly into INFO 200. To date, 104 students are enrolled in the basic computer applications course (INFO 100). The Word portion of the assessment measures the twenty basic tasks listed in Table 2. All tasks were presented to the students. Of the tasks presented, fourteen were successfully completed with a score of 80 or higher. Tasks’ scores below 80 include the ability to change the page orientation, insert and format the current date, change page margins, indent a block of text, use find and replace to replace existing text, and insert a clip art image. The Excel portion of the assessment consists of 29 basic tasks as described in Table 3. Five of these tasks (open a workbook, apply bold, italics, underline, change font color, change worksheet orientation, and change page margins) are accomplished in a similar manner as their Word counterpart. Six tasks relate to creating formulas and functions. Two tasks relate to chart creation and editing. The remaining tasks involve worksheet, row, column, and cell manipulation tasks including insertion, deletion, rearranging, and formatting tasks. Only seven of 29 tasks were successfully completed by 80% of the participants (see Table 3). With the exception of deleting selected cells (15% correct), the lowest percentages correct pertain to the use of formulas: absolute references (1% correct), current date and time (11% correct), and relative cell references (19% correct). Tasks most frequently skipped include creating a formula using absolute cell references (54%), creating a formula using relative cell references (48%), and inserting the current date and time (46%). Table 4 lists the 15 PowerPoint tasks measured in this assessment. Six of the 15 tasks were successfully completed by 80% Table 2 Microsoft Word Skills Assessed Skill Correct Incorrect Skipped Not Presented % Correct Open a document 164 0 0 0 100% View a document in Print Preview 164 0 0 0 100% Print a document 164 0 0 0 100% Save a document 164 0 0 0 100% Change font size 163 1 0 0 99% Change font type 163 1 0 0 99% Correct spelling 162 1 1 0 99% Bold text 163 1 0 0 99% Select a paragraph 161 1 2 0 98% Insert text within existing text in document 154 3 7 0 94% Copy table data into adjacent columns 150 9 5 0 91% Use drag-and-drop to move text 135 16 13 0 82% Cut and paste text 134 24 6 0 82% Add bullets to a list 134 25 5 0 82% Change document orientation 114 23 27 0 70% Insert/format current date 102 56 6 0 62% Change page margins 101 32 31 0 62% Indent a block of text 96 34 34 0 59% Use find and replace to replace existing text 73 56 35 0 45% Insert a clip art image in a document 17 131 16 0 10% of the participants. These tasks included creating a new blank presentation, deleting a slide, adding a slide, running a slide show, editing slide contents, and inserting a clip art image. The use of find and replace is the least successful task (34% correct). Table 5 lists the 11 tasks that test Microsoft XP file and program management skills. None of the 11 tasks are successfully completed by 80% or more of the participants. The most frequently missed (98 incorrect) question pertains to viewing the contents of the Documents and Settings folder on the C: drive. For each task more questions were not presented than for any other application set indicating that students were not offered the opportunity to attempt these questions due to time constraints. 5. DISCUSSION Basic skills assessment for incoming freshmen provides an opportunity for placement appropriate to the students’ present skill level. At this university, basic skills assessment for incoming freshmen was begun in the College of Business in order to ensure that students entering the information systems core course have a set of requisite skills in file management, word processing, spreadsheet, and presentation graphics processing before entering the MIS core course in the College of Business. Computer-based skills assessment as described here is successful. Of the 166 test attempts, only 2 scores (1%) are unsalvageable. With as many as 37 people taking the test at any given time, few problems occurred. In all cases where a Windows interrupt occurred during testing, students were able to successfully re-enter the exam without losing their completed work. Each instructor teaching the basic computer applications course will receive a frequency analysis for each student. The intent is that this information will provide a useful way to assist the student in navigating through deficient core computer skills. Taught in a computer-based training (CBT) environment, students will contract to complete necessary Table 3 Microsoft Excel Skills Assessed Skill Correct Incorrect Skipped Not Presented % Correct Apply bold, italics, underline 152 7 1 4 93% Edit numbers in cells 151 7 2 4 92% Locate and open existing workbooks 146 10 2 6 89% Delete rows and columns 138 20 1 5 84% Move a worksheet to a new location 136 10 13 5 83% Insert a new, blank worksheet to the beginning of a workbook 135 11 15 3 82% Change font color of text in a cell 131 24 4 5 80% Modify worksheet orientation to landscape 113 11 35 5 69% Format chart areas 111 39 7 7 68% Hide a column 109 35 19 1 66% Use the ‘Go to’ method to move to a cell 106 30 26 2 65% Clear cell content 104 44 12 4 63% Change margins in Page Setup 102 27 33 2 62% Insert columns and rows 96 58 3 7 59% Add a double bottom border to a cell range 72 49 39 4 44% Use the MAX function 72 33 55 4 44% Sum the contents of a column using the SUM function 63 53 47 1 38% Apply number formats (currency and percent) 62 63 36 3 38% Center across selection 53 74 31 6 32% Create a stacked column chart to compare values over time using the Chart Wizard 43 54 62 5 26% Use the fill handle to copy a cell 34 80 47 3 21% Create a formula to subtract one value from another 32 68 61 3 20% Create a formula using relative cell references 31 50 78 5 19% Delete selected cells 24 123 14 3 15% Create a formula to insert the current date and time that is updated automatically 18 66 75 5 11% Use an absolute reference in a formula to calculate a value as a percent of another value 2 67 89 6 1% training and have the opportunity to contract to extend their skills beyond core concepts in each area. Regardless of their degree of participation in class, this course will require the purchase of SAM 2003. This software provides skills training in Word, Excel, Access, PowerPoint, Outlook, and Publisher, and basic file operations in Windows 2000, XP, and 2003, as well as objective-based training and assessment in computer concepts, MIS, and the Internet. The results of this first effort to ensure a more homogenous skill base in the business core IS class will not be fully evident until at least the fall of 2005 when the majority of students have either tested out of or completed a basic computer skills training. Particular effort must be made in the INFO 100 course to ensure a significant level of training in Excel so that problem solving exercises in INFO 200 and other business classes will be successful. Table 4 Microsoft PowerPoint Skills Assessed Skill Correct Incorrect Skipped Not Presented % Correct Create a new, blank presentation. 146 1 1 16 89% Delete Slide 5. 144 3 1 16 88% Add a new slide to the end. 140 5 1 18 85% Run the slide show. 134 5 2 25 82% Correct the capitalization of the bulleted items on Slide 2. 131 4 9 20 80% Use ClipArt's search feature to add a clip art image to Slide 4. 131 6 11 16 80% Create a new slide just after the title using the layout with a bulleted list on the left and clip art on the right 117 16 15 16 71% Add the slide title, Great Rewards, to Slide 2. Add a four-item bulleted list. Be sure to use the correct spelling and capitalization 112 8 26 18 68% Create a presentation using the Mountain Top design template. 102 32 13 17 62% Create a text box on Slide 3 in the lower left corner. Type "Visit Galveston Island" in the text box. 97 34 16 17 59% Add "DrivingForce.com" to the footer on every slide. 91 25 31 17 55% Apply the Fade in one by one animation scheme to all the slides in the presentation at one time. 86 33 24 21 52% Print portrait handouts of the presentation, placing six slides per page. 83 38 21 22 51% Remove the slide transitions. 84 31 27 22 51% Replace "2003" with "2004" throughout. 55 39 50 20 34%` 6. CONCLUSION Every new initiative must survive the “first time” in terms of new implementations. Rather than continue round table discussions at the university and college level concerning the need for and appropriate placement of basic computer skills assessment and training mechanisms, this information systems faculty decided the need was now and secured necessary resources to implement skills assessment during freshmen orientation. As a result, computer-based training implementation will begin in the upcoming semester. This implementation represents a first step towards restoring the college information systems core course to its initial computer applications objective of concentrating on problem solving and decision making skills using technology. Table 5 Microsoft XP Skills Assessed Skill Correct Incorrect Skipped Not Presented % Correct Open the Start menu. 127 0 4 33 77% Select WordPad on the Accessories menu. 122 3 3 36 74% Explore the contents of the floppy disk using the My Computer window. 120 1 3 40 73% Expand the My Computer folder 117 3 7 38 71% Create a folder named "New Projects" on the floppy disk. 111 8 7 38 68% Increase the width and height of the My Computer window. 105 12 11 36 64% Save the Learning to Save document on the desktop (it has previously been saved to the My Documents folder) 104 16 4 40 63% Switch between WordPad and Notepad. 95 22 9 38 58% Copy files to a folder on the floppy disk 89 22 10 43 54% Change the name of the Practice Test file to "SampleTest". 71 50 3 40 43% View the contents of the Documents and Settings folder on the C: drive. Contents should be visible in the right pane. 24 98 3 39 15% 7. REFERENCES Baugh, Jeanne M and Robert Morris, 2003, “Assessment of Spreadsheet and Database Skills in the Undergraduate Student.” Proceedings of ISECON2003, November 6-9. Bretz, Robert and Linda Johnson, L.,2000, “An Innovative Pedagogy for Teaching and Evaluating Computer Literacy,” Information Technology and Management, 1(4), pp. 283-292. Edmiston, Elizabeth. and Marilyn McClelland, May 2001, “A Conversation about ensuring computer literacy of first-year students,” Journal of Computing in Small Colleges, 16(2), pp. 306-316. Hindi, Nitham M., Don Miller, and James Wenger, Summer 2002, “Computer Literacy: Implications for Teaching a College-Level Course,” Journal of Information Systems Education, 13(2), pp. 143-151. Hoffman, Mark and Jonathan Blake, 2003, “Computer literacy: today and tomorrow,” The Journal of Computing in Small Colleges, 18(5), pp. 221-223. Ives, Blake, Joe Valacich, Rick Watson, and Robert Zmud,2002, “What every business student needs to know about information systems.” Communications of the Association for Information Systems, 9(30), December 2002, pp. 1-18. Larres, Patricia., Joan Ballantine, and Mark Whittington, June 2003, “Evaluating the validity of self-assessment: measuring computer literacy among entry-level undergraduates within accounting degree programmes at two UK universities,” Accounting Education, 12(2), 97-112 Miertschin, Susan and Cheryl Willis, October 2003, “A Freshman Course in Emerging Information Technologies,” CITCA 2003 Lafayette, IN, pp. 115-118. “Overcoming IT Illiteracy,” Personnel Today, 28 January 2003, pp. 18. Reffel, Pete and Andrew Whitworth, 2002, “Information fluency: critically examining IT education,” New Library World, 103(1182/1183), pp. 427-435. Simis, Peter and Ivan Hoong, 2001, “A Comparison of Using CBT and Teaching Assistants to Teach Microsoft Office 2000,” ISECON 2001, 18th Annual Conference on Information Systems Education, Cincinnati, OH Torkzadeh, Gholamereza, Xenophon Koufteros, and Kurt Pflughoeft, 2003, “Confirmatory Analysis of Computer Self-Efficacy,” Structural Equation Modeling, 10(2), pp. 263-27. ?? ?? ?? ??