Assessing the Learning Outcomes of a Computer Information Systems (CIS) Program Samuel Abraham Computer Information Systems Department Siena Heights University Adrian MI 49221 sam@sienahts.edu Track: Information Systems Assessment Abstract In recent years greater attention has been paid to develop learning outcomes for academic programs and then to develop methods to assess these learning outcomes. Generally speaking, there are two kinds of outcomes: course outcomes and program outcomes. Assessments of these learning outcomes in institutions of higher education are mandated by the accrediting organizations. This paper describes a methodology used by a Computer Information Systems program in a small undergraduate institution to develop its learning outcomes, to collect assessment data, and to evaluate or assess its course and program outcomes. Key words: course outcomes, learning outcomes, learning outcome assessments, measurement, outcome based education, program outcomes, and programs metrics 1.INTRODUCTION In recent years greater attention has been paid to develop learning outcomes for academic programs and then to develop meaningful assessment methods to evaluate these outcomes. Assessment is a systematic and on-going process of collecting, interpreting, and acting on information relating to the goals and outcomes developed to support the mission and purpose of an institution (Osters, 2003). According to Acharya (2003), assessments should help us to answer the following questions: (1) What do we want the students to learn? (2) Why do we want them to learn it? (3) How can we help them to learn it? (4) How do we know what they have learned? Also Osters (2003) pointed out that assessments should help us to improve what we are doing. Assessment begins with the articulation and development of measurable outcomes. Generally speaking, there are two kinds of learning outcomes: course outcomes and program outcomes. The course outcomes should describe what students are expected to learn from an individual course, while program outcomes should describe what a student is expected to accomplish after completing the coursework from the program. Maki (2002) pointed out that learning outcome assessments must be based on institutional curiosity to seek answers to questions about student learning, why they learn, how well they learn, when they learn, and explores how pedagogies and educational experiences develop, and foster student learning. Maki (2002) also pointed out that innovations in pedagogy or integration of diverse methods of teaching and learning into a program of study, redesign of a program, reconceptualizing the role of advising, or establishing stronger connections between curriculum and non-curriculum represents some of the kinds of changes that faculty and staff may undertake to improve student learning and development based on their interpretations of learning outcome assessment results. 2. PROGRAM LEARNING OUTCOME DEVELOPMENT Learning outcomes should describe what students will be able to demonstrate in terms of knowledge, skills, and values upon completion of a course, a span of several courses, or a degree program (Osters, 2003). Clear statement of learning outcomes serves as the foundation to assess the effectiveness of the teaching and learning process. According to Osters (2003), the three essential components of a measurable learning outcome are: (1) Student learning behaviors, (2) Appropriate assessment methods, and (3) Specific student performance criteria. Student behaviors describe what students are expected to demonstrate by the completion of the course. Action verbs like demonstrate, apply, define, analyze, etc. are used to describe student behaviors. Assessment methods are tools and techniques used to determine the extent to which the stated learning outcomes are achieved. Student performance criteria should be expressed in specific and measurable terms that are acceptable to a specific course or series of courses. A variety of methods, qualitative and quantitative, direct and indirect, should be used to assess the learning outcomes. Keep in mind that a simple letter grade alone does not provide adequate feedback to student's performance, because the letter grade alone does not sufficiently identify with the strengths and weaknesses of individual learning outcomes. If the grading system is accompanied by a rubric where the individual outcome components are addressed, then this tool can be used to pinpoint the weakness and strengths of the student's performance. It is very important to define the learning outcomes of a program/course in specific and precise manner. Spady & Marshall (1994) wrote: "Outcomes are clear, observable demonstrations of student learning that occur after a significant set of learning experiences?Typically these demonstrations, or performances, reflect three things: (1) what the student knows; (2) what the student can actually do with what he or she knows; (3) the student's confidence and motivation in carrying out the demonstration. A well-defined outcome will have clearly defined content or concepts and be demonstrated through a well-defined process beginning with directive or requests such as explain, organize, or produce." The CIS program at Siena Heights University decided to develop a set of program learning outcomes first. Faculty members were asked to develop a list of outcomes that they thought were important for the program. They were asked to consult with the pertinent literature to develop meaningful and measurable outcomes. They were also asked to consult with other educational institutions where similar programs were available, professional organizations, and accreditation agencies to learn more about the outcome development process. After an exhaustive research the faculty members developed a number of outcomes for the CIS program and from this list we were able to select six measurable outcomes for our program. The American Association of Higher Education's (AAHH) (1996) nine principles of good practices for assessing student learning were used in the selection process. We also used a number of other research documents from the AAHE's assessment web site. Our hope is that the graduates of our program will be able to show that they have accomplished these six outcomes by receiving a degree from the CIS program. The following list shows the learning outcomes developed by the CIS program. 1. Students will demonstrate the skill to write complete, complex programs that are fully tested. 2.Students will demonstrate the skill to develop a complete information system that incorporates feasibility study, analysis, design, systems development, testing, implementation and maintenance. 3.Students will demonstrate the ability to solve problems using the computer as a tool, using either application packages or custom programs. 4. Students will demonstrate the ability to work as a team member in a problem-solving situation. 5.Students will demonstrate the ability to investigate existing literature in Information systems. 6.Students will demonstrate the ability to communicate effectively. Fig 1 3. COURSE LEARNING OUTCOME DEVELOPMENT Once these outcomes were developed, we set out to see how these outcomes can be accomplished through our course offerings. We know that we have to develop a set of outcomes for each of our courses, keeping in mind that there must be a match between these course outcomes and the program outcomes. In other words, the stated program outcomes must be accomplished through the course outcomes. Faculty who are teaching the individual courses are asked to take the program outcomes and see how these outcomes can be accomplished through their courses. Also these are the outcomes a faculty would like his/her students to know at the completion of that particular course. The importance of measurability and clarity of the course outcomes were emphasized. Faculty members developed a set of learning outcomes for each course from which we selected five or six outcomes for each individual course. We then developed a table to show the relationship between program outcomes and courses offerings. We also agreed that when we develop new courses in the future, we need to pay greater attention to the course outcomes to see how the new course will satisfy the program outcomes. By adding new rows in Table 2 we will be able to get a quick view of the relationship between the course and program outcomes. Course Number Course Title CIS 119 Visual Basic Programming CIS 218 Introduction to Information Systems CIS 252 Introduction to C++ Programming CIS 260 Cobol Programming CIS 353 Systems Analysis CIS 363 Data Base Structures CIS 443 Data Communication CIS 465 Management Information Systems CIS 495 Senior Project Electives CIS 352 Data Structures Using C++ CIS 340 Java Programming CIS 370 Network Operating Systems CIS 460 Web Development CIS 480 Internships CIS 485 Emerging Technology Table 1 Required Courses Learning Outcomes 1 2 3 4 5 6 CIS 119 X X X CIS 218 X X X X CIS 252 X X X CIS 260 X X X CIS 353 X X X X X CIS 363 X X X X X CIS 443 X X X X CIS 465 X X X X CIS 495 X X X X X Electives CIS 352 X X X CIS 340 X X X CIS 370 X X X X CIS 460 X X X X X CIS 480 X X X CIS 485 X X X Table 2 The current CIS course offerings (course numbers and corresponding course titles) are listed in Table 1 for reference. Table 2 shows a mapping of the courses and the CIS program outcomes. In Table 2, the program outcomes are listed as column headings while the course offerings are listed as row headings. The courses are divided into two groups: required CIS courses and elective courses. (Students are required to take eleven CIS courses from the CIS program to receive a CIS degree). An "X" mark is placed in a cell to show which course is used to achieve a particular outcome. For example, CIS 119 (Visual Basic Programming) is used to achieve outcomes one, three and six. The same is true with CIS 260, CIS 252, and CIS 340. A further analysis of the table shows that outcome two is accomplished through CIS 353, CIS 363, CIS 460 and CIS 495. We are using computer as a tool to solve problems in all our courses and thus, we decided that outcome three will be addressed in all our classes. Students are expected to complete team projects in CIS 460, CIS 370, CIS 465, CIS 443, CIS 363, CIS 353, and CIS 218 and this requirement will address outcome four. Research papers are required in CIS 218, CIS 353, CIS 363, CIS 443, CIS 360, CIS 460, CIS 465, and CIS 495 and this requirement will address outcome five. Some kind of formal presentations (Oral, written, and/or posters) are required in our entire courses, and this requirement will address outcome six. As mentioned above, we have developed a number of outcomes for each course. Faculty members also developed a number of rubrics for each course to assess the achievement of each student. Upon completion of this course, students will be able to demonstrate proficiency in: 1. A disciplined approach to problem solving methods and algorithm development (CIS-O#1, 3) 2. The syntax and vocabulary of Visual Basic.Net (CIS-O # 1) 3. The usage of Visual Basic.Net Programming Environment (CIS-O #1) 4. Developing complete Visual Basic programs that include specification, design, code, debugging, testing, and documentation. (CIS-O #1) 5. Using computers as a tool in problem solving (CIS-O # 3) 6. Communicating the program development process in a predetermined format (CIS-O #6) Fig 2 The outcomes developed for CIS 119 (Visual Basic Programming) are given above. Similar outcomes were developed for all the other courses in our curriculum. 4.LEARNING OUTCOMES ASSESSMENT For the purpose of this paper, I will be using the Visual Basic class as an example. From Table 2, it is clear that the Visual Basic Course is addressing three (one, three, and six) of the program outcomes. The course outcomes stated in Fig 3 are developed to augment the program outcomes. We decided to concentrate on outcome one and six for this course as outcome three will be assessed in other classes. In order to assess outcome one and six we further divided the program development process into specification, design, listing, testing, output, and presentation. Students are expected to write seven to eight programs for this class. For convenience each program is graded out of 100 points and these points are divided into the six components above. The point allocation among the six components was somewhat arbitrary, but we thought that these allocations will show the order of importance between them. We allotted 15 points for specification, 15 points for design, 15 points for program listing, 10 points for program testing, 30 points for the correct output and 15 points for the presentation of the program material. A rubric is developed for grading the program using these six items. This rubric has a column for student grade and another column for the instructor to provide written feedback to the student. Students are encouraged to meet with the faculty member to discuss this feedback to improve their next assignment submission. Table 3 shows the sample rubric that we are using for all our programming assignments. Items Points (100) Student Grade Comments Specification 15 Design 15 Listing 15 Testing 10 Output 30 Presentation 15 Table 3 The first five items (Specification, Design, Listing, Testing, and Output) in Table 3 are to address outcome one, while the sixth item (Presentation) is to address outcome six. At the beginning of the semester, these items were explained and a handout is distributed in the class describing the expectations in detail. For specification, students are expected to write a program/problem description with input, output and processing requirements. Students are required to design an interface together with a flowchart or a pseudo-code. The listing of the program must be easily readable with internal documentation and ample comments. Students must provide proof that they have tested the program with all possible data sets. The program must also produce the correct output with sufficient formatting features. The presentation component is concerned about the communication aspect of the program. Students are required to submit the program as a package (specification, design, listing, test data, and the program in computer readable form). Students normally write seven to eight small programs and they are required to submit a specification, design, listing, and the output for each program. At the beginning of the assessment measurement process, we decided to use only the last program to assess the outcomes. We are currently in the process of using the same kind of assessment for all the programs. We used a spreadsheet to collect this data so that we can analyze the data at a later time. Student names are listed in the left hand column, the next five columns to measure outcome one, the next column is to calculate the percentage of these five items, and another column is added for the for the presentation at the end. The instructor grades the program using the template (Table 3) above and then the corresponding grades are inserted into the spreadsheet. The percentages for the first five items are calculated by adding those scores for the five items; dividing by 85 and then multiplying by 100. The percentage for the presentation is calculated by dividing the score by 15 and then multiplying by 100. We used Excel formulas to do these calculations. The last row shows the average score for each column which is a measurement for the whole class. A sample assessment sheet for the class is shown as Table 4 with all the cells completed with data, except for the actual student names. The last column shows the score the students received for this programming assignment. Column 7 shows the percentage students received for assessing outcome one, while column 9 shows the percentage the students received for outcome six assessments. The last two rows show the class average for each item within outcome one and six. In this case the class average for specification is 74%, while that of design is 76.67%, listing 77.33%, testing 79% and that of the output is 83.33%. These numbers can be used to assess whether the course outcomes and programs outcomes are met. In our case we decided that a 70% is required for the student to achieve the course outcome. This number was arbitrary but a 70% equates to an average "C". We also decided that a percentage above 90% means that the particular student exceeds the requirements. We further expanded the concept by theorizing that if 80 % of the students achieved the outcome then this class must have satisfied the program outcome. Table 4 In the above case, a number of inferences can be developed: 1. Two students (20%) did not meet the outcome one requirements. 2.One student (10%) did not meet the outcome six requirements. 3.One student (10%) exceeded the requirements for outcome one. 4. One student (10%) exceeds the requirements for outcome six. 5.The class average for outcome one score is below 80 %. Even though this is above the 70% benchmark, there is room for improvement here. 6. The averages for outcome six is below 80%, and that shows there is room for improvement for these items. 7. The class average for the total score is below 80%. This shows that more effort is required for the completion of these assignments. 8. As 80% of the students satisfied the outcome one requirements, the class satisfied the program outcome one. 9. As 90% of the students satisfied the outcome six requirements, the class satisfied the program outcome six. The percentages in Column 7 and Column 9 show that 78.94 % is the average score for outcome one, while 80% is the average for outcome six. This shows that the class has met the requirements for both outcomes. Please note that this class only has 10 students and thus the statistics may not be very meaningful. On the other hand, this kind of analysis will help a student to assess his/her strengths and weaknesses in a particular area. This analysis will also help faculty members to concentrate on areas where more attention is required. We have created another table as shown in Table 5 to show a list of students who have met the outcome requirements of CIS 119. Student Names Outcome One Outcome Six One M M Two M M Three M M Four M M Five E E Six NM NM Seven M M Eight M M Nine M M Ten NM M Class M M M - Met the requirements E - Exceeds the requirements NM - Not met the requirements Table 5 The same kind of data collection and analysis was performed for all other classes, and a table similar to Table 4 is generated for each of these classes. The data from all these tables are compiled into one table to show the final result as shown below as Table 6. Required Courses Learning Outcomes 1 2 3 4 5 6 CIS 119 M M M CIS 218 M M M M CIS 252 M M CIS 260 M M CIS 353 M M M M M CIS 363 M M M M M CIS 443 M M M M CIS 465 M M M M CIS 495 M M M M M Electives CIS 352 M M CIS 340 M M CIS 370 M M M M CIS 460 M M M M M CIS 480 M M M CIS 485 M M M Table 6 Table 6 shows that the CIS program is meeting the outcome requirements for this particular semester. We are in the process of developing a document to show the progress from semester to semester. 5. LESSONS LEARNED To draw reasonable conclusions from learning outcome assessments, we should make our assessments as fair as possible. Lam (1995) pointed out that a fair assessment is one in which students are given equitable opportunities to demonstrate what they know. Suskie (2000) suggested the following steps to make our assessments methods as fair as possible: (1) Have clearly stated learning outcomes and share them with your students, so they know what you expect from them, (2) Match your assessment to what you teach and vice versa, (3) Use different measures and many different kinds of measures, (4) Help students learn how to do the assessment tasks, (5) Engage and encourage your students, (6) Interpret assessment results appropriately, (7) Evaluate the outcomes of your assessments. Learning outcome assessment must be an ongoing process. According to Rodrigues (2002), assessment must become a part of an institution's culture. We are in the beginning stages of completing the first round of outcome evaluation for our program. Some courses passed through four or five semesters of assessments, while some other classes passed through only one semester of data collection and analysis. As we offer some courses infrequently, the learning outcomes assessment for these courses will also become infrequent. The faculty in our program felt that the experience of going through the process was very worthwhile, even though it was very time consuming and frustrating. The data collection and the subsequent data analysis show our strengths and weaknesses and we were able to address a number of those weaknesses. We used a number of other assessment techniques other than those described in this paper. All our graduating senior students are required to attend an exit interview. During the interview, a faculty member and the student address the program and course outcomes and solicit recommendation from the students. In addition to oral, written, and poster presentations, faculty members usually visit internship sites to evaluate the performance of the student interns. 6.CONCLUSIONS Outcome based education promises a better way of understanding student learning, and in turn provide ways to improve the quality of education. To measure or assess the learning outcomes effectively, we need to start with measurable, concise, and specific learning outcomes for our program and individual course that must be shared and explained to the students. Clear and concise measuring tools, techniques, instruments, and methods must also be developed and must be conveyed to the students to avoid confusion and frustration. Assessment data must be collected in an ongoing basis using multiple methods and instruments. Collected data must be analyzed to understand the strengths and weaknesses of the program, courses, teaching, and learning. This information must be used to improve teaching and learning, incorporate innovations in pedagogy, redesign programs and courses, redevelopment of the outcomes, and the development of new tools for assessment. For outcome assessment to be successful it must be ongoing and must be part of the institution's culture. Administrators must recognize the importance of this process by providing financial and collateral support. Outcome based education is here to stay and it is important for educators to be prepared to accept the challenge of developing measurable outcomes for their programs/institutions, assess these outcomes, and then use the assessment data to improve what they are doing. 7.REFERENCES American Association for Higher Education. (1996). "Nine principles of good practice for assessing student learning" [Online] .Available: http://www.aahe.org/assessment/principl.htm Acharya, Chandrama 2003, "Outcome-based Education (OBE): A new Paradigm for learning." CDTLink, November. Retrieved August 15, 2004, from http://www.cdtl.nus.edu.sg/link/nov2003/obe.htm Lam, T.C.M. (1995) "Fairness in performance assessment": ERIC digest [Online]. Available http://ericae.net/db/edo/ED391982.htm. Maki, Peggy L 2002, "Developing an Assessment Plan to Learn About Student Learning", AAHE, January. Retrieved August 15, 2004, from http://www.aahe.org/assessment/assessmentplan.htm Osters, Sandi 2003, "Writing measurable Leaning Outcomes." Texas A&M 3rd Annual 2003 Assessment Conference, February. Retrieved August 15, 2004, from http://www.tamu.edu/qep/conference/2003/Writing-Measurable-Learning-Outcomes.pdf Rodrigues, Raymond J, 2002, "What campus Buy-In Your Assessment Efforts?" AAHEBulletin.com, October. Retrieved August 15, 2004, from http://www.aahebulletin.com/member/articles/2002-10-feature02_pf.asp Spady, W. and Marshall, K. (1994). Light, not Heat, on OBE. The American School Board Journal. Vol. 181.pp 29-33. Suskier, Linda 2000, "Fair Assessment Practices." AAHEBullitin., July 16. Retrieved August 15, 2004, from http:// www.aahebulletin.com/public/archive/may2.asp William, K. C and William, S. D. 2004, "Planning Assessment of Student Leaning Outcomes: A process Within Your Grasp". International Journal of Nursing Education Scholarship, January.. Retrieved August 15, 2004, from http://www.bepress.com/ijnes/vol1/iss1/art3