The Role of Assessment in Accreditation: A Case Study for an MIS Department Luann Stemler lstemler@millikin.edu Cheryl Chamblin cchambin@millikin.edu Tabor School of Business, Millikin University Decatur, Illinois 62522 USA Abstract In preparation for Millikin University accreditation assessment committees were formed at the university and college levels to facilitate the process. The learning goals/objectives of each college and department need to be soundly rooted in the university’s mission statement. This paper shares some of the University’s and department’s experiences, and outlines a procedure for developing an assessment strategy to achieve accreditation and to improve the program. Keywords: assessment, objectives, learning goals, accreditation 1. INTRODUCTION Over the past years accrediting institutions have placed a significant emphasis on assessment as critical to the University’s educational mission. According to Acharya (2003), assessments should answer the following questions: (1) What do we want the students to learn? (2) Why do we want them to learn it? (3) How can we help them to learn it? (4) How do we know what they have learned? Blaha and Murphy (2001), describe several principles that should be followed when creating assessment plans. It is advised to determine objectives before selecting measures to assess student learning. They also suggest that assessment tasks needs to be part of normal departmental operations. Blaha and Murphy (2001), recommend the following steps: * Determine the target * Develop goals/objectives * Determine how and where objectives will be assessed. * Determine how the assessment information will be reviewed and used to make program improvements. Further information on assessment as it relates to accreditation can be found at the Higher Learning Commission of the North Central Association, www.ncahigherlearningcommission.org/ The assessment process can be viewed as a tree diagram. The University mission and goals are the root of the process. The school or college goals develop from those. Finally, departmental goals grow from the college goals. This paper will demonstrate how the Management Information Systems Department of the Tabor School of Business at Millikin University developed its assessment plan using this process. Millikin University is a small private comprehensive university with approximately 2300 students. The structure consists of four colleges/schools: Arts and Sciences, Fine Arts, Nursing and the Tabor School of Business. The College of Arts and Sciences is further divided into three divisions: Natural Science and Mathematics, Humanities, and Social Sciences. The University also includes an adult degree completion program (PACE). The Management Information Systems (MIS) department is housed in the Tabor School of Business. Currently the MIS department offers nine courses for their majors. Additionally the department teaches two introductory courses required for all Tabor students. Finally, several of the MIS department courses are cross-listed with those in the Computer Science (CS) department and are taken by those majors. The department has two full-time tenure-line faculty, who teach all of the information system courses. 2. THE ASSESSMENT PROCESS AT MILLIKIN The assessment process began in the spring of 2004 as the University began preparations for its North Central Association accreditation visit scheduled for the fall of 2006. One of the areas that was somewhat problematic at the last visit was assessment. Thus the Millikin accreditation committee decided to make assessment one of the major focal points of the report. A University Assessment Committee (UAC) was formed with one representative from each of the three divisions of Arts and Sciences, a representative from the School of Music (located in the College of Fine Arts), one representative each from Tabor, Nursing and Fine Arts, a representative from the Education department, a representative from PACE and the Coordinator of Institutional Research and Assessment. The UAC began meeting in the spring of 2004 to begin the planning process The initial task of the UAC was to facilitate a one-day workshop designed to help faculty begin the assessment process. After this workshop each representative of the UAC selected a team of faculty to assist in the assessment plans for their respective units. In the Tabor School the team consisted of faculty from Accounting, Management and MIS. Step 1: Developing the Tabor School Objectives The University had recently adopted a new mission statement as follows: To Deliver on the Promise of Education At Millikin, we prepare the student for * professional success; * democratic citizenship in a diverse and dynamic global environment; * a personal life of meaning and value. Each unit team examined their mission statement and current learning objectives to determine if they were still relevant in light of the University’s changes. The mission of the Tabor School is to deliver an entrepreneurially-focused integrated educational foundation for graduates’ professional and personal achievement, and will be a learning partner valued by the business community. The committee found that the mission statement was still relevant and suggested no changes. Crouch & Schwartzman (2003) suggest objectives should be kept simple and be limited in number reflecting the strength and/or uniqueness of the program. The Tabor school originally had fourteen objectives, many which could not be measured. Two such objectives were “Maturity and self-confidence” and “A personal philosophy and a framework of professional ethics”. While these were attributes that the Tabor School desired of its graduates, the extent to which these objectives were achieved could not be assessed. A second criterion for developing a learning goal was how it worked to achieve the mission of the University and the mission of the Tabor School. When the assessment team finished this stage of their process, they had developed a set of six learning goals. Tabor Learning Goals: (1) Students will demonstrate competent application of business theory and concept to practical situations in communities outside the formal classroom. (2) Students will communicate facts and ideas in written and verbal formats using language, grammar, and organizational skills appropriate to business situations. (3) Students will be actively engaged citizens using their education and skills to serve the community. (4) Students will demonstrate a strong sense of individual, leader, and team roles and responsibilities. (5) Students will discover the global nature of business, including immersion or familiarization with diverse cultures and cultural environments. (6) Students will apply those problem-solving and decision-making skills expected of entry-level business professionals. Step 2: Assessment of the Tabor School Core Once the new learning goals were adopted by the faculty in the Tabor School, the team began examining the business core courses to see where the objectives could be assessed. The assessment process is only valuable if it is completed. Blaha & Murphy (2004) note that responsibility for assessment tasks must be clearly defined and, whenever possible, be incorporated into regular departmental policy. Also, Maxim (2004) advised that record-keeping be kept to a necessary minimum and use the smallest amount of class time that was practical to still acquire meaningful assessment data. So, while many of the courses in the business core met one or more of the learning goals, the committee decided to assess the objectives at few logical points. This was done for several reasons. First, faculty may have difficulty fitting assessment activities into an already busy schedule. Second, the analysis of assessment data becomes more time-intensive when too many observations are gathered. Thus it may be difficult to find faculty willing to serve on departmental assessment teams. For example, the committee decided to assess written communication (objective 2) at the sophomore level in the MS 240 Organizational Information Systems course (a core business course). Students in this course analyze spreadsheet results and project the best way for a company to proceed. The paper would be comparable to one a businessperson would use in a consulting engagement. Other sophomore courses also use writing assignments, but the team believed this would be the most appropriate one to use for assessment. Step 3: Assessment of the Majors After the core assessment plan was completed, the team began to examine each of the majors in the Tabor School. Each major would develop learning objectives that were linked to Tabor School objectives. The process from this point followed the same procedure as in the first two steps moving from the University mission statement to the Tabor objectives. In keeping with the philosophy of developing easily assessable goals, the MIS faculty developed five central objectives for the MIS major. The assessment team believed that since the Tabor goals were implied throughout the entire business curriculum, they should not be repeated as part of the departmental or major learning goals. For example, one of the Tabor learning goals is that “students will apply those problem-solving and decision-making skills expected of entry-level business professionals.” (objective #6) While all MIS courses focus on these skills, the MIS department will assess this objective a Tabor goal rather than an MIS major goal. MIS Objectives: Using the philosophy above, the MIS faculty developed the following five learning goals: (1) Students will demonstrate their database competencies in designing and building a relational database of moderate complexity through database management system software. (2) Students will use the System Life Cycle to develop a system design through the use of case modeling and system modeling that will solve a moderately complex business problem; then develop a program from the modeling specifications in any program language. (3) Students will be able to understand the basic concepts of security with regards to operating systems and access control. (4) Students will apply networking and telecommunication knowledge to specific applications and situations, such as the Internet, intranet, and e-commerce. (5) Students will organize and write user documentation, system documentation and IS reports in a language and style appropriate to the profession. Once the departmental/major objectives were created, the course objectives could be completed. As an example, in the MS370 Database course the objectives are listed below. At the end of each objective is a reference to the MIS major or Tabor School learning goals. Some courses may also have more specific course-only objectives. MS370 Database Application Development Course Objectives: A) Students will demonstrate their database competencies in designing and building a relational database of moderate complexity through database management system software. (MIS #1) B) Organize and write user documentation, system documentation and IS reports in a language and style appropriate to the profession. (MIS #5) C) Students will demonstrate competent application of business theory and concept to practical situations in communities outside the formal classroom. (Tabor #1) D) Students will communicate facts and ideas in written and verbal formats using language, grammar, and organizational skills appropriate to business situations. (Tabor #2) E) Students will demonstrate a strong sense of individual, leader, and team roles and responsibilities. (Tabor #4) F) Students will apply those problem-solving and decision-making skills expected of entry-level business professionals. (Tabor #6) Note that the first two MS370 objectives are MIS departmental objectives and the last four reflect Tabor objectives. Thus the students see Tabor objectives repeated in the majors as well as the core courses. Step 4: Type of Instruments, Data Collection and Evaluation Once the objectives have been developed and the data collection points have been established, the departments must decided on the instruments that will be used to provide the data. When the team began selecting assessment points, it realized that there should be a common set of criteria on which to judge writing, oral communication and team skills. Thus, the Tabor School faculty developed common rubrics for writing (appendix A), oral communications (appendix B), and team skills (appendix C), that the students will see in nearly every class. The team also developed (with the approval of the faculty) a common format for the first page of every syllabus. This page includes the Tabor objectives and departmental objectives for the course. The rubrics and syllabi format (appendix D) are included in appendix. At the MIS departmental level, group projects will be used as the assessment tool in three required courses. The MS370 Database Application Development course focuses on the database design and development. In this course students will have a real client who has a database need. The project is ongoing throughout the semester designing and building this database for the client’s application. In the two-semester MS 321/322 System Analysis and Design course the same pedagogy is applied. In the first semester the team designs a system for a client and then in the second semester the team develops a fully functioning system. The group rubric is used at the end of each semester to evaluate the group and individual’s work. In the MS370 Database course a series of 5 Milestones/Rubrics were created for grading purposes. Specific areas of Milestone #2, #4 and #5 were identified to be used for the assessment points. The milestones can be found in the appendix E with the appropriate MS370 objective label in the left-hand column of the page. In Milestone #2, MS370 objectives A, B, and D will be measured. In Milestone #4 the objectives A and F will be measured. Milestone #5 is typically graded by the user, so consequently objective C is measured by the client themselves. To measure objective E the team rubric will be used. The rubric is in appendix E. The data collected will be measured using a green light, yellow light, and red light scale measuring the level of achievement desired by the MIS faculty. The rubrics for the process are as follows: Green an acceptable level or clearly heading in the right direction and not requiring any immediate change in course of action. Continuing support should be provided. Yellow not an acceptable level; either improving, but not as quickly as desired or declining slightly. Strategies and approaches should be reviewed and appropriate adjustments taken to reach an acceptable level or desired rate of improvement. Red our current status or direction of change is unacceptable. Immediate, high priority actions should be taken to address this area. A Tabor Assessment Standards guide can be found in appendix F for the rubrics and measurement of the Tabor objectives. The MIS objectives for the MS370 project will be measured using the following criteria. The project must obtain an 85% or above to receive a green light, 75% - 84% to receive a yellow light; projects earning below 75% will receive a red light. Student criteria G (good) score or above Green M (mediocre) score Yellow