Using Blackboard’s Learning Suite in ABET – CAC Outcomes Assessment and Accreditation J. Packy Laverty laverty@rmu.edu David Wood wood@rmu.edu John Turchek turchek@rmu.edu Computer and Information Systems Robert Morris University Moon Township, Pennsylvania 15108, USA Abstract This paper presents an easy-to-use and flexible approach for data collection supporting the ABET-CAC accreditation cycle. This approach to accreditation requires program objectives, program outcomes and continuous improvement. ABET-CAC accreditation requires continuous improvement between accreditation visits. Closing the loop on Outcome-Based Assessment is a challenging and difficult problem for faculty and institutions seeking accreditation. The paper presents a description of other outcomes-based Management Systems, both internally developed and commercially available, and proposes an easy way of integrating existing ABET-CAC templates into a coherent system. The system adapts the Blackboard Learning Suite for Outcomes Assessment. The detailed assessment mechanisms currently being used in an ABET accredited program are presented, along with methods of integrating them with Blackboard allowing each faculty member to use his/her own rubrics or assessment mechanisms without demanding significant faculty time in system configuration. The procedures taken by a faculty member are explicitly presented, along with the program level procedures to track progress across terms and years. Built in Blackboard assessment instruments and analysis tools are used, even if a faculty member is not using Blackboard for course delivery. Keywords: ABET-CAC Accreditation, Outcomes Assessment, Blackboard 1. INTRODUCTION The preparation for any accreditation visit is labor-intensive and places a burden on faculty (Booth, 2006, Kaczmarczyk, 2001). Since 2008, the ABET-CAC accreditation criteria have been enhanced to have separate criteria for program objectives, program outcomes and continuous improvement. The intent of Computing Accreditation Committee (CAC) of ABET was to encourage curriculum innovation and provide flexibility within the accreditation process (Booth, 2006). Providing continuous assessment of course outcomes, which are aligned with program outcomes, is a major shift in focus from detailed criteria to satisfy check sheet requirements. (Cassel, 2005) The implication of a “continuous” assessment standard implies a “continuous” collection of assessment data, storage, analysis and improvement of the curriculum (Kaczmarczyk, 2001). ABET-CAC accreditation requires continuous improvement between accreditation visits. Closing the loop on Outcome-Based Assessment is a challenging and difficult problem for faculty and institutions seeking accreditation (Maxim, 2004). 2. OUTCOMES-BASED MANAGEMENT SYSTEMS A brief overview of outcome-based assessment may include the following tasks: 1) Specification of program outcomes, which are consistent with the institutional outcomes and ABET curriculum criteria; 2) Specification of course outcomes which are consistent with the program outcomes; 3) Assessment of course outcomes and storage of assessment data, 4) Analysis of the assessment data, and 5) Recommendations for course and curriculum improvement. When faced with these continuous labor-intensive and burdensome tasks, any CS/IS/IT faculty member “worth their salt” would want to automate and manage the outcomes assessment process. The conclusion may be, “There must be an easier way to do it.” Many introductory management courses introduce the “four functions of management”: Planning, Organizing, Leading and Controlling (Barnat, 2009). These management functions may be extrapolated to management of the outcome assessment process. The planning process may be adapted to ensure consistency between program, course and model curriculum standards. The organizing function may be adapted by assigning faculty responsibilities for specific courses, course outcomes and course outcome assessment. The controlling function should continuously monitor outcomes assessment process and ensure accountability. And the leading function should be adapted to include the mentoring and feedback to the faculty member on a continuous basis. While these extrapolations are only suggested, it does highlight some differences between the outcomes assessment process and the management of the outcomes assessment process. ABET-CAC provides various Microsoft Word templates to guide the outcomes assessment accreditation process. Exchanging documents via email attachments and storing accreditation and outcome assessment documents in a carefully designed directory structure is one approach to manage outcome assessment. ABET does not prescribe any details for an outcomes assessment management system. An outcomes management system can range from a low-cost approach, e.g., documents and email attachments stored in a shared directory structure, to an expensive and comprehensive, commercial approach like Blackboard Outcomes System. 3. INTERNALLY DEVELOPED OUTCOMES ASSESSMENT MANAGEMENT SYSTEMS A review of literature provides many examples of internally-developed, ABET-CAC outcomes assessment management systems. Pallapu (2005) converted exported Blackboard grade book assessment data and other assessment rubrics to a XML format using TrueOutcomes software. Exported Blackboard student assignment submissions were archived and stored in a traditional directory structure. The study did not address the issues relationship between program and course outcomes with assessment results. Booth (2006) proposed a database design for continuous program improvement. The database design focused on the relationships between assessment mechanisms, assessment artifacts and ABT, program and course outcomes. Using this database design, Booth, Preston, & Qu (2007) improved both a more existing outcomes assessment database (WebSubmit) and a web interface developed by Preston and Wilson (2004). The improvement mapped assignment elements to course and program outcomes using a rubric assessment instrument. Using a Microsoft Access Database, Boff, DeLorenzo, Kovalchick, & Sible (2009) designed, developed and implemented the CISaccred accreditation tracking tool. While the database design improved the storage of outcome assessment data, the ability to use Access forms and reporting tools provided for the flexible analysis of assessment data. The study recognized the limitations encountered when entering assessment data and knowledge requirements of the faculty members necessary to use the CISaccred accreditation tracking tool. Owen, Scales, & Leonard (1999) developed a system designed to track course outcomes and assessment tools to program outcomes in an engineering program. Tungare, et al. (2007) proposed and designed a syllabus storage system that linked course outcomes to programs outcomes using a web interface. Other efforts are underway to design and develop other outcomes assessment management systems. (Konsky et al., 2006: Poger, Schiaffino, & Ricardo, 2005) 4. COMMERCIAL OUTCOMES ASSESSMENT MANAGEMENT SYSTEMS Neither the review of Assessment Systems (Assessment Systems, 2009) nor that by Rediker Software (2009) provides a comprehensive outcomes assessment management solution. Blackboard, which dominates the eLearning industry, (Jashcik, 2007) introduced the Blackboard Outcomes System to provide institutional, program, course and student level assessment. While Blackboard’s Outcomes System does not require the use of Blackboard’s Learning Suite, it does leverage the student data gathering features to support outcome assessment. While the Blackboard Outcomes System is quite comprehensive, it may be deemed too expensive. 5. ADAPTING THE BLACKBOARD LEARNING SUITE FOR OUTCOMES ASSESSMENT Many universities deliver course content using a Blackboard Course. Many faculty members have experience using Blackboard. And even for the most challenged student, a Blackboard course shell is easy to use. The emphasis of outcomes assessment in ABET-CAC accreditation has presented new challenges. To address some of these changes the Computer Information Systems (CIS) department tested the feasibility of using Blackboard’s Learning Suite as an outcomes assessment and outcomes assessment management tool. In preparation for the ABET-CAC accreditation visit, the CIS department: a) used a Blackboard course shell as a outcomes assessment management system, b) used several Blackboard assessment instruments and analysis tools to provide course outcomes assessment data, and c) provided a Blackboard-administered Senior test to assess the BS-CIS and BS-IS programs. 6. BLACKBOARD COURSE SHELL AND OUTCOMES ASSESSMENT MANAGEMENT As an alternative to the previously cited outcomes assessment management systems, a standard Blackboard Learning Course Shell was designed to support an ABET-modeled outcomes assessment management system. Using a Blackboard Course Shell to manage ABET’s outcomes-based assessment does not require a faculty member to use a Blackboard Course Shell to deliver a course. For those programs seeking ABET accreditation who are already using the Blackboard Learning Suite, using a Blackboard Course Shell may provide a low-cost, easy-to-use, flexible and accountable approach that may document the outcomes assessment process over multiple semesters. 6.1 Design of ABET Blackboard Master Shell Each Blackboard Course Shell displays a pre-defined Course Menu. The ABET Outcomes Assessment Shell may customize the Course Menus to the following: Course Document Submissions, Course Documentation, University Documentation, Program Documentation, ABET Resources and Templates, Outcomes Assessment Resources, ABET Self-Study Report, and Internal and External Program Assessment Measures (See Appendix). The Course Document Submission menu will contain Blackboard Assignments which will permit a faculty member to submit ABET Outcomes Assessment Documentation, e.g., ABET Course Description, Teaching Syllabus, ABET Assessment Plan, ABET Outcomes Assessment Mechanisms, ABET Outcomes Assessment Results, and Assessment Recommendations. The advantages of using a Blackboard Assignment is that the submitted documentation can be monitored, reviewed and returned to the faculty member with recommendations for revision. Blackboard’s Learning Suite Version 9 will provide an audit trail for multiple submissions and reviews when the option is selected. However, it is recommended that Blackboard Assignment Grade be displayed as a check mark instead of a percentage or letter grade to avoid offending sensitive faculty members. Blackboard provides an Early Warning System that will provide a report listing assignments (outcomes assessment requirements) not submitted within an acceptable timeframe or has not been submitted in an acceptable format. To use the Blackboard Early System the ABET submission an assignment must have either a due date or assigned points. 6.2 ABET Outcomes Assessment Shell Usernames Using your institution’s Blackboard faculty name will not be adequate to provide accountability for ABET document submissions. A faculty member may be responsible to provide documentation for many courses and a course may have many course sections staffed by different faculty members. To maximize a Blackboard’s course shell to provide accountability for ABET submissions, a Blackboard user name with student access rights needs to be created for each course section. For example, consider the course with the name of INFS2210 and Section B. A Blackboard username of “INFS2210B” with the last name of “INFS2210” and first name of “B” would be acceptable for this design. If a faculty member was responsible for three different course sections for the current semester, then he/she would be assigned three different Blackboard user accounts with a default password. When the faculty member logons on to Blackboard using the username of INFS2210B, the faculty member will only see the ABET course shells, not their course delivery shells that they are currently using for course delivery. While the Blackboard content management system can restrict access to folders, items, assignments and links by username, this level of security does not seem necessary. The course/section username simplifies the process of monitoring documentation submission. It is recommended that Blackboard faculty usernames only be used to access ABET outcomes assessment shell when administrative need dictates the access. 6.3 ABET Course Documentation Menu The Course Documentation Menu contains folders for each course, e.g., INSF1010 Principles of Information, INFS2210 Operating Systems, etc. Each course subfolder contains a link to the online course description for consistency, the department’s ABET Course Description, and subfolders for each section for the current semester and sample documentation for the course. (See Appendix.) ABET Shell user (student) accounts will not be able to add content to this folders. Only a Blackboard faculty user may add, update or delete content. The Blackboard Shell is rather light-weight as compared to other content management systems. There are only two levels of security: the course/section user (a Blackboard student user) and a Blackboard faculty user (who can do it all). It is recommended to document in the Blackboard object’s description for the individual and date of the last change, e.g., Last Updated by Smith of 7/7/2009 or Last Retrieved by Smith on 7/7/2009. The design of the relationship between the Course Document Submission Menu and Course Documentation Menu needs to be considered. The design assumes that documentation is submitted by a course/section username. This documentation is then monitored, reviewed, accepted and downloaded by faculty user account (assigned to be an ABET documentation administrator). In order to make the documentation available to other users a faculty user account then must upload the accepted documentation to course/section and documentation item. While the process of submitting and uploading documentation is as easy as adding an attachment to an email message, the uploading process may be time consuming for one faculty member. One possible solution is add all full-time faculty members to the ABET shell and assigned responsibilities for uploaded accepted documentation to assigned courses for all sections. Throughout this design external HTML links will be used to display online course descriptions, program outcomes, university objectives and ABET – CAC Accreditation Criteria. It is important that ABET course descriptions, teaching syllabuses and program outcomes are consistent with online course catalogs. Likewise, ABET program and institutional outcomes should be consistent with institutional web sites and program catalogs. 6.4 Other ABET Outcomes Shell Menus It is important to organize the Blackboard shell for other ABET accreditation and outcomes assessment requirements. Course Outcomes Assessment is only a part of a ABET Self study document and accreditation visit. ABET accreditation also requires: a) program objectives that are consistent with institutional objectives, b)program outcomes that are consistent with program objectives, and c) course outcomes that are consistent with program outcomes. Not all course outcomes are required to be traceable to program outcomes and ABET accreditation standards. ABET does recognize the importance of flexibility. It is important to share and document University objectives, program objectives, program outcomes and ABET accreditation criteria with all faculty members. Program outcomes assessment instruments, analysis and recommendations also need to be documented. The CIS department used the following program assessment instruments: a) Senior Exam (a Blackboard Test), b) Alumni Survey, c) Board of Visitors Survey, Senior Exit Survey, and e) an Employer Survey. The selection of program outcomes assessment instruments may vary significantly among institutions. 7. MULTIPLE SEMESTER SUPPORT ABET Outcomes Assessment is a continuous process. This means that once a semester outcomes assessments has been analyzed, documented and the assessment process closed, then a new semester begins. It is recommended to create a Master ABET Outcomes Assessment Shell. Some of the Course Outcome Assessment Documentation may be blank in the Master Shell. Prior to the start of each semester the Blackboard system administration will be required to create a new “blank” Blackboard Outcomes Assessment course shell for the new semester. The contents of the Master ABET course shell will then be copied into the new semester shell. Most institutions will not have ability to automatically enroll course/section users. A faculty user assigned as the ABET outcomes assessment administrator can add course/section users applicable to the current semester. If INFS2210 Section B is not scheduled for the current semester, then the username INFS2210B should not be added to the current semester shell. The design of the master ABET shell should include all possible courses and all possible section whenever possible. Some may decide to hide or delete un-scheduled courses and sections from the current semester shell. This is not really necessary unless it is an important objective to be consistent with the current course schedule. It is the selection of course/section users that controls the submission, monitoring and feedback process. 8. BLACKBOARD ASSESSMENT INSTRUMENTS AND ANALYSIS TOOLS While it is not necessary for a faculty member to use a Blackboard course shell to deliver a course, several CIS faculty members use Blackboard course shells even to supplement on-ground courses. A Blackboard course shell provides many sources of ABET course assessment data. Detailed, summarized, selective or aggregate time-on-task and access data is readily available. The most important assessment tools are Blackboard tests and assignments. Blackboard tests support a variety of questions formats, e.g., multiple choice, true/false, short-answers, essays, matching, etc. While one can edit questions directly in Blackboard, it is much easier to import test questions from test banks, other course shells or from a Word document using a third-party product like Respondus. Each Blackboard test may be automatically graded and entered into the Blackboard Grade Book. Each Blackboard Grade Book item may be weighted to be consistent with the course syllabus. Security is always a concern within any online test. Blackboard tests can be randomly-ordered, randomly-selected from a question pool, timed, and password protected. Each feature has its own security limitations. If security is a major concerned that the Blackboard test should be administered under faculty supervision. It should be noted that the security feature of randomly-selecting questions from a question pool will limit Blackboard’s ability to perform some types of question item analysis. Perhaps the greatest contribution of using a Blackboard test as an assessment mechanism is the built-in test analysis capabilities and export features of the test results. Besides providing popular test summary statistics, Blackboard will provide a test item analysis report. Blackboard can also export the test results by item or user to an Excel workbook. Once in a saved in an Excel workbook the data becomes a candidate for Excel’s Data and Pivot Tables. There is one limitation on Blackboard Test Analysis. A given Blackboard test may contain questions that assesses more than one course outcome. While Blackboard associates a category attribute with each question item which may be associated with a course outcome, currently the test item category can not be used in Blackboard Test analysis or export. Restructuring a test’s content to be consistent with a specific course outcome may be useful from an assessment and analysis perspective, but it may be too limiting for the design of a course. Automatically assigning test items to a course outcome is a limitation. Individual test items can not be automatically traced to individual course outcomes. Any type of assignment or course project may be distributed and graded using a Blackboard assignment. Since a Blackboard assignment is connected to the Blackboard grade book it may be weighted and included in the calculation of the final grade. Many instructors use a rubric assessment instrument to grade and provide feedback to an assignment or project. The Blackboard’s Learning Suite does not directly provide any rubric grading tool. Any rubric document or worksheet may be attached to the assignment and returned to the student. The instructor may enter a manual grade for each Blackboard Assignment. Rubric grading tools may be used to provide qualitative measurement and feedback. However, it is recommended that for the purposes of outcomes assessment that the instructor provides a rubric measurement that uses an Excel Workbook. By properly designing an Excel assignment rubric workbook, the easy-to-use Excel’s Consolidation tool may be used to summarize multiple student rubrics into one consolidate workbook using a variety of mathematical operations. After saving the Excel Rubric workbook locally, the instructor attaches the workbook to the Blackboard assignment similar to attaching any document to an email message. 8.1 Blackboard Senior Test Given the analysis features and administrative convenience of a Blackboard test a Senior Pilot Test was developed to assess the CIS program outcomes. The Senior Pilot test was administered to a selected group of seniors online. The test was timed, password protected and randomly ordered. CIS instructors submitted five to ten candidate multiple choice questions for each core course using a Word document. Since there was no way to control the programming language for each student, it was decided not to assess an individual programming language in this pilot test. The submitted questions were then reviewed by a committee of CIS faculty members. Fifty questions were selected and imported into Blackboard using Respondus. The results of test were analyzed by Blackboard. No plan was developed to gather student background data using Blackboard. Data concerning whether a student took the CIS core course at the university or transferred the credits was considered to be important and was initially overlooked in the pilot test. The test data was exported to an Excel workbook and merged with other sources of student background data for further analysis. Using the Blackboard Survey instrument may be an option to gather student background data in the next revision or the Blackboard Senior test. The provision for better guidance to instructors in the submission of candidate questions should also be improved. 9. CONCLUSIONS Gloria Rodgers (2003) provided the following advice for implementing Outcomes-based Assessment. “There is no one right way to do program assessment, and no two programs or institutions are alike. … When involved in program assessment, it is important to remember that there is neither time nor resources to do everything. “This paper agrees with this advice. Many faculty members and programs may be comfortable with a template-email-directory approach to managing outcome assessment. This solution may be acceptable for one-time accreditation visits, it remains to be seen if this opinion remains when outcomes assessment is truly implemented as vehicle for continuous improvement. The efforts to develop an automated system to managed outcomes assessment by the previously cited individuals and teams are impressive. But, one must ask whether the effort was worth the value of the time to develop the assessment management system. Only Boff et al. (2009) provided insight of the limitations of their assessment tracking tool. One would expect that these early prototypes would encounter limitations such as: ease-of-use, comprehensibility, monitoring, feedback, data entry and flexibility to adapt to new assessment instruments and delivery systems. However, the important question may be to ask whether the creative effort and time expended to create these solutions may have been better spent assessing and improving their own courses and curriculum? Commercial Outcome Assessment Management systems, like Blackboard’s, will increasingly come available. Larger ABET-CAC accredited programs may be able to justify the cost of a commercial system. But, questions remain whether these systems will be flexible to support the needs of individual faculty members. Training, support and maintenance also need to be considered. For those institutions who are currently using the Blackboard Learning Suite, adapting a course shell to manage the ABET-CAC Outcomes-based assessment process may be a viable alternative. Controlling assessment documentation submission, organization of the assessment process and display, the flexibility of design and content, and leveraging existing assessment tools and analysis are some advantages. Automating consistency between program outcomes, course outcomes and assessment results is limited. Importing, entering, or analyzing not Blackboard assessment data faces the same limitations as other alternatives. And while all faculty members can attach documents to email, there will be those who will never want to use Blackboard. Like ABET-CAC Outcomes-based assessment, evaluating the use of a Blackboard course shell as an assessment management system will also be a continuous process. 10. REFERENCES Assessment Systems (2009) Assessment related software. Retrieved July 15, 2009. Available: http://www.assess.com. Barnat, R. (2009). “Introduction to Management”. Retrieved 5/14/2009 from: http://www.introduction-to-management.24xls.com/en100 Boff, G., G. DeLorenzo, L. Kovalchick, & P. Sible (2009). “Leveraging Academic Resources in the ABET Accreditation Process: A Case from California University of Pennsylvania.” Information Systems Education Journal, 7 (79). Booth, L. (2006). "A Database to Promote Continuous Program Improvement." Proceedings of the 7th Conference on Information Technology Education SIGITE'06. 83-88. Booth, L., J. Preston, J. Qu, (2007). "Continuous Program Improvement: A Project to Automate Record-keeping for Accreditation". Proceedings of the 8th ACM SIGITE Conference on Information Technology Education SIGITE'07. 155-159. Cassel, L. ( 2005). "Outcomes-based Computer Science Education". Position Statement of Panel Session (SIGCSE'05). Retrieved 62/ 2009 from: http://delivery.acm.org/10.1145/1050000/1047437/p260-cooper.pdf?key1=1047437&key2=3485888421&coll=GUIDE&dl=GUIDE&CFID=46943245&CFTOKEN=48308183 Jaschik, S. (2007). “The New Assessment Market”. Retrieved 5/15/2009 from http://www.insidehighered.com/news/2007/01/17/blackboard Kaczmarczyk, L. (2001). “Accreditation and student assessment in distance education: Why we need to pay attention." in 6th Annual Conference of Innovation and Technology in Computer Science Education, ACM Press, 113-116. Konsky, B., A. Loh, M. Robey, S. Gribble, J. Ivins and D. Cooper (2006) “The Benefit of Information Technology in Managing Outcomes Focused CurriculumDevelopment Across Related Degree Programs.” Proceedings of ACE’06, January 16-19, pp. 235-242. Maxim, B. (2004). "Closing the loop: assessment and accreditation." Journal of Computing Sciences in College, 20(1), 7-18. Owen, C., K. Scales, & M. Leonard (1999). “Preparing for Program Accreditation Review under ABET Engineering Criteria 2000: Creating a Database of Outcomes and outcome Indicators for a Variety of Engineering Programs.” Journal of Engineering Education, 88(3), 255-259. Pallapu, S. (2005). "Automating Outcome Based Assessment". Retrieved 7/12/2009: http://technology.asu.edu/files/documents/tradeshow/May04/SuseelPallapu.pdf Poger, S., R. Schiaffino and C. Ricardo (2005) “A Software Development Project: A Student-Written Assessment System.” Journal of Computing Sciences, 20(5), 229-238. Preston, J. and Wilson, D. (2004). "Achieving Consistency and Communication in Online Education via an Online Grading System." Proceedings of Teaching Online in Higher Education (TOHE). Rediker Software, Inc. (2009) “Assessment related software.” Retrieved July 15, 2009. Available:http://www.rediker.com. Rodgers, G. (2003). "Lessons Learned:Things I Wish I had Known . . ." Retrieved 6/14/2009 from: http://abet.org/Linked%20Documents-UPDATE/Assessment/Assessment%20Tips5.pdf Tungare, M., X. Yu, W. Cameron, G. Teng, M. PerezQuinones, L. Cassel, W. Fan, E. Fox (2007). "Towards a Syllabus Repository for Computer Science Courses". http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.101.2797&rep=rep1&type=pdf Appendix ABET Outcomes Assessment Master Shell Design Course Menus Folders and Objects Subfolders and Objects Comments Announcements Standard Blackboard Announcements None   Course Document Submissions * ABET Course Description (A) * Teaching Syllabus (A) * ABET Assessment Plan (A) * ABET Outcomes Assessment Mechanisms (A) * ABET Outcomes Assessment Results (A) * Recommendations (A) None   All objects are Blackboard Assignments. Faculty can submit documentation using an Assignment. The submission process can be monitored and reviewed in the Blackboard Grade Book   Course Documentation * INFS1010 Principles of Info (F) * INFS2210 Programming Logic (F) * INFS2210 Operating System (F) * Online Course Description (L) * Approved ABET Course Description (I) * Sample Course Section (F) * INFS Section A (F) * INFS Section B (F) Each course folder contains an approved ABET Course Description, Online Course Description and a Subfolder for each Course Section. Each Course Section folder contains approved documentation submitted by a faculty member for the current semester and course section University Documentation Online University Links (L) Custom University Items (I)    University/School Objectives and Outcomes  Program Documentation Online Program Links (L) Custom Program Items (I )   ABET Program Outcomes  ABET Resources Useful ABET Links (L)    Templates, examples  Outcomes Assessment Resources Useful Outcomes Assessment Links (L) Custom Resources (I)    Research document and PDFs, rubrics, tutorials, Power Points  ABET Self-Study Report ABET Self -Study Report (I)   Copy of the most current ABET Self Study Report  - May contain subfolders for each section Internal and External Program Assessment Measures * Senior Exam Overview and Assessment (I or F) * Alumni Survey and Assessment (I or F) * Board of Visitors Report (I or F) * Employer Survey and Assessment (I or F) Discussion Board Standard Blackboard Discussion    Provide Discussion Forums for ABET Issues  (A) – assignment (I) Item: Text, Document or Presentation (F) Subfolder (L) External HTML Link