The Work System Model as a Tool for Understanding the Problem in an Introductory IS Project Doncho Petkov petkovd@easternct.edu Department of Business Administration, Eastern Connecticut State University Willimantic, CT 06226, USA Olga Petkova petkovao@ccsu.edu Department of MIS, Central Connecticut State University New Britain, CT 06050, USA Abstract The work system method is a recent approach for better holistic analysis of organizational problems involving the use of information technology. We present the results from an exploratory field experiment on the impact of the work system framework as a vehicle for improvement of student understanding of a business situation involving an information system (IS) implementation problem. We provide a brief overview of the work system method, present the setup of the field experiment and discuss the results and a few directions for further research. Keywords: Work system method, Information Systems, Information Systems Education. 1. INTRODUCTION Suitable theoretical frameworks to support teaching of information systems have been a concern for information systems (IS) educators but the reported results are scarce (for further discussions see Silver et al. (1995), Alter (2006a), Ramiller (2005)). A popular approach to introduce information systems to students is the Interaction Model which focuses on the relationships between IS, their environment and the organization (see Silver et al. (1995)). A more recent set of concepts around the Work Systems Method (Alter, 2006a) can be used both for IS teaching and research from a systemic perspective and that is why we got interested in it. The work system method is an approach for understanding and analyzing systems in organizations whether or not information technology (IT) plays an essential role (Alter, 2002). Both mentioned approaches were developed and tested within postgraduate IS courses. A broader comparison of the two approaches is outside of the scope of this paper and requires further research. We will focus our attention mainly on the work system method (WSM) and its role for understanding IS related problems in an introductory undergraduate IS course investigative project. Previously WSM was explored as a teaching tool mostly at masters level by a few authors discussed briefly in Alter (2006a). Ramiller (2005) describes the use of the work systems concept in an undergraduate IS course for the purpose of understanding the notion of business processes. Alter(2006a) however, stresses that past dominance of single ideas like Total Quality Management and Business Process Reengineering are not sufficient to influence profoundly the IS field and hence the students need to understand more than processes. We consider the systemic nature of the work system method to be its most distinctive and important characteristic both theoretically and for IS practice. This view is in line with the suggestions of Alter (2004) and others who argue recently for the revival of attention to systems thinking in the IS discipline. There are indications that the work systems concepts are finding recently more attention within the IS community (see Alter, 2006b; Korpela et al., 2004; Siau et al. (2004)). A Google scholar search (as of September 29th 2006) on the use of the phrase “work system method” in refereed papers for 2005 and 2006 showed that it was used already in 12 papers in a broad range of areas like professional communication, service science, knowledge management, measurement of information systems use. The contribution of this paper is that it presents the results of the first-ever conducted controlled field experiment on the impact of the Work System Framework for better understanding of an IS related business problem. In that way it contributes to the body of knowledge related to the WSM, one of the most recent important developments in IS. Our work differs from Ramiller (2005) and the cases discussed in Alter (2006a) as we have used WSM within a foundational business undergraduate course on information management and within a controlled field experimental environment. The paper proceeds with a brief overview of the work system concept, followed by a description of a project within the course that is used to explore the impact of the WSM on student learning. The setup of the field experiment to examine the impact of the work system framework on the student understanding of the problem is discussed followed by a summary of the results and directions for further research. 2. THE WORK SYSTEM CONCEPT AND ITS ROLE FOR UNDERSTANDING INFORMATION SYSTEMS RELATED WORK PROBLEMS The work system method provides a rigorous but non-technical approach to any manager or business professional to visualize and analyze systems related problems and opportunities (Alter, 2006a). A very detailed justification for the work systems method and how to apply it to define a work system, analyze it, formulate recommendations for improvement and guide its evolution is presented in Alter (2006a). This method is more broadly applicable than techniques used to specify detailed software requirements and is designed to be more prescriptive and more powerful than domain-independent systems analysis methods such as soft system methodology (Alter, 2002). The work system method combines a static view of a current or proposed system in operation (known also as the work system framework) and a dynamic view of how a system evolves over time (the work system life cycle model). Both views have a complementary role (see Alter (2002, 2006a)). Table 1 is derived from Alter (2002) and defines basic terms underlying the work system method. Further elaboration on important definitions of related concepts is presented in Alter (2006a, 2006b). Table 1. Some Basic Terms Underlying the Work System Method (after Alter (2002)). Basic Term Definition Work system A view of work as occurring through a purposeful system Work system framework Model for organizing an initial understanding of how a particular work system operates and what it accomplishes. Organization Multiple work systems coordinated to accomplish goals that these work systems cannot accomplish individually Static view How a work system operates, based on a particular configuration Dynamic view How a work system’s configuration evolves over time Work system life cycle Process through which a specific work system is created and changes over time through planned and unplanned changes. In view of the continuing debate in the IS literature over rigor and relevance of IS research and the role of the IT artifact in it (See Benbasat and Zmud (2003) and the multiple responses to that paper published in the journal Communications of the Association of Information Systems), Alter’s proposal for work systems to replace the IT artifact as the focus of the IS discipline is a very interesting innovative idea (see Alter (2003) and Alter (2006b)). So far most of the publications related to the work system method have been related to the potential application of its concepts (e.g. see Siau et al. (2004), Casey and Brugha (2005) and others). There have been also some attempts for a critical analysis of the WSM (see Korpela et al. (2004)) or for linking it to the “work practice approach” (see Petersson (2005)). Those sources are listed here only for completeness of this brief review of WSM related research. However they are not relevant for the purposes of this paper and will not be discussed further here. We focus in this paper mainly on the work system framework or the static view of a work system, as it is shown in Figure 1 (based on Alter, 2002). That is justified by the introductory nature of the IS course of concern here and the purpose of the project that is part of it. Detailed definitions of the components of Figure 1 are presented in Alter (2002) and Alter (2006a). The interrelationships between the various elements of a work system and their boundaries are useful for generating an analysis of a specific business problem. Information systems constitute a special case of work systems in which the business processes performed and the products and services produced are devoted to information (Alter, 2002:95). Information systems exist to support other work systems and there could be some overlap with them. Various possible relationships between an IS and a work system are described in Alter (2002:96). Guidelines for analyzing work systems are presented in Alter (2002, 2006a). Alter (2006a) provides evidence that work systems ideas provide support for better understanding of business and systems problems when used with masters students who usually have a broader IT background (see Alter(2006a)). In contrast, our research explores the role of the work system framework for improvement of student understanding of an IT related work system problem in an introductory business course on IS. We decided to measure how well the students grasped the problem through assessment of student learning in a team project that is discussed below. 3. ON THE NATURE OF THE INTRODUCTORY IS PROJECT SERVING AS A MEASURE OF STUDENT PERFORMANCE The introductory IS course in which we tested the impact of work systems concepts is part of the IS program at a Northeastern State University. For historical curriculum reasons it corresponds partly to the IS2002 curriculum modules on personal productivity tools and on the foundations of IS (see Gorgone et al., 2003). The students work on two team projects during this course. The first one, which concerns this paper, is about understanding conceptual issues on Information Systems in organizations. The project goal is to show that students can analyse a particular IT problem and suggest possible ways for improvement. In order to ensure equal conditions for all students, a published case study with rich details on a business situation was given to them (see Volkoff, 2003). In the past as objects of analysis were used either small information systems in real organizations or other cases. At the completion of this project, students in the class are expected to produce a report reflecting their work on three sub-tasks: 1. Analysis of the problem situation. 2. An overview of the best practices in industry regarding relevant solutions to a similar problem using available library resources. 3. Production of recommendations on how the problem situation can be effectively improved with considerations for the necessary resources. The groups are usually of three to four students. Besides the submission of a final report, each group presents their findings to the class. The report and the presentations are evaluated through a rubric that was developed by the authors. The project assessment criteria were derived from the IS program goals and the existing project assessment literature (see Brown et al. (1997) and Wiggins (1998) : * Ability to analyze a business and the related information systems issues. * Ability to identify best business practices in similar situations through relevant publications. * Ability to interact within a team and manage time and other resources. * Ability to provide a convincing presentation. We have accepted four levels of distinction for student performance and competency: beginning, beginning, developing, accomplished and exemplary (similar to Mertler (2001)). See Petkov and Petkova (2006) for more details on the justification for the analytic rubric shown in Appendix 1. It enabled better measurement of the student performance in the project within the field experiment which is discussed in the next section. 4. A FIELD EXPERIMENT TO DETERMINE THE EFFECT OF THE WORK SYSTEM METHOD ON UNDERSTANDING OF A BUSINESS PROBLEM To illustrate the impact of using the Work System Framework in an undergraduate business course on the level of understanding of a business problem related to IS, a controlled field experiment was carried out. The problem description The problem was based on a case study on the analysis of a business situation resulting from ERP implementation difficulties (Volkoff, 2003). The case was about the complex implementation issues associated with the introduction of an enterprise information system in a large photographic supplies corporation. It illustrates the typical problems of many ERP projects: poor planning, lack of proper analysis of the existing business processes, inadequate training, project delays resulting in user dissatisfaction and business difficulties. The problem area is interesting and suitable because of the significant amount of literature documenting typical difficulties in ERP implementation projects and related best practices. These provide a rich amount of information for the students to compare with the situation described in the case of concern. Population sample and treatment The population sample involved two similar sections of approximately 25 students within a first course on Management of Business Information. The course is a compulsory component of the business core courses within their Business Administration major. The students had similar previous educational background; similar number of credit hours taken at the university and similar, very limited exposure to any IS knowledge. One section was taught in the fall semester of 2005 while the second – in the spring of 2006. Both classes were taught in two sessions a week by one of the authors following a very similar syllabus and course content. The two groups were using similar text materials on the introduction to Information Systems. The only significant difference in the treatment of the groups was that the fall 2005 section was introduced to the principles of the work system framework following Alter (2002). The project assignments had the same scope and goals. In both semesters student achievements were evaluated using the rubric shown in Appendix 1. However, only the fall 2005 students were supposed to apply the work system framework as a method for their analysis of the problem situation. Hence the rubric for the spring 2006 group was slightly modified to exclude any mentioning of work systems. It is essential to note that the Work System Framework (see Figure 1) may be used only for a static analysis of a business situation while the WSM is much broader and involves among other things tools for analysis of the problem at three levels and also for modeling the dynamic evolution of a work system (see Alter (2006a)). The latter is achieved through the Work Systems Life Cycle which was not used in our experiment because of lack of space in the undergraduate business curriculum and because of the introductory nature of this IS course. The null hypothesis was that the group which was not taught the work system framework and did not use it in the analysis of the business situation obtained results that are equal or better than those of the group which applied in its problem analysis the work system method. Discussion of the results The spring 2006 section was divided into eight project teams. Their project results comprised the data for the control group in our field experiment (see Appendix 2). These teams achieved a mean percentage grade of 94.13. The fall 2005 section which was exposed to the WSM was split in nine teams. Those had a mean grade of 96% for their projects. The detailed results from the application of the earlier discussed rubrics are shown in Appendix 3. The question was how statistically significant is this difference to prove or reject the null hypothesis of our experiment. The one tail t-test for the means of two independent samples with similar variance was applied using the statistical functions of Microsoft Excel (see also Pollard, 1977:129). The calculated t-statistic was 1.885. It was greater than the critical t-value (1.753) corresponding to 15 degrees of freedom (n1+n2-2) taking into account the number of projects in both groups. Hence we may conclude that the null hypothesis is rejected at the 0.05 level of confidence and that the use of the Work System Framework has a positive impact on improving the understanding of business problems involving ERP implementation issues. A strong aspect of our field experiment was that the case we used was a typical and complex work system that involved a difficult IT problem. It was not conducted in a sterile laboratory setting. A potential limitation that does not allow generalization of our findings is the exploratory nature of our research due to the small number of projects in both sections. Nevertheless we hope that our work will inspire future similar work and wider application of the work systems method in IS education and practice. Further statistical results from future larger field studies are needed to provide a better answer with respect to whether the influence of using the Work System Framework to improve the understanding of work systems involving IT problems is a statistically significant factor. The use of the analytic rubrics in our field experiment improved the precision of the assessment process. Thus was reduced the subjectivity in awarding the percentage grade to a project. In addition, by observing the analytic rubrics and the results (Appendices 1-3), we can easily conclude that student performance in the fall 2005 group might be indicative for the improved understanding of the business situation and the related IT problem. Since all other conditions for the two groups were the same, this was a result of the additional knowledge of the Work System Method. The group exposed to the WSM performed better than the other students along sub-criteria 1.3.”Are the conclusions in line with the factors for success or failure of IS”; 2.1.” To what extent is the review of the best business practices relevant”; and criterion 4.4., dealing with the quality of the responses to questions during presentations. Both groups had the same average evaluations on criterion 4.1.” Clarity of the explanations and conclusions”, while on one criterion, 3.1. “Have the main points to emerge from the project being picked up for discussion”, the control group had a higher average than the group applying WSM. The latter can be explained probably by other factors than the knowledge of the work systems concepts. The remaining criteria in the analytic rubric (see Appendix 1) are unrelated to whether a student has a prior knowledge of the work system framework and hence the related results are not discussed here. The above discussion shows that the students in the section that was not introduced to the WSM (spring of 2006) had performed slightly worse than their counterparts in the experimental group. The actual project reports of the latter group demonstrated a better structure of the argument and better conclusions as well – two important indicators for improving student critical thinking skills, another essential goal of the course. These observations further enrich the interpretation of the above results on the impact of using the WSM in undergraduate business programs within an introductory IS course. Our findings show that the Work Systems Framework has a positive impact on student learning on the basis of the statistical testing of the formulated hypothesis and through the comparisons of the evaluation scores along the appropriate sub-criteria in the project assessment rubrics for both groups. The student projects in the fall 2005 group demonstrate an improvement of their understanding of a business situation involving a complex IS problem as a result of applying WSM. 5. CONCLUDING REMARKS Finding better ways to explain the concepts of information systems to undergraduate business majors is a continuous challenge to IS educators. The research reported in this paper is about the first controlled field experiment (to the best of our knowledge) on the role of the Work System Framework (see Alter, 2006a) for better understanding of a business situation involving an information system. We acknowledge that the results are only of exploratory nature due to the small size of the samples of student projects in the control and the experimental group. In spite of that limitation we obtained evidence to support the claim that the use of the Work Systems Framework provides the preconditions for development of a better understanding of business and IS problems. Our statistical analysis was further supplemented by a qualitative discussion of the projects in both groups based on their reports and on the analytic rubrics we applied for assessing student performance. That analysis demonstrated again the role that the Work System Framework had on improving the grasp of the essence of a complex business situation involving the implementation of an information system. Possible directions for future research on the work system method are outlined in Alter (2004). A potential project may focus on the comparison between the impact of the work system framework and the Interaction Model (Silver et al., 1995) on student interpretations of business or IT problems in an introductory undergraduate course. Further extensions of our work, involving a larger number of projects following a similar methodology may provide evidence for broader generalizable conclusions. We may note however that our exploratory findings are in line with the conclusions from observations in other settings by Ramiller (2005) and Alter (2006a) about the positive influence of introducing work system concepts on understanding business and IS issues in IT education. 6. ACKNOWLEDGEMENTS The authors are grateful to the anonymous reviewers and the editor for their helpful comments on improving the paper. 7. REFERENCES Alter, S. (2002) “The Work Systems Model and Its Role for Understanding Information Systems and Information Systems Research”, Communication of the Association for Information Systems, 9, 90-104. Alter, S. (2003) “18 Reasons Why II-Reliant Work Systems Should Replace “The IT Artifact” As The Core Subject Matter Of The IS Field”, Communication of the Association for Information Systems, 12, 366-395. Alter, S. (2004) “Desperately Seeking Systems Thinking in the Information Systems Discipline”, Proceedings of Twenty Fifth ICIS Conference.December, 2004, 757-769. Alter S. (2006a) The Work Systems Method. Connecting People, Processes and Information Technology for Business Results. The Work Systems Press. Alter S. (2006b) “Work Systems and IT Artifacts – Does The Definition Matter”, Communication of the Association for Information Systems, 17, 299-313. Benbasat, I. and R.W. Zmud (2003). “The Identity Crisis within the IS Discipline: Defining and Communicating the Discipline’s Core Properties”, MIS Quarterly, 27(2), 183-194. Brown, G, J Bull & M Pendleburry (1997). Assessing student learning in Higher Education, Routledge, London and New York. Casey, D.   Brugha, C, (2005) “From fighting fires to building bridges: the role of metaphor in systems requirements”, Proceedings International Professional Communication Conference, 2005. IPCC 2005. available at: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1494256 Gorgone,J.T., G.B., Davis, J.S.., H.Topi, D.L Feinstein & H.E. Longenecker, (2003) “IS 2002: model curriculum and guidelines for undergraduate degree programs in Information Systems”, Database, 34(1). Korpela M, Mursu A, Soriyan A, Eerola A, Häkkinen H, Toivanen M. (2004) “Information systems research and development by activity analysis and development: dead horse or the next wave?” In: Kaplan B, Truex III DP, Wastell D, Wood-Harper AT, DeGross JI,(eds.) Information Systems Research. Relevant Theory and Informed Practice. p. 453-471. Kluwer Academic Publishers, Boston. Mertler, C.A. (2001) “Designing scoring rubrics for your classroom”, Journal of Practical Assessment, Research and Evaluation, 7(25). Petersson, J (2005) “Aren't the fundamental concepts of work systems about actions?”, Fifth Conference for the Promotion of Research in IT at New Universities and University Colleges in Sweden, 11-13 Maj, 2005, Borlänge, available at: http://www.ida.liu.se/~gorgo/div/JohanP.pdf Petkov D and Petkova O. (2006) “Development of Scoring Rubrics for Projects as an Assessment Tool Across an IS Program”, Issues in Informing Science and Information Technology, 3, 499-510, available at: http://informingscience.org/ Pollard, J. H. (1977). A Handbook of Numerical and Statistical Techniques. Cambridge University Press, Cambridge. Ramiller, N.C. (2005) “Animating The Concept Of Business Process In The Core Course In Information Systems”, Journal of Informatics Education Research, 3(2). Siau, K., Sheng H. and Nah, F. (2004) “The Value of Mobile Commerce to Customers”, Proceedings Pre ICIS SIGCHI Symposium, available at: http://sigs.aisnet.org/sighci/ Silver, M, Marks, M.L, Beath, C. (1995) “The Information Technology Interaction Model: A Foundation For The MBA Core Course”, MIS Quarterly, 19(3), 361-390. Volkoff, O. (2003) “Configuring ERP Systems: Introducing Best Practices or Hampering Flexibility”, Journal of Information Systems Education, 14(3), 313-324. Wiggins, G. (1998) Educative assessment. Designing assessments to inform and improve student performance, Jossey Bass Publishers. APPENDIX 1. Analytic Rubrics for Project Assessment in an Introductory Course on Management of Business Information Criterion Definition of rubrics on a scale of proficiency (1-4) Beginning 1 Developing 2 Accomplished 3 Exemplary 4 1. Ability to analyze a business as a work system 1.1. Correct application of the work system model Inappropriate Partial Well-defined Results analyzed 1.2. Appropriate data collection No evidence Secondary Interviews Integrated sources 1.3. Are the conclusions in line with the factors for success or failure of IS No evidence Occasional Good evidence Evidence and good analysis 2. Review of best business practices: 2.1. To what extent is the review relevant No sources Up to 2 sources At least 5 sources-no justification Sources well justified 2.2. Is there evidence of critical appraisal of other work or is it just descriptive No appraisal Occasional Attempted minor errors Critical appraisal no errors 2.3. Is there a summary linking the review to the problem on hand No attempt Somewhat Attempted Well defined 3. Recommendations of the project 3.1. Have the main points to emerge from the project being picked up for discussion? No evidence Occasional Good evidence Evidence and analysis 3.2. Is there a consideration on the resources needed for the suggested transformation and the schedule? No appraisal Occasional Attempted minor errors Well defined - no errors 3.3. Was the project developed within the time allocated for the phases? No Mostly on time On time On time and with no errors 4. Presentation 4.1. Clarity of explanation and conclusions Lacking Satisfactory Good Excellent 4.2. Visual impact of the presentation No Only text A plan is evident Excellent 4.3.Use of audio visual aids, body language Poor Satisfactory Good Excellent 4.4. Response to questions Poor Satisfactory Good Excellent APPENDIX 2. Results for the experimental (Fall 2005) group which was introduced to the principles of the Work Systems Framework Criteria Proj1 Proj2 Proj3 Proj4 Proj5 Proj6 Proj7 Proj8 Proj9 AVG 1. Ability to analyze a business as a work system 1.1. Correct application of work system model principles. 3 3 3 4 3 3 2 4 4 3.22 1.2. Appropriate data collection 4 3 3 4 3 3 2 4 3 3.22 1.3. Are the conclusions in line with the factors for success or failure of IS 4 3 3 3 2 3 2 4 3 3.00 2. Review of best business practices: 2.1. To what extent is the review relevant 4 3 3 3 3 3 2 4 3 3.11 2.2. Is there evidence of critical appraisal of other work or is it just descriptive 3 2 3 4 2 2 2 3 3 2.67 2.3. Is there a summary linking the review to the problem on hand 4 3 3 3 3 2 2 3 3 2.89 3. Recommendations of the project 3.1. Have the main points to emerge from the project being picked up for discussion? 3 3 3 4 3 3 2 3 3 3.00 3.2. Is there a consideration on the resources needed for the suggested transformation and the schedule ? 3 2 4 3 2 3 2 3 3 2.78 3.3.Was the project developed within the time allocated for the phases? 3 3 3 3 3 3 3 3 3 3.00 4. Presentation 4.1. Clarity of explanation and conclusions 3 2 4 4 3 3 2 3 3 3.00 4.2. Visual impact of the presentation 3 3 4 4 3 3 3 3 3 3.22 4.3.Use of audio visual aids, body language 4 2 4 4 3 3 3 3 3 3.22 4.4. Response to questions 3 2 3 4 3 3 2 3 3 2.89 Overall grade % 98 94 96 98 95 95 93 98 97 96.00 N.B. Please note that the last column contains the average grade of the scores corresponding to the descriptive rubrics. There is no direct mapping between the rubrics used for the evaluation of performance and the percentage grade for a project that is shown in the last row. This issue generally is an open research question and we do not attempt addressing it. DEFINITIONS OF Beginning Developing Accomplished Exemplary ACHIEVEMENT 1 2 3 4 APPENDIX 3. Results for the control (Spring 2006) group which was not introduced to the work system framework Proj1 Proj2 Proj3 Proj4 Proj5 Proj6 Proj7 Proj8 AVG 1 Ability to analyze a business IT problem 1.1. Correct application of IS principles. 3 3 3 4 2 3 3 3 3.00 1.2. Appropriate data collection 4 2 2 4 2 4 2 2 2.75 1.3. Are the conclusions in line with the factors for success or failure of IS 3 3 3 3 2 3 2 2 2.63 2. Review of best business practices: 2.1. To what extent is the review relevant 4 2 3 4 2 4 2 3 3.00 2.2. Is there evidence of critical appraisal of other work or is it just descriptive 3 2 2 3 2 3 2 2 2.38 2.3. Is there a summary linking the review to the problem on hand 4 2 3 3 3 3 2 2 2.75 3. Recommendations of the project 3.1. Have the main points to emerge from the project being picked up for discussion? 3 3 3 4 3 4 3 2 3.13 3.2. Is there a consideration on the resources needed for the suggested transformation and the schedule ? 2 2 2 2 2 2 2 2 2.00 3.3.Was the project developed within the time allocated for the phases? 3 3 3 3 3 3 2 3 2.88 4. Presentation 4.1. Clarity of explanation and conclusions 3 3 3 4 3 3 2 3 3.00 4.2. Visual impact of the presentation 3 3 4 4 2 3 3 3 3.13 4.3.Use of audio visual aids, body language 3 3 4 4 2 4 3 3 3.25 4.4. Response to questions 3 3 3 4 2 3 2 2 2.75 Overall grade % 97 94 96 96 91 95 92 92 94.13 N.B. Please note that the last column contains the average grade of the scores corresponding to the descriptive rubrics. There is no direct mapping between the rubrics used for the evaluation of performance and the percentage grade for a project that is shown in the last row. This issue generally is an open research question and we do not attempt addressing it. DEFINITIONS OF Beginning Developing Accomplished Exemplary ACHIEVEMENT 1 2 3 4