A Survey Analysis of the Perceptions of IS Educators and IS Professionals Appertaining to the Course Content of an Undergraduate Information Systems Capstone Course Jack Russell, Ph.D., CCP jrussell@nsula.edu Barbara Russell, M.S., CCP brussell@nsula.edu Computer Information Systems Northwestern State University 201 Russell Hall Natchitoches, Louisiana 71597, USA Abstract The purpose of this paper is to summarize and evaluate the content of a proposed undergraduate IS capstone course by the authors. A statistical analysis of a survey regarding the behavioral objectives of the proposed IS capstone course is used to augment the evaluation aspect of the capstone course content. The IS 2002 Model Curriculum and Guidelines for Undergraduate Programs was used as a guideline in developing the capstone objectives. A review of the proposed IS capstone course is presented. The IS capstone course behavioral outcomes are summarized. Based on the behavioral outcomes a survey instrument was prepared that would address each of the outcomes. The purpose of the survey questionnaire was to acquire the perceptions of both IS educators and IS professionals who responded to the survey. From eleven research questions a set of eight (8) research hypotheses were listed. The research hypotheses are aimed at learning if IS educators and IS professionals agree or disagree on key course objectives identified for the capstone course. Demographic results are presented along with separate statistical tests for each of the null hypotheses. Both Chi Square tests for independent proportions for 2X2 tables and Two-Sample T-Tests where appropriate are performed to determine if survey item means for each group were significantly different and to also examine the magnitude of the means to determine if the respondents strongly agreed with the survey item that maps to a stated course outcome Keywords: IS capstone course, IS 2002 Model Curriculum, business modeling, data collection interviews, soft skills, object-oriented analysis (OOA), Unified Modeling Language (UML), structured analysis, and systems proposal. 1. INTRODUCTION The purpose of this paper is to summarize and evaluate the content of a proposed undergraduate IS capstone course. The evaluation component of this study consists of a statistical analysis of a survey appertaining to the behavioral objectives of a proposed IS capstone course. The survey was administered to both IS educators and IS professionals to gain a balanced view of the elements that should be taught in the capstone course. Specifically, the authors were interested in understanding how the two groups (IS Educators and IS Professionals) agreed or disagreed with key course content. The content of the IS undergraduate capstone course remains to be a point of argument among IS educators and IS professionals in the field. Research on IS capstone courses (Davis et al 2002) (Russell and Russell 2006) and (Schatzberg 2003) suggest a strong balance of competencies among soft skills, teaming, and software design and development. The authors have reviewed the IS 2002 Model Curriculum and Guidelines for Undergraduate Programs in an attempt to incorporate the proper balance of competencies related to project management, soft skills, business modeling, systems design, implementation and testing. The authors believe that IS programs should be actively involved in insuring that the IS graduate is not only a competent technical graduate but one who also understands and is capable of performing well within each phase of a project. Students who tend to excel are those who are well trained in project management and business modeling, and are often the ones who are also involved in making presentations and helping author feasibility analysis reports and project proposals (Strader 2004). The argument continues as to whether IS graduates should learn structured methods as well as object-oriented methods and UML. Various studies suggest that IS graduates should have a strong understanding of both structured analysis and object-oriented analysis (Russell and Russell 2006). Various studies continue to suggest that the capstone course should emphasize teaming, project interviewing, project presentations and reporting along with the systems development technical skills (McGinnis 2001). Although much as been written regarding the content of the undergraduate IS capstone course very little analysis has been done to determine the degree to which IS professionals agree or disagree with IS educators on various proposed learning objectives within the capstone course. 2. PROPOSED TEACHING MODEL FOR A CAPSTONE COURSE IN INFORMATION SYSTEMS The IS 2002-10 Course Objectives used as a Framework for a Capstone Course A proposed teaching model for a capstone IS Course (Russell and Russell 2006) describes proposed course content and a strategy for teaching a capstone course in the IS curriculum that addresses both cognitive and affective learning experiences. The IS 2002 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems (Davis et al 2002) was used as a guide in architecting the capstone course. A greater emphasis was placed on the soft skills component, but many of the IS 2002-10 Project Management and Practice course learning outcomes were included in the proposed capstone course. The need for soft skills in the capstone course as well as across the curriculum are emphasized and discussed in various studies (Tastle and Dundum 2000; Russell et al 2005; Schatzberg 2003). The learning outcomes includes the management of the systems life cycle; cost-benefit analysis; requirements determination; systems design; systems implementation; system and database integration; project tracking; metrics; expectations of managers, clients, team members; and reporting and presentation techniques. IS Capstone Course Outcomes The IS capstone course maps to virtually all the learning outcomes outlined in the IS 2002 Model Curriculum previously discussed. The learning outcomes are listed below. 1. Complete a business modeling review packet of assignments. 2. Participate as a team member or project leader in a semester project. 3. Participate in a “mock” data collection interview based on the semester project narrative. 4. Complete a feasibility analysis and report complete with payback analysis. 5. Complete the ERD for the semester project’s business narrative using Visible Analyst (VA). 6. Complete a Class Diagram for the semester project’s business narrative using VA. 7. Complete the Decomposition Diagram and Data Flow Diagrams using VA. 8. Complete a Use Case Diagram, a sequence diagram and state chart diagram using VA. 9. Compose and Present the Proposal to Perform Systems Design. 10. Design the business system. The deliverables will include: graphical user interface design, navigation design, database design and program design. 11. Program the sub-system from the design elements and test the sub-system. 12. Present the systems specification and demonstrate the sub-systems functionality. 3. THE SURVEY QUESTIONNAIRE: ADDRESSING COURSE OUTCOMES The purpose of the survey questionnaire was to acquire the perceptions of both IS educator and IS professional respondents appertaining to the importance of the various learning outcomes bulleted above. The survey was distributed to the distribution list of the 2005 Association of Information Technology Professionals (AITP) National Collegiate Conference educator attendees in Dallas, Texas (2006). The instrument was sent out to IS professionals from Fortune 100 companies across the nation. The survey instrument and results can be found at http://users.nsula.jrussell. Except for the first objective of requiring students to complete a review packet of assignments related to business modeling the remaining objectives were used in developing thirty-four (34) statements. Respondents were asked to indicate their agreement or disagreement with each of the 34 statements using a 5 point Likert scale where a score of 5 indicated a strong agreement and a 1 indicated a strong disagreement. For example, the second course outcome previously listed is “participate as a team member or project leader in a semester project.” This outcome is addressed as Survey Question 37 which reads “requiring students to successfully participate as a team player in a project...is very important.” Both responding groups were asked to respond with their level of agreement or disagreement using the Likert scale. Questions 1 – 6 helped gain demographic information for data classification purposes. Survey items 7- 41 are specific statements that map to various course learning outcomes. First, the authors wanted to know whether they were an IS educator or an IS professional, and if they were educators, did they teach in a four-year undergraduate degree program? Third, if the respondent taught at a four-year undergraduate program then what was the title (CIS, IS, MIS, IT or other) of the program? Fourth, the authors were also interested in knowing if the program within which the educator respondent taught had a capstone IS course. In addition, the authors wanted to know if the educator respondents taught within a program that was aligned with a specific model curriculum (IS 2002, ACM 2001 or other). The authors were interested in understanding if the type of program had an influence on the level of agreement with specific objectives of the capstone course. 4. RESEARCH QUESTIONS Primarily the authors were interested in gaining a better insight into the following questions. 1. Was there agreement on the basic course content of the capstone course? 2. Do the educators and professionals agree that both structured analysis and object-oriented analysis be taught in the capstone course? 3. Do the educators and professionals agree that only object-oriented analysis be taught in the capstone course? 4. Did the IS educators as a group provide similar Likert responses for key statements? The answer to this helps determine the validity of the mean scores. 5. Did the IS leaders provide similar Likert responses for key statements? 6. Which survey statements tended to score differently between the groups (in other words, did one group tend to respond differently from the other group for a particular survey question? The answer to this helps determine the validity of the mean scores. 7. Was there agreement that the complete life cycle be taught in the capstone course? 8. Was there agreement as to the importance of teaching interviewing and presentation skills in a capstone course? 9. How similar were the responses from MIS educator group compared to the responses from CIS educator group to the importance of teaching soft skills in a capstone class? 10. How similar were the responses from the MIS educator group compared to the responses from the CIS educator group as to the importance of requiring students to present a systems proposal in front of the class? 11. How similar are the responses provided by the IS educator respondents as compared to the responses from the IS professional respondents. From a research perspective the authors identified the various null hypotheses for the author’s research questions. It is important to this study to know if both IS educators and IS professionals agree on the importance of the outlined key objectives of the proposed IS capstone course. It is also important to know if MIS educators and CIS educators respond the same on key survey questions; otherwise, the results would be questionable since the two subgroups within the group with highly different perceptions could render the mean scores as useless. It is important that both types of curriculums address these important elements within the capstone course. 5. RESEARCH HYPOTHESES The stated hypotheses below will be tested using the survey results and an appropriate statistical test, and based on their calculated level of significance the authors will either accept or reject the following null hypotheses. The .05 level of significance was used to enable the authors to reject the null hypotheses below. H1 IS educators and IS professionals agree on the general content of the author’s proposed IS capstone course (all questions). H2 IS educators and IS professionals agree that both structured methods and object-oriented analysis methods be taught in the capstone course (survey question #40). H3 IS educators and IS Professionals agree that project management is an important component within the capstone course (survey question #7). H4 CIS educators and MIS educators agree that requiring students to prepare and present a system proposal in class is important (survey question #32). H5 IS educators and IS professionals agree that presenting a proposal in the classroom is an important objective (survey question #32). H6 IS educators and IS professionals agree that students should be required to perform data collection interviews is an important course objective (survey question # 31). H7 IS educators and IS professionals agree that requiring students to participate in all phases of the SDLC is very important (survey question # 16). H8 IS educators and IS professionals agree that the capstone course require students to be able to develop complex logic using decision tables (survey question #29). 6. DEMOGRAPHIC RESULTS Twenty seven IS professionals and twenty IS educators provided response packages. Seven of the educator’s survey responses were partially incomplete. From the incomplete responses, the authors employed the survey responses that were completed for individual item (question) analysis (H2 through H8 hypotheses), but were excluded in the H1 null hypothesis later discussed. All but one of the IS educator respondents reported that they taught at a four year undergraduate institution of higher learning. The results indicated that seventy-five (75) percent of the programs were CIS, sixteen (16) percent were MIS and the remainder was listed as other (IS, CS). Four-Year Programs Following a Model Curriculum Fifty (50) percent of the IS educators surveyed indicated that they followed the IS 2002 Model Curriculum while the remaining half did not follow any specific model. The limited response for this survey fails to represent a true proportion to the population, and the authors plan to replicate the survey at a later date to a wider sample audience. Four-Year Programs Teaching a Capstone Course The authors were interested in learning how many programs had an official capstone course in place at their university or college. The authors believe that the smaller than expected response rate was at least partially due to the possibility that many programs do not have a capstone course and as a result may have not even provided a partial survey submittal. Of the responses, fifty-eight (58) percent reported that their program had a capstone course currently being taught. Four-Year Program Respondents Believing that Only OO and UML Should be Taught Survey Statement #38 – “Requiring student to learn only object-oriented analysis (OOA) and UML since structured analysis and design methods are rarely used is very important” received adamant disagreement. Only two respondents felt that only OO and UML should be taught in the business modeling area. Eight-seven (87) percent of the respondents felt strongly that OO and UML should not be the only methodology being taught. This finding will be a catalyst for future research because this seems to contradict the general movement toward object-oriented development. Four-Year Programs Teaching Only Structured Analysis Survey Statement #39 – “Requiring students to learn only structured methods since OOA and UML is still in its infancy is very important” received strong disagreement with all but one response from IS educators and all but two responses from the IS professionals. It seems that the capstone course must include a movement toward objects as well. Four Year Programs Teaching Both Object-Oriented methods and UML and also Structured Analysis Survey Statement #40 – “Requiring students to learn both OOA and UML and structured methods is very important” received mixed results with seventy-two (72) percent agreeing and twenty-seven (27) percent disagreeing. Twenty of the twenty-one respondents from the IS professionals agreed that both methods should be taught. This survey result was a curious finding with a somewhat higher percentage of professionals than educators that felt both methods should be taught. As the reader will ascertain from the statistical analysis this difference in proportions became significant at the .10 level, but not enough significance to reject the H3 null hypothesis previously stated. 7. STATISTICAL ANALYSIS OF THE SURVEY RESULTS A previous statistical analysis was performed on the survey data (Russell and Russell 2006) for the H1 and H2 null hypotheses. The results of the previous study are summarized in the respective sub-sections below. The H3, H4, H5, H6, H7, and H8 null hypotheses were unique to this study alone. Testing the H1 Null Hypothesis To test the H1 null hypothesis a t-test of independent groups was performed between the mean scores of IS educators compared to IS professionals. Figure 1 illustrates the result of the t-test of the two group’s mean scores to determine if there was a general agreement or disagreement between the group’s mean scores as to the content of a capstone course. There was a response mean of 3.971 for educators versus a 3.879 response mean for IS professionals. A p-value of 0.525 indicated that no significant difference exists in the means of the two groups. Therefore, the authors will retain the H1 and accept that both groups agree on the general content of the author’s proposed IS capstone course. Additional statistical analysis of the standard deviations was performed. With the 0.52 p-value coupled with the small standard deviation of 0.4, the authors feel strongly that the two groups agreed on the curriculum content. Two-Sample T-Test on Group Mean-Scores Grouped by IS Educator & IS Professional Group Significance Mean SD Educators N = 12 3.971 0.354 IT Professionals N = 22 3.879 0.420 Mean Difference 0.092 t-value 0.643 Degrees of Freedom 32 p-value .525 Figure 1: Two-sample T-Test of Independent Groups: Educators and Professionals Testing the H2 Null Hypothesis The level of agreement or disagreement between the IS educator and IS professional groups was determined regarding the importance of teaching both structured analysis and object-oriented analysis and design. A ?2 of the Difference Between Independent Proportions was determined related to a forced choice response (agree or disagree) from the two groups. The ?2 calculation pertains to Question 40 that is highlighted below. Question #40: Do you agree or disagree that both structured methods and object-oriented analysis and UML be covered in a capstone class? Figure 2 describes the 2X2 tables describing the survey responses for the question above. The ?2 for Question 40 is equal 3.34. The calculated ?2 of 3.34 is less than the 3.84 proportion of the area in the tail of the distribution of ?2 and falls short of being significant at the .05 level with one degree of freedom; therefore, since 3.34 < 3.84 the H2 hypothesis is retained that both IS educators and IS professionals agree that both object-oriented analysis and structured methods should be taught in the capstone class. Since the calculated ?2 of 3.34 and a probability of .067 was almost significant at the .05 level, it is important to notice that there is a significant difference in the proportions at the .10 level (P < .10) and the authors are surprised at this level of significance; and will investigate this finding in greater detail in future analysis. Frequency Proportion Agree Disagree Agree Disagree IS Educators 8 3 .73 .27 IS Professionals 20 1 .95 .05 Test statistic Value df Probablity Pearson Chi-square 3.344 1.000 0.067 Figure 2: 2X2 Chi Square Table Illustrating Frequency and Proportion of Agreement or Disagreement Testing the H3 Null Hypothesis A two-sample t-test was performed on survey question #7 to determine if there was agreement of IS educators and IS professionals regarding their perceptions of the importance of the over-all application of project management concepts, skills or tools to either a contrived or real system project. The t-test calculations indicate that there was no significance difference between the mean scores of the two groups for this question. The p-value of .301 far exceeded the .05 level of significance required to reject the null hypothesis. The IS educator mean score for this question was 4.333 with a standard deviation of 0.492. The IS professional mean score was 4.545 with a standard deviation of 0.595. From these results the authors retained the null hypothesis that the two groups agreed that project management skills were very important. Figure 3 highlights the t-test results. Two-Sample T-Test for Question 7 Grouped by IS Educators and IS Professionals Group N Mean SD IS Educator 12 4.333 0.492 IS Professional 22 4.545 0.595 Pooled variance Difference in means = -0.212 95.00 Confidence Interval = -0.623 to 0.199 t = -1.051 df = 32 p-value = 0.301 Figure 3: Two sample T-Test for Survey Question 7(importance of project management) Grouped byISEducators and IS Professionals Testing the H4 Null Hypothesis A two-sample t-test was performed on survey question #32 to determine if there was agreement of educators within a CIS program versus educators within a MIS program regarding their perceptions of the importance of presenting a systems proposal in front of the class. The results indicate a strong agreement between the two groups that this is a very important capstone objective. The two-sample t-test results are highlighted in Figure 4. The p-value of .487 (p > .05) far exceeds the .05 level required to reject the null hypothesis; therefore, the H4 null hypothesis is retained. Testing the H5 Null Hypothesis A two sample t-test was performed also on survey question #32 but grouped by IS educators as a single group and IS professionals to determine if there was agreement between the two groups. Figure 5 reveals that no significant difference existed between the groups as to their perceptions of the importance of the objective of requiring students to present a systems proposal in front of the class. Since the mean scores were near or above 4.0 it is safe to say that the two groups felt this objective was important. It is worth noting that educators had a .445 greater mean score than IS professionals. This is a curious finding in that the author’s past research (Russell, et.al. 2005) discusses how strongly industry feels that students be able to make persuasive presentations. The authors plan to expand the survey population to find out if this was due to an inadequate stratification of the survey group. From the 0.176 p-value (p > .05) and the fact that the mean scores were near or above 4.0 the authors will retain the null hypothesis previously discussed. Two-Sample T-test on Q32 Grouped by MIS Educators or CIS Educators Group N Mean SD CIS Educ. 7 4.429 0.535 MIS Educ. 2 4.000 1.414 Pooled variance: Difference in means = 0.429 95.00% CI = -0.952 to 1.810 T = 0.734 Df = 7 p-value = 0.487 Bonferroni adj p-value = 0.487 Figure 4: Two sample t-test for Survey Question 32 (importance of presentations) grouped by CIS Educators versus MIS Educators Two-sample T-Test on Q32 Grouped by IS Educators and IS Professionals Group N Mean SD IS Educ. 0 4.400 0.699 IS Prof. 22 3.955 0.899 Pooled variance: Difference in means = 0.445 95.00% CI = -0.212 to 1.103 T = 1.384 Df = 30 p-value = 0.176 Figure 5: Two Sample T-Test for Survey Question 32 (Proposal Presentation Requirement) Grouped by IS Educator and IS Professional Testing the H6 Null Hypothesis A two-sample t-test was made for survey question #31 that measured the level of Perceptions regarding the importance of requiring capstone students to conduct data collection interviews in the classroom. The p-value of .585 (p >.05) shown in Figure 6 below far exceeded the .05 p -value needed to reject the null hypothesis; therefore, the authors retained the null hypothesis that both groups agreed that it was important to require students to conduct interviews in the classroom as a part of the over-all capstone experience. Two-Sample T-Test on Q31 Grouped by IS Educators and IS Professionals Group N Mean SD IS Educ. 0 3.900 0.876 IS Prof. 22 4.091 0.921 Pooled variance: Difference in means = -0.191 95.00% CI = -0.898 to 0.516 T = -0.551 Df = 30 p-value = 0.585 Bonferroni adj p-value = 0.585 Dunn-Sidak adj p-value = 0.585 Figure 6 Two-Sample T-Test on Survey Question #31 (importance of interviewing) grouped by IS Educator and IS Professionals Testing the H7 Null Hypothesis Figure 7 illustrates a two-sample t-test to determine if there was a significant difference in the mean scores for survey question # 16 that measures the degree to which the respondent agreed or disagreed that the capstone course should require students to be involved in all phases of the systems development life cycle. The authors expected a higher mean score for both groups for this survey item. IS educators scored the lowest with a mean score of 3.833 with IS professionals scoring a 4.273. The two scores indicated that both groups felt that exposing students to all phases was important, but fell short of being a very important element. Nonetheless, from the p-value of .132 (p > .05) there was no significant difference in the mean scores. Testing the H8 Null Hypothesis The H8 null hypothesis states, “IS educators and IS professionals agree that the capstone course require students to be able to develop complex logic using decision tables” (measured with survey question #29). Figure 8 illustrates the two-sample t-test on survey question #29 grouped by IS educators and IS professionals. The reported p-value of .065 (p > .05) is greater than the required .05 level of significance to reject the null hypothesis; but the authors were surprised at this level of significance with IS educators with a mean score of only 3.3 while IS professionals scored a 4.0. IS professionals agreed that requiring students to use decision tables to develop complex logic, but IS educators only somewhat agreed. The authors plan to research why there was this level of disparity between the two groups regarding the importance of this capstone objective. Two-Sample T-Test on Q16 Grouped by IS Educators and IS Professionals Group N Mean SD IS Educ. 12 3.833 0.835 IS Prof. 22 4.273 0.767 Pooled variance: Difference in means = -0.439 95.00% CI = -1.018 to 0.139 t = -1.548 df = 32 p-value = 0.132 Bonferroni adj p-value = 0.132 Dunn-Sidak adj p-value = 0.132 Figure 7 Two-Sample T-Test Determining Level of Significance in Mean Scores of Question 16 (Importance of Involving Students in All SDLC Phases). Two-Sample T-Test on Q29 grouped IS Educator and IS Professional Group N Mean SD IS Educ. 10 3.300 1.059 IS Prof. 21 4.000 0.894 Pooled variance: Difference in means = -0.700 95.00% CI = -1.445 to 0.045 t = -1.920 df = 29 p-value = 0.065 Bonferroni adj p-value = 0.065 Dunn-Sidak adj p-value = 0.065 Figure 8: Two-Sample T-Test on Survey Question #29 (Importance of Decision Tables and Complex Logic) Grouped by IS Educators and IS Professionals Figure 9: A Comparison of IS Educator’s and IS Professional’s Mean Scores by Survey Item 8. CONCLUSION Based on the statistical analysis performed the authors believe that, in general, there is strong agreement between IS Educators and IS Professionals regarding the importance of the learning objectives of the proposed capstone course. All null hypotheses were retained with H8 almost being rejected with a p-value of .06. The authors realize that to perform a test of significance of a difference in the means of the individual survey item mean scores then the degree of variation in various survey item mean scores can impact the validity of the t-test. Figure 9 below illustrates the IS educator’s and IS professional‘s mean scores by survey item. The reader may observe a very acceptable variation in mean scores with all scores falling between a 4.5 and a 3.2. The observed proximity of paired mean scores of the two groups also helps support the validity of this research. A noticeable disparity in paired mean scores can be observed in survey items 31 through 35.The authors are fully aware that the small response rate especially for educators could have distorted the Chi Square calculation for the H2 hypothesis test. The Pearson Chi Square test generated a warning message that one-fifth of the fitted cells were sparse with a frequency < 5. As a result, the authors plan to replicate this study in the fall or spring semesters with a broader stratified response audience. The authors suspect that a poor response rate was due to the survey being sent out during the summer months when educators are on holiday. In general, the authors feel comfortable with the proposed IS capstone learning objectives, and as a result plan to continue incorporating them into the IS capstone class at their university. 9. REFERENCES Davis, Feinstein, Gorgone, Longenecker, and Valacich, (2002), IS 2002 Information Systems Model Curriculum, College of Business and Economics, Washington State University, Pullman, WA 2002. McGinnis, D R and G J Slauson, (2001), “An Information Systems Course Model That Emphasizes Non-Technical Skills.” In the Proceedings of ISECON 2001, v 18 (Cincinnati): § 33a. Russell, J. P. and B. Russell (2006), “Proposed Model Curriculum for a Capstone Class in the IS Curriculum: A Survey Analysis of IS Educators and IS Professionals”, Issues in Information Systems and the Proceedings of the IACIS Conference, Reno, NV. Russell, J., Russell, B. and Tastle W. (2005), “Teaching Soft Skills in a Systems Development Capstone Class”, Information Systems Education Journal, v 3 n 19 ISSN: 1545-679X. Also appears in the Proceedings of ISECON 2004. 2224 ISSN: 1542-7382. Schatzberg, L. A. (2003), “Capstone Introductory IS Course: Strengthening Coverage of IS 2002.1 and Disentangling it from IS2002.p0”, The Proceedings of ISECON 2003, v 20 (San Diego): § 2421. ISSN: 1542-7382. (Also appears in Information Systems Education Journal 1: (1) ISSN: 1545-679X.) Strader, T. (2004), State Farm Insurance Information Systems Student Recruiter, Interview, Northwestern State University, September 24, 2004. Tastle, W J and U R Dumdum, (2000), “Enabling Systems Analysis and Design: A Case for Extending the IS Curriculum”, The Proceedings of ISECON 2000, v 17 (Philadelphia): § 208.