Post-Implementation Review of an Introduction of WebCT Douglas Michele Dept of Business Administration Eastern Connecticut State University Willimantic, CT 06226 USA Doncho Petkov petkovd@easternct.edu Dept of Business Administration Eastern Connecticut State University Willimantic, CT 06226 USA Patricia Kleine Eastern Connecticut State University Willimantic, CT 06226 USA Abstract This paper describes the process of a post-implementation review of WebCT introduction at Eastern CSU. It is based on structured interviews of twenty faculty members and nineteen students who were are using this technology. These interviews were summarized using a software program for processing unstructured text, NVivo. The report analyzes the usage of WebCT among faculty and students within the University, the organizational structures and procedures that were established at ECSU to implement WebCT, its impact on the teaching practices at the university, and shows some problematic areas that the users of WebCT are having, which might be of usefulness to other similar endeavors elsewhere. Keywords: post-implementation review, on-line education, WebCT, qualitative research, NVivo 1. INTRODUCTION The introduction of integrated teaching environments has been in the attention of IT educators for a number of years. It has as a phenomenon many dimensions. Thus Rutter and Hunt (2003) investigate the design of a managed learning environment with WebCT. Similar goals are pursued by Dabbagh and Schmitt (2001) and Turchin et al. (2000). Ellis and Okpala (2004) investigate the use of digital technology among business professors. This requires training which is often lacking. Clegg et al (2000) investigate the issues of educating professors how to use Information and Communications Technologies to transform the way how education is delivered. Another aspect of the implementation of Internet based teaching tools is the evaluation of on-line teaching materials (see Robson, 2000 and Cann, 1999). A conclusion emerges from the above studies that they focus only on some of the aspects of the problem of introduction of tools like WebCT. To the best knowledge of the authors very little or no attention has been paid to the systemic post implementation evaluation of the introduction of a teaching environment at a teaching institution and that was one of the motivations behind this research. This paper describes the process of the investigation and a case study about a project designed to find what improvements and concerns need to be addressed after the implementation of WebCT at Eastern Connecticut State University, conducted in the second half of 2003. The University bought a license for the WebCT package and it was introduced as a pilot in March of 2003. The full implementation was completed in the Fall Semester 2003. The sub-goals of the research were as follows: * Investigate if there were suitable organizational structures and processes established at the University, including clear channels of communication to the faculty about the implementation of WebCT. * Investigate if the faculty felt that they were offered training and adequate assistance with WebCT. * Investigate how the introduction of WebCT influenced the teaching culture at the university. * Establish what is missing in terms of the use of the system by students. * Establish what is missing in terms of the management of the use of the system. The investigation was systemic and ensured completeness in the analysis by covering all aspect of this organizational problem using four organizational dimensions suggested by Flood: organizational structures associated with the problem, establishment of relevant processes, organizational politics affecting the problem and how the organizational culture affects the problems and vice versa (see Flood, 1995 for more details). The interview questions were structured around these four dimensions. The analysis of the interviews was undertaken using a program which analyzes qualitative data in the form of unstructured text. The program is called NVivo, and has to our knowledge never been used for this type of research before at this university. Once the analysis of the interviews was undertaken, themes and trends were identified which were used in making conclusions about what was discovered. The following section will outline the suitable research methods to achieve the above goals. 2. METHODOLOGY Before proceeding with the investigation, it was necessary to select an appropriate research approach. Due to the complexity of the problem it was felt that the investigation should go beyond a quantitative survey in order to reveal deeper issues about the impact of the WebCT implementation on the teaching culture at the university. There are several different types of qualitative research. Qualitative research involves the use of qualitative data, such as interviews, documents, and participant observation data, to understand and explain social phenomena. In Information Systems (IS), there has been a general shift in IS research away from technical to managerial and organizational issues, hence an increasing interest in the application of qualitative research methods (Myers, 1997). It should be emphasized that the word “qualitative” is not a synonym for “interpretive.” Qualitative research may or may not be interpretive, depending upon the underlying philosophical assumptions of the researcher. Qualitative research can be positivist, interpretive, or critical. The choice of a specific qualitative research method (such as the case study method) is dependent on the underlying philosophical position adopted. For example, case study research can be positivist, interpretive, or critical (Myers, 1997). A case study approach was used in this investigation. This methodology was selected due to its strength and simplicity. The complexity of the case under concern makes it revelatory. By following the guidelines of Benbasat et al. (1987: 372), it can be claimed that the design for this exploratory research can be based on a single case. Case study research is the most common qualitative method used in information systems (Orlikowski and Baroudi, 1991). Although there are numerous definitions, Yin (1994) defines the scope of a case study quite well. A case study is an empirical inquiry that: “Investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident" (Yin 1994, p. 13). A case study typically spans over two to three months while an investigation can sometimes go longer and is usually of ethnographic nature. The research conducted on the WebCT implementation at ECSU is an interpretive case study, based on deep structured interviews. The data is processed with a special software package for text processing (NVivo) to be discussed in the next section. We adopted a case study approach as the post-implementation review had certain time limits related to it and hence there was no possibility to conduct an ethnographic analysis. The purpose was to improve the understanding of the university community and the management of information technology at ECSU about the potential problems associated with it. Following Lee (1989: 41), even though the observations in a particular case study are non-replicable, the case study’s findings (that a particular theory is confirmed or disconfirmed) would be replicable. Further applications to other case studies will provide evidence to test the findings of this case against the empirical circumstances of those other case settings. However, further applications of a similar approach to investigate other WebCT implementations could serve as a basis for possible generalization of a theory to explain some of the findings. The philosophical base of interpretive research is hermeneutics and phenomenology (Boland, 1985). Interpretive studies generally attempt to understand phenomena through the meanings that people assign to them and interpretive methods of research in IS are "aimed at producing an understanding of the context of the information system, and the process whereby the information system influences and is influenced by the context" (Walsham 1993, p. 4-5). Interpretive research does not predefine dependent and independent variables, but focuses on the full complexity of human sense making as the situation emerges (Kaplan and Maxwell, 1994). The processing approach employed in this study using the NVivo package is based on hermeneutics (see for details Myers, 1997). The NVivo package was chosen, because it is an application widely used for the analysis of qualitative data. Its primary feature is that it integrates coding with qualitative linking, shaping and modeling. It is able to assist in making sense of the whole, and is able to show the relationship of the meaning to the questions. Using NVivo NVivo does two things: it supports the storing and manipulation of texts or documents; and it supports the creation and manipulation of codes, known in NVivo as “nodes.” Around these two basic functions the program also provides tools for creating and examining new ideas about the data for example, through searching, linking and modeling and for reporting results. The dual functions of supporting documents and nodes are reflected in two major tools in NVivo, the Document Explorer and the Node Explorer. The first is used to manage the collection of documents, information about all documents including proxy documents, and anything linked to them such as other documents can be coded, and NVivo will keep track of the codes and associated text. Documents can also be edited, formatted and reported on. The Node Explorer is used to manage nodes. Nodes are names or labels for a concept or data about data. Coding text at a node is the process of establishing a relationship or connection between a node and one or more passages of text. Nodes can be free nodes, just kept as a list, or organized into a free or hierarchical structure. The Node Explorer enables one to create, delete, merge and move nodes and to change the text to which they refer. At any time one can browse the text coded at a node and change that coding or view it in context. Nodes can also be searched and in this way, along with an inspection of linked data, such as memos, the researcher can ask questions of the data and build and test theories (Gibbs, 2002). For the purposes of this project, all of the documents for the WebCT Project were typed into Microsoft Word and saved in the rich text format (RTF) and then imported into the NVivo package. They were then later separated into ‘Sets’ separated by either being part of ‘the School of Arts and Sciences’, or by the ‘School of Education and Professional Studies’. There are four options to create documents in NVivo. They are: Locate and import readable external files; make proxy documents representing external files; make a proxy document for non-file data, and Make a new blank document. A node in NVivo is a way of bringing together ideas, thoughts and definitions about data, along with selected passages of text. Passages of text from one or more documents are connected to a node because they are examples of ideas or concept it represents. This process is called coding the text at a node. This brings together passages of text that are about the same thing or indicate similar ideas, concepts, actions, descriptions, and so on. For example, by coding them at a node one could connect together all the passages of text where a surprising theme was discovered in the documents where it has been recurring (Gibbs, 2002). As well as its name and the passages of text to which it is connected, a node can have a definition and any number of memos that contain ideas, hunches, and other notes about the node or the text it codes. Having established the node in NVivo, one could then start to look for examples in the text that could be linked with the node (Gibbs, 2002). NVivo distinguishes three ways of keeping nodes: free nodes, tree nodes and case nodes. Free nodes are the simplest and appear as a simple list in the program. Like all nodes, they have a title, icon, description, and one or two other properties such as the dates they were created and modified. Tree nodes, have all of the properties of free nodes, but in addition are organized into a hierarchy or tree shown in the Node Browser like the file and folder hierarchy in Windows Explorer. Free nodes can be made into tree nodes (and vice versa). The process of constructing a hierarchy of nodes is often a central activity in qualitative analysis. Case nodes are used for organizing coding about cases. They can refer to all the text for a particular case or can be used to organize these cases into case types (Gibbs, 2002). An Example of How the Data Was Coded Question 20 of the questionnaire reads as follows: “Are students having problems in your class using WebCT?” The responses obtained from the interviews for this question could be grouped into four groups, namely; “Yes”, “No”, “Not Sure”, and “Not Applicable”. These four groups would then be captured into NVivo as the nodes for the question. The nodes for a specific question are the actual grouping of the responses for that question. If a respondent reply’s that he or she does not know the answer or is unsure of the answer, this would fall under the node, “Not Sure”. Because NVivo uses a tree structure for its nodes, the base of the tree would be question 20, the next four parallel nodes would be “Yes”, “No”, “Not Sure”, and “Not Applicable”. If the response was “Yes”, then this node will have seven sub-nodes of: “Password Problems”, “Login Problems”, “Printing”, “Does not know how to use it”, “Off Campus Problems”, “Other”, and “Download Problems”. The task of assigning the nodes to the respondents’ comments is known as coding. Once all of the responses are captured and coded, the system is able to both qualitatively and quantitatively analyze the information. This analysis results in the further identification of the relationships to the responses; statistical information is able to be derived from the information, such has how many of them said ‘Yes’ or ‘No’; and trends identified. 3. DESIGN OF THE QUESTIONAIRES FOR THE INTERVIEWS The questionnaires for the interviews were separate for the faculty and for the students. The questionnaire used in the interviews with the faculty was grouped into the four sections. The first section questions were setup to find out if there were any suitable organizational structures in place relevant for the implementation of WebCT. This section for the questions correlating to the structure was designed to find out if the users felt that the people who were responsible, and who controlled the mechanism of WebCT contributed to their involvement of using it in their courses. The second set of questions was designed to help in the investigation of the processes and procedures that were introduced for the purposes of the implementation of WebCT. They can be summarized under the category of Processes – another dimension of any organizational problem. These questions were used to find if there were proper processes and procedures established for better usage of WebCT. They determined whether manuals or tutorials were distributed to look at. The third section involved issues regarding the WebCT implementation and the organizational culture at Eastern. These were designed to find out if the implementation of WebCT had affected the teaching culture of the university by including an additional piece of technology into the educational practices at the university. The last set of questions looked at additional issues of the WebCT implementation that could be related to some political issues – a fourth dimension of the investigation of any organizational problem according to Flood (1995). They were asked to see if the faculty felt that WebCT may eventually replace the traditional teacher and to find out if they had any additional comments that they would like to have made about WebCT. These categories were chosen in order to perform the investigation of the post-implementation of WebCT at the university in a systemic manner according to Flood (1995), as well as to identify those problem areas which are in need of possible attention. The questions were then examined with the assistance of the manager of the Center for Instructional Technology. Among other things they reflected what types of concerns CIT is trying to find as a result of the investigation, and to possibly bring results to them. Several of the questions were setup to see if the participants knew what WebCT was, and to see how much of it they have incorporated into their courses. Other questions were created to find out what types of problems students and faculty are facing, and what kinds of solutions they would like to see. The questionnaire for the faculty interviews was experimentally tested in a pilot setting with four faculty members. A similar approach was followed for the student questionnaire. 4. INTERVIEW PROCESS AND HOW THE RESULTS WERE DERIVED The sample of participants to be interviewed was provided by the CIT department. They had supplied a list of all of the professors using WebCT. From the thirty-three of them were contacted to be part of this investigation. Of the thirty-three invited to be interviewed, twenty of them responded, and those were the ones that are part of this investigation. Some of the students that were part of the student questionnaire were interviewed from the place of work of the author and from courses that he was in. Fifteen of the interviews were then distributed in an English course in which all of the students were using WebCT. The faculty members that were interviewed during the investigation of WebCT were told that the process of interviewing would take 20 to 25 minutes. Each person that was interviewed was tape recorded and verbally asked each question. Their responses were jotted down on the paper and were further elaborated by the playback of the recorded interview. The students that were interviewed were handed the questionnaires, and they filled out the responses to the questions. Once the documents were loaded into NVivo, the coding all of the responses to the questions started. The first task was to setup a node system in which each of the responses would fall under. When creating a node system, a base (or a tree) must be created. The tree for the node was each question. Sub-nodes (branches off the tree) were then created. Each sub node was created so that the responses could coincide with each branch. If the respondents answered “Yes” to a question, then the response would be coded into the “Yes” sub-node. If the respondent said ‘Yes’ and had additional comments, then under the “Yes” there would be another sub-node. This sub-node would have all of the additional comments that were left as a result of the respondent saying “Yes”. The next section will deal with a brief summary of the interview results. 5. DISCUSSION OF THE RESULTS Summary of Findings on Whether the Organizational Structure and Processes Were Established For the Implementation of WebCT The majority of the faculty felt that WebCT is a useful software tool that allows them to interact with their students in a structured way. They can post their classroom materials, syllabuses, quizzes, documents, grades and assignments. Many of them felt that it is an useful way of communicating with the students outside of the classroom though not too many were using the opportunity for chat and threaded discussions. Faculty were informed about the introduction of WebCT and CIT allocated the necessary human resources and communication channels for this task. Many of the faculty referenced the online manuals that were distributed because there were not enough hard copy manuals were distributed, but these days most of the documentation of software products in on-line. The CIT department did in fact setup very good procedures for training sessions. Its staff had offered training sessions at different times of the year. They offered individualized training as well. Faculty members who attended CIT training sessions stated that the training had benefited them. CIT offered an adequate amount of training. A suggestion was made however to introduce training for students at the beginning of each semester and to improve the documentation for them. Another suggestion was that it may be helpful to involve more of the adjunct professors to use WebCT through offering training in the evening as well. Areas for further specialized training were: how to set up chat sessions, exchanging data with Excel and how to organize threaded discussions. This level of training would involve more sophisticated usage of WebCT and can be offered in subsequent advanced courses. A WebCT user group at ECSU to exchange ideas for better teaching with WebCT could be setup as well. In conclusion to this section of the investigation it may be stated that CIT initiated the introduction of WebCT adequately from the point of view of organizational structure and processes necessary for a successful implementation of such a technology. Summary of Findings on How the WebCT Adoption Influences the Teaching Culture and Other Issues at ECSU A summary of the responses to the questions that were related to whether the implementation of WebCT has affected the issues relating to the teaching culture shows that the introduction of WebCT did not really create any major problems with the teacher and student culture. A potential problem is that only a small percentage of the respondents mentioned that they had used all of the benefits of WebCT. Several said that they used it progressively while some to a limited degree. There could be an increase in the usage of WebCT by faculty members if there was some additional higher level training offered. The majority of the respondents said that they planed on incorporating additional WebCT technology into their work. Some were going to use it to cut down the amount of paper they distribute in class, while others would like to incorporate more of the chat and collaborative features. When asked how WebCT fits into their teaching style and practices, half of the respondents said that it works for them and they do not have any problems with it. A quarter of them said it makes teaching easier, while several said that they can now get better feedback from the students. Many of the respondents said that they did not find any resistance to using WebCT among their colleagues, while a few said they did not ask. Half of the interviewed did not feel any threat to their discipline from fully online courses being offered elsewhere. They did not think that online courses could replace the teacher in the classroom, as there still needs to be student-to-teacher and student-to-student interaction. Findings from the Student Questionnaire The nineteen interviews with the students showed that they accepted well this innovation into the teaching process. At the same time the participating students indicated that there were no formal training sessions for the students and that there were no manuals that were distributed for their use. There could be some form of training for students to attend on how to use WebCT. This could be done at the beginning of each semester. Alternatively CIT may create a student manual that professors could distribute to their students. CIT have posted on their website possible problems that may be encountered, and then provided solutions to these problems. Most of the students had an informal idea of what WebCT was, but they only knew as much about it as the teacher told them and offered. Most students only used WebCT for minor things, such as checking up assignments, while others used it to submit papers and assignments. That demonstrates that the resources of the environment are not utilized fully yet in the teaching process. It may be concluded that CIT had done a very good job with being able to inform students on how to solve any problems that they may encounter with using WebCT. They have posted on their site possible problems that may be encountered, and then mentioned solutions to these problems. This can be very helpful for the students that are doing work late at night and need an immediate response. They can access the site and see if the problem they are facing is mentioned with the solution. The CSU system office, responsible for fully on-line courses offered through any of the four universities in the system, may have to consider the implications of other universities offering cheaper online courses and find ways to keep its position competitive through not charging higher fees for fully online courses as the student responses demonstrate the need not to increase the price of fully online courses. When asked if they would pay more for a fully online course, almost all of the students said that they would not. 6. CONCLUSION This investigation dealt with the systemic post-implementation review of the WebCT deployment at Eastern Connecticut State University. It aimed to analyze what kinds of improvements need to be added and what concerns need to be addressed to expand the usage of WebCT considering the four dimensions of a systemic organizational problem: structures, processes, culture and politics (Flood, 1995). WebCT has only been operating on campus for one full semester. As time goes on more and more faculty members will become accustomed to WebCT. The process of setting up the pages and the minor problems that they face will soon become extinct. The overall responses of most of the faculty members and students interviewed showed that they enjoyed using WebCT. There were however, some minor problems that faculty members and students encountered, which are briefly outlined above and discussed in more detail in the full report. WebCT has only been used on campus for one full semester. As time goes on, more and more faculty members become accustomed to the process of setting up their courses using this technology, the minor problems that they face now will soon become extinct. It is evident that WebCT has benefited not only the teachers but the students as well. A conclusion from the faculty and student interviews is that the Center for Instructional Technology implemented successfully WebCT at ECSU. The necessary communication channels, training and organization for the blending of the new technology into the teaching practices at Eastern were in place. A limitation of the results in our analysis is the fact that the sample of the interviewed instructors was not big enough and hence the results are only exploratory in nature. Still they served the purpose of designing subsequent measures at the university for improvement of the utilization of the WebCT resource. A further possible refinement would have been if the duration of previous expertise of the instructors involved in the interviews was taken into account in analyzing in more details the different needs for support and training of different categories of professors using WebCT. Possible further investigation in a longitudinal setting would enable to measure whether future improvements in the usage of WebCT at the university are taking place and at what pace. Future potential improvements of WebCT usage at an University may be achieved through the greater involvement of academic management in that process taking it beyond the traditional cooperation between an academic computing unit such as CIT at ECSU and the faculty members. This may happen through the introduction of suitable organizational measures and incentives for faculty promoting the usage of this new educational technology. That is taking place already to a degree. In conclusion it can be stated that the reaction of the CIT and the university community to the report showed that in general the feedback to the IT and academic management provided through the results from a systemic post implementation review as the one discussed here is useful. The process of the investigation may be transported to other educational institutions aiming at analyzing its implementation of a new educational technology such as WebCT. 7. ACKNOWLEDGEMENTS We would like to thank our colleagues at ECSU: Mr G Kahkedjian (CIO), Steve Frazier (CIT), Rick Riese (ITS) and Dr D. Pachis (Vice President of Academic Affairs) for their advice and encouragement. 8. REFERENCES Benbasat, I., Goldstein, D.K., and Mead M., (1987). The Case Research Strategy in Studies of Information Systems, MIS Quarterly, 11(3), 369-386. Boland, R., (1999). "Phenomenology: A Preferred Approach to Research in Information Systems," in Research Methods in Information Systems, E. Mumford, R.A. Hirschheim, Cann A. Approaches to the Evaluation of Online Learning Materials, Innovations in Education & Training International, Vol. 36 Issue 1, 44-52. Clegg S., (2000). Preparing academic staff to use ICTs in support of student learning, The International Journal for Academic Development , Vol 5 No 2, 138-148 Dabbagh N. and Schmit J., (2001). Redesigning Instruction Through Web Based Course Authoring Tools, Educational Media International, Jun98, Vol. 35 Issue 2, 106-110. Ellis R. and Okpala, C., (2004). Evaluation of Digital Technology and Software Use Among Business Education Teachers, Journal of Instructional Psychology, Vol 31 No 1, 53-59. Flood, R. (1995). Solving Problem Solving, Wiley, Chichester, pp 65-75. Gibbs, Graham R., (2002). Qualitative Data Analysis Explorations with NVivo. Philadelphia: Open University Press Kaplan, B. and Maxwell, J.A., (1994). "Qualitative Research Methods for Evaluating Computer Information Systems," in Evaluating Health Care Information Systems: Methods and Applications, J.G. Anderson, C.E. Aydin and S.J. Jay (eds.), Sage, Thousand Oaks, CA, pp. 45-68. Lee, A., (1989). A Scientific Methodology for MIS Case Studies, MIS Quarterly, 13 (1), 33-50. Myers, Michael D., (1997). "Qualitative Research in Information Systems," MIS Quarterly, Vol. 21, No. 2, pp. 241-242. MISQ Discovery, archival version, June 1997, http://www.misq.org/misqd961/isworld/ Orlikowski, W.J., (1991). "Integrated Information Environment or Matrix of Control? The Contradictory Implications of Information Technology", Accounting, Management and Information Technologies, (1:1), pp. 9-42. Orlikowski, W.J., Markus, M.L. and Lee, A.S., (1991). "A Workshop on Two Techniques for Qualitative Data Analysis: Analytic Induction and Hermeneutics," Proceedings of the Twelfth International Conference on Information Systems, p. 390-1. Robson, J., (2000). Evaluating Online Teaching, Open Learning, Vol. 15, No. 2, 151-172. Rutter, P, and Hunt A., (2003). The Development of a Managed Learning Environment Using WebCT to Facilitate 4th Year M.Pharm Undergraduates Ability to Counsel Patients in Preparation for OSCEs, Pharmacy Education, Vol. 3 (1), pp. 63–66 Seidel, John V., (1998). “Qualitative Data Analysis.” Accessed 30 Sept. 2003, http://www.qualisresearch.com/ Turchin, A., Lehmann H., Flexner C, Hendrix, C, Shatzer, J., and Merz, W., (2000). Active Learning Centre: potential uses and efficacy of an interactive Internet-based teaching tool ,Medical Teacher, Vol. 22, No. 3,271-275. Walsham, G., (1993). Interpreting Information Systems in Organizations, Wiley, Chichester. Yin, R. K., (1994). Case Study Research, Design and Methods, 2nd ed. Newbury Park, Sage Publications.