Evaluation of a Virtual Lab Environment for Teaching Web Application Development

Jens Liegle
jliegle@cis.gsu.edu
Peter N. Meso
pmeso@cis.gsu.edu
Dept. of CIS, Georgia State University
Atlanta, Georgia  30319  USA
ABSTRACT
In this paper we explore how one aspect of virtual computing – the virtual lab – effectively addresses many of the challenges of teaching web application development. Based on a case study at a large south-eastern university, we begin by providing a description of the technical resources needed to teach such a course. We then briefly describe the shortcomings of previous approaches for providing a suitable environment, followed by a description of the recently implemented virtual lab approach. Thereafter, we report results of a survey that asked students exposed to this environment about heir experience and perception of the virtual lab. The paper concludes with a discussion on the benefits, drawbacks, and lessons learned from the virtual lab approach.
Keywords: virtual lab, web design, computer programming

1.  INTRODUCTION
The ongoing evolution of information technology has a direct bearing on the nature of instruction provided to business students, in particular within the computing and information sciences. The teaching of courses in this field requires the use of definite computing technologies either within a laboratory setting or individually outside of the classroom (Liegle and Madey, 2003). The perception that students have about the effectiveness of the instructor and the course are also influenced by students’ perception of and satisfaction with the technology used (Meso and Liegle, 2003, 2005). These perceptions have a direct bearing on the instructor’s evaluation by students – evaluations that are heavily relied upon at most institutions to assess the teaching effectiveness of instructors (Liegle and Johnson, 2003). Technologies that simplify the teaching of IT enhance pedagogical quality.
In the case of application development, the complexity of the technology, or its cumbersomeness, mitigates the students’ ability to grasp and understand the core body of knowledge being disseminated in the course. This is due to the fact that the computing environment of application development, in particular for the web, typically involves a) multiple physical tiers in form of client/browser, web-server, application server, and database server, b) different perspectives of programming in terms of client vs. server side, c) infrastructure related issues such as security/firewalls, downloading/uploading software, web hosting, and so forth.
Effective teaching tools enhance the learning capability of students and make the mastery of difficult principles simpler (Liddle, Brown et al., 1995; Janicki and Liegle, 2001). Research in this area also points out that the teaching tools and technologies that prove to be effective in most cases are those that are easy to use and easy to learn (Meso and Liegle, 2005), minimize the technological barriers between student and the core-knowledge or principles being disseminated to the student (Janicki and Liegle, 2001).
Therefore, our objective in this study is to assess how one aspect of virtual computing – the virtual lab – effectively addresses many of the challenges of teaching application development. Based on a pilot study at a large south-eastern university, we provide a description of the recently implemented virtual lab approach. Thereafter, we report results of a survey that asked students exposed to this environment about heir experience and perception of the virtual lab. The paper concludes with a discussion on the benefits, drawbacks, and lessons learned from the virtual lab approach.
2.  VIRTUAL LAB
In spring 2005, the department installed a virtual lab environment (VL). The VL is configured in a way that each class is given a number of virtual workstations (VWS). Students can access these VWS via MS remote desktop connection and then interact with them as if they were in front of the real machine. Each of these VWS is presently so configured that the student has complete administrator rights, and has both IIS and Visual Studio running. Instructors are given administrator rights to all these workstations as well, allowing them to review and test applications. The VL was first used for teaching in the summer semester of 2005.
2.1 Configuration of Virtual Lab
The virtual lab cost a total of $115 000. It consists of two servers (IBM XSeries 365) with the following software in place for the architecture: VMware ESX 2.5, VMware Virtual Center 1.2, IBM Director 4.2.02, and IBM Virtual Machine Manager 1.0. Each server has 10Gb memory, four Xeon 2.2 GHz processors, three 100 GBits network cards, six hard drives with 146GB each, configured as a RAID5 array. This setup enables the concurrent existence of forty Win2k3 server and/or XP-workstation virtual machines with 256MB memory.
2.2 Operation of Virtual Lab
The system administrator creates a template virtual workstation for each class, installing and configuring the required software as per the specifications of the instructor. The instructor is then given a number of student accounts, which are replicas of this template. Students are then provided with the virtual machine name, and given a login ID and password. Unlike with a shared server, the names of the virtual machines are different for each account, and this account has full administrator rights. Students can connect to their VWS either from conventional teaching lab workstations (that by themselves do not have IIS running), or from home, as long as they have remote desktop connectivity. This allows students to develop, host, and test their web applications directly without being constraint by the settings of the local machine.
The major advantages of this setup are:
* It becomes possible to support in-class web development exercises using conventional computer labs
* Students don’t have to install and configure IIS and necessary software development environments like Visual Studio at home
* Students don’t have to submit projects, sine the instructor has access to all their virtual accounts
* Once a VWS is created, each student/instructor can install their own personal software
* Special software that requires individual licenses can now be installed on the right number of virtual machines
* System administration is made much easier:
* Each semester, the original template is used to re-create the class accounts by simply cloning it
* Both the students and the instructor have full administrative rights and therefore do not have to wait for tech support
* Should an account become corrupted, only the project files need to be saved while the workstation is re-cloned
3.  EVALUATION OF THE VIRTUAL LAB
The virtual lab was installed and tested during the spring 2005 semester, and two faculty members were given access to a small number of VWS to allow them to learn the system, configure “template” workstations for their respective courses, and write instructions for students. During the summer 2005 semester, these two faculty members were assigned a total of 25 virtual workstations for use in their three web development and systems design courses.
Toward the end of the semester, the primary question became: how effective is the VL setup as a pedagogical resource for application development. To address this question, we administered a survey based on the theory of TAM’s (Technology Acceptance Model) key efficacy constructs: “ease of use” and “usefulness” (See Figure 1) (Gallivan, 2001; Chircu et a., 2000; Straub et al., 1997).

Figure 1: Technology Acceptance Model  (Chin, 2000)
TAM has been used to explain the selection of an IT tool or technology to support the teaching of a technical IT course (Meso and Liegle, 2005) and to compare particular technologies or skill sets with respect to predefined outcomes such as subject’s productivity, subject’s cognitive performance and some output artifacts (e.g. higher quality analysis diagrams, higher quality program code, etc) (Basili et al., 1999; Burton-Jones and Meso, 2002; Havelka, 2003; Howard et al., 1999; Wang, 2003; Moris et al., 1999; Vessey and Conger, 1994).
Within the context of an IT course, we expect that students will be attracted to a technology that is easy to use and directly relevant to the course requirement tasks that they must complete, or to a technology that they perceive as bearing these traits. Therefore, assessing the reactions of students toward a particular technology can determine the effectiveness of that technology as a pedagogical tool for the course in question (Meso and Liegle, 2005).
In past TAM studies, the ease of use and usefulness variables have been operationalized as either perceived or actual measures (Davis, 1989; Deane, Podd, and Henderson, 1998; Henderson and Divett, 2003; Szajna, 1996;). Perceived measures have been more frequently employed than actual measures (Deane, Podd, and Henderson, 1998) and are said to be appropriate in situations where users have yet to use the technology, in other words, pre-implementation (Deane, Podd, and Henderson, 1998). Therefore, we selected to use perceived measures in this study.
TAM as a tool is designed to evaluate a technology in isolation. However, based on preliminary observations and past experience with teaching this type of course, we expect a number of factors to have an influence on a student’s perception of the usefulness and ease of use of the VL. In particular, some students would have successfully installed a web-server at home or on their laptop, while others would fail to do so for various reasons. Anecdotal experience from past semesters pointed to the expectation that the latter type of students would not be able to complete individual assignments and would heavily rely on teammates for group assignments. We expect that students who were unable to install a web server would find the virtual lab particularly useful and perceive it as being easy to use. On the other hand, students with their own web server would find a virtual lab more cumbersome and therefore would rate it lower on both ease of use and in particular usability. This is due to the fact that the virtual lab a) requires additional steps to be taken to connect to the virtual server, b) will be slower in terms of performance than most student’s own web server, c) involves things like VPN (virtual private network) to bypass campus firewalls for use at home, and d) may have a layout/configuration that is different from what they are used to. Therefore, as a second part of the study, we adres the question what factors influence the perception of the ease of use and usefulness of the VWS.
a) Hypothesis
The flexibility of the virtual lab, which not only enables instructors to use standard electronic classrooms for in-class exercises, but also allows students to log-into them from anywhere, suggests that this environment would be perceived as extremely useful. In addition, we expect them to find the virtual computers to be easy to use. Based on this, we hypothesized as follows.
Students will rate the virtual machine as
H1: useful for individual exercises
H2: useful for group projects
H3: easy to use for individual exercises
H4: easy to use for group projects
b) Research Instrument
We used a questionnaire (See appendix 1) to collect the data for evaluating the VL. The questions used in the questionnaire were derived from standard TAM questions (Meso & Liegle, 2005; Gallivan, 2001; Chircu et al., 2000; Straub et al., 1997) and further included standard demographic questions. Finally, data about the specifications of the primary computer used by each student and how – if at all – they had configured their web server was also collected.
c) Subjects
The subjects for this study were drawn from two related graduate courses. One course (Principles of Web Design) had 13 respondents, while the other (Web application development) had 10 respondents. There was no significant differences in the composition of the two sections with respect of gender, major (see Appendix 2), IT related work experience, years of programming experience, or years of web-design experience (see Appendix 3). There was also no difference across the groups in their programming experience with relevant languages (Java, VB, C, and C#, see Appendix 3). Therefore, the data from both sections was merged for the analysis of the study’s hypotheses.
Students have the option to download Visual Studio.NET and WindowsXP upgrade for free as part of the Microsoft Academic Alliance (MSDNAA) program. Out of 23 subjects, 78% (18) downloaded the software from MSDNAA, 13% (3) purchased it, and 4% (1) did not respond to this question.
The vast majority (91% or n=21) reported that they installed Visual Studio themselves, 4% (n=1) had it pre-installed, the rest (4% or n=1) did not respond. To allow the development of web-applications, Internet Information Server (IIS) needs to be installed and configured. Of the respondents, 78% (n=18) reported that they configured it themselves, while 9% (n=2) relied on 3rd parties for the configuration. Similarly, both IIS and Visual Studio need to be configured to work with the .NET 1.1 framework. Here, 74% (n=17) reported that they configured it themselves, while 9% (n=2) relied on 3rd parties.
d) Statistical methods
To test hypotheses H1-H4, we report the results of the survey (Lickert-scale) of the related questions, while for hypothesis H5-H8 we used the non-parametric two-sample Kolmogorov-Smirnov test and further regression to analyze the differences in terms of “haves” and “have-nots”. The Kolmogorov-Smirnov test is used to compare means in case of small sample sizes. Our study had only 23 subjects, making it appropriate for this test. The regression method enables researchers to identify the factors that most contribute towards the variance in the dependant variable. In our case, the dependent variable is ease of use and usefulness. The independent variables are students programming experience (in years), whether the student had a running web server (t/f), and his/her computer’s sophistication. As a proxy for the sophistication of the computer, we used the amount of RAM the student had installed. Other questions that we asked to use as a proxy were age (years) and processor (type/speed). Unfortunately, very few students knew much about their computer, but nearly everyone knew the amount of RAM they had. A visual inspection of the data showed that newer, faster computers also were equipped with more ram, while older ones used less ram. We therefore feel comfortable with using RAM as a proxy.
4.  RESULTS AND DISCUSSION
4.1 Perception of VL
We used the Lickert-scale scores provided by the subjects to assess the first four hypotheses. Specifically, we determined the mean of the responses for each respective hypothesis. The results indicate that students found the virtual computers most useful for individual exercises, and somewhat useful for group projects. Similarly, but with lower scores, students found the virtual computers easy to use for individual exercises, and to a much lesser extend, for group projects. The specific mean (standard deviation) ratings were as shown in Table 1.
While these results could be interpreted as marginal, one has to consider that currently there are only a small number of virtual machines available per section, meaning that not every student has his/her own virtual machine. Additionally, they are configured that they do not support multiple concurrent logins, which limits their use for group-projects in that only one person can be logged in at the same time. Further, the mean scores were all above 3 (neutral) and mostly leaning towards 4 (agree) – indicating that students felt that the virtual computers were easy to use or very useful respectively. Therefore, there is marginal support  for hypothesis H1-H4.
Table1: Mean and Standard Deviation
of questions 16-19.
Hypo-
thesis
Question
Mean (StDev)
H1
VL useful for individual exercises
4.00 
(0.97)
H2
VL useful for group projects
3.50 (1.10)
H3
VL easy to use for individual exercises
3.67 (1.03)
H4
VL easy to use for group projects
3.12
(0.99)
Dependent variable
Model strength
Coefficients of independent variables
#
Question
r2 (p)
PExp.
WSxPExpy#
WS
WSxComp#
Comp
16
ease-of-use
(in-class)
5.66 (.093) *
5.526 (.005)**
-5.466 (.006)**
-0.640 (.224)
3.746 
(.012)**
-3.241 (.009)**
17
ease-of-use
(group work)
.716 (.024) **
5.292 (0.002)**
-5.345 (.002)**
-.524 (.242)
3.370 
(.010)**
-3.823 (.004)**
18
usefulness
(in-class)
.745 (.009) **
6.226 (.000) **
-6.152
 (.000) **
-.474 (.239)
3.860 
(.002) **
-4.193 (.002) **
19
usefulness
(group work)
0.626 (.049) **
5.381 (.004) **
-5.459 
(.004) **
-.268 (.571)
2.906 
(.028) **
-2.994 (.013)**
Table 2 :  Questions & adjusted beta (p-value) significant at p<=.1 *  p<=.05 **
Legend:PExpy (PROGY): Programming experience in years
WS (RUN): True/False whether subject has personal web server
Comp (RAM): Power of subject’s computer in MB of RAM
#:  Interaction effect variables
While these results could be interpreted as marginal, one has to consider that currently there are only a small number of virtual machines available per section, meaning that not every student has his/her own virtual machine. Additionally, they are configured that they do not support multiple concurrent logins, which limits their use for group-projects in that only one person can be logged in at the same time. Further, the mean scores were all above 3 (neutral) and mostly leaning towards 4 (agree) – indicating that students felt that the virtual computers were easy to use or very useful respectively. Therefore, there is marginal support  for hypothesis H1-H4.
4.2 Factors influencing the perception of the VL
We run four regressions to identify what factors significantly influence student’s perception of the VL. The dependent variable was ease of use or usefulness for individual or group-work assignments respectively. The significant independent variables were programming experience, the power of one’s home computer, and whether one had a running web server on it. We also determined the interaction terms of these variables. Tables 2 and 3 summarize the results.
The results indicate that simply having a running web server does not influence a student’s perception of the VWS. However, the extent of programming experience (in years), and the power of the computer owned by a subject seem to have a significant influence on the subject’s perception of the virtual lab. These observations were consistent for all four regressions.
A closer examination of the regression models revealed that there were significant interaction effects between programming experience and the presence of a running web server on a subject’s home computer; and also between the power of a subject computer and the presence of a running web server on that computer. This necessitated a refined interpretation of the regression models.
For those students that have a web server running on their personal computer, programming experience had no real influence on their perception of the VL. However, programming experience significantly impacted the perception for those students that had no running web server at home. These results indicate that the more programming experience a subject had, the more they appreciated the VL. We interpret that as that “novice” students may not have had the knowledge to take advantage of the VL.
Table 3: Regression formulae with WS being either 0 or 1 (rounded)

Has personal web server
Has no personal webserver
Q16
E =  .5C  -.6
E = 5.5 Pexp – 3.2 C
Q17
E = -.4C  - .5
E = 5.3 Pexp -3.8 C
Q18 
U = - .3C -.5 
U = 6.2 Pexp – 4.2 C
Q19
U = -.1 C -.1 P - .3
U = 5.4 Pexp – 3.0 C
The results with respect to computing power were surprising, since we expected this to have no effect for students who do not have a running web server. Yet, in all four regressions, the computing power strongly and negatively influenced the student’s perception of the VL. Our interpretation here is that students tended to compare the performance of the VL to that of their personal computers. Therefore, students with powerful computers, perceiving the VWS as running more slowly than their personal computers, were inclined to rate the VL relatively lower. Those having slow personal computers, on the other hand, tended to rank it relatively higher, for the same reason. This explains the negative directionality of the influence of computer power on student’s perception.
5.  CONCLUSION AND FUTURE RESEARCH
Our initial assessment of the virtual lab is that it was a success. Students on average found it to be useful (3.75 out of 5) and easy to use (3.45 out of 5). The results, though preliminary, also indicate that the power of the personal computer, presence of a running personal web server, and programming experience have a significant influence on the perception of the students.
A limitation of this study was that it was conducted over a compressed time period (summer semester) with a small number of students, resulting in very few data points. Additionally, the limited use per subject of the virtual computers due to the fact that only a small number of virtual workstations were available for the two sections may have affected the power of the results.
Informal feedback from the students and the fact that they were able to do in-class exercises as well as group work showed that the system worked. Those who could use them, i.e. students with programming experience, in particular appreciated the virtual computers. In addition, those students who had less sophisticated computers at home and/or did not have a computer with IIS installed at all, appreciated the virtual computers and found them easy to use and useful. Since easy of use and usefulness are predictor variables for technology acceptance, we feel confident that the virtual servers will be a huge success.
The next step is to evaluate the system in longer-term studies with larger number of students to see how it performs “under load.” In addition, other system features such as the ability to install limited number of copies of special software to comply with licensing restrictions need to be tested and evaluated.
6.  REFERENCES
Basili, V., Shull, F., and Lanubile, F., (1999). “Building Knowledge Through Families Of Experiments,” IEEE Transactions on Software Engineering, Vol. 25 No.4, pp. 456-473
Burton-Jones, Andrew and Peter Meso, “How Good are These Uml Diagrams? An Empirical Test of the Wand and Weber Good Decomposition Model,” 23rd ICIS , 15 – 18 December 2002, Barcelona, Spain
Chircu, Alina M; Kauffman, Robert J, “Limits to value in electronic commerce-related IT investments, JMIS, Vol. 17 No 2, 2000, 59-80
Davis, F.D., “Perceived Usefulness, Perceived Ease-of-Use and User Acceptance of Information Technology,” MIS Quarterly, Vol. 13 No. 3, 1989, pp. 319-339. 
Deane, F., Podd, J., Henderson, R. “Relationship between self-report and log data estimates of information system usage,” Computers in Human Behavior, Vol 14, 1998, pp. 621-636.
Gallivan, Michael, “Organization Adoption and Assimilation of Complex Technological Innovations: Development and Application of a New Framework,” Database for Advances in Information Systems, Vol. 32 No. 3, 2001, pp. 51-85
Havelka D., “Predicting software self efficacy among business students: A preliminary assessment,” Journal of Information Systems Education, Vol. 14 No. 2, 2003, pp. 145-152
Henderson, R., and Divett, M. “Perceived usefulness, ease of use and electronic supermarket use,” International Journal of Human-Computer Studies, Vol 59 No. 3, September 2003, pp. 383-395   
Howard, G., Bodnovich, T., Janicki, T., Liegle, J., et. al. “The Efficacy of Matching Information Systems Development Methodologies with Application Characteristics - An Empirical Study.” Journal of Systems and Software, Vol. 45 No. 3, 1999, pp. 177-195
Janicki, Thomas N. and Liegle, Jens O. “Development and Evaluation of a Framework for Creating Web-based Learning Modules.” Journal of Asynchronous Learning Networks, Vol. 5 No. 1, June 2001, pp. 58-84
Liddle, J., Brown, K., Slater, A., MacDonnchadha, S. : “Utilizing Multiple Training Strategies within Intelligent Industrial Training Systems,” AI-ED 95, Washington D.C August 1995, 1-7
Liegle, J. and Johnson R. “A Review of Premier Information Systems Journals for Pedagogical Orientation,” Information Systems Education Journal, Vol. 1  No. 8, 2003. 
Liegle, J., Madey, G.: “A Classification of Programming Knowledge and applicable Teaching Strategies,” IEEE Transaction on Education, pp. 1-27, resubmitted Sept. 2003 (under review)
Meso, P and J. O. Liegle “An Exploratory Comparative Assessment of .NET as a Pedagogical Tool for Teaching Object-Oriented Systems Design ISECON 2002, v19 (San Antonio): 244a.
Meso, P. and J.O. Liegle “An Exploratory Assessment of the Pedagogical Effectiveness of a Systems Development Environment” Journal of Information Systems Education, Vol. 16(2), pp. 157-166, 2005
Morris, M. G., Speier, C., and Hoffer, J., , “An Examination of Procedural And Object -Oriented Systems Analysis Methods: Does Prior Experience Help Or Hinder Performance?” Decision Sciences, Vol. 30 No. 1, 1999, pp. 107-136
Vessey, I., and Conger, S., “Requirements Specification: Learning Object, Process, and Data Methodologies.” CACM, Vol. 37 No. 5, 1994, pp.102-113
Straub, Detmar W., Mark Keil and Walter Brenner, “Testing the Technology Acceptance Model across Cultures: A Three Country Study, Information & Management,” Vol. 31 No. 1, 1997, pp. 1-11
Szajna, B., “Empirical Evaluation of the Revised Technology Acceptance Model,” Management Science, Vol. 42 No. 1, 1996, pp.85-92.


Appendix
VIRTUAL INTERNET SERVER SURVEY
Participation in this survey is voluntary and anonymous. The results of this survey will only be published in summary form and be used to improve the teaching at GSU.

Your Background:
Standing: ?Graduate ?Undergraduate
Major: ? CIS(major/concentration/related)  ? Other: _____________________
Gender: ? Male ?Female ?Other
Years of IT related work experience:      _______
Years of programming experience: 	 _______
Years of web-design experience: 	 _______

Please rate your PRIOR programming experience  
[1= none, 2 little, 3 some 4 proficient, 5 expert, 6 N/A]
___ with Visual Studio
___  prior experience with virtual labs
___  ASP.NET 
Languages:	___  Java, 	___  C++, 	___  VB,	 ___  C#
Any Comments:

Please rate the Computer that you primarily use for programming assignments:

Location: ? PC at home       ?Laptop       ? PC at work ? University Lab	? other: 

Amount of RAM Memory
(i.e. 512 MB)

Processor type and speed
(i.e. Intel Celeron 2.4 Mhz)

(be as specific as you can. If “unknown”, specify age of computer or unkown)

If your main computer is at home/office, how did you obtain VisualStudio.NET?
? Downloaded from Microsoft (Academic Alliance)
? Bought software
? Came already installed
? Other: _________________________________________________

Regarding your primary computer
Yourself
3rd party
N/A
Who installed Visual Studio?:



Who configured IIS to work with Visual Studio?:



Who configured IIS/VS to work with .NET 1.1 ?





* Please turn over –
Please answer the following section if you obtained Visual Studio through the 
Academic Alliance download:

Strongly Disagree 
Disagree
Neutral
Agree
Strongly Agree
N/A
Accessing the download site was easy 
(ie finding site, getting password)






My internet connection is reliable 
(in regards of data loss/timeouts)






Downloading the software was easy






Installation of the software was easy






Configuring the software (i.e. IIS) was easy






Learning how to use Visual Studio was easy






Using Visual Studio was easy







My computer was very responsive with IIS and Visual studio running at the same time






It was easy to locate applications/files on my primary computer






It was easy to use my primary computer for group work






Please answer the following section if you have used the Virtual Servers
as part of your in-class/homework experience

Strongly Disagree 
Disagree
Neutral
Agree
Strongly Agree
N/A
Connection to the virtual lab is easy (i.e., mstsc)






My internet connection was reliable 
(in regards of data loss/timeouts)






The virtual lab computer was very responsive with IIS and VStudio running at the same time






It was easy to locate applications/files on the Virtual Computer






It was easy to upload/download data from the Virtual computer






It was easy to use the virtual computer for in-class exercises






It was easy to use the virtual computer for group projects






The virtual server is very useful for in-class exercises






The virtual server is very useful for group projects