Student Perceptions of Information Technology Preparedness and Important Job Skills Belle Woodward, Assistant Professor bellew@siu.edu Susanne Ashby calgal@siu.edu Adam Litteken adaml@siu.edu Sue Zamora szamora@siu.edu Department of Information Systems and Applied Technology Southern Illinois University Carbondale, Illinois 62901, USA Abstract This study summarizes the analysis performed on the results of a survey which was administered to university students competing in a Midwest state and later a regional Collegiate Cyber Defense Competition. The purpose of the study was to ascertain students’ perceptions regarding their preparedness for the Competition as well as their perceptions concerning the most useful skill sets to bring to the competition, and the most helpful means of preparation. Descriptive data was collected regarding number of hours in preparation and number of courses taken and analyzed as a predictor of competition success. Participants identified as critical to success in the competition soft skills (with the exception of leadership) above technical skills. Key Words: Information technology, soft skills, IT education 1. INTRODUCTION 2. Annually the National Science Foundation Regional Center for Systems Security and Information Security Assurance Site sponsor the Collegiate Cyber Defense Competition (CCDC). This competition creates, as closely as possible, a realistic corporate administration and security experience as a way to measure the ability of a team of college students to operate, secure, manage, and maintain a corporate IT network. Preparedness is crucial to success in this competition. This preparedness is not just in the knowledge of network infrastructure and hardware or the technical skill needed to operationalize software applications; this preparedness also concerns the ability to think critically, communicate clearly, and to function with one’s colleagues as a cohesive team. During one state and one regional competition, members of the participating teams were surveyed regarding their perspectives of preparedness for not only the competition, but also for the workforce. Despite the small, select sample size, these survey results have implications for postsecondary curriculum and workforce skill sets. 2. COMPETITION OVERVIEW The fundamental task of the CCDC is to design and implement a network to provide specified information technology (IT) services for a fictitious company. Once implemented, each team must then operate the system and defend it against an onslaught of cyber attacks and other naturally occurring phenomena. The competition begins at the state level with winners advancing to the regional level. The first place team from each region then advances to the national competition. CCDC competitions are unique in that they focus on business operations while incorporating the operational aspects of managing and protecting an existing network infrastructure. Teams of students assume control of an "operational" network from a fictitious business complete with email, web sites, data files, users, and network traffic. Each team must integrate the correction of network problems and the performance of typical business tasks while defending their networks from hostile activity, just as companies worldwide do everyday. Due to the team environment where members each bring their own specialized knowledge to competition, students require not only technical knowledge and skills, but also “soft skills” that is, people skills. These soft skills include knowledge of group dynamics and how to work collaboratively, motivational techniques, and group decision-making processes as well as mediation and negotiation skills. 3. LITERATURE REVIEW In his November, 2006 address during the Inaugural Civic Scientist Distinguished Lecture Series, Dr. Arden L. Bement, Director of the National Science Foundation, emphasized the necessity of hands-on learning experiences for students in our nation’s quest to elevate learning beyond the rote methods of earlier generations. He stated, “With hands-on learning, the student, or team of students, is presented with a problem to solve. On the student’s part, this automatically evokes a series of questions which call upon creativity, cumulative learning and innovation” (Bement, 2006). Such a hands-on approach places problems into highly contextualized situations. Learners work collaboratively while actively investigating these situations. The learners build new knowledge upon prior knowledge while striving to find the most appropriate solutions. This type of meaningful learning task encourages independent investigation as it fosters the ability to adapt quickly to novel situations. These hands-on learning situations are also known as authentic learning, problem-based learning, project-based learning and apprentice learning. Using a variety of approaches, each seeks to place the novice into a community of practice for the purpose of engaging them in real world activities. These genuine activities are specially designed to scaffold learning while insulating the learners from extraneous experiences not germane to the tasks at hand that could impede their learning (Stein, Isaacs & Andrews, 2004). Authentic learning provides a forum for problem solving that activates critical thinking skills at the highest levels while facilitating reflection-in-action (Schön, 1987). Upon examination of professionals at work in their fields, Schön determined that a competent professional inquires into a problem, attempting to find a solution to the problem as it is initially set. The professional, remaining open to the possibilities that incongruence might develop with the original problem’s setting, thus, steps directly into the problematic situation for the purpose of imposing a frame on the problem. All the while, the professional is cognizant of any consequences that could occur due to this reframe and is willing to reflect during this action (reflection-in-action), formulate new hypotheses, and test these hypotheses through further action (followed by further reflection). These actions function to frame and re-frame the problematic situation as well as explore new junctures or phenomena as they surface for further consideration and reflection until the problem is adequately solved. This “professional artistry” (Schön, 1987, p. 22) is the type of proficiency exhibited in “unique, uncertain, and conflicted situations of practice” (p. 22). Developing these professional skills in novices can be accomplished during authentic learning (i.e., a hands-on experience) using the “reflective practicum” (Schön, 1987, p. 157) during which the instructor facilitates learning through coaching techniques that embody and model reflection-in-action. Schön’s model also underscores the important role a more experienced professional (such as a faculty advisor, technical lead or mentor from the field) plays in developing competence in novices. Framing the learner within the authentic learning experience is the community of practice (Wenger, 1999). It is within this frame that the novice acquires greater non-technical skills such as teamwork and collaboration, problem solving, decision-making and “soft skills” or skills related to effective communication practices used when interacting with co-workers and clients (Hissey, 2000). As students enter the workforce, professional expectations well exceed those of core technical skills to include those skills that will enable them to successfully engage with others in the business environment, inclusive of technical as well as non-technical professionals. (Hissey, 2000; Bunt, McAndre & Kuechel, 2005; Jenkins & Wolf, 2005). Many researchers have clearly identified five core soft skills proposed as required curriculum for all concentrations of Information Systems (IS) students: writing, working in a team environment, delivering presentations, managing projects, and developing interpersonal relationships (Todd, McKeen & Gallupe, 1995; Denning & Dunham, 2001; Noll & Wilkins, 2002; Newton, Hurstfield, Miller, Page & Akroyd, 2005). According to a research study conducted by Winterbotham, Adams & Hasluck (2001), employers regard candidates lacking technical skills as trainable provided these same candidates possess soft skills and other qualities deemed as positive by the employer. In preparing for the workplace, this creates an obligation on the part of the student to present as a key part of their employable skill set these non-technical skills. O*Net Online, an occupational information network (http://online.onetcenter.org/find/), provides research on the importance of skill sets for many occupations. For example, computer security specialists engage in careers which demand abilities and activities requiring reasoning, time management, writing, oral and written comprehension, critical thinking, judgment and decision making, and flexibility. A lack of soft skills can jeopardize the success of a business and is therefore a critical factor for viability. Acquiring these skills in the classroom is more likely to happen if the student engages in activities that promote the use of such skills. Two key goals of the CCD Competition include not only to provide an educational venue in which students are able to apply the theory and skills they have learned in their course work, but also to foster a spirit of teamwork, ethical behavior, and effective communication both within and across teams. According to a 2006 job outlook survey conducted by the National Association of Colleges and Employers (http://www.naceweb.org), employers are reporting that skills most sought after are soft skills such as communication skills, teamwork skills, interpersonal skills, motivation, initiative, detail orientation, organization skills, and leadership skills. 4. METHODS To gather data regarding students’ preparation for and experiences during the Collegiate Cyber Defense Competition, a researcher-designed survey was administered during two CCDC competitions: one at the state level (n1=50) and one at the regional level (n2=33). The participants surveyed were the CCDC’s undergraduate student competitors. The survey questions were forced choice, Likert-scale with some open-ended. Questions addressed student participants’ demographics, preparation for competition, perceptions of skill requirements, learning expectations of the event, and perceptions on competition experiences. During one state competition conducted at a Midwest university over a period of three days, the survey was administered on the second day as part of the competition through a business inject. Monitored by a member of the competition event staff, the paper-pencil formatted survey was completed in a postsecondary classroom setting. The participants were allowed four hours to complete the survey in order to ensure that the participants did not feel rushed. At the end of the time period, the surveys were collected and given directly to the researchers. The regional competition also took place at a Midwest university. The survey administered at this competition was also given as part of the competition’s business inject. Participants (n2=33) completed the survey via an online website previously programmed and tested for accuracy by the researchers. The data was automatically tabulated and formatted directly into the statistical analysis software program SPSS v.15. The descriptive data was analyzed in regard to frequency of responses by the two groups: state competitors (n1) and regional competitors (n2). Means and standard deviations are given within each of the tables that follow. By separating the state competition results from the regional results, researchers could analyze not only the difference between the teams’ levels of success (all teams in the regional competition had won their state competition), but also the differences in the teams’ preparation and importance place on various skill sets as they moved from the state level of competition to the regional level. 5. RESULTS Participants were asked to rank skill sets in terms of importance to competition success. They were provided a listing of six skills delineated by the literature as critical to success in the IT workplace: communication, leadership, thinking, teamwork, personal attitude and motivation, and technical. They were asked to rate each on a scale of 1(not important) to 5 (extremely important). Table 1 (see Appendix A) displays the means and standard deviations for the importance ranking by participants at the state competition. These ratings indicate the competitors’ perspectives on the second day of competition. At the regional level, all soft skills were rated higher than technical skills. Table 1 (see Appendix A) provides the means and standard deviations for each skill rated by the teams from the regional competition. All of these teams were teams that had already been successful at the state level. This group of teams rated all soft skills ahead of technical skills with the exception of leadership skills. To ascertain the participants’ level of coursework experience in internet security, competitors were asked to indicate the number of classes they had taken prior to competition. Over half (60.0%) the participants in both competitions had completed two or more courses regarding internet security. A lower percentage (54.5) of the participants in the regional competition indicated they had completed two or more courses in internet security. Participants were asked to consider the best means of preparation for the competition. All participants were given a list of five activities that have been noted in the literature as being typically used in preparation for such competitions: faculty advisor, individual research, coursework, internship and other. Participants were asked to rate on a scale of 1 (most helpful) to 5 (least helpful) what they used to prepare for the competition. Participants in both the state and regional competitions rated the faculty advisor and their own individual research as the two most helpful means of preparation. To better understand participants’ perspective regarding their preparation, regional competitors were asked to identify from a list of experiences one preparation experience they wished their school had offered or had offered more of during their training time. The following list was culled from the literature describing typical training experiences of team’s involved in such competitions: teamwork building, hands-on experience, project management, communication skills and thinking skills. Participants at the state level were asked the same question however they were allowed to select more than one answer. Most participants from both groups (73% of the respondents) indicated they wished their school had offered more hands-on experiences. The number of respondents asking for teamwork building (6%), communication skills (3%), thinking skills (6%), and project management (12%) were comparatively much smaller. To ascertain the amount of time participants perceived was necessary for adequate preparation and its relationship to success in the competition, respondents at the state competition were asked to estimate the average number of hours their team spent in preparation. Based upon these estimations, the most successful team did not spend the most time preparing. Analyzing these data using a Pearson Correlation (?= .01, two-tailed), no significant relationship was indicated between the number of hours a team spent preparing for the competition, and the overall success of the team in the competition. When the competitors were asked if they believed that the hours they spent preparing were adequate, nearly 32% of state competition respondents and 30% of regional competition respondents indicated the amount of training time was adequate. Respondents at the regional competition were asked how they believed their team performed during the competition. Although the teams were unaware of their scores at the time the survey was administered, the most successful teams indicated the greatest amount of confidence in their level of performance. This descriptive data is displayed in Table 2 (see Appendix A). Analysis of this data using a linear regression (p = .000) as displayed in Table 3 (see Appendix A) indicates that a team’s success can be predicted from their level of confidence in their self-identified level of performance. 6. DISCUSSION The CCDC is not only a good test of a person’s technical skills, but it also stresses the team’s ability to use their soft skills to ensure that their technical skills are being used in the most effective manner. In fact, technical skills were rated by the state competition respondents as the least important and the second least important (with leadership skills rated the least important) among the respondents at the regional competition. The most successful teams indicated that without the commensurate soft skills their technical skills were not as effectively utilized. A team can have members with great technical knowledge, but without the ability to communicate their information regarding the surrounding problems that inevitably arise; the team will not perform at their highest potential. Each competing team possessed a different set of strengths and weaknesses. By stressing soft skills during preparation time, a team can learn to enhance its strengths and minimize its weaknesses. The data analysis of the survey results from the state-level competition indicated that a team’s overall success in the competition was related to the importance they placed on technical skill. At the state competition, the teams who perceived their greatest strength to be technical skills were the two teams that performed best. The regional competition, however, yielded counter results. Bearing in mind that the teams from the regional competition had all been the most successful teams at their respective state competition, almost twice as many respondents selected teamwork over technical skills as their team’s greatest strength. This indicates that through the experience of the competitions, these participants realized the importance of collaboration to the entire process as well as the communication skills critical to successful teamwork. When preparing for such competitions, the research on team formation, group dynamics and leadership should be heeded by faculty advisors. Greater emphasis should also be placed during practice time on intra/inter-personal communication techniques. Simulation activities that incorporate cooperative learning techniques would also provide increased awareness among team members in how to work collaboratively. Teams should also be encouraged to spend time together in non-competition-related activities in order to establish and strengthen camaraderie. As the results indicated, a team’s success does not necessarily depend on their amount of classroom experience. In fact, the competitors in the regional competition indicated less class experience than the competitors in the state competition. Apparently, although helpful to have team members with a greater amount of class experience, there are many other areas from which students can gather useful information that enhances their ability to perform. The results showed that students from both the state competition and the regional competition indicated that their own research was the most valuable to their preparation for the competition, as well as the guidance they received from their faculty advisor. In future competitions it would be interesting to delineate exactly what type of research the individuals used as well as the source of this information. It would also be beneficial to know in exactly what ways their faculty advisor was helpful. Did the faculty advisor model critical thinking during problem solving sessions? Did the faculty advisor promote the use of additional resources for information gathering? Did the faculty advisor use certain instructional techniques to assist in gaining greater familiarity with the processes involved in this type of IT work, such as case study? Finding the answers to these questions would provide greater insight into the most beneficial types of advisor assistance. Overwhelmingly, respondents in these competitions indicated that hands-on experiences were greatly needed as part of their preparation. This type of learning requires that the teams have access to the most up-to-date equipment used by companies in the field. It also requires additional personnel at each school site to assist the faculty advisor in developing and running these high-tech scenarios in which students can practice. Oftentimes faculty rely too much on teaching theory and how to perform a function absent of application, dismissing the importance of experiential learning through which the student applies theory and knowledge directly into a real world situation. Experiential learning activities (that is, hands-on experiences) increase familiarity of technical components and their direct applications. This familiarity leads to increased confidence in one’s ability to manage the technical components in new situations. Experiential learning also increases one’s ability to problem solve provided that the activities are led by instructors who model the problem solving processes by thinking aloud along with the students as the solution is being sought out. As the results indicated the amount of preparation factored little into a team’s actual competition performance. Teams on average spent 120 hours preparing for the competition, yet less than 35% of all participants indicated that the amount of preparation time was enough. Knowing that the resource of time is greatly limited due to the participants’ status as students (participating in non-technology coursework, extra-curricular activities, etc.), developing confidence in their performance ability appears to play an important role in the team’s overall success. According to the data analysis, a relationship was indicated between the actual success of a team and how the team perceived their performance during the competition. This confidence could be related to regulatory thinking skills (such as metacognition) and not merely an inflated perception of their own abilities. Unlike physical sporting competitions, there is no scoreboard continually announcing the real time score of the game. At specified intervals throughout the CCDC, teams are provided their scores only for the last system review. The teams do not know the scores of the other teams during the competition and therefore, do not know how their score rates in comparison. Maintaining a high level of confidence in their knowledge and skills throughout the competition would perhaps lessen the potential for a breakdown in their ability to function as a team. 7. CONCLUSION Additional research of both a qualitative and quantitative nature needs to be performed in order to better ascertain such factors as problem solving skills, metacognition, and confidence upon success in such competitions. Related to these would also include additional research regarding the exact nature and type of experiential learning (hands-on experiences) that would provide the greatest learning per engagement time and faculty preparation time. As development and implementation of such high tech experiences is at times cost prohibitive, perhaps quasi-experiential learning activities such as case study could provide an equal benefit at a lesser cost. These case studies, of course, would be dependent upon the quality and applicability of each scenario to the skills, processes and knowledge required for internet security. Preparedness is not only paramount for each team as they design and implement a network to withstand an onslaught of cyber attacks; it is also critical to our nation’s security and economy. Universities are key to maintaining a workforce prepared to work not just immediately in today’s workplace, but also prepared for the workplace of tomorrow. As this workplace evolves at the speed of wireless connections, universities must not only include in their curriculum the hard skills of technical expertise in hardware knowledge and software applications, but also the soft skills of interpersonal communication, intrapersonal knowledge, leadership and collaborative skills that lead to a cohesive team. Woven throughout this knowledge are the skills of critical thinking and metacognition that facilitate the crucial reflection-in-action needed in a constantly evolving workplace. 8. REFERENCES Bement, A. L. (2006). “When the Jobs in the Nation Change, So Does the Job of the Scientist.” Inaugural Civic Scientist Distinguished Lecture Series presented at Baker Institute, Rice University, Houston, TX, November 9, 2006. Retrieved on July 4, 2006 from http://www.nsf.gov/news/speeches/bement/06/alb061109_rice.jsp Bunt K, McAndre, F, and Kuechel A, (2005). “Jobcentre Plus Employer (Market View) Survey 2004” Sheffield: DWP. Denning, P. J. & Dunham, R. (2001). The core of the third-wave professional. “Communications of the ACM,” November, 44(11), 21 – 25. Hissey, T. W. (2000). Education and careers 2000: Enhanced skills for engineers. “Proceedings of the IEEE,” 88(8), 1367 – 1370. Jenkins A, and Wolf A, (2005). Employers’ selection decisions: the role of qualifications and tests in Machin, S. and Vignoles, A. (Eds.) “What’s the Good of Education: The Economics of Education in the UK?” Oxford: Princetown University Press. Newton, B., Hurstfield, J., Miller, L., Page, R., & Akroyd, K. (2005). “What Employers Look for When Recruiting the Unemployed and Inactive: Characteristics, Skills and Qualifications,” DWP Research Report DWPRR 295. Noll, C. L. & Wilkins, M. (2002). Critical skills of is professionals: A model for curriculum development. “Journal of Information Technology Education,” 1(3), 143 – 154. O*Net OnLine. (2007). Retrieved July 24, 2007 from http://online.onetcenter.org/find/. Stein, S. J., Isaacs, G & Andrews, T. (2004). Incorporating authentic learning experiences within a university course. “Studies in Higher Education,” 29(239 – 258). Schön, D. (1987). “Educating the reflective practitioner.” Jossey-Bass: San Francisco. Todd, P. A., McKeen, J. D., & Gallupe, R. B. (1995). The evolution of is job skills: A content analysis of IS job advertisements from 1970 – 1990. “MIS Quarterly,” 19(1), 1 – 27. Wenger, E. (1999). Communities of practice: Learning, meaning and identity. New York: Cambridge University Press. Winterbotham, M., Adams, L. and Kuechel, A. (2001). “Evaluation of the Work Based Learning for Adults Programme since April 2001: Qualitative interviews with ES Staff, Providers and Employers, Sheffild: DWP.“ APPENDIX A ___________________________________________ Table 1 State and Regional Competition: Importance Ranking of Skill Sets Skill Set Sample Size Mean Standard Deviation State Regional State Regional State Regional Communication 45 33 4.53 4.61 0.625 0.496 Leadership 45 33 4.36 4.09 0.645 0.723 Thinking 45 33 4.64 4.76 0.529 0.435 Teamwork 44 33 4.77 4.73 0.424 0.517 Personal Attitude/ Motivation 45 33 4.31 4.49 0.668 0.619 Technical 45 33 4.29 4.39 0.695 0.704 __________________________________________________ Table 2 Team Regional Competition Performance Ranking Competition Sample Mean Standard Rank Size Deviation First 9 3.667 1.225 Second 8 3.625 0.744 Third 7 3.143 0.378 Fourth 9 3.000 0.867 Fifth 12 2.500 0.522 Total 45 3.133 0.894 Table 3 Regression Analysis for Confidence Levels and Competition Rank (N = 78) Variable B SE B ? Success -0.300 0.078 -0.505* *p < 0.05