Ask the Audience … Using Clickers to Enhance Introductory Business Statistics Courses Nicole Koppel koppeln@mail.montclair.edu Mark Berenson berensonm@mail.montclair.edu Department of Management & Information Systems Montclair State University Montclair, NJ 07043, USA Abstract Teaching introductory business statistics presents many challenges, including motivating students to engage in the learning process. This research presents a real-life application of incorporating radio-frequency clickers into the course delivery of introductory business statistics. These “clickers” are part of a student response system that allow students to answer questions and records their responses in real time during class meetings. Using clickers provides a fun and interactive way for faculty to get immediate feedback from students and gauge their level of understanding of class material. It permits the instructor to adjust the pace of the class according to students’ understanding. Clickers also allow students to gauge their own understanding of the material and provide an opportunity to respond to in-class questions anonymously. This paper discusses an application of using clickers in an introductory business statistics courses and an analysis of the results of a student survey on perceived clicker classroom usage, along with a discussion of the lessons learned. Keywords: Student Response Systems, Clickers, Business Statistics, Active Learning 1. INTRODUCTION Teaching introductory business statistics presents many challenges, including motivating students to engage in the learning process. Since business statistics is known for being a course that requires quantitative skills and provides challenging concepts, the majority of business students are, unfortunately, enrolled based solely on the curriculum requirements and lack the necessary motivation to thoroughly learn the material. Utilizing traditional teaching practices often exacerbates this problem. Typically, the instructor lectures and select students respond to questions while the majority, all too many of whom having weak or insufficient preparation in their mathematics, remain disinterested, do not understand, or do not pay careful attention. The instructor often does not realize the extent to which students do not comprehend the material until after grading student quizzes and exams. This research presents a real-life application to address some of these challenges by incorporating radio-frequency clickers into the course delivery of introductory business statistics. These “clickers” are part of a student response system that allows students to answer questions and records their responses in real time during class meetings. Using clickers provides what could be a fun and interactive way for faculty to get immediate feedback from students and gauge their level of understanding of class material. It permits the instructor to adjust the pace of the class according to students’ understanding. From the opposite perspective, the students are more alert in class since they are anticipating clicker questions. In addition, students using the clickers are able to gauge their own understanding of the material and thus they may ask more timely questions. Finally, students are more willing to respond to “clicker questions” because their answers are received anonymously, avoiding the potential for embarrassment that might occur if erring in the presence of peers. This paper will present an overview of the clicker-system, a review of the existing literature on incorporating clickers into the classroom, a discussion of our real-life application using clickers in introductory business statistics courses, the results of a student survey on perceived clicker classroom usage, along with the discussion of the lessons learned, future research, and our conclusions. 2. OVERVIEW OF THE STUDENT RESPONSE SYSTEM A student response system is a technology that allows an instructor to present a question or problem in class, allows students to enter their answers into a device; and instantly aggregates and summarizes the students’ responses. Clickers are the devices used by the students to enter the responses. Beatty (2004) provides an overview of student response systems. Although clickers have been used in classroom settings since 1985 (Beatty, 2004), the technology has improved and their use has become more widespread over the last few years. The clickers used in this study are the Interwrite PRS RF. They utilize radio-frequency technology, so line-of-sight with the instructor’s receiver device (similar to a flashdrive plugged into the instructor’s USB drive) is not necessary. Since each student purchases his or her own clicker, each clicker can be programmed with the student’s identifying information (name, student ID, etc.). As a result the instructor can use the data collected by the system to track student performance and attendance. The Interwrite PRS instructor software allows instructors to present “clicker questions” within a PowerPoint presentation. Each clicker has numeric buttons 0 through 9, a negative sign, and a decimal point. In addition, each clicker has buttons to indicate true/false and multiple choice responses – A, B, C, D and E. Therefore, questions can be numerical and require a mathematical calculation, true/false, or multiple choice. Figure 1 contains a picture of the Interwrite PRS RF clicker. Figure 1: Interwrite PRS RF Clicker Each time a question is presented to the class, students are given a pre-set amount of time to respond to the question using their personal clicker. While the question is being administered, the number of responses is counted so the instructor may choose to increase or decrease the amount of time needed to complete the question. The clicker would then send a radio-frequency signal containing the student’s identifying information and the student’s response to the instructor’s receiver device. At the end of the allotted time, the class results are tabulated and presented in the form of a vertical bar chart and the correct response is indicated. There is no public display identifying which students responded correctly or incorrectly. However, such information is saved on the instructor’s computer for later use. 3. LITERATURE REVIEW Since clickers have only been integrated into classroom pedagogy over the past few years, much of the literature on student response systems is still in its infancy. Existing literature focuses either on case studies of implementations exploring student feedback and/or student learning, or the literature focuses on the goals of incorporating student response systems in a classroom setting. Instructors in a variety of disciplines have been experimenting with clickers in the classroom. As examples, Nelson and Hauck (2008) discuss their experiences with clickers in a management information systems course, Taylor (2007) implemented a clicker experiment in an accounting course, McKenzie, et al (2006) described the use of clickers in a very large section of an introductory business statistics course, Presby and Zakheim (2006) utilized clickers in a quantitative methods course, and Hoffman and Goodwin (2006) implemented a clicker system for library instruction. Other studies, such as Mayer (2008) and Yourstone, et al (2008) explore whether or not students using clickers to reinforce topics were able to retain and transfer material better than students taught without using clickers. Nevertheless, much of the literature on student response systems focuses on the goals of incorporating them into the classroom. One of the main goals for implementing the system is to provide an active learning environment. Active learning refers to techniques that require students to actively process and apply the information to learn as opposed to passive listening in a typical lecture setting (Meyer and Jones, 1993). Active learning requires students to analyze, synthesize, and evaluate material resulting in better understanding and longer retention. One of the main goals of incorporating clickers into the classroom is to provide this active learning environment (Cunningham (2008), Hoffman and Goodwin (2006)). 4. CLICKER EXPERIMENT This “clicker experiment” was conducted during the Spring 2008 semester by the two authors teaching a total of four introductory business statistics sections in a School of Business at a mid-size state university on the East Coast. Of the four business statistics courses, three were day session undergraduate courses and the fourth section was an evening session MBA-level course. Each undergraduate class met twice a week during the day for one hour and fifteen minutes and the graduate class met once a week in the evening for two and a half hours. At this university, clickers are a fairly new phenomenon and less than 5% of the students have used clickers in previous classes. This had also been the first experience with the Interwrite PRS RF clickers for both of the instructors. Students purchased the required clickers through the university’s book store and were required to bring the clickers to each class meeting. All class presentations began with a PowerPoint slide containing a clicker question for the purpose of attendance. Depending on the lecture material, an additional two to five clicker questions were embedded into each Power Point presentation in the undergraduate class and an additional four to eight clicker questions were embedded into each Power Point presentation in the graduate class. The format of questions ranged from true/false to multiple choice to numeric analysis. The true/false and multiple choice questions were typically qualitative and would reinforce major concepts; however, the majority of questions did require numeric analysis since the statistics courses is quantitative and students needed to practice solving problems in order to comprehend the material. The amount of time provided to complete each question varied and the instructor was able to increase the available time as needed until the majority of the students recorded their responses via their clickers as indicating in a counter on the instructor’s computer. After each clicker question, the correct response and a vertical bar chart anonymously presenting all student responses were automatically displayed. Figure 2 shows a sample of a clicker question embedded into a PowerPoint presentation. Figure 2: Sample Clicker Question Embedded into a PowerPoint Presentation After each class meeting, the instructor software for the Interwrite PRS RF clickers allowed the instructor to generate reports pertaining to each student within a course section, recording whether or not the responses to each question in the class session were correct. Although it is possible to incorporate these results in calculating the students’ overall grades, in this “clicker experiment” the information was primarily tabulated for the purpose of attendance. At the end of the semester, students were asked to complete an anonymous on-line survey regarding their experiences with using clickers in their business statistics course as well as to express their opinions about the potential for using clickers in future business statistics courses. The 25-question survey instrument is presented in Figure 3 (see Appendix). 5. HYPOTHESES Given that the focus of this present study is to measure the level of satisfaction students experienced using the clickers in the classroom as well as to assess their opinions regarding the potential use of clickers to enhance learning, the following hypotheses are to be tested: H1: Students will find the use of the clicker to have been a positive classroom experience. That is, there will be more agreement than disagreement with respect to responses on the first set of questions regarding the use of clickers in the introductory business statistics course over the past semester. H2: Students will perceive that using the clicker in future introductory business statistics classrooms has the potential to enhance the learning of the subject matter. That is, there will be more agreement than disagreement with respect to responses on the second set of questions regarding the potential use of clickers in the introductory business statistics course. Given personality differences among faculty and the different approaches faculty use in teaching, a question should be raised regarding the impact of the instructor on student satisfaction with the clicker as well as on student perception of the potential use of the clicker to enhance learning. Thus the following hypotheses are to be tested: H3: There is no instructor effect on student satisfaction with using the clickers in an introductory business statistics classroom. H4: There is no instructor effect on student perception of the potential use of clickers in an introductory business statistics classroom. Given that differences in student “readiness” (i.e., age, experience, technological proficiency and maturity) may be observed in day session versus evening session introductory business statistics classes or in undergraduate level introductory business statistics classes versus MBA-level introductory business statistics classes, the following hypotheses are to be tested: H5: There is no student “readiness” effect on student satisfaction with using the clickers in the introductory business statistics classroom regardless of course level (i.e., undergraduate versus MBA). H6: There is no student “readiness” effect on student perception of the potential value for using clickers in the introductory business statistics classroom regardless of course level (i.e., undergraduate versus MBA). 6. METHOLOGICAL ISSUES Two constraints on the data collection process affected the methodology used to test the aforementioned six hypotheses (H1 – H6): 1. The overall sample size available for this study. 2. The use of potential covariates that might reduce experimental error (Berenson, et al (1983)) and thereby provide for a more powerful analysis of the results. Total Sample Size: During the last week of the semester and prior to the final examination students in each of the four class sections were given a constrained time period window of three days to complete the survey on-line. Thus the potential total sample size for this study was limited to student enrollees in each of the four sections who had not withdrawn from class during the semester. Although the response rates to the anonymous on-line survey exceeded 90 percent in each of the four sections, the overall study size was limited to the 105 student respondents – 33 from the section N1, 27 from section N2, 26 from section MU and 19 from section MG. Classes “N1” and “N2” represent undergraduate sections taught by faculty member “N,” class “MU” represents an undergraduate section taught by faculty member “M,” and “MG” represents a graduate section taught by faculty member “M”. These four section sample sizes are certainly sufficient for making comparisons of survey responses across the four class groups but, unfortunately, a total sample size of only 105 students precludes the development of a useful factor analysis on the responses to a survey with as many as 25 questions (see, for example, Hair, et al (1995)) that would have attempted to cull out the important factors or characteristics pertaining to both clicker-use satisfaction and potential. Rather than a “formal” factor analysis, this study used the correlation matrix along with some preliminary descriptive statistical analyses to subjectively map particular survey questions into the important characteristics. Potential Covariates: Institutional Review Board policy on campus precluded the obtaining of demographic information on gender, major, cumulative grade point average and other measures that could have served as covariates to the analysis. The only covariates that could be obtained as part of the anonymous on-line survey were the (self-selected) student responses on the 7-point Likert scales to question 13 (I am a technologically savvy person) and question 14 (The clicker was within the limits of my technological ability) and for this purpose a covariate was constructed as the sum of this pair of responses. Unfortunately, however, preliminary analyses incorporating this covariate demonstrated significant multicollinearity with the four class sections used for comparing the Likert-scale ratings of both clicker experiences and clicker potential. Thus, in this study a “formal” analysis of covariance (ANACOVA) was not conducted. Instead, the above mentioned covariate was used to refine the data set. Only those students who considered themselves “technologically saavy” to some degree (i.e., those responding with Likert-scale scores of 5, 6 or 7 to question 13) and also felt “at ease” to some degree using the clicker (i.e,, those responding with Likert-scale scores of 5, 6 or 7 to question 14) were included as part of the final analysis comparing the results among the four class sections in an effort to strengthen the conclusions. Table 1: Some Useful Pearsonian Pairwise Correlations to Survey Questions Questions Pearsonian r * Understanding and Learning Q1 and Q2 0.862 Q1 and Q16 0.699 Q2 and Q16 0.748 * Enjoyment Q3 and Q17 0.655 * Need to Submit an Assignment Q4 and Q5 0.778 * Reduce Anxiety Q6 and Q7 0.844 Q6 and Q8 0.841 Q7 and Q8 0.870 Q18 and Q19 0.882 Q6 and Q18 0.660 Q6 and Q19 0.584 Q7 and Q18 0.679 Q7 and Q19 0.647 Q8 and Q18 0.661 Q8 and Q19 0.684 * Immediate Feedback Q9 and Q10 0.704 Q9 and Q20 0.519 Q10 and Q20 0.706 * Instructor Gauging and Reinforcing Q11 and Q21 0.644 * Potential Teaching Tool Q22 and Q23 0.826 Q22 and Q24 0.571 Q22 and Q25 0.826 Q23 and Q24 0.530 Q23 and Q25 0.846 Q24 and Q25 0.524 7. METHODOLOGY Given the two groupings of questions aimed at measuring the level of satisfaction students experienced using the clickers in the classroom (i.e., questions 1 through 12) as well as assessing their opinions regarding the potential use of clickers to enhance learning (i.e., questions 15 to 25), Table 1, extracted from the complete correlation matrix, is a listing of Pearsonian pairwise correlations of the 105 student responses to the on-line survey questions having similar characteristics. Each of these correlations is highly statistically significant. The above listing and labeling of characteristics of the clicker technology, obtained from the overall correlation matrix, is intended to act as a proxy for a factor analysis that could have grouped the variables representing the 25 questions in the on-line survey into a set of constructs – had the total sample size been much larger than 105. For formal analysis of the aforementioned hypothesis, we developed the necessary summated Likert scales based upon the clicker characteristics identified in Table 1. Table 2 provides a listing of the particular survey questions selected to form these two summated Likert scales. Table 2: Components of the Two Summated Likert Scales Characteristic Experience Potential Understanding and Learning Q1 Q16 Enjoyment Q3 Q17 Need to Submit an Assignment Q5 Reduce Anxiety Q8 Q19 Immediate Feedback Q10 Q20 Instructor Gauging and Reinforcing Q11 Q21 Potential Teaching Tool Q22 Note that each selected question represents one of the highlighted clicker characteristics displayed in Table 1. To examine Hypotheses 1 – 6 from an “exploratory” perspective, Table 3 provides the means on the 7-point Likert scales for each of the 12 questions in the first grouping and for 10 questions in the second grouping over all four class sections and Figure 4 provides a plot of the mean profiles for each of these class sections. Table 3: A Comparison of Mean Scores on 22 Questions Across 4 Class Sections Ques No. Mean N1 Mean N2 Mean MU Mean MG 1 5.52 5.00 3.62 4.74 2 5.42 4.85 3.35 4.58 3 5.97 5.63 4.31 4.84 4 5.82 5.81 4.65 5.53 5 5.76 5.56 4.23 5.16 6 5.42 5.59 4.19 5.58 7 5.58 5.52 4.04 5.21 8 5.36 5.52 4.12 5.32 9 5.85 5.74 4.58 4.89 10 6.03 5.67 4.73 5.37 11 6.52 6.11 4.62 5.21 12 5.33 5.15 3.92 4.74 16 5.67 5.81 4.08 5.42 17 5.67 5.44 4.58 5.63 18 5.85 6.07 4.65 5.79 19 5.91 6.04 4.77 5.89 20 6.09 5.63 4.50 5.89 21 6.12 6.19 5.00 5.89 22 6.06 5.93 4.12 5.58 23 5.39 5.67 3.50 5.05 24 5.94 6.22 4.96 5.95 25 5.85 5.81 3.96 4.95 Figure 4: A Comparison of Mean Profiles on 22 Questions Across 4 Class Sections H1 and H2: From responses to questions 1 through 12 as displayed in Table 3 and Figure 4 it is clear that students found the use of the clickers to have been a positive classroom experience. There was more agreement than disagreement with respect to responses on this first set of questions regarding the use of clickers in the introductory business statistics course over the past semester. The mean Likert scores on the 7-point scales exceeded 4.0 for all questions in three of the class sections (labeled N1, N2 and MG) and exceeded 4.0 on nine of the 12 questions in the remaining class section (labeled MU). Overall, 71.3 percent of the 105 total sample responses to each of the 12 questions demonstrated agreement with Likert scores of 5 (slightly agree), 6 (moderately agree) or 7 (strongly agree). Similarly, from responses to questions 16 through 25 as shown in Table 3 and Figure 4, there was also more agreement than disagreement with respect to responses on the second set of questions regarding the potential use of clickers in the introductory business statistics course with respect to the enhancement of learning. Again, the mean Likert scores on the 7-point scales exceeded 4.0 for all questions in class sections N1, N2 and MG and exceeded 4.0 on eight of the 10 questions in class section MU. Overall, 77.3 percent of the 105 total sample responses to each of the 10 questions had Likert scores of 5, 6 or 7—indicating some level of agreement. H3 and H4: A comparison of the mean scores across the four class sections, however, indicates a substantial instructor effect on student satisfaction with using the clickers in an introductory business statistics classroom as well as in the perception of the potential use of the clicker to enhance learning. From Table 3 and Figure 4, note that the mean scores for section MU, an undergraduate class taught by faculty member “M,” are lower on each of the 22 questions than are the corresponding mean scores for section MG, a graduate class taught by that same faculty member, as well as the corresponding mean scores for sections N1 and N2, undergraduate classes taught by faculty member “N.” To formally test the hypotheses H3 and H4, one-way ANOVA F tests with Tukey HSD multiple comparisons were employed, separately, using the summated Likert score ratings for a subset of six of the 12 experience questions shown in Table 3 and then using the summated Likert score ratings for a subset of six of the 10 perception questions, also shown in Table 3. The corresponding results are recorded in Tables 4 and 5 (see appendix). The results obtained in Tables 4 and 5 confirm a substantial instructor effect on student satisfaction with using the clickers in an introductory business statistics classroom as well as in the perception of the potential use of the clicker to enhance learning. Faculty member “N” was more successful in both using the clicker in the undergraduate classroom and in developing student perception of the potential values of the clicker than was faculty member “M.” Interestingly, faculty member “M’s” graduate class (MG) scored significantly higher than did the undergraduate class (MU) in the perception ratings of the potential use of the clicker but there was no significant difference in these two classes with respect to student experiences with the clicker during the semester. H5 and H6: What impact might student “readiness” (i.e., age, experience, technological proficiency and maturity) have on their responses to the 22 questions? From Table 3 and Figure 4 three clues emerge. First, note that the mean scores achieved by faculty member “N” are more homogenous than are the mean scores attained by faculty member “M.” One reason might be the discrepancy in both experiences (questions 1 through 12) and perceptions (questions 16 through 25) described in the ratings by the graduate students (MG) versus the undergraduate students (MU). Second, note that the discrepancies in responses between the two groups seem to widen with respect to mean ratings of the perception questions 16 to 25 as opposed to the experience questions 1 through 12. Perhaps the older, more experienced and mature graduate students can better assess the potential positive use of the clicker in the introductory statistics classroom even though their current experience over the semester was not so positive. Third, note the different pattern in the ratings achieved by faculty member “N” in both undergraduate sections displayed in Table 3 and Figure 4. Clearly, students in both sections had positive classroom experiences with the clicker during the semester and perceive its value as an enhancement to learning. However, upon closer scrutiny it is observed that students in section N1 enjoyed their experiences with the clicker more than did the students enrolled in section N2. The mean scores for section N1 were higher in 10 of the 12 experience questions. Nevertheless, with respect to the potential use of the clicker as an enhancement to learning, students in section N2 gave the higher mean scores in six of the 10 questions. Perhaps this is due to differences in student “readiness” in the two class sections. Unfortunately, the summated mean scores for questions 13 and 14 could not be employed as the aforementioned “readiness” covariate in a more formal ANACOVA. This is because a preliminary analysis showed these covariate scores were significantly different over the four class section groups. Such multicollinearity between the covariate and the class sections would only confound results and preclude more light being shed on any “among group” distinctions with respect to clicker experience or clicker potential. To formally test hypotheses H5 and H6 in order to evaluate the possible impact of “readiness” on student satisfaction with their experiences using the clickers and student perception of the potential value for using clickers, one-way ANOVA F tests with Tukey HSD multiple comparisons were conducted, as was previously done for testing hypotheses H3 and H4. The response variable was the same – the former used the summated Likert-score ratings for a subset of six of the 12 experience questions (see Table 3) and the latter used the summated Likert-score ratings for a subset of six of the 10 perception questions (see Table 3). Here, however, in an attempt to emulate ANACOVA, the “readiness” covariate previously described as the summated Likert-score ratings to questions 13 and 14 was used to reduce the previous data set by filtering out students whose responses indicated they did not perceive themselves to be technologically savvy or did not feel comfortable using the clickers in the classroom. Thus the ANOVAs on the reduced data set controlled for “readiness” and the conclusions drawn when comparing differences in satisfaction with clicker experiences or differences in perception of potential clicker value among the four class groups would be conditioned on those students who were “ready” to use them. The reduced data set contained 77 clicker “ready” student responses. However, to eliminate the possibility of any extreme responses affecting the analysis, within each of the four class sections both the highest and lowest of the summated-Likert scores were removed from the study. Following this “trimming,” the final reduced data set contained 69 student respondents – 22 from the section N1, 20 from section N2, 12 from section MU and 15 from section MG. The results of these analyses are presented in Tables 6 and 7 (see appendix). From Table 6, “controlling” for clicker “readiness,” it is clear that there is a highly significant instructor effect. Clicker “ready” students taking faculty member “N” were significantly more satisfied with their clicker experiences than were students taking faculty member “M.” Course level, undergraduate or MBA, did not matter. This new result, a refinement of what was found in Table 4, occurred by taking into account the clicker “readiness” of the student respondent. On the other hand, comparing the earlier analysis shown in Table 5 with the more refined analysis displayed in Table 7, the conclusion here remains the same. With respect to perception of potential enhancement to learning, the undergraduate students in section MU were significantly less confident than were the graduate students in section MG taught by the same instructor or the undergraduate students in sections N1 and N2 taught by the other instructor. Here, the “readiness”/maturity of the graduate students in section MG appears to have enabled them to separate their past clicker experiences from their perception of potential clicker value. The undergraduate students in section MU did not, or perhaps could not, do this. 8. LESSONS LEARNED The anonymous on-line survey evaluated students’ perceptions regarding their experiences with using clickers in their business statistics course as well as to express their opinions about the potential for using clickers in future business statistics courses. The major characteristics that the survey focused on were whether or not students felt that the clickers increased their understanding and learning of introductory statistics; whether students enjoyment of their statistics course increased as a result of clickers; whether or not students were more inclined to attempt in-class assignments as a result of the clicker; whether students felt less anxiety as a result of being able to respond to questions anonymously with their clicker; whether students benefited from immediate feedback based on their clicker submissions; whether students felt that the instructor gauged student learning and reinforced certain topics based on clicker feedback; and finally, whether students perceived clickers to potentially be a positive teaching tool. Based on the responses to the surveys, overall the majority of students did “agree” with each of these areas. However, the study did reveal that there clearly is a “teacher effect” and a “student readiness effect” that factor into student’s perceptions of clickers in the classroom. Overall, based on these results, there are a number of lessons learned. 1) Faculty must be adequately trained and prepared to use the clickers. If the faculty member is unable to properly present clicker questions and control the allocated time period available to complete each question, student frustration levels increase and their dissatisfaction with the clickers will also increase. 2) Clickers will not appeal to all students. Based on individual learning styles some students found clickers added anxiety and frustration. For example, if a student typically spend his or her class time simply taking notes that will later be used to absorb and understand the material, he or she may not be comfortable responding to questions in-class on new material just presented, but not yet absorbed. 3) The cost of the clickers may affect students’ perception of them as a classroom tool. In addition to purchasing a required textbook for the introductory business statistics course, students were required to purchase the clickers. The additional cost may have given students a preconceived negative bias. It should be noted that students did receive a significant rebate if the clicker was purchased as a bundle with the textbook. In addition, the campus bookstore agreed to buy back clickers at the end of the semester and sell them as used. 4) Using the clickers as an attendance tool was extremely helpful but it was not sufficient. Frequently, students forgot to bring their clickers to class. Although bringing clickers to every class was a requirement, instructors did not want to mark these students as absent since coming to class without a clicker was better than not attending class at all. As a result, manual adjustments to attendance listings were required. 5) Incorporating clickers into the introductory business statistics course requires substantial time. Not only is there an initial time investment required for preparing “clicker questions” and incorporating them into PowerPoint lecture notes, there is also a significant amount of class time utilized by the clickers. This amount of time may depend on students’ technological experience and “readiness.” During the first few class meetings, time is required to introduce the clickers, allow students to set-up their clickers and then teach students how to use them. Time is also required in the beginning of every class meeting to allow all students to turn on their clickers and join the section. Finally, and most significantly, much time is needed to allow students to respond to clicker questions. During a non-clicker class, an instructor may suggest that students attempt an in-class problem. When the instructor feels that a sufficient amount of time has elapsed, he or she may begin going over the problem even though students may not have completed the entire problem. In a “clicker classroom,” an instructor will need to wait until the majority of students have completed the problem and have submitted their response via their clicker. 9. FUTURE RESEARCH In addition to addressing some of the lessons learned, in future semesters clickers can play a greater role in the introductory business statistics classroom. The responses to questions can be better used to track student performance throughout the semester and incorporate responses into students’ course grade. Also, clickers can be used for in-class quizzes. In this case, students enter responses to questions but student responses will not automatically be displayed. Moreover, this clicker study can be enhanced by evaluating student performance and retention in a clicker class as compared to the performance and retention of students in non-clicker classes. 10. CONCLUSIONS Addressing the many issues inherent in teaching an introductory business statistics course has challenged instructors for decades. While incorporating a student response system has provided a means to further engage students in the learning process and motivate students to participate in class, using clickers is by no means a panacea. Overall, students found the use of the clicker to have been a positive classroom experience, but a number of students may have preferred a more traditional classroom approach. Based on our survey results, the successful use of clickers is dependent upon both an instructor effect and a student “readiness” effect. Clickers by no means can be a substitute for a well-prepared, motivated, and dynamic instructor. To incorporate clickers into the classroom, an instructor must be well-trained using the clicker software, must be prepared with contingency plans to circumvent unexpected technological problems, invest substantial time into preparing and presenting relevant clicker questions that would enable the instructor to gauge student learning and be flexible to adjust lesson plans, and know how to manage student issues pertaining to lost or forgotten clickers. When prepared and used properly, clickers reduce student anxiety, provide a fun and interactive classroom experience, provide students with immediate feedback on what they have learned, and provide instructors with a tool to gauge student learning and reinforce certain topics based on clicker feedback. 11. REFERENCES Beatty, I, (2004), “Transforming student learning with classroom communication systems.” Educause Research Bulletin. Vol. 3, pp. 2-13. http://net.educause.edu/ir/library/pdf/erb0403.pdf. Berenson, ML, DM Levine, and M Goldstein, (1983). Intermediate statistical methods and applications: A computer package approach. Englewood Cliffs, NJ: Prentice Hall. Cunningham, BM, (2008). “Using action research to improve learning and the classroom learning environment.” Issues in Accounting Education, Feb, Vol. 23, Issue 1,  pp. 1-30. Hair, JFJ, RE Anderson, RL Tatham, and WC Black (1995). Multivariate data analysis. (4th ed.). Saddle River, NJ: Prentice Hall Hoffman, C and S Goodwin (2006). “A clicker for your thoughts: Technology for active learning.” New Library World, London, Vol. 107, Issues 9/10. pp. 422-433. McKenzie, JD, RT Rogers, and S Taylor (2006). “Using clickers: A live survey of today’s first course in statistics.” Making Statistics More Effective in Schools and Business (MSMESB) Conference, Chicago, IL. Meyers, C and TB Jones (1993). Promoting active learning: Strategies for the college classroom. Jossey-Bass, San Francisco. Nelson, ML and RV Hauck (2008). “Clicking to learn: A case study of embedding radio-frequency based clickers in an introductory Management Information Systems course.” Journal of Information Systems Education. Vol. 19,  Issue 1,  pp. 55-64. Presby, L and C Zakheim (2006). “Enhancing student learning with only a click.” The Business Review, Cambridge, Vol. 6, Issue 1, pp. 153-156. Taylor, PS (2007). “Can clickers cure crowded classes?” Maclean's, Jul 9-Jul 16, Vol. 120,  Issue 26/27, p. 73. Yourstone, SA, HS Kraye, and G Albaum (2008). “Classroom questioning with immediate electronic response: Do clickers improve learning?” Decision Sciences Journal of Innovative Education, Jan, Vol. 6, Issue 1, pp. 75-88. APPENDICES FIGURE 3: On-Line Clicker Technology Survey Clicker Technology Survey ________________________________________________________________________Since your class experimented with the use of clickers in the statistics course we would like to know your candid assessment of their value as a tool for assisting you to learn. The first set of 14 questions attempts to assess your classroom experiences with the clicker this past semester. Assuming that the instructor will be properly trained and that there will be minimal technological glitches in using the clicker software, the second set of 11 questions attempts to assess your opinion on the potential value of this device in future classes. Instructions: Please answer the following questions concerning your use of the clickers this past semester according to this scale: 1 2 3 4 5 6 7 Strongly Disagree Moderately Disagree Slightly Disagree Neutral Slightly Agree Moderately Agree Strongly Agree _____ 1. Using the clicker enhanced my learning of the subject of statistics. _____ 2. The clickers helped me understand the material presented in this class. _____ 3. I enjoyed using the clickers in this class. _____ 4. I was more willing to attempt in-class problems because I knew I needed to submit a clicker response. _____ 5. I concentrated more in class because I knew I would have to respond to clicker questions on the material. _____ 6. The clickers were useful because I did not have to answer a question in public and perhaps embarrass myself. _____ 7. The clickers made me more confident to participate in this class. _____ 8. The anonymity of the clickers made me less fearful to be an active participant in class. _____ 9. When getting clicker questions incorrect, I realized that I did not understand the concept. _____ 10. Clickers gave me immediate feedback on whether I knew the material presented in the classroom _____ 11. My instructor used the results from clicker questions to gauge class understanding and reinforce material that was not understood. _____ 12. I would prefer more clicker questions during each class meeting. _____ 13. I am a technologically savvy person. _____ 14. The clicker was within the limits of my technological ability. Instructions: Please answer the following questions concerning your thoughts on the potential use of the clickers according to this scale: 1 2 3 4 5 6 7 Strongly Disagree Moderately Disagree Slightly Disagree Neutral Slightly Agree Moderately Agree Strongly Agree _____ 15. I would prefer to be in a class that uses clickers than in a class that does not use clickers. _____ 16. Appropriate use of the clicker should enhance the learning of the subject of statistics. _____ 17. Appropriate use of the clicker should result in a more enjoyable experience for the statistics class. _____ 18. Using clickers should reduce anxiety in the classroom because students’ answers to a question are not public and thus not potentially embarrassing. _____ 19. The anonymity of the clickers should make students less fearful to be active participants in class. _____ 20. Clickers should be valuable to the students because they can provide immediate feedback on whether they knew the material presented in the classroom _____ 21. Instructors should be able to use the results from clicker questions to gauge class understanding and reinforce material that was not understood. _____ 22. Clickers have the potential to be a valuable teaching tool. _____ 23. More faculty members should use the clickers. _____ 24. I believe that technology in general helps the educational process. _____ 25. I would recommend that clickers continue to be used in the introductory business statistics classes. Table 4: Comparing Summated Likert Scores on Clicker Experiences over 4 Groups SUMMARY: Summated Likert Score Ratings on 6 Questions Groups Count Sum Average Variance N1 33 1160 35.15 29.820 N2 27 904 33.48 73.182 MU 26 666 25.62 65.606 MG 19 582 30.63 45.690 ANOVA Source of Variation SS df MS F P-value Between Groups 1460.499 3 486.833 9.243 < .0001 Within Groups 5319.558 101 52.669 Total 6780.057 104 Tukey HSD Results: * N1 and N2 differ only by chance. * N1 is significantly better than MU. * N1 and MG differ only by chance. * N2 is significantly better than MU. * N2 and MG differ only by chance. * MU and MG differ only by chance. Table 5: Comparing Summated Likert Scores on Perception Potential over 4 Groups SUMMARY: Summated Likert Score Ratings on 6 Questions Groups Count Sum Average Variance N1 33 1172 35.52 25.070 N2 27 946 35.04 61.960 MU 26 703 27.04 60.759 MG 19 652 34.32 51.117 ANOVA Source of Variation SS df MS F P-value Between Groups 1277.118 3 425.706 8.861 < .0001 Within Groups 4852.272 101 48.042 Total 6129.390 104 Tukey HSD Results: * N1 and N2 differ only by chance. * N1 is significantly better than MU. * N1 and MG differ only by chance. * N2 is significantly better than MU. * N2 and MG differ only by chance. * MG is significantly better than MU. Table 6: Comparing Clicker Experiences over 4 Groups Controlling for “Readiness” SUMMARY: Summated Likert Score Ratings on 6 Questions Groups Count Sum Average Variance N1 22 801 36.41 16.825 N2 20 714 35.70 24.011 MU 12 338 28.17 31.061 MG 15 457 30.47 32.124 ANOVA Source of Variation SS df MS F P-value Between Groups 762.299 3 254.100 10.317 < .0001 Within Groups 1600.918 65 24.623 Total 2363.217 68 Tukey HSD Results: * N1 and N2 differ only by chance. * N1 is significantly better than MU. * N1 is significantly better than MG. * N2 is significantly better than MU. * N2 is significantly better than MG. * MU and MG differ only by chance. Table 7: Comparing Perception Potential over 4 Groups Controlling for “Readiness” SUMMARY: Summated Likert Score Ratings on 6 Questions Groups Count Sum Average Variance N1 22 818 37.18 13.299 N2 20 746 37.30 17.905 MU 12 358 29.83 31.061 MG 15 526 35.07 17.067 ANOVA Source of Variation SS df MS F P-value Between Groups 515.144 3 171.715 9.301 < .0001 Within Groups 1200.073 65 18.463 Total 1715.217 68 Tukey HSD Results: * N1 and N2 differ only by chance. * N1 is significantly better than MU. * N1 and MG differ only by chance. * N2 is significantly better than MU. * N2 and MG differ only by chance. * MG is significantly better than MU.