Enhancing Decision Sciences Education
through Intelligent Tutors
Owen P. Hall, Jr., P.E., Ph.D.
ohall@pepperdine.edu
Graziadio School of Business and Management
Pepperdine University
Ken Ko., Ph.D.
Kenneth.Ko@pepperdine.edu
Graziadio School of Business and Management
Pepperdine University
ABSTRACT
Globalization is bringing about a radical “rethink” regarding the delivery of graduation management education particularly in the discipline of decision sciences. Today, many students entering an MBA program do not possess an undergraduate degree in business and thus have a limited background in decision sciences.  Furthermore, there is a growing trend toward de-emphasizing decision sciences in MBA programs. As a result, many business schools are turning to the Internet to provide “customized” instructional content to insure that graduates possess the requisite technical skills to meet the demands of the marketplace. Intelligent tutors represent one approach to address these challenges.  These systems provide customized learning and feedback based on student characteristics and performance.  The purpose of this paper is twofold: 1) to review the current direction in Internet-based intelligent tutors; and 2) to report on a preliminary evaluation regarding the performance and effectiveness of an intelligent tutor system to support decision sciences education. The tentative results from the study indicate that students that used an intelligent tutor in a decision science course significantly increased examination performance.
Keywords: MBA programs, decision sciences, learning support systems, intelligent agents.


1. INTRODUCTION
In this regard, the debate continues on the appropriate level of technical emphasis in graduate management education.  Many “new” MBA curriculums have reduced the number of credit hours for quantitative courses in favor of more managerial themes such as leadership, innovation, and information technology (Kleiman, 2007; Richards-Wilson, 2006; Bennis, 2005; Mangan, 2003).  The compelling argument is that management is inherently qualitative. Decision science educators recognize the need to “optimize” the limited amount of classroom time devoted to quantitative analysis courses (Arain, 2007; Shore, 2007). The task is determining the appropriate quantity and content level given a finite amount of space in the curriculum. This is particularly challenging for the widely varying educational and work experience backgrounds of students entering an MBA program.  This is where artificial agents can play a role. Today, the general pedagogical direction in management education is moving increasingly towards a learning-centric perspective (Driver, 2002). Accordingly, the roadmap to effective learning is a flexible and customized curriculum.  This perspective is predicated on the fact that many students do not come to a MBA program with the same technical, academic or work experience background. Therefore, providing self-paced, “customized” content, as part of the overall curriculum design can further enhance the learning experience. Furthermore, students tend to participate more in learning systems that are content-rich and that feature extensive variety (Neo, 2004).
The Internet has emerged as an effective platform for providing online instruction and content for the decision sciences discipline (Dutton, 2005; Suanpang, 2005; Grandzol, 2004). The nature and specificity of the technical content associated with the decision sciences tends to be more aligned with the current state of distance learning than in other disciplines such as organizational behavior and leadership (Arbaugh, 2007; Whittingham, 2006). The use of the Internet in this fashion provides the gateway to enhancing online learning through the use of intelligent tutors. These systems can deliver customized content based on the specific characteristics of the student (Cheung, 2003). These artificial intelligence based models are defined as purposeful autonomous entities capable of adapting to changing demands such as found in many unstructured and semi-structured educational applications. Intelligent agents allow the active reconfiguration of the learning presentation according to current requirements and the availability of information sources of varying quality. It is within this design context that the learning objectives can be best achieved. Intelligent tutors are seeing increased use throughout industry and government for delivering educational content (Gregg, 2007; Hubl, 2006).  In this regard, intelligent tutors are also receiving increased attention throughout academe with particular emphasis on management education (Graesser, 2005; Schleiffer, 2005).  Accordingly, students would benefit from exposure to these systems because they can assist in the learning process as well as orient them to this new class of technology as it relates to their future careers.
2. INTELLIGENT TUTORS
One key to effectively learning decision science principles is a customized lesson plan wherein the specific strengths and weaknesses of each student are identified and measured and appropriate feedback is provided. This is where intelligent tutors, a major branch of artificial intelligence (AI), can play a helpful role. Typically, mentoring agents used in this context are defined as pedagogical structures that provide information including suggestions to the learner (Ghaoui, 2004; Baylor, 2000). These systems can even take on human like qualities (Veletsianos, 2007). Among other things, agents can be used to design lesson plans and learning experiences based on student performance and background (Li, 2007).  For example, if a student is having difficulty mastering a particular subject or theme as detected by testing, simulation or self-assessment, the synthetic agent could prescribe specific additional learning content. This content can take the form of videos, computing tutorials or simulations. Agents also offer the capability to provide customized feedback based on student characteristics and performance in a semi-structured environment (Schleiffer, 2005).  Figure 1 illustrates the overall design structure of a mentoring agent as related to the learning process in a school of business. The student-agent interface would display previous performance as measured by examinations and simulations. These data can be used to construct a customized lesson plan as well as provide for specific learning suggestions as gleaned from a case-scenario consultation. The agent would also link to the overall curriculum to insure consistency.

Figure 1 – Learning Agent Design
Typically, intelligent tutors should possess four basic characteristics: autonomous, proactive, flexible and user-friendly.  The “social” interface between the agent and the student should be highly visual with limited user-required inputs. It is within this type of design context that the operational performance of the system can be achieved and maintained (Matsatsinis, 2003). The capability of providing customized content based on specific factors is particularly useful for students whose job assignments mirror the specifics found in the identified content. Used in this way a student can directly apply the lesson plan material to the workplace.
Table 1 shows more specifically how intelligent agents can be used to support a lesson plan involving linear programming (LP). The first column lists some basic learning plan objectives. The second column identifies the primary resources used by a student in connection with each session objective. The third column shows additional material identified by the instructor that is designed to support each lesson objective. For example, after developing a linear program solution using a virtual applet, a student could be directed to an interactive LP simulation to better understand the solution process. This type of capability is sufficiently general to evoke a wide range of interest among students. In addition customized content can be presented to the student based on both student performance and characteristics. For example, after reviewing and discussing a case on LP based capital budget allocation a student can choose to view specific budgeting applications.
Table 1 – Learning Agent Example – LP

Session Learning
Goals
Primary Aids
Support Aids
Appreciate the role of LP 
Lecture notes
Video
Understand LP inputs
e-chapter
Articles
Solve LP problems
LP applet
Sim Demo
Interpret LP solutions
Slides
Linear chatroom
Figure 2 shows sample prompts for a linear programming analysis learning consultation. The student is guided through a series of prompts regarding the case and explanations are provided for each prompt. Each consultation consists of approximately 20 prompts involving single-choice, multiple-choice, and fill-in-the-blank questions. The consultation can be taken more than once since, among other things, some of the prompts are randomized. The consultation also provides a PowerPoint recap at the end to further reinforce the basic principles. 
A similar LP learning resource allocation tutorial system is being used for assisting law enforcement in manpower allocation (Furtado, 2006). In this application, students allocate police resources to a selected geographic region using an interactive simulation. The student interprets the results of the allocation with the help of an intelligent tutor by observing how crime behaves in the presence of the policing.

Figure 2 – Linear Programming Tutor
The use of an intelligent agent based expert system is not limited to entering students requiring preparatory work but can also be used as a refresher by continuing students. For example, a student in a decision sciences course that is struggling with the notion of multiple regression could be directed to a consultative system that outlines the basic process. This type of learning construct has been used successfully in a variety of other business disciplines including the field of accounting (McDuffie, 2006). Specifically, an auditing expert system was constructed to assist students to better understand and apply Generally Accepted Accounting Principles (GAAP). The results showed that students who used this agent-based learning system performed better on course examinations. Expert systems have also been employed to assist students in mastering database design (Post, 2005). This system provides the student with the capability to create database designs and receive feedback in real time. Another agent, called AutoTutor, engages in a conversation with the student using three-dimensional interactive simulations (Graesser, 2005). Students that used this tutoring system demonstrated a nearly one letter grade improvement in course performance.
There are a number of reasons for using intelligent tutors throughout an MBA program including: 1) Assisting students to better understand the core material, 2) Serving as a vehicle to provide refresher material over the course of study, and 3) Exposing students to technology that they will likely encounter in the business community. Recent findings suggest that students who used intelligent agents learned more and had more confidence about subject material (Ingebretsen, 2007; Changchit 2003). These results indicate both the feasibility and effectiveness of the Internet-based intelligent systems as a substitute or supplement to traditional teaching methods in management education.  The following section presents the results of an analysis on the use of intelligent agents in graduate management decision sciences courses.
3. DATA ANALYSIS
A preliminary study was undertaken to assess the efficacy of learning agents in graduate management decision sciences courses. The following hypotheses were developed based on the foregoing literature review.
Hypothesis 1: Students using intelligent tutors multiple times had higher examination grades compared to those students who did not have access to these systems.
Hypothesis 2:  Students report that using intelligent tutors helped improve their understanding and mastery of the content.
These hypotheses were evaluated using two comparable graduate-level decision sciences classes.  In all five studies, only one section had access to the consultation systems as described in the previous section.  While students in the test group were encouraged to use the learning support system it was not mandatory. Table 2 summarizes the relevant information for the five studies. The final column reports the percentage that used the tutor.
Table 2 – Preliminary Study Information

Performance Metric
Intelligent Tutor(s)
% of Class
Midterm
Linear Programming
17%
Midterm
Linear Programming, Decision Analysis
77%
Final
Project Management
73%
Final
Project Management, Decision Analysis
40%
Each class module had learning modules which were supported by the corresponding intelligent tutor.  For example, the learning objectives for linear programming included:
* How to recognize resource management applications
* Understanding the role of LP in resource management
* How to formulate LP models
* How to solve LP problems
* How to interpret LP solutions
Each of these objectives was incorporated in the class lectures leading to the midterm examination. Table 3 highlights the statistics results from the examinations. Using t-tests, these data show that there is a statistical difference in performance between the control and test groups at the 0.05 level for all five studies:  p=0.047 for study 1, p=0.044 for study 2, p=0.028 for study 3, p=0.004 for study 4, and p=0.003 for study 5.  Care should be exercised in extrapolating this conclusion given the relatively small sample sizes of the test groups. Nevertheless, these results suggest the potential for enhancing the learning process.
For studies 1, 2, 4 and 5, the intelligent tutors spanned the entire examinations so the total possible for each group was 100 points.  For study 3, the intelligent tutor only pertained to one 25 point project management question on the examinations.
In addition to assessing student examination performance (Hypothesis #1) student responses were recorded on the perceived usefulness of the learning tutors (Hypothesis #2).  Table 4 summarizes student comments for selected questions. Performance assessment and student feedback, as illustrated in Table 4, will continue to be used as a basis for refining both the content and delivery process of this learning system.
Table 3 – Impact of Learning Agent Usage on Student Performance (SD = standard deviation, SS= Sample Size)


Mean
SD
SS
Control Group 1
88.0
6.2
29
Test Group 1
93.7
3.5
3
Control Group 2
82.7
7.6
28
Test Group 2
87.2
9.4
20
Control Group 3
19.4
6.3
30
Test Group 3
22.2
3.4
19
Control Group 4
89.3
7.2
27
Test Group 4
93.5
0.7
2
Control Group 5
79.6
15.6
20
Test Group 5
91.3
3.2
4


Table 4 – Selected Summary of Student Comments

Topic
Was the application helpful?
Comments
1
It helped reinforce what we had learned.
The questions help me understand LP much better.
2
The appl. was helpful; found it more challenging than book problems.
The questions are a little tricky; however, it is more interesting.
3
The appl. helped with review.
The questions covered both theory and application.
4
It is definitely a helpful exercise as it provides more review of the material.
Some questions were a bit unclear at first, but some thinking and the hints helped me understand.
4. CONCLUSION
Today, the trend in graduate management education appears to be de-emphasizing decision sciences courses in favor of more managerial themes like leadership, innovation and information systems. Furthermore, many students entering MBA programs do not have undergraduate degrees in business. These two dynamics suggest the need for specialized learning systems for assisting the student over the course of an MBA program. The purpose of this paper is twofold: 1) to review the current direction in Internet-based intelligent tutors; and 2) to report on a preliminary evaluation regarding the performance and effectiveness of an intelligent tutor system to support decision sciences education. Web-centric intelligent tutors, like the ones outlined in this paper, are particularly attractive for assisting non-business majors in MBA programs and for optimizing the decision sciences content given the limited amount of credit hours.  Specifically, intelligent tutors offer the student a customized learning experience and a gateway for effectively applying decision sciences analysis throughout the MBA curriculum. Intelligent agents are one promising technology to support the customization of the learning process in management education.  These tutorial systems provide learning content based on student performance and background characteristics. Similar systems are being used throughout industry to improve both productivity and effectiveness and can play a role in enhancing student learning.
A preliminary study was undertaken to assess the efficacy of learning agents in graduate management level decision sciences courses. The tentative results show that students that repeatedly used the tutors scored higher on examinations compared to students that did not use the tutors. The issue of the small sample size will be addressed as additional data is collected over the next several terms. Furthermore, students that used the tutors found them supportive and complementary to the course material as measured by the student responses. The current plan is to expand the availability of the intelligent tutors to other aspects of the decision sciences course (e.g., regression and decision analysis) and to extend the performance measurements to the final examination. Another research objective is to significantly increase the sample size by introducing the intelligent tutors to additional decision sciences class sections. The ultimate goal of this research is to better understand the role of intelligent tutors in decision sciences education.
5. REFERENCES
Arain, F.; Tipu, S. 2007. Emerging trends in management education in international business schools. Educational Research & Review, 2(12), 325.
Arbaugh, j.; Rau, B. 2007. A study of disciplinary, structural, and behavioral effects on course outcomes in online MBA courses. Decision Sciences Journal of Innovative Education, 5(1), 65.
Baylor, A. 2000. Beyond butlers, Intelligent agents as mentors. Journal of Educational Computing Research,, 22(4), 373-382.
Bennis, W; O’Toole, J. (2005). How business schools lost their way. Harvard Business Review, 83(5), 96.
Changchit, C. 2003. An investigation into the feasibility of using an internet-based intelligent system to facilitate knowledge transfer. Journal of Computer Information Systems, 43(4), 91.
Cheung, B.; Hui, L.; Zhang, J.; Yiu, S .2003. SmartTutor: An intelligent tutoring system in web-based adult education, Journal of Systems & Software; 68(1), 11.
Driver, M. 2002. Investigating the benefits of web-centric instruction for study learning: An exploratory study of an MBA course.  Journal of Education for Business, 77(4), 238.
Dutton, J., Dutton, M. 2005. Characteristics and performance of students in an online section of business statistics. Journal of Statistics Education, 13(3).
Furtado, V. 2006. A multi-agent simulator for teaching police allocation. AI Magazine, 27(3). 63.
Ghaoui, C.; Janvier, W. 2004. Interactive E-Learning. International Journal of Distance Education Technologies, 2(3), 26.
Grandzol, J. 2004. Teaching MBA statistics online: A pedagogically sound process approach. Journal of Education for Business, 79(4), 237.
Graesser, A. 2005. Auto Tutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4), 612.
Gregg, D. 2007. e-learning agents. Learning Organization, 14(4). 300.
Hubl, G.; Murray, K. 2006. Double agents.  MIT Sloan Management Review, 47 (3), 8.
Ingebretsen, M. 2007. Intelligent tutors making the grade. IEEE Intelligent Systems, 22(4), 8.
Kleiman, L. 2007. Giving MBA programs the third degree. Journal of Management Education, 31(1), 81.
Li, X. 2007. Intelligent agents – support online education. Decision Sciences Journal of Innovative Education, 5(2), 311.
Mangan, K., 2003. The new MBA: Business school professors shift programs to emphasize relevance and flexibility. The Chronicle of Higher Education, 12.
Matsatsinis, N.; et. al. 2003. An agent-based system for product penetration strategy selection. Applied Artificial Intelligence, 17(10), 901.
McDuffie, R.; Smith, S.; Murphy, L. 2006. Impact of an audit reporting expert system on learning performance: a teaching note. Accounting Education, 15(1), 189.
Neo, T., Neo, M. 2004. Classroom innovation: engaging students in interactive multimedia learning. Campus – Wide Information Systems, 21(3), 118.
Post, G. 2005. An Expert System Helps Students Learn Database Design. Decision Sciences Journal of Innovative Education, 3(2), 273.
Richards-Wilson, S. 2006. What every business school needs to know about its master of business administration. Journal of Education for Business., 82(2), 95.
Schleiffer, R. 2005. An intelligent agent model. European Journal of Operational Research, 166 (3), 666.
Shore, B.; Briggs, W. 2007. Competitive Analysis of MIS in the MBA Core: Are Trends Putting Pressure on the MIS Course? Journal of Information Systems Education, 18(1), 63.
Suanpang, P.; Petocz, P.; Kalceff, W. 2005. Student attitudes to learning business statistics: Comparison of online and traditional methods. Educational Technology & Society, 7(3), 9.
Veletsianos, G. 2007. The role of intelligent agents on learner performance. World Conference Proceeding on Educational Multimedia (Ed-Media), Montreal.
Whittingham, K. 2006. Impact of personality on academic performance of MBA students: qualitative versus quantitative courses. Decision Sciences Journal of Innovative Education, 4(2), 2006.