A Review of Premier Information Systems Journals for Pedagogical Orientation

Jens O. Liegle
and
Roy D. Johnson

Department of Computer Information Systems, 
Robinson College of Business, Georgia State University,
35 Broad Street, Atlanta, GA  30303



Abstract

Tenure, promotion, and merit pay at most universities are a direct function of the quantity and quality of a faculty’s publication. In particular, recent changes in the treatment of non-tenure track (NTT) faculty at a large south-eastern University suggest that for NTT faculty, whose assigned workload consists primarily of teaching responsibilities, the criteria for promotion are not only excellence in teaching, but also demonstrated evidence of teaching scholarship. This will likely apply to tenure track faculty at teaching institutions. Accomplishments in discipline-based scholarship of discovery are considered complementary, but not a substitute for accomplishments in the scholarship of teaching; however, publications in pedagogical journals are considered a strong form of public dissemination. 

This requirement was the motivation behind this study to review the premier Information Systems journals for pedagogical orientation. Sixty-one (61) top ranked IS journals were selected for the initial study based on the cumulative IS journal rankings by Saunders (2003). In an e-mail survey with telephone follow-up, journal editors were asked to identify the extent of the pedagogical orientation of their respective journals. Results revealed that with the exception of three (3) journals, these top-ranked journals published on average less six (6) percent or less of their articles with a pedagogical focus, and only two (2) journals declared a pedagogical focus. This means that there are very few outlets for IS pedagogy research in the leading IS journals, which can explain the difficulties that NTT and tenure-track faculty at teaching institutions have to gain promotion and/or tenure. This information may be useful to candidates for tenure, promotion, and merit, as well as those who serve on committees charged with making decisions about tenure, promotion, and merit.

Keywords:  survey, information systems, pedagogy, education, journal, tenure, promotion, merit


1. INTRODUCTION

There are many motivations for publishing in professional journals. While many authors wish to extend the knowledge base in their discipline, the majority of university faculty may have an even more compelling motivation: journal publications are easily quantified and are therefore taken into consideration as a measure of faculty productivity for decisions such as tenure, promotion, and pay raises (Newall & Price, 1983; Euster & Weinbach, 1986; McNabb, 1994; Marchant & Newman, 1994; Whicker, Kronenfeld, & Strickland, 1993).

According to Whicker, Kronenfeld, and Stickland, “the first commandment of tenure success is to publish” (1993, p. 138). At many institutions, publications are considered a major factor in decisions about promotion and tenure (Euster & Weinbach, 1986; Marchang & Newman, 1994). Even getting a job in higher education may hinge upon an established publication record prior to graduation (Dill & Morrison, 1985).

Tenure and promotion are related subjects, and in both instances the number of publications has a profound impact on how a faculty member is evaluated: “Tenure demonstrates a belief in a faculty member’s future promise, largely judged on evidence of potential contributions to his/her profession and institution. It symbolizes an institution’s long-term investment and commitment to the future career and success of that faculty member” (Hiller & Ritvo, 1991, p. 87). On the other hand, “promotion institutes an award earned for past accomplishments and performance” (Hiller & Ritvo, 1991, p. 87). In many universities, publications may also be rewarded with salary increments (Euster & Weinbach, 1986) commonly referred to as merit pay.

It is generally accepted that teaching, research and service are three categories by which candidates for tenure, promotion, and merit are evaluated. One study of Information Systems (IS) education programs found that teaching, research, and service were indeed the primary criteria influencing tenure, promotion, and merit decisions (Whicker et al, 1993). Standards and expectations, however, may vary from university to university. Although information on criteria for tenure, promotion, and merit may vary, these criteria often fail to operationalize standards. Therefore, a candidate must actively seek information from the dean, department chair, senior faculty, and those who have recently gone through the process (Sundre, 1992; Whicker, Kroenfeld, & Strickland, 1993). More specifically, candidates must seek to discover the research and publication expectations that they will be expected to meet, and then develop a research plan leading to a research publication record.

The information gained from this research paper may make it easier for potential authors to make decisions about which IS journals to target for publication of pedagogically oriented articles. This study was undertaken with those who are, or will be, trying to build their publication record in mind.  Specifically, those who may benefit from the following information include: candidates for faculty positions; candidates for tenure, promotion, and merit increases; those who advise such candidates with regard to their publication plan (e.g., department chairs and mentors); and those responsible for making decisions on awarding of tenure, promotion, and merit pay (e.g., deans, department chairs, and tenure, promotion, and merit committees).

2. METHODOLOGY

Sixty-one (61) top ranked IS journals were selected for this study based on Carol Saunders combined meta-rankings of six articles that ranked Management Information Systems journals (Saunders, 2003). This is the most comprehensive ranking list of IS journals, comparing the rankings of five studies by Mylonopoulos & Theoharakis (2001), Whitman et al. (1999), Hardgrave & Walstrom (1997), Walstrom et al. (1995), Holsapple et al. (1994), and by Gillenson & Stutz (1991). 

These journals were selected because they are mainstream professional IS journals, journals of professional IS organizations, or refereed IS education related journals. Subsequently, the authors identified the journal editor’s name, e-mail address and phone numbers. Four (4) journals are no longer in print, two (2) “journals” were not surveyed as they represented boards; the remaining 57 journal editors were contacted by e-mail. Two (2) journals opted out of the survey after which one follow-up e-mail and a telephone call were made to bring the total number of respondents to 50 of 53 or 94% of the total number surveyed.

Each editor was asked three (3) questions about the types of articles published by the journal (see appendix I). The first question (#1) asked “Typically, what percentage of your accepted articles fall into each of these IS domains?” with the options “Applied, Theoretical, Pedagogical, and Other” given. Instead of giving definitions of these categories, the authors provided them with examples: Applied (i.e. surveys, research case studies, etc.), Theoretical (i.e. examine/develop research methodologies, etc.), and Pedagogical (i.e. teaching tools, teaching cases, etc.), Other.

The second question (#2) asked, “Out of the total number of pedagogical articles that you publish, what percentage of accepted articles fall into each of these three sub-domains for Scholarship of Teaching in…”  with three choices being offered: a) Information Systems (teaching and learning in IS specifically), b) Business (teaching and learning in a general business school setting), and c) Information Technology (educational technology to enhance teaching and learning). These categories were chosen to reflect the three categories of the target journal list of a large southeastern research-oriented university. 

The last question (#3) asked “Would you classify your journal focus as pedagogically oriented?”, offering a dichotomous yes/no response. Since there is no agreed upon definition of pedagogically oriented, it was left open for the individual respondent’s interpretation. The goal of this question was to allow the authors to validate the responses of question #1 and #2.

3. RESULTS

The results of the survey are listed in Appendix II. The first question of the survey asked the journal editors to identify how many articles they publish on average in one of three categories. The results show that the majority of the articles focused on Applied Research (50%), followed by Theoretical (37%), Pedagogical (6%), and Other (6%). This means that these premier journals published on average only six percent (6%) of their articles focusing on pedagogical material, with the top three journals – MISQ (1%), ISR (0%), and JMIS (0%) –  publishing an insignificant number of pedagogical articles. 

Only 20 out of the 50 journals (40%) do publish pedagogical material, and only two journals (or 4%) identified themselves as having a pedagogical focus (Journal of Information Systems Education and IEEE Transactions). A closer look at the IEEE transactions response reveals two things: First, only one of the numerous transactions seems to focus on pedagogy, namely IEEE Transactions on Education. And second, of the 50% pedagogical content, only five percent (5%) is IS or Business related. 

The two journals making up the majority of IS pedagogical publications are the Journal of Accounting Information Systems with an overall 21% IS pedagogical content (with 85% IS related articles of its 25% pedagogical portion), and the Journal of Information Systems Education (JISE) with an overall of 36% IS pedagogical content (90% out of 40%). Based on these numbers, these two journals account for nearly half of all IS pedagogy publications (not taking the different number of articles, article size, circulation, or publication frequency into account). 

A distant third is journal in IS pedagogical publications is Information and Management, which publishes 15% of its articles in pedagogical areas, but of those, 80% are IS related, resulting in an overall 12% IS pedagogical content. The reader may note that all other journals have 6% or less IS pedagogy content.

These percentages might have be further skewed by the response of the editor of the Journal of Information Systems Education. As the journal of the SIG for IS education of AITP, one might expect a 100% IS pedagogical content, however, the numbers reported were 50% Applied, 10% Theoretical, and only 40% Pedagogical content, of which, as expected, 90% is IS related. The authors can only speculate that articles related to IS curriculum design or IS technology were considered as Applied. 

A second question asked the editors to further specify the type of pedagogical articles they publish, with the categories of Information Systems, Business in General, IS Technology, and Other. Of the 20 journals that do publish pedagogical articles, 17 publish articles with a focus on IS (of those, an average of 53% deal with IS Pedagogy), 8 with a focus on Business (51%), 17 with a focus on IS Technology (40%), and 1 Other (95%). This shows that IS Pedagogy and IS Technology as it relates to pedagogy make up the majority of pedagogical publications in terms of both numbers and percentages.

4. DISCUSSION

The results of the survey were a surprise to the authors. The data in Appendix II revealed that overall 2.5% of the articles published by the top 50 IS journals deal with IS pedagogy, and even when looking only among the 20 journals that do in fact publish pedagogical articles, this number climbs to only 6.73%. That means that on average only one in 40 articles of these top IS journals deals with IS pedagogy. These numbers present a bleak picture, made even worse by the fact that the top three journals: MISQ, ISR, and JMIS (not including CACM, see Appendix II), publish an insignificant number of pedagogical articles. Considering the case of MISQ:  1% means that with 4 issues per year and about 5 articles per issue, MISQ publishes on average one (1) pedagogical article every five (5) years!
 
Two journals, the Journal of Accounting Information Systems and Journal of Information Systems Education, account for nearly half of all IS pedagogical publications (not taking into account differences in the number of articles, article size, circulation, or publication frequency), but their relatively low ranking (44th  and 45th out of the 50 respondents respectively) also means that they would not typically be considered “A” journals. Of the top 30 respondents, only 1.5% of the articles deal with IS pedagogy. Thus it is clear that a researcher who publishes IS pedagogical articles will seldom be able to publish in one of the commonly accepted top tier journals.

5. CONCLUSIONS

A survey of the top 61 ranked IS journals revealed that of the 50 respondents, 2.5% of the accepted articles were IS pedagogy related, and only two (2) journals consider themselves to have a focus on IS pedagogy. The authors conclude that IS educators who want to publish in this area have little hope to publish their work in one of the commonly top ranked IS journals. Considering the fact that faculty are expected to do three things, namely research, teach, and service, one could expect that pedagogy – as a main component of a faculties responsibility – also finds its way into research and in turn gets published. However, the results of this study show that this type of research is not published in top-tier journals. Which raises the question of who ranks these journals? 

One could speculate that these rankings are more important to, and thus dominated by, research-oriented as opposed to teaching oriented universities. Since faculty in research institutions are expected to do more research than teaching, they might also do less research on teaching. This can have an influence on them as rankers, which in turn will influence the number of IS pedagogy articles published in these top journals. 

The purpose of this paper was to determine the pedagogical orientation of research journals. The authors have shown that, with the exception of two (2) journals, few IS pedagogical articles find their way into the ranked journals. The authors believe that there are other venues for the dissemination of pedagogical articles that are not represented in the current rankings. 

While the authors are currently conducting such a study to identify and rank these journals, this exceeds the scope of this paper. Instead the authors hope that the results of this study will serve as a catalyst to open a dialogue among those who have a stake in, or are involved in, the evaluation of publications when assessing faculty productivity. 


6. REFERENCES

Dill, D.D., & Morrison, J.L. (1985), Ed.D. and Ph.D. “Research training in the field of higher education: A survey and a proposal.” The Review of Higher Education, 8(2), 169-186.
Euster, G.L., & Weinbach, R.W. (1986). “Deans’ quality assessment of faculty publications for tenure/ promotion decisions. “Journal of Social Work Education, 22(3), 79-84.
Gillenson M. and Stutz J. "Academic Issues in MIS: Journals and Books," MIS Quarterly, 1991, vol.15, no.4, 147-452
Hardgrave B. and Walstrom K.: "Forums for MIS Scholars," Communications of the ACM, November 1997, vol. 40, no.11, 119-124
Holsapple C., Johnson, L. , Manakyan H. , and Tanner J.: "Business Computing Research Journals: A Normalized Citation Analysis,”  Journal of Management Information Systems, 1994, vol. 11, no. 1, 131-140
Hiller, M.D., & Ritvo, T.A. (1991). “Rethinking the promotions and tenure process.” Journal of Allied Health, 20(2), 85-93.
McNab, W.L. (1994). “Pursing promotion and tenure in higher education: Questions, answers, and suggestions.” The IS Educator: Journal of Eta Sigma Gamma, 26(1), 25-28.
Marchant, G.J., and Newman, I. (1994). “Faculty activities and rewards: Views from education administrators in the USA.” Assessment & Evaluation in Higher Education, 19(2), 145-152.
Mylonopoulos, N and Theoharakis, V.: “On-Site: Global Perceptions of IS Journals,” Communications of the ACM, Sept. 2001, vol. 44, no. 9, 29-33
Newell, S., & Price, J.H. (1983). “Promotion, merit, and tenure decisions for college IS education faculty. “IS Education, 14(3), 12-15.
Sundre, D.L. (1992). “The specification of the content domain of faculty scholarship.” Research in Higher Education, 33(3), 297-315.
Saunders, Carol (2003): Journal Information, http://www.bus.ucf.edu/ csaunders/newjournal.htm, reviewed June 13th, 2003.
Walstrom K., Hardgrave B., Wilson R. "Forums for Management Information Systems Scholars," Communications of the ACM, 1995, vol. 38, no.3, 93-102
Whicker, M.L., Kroenfeld, J.J., & Strickland, R.A. (1993). Getting tenure. Thousand Oaks, CA: Sage Publications.
Whitman, M., Hendrickson A. and Townsend A.: “Research Commentary. Academic Rewards for Teaching, Research and service: Data and Discourse,”  Information Systems Research, June 1999, vol. 10, no. 2, 99-
Appendix I: Survey (reformatted)
Professor XXX,
As editor of XXXX, we need your help since many states, including Georgia, are now mandating vehicles for promotion based on the Scholarship of Teaching both for Tenure and Non-Tenure Track faculty. Therefore, we are conducting a study to identify journal outlets for the dissemination of Scholarship of Teaching in Information Systems. 
Please take five (5) minutes to “reply” and answer the following four questions:

Question #1:
Typically, what percentage of your accepted articles fall into each of these IS research domains:
___ %  A) Applied (i.e. surveys, research 
                case studies, etc.)
___ %  B) Theoretical (i.e. examine/develop 
                research methodologies, etc.)
___ %  C) Pedagogical (i.e. teaching tools, 
                teaching cases, etc.)
___ %  D) Other:__

Question #2:
Out of the total number of pedagogical articles that you publish (see Question #1.C above), what percentage of accepted articles fall into each of these three sub-domains for Scholarship of Teaching in:
___ %  A) Information Systems: teaching 
                and learning in IS specifically
___ %  B) Business: teaching and learning 
         in a general business school setting
___ %  C) Information Technology: educa
                 tional technology to enhance  
                 teaching and learning
___ %  D) Other:___________
     
Question #3:	Would you classify your journal focus as pedagogically oriented?
___ Yes	___ No
Thank you for your participation.



Appendix II: Results of Survey


Q1:
Percent 
Accepted
Q2: Percent Pedagogic 
Accepted
Q3: 
Pedagogic Oriented
 Journal Name
Applied
Theory
Pedagogic
Other
IS 
Business
Info Tech
Other
Yes
No
MIS Quarterly
95
4
1
0
95

5


1
Communications of the ACM (*)
35
25
25
15
30
35
35
0

1
IS Research
50
50
0
0
 
 
 
 

1
Mgmt. Science
10
90
0
0
 
 
 
 

1
Journal of MIS
40
60
0
0
 
 
 
 

1
Decision Sciences
50
50
0
0
 
 
 
 

1
IEEE Transaction (**)
50
0
50
0
5


95
1

ACM Transactions on IS
0
100
0
0
 
 
 
 

1
Journal of the ACM
0
100
0
0
 
 
 
 

1
Harvard Business Review
33
0
0
67
 
 
 
 

1
ACM Computing Surveys
20
75
5
0
80
20



1
Decision Support Systems
30
70
0
0
 
 
 


1
Computer(IEEE)
60
30
10
0


100


1
Info. and Mgmt.
25
50
15
10
80

20


1
Euro. Journal of IS
56
12
6
26


100


1
Sloan Mgmt. Review
100
0
0
0
 
 
 


1
Communications of the AIS
65
11
11
13
60

40


1
Omega
50
50
0
0
 
 
 


1
Academy of Mgmt. Review
0
100
0
0
 
 
 


1
IS Management
90
0
5
5
90

10


1
Info. Systems
70
30
0
0
 
 
 


1
Info. Resources Management J,
70
20
10
0
50
30
20


1
Org. Science
90
10
0
0
 
 
 
 

1
Admin. Science Quarterly
0
100
0
0
 
 
 
 

1
ACM J. on Edu. Res. in Comp.

 
 
 
 
 
 
 

 
Operations Research
90
0
10
0

100



1
Org. Behavior & Human
 Decision Processes
85
15
0
0
 
 
 
 

1
Int'l Journal of Info. Mgmt.
80
20
0
0
 
 
 
 

1
Int'l J. of Human-Comp. Studies
65
35
0
0
 
 
 
 

1
ACM Transact. on 
Comp.-Human Interact.
50
50
0
0
 
 
 
 

1
J. of Strategic Info. Systems
65
35
0
0
 
 
 
 

1
J. of Database Management
35
50
0
15
 
 
 
 

1
J. of Computer Info. Systems
65
5
20
10
50

50


1
Interfaces (INFORMS)
80
10
10
0
5
90
5


1
DATA BASE
50
50
0
0
 
 
 
 

1
Knowledge Based Systems
 
 
 
 
 
 
 
 

 
Journal of Info. Science
50
40
0
10
 
 
 
 

1
Journal of Mgmt. Sys.
75
15
10
0
30
60



1
Journal of Operations Research
10
90
0
0
 
 
 
 

1
Journal of Systems and Software
 
 
 
 
 
 
 
 

 
Expert Systems with Applications
40
50
10
0
34
33
33


1
Behavior and Info. Tech.
40
25
15
20
30
40
30


1
INFOR J. (Info. Sys. and OR)
20
80
0
0
 
 
 


1
Int'l Journal of Intelligent 
Sys. in Acc., Fin. & Mgmt.
(formerly Expert Sys. Review)
75
25
0
0
 
 
 


1
Journal of IS (Acct.)
60
15
25
0
85

15


1
Journal of IS (Educ.)
50
10
40
0
90

10

1

Computers in Human Behavior
35
45
20
0
40

60


1
Communication Research
50
50
0
0
 

 


1
Journal of End-User Computing
80
10
10
0
50

50


1
Simulation (J. of the 
Society for Comp. Sim.)
35
50
15
0
 
 
 100
 

1
Journal of Software Maintenance
40
60
0
0
 
 
 
 

1
Datamation
95
0
0
5
 
 
 
 

1
PC World
0
0
0
100
 
 
 
 

1











Count
50
50
50
50
17
8
17
1
2
48
Avg
50.2
37.4
6
5.9
45.2
20.4
29.2
40.2
0.04
0.96
Minimum
0
0
0
0
0
0
0
0
0
0
Maximum
100
100
50
100
95
100
100
95
1
1
(*) CACM underwent many changes, which lead to its removal from the “A” list of a number of universities, incl. this institution.

(**) There are numerous IEEE publications. The reported numbers represent the “overall” response of a senior editor in reference to all the IEEE publications.

Journals that opted out of our survey: Quality Progress, Academy of Management Journal

Journals that did not respond: ACM Journal on Edu. Resources in Comp, Knowledge based Systems, Journal of Systems and Software

Journals that are no longer published: Journal of Systems Management, AI Expert, Interface (The Education Journal), IBSCUG Quarterly (Journal of the International Business School Computer Users Group)

Journals where no information could be found: ACM Educational Board, IEEE Computer Society -Educational Activities, Computers and Automation