Electronic copy available at: http://ssrn.com/abstract=2194232 1 How to Review a Paper Derek K. Oler [email protected]William R. Pasewark [email protected]Texas Tech University December 22 nd , 2012 Abstract This paper introduces the review process to new researchers, describes the institutional background for reviews, and provides advice to reviewers. The review process is a critical element of the dissemination of research that ultimately provides credibility to the body of academic knowledge. This process relies on expert, anonymous reviews by established researchers (typically those who have already published in top journals on similar topics), but the scarcity of expertise combined with anonymity results in agency problems. Invitations to review should only be accepted if the researcher has the skill, time, objectivity, and inclination to do a quality review. The objectives of a quality review are (1) to assist the editor in his/her gatekeeping role and (2) to assist the authors in improving their work. A quality review should consider the contribution, execution, and conclusions of the paper. We conclude by discussing pitfalls that reviewers should avoid. We thank Denton Collins and Dana Hermanson for helpful suggestions and comments.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Electronic copy available at: http://ssrn.com/abstract=2194232
Abstract This paper introduces the review process to new researchers, describes the institutional background for reviews, and provides advice to reviewers. The review process is a critical element of the dissemination of research that ultimately provides credibility to the body of academic knowledge. This process relies on expert, anonymous reviews by established researchers (typically those who have already published in top journals on similar topics), but the scarcity of expertise combined with anonymity results in agency problems. Invitations to review should only be accepted if the researcher has the skill, time, objectivity, and inclination to do a quality review. The objectives of a quality review are (1) to assist the editor in his/her gatekeeping role and (2) to assist the authors in improving their work. A quality review should consider the contribution, execution, and conclusions of the paper. We conclude by discussing pitfalls that reviewers should avoid.
We thank Denton Collins and Dana Hermanson for helpful suggestions and comments.
Electronic copy available at: http://ssrn.com/abstract=2194232
2
1. Introduction
The intent of this paper is to facilitate and improve the review process by discussing
the advantages and weaknesses inherent in the process and by providing
recommendations on how a quality review can be executed. We build on prior
commentaries and advice from accounting researchers (e.g., Moizer, 2009; Bailey et al.,
2008; Kachelmeier, 2004; Omer et al., 2004) by providing greater detail on the
mechanics of the review process and its institutional background and by providing
specific, concrete guidance on how to conduct a quality review. This paper can be
especially useful to PhD students and new faculty, but may also be helpful to seasoned
faculty as they consider future reviewing activities.
In the academic accounting discipline journal publications are the main channel for
the distribution of new ideas. Journal publications are also the most important factor in
tenure decisions at top research schools, and even at schools more oriented towards
teaching, publications play a significant role. Academic journals generally require peer
reviews of submissions before they are published, making the review process one of the
most important tasks in the academic profession.
Waymire (2012) argues that developing reviewer talent benefits the discipline by
encouraging greater innovation by researchers. Yet, in spite of the singular importance of
reviews, many journals provide only brief guidance to their reviewers (and others provide
none at all).1 The purpose of this paper is to introduce new researchers to the review
process by describing the institutional background of the review process and providing
advice on how to do a quality review.
1 Generic review guidance does exist, for example, Pyrczak (1999).
3
Reviews play a critical role because they (1) assist the editor in fulfilling a
gatekeeping role, and (2) assist the authors in improving their research. However, the
review process is subject to considerable agency problems because reviewers are
anonymous to the authors and because there is often a shortage of qualified reviewers
(Gilmore et al., 2006).2 Accordingly, the review process is vulnerable to failure because
reviewers can shirk with little or no immediate negative consequences.3 In spite of these
difficulties, there is no obvious alternative to the review process, and at present no major
journal has proposed a substitute for the traditional review process. Accounting
academics have a vested interest in ensuring that the review process works as best as it
possibly can. Accordingly, with this paper, we seek to improve the way we build
knowledge by facilitating one of our critical academic processes.
In this paper we discuss how the review process can be impaired when reviewers
accept review opportunities for which they lack the skill, time, objectivity, or interest.
We also demonstrate how researchers should not use reviewing opportunities to “dish
some back out” or to make the reviewer look smart at the authors’ expense; rather, they
should be viewed as a chance to participate in honing cutting edge research.
In this paper we show how bad reviews potentially hurt the journal’s reputation,
frustrate editors, and discourage new research. On the other hand, good reviews enhance
the journal’s reputation, make the editor’s job easier, and encourage innovative and
interesting research. In our experience, even when the editorial decision was
unfavorable, receiving a thoughtful review was encouraging and helpful.
2 “Agency problems” refers to problems from individuals maximizing their own utility at the expense of others. For example, a manager may maximize his personal utility by surfing the web during work, and this hurts shareholders who bear the cost of his salary. 3 As we discuss later, there are significant longer-term reputational consequences to providing a poor review.
4
We attempt to provide new reviewers with direction on providing review comments
regarding: (1) the contribution, (2) the motivation and literature review, (3) hypotheses
and theoretical support, (4) data, analysis, and presentation, and (5) the conclusions
drawn. We recommend beginning a review with a quick overview, focusing first on the
abstract and introduction (where the contribution and motivation can be broadly
assessed), and then a detailed read of the text and tables. Once the reviewer collects
his/her initial thoughts, often on a scratch pad or a rough electronic draft, we recommend
setting the review aside for a day or two before finalizing the review and returning it to
the editor.
The final portion of this paper identifies pitfalls that the reviewer should try to avoid.
For archival research, these include unsupported assertions of bias, which often manifests
as a correlated omitted variable. There will always be some sort of bias in the data or
analysis (and often multiple forms of bias), and potential bias should be addressed by
considering (1) whether the bias works for or against the hypotheses, and (2) how strong
of an effect that bias might have. Experimental work may also sometimes face the
criticism that the (rarely helpful) setting is not “realistic.”
2. Background
Ellison (2011) contends that journals have two functions, the dissemination of
knowledge and quality certification.4 The publication process involves authors, who
produce papers and are ultimately responsible for their content; editors, who make the
ultimate decision on papers that are accepted for the journal; and reviewers, who help the
4 Ellison (2011) also notes the review process may be in decline because of increased use of electronic media such as Social Sciences Research Network (SSRN). Thus, authors (especially established authors) feel less need to utilize journals to disseminate their work.
5
editor make his decision. Reviewers might not catch all of a paper’s flaws (Spiegal,
2012), but they still play a vital role in the quality assurance aspect of the journal.
Informal faculty and PhD student discussions sometimes gravitate to a critique of the
review process, especially if a participant was recently on the receiving end. As the top
accounting academic journals generally have acceptance rates of around 10% (or less),
the outcome of most submissions is, unfortunately, negative, and it is easy to blame the
reviewer(s) when this happens. However, when the sting of rejection has passed, it is
useful to consider the broader picture of why we do reviews in the first place…and why
the review process continues to be utilized in spite of apparent shortcomings.
Examinations (and critiques) of the review process have been published in many
fields. Peters and Ceci (1982), in their famous (perhaps infamous) paper, describe how
they sent twelve papers previously published in highly regarded psychology journals
back to these journals as new submissions with fictitious author names. Only three of the
twelve articles were detected as resubmitted papers; out of the nine remaining, eight were
rejected (mostly for having “serious methodological flaws”). While we hope that such an
exposé could not happen in the academic accounting profession, the perception remains
that the process of publishing a paper in a top accounting journal is not always fair
(Moizer, 2009). This is confirmed by the survey results reported by Bailey et al. (2008).
Their results suggest that only 59.8% of respondents believe that the review process is
fair for top-tier journals, and that assistant professors (the ones under the most pressure to
publish) believe the review process is more unfair than associate and full professors.
Several papers have been published in management (e.g., Bedeian, 2003; Harrison,
Hermanson, and Tomkins, 2008), and psychology (e.g., Epstein, 1995; Peters and Ceci,
1982) describing flaws in the academic review process and how it can be improved. We
attempt to integrate the relevant points and suggestions from other fields into our
recommendations here. Prior work has also been done in accounting (e.g., Moizer, 2009;
Bailey, Hermanson, and Louwers, 2008; Omer et al., 2004; Kachelmeier, 2004; Demski
and Zimmerman, 2000; Brinn et al., 1996), and we attempt to integrate (and
acknowledge) contributions from these authors here.
3. Why Are Reviews Important?
Reviews are critical to our academic profession because they help screen journal
content, and thereby vet what is added to our body of knowledge. The reader might
imagine what would happen if reviews were not done – perhaps if all academic
accounting papers were merely accepted for publication. This would be similar to
handing out gold medals to all athletes instead of only the very best. Readers would not
be able to sort through the deluge of marginal, contradictory, and inaccurate papers, and
the profession would not survive the precipitous loss of credibility. Future work would
lack a firm foundation to build on. In short, the “market” for research would be
compromised and the entire academic accounting discipline would suffer (Kachelmeier,
2004).
3.1 Consequences of Editorial Mistakes
Editors, who are the gatekeepers of what is accepted in academic journals, rely on
reviewers to help them determine whether to accept, reject, or consider the manuscript for
revision. The effectiveness of this feedback has implications for reputation of the
7
journal, but it also influences the continuity of fruitful streams of research within the
discipline. The acceptance of a manuscript potentially impacts both the academic and
practitioner communities. In addition, it provides a basis for future research and its
contribution will be manifested in subsequent citations.
The importance of reviewer input in the publication decision is best illustrated by
consequences of editorial mistakes. Reviewers sometimes recommend acceptance of
poor quality manuscripts.5 For example, a reviewer might feel unqualified and, at the
same time, reluctant to admit to his or her lack of expertise. As a result, the reviewer
might provide an inadequate review and recommend acceptance based on the inability to
demonstrate justifiable reasons for rejection.
Another example might be a reviewer who is unwilling to invest the time needed for a
comprehensive review. As a result, the reviewer might provide cursory feedback and a
favorable recommendation to discourage the need for additional review effort. In this
case, the editor might interpret the lack of effort by the reviewer as an acknowledgement
that the manuscript lacks flaws.
A reviewer might also recommend acceptance of a poor manuscript if he or she
suspects the manuscript is written by a reputable author (Bailey et al, 2008). This lack of
objectivity may make the reviewer reluctant to point out weaknesses in the manuscript.
These behaviors potentially result in the acceptance of a poor quality manuscript (a Type
II error). This can result in the profession using weak (or simply false) research findings
as a basis for future research and may lead subsequent researchers down unproductive
research paths. Eventually, the reputation of the journal is damaged (see Figure 1).
5 Reviewers are also referred to as “referees.” For consistency, we will use the term “reviewer” here.
8
______________________
Insert Figure 1 ______________________
Similarly, reviewers might recommend the rejection of a high quality manuscript (a
Type I error). This editorial inefficiency is sometimes related to reluctance of the
reviewer to accept innovative methodologies or unexpected results. In other cases, the
reviewer may be unwilling or unable to perform a quality review and merely lists reasons
to reject the paper (and, as Peters and Ceci, 1982, show, there are always reasons to reject
any paper).
When this occurs, the readership is deprived of useful knowledge that can increase
the body of knowledge and provide guidance to practice. The rejection of a suitable
paper discourages future research and ultimately hurts the reputation of the journal. In all
likelihood, the manuscript will ultimately be published in a competing journal and the
impact of the research findings is delayed. In the end, editor, reviewer, and author effort
and time is wasted.
The editor also depends on reviewers to provide feedback to authors that will improve
their work. This reviewer role is important whether or not the manuscript continues in
the review process. If a revision is requested, the editor typically utilizes reviewer
suggestions to formulate expectations of the revision. If the manuscript is rejected, the
expectation is that the authors will use the feedback before submitting to another journal.
In this case, useful comments will provide new direction for collecting additional data,
redesigning an experiment, conducting different analyses, or improving a write-up.
9
3.2 The Difficulty of Obtaining Quality Reviews
The benefits of a quality review are not without cost. Referees are usually not
compensated for their work (paying reviewers more than a token amount would quickly
bankrupt academic journals).6 The fact is that qualified reviewers are a scarce resource
(Gilmore et al., 2006) and top researchers are often inundated with more requests to
review than they could complete without neglecting their own research, teaching, and
service obligations.
A quality reviewer is valuable. From an editor’s perspective, reviewers develop a
reputation as helpful or not helpful, brash or courteous, and timely or tardy.7 As might be
expected, individuals who perform high quality and timely reviews are likely to receive
subsequent requests. In essence, talented reviewers can be “punished” by more requests
for review work.
Likewise, there is the potential for bad behavior to be rewarded in the review process.
By suggesting a “reject” recommendation the reviewer likely frees himself from an
additional review of a resubmitted manuscript. Thus, rejecting a manuscript can reduce
the reviewer’s workload.
The anonymity of reviewers may also contribute to lower quality reviews. By
remaining unknown to the authors, reviewers may believe that they are free from the
6 Some journals compensate reviewers if they complete a review within a certain period of time. Compensation can take the form of direct payments, waived future submission fees, etc. Compensating reviewers can increase turnaround time, but journals usually charge higher submission fees to offset compensation costs. 7 The software utilized by the American Accounting Association journals offers the editor an opportunity to rate the reviewer effectiveness and archives the responses.
10
consequences of a poor review.8 In combination, these factors provide some reviewers
with the incentive to shirk – to provide sloppy and/or inappropriate reviews.
Unfortunately, poor reviews impose costs on others. Editors that receive poor
reviews must decide whether to do the review themselves or to proceed using low quality
feedback. The first alternative is usually not possible because the editor’s time is limited.
The second alternative increases the chances that an unsuitable paper will be published
or, more likely, that an important paper will be rejected.
4. How the Review Process Works
The editor manages the flow of manuscripts, which includes tasks such as
corresponding with authors, assigning reviewers, advising authors regarding revision, and
working with publishing organizations to develop electronic and hard copy for
distribution. When a manuscript is received, the editor or editorial staff member usually
reviews the manuscript for compliance with journal formatting standards (sometimes
referred to as “quality control”). For larger journals, the editor assigns the manuscript to
an associate editor who has an expertise in the topical area.9 The associate editor is
usually expected to skim the manuscript and recommend potential reviewers. The
associate editor might recommend a “desk rejection,” (i.e., rejection without formal
review.) Desk rejections usually occur if the content is similar to previously published
8 The negative effects of anonymity can also work the other way, when authors submit papers that are not proofread, not previously presented, and/or contain multiple micro-font tables, and expect an anonymous reviewer to serve as their mentor and writing coach. It should also be noted that while the reviewer is anonymous to the authors, he is not anonymous to the editor, who can respond by reducing future review opportunities. 9 Titles differ depending on the journal. For example, for the Accounting Review, the “senior editor” assigns the manuscript to an “editor.”
11
manuscripts, the analysis exhibits obvious flaws, or the subject is outside of the scope of
the journal’s mission.10
Referees are typically selected from a pool of authors who have published in the in
the journal (or comparable journals) on similar topics. These individuals are usually
selected in one of two ways. First, the editor may utilize members of the editorial board.
These individuals are familiar with the journal and have made a commitment to
reviewing for the journal.11 Alternatively, the editor might search for individuals not on
the editorial board, referred to as “ad hoc” reviewers. These individuals are typically
cited in the submitted manuscript or have demonstrated an expertise in the subject of the
manuscript.
Typically two reviewers are assigned to a manuscript. Three reviewers are
sometimes assigned when one reviewer is less experienced. In all, a submitted
manuscript is usually read by three to five individuals, two or three reviewers who
scrutinize the content, analysis, and contribution of the manuscript, an associate editor,
and/or an editor who synthesizes the conclusions of the reviewers and provides the
authors with a consensus of the recommendations and makes suggestions for the revision.
To facilitate an unbiased review the identity of the reviewer is not provided to the
authors, and the authors are also unknown to the reviewer.12 Because neither party is
aware of the identity of the other, this technique is known as a “double-blind review.”
10 For journals of the American Accounting Association submission fees are refunded for desk rejections because the manuscript did not undergo formal review. While the outcome of a desk rejection, on surface, seems harsh, it benefits the editorial process by expediting a highly probable negative outcome. The authors are benefitted by a refunded submission fee and avoiding the wait associated with the review process. 11 The American Accounting Association journals utilize software that facilitates a search for qualified reviewers using key words. 12 Some have suggested reviewers not be anonymous (e.g., Epstein, 1995) while other suggest it is difficult to achieve a truly anonymous review (Bailey, et al. 2012).
12
For journals that utilize associate editors, journal policy varies on whether the associate
editor is revealed to the authors.
Spiegal (2012) believes reviewer delays and extensive revision requests are a major
reason for delaying the review process. The reviewers are usually asked to complete the
review within four to six weeks. It is significant to note that the time taken to respond to
authors is primarily controlled by individual reviewers and that reviewers vary
significantly in the time taken to respond to journal duties. Table 1 shows median days
required for each stage of the review process for Issues in Accounting Education for a
two and one-half year period. The median turnaround time for the journal is 66 days.13
The median time for a reviewer to respond to review request is five days and the median
time to complete a review, once the reviewer has accepted, is 47 days. That means
78.8% of the total median review time relates to activities under the control of the
individual reviewer.
______________________
Insert Table 1 ______________________
Wide deviations in the reviewer procurement and reviewer evaluation stages indicate
reviewers vary significantly in the time taken to complete a review. In other words,
while some reviewers are quick to respond and complete a review, others tend to
significantly delay the review process. A conscientious editor will make efforts to send
reminders of upcoming deadlines, follow up when deadlines are not met, and avoid using
reviewers who are chronically tardy.
13 First response times in other areas are typically longer. For example, economics journals average 6.5 months as an average response time (Ellison 2002).
13
When all reviews are received, the editor and/or the associate editor summarizes the
comments and make a recommendation for acceptance, revision, or rejection. If the
recommendations of the reviewers are mixed it is the responsibility of the editor to make
a final recommendation. If a revision is requested, the most important role of the editor is
to clarify the major requests of the reviewers. In some cases, the editor may ask a
reviewer to reconsider a request that does not seem logical or that is contrary to another
suggestion.
In some cases, a reviewer may make a request that requires significant effort, for
example changing the purpose of the research or collecting significant additional data.
Accordingly, the editor must decide whether to “relax” the reviewer request or reject the
paper because the required effort justifies a new submission, rather than a revision.
Most journals have the policy of sending revisions back to the same reviewers. As
such, when a revision is requested, the reviewer should expect to expend additional effort
in the review process. The median number of days to complete a revision for Issues in
Accounting Education is 108 days. This is plenty of time for the reviewer to lose
familiarity with the original content and suggested revisions. Consequently, it is good
practice for reviewers to reread the original manuscript and the previous review before
considering the revision.
5. How to Review a Paper
To reiterate, a review has two main objectives (1) to assist the editor in making a
decision on whether or not to publish the paper, and (2) to assist the authors in improving
their work. This section examines issues to consider before accepting a review
14
assignment and guidelines for conducting a thorough review. This section also relates
pitfalls that tempt reviewers.
5.1 Before Accepting a Review Assignment
The invitation to review a paper suggests that the manuscript has already passed the
initial screen(s) from the editor (and possibly an associate editor). However, it is unlikely
the manuscript has been read in detail before it reaches the reviewer. Before accepting an
invitation to review, a researcher should consider whether he or she has the skills, time,
objectivity, and inclination to review the paper.
Skills Invited reviewers should first evaluate their skill in reviewing a paper on two
bases: subject matter and, if applicable, methodology. The invited researcher should
read the abstract and skim the tables (and the rest of the paper, if necessary) and assess
whether he or she is familiar enough with the literature and the relevant statistical tools to
adequately evaluate the paper. If the potential reviewer does not have the skills to do the
review, the editor should be made aware. The editor may assign the paper to a different
reviewer, or may ask an additional reviewer to join the process.
Time Editors want efficient processing of manuscripts because turnaround time has
implications for the journal reputation. Quick turnaround is also important because
manuscripts utilize data that may become “stale” or the content may address issues that
lose importance as time passes (such as pending financial accounting pronouncements or
tax legislation).
As noted earlier, three-quarters of the review time is typically related to the reviewer.
As such, a reviewer should try to respond as quickly as possible to a reviewing invitation,
and make a realistic evaluation of his or her ability to complete the review in a timely
15
manner. Unfortunately, reviewers have a tendency to procrastinate. Issues in Accounting
Education asks reviewers to complete a review within five weeks. Historically,
approximately 56% of reviewers complete the review on time. Of these, approximately
half will complete the review in the last week before the review is due (see Table 2).
______________________
Insert Table 2 ______________________
Reviewers should also be sensitive that a manuscript may be written by a person who
is preparing a dossier for promotion or tenure. While a solid research record is ultimately
the responsibility of the author, it is unfair to put candidates for promotion at a
disadvantage because of inefficient manuscript processing.
Effective editors are first concerned with content, but are also required to be good
process managers. Most editors send notices reminding reviewers of upcoming
deadlines. In addition, they usually send stern correspondence when reviews are
overdue. Editors and authors appreciate reviewers who respect deadlines.
Objectivity For most journals, the review process is perceived as independent only
under “double-blind” conditions. In other words, the author names or institutions should
have no influence on how the value of the manuscript is assessed. Potential reviewers
should ask themselves whether they have seen the paper presented or if they know the
authors. Many journals ask reviewers to refrain from searching for papers online (such as
SSRN) to determine author identity. A potential problem with of objectivity should be
discussed with the editor.
16
Inclinaton In other situations, the researcher simply may not have the inclination to
review the paper…for example, the topic simply might not be interesting to the
researcher, or the researcher might have recently received multiple requests for many
reviews and is burned out. In these cases, it is also appropriate to refuse the offer. Some
researchers may feel that refusing the offer will hurt their reputations, but our
conversations with editors suggests that it is far more damaging when a reviewer accepts
an assignment, postpones a response, and then ultimately admits not being interested in
doing the review (or does a sloppy job).
5.2 The Review Task
After agreeing to do the review, a conscientious reviewer will schedule time to
complete the job well before the deadline. In reviewing the paper, we suggest first
reading the title, abstract, and introduction, and then doing a quick overview of the
tables.14 Then, once the paper’s motivation, contribution, and method of analysis is clear,
the reviewer should do a detailed read of the entire paper.
In the initial overview, the manuscript should be evaluated for consistency with the
mission of the journal, interest to the readership, general adherence to the journal format,
and writing effectiveness.15 In most cases, the editor or associate editor will have
screened for these qualities. However, significant deficiencies may warrant a request that
the editor reconsider a “desk rejection.”
14 Recommendations differ here – Magro looks at the tables first and attempts to infer the hypothesis that are being tested before reading the entire paper; Mills appears to begin with the title, abstract, and introduction (Omer et al., 2004). Given the variance in practices by experts, we recommend that new researchers consider various recommendations in developing their own preferences. 15 However, in some cases authors should consider whether it is worthwhile to comply with more obtuse requirements before the paper is accepted. For example, some journals require an accepted manuscript in 11-point font, but this is more difficult for reviewers to read.
17
In our opinion, a reviewer is not obligated to invest hours on a paper that is poorly
motivated, laden with errors, or rife with grammatical mistakes. However, reviewers
should be honest in their assessment: if the paper was insufficiently motivated, they
should say so, and not go hunting for additional, often contrived, reasons to reject the
paper. Of course, nothing prevents the reviewer from continuing to read through the
paper and providing additional comments and suggestions, especially if it is clear that
authors could use the help or if the motivation could easily be improved and is coupled
with competent analysis.
The approach taken by a reviewer should differ depending on whether the research is
empirical, analytical, or a commentary. However, the following is a general approach a
reviewer might consider. The appendix is a checklist of items that can be useful in
evaluating a manuscript.
Title The title is the first item typically viewed by the reader and it should serve as a
tool for determining whether the manuscript will be of interest to the reader. A good title
should concisely, but adequately, describe the manuscript content.
Abstract The abstract describes the purpose of the manuscript and highlights the key
take-aways from the paper. In addition, the abstract should provide brief information
regarding the methodology and findings. The abstract should also summarize the
conclusions and may also note implications for practitioners and future research.
Motivation and Literature Review The paper should identify a specific and
important problem, and should provide a current and complete overview of the relevant
prior literature. The prior work cited should provide an adequate foundation for the
current paper’s research questions or hypotheses.
18
The motivation of the paper must be spelled out clearly and persuasively – it is not
sufficient for the authors to simply assume the reviewer will find the paper “obviously”
interesting. Moreover, motivation must be carefully linked to the prior literature. The
paper should explain the relevant facts from prior work, how those facts relate to the
current paper, and what the current paper does to build on those facts. Merely providing
a laundry list of prior work without any effort to link prior work to the current paper is
inadequate.
A well-crafted motivation will convince the reader to continue reading. When
motivation is lacking a good reviewer will point this out and, where possible, suggest
improvements.
Hypothesis Development While papers (and journals) vary in their use of formally
stated hypotheses, quality research always includes theoretical backing for the analysis.
In more recent work, this is often done simultaneously while describing the contribution
and motivation of the paper. A quality review will consider the strength and
appropriateness of the theoretical development.
Good theoretical development provides the framework within which the paper’s
results can be understood. Generally, top journals are not receptive to studies that jump
directly to empirical results without explaining their context, because this can often result
in “data mining” (an ex-post, ad-hoc fitting of a “story” to the observed results). Tension
between competing explanations is also often needed – that is, a paper should try to find
some middle ground between knowing exactly what the results will be prior to the
investigation and having no idea what the results could be and what they could mean.
For example, a paper investigating whether the market misprices accruals in a particular
19
industry would not be interesting because we already know the market misprices accruals
in general. However, if the researcher could point out plausible reasons why the market
might not misprice accruals in a particular setting, that could introduce sufficient tension
to make the paper interesting.
Data and/or Subjects The reviewer should consider whether the data is appropriate
and free from bias. Where bias exists, the reviewer should make an evaluation of the
potential impact of the bias and whether the bias favors or opposes the expected results.
When reviewing an archival paper, the reviewer should consider the data sources and
whether there are known problems. Data sources that are familiar to researchers, such as
CRSP and Compustat, generally require little explanation, but data that is unique or has
known problems may require more description (and possibly more discussion on how the
authors dealt with any problems).
In an experiment, the reviewer should consider the representativeness of the subjects;
for example, whether first-year accounting students are appropriate subjects when testing
for the effect of cognitive biases in analytical review. Also, in an experiment, the
manipulation should be clear, and the experimental results should not boil down to
obvious findings such as the subjects prefer more money over less.
Analysis and Methodology The study should employ statistical tools that are both
appropriate and free from bias. Archival work should use the best tools available to
overcome known problems (especially for studies examining long-window returns). For
example, papers like Gow, Ormazabal, and Taylor (2010) discuss methodologies to
control for cross-sectional and time-series dependence. Lennox, Francis, and Wang
(2012) discuss self-selection models, and Beaver, McNichols, and Price (2007) discuss
20
methods of dealing with delisted firms. Papers that don’t use the most up-to-date tools
don’t deserve automatic rejection; in our experience, results under simple OLS typically
hold up under more exotic approaches, but the reviewer should make recommendations
to improve the analysis when needed.
Results and Contribution The results of the paper should be connected to
hypotheses. Are the hypotheses supported? Do the results suggest changes to established
theory? The authors should highlight the important aspects of their results (and not
assume that these are obvious). Results should be clear; most journals now require that
tables be able to “stand on their own,” meaning that a reader should be able to understand
them without having thoroughly read the paper. This means identifying the dependent
variable and providing definitions for all variables used (or referring the reader to another
table or appendix that is close at hand).
The results of the paper (and their discussion) should lead smoothly back into the
main contribution of the paper, and should not lead the reader down meandering side
issues. The contribution of the paper should be sufficient to warrant publication in the
journal. This question is often considered by the editor before the paper is sent to the
reviewer(s), but a conscientious reviewer will also address this “big picture” question.
Contribution has two elements: (i) does the paper, by itself, make a sufficient
contribution to our understanding? and, (ii) does the paper make a sufficient marginal
contribution relative to other work that is already published, or is found in working
papers already in circulation? The answer to the first question must always be
affirmative to warrant proceeding: if no other paper has investigated the topic, the reason
might be that the topic is simply not interesting or important. If the contribution of the
21
paper is wanting, a good reviewer should recommend where the contribution should be
improved or perhaps journals where the paper might be more appropriate.
Conclusions – The conclusion of the study should be supported by the analysis and
results. In some cases, authors may overreach (or, less frequently, undersell their
findings). The authors should discuss important limitations of the study, avoiding
mention of obvious points, such as a generic “future results may be different.” Finally,
the authors should discuss legitimate directions for future research.
5.3 Preparing a Reviewer Report
A quality review report begins with a succinct overview of the paper and its findings;
i.e., it will identify the paper’s incremental contribution (Kachalmeier, 2004). The report
should then introduce the most important elements of the paper – both positive and
negative. For example, a paper may address a particularly important question and be well
motivated, but might suffer from a fatal bias in the data used. After laying out the most
significant issues of the paper, the report should list minor issues in order from most
significant to least significant.
Whenever possible, reviewers should provide suggestions how to overcome problems
noted. If a revision is requested, the associate editor or editor will likely utilize these
suggestions when responding to the author. If the manuscript is ultimately rejected, the
reviewer’s suggestions should reflect how the study might by reconducted to achieve
more publishable results. If practical, the reviewer may want to offer suggestions for
publication in alternative journals. However, as Demski and Zimmerman (2000) point
out, the reviewer is not an anonymous co-author; the authors are not obligated to take the
reviewer’s advice (although they should explain why they don’t in their response), and
22
reviewers are not obligated to solve the paper’s problems (although some suggestions are
often appreciated). Further, the reviewer should distinguish between correctable and
uncorrectable flaws (Moizer, 2009).
Most journals ask reviewers to express their recommendation for a manuscript. The
software utilized by the journals of the American Accounting Association allows the
reviewer to choose between five recommendations (although these may be altered by
individual editors):
1. accept for publication with no changes 2. accept for publication subject to minor improvements 3. revise utilizing review comments before a final decision can be made 4. significantly revise before a final decision can be made 5. reject without further consideration
In general, reviewers should not put their recommendation in their review report, as
this potentially preempts the editor (who has the ultimate responsibility to make these
calls). The reviewer should make a recommendation to the editor outside of the review
report.
In the first round, if the reviewer provides a positive report and the editor concurs, the
paper typically will receive a “revise and resubmit” recommendation.16 The authors,
following the guidance from the editor, should provide a detailed memo discussing how
they dealt with each review point (although the authors need not follow every suggestion,
they should explain their decision when they do not). Most journals now discourage
multiple-round sessions and prefer to make an up-or-down decision after one or two
rounds. In subsequent rounds, we recommend that reviewers should be careful not to
“move the goalposts” for authors. That is, reviewers should avoid the impulse to produce
16 Journals rarely accept a paper for publication without requiring at least some changes.
23
new, significant recommendations beyond their first review, but should focus on whether
their original review points were adequately addressed.17
In some cases the authors may fail to address the editor and reviewer comments
(perhaps because the authors might not have fully understood them), and do not make
progress in their revision. When this happens it should be clear to the editor that their
earlier review points were not adequately addressed. In rare cases the reviewer may
realize he made a mistake and that his recommendations appear to have diminished the
value of the paper; in these cases, we think complete honesty is appropriate (rather than a
somewhat dishonest revision of history where the reviewer pretends he didn’t make the
recommendation in the first place and is critical of the change in the revision). Reviewers
should also pay attention to the comments from other reviewers and from the editor
(Moizer, 2009).
5.4 Reviewing Pitfalls
It is important to recognize that a review is not an opportunity to get revenge for past
slights from poor reviews (“dishing some back out”), nor a game in which the reviewer
merely lists all possible reasons to reject the paper (perhaps thinking that such behavior
will impress the editor by showcasing the intelligence of the reviewer).18
There are a number of valid criticisms that can be made of any paper. There are also
some “cheap shots” that seem to serve as substitutes for a careful reading and thoughtful
analysis of the paper.
17 There are likely many “horror stories” of multiple-round submissions; we are personally familiar with one paper going five rounds, where an incompetent reviewer eventually cycled back to old review points that he had earlier acknowledged as resolved. 18 Bailey et al. (2008) report that behaviors like “getting even,” and even delaying reviews for selfish reasons, rank among the most serious perceived threats to the review process.
24
Research Bias They are many variants of the bias problem, but most represent a
misinterpretation (or more accurately, an overstatement) of the problem. All data,
experimental or archival, is subject to error and bias. The key questions are (1) does the
bias work for or against the results, and (2) is the bias significant enough to warrant
concern? Bias in the results or analysis could be sufficient to warrant disclosure and
discussion, a robustness check, or perhaps even rejection if the paper’s results are
misleading (hopefully with suggestions as to how the bias could be overcome or at least
mitigated).
One potentially significant form of bias is a correlated omitted variable. However,
merely stating that the possibility exists is unhelpful, because all studies are subject to
correlated omitted variables. There may be cases where the authors missed an important
variable, and if so the reviewer should point this out and consider the effect that the
omission might have. For example, Richardson, Sloan, Soliman, and Tuna (2005)
discuss the issue of a correlated omitted variable problem (see their footnote 3), and
Dechow, Hutton, Kim, and Sloan (2011) provide more insights on the effects of a
correlated omitted variable in detecting earnings management.
Another form of bias is the possibility of self-selection. Self-selection is present
whenever the phenomenon studied includes management choice; for example, accruals,
earnings manipulation, discontinued operations, issuing equity, and pursuing an
acquisition. The presence of self-selection rarely invalidates the results, but it can affect
the interpretation of the results. Consider the following sequence that has been studied
by Xie (2001):
Discretionary Accruals Abnormal Future Returns
25
Xie (2001) finds that discretionary accruals have predictive power over future returns,
building on Sloan’s (1996) hypothesis that the market fails to fully consider the
differential persistence of the accruals portion of income versus the cash portion of
income. However, the above relationship reflects only the observable factors involved.
Moreover, management choices are not randomly distributed either: they are
functions of management incentives, which are in turn affected by management’s ethical
capacity and cultural factors, and these additional factors are much more difficult to
objectively measure. This information in no way invalidates the relationship between
discretionary accruals and future returns that is well documented in the literature.
In some cases self-selection can affect the interpretation of results. For example, it
may be that whatever causes discretionary accruals can also affect future returns, so one
cannot be certain that discretionary accruals cause future returns…they are merely
associated. This is a weakness common to all archival research: causality is much harder
to establish than association. This inherent weakness can often lead to interesting
research opportunities for experimentalists, who have tools better suited to determining
causality. Self-selection issues can also possibly open the door to asking (and answering)
additional interesting questions, such as “what motivates firms to acquire other
companies in the first place?” or “why would managers want to inflate earnings in some
settings and deflate them in others?”
26
Sweeping Generalizations Reviewers may be tempted to make broad statements of
criticism without strong evidence to support these claims. Some of the most common
claims in this regard are that a paper is “not interesting” or “lacks contribution.”
While these claims may be true, neither stands on its own. For example, the claim of
uninteresting should be accompanied by examples of what the readership is expecting
and how they typically use journal content. Then the reviewer should state why the
current article fails to meet those expectations.
When unsupported, the criticism of a “lack of contribution” can be particularly
damaging because the criticism is often characterized as a “fatal flaw” that justifies
rejection. The criticism of “lack of contribution” should also be supported. For example,
the reviewer might identify previous studies that already address the same issue.
Alternatively, the reviewer might indicate why the results would be of little consequence
to practitioners or future researchers.
A Need for Open-Mindedness A final “cheap shot” is the improper assertion that broad
interpretations of findings from prior work means that future research cannot make a
contribution. This is more an indication of a reviewer who is simply not interested in
learning new things. For example, a reviewer could observe that a growing body of
research finds that the market is not completely efficient with regard to incorporating
financial statement fundamentals into market prices, and therefore research showing the
market fails to understand the implications of changes in pension assumptions does not
make a marginal contribution. This is similar to claiming that the study of disease is not
useful because we already know that people die. A similar approach is the assertion that
27
“I just don’t buy it,” without any explanation of why the results of the paper might be
wrong or what might be missing.
These pitfalls can be often be avoided by a thoughtful reviewer reflecting on what he
is writing. Sometimes it can be useful to apply the criticism one is contemplating to
already published work. For example, if the reviewer believes the paper he is reviewing
has a critical flaw, would that “critical flaw” also suggest that papers like Fairfield,
Whisenant, and Yohn (2003) or Feltham and Ohlson (1995) should have not been
published? If so, it might be the criticism and not the paper that is at fault.19
6. Summary and Conclusions
Reviews are an essential part of the academic process, and while sloppy reviews hurt
the authors, the journal, and ultimately the profession, good reviews can encourage the
authors, support the journal, and improve the profession. Accordingly, we provide an
overview of the review process and discuss its inherent weaknesses, and we provide
specific guidance on how to do a good review. Finally, we discuss pitfalls (“cheap
shots”) that a reviewer should try to avoid.
The observations and recommendations in this paper are based on our opinion,
although this has been shaped by our personal experiences and those of our colleagues.
Ultimately, a good review reflects a conscientious effort by the researcher. In some
ways, the review process is affected by the tragedy of the commons: reviewers may
individually benefit from shirking, while the benefits of a good review accrue to the
19 We have purposely selected these papers because they were rejected by the journals they were initially submitted to, as was Ball and Brown (1968).
28
profession as a whole. Thus, the quality of the peer-review system depends on the skills
and integrity of reviewers.
We do not wish to imply that responsibility for the review process (and its outcome)
lies principally with the reviewer, or even the editor. Although these individuals
certainly have a strong impact, we believe the authors have the greatest influence over
whether their paper is accepted or rejected. Editors and reviewers must work with what
they are given, and even if they make a mistake in rejecting a paper that should have been
accepted, there are many other competing accounting journals that will be happy to
publish good research. Sloppy work will, in general, receive negative reviews; authors
who would prefer good reviews followed by publication should avail themselves of
Zimmerman’s (1989) solid advice.
In our experience, reviews from accounting journals are generally of better quality
than reviews from other business academic journals. This is a reflection of the
collegiality of the academic accounting in general. In recent years the profession has
received calls for more diversity in accounting research, and calls for improvements to
the review process (Waymire, 2012). Recent research suggests that concentration (the
propensity of publications in top journals to be dominated by a few elite schools) has
decreased in top accounting journals (Oler, Oler, Skousen, and Talakai, 2012), suggesting
that diversity is indeed increasing. We believe that quality reviews will help this positive
trend continue into the future.
29
References Bailey, Charles D., Joe F. Hair, Dana R. Hermanson, and Victoria L. Crittenden, 2012,
Marketing academics’ perceptions of the peer review process, Marketing Education Review 22(3), 265-280.
Bailey, Charles D., Dana R. Hermanson, and Timothy J. Louwers, 2008, An examination
of the peer review process in accounting journals, Journal of Accounting Education 26, 55-72.
Bailey, Charles D., Dana R. Hermanson, and James G. Tomkins, 2008, The peer review
process in finance journals, Journal of Financial Education 34(2), 1-27. Ball, Ray and Paul Brown, 1968, An empirical evaluation of accounting income numbers,
Journal of Accounting Research Autumn, 159-178. Beaver, William, Maureen McNichols, and Richard Price, 2007, Delisting returns and
their effect on accounting-based market anomalies, Journal of Accounting and Economics 43, 341-368.
Bedeian, Arthur G., 2003, The manuscript review process: the proper roles of authors,
referees, and editors, Journal of Management Inquiry 12(4), 331-338. Dechow, Patricia M., Amy P. Hutton, Jung Hoon Kim, and Richard G. Sloan, 2011,
Detecting earnings management: A new approach, Working Paper, Berkley. Demski, Joel S. and Jerold L. Zimmerman, 2000, On “research vs. teaching:” A long-
term perspective, Accounting Horizons September, 343-352. Ellison, Glen, 2011, Is peer review in decline? Economic Inquiry, 49(3), 635-657. Ellison, Glen 2002, The slowdown of the economics publishing process, Journal of
Political Economy 100(5), 947-993. Epstein, Seymour, 1995, What can be done to improve the review process, American
Psychologist October, 883-885. Fairfield, Patricia, Scott Whisenant, and Teri Yohn, 2003, Accrued earnings and growth:
Implications for future profitability and mispricing, The Accounting Review 78, 353-371.
Feltham, Gerald A., and James A. Ohlson, 1995, Valuation and clean surplus accounting
for operating and financial activities, Contemporary Accounting Research 11, 689-731.
30
Gilmore, Audrey, David Carson, and Chad Perry, 2006, Academic publishing: Best practice for editors, guest editors, and reviewers, European Business Review 18(6), 468-478.
Gow, Ian D., Gaizka Ormazabal, and Daniel J. Taylor, 2010, Correcting for cross-
sectional and time-series dependence in accounting research, The Accounting Review 85, 583-612.
Harrison, David, 2002, Obligations and obfuscations in the review process, Academy of
Management Journal 48(6), 1079-1084. Holbrook, Morris B., 1986, A note on sadomasochism in the review process: I hate when
that happens, Journal of Marketing July, 104-108. Kachelmeier, Steven J., 2004, Reviewing the review process, Journal of the American
Taxation Association 26, 143-154. Lennox, Clive S., Jere R. Francis, and Zitian Wang, 2012, Selection models in accounting
research, The Accounting Review 87, 589-616. Moizer, Peter, 2009, Publishing in accounting journals: A fair game? Accounting,
Organizations, and Society 34, 285-304. Omer, Thomas C., Susan L. Porter, Robert J. Yetman, Anne M. Magro, Lillian F. Mills,
Richard C. Sansing, and Benjamin C. Ayers, 2004, A discussion with reviewers: Insights from the midyear ATA meetings, Journal of the American Taxation Association 26, 135-141.
Oler, Derek K., Mitchell J. Oler, Christopher J. Skousen, and Jayson Talakai, 2012, Has
concentration in the top six accounting journals changed over time? Working Paper, Texas Tech University.
Peters, Douglas P., and Stephen J. Ceci, 1982, Peer-review practices of psychological
journals: the fate of published articles, submitted again, Behavioral and Brain Sciences 5, 187-195.
Pyrczak, Fred, 2005, Evaluating Research in Academic Journals: A Practical Guide to
Realistic Evaluation, 3rd edition, Pyrczak Publishing, Glendale, CA. Richardson, Scott A., Richard G. Sloan, Mark Soliman, and Irem Tuna, 2005, Accruals
reliability, earnings persistence, and stock prices, Journal of Accounting and Economics 39, 437-485.
Sloan, Richard G., 1996, Do stock prices fully impound information in accruals and cash
flows about future earnings? The Accounting Review 71, 289-315.
31
Spiegel, Matthew, 2012. Reviewing less – progressing more, The Review of Financial Studies 25(5) 1331-1338.
Waymire, Gregory, 2012, Seeds of Innovation: Can We Alter the Status Quo?
Accounting Education News 40:1, 3-4. Xie, Hong, 2001, The mispricing of abnormal accruals, The Accounting Review 76, 357-
373. Zimmerman, Jerold L., 1989, Improving a manuscript’s readability and likelihood of
publication, Issues in Accounting Education 4, 458-466.
Median Duration per Stage of a First Round Review Stage Median Days Percent Std DevQuality Control 1 1.5% 3.2Reviewer Procurement 5 7.6% 7.2Reviewer Evaluation 47 71.2% 18.9Associate Editor Evaluation 9 13.6% 15.2Editor Decision 4 6.1% 23.2 Total Days for First Round Decision 66 100.0% 40.59 Author Preparation of Revision 108 Stage Activities Quality Control Assistant editor inspects uploaded documents for completeness and
conformity to journal format Editor makes initial determination of appropriateness for journal (makes
“desk rejection” if necessary Editor assigns associate editor and selects potential reviewers
Reviewer Procurement Requests to review are sent Reviewers accept or decline request to review
Reviewer Evaluation Reviewers evaluate manuscript Reviwer report and recommendation is prepared
Associate Editor Evaluation
Associate editor evaluates manuscript and reviews comments of reviewers
Associated editor determines whether there is consensus among readers and makes a recommendation to the editor
Editor Decision Editor makes first round decision Authors are notified of first round decision
Based on 588 manuscripts submitted to Issues in Accounting Education between January 1, 2010 to July 26, 2012.
Is the manuscript content consistent with the mission of the journal? Will the subject matter be of interest to the journal audience? Does the manuscript adhere to the prescribed journal format? Is the quality of the writing effective?
Title
Does the title adequately describe the subject matter? If applicable, does the title identify the participants and findings? Is the title specific and concise?
Abstract
Is the purpose manuscript adequately identified? If applicable, are the participants, methodology, and findings identified? Is the usefulness of the results to practitioners and future researchers identified?
Motivation / Literature Review
Is a specific problem identified and supported as important? Have the conclusions of past research been accurately represented? Is the literature review adequately extensive and current? Are the research questions or hypotheses developed from the theory represented in
the literature review? Is the identified literature pertinent (rather than a laundry list of prior with without
proper links to the proposed study)? Is the motivation for the study clear by the end of the literature review?
Hypothesis Development
Are the hypotheses or research questions based on theory from previous work? Does the study attempt to anticipate findings based on the findings of previous
research?
36
Sample / Subjects / Data
Are the data from an appropriate source? Is the data source reliable? Are the data or participants free from bias? If bias exists, does it potentially affect
the results? Is the sample randomized and stratified? Are non-responses or eliminations adequately justified? Are the sample demographics adequately described? Is the sample size adequate? If human participants were utilized, was adequate consent obtained?
Instrument
Has the full instrument been provided for observation? Is the question format appropriate? Do the questions demonstrate adequate content validity? Does the instrument contain sensitive inquires that potentially bias results?
Experiment
Are participants selected in a random manner? Are treatments alternated? Are administrators properly trained and unbiased?
Methodology and Analysis
Are proper statistical methods utilized? Has statistical significance been properly established? Are the tables describing the results comprehensive and descriptive? Do the statistical results properly relate to the research questions or hypotheses?
Results and Contribution
Are the results related to the purpose of the study? Are the results sufficiently generalizable to the external environment? Are the results described in perspective to previous studies?
Conclusions
Are the conclusions supported by the analysis? Are the limitations of the study discussed? Are implications for practitioners and future research adequately addressed?