Tutorial on reviewing your own paper (and those of others)

Reviewing Your Own Paper(and those of others)

Tim GrantRetired But Active Researcher (R-BAR)

Chair, ISCRAM Publications & Academic Standards Committee (PASC)

r̅�

Overview

• Goal:– To help listeners to produce better submissions to

conferences, journals, & books by gaining knowledge of reviewing own & other papers

• Outline:– Reviewing: history, process, products, roles– Reviewing criteria & how to review

– Shortcomings, criticisms & failure of reviewing– Online possibilities

– Key resources

– An exercise for the attendee

Linköping, 5 Jun 13

Tutorial on reviewing

2

Introduction: about me

• Career:– 1966-87: Royal Air Force officer (UK & SG)– 1987-2004: Consultant, ICT services industry (NL)– 2001-09: Visiting Professor, University of Pretoria (ZA)– 2004-12: Professor, Netherlands Defence Academy (NL)

• Qualifications:– 1969: Bachelor of Science, Aero Engineering, Bristol (UK)– 1984: Defence Fellowship (Masters), Brunel (UK)– 1996: PhD, Artificial Intelligence, Maastricht (NL)

• ISCRAM:– 2009-date: Board member & PASC chair



3

Introduction: reviewing (1)

• Reviewing (aka refereeing):– Subjecting draft proposals, papers, posters, demonstrations,

theses, courses, subjects, etc to critical evaluation

• Why?– Quality control; help authors to learn– Admission to “Body of Knowledge” (BoK)– Assignment of credit & priority to authors

• By whom?– Independent experts working in same field– Can be board / panel / tribunal of experts– More usually peers (other researchers / authors)



4

Introduction: reviewing (2)

• Where?– Evaluation of applications for funding– Review of project reports by researchers to assess

successful progress / completion– Review of draft conference presentations, journal articles,

and monographs to check they meet quality standards– Evaluation of set of papers after publication for review article– Evaluation of quality of work produced by individuals, teams,

departments, and institutions to help determine:• Appointments

• Promotions

• Levels of funding



5

RIN, 2010

Brief history (1)

• 854-931: Ishap bin Ali Al Rahwi, “Ethics of Physician”:– 1st description of (medical) peer review

• 1453: printing press• 1543-1564: Copernicus, Servetus, Galileo, Versalius:

– Review by non-peers

• 1620: Francis Bacon’s New Philosophy:– Scientific method

• 1662: Royal Society, London, chartered:– 1st learned society

• 1665: Philosophical Transactions:– 1st scientific journal; editor selects publications



6

Spier̅, 2002

Brief history (2)

• 1752: Royal Society adopts review procedure:– Pioneered in Edinburgh since 1731

– Need for editorial committee (initially to fill excess space)

• 1890: Typewriter & carbon copies (1959 Xerox):– Easier to circulate submissions to reviewers

• Approx 1940: Diversity & specialization of material:– Excess journal space vanishes; need to discriminate

– Need for external reviewers

• 1989-98: International Congresses on Peer Review

• Since approx 1990: PC, Internet, email, WWW:– Online journals; Open Access movement



7

Spier̅, 2002

Reviewing process (1)

Linköping, 5 Jun 13 8

Problem Research Write up Review Publish

Resear̅chFunding

Teach Evaluate

Teaching

Design

Develop

Product Service

Valor̅ization

Evaluate

citations / impact

patentsuse in cour̅ses

Ideally should pr̅edictcitations / impact, patents,

& use in cour̅ses

BoK


• Steps in review process:1. Authors submit draft paper to editors/publishers

2. Publisher’s staff log submission & acknowledge receipt

3. Editors assign draft paper to reviewers

4. Reviewers assess draft paper, recommending acceptance, rejection, or modification

• Typically 2 or 3 reviewers

5. Editors weigh recommendations, make decision, & inform authors

6. Authors modify paper & resubmit revised version



9

RIN, 2010


• Steps in ISCRAM review process:1. Authors upload draft paper to ConfTool

2. ConfTool logs, acknowledges receipt, & informs Track chair

3. Track chair assigns draft paper to reviewers in ConfTool

4. Reviewers assess draft paper, recommending acceptance, rejection, or modification in ConfTool

5. Meta-reviewers weigh recommendations in ConfTool

6. Track chair makes decision & informs authors via ConfTool

7. Authors modify paper & upload revised version, details of changes made & copyright form to ConfTool

8. ConfTool logs, acknowledges receipt, & informs Track chair



10

(Adapted fr̅om) RIN, 2010


• Variations on reviewing process:– Submission:

• (Extended) abstract + paper; paper only

– Submission types:• Full paper; short/work-in-progress paper; practitioner paper;

poster; demonstration; exhibit

– Reviewing:• By committee; blind; double-blind; restricted; crowd (open)

– Medium:• Paper; CD-ROM; USB; online• Restricted (to attendees or to members) vs open access



11


• Steps in review process (with abstract, eg for book):1. Authors submit abstract to editors/publishers

2. Editors assess abstract, make decision, & inform authors

3. Authors submit draft paper to editors/publishers

4. Publisher’s staff log & acknowledge receipt







12



• Steps in review process (for journal):1. Authors submit abstract to editors/publishers

2. Editors assess abstract, make decision, & inform authors

3. Authors submit draft paper to editors/publishers

4. Publisher’s staff log & acknowledge receipt




8. Authors modify draft paper & resubmit revised version

9. Reviewers assess revised paper & make new recommendation





13



• Conferences:– Usually 1 iteration; paper only; 8-10 pages– (Less often) 2 iterations: first abstract, then paper

• Journals:– 2 or more iterations; paper only; 20-30 pages– At least 50% new material for conference paper

• Books:– 2 iterations:

• Extended abstract or chapter proposal (typically 2 pages)

• Chapter / contribution; 20-30 pages



14


• Differences between disciplines:– Humanities & social sciences:

• Double-blind common

• Authors may be allowed to nominate reviewer(s)

– Sciences:• Single-blind common; increasingly double-blind

– Medical field:• Increasingly open-review



15

Products from reviewing

• Paper template• Copyright form• Review template• Email templates for:

– Acknowledging receipt– Informing Track chair of upload– Assigning papers to reviewers– Rejection, acceptance, & conditional acceptance– Final acceptance of camera-ready copy



16

Roles in reviewing

• Author: writes, submits & modifies paper• Track chair: assigns submissions to reviewers• Reviewer: reviews set (2..6) of submissions• Meta-reviewer: moderates (2..4) reviews• (Editor: combines Track chair & meta-reviewer roles)• Programme Committee = {Editors}

– ISCRAM: Scientific Committee = {Track chairs}– Moderates across tracks

• Programme Chair (& co-chairs):– Decides conference schedule



17

Reviewing procedures

• Review procedure documents (eg ISCRAM):– Review process:

• Overview for conference organizers

– Review timeline:• Helps conference organizers schedule activities

– Review guidelines:• For reviewers of full, short / work-in-progress, & practitioner

papers

– Meta-review & final decision:• For meta-reviewers, track chairs, & Scientific Committee

– Best (Student) Paper procedure:• For Best Paper sub-committees & conference organizers



18

Criteria for reviewing (1)Criterion ISCRAM BNAIC PlanSIG ICIW

(Discipline) IS + CRAM AI Planning Cyber/info war

Relevance x x x x

Significance/contribution x - x x

Originality x (full only) x x -

Validity x (full only) x x x

Clarity/readability x x x x

(Other̅s) Best Paper? Award? Method, drawings, abstract, keywords,

references

Overall score x x x x

Reviewer’s confidence x x x -

Remarks for author(s) x x x x

Remarks for PC x x x -



Criteria for reviewing (2)

• Relevance:– Papers should be clearly relevant to conference’s subject area,

not generalist– ISCRAM (= IS + CRAM):

• Technical papers (information systems, computer science, IT) should describe implications for crisis response and/or emergency management

• Vice versa for crisis response / emergency management papers

– Tips:• Read Call For Papers!• Ensure paper fits into topic or track• Does this year’s conference have special theme?

– If so, try to make link to theme in paper (N.B. nice to have)



20


• Significance / contribution:– Does paper make a significant contribution to literature?

• Significance means contribution opens up (or closes off) previously unexplored lines of research or of development (valorization)

– Does paper state the contribution it makes?– Does paper link this contribution to pre-existing literature?– Does paper identify the limitations of this contribution?

• From method & from what is outside scope of paper• Limitations can always be topics for further research!

– Tips:• Know the relevant literature!

• Don’t overclaim, i.e. overstate contribution



21


• Originality / novelty:– Ideas must be non-trivial, new & timely, not “more of same”

• Case study can be new

• Literature survey paper can be new if it “adds value”, eg structuring literature in a new way

• Application of established ideas to another field can be new:– Idea may be known in IS literature, but not yet applied to CRAM.– And vice versa - which is why practitioner papers are so important

– Tips:• Know the relevant literature!

• Where can, exploit multi-disciplinary nature (e.g. IS + CRAM)



22


• Validity / soundness / thoroughness:– Is purpose of paper clearly stated? And paper’s scope?– Is research question / hypothesis clearly stated?– Are appropriate methods used?

• Is best practice for method followed? (eg case study, literature survey)

– Is argument logically sound? Statistics & equations correct?– Is treatment /discussion thorough?

• In enough detail for another researcher to reproduce what you did?

– Tips:• Ensure paper tells story:

– Not necessarily the order in which you did your research!

• Reflect on your contribution in context of related literature



23


• Clarity / readability / structure:– Well structured? Grammatically correct? Easy to read?– Abbreviations in full on 1st use? Jargon known to readers?– Citations given in full & easy to find?– Can oldies (60+) read graphics without magnifying glass?– Tips:

• Apply “magic number of 5 +/- 2” to number of sections

• First section should be “Introduction” or “Motivation”

• Last section should be “Conclusions & Further Research”

• Review relevant literature in 2nd or 2nd-to-last section

• Do not exceed page or word limit!

• Use spelling & grammar checker!



24

How to review (1)

• Reviewing is extension of reading paper• Keshav’s three-pass method for reading:

– Pass (1): Quick-scan (5-10 mins):• Read title, abstract, & introduction• Read section & sub-section headings (structure)• Glance at equations (if any)• Read conclusions• Glance over references to see which ones you know

– Should be able to answer “five Cs”:• Category; context; correctness; contribution, clarity



25

Keshav, 2012

How to review (2)

– Pass (2): Read paper with more care (1 hour):• Ignore details such as proofs• Jot down key points, or make notes in margin• Look carefully at figures, diagrams, & other illustrations• Mark relevant unread references for further reading• Answer:

– What does paper do? (Rationale/motivation, aims, hypothesis)– How does it do it? (Participants, method)– What did it find? (Findings, implications, falsifiable, limitations)



26

Keshav, 2012

How to review (3)

– Pass (3): Virtually re-create paper (4-5 hours):• Making same assumptions, re-create paper in own way• Compare re-creation with actual paper:

– Identify & challenge every statement

– Jot ideas down as you go

• Self-reflect: watch out for community groupthink / bias• Answer:

– Weaknesses:» Implicit assumptions; experimental / analytical errors; missing

citations

– Strengths:» Presentation technique; support for conclusions; internal efficacy


Keshav, 2012Tutorial on reviewing

How to review (4)

• Now add reviewing-specific steps:– Scoring according to reviewing template– Writing “Remarks to authors”:

• Briefly summarize paper in own words

• State what you think contribution(s) are:– Not those stated by author(s)

• Give specific comments based on jottings / margin notes:– Separate major points from minor ones (eg typos)

– By relevance; significance; originality; validity; clarity

– Justifying your scores

• Conclude comments (eg summarize good & bad points)

• Recommend modifications



28

Roscoe, 2007

How to review (5)

• Approaches to reviewing own paper:1. Write paper & then review it, modifying as needed

2. Write paper, already building in information reviewer needs

• I prefer 2nd approach (next 6 slides)• Mindset: Reviewer is always right!

– True: modify as reviewer recommends (Relevance, Significance, Originality, & Validity)

– False: reviewer misunderstood because draft paper is poorly expressed; express more clearly (Clarity)

• You’ll still get Remarks to Authors:– But they should be fewer and less major



How to review (6)

• How I plan a paper (1/6):– Look at Call For Papers– Have I done research that is worth presenting?

• If none, STOP

– Which topic / track does it fit into?• If none, STOP

– What message do I want to give?• Sketch story (5 plus/minus 2 bullets)

– Write single sentence describing purpose of paper:• “The purpose of this paper is to present / survey / show ...”



30

How to review (7)

• How I plan a paper (2/6):– Extract title from purpose of paper:

• This must “sell” your paper!Eg “Tweak the tweet: leveraging proliferation with a prescriptive syntax to

support citizen reporting” (Starbird & Stamberger, ISCRAM 2010)

– Extract top-level headings from story:• Don’t forget 5 plus/minus 2:

1.Introduction

2.Relevant theory

3.Experiment

4.Discussion

5.Conclusions & Further Research



How to review (8)

• How I plan a paper (3/6):– Expand 1. Introduction:

1.1 Background (or Motivation)– Identifies “gap” in current Body of Knowledge– Motivates why research is worth doing– Shows relevance to conference & topic / track– Outlines what paper covers, eg

“This paper focuses on technical capabilities for cyber warfare.”

1.2 Purpose & scope– Insert one-sentence purpose here– Scope is what your research & this paper does not include, eg

“Legal issues are outside the scope of this paper.”

1.3 Paper structure (or Layout)– So reader knows what to expect (sections, appendices)



How to review (9)

• How I plan a paper (4/6):– Expand 3. Experiment:

Eg 3.1 Design; 3.2 Execution; 3.3 Results & analysis

– Expand 5. Conclusions & Further Research:(5.1) Summary

– Not same as Abstract

(5.2) Contribution(s)– Just most important ones– Don’t overstate

(5.3) Limitations– From method used– From what is outside scope

(5.4) Further research



How to review (10)

• How I plan a paper (5/6):– Allocate page budget (see CFP or template for limit):

– Write Abstract:• Note word limit (in CFP, author instructions, or template):

– Two-thirds background & motivation:» Conclude with one-sentence purpose of paper

– One-third (intended) structure of paper


Title, abstract & keywords ½ page or 5%

Introduction 5%

Conclusions & Further Research 5%

References 10%

(other sections) divide up 75%


How to review (11)

• How I plan a paper (6/6):– Write one-sentence objective for each section:

1. Introduction “The objective of this section is to describe the background,

motivate the research, state the purpose and scope of the paper, and outline its structure.”

2. Relevant theory

“The objective of this section is to summarize the theory relevant to the content of this paper.”

• N.B. To guide writing only; may be deleted once written

– Allocate (sub-) sections to co-authors (if any):• In consultation, taking into account roles & abilities



Key resources (1)

• Overviews (recommended):– RIN, 2010. Peer Review: A guide for researchers. Research

Information Network (March), http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/peer-review-guide-researchers. Accessed 2 May 13. *

– Wikipedia. 2013. http://en.wikipedia.org/wiki/Peer_review. Accessed 2 May 2013

– Rowland, F. 2002. The Peer Review Process: A report to the JISC Scholarly Communications Group. http://www.jisc.ac.uk/uploaded_documents/rowland.pdf. Accessed 2 May 2013. *



40

Key resources (2)

• Papers/articles (1/3):– Calcagno, V., Demoinet, E., Gollner, K., Guidi, L., Ruths, D. & de

Mazancourt, C. 2012. Flows of Research Manuscripts Among Scientific Journals Reveal Hidden Submission Patterns. Science Magazine, 338, 1065-9 (November)

– Cormode, G. 2008. How NOT to Review a Paper: the tools and techniques of the adversarial reviewer. SIGMOD Record, 37, 4. *

– Griswold, W.G. (not dated). How to Read an Engineering Research Paper. http://cseweb.ucsd.edu/~wgg/CSE210/howtoread.html. *

• See also link to paperform.pdf. *

– Hill, S. & Provost, F. 2003. The Myth of the Double Blind Review? Author identification using only citations. SIGKDD Explorations, 5, 3, 179-184

– Jefferson, T., Alderson, P., Wager, E. & Davidoff, F. 2002. Effects of Editorial Peer Review: A systematic review. JAMA, 287, 2784-6



Key resources (3)

• Papers/articles (2/3):– Kershav, S. 2012. How to Read a Paper.

http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf. (June 26). *

• See Ian McLean’s 2012 Literature Review Matrix, based on Kershav, at http://www.psychologyinc.org/2012/06/literature-review-matrix.html. *

– Mahoney, M.J. 1977. Publication Prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 2, 161-175

– McCook, A. 2006. Is Peer Review Broken? The Scientist Magazine (February 1)

– Roberts. 1999. Scholarly Publishing, Peer Review and the Internet. First Monday, 4, 4 (5 April)

– Roscoe, T. 2007. Writing Reviews for System Conferences. http://people.inf.ethz.ch/troscoe/pubs/review-writing.pdf. *



Key resources (4)

• Papers/articles (3/3):– Smith, R. 2006. Peer Review: A flawed process at the heart of

science and journals. J R Soc Med, 99, 178-182

– Spier, R. 2002. The History of the Peer Review Process. Trends in Biotechnology, 20, 8, 357-8 (August)

– Van Rooyen, S., Godlee, F., Evans, S., Smith, R. & Black, N. 1998. Effect of Blinding and Unmasking on the Quality of Peer Review: A randomized trial. JAMA, 280, 3, 234-7

– Walt, S.M. 2013. On Writing Well. Foreign Policy (February 15). http://walt.foreignpolicy.com/posts/2013/02/15/on_writing_well. *

• Itself an example of good writing.

– Weiss, R. 2005. Many Scientists Admit to Misconduct: Degrees of deteption vary in poll. Washington Post (June 9)



Key resources (5)

• Journals:– Learned Publishing– Journal of Scholarly Publishing– Journal of American Society for Information Science– Journal of Documentation

• Bibliography:– Bailey, C.W. 2011. Scholarly Electronic Publishing Bibliography.

www.digital-scholarship.org/sepb/. Version 80 (30 November), accessed 2 May 13

• Learned society:– Association of Learned and Professional Society Publishers

(ALPSP): www.alpsp.org. Accessed 2 May 13



44

Key resources (6)

• Books:– Peek, R.P. & Newby, G.B. (eds). 1996. Scholarly Publishing: The

electronic frontier. MIT Press, Cambridge, MA– Page, G., Campbell, R. & Meadows, A.J. 1997. Journal Publishing.

Cambridge University Press, Cambridge, UK– Meadows, A.J. 1998. Communicating Research. Academic Press, San

Diego, CA– Tenopir, C. & King, D.W. 2000. Towards Electronic Journals: Realities

for scientists, librarians and publishers. SLA Publishing, Washington DC– Fredrikson, E.H. (ed). 2001. A Century of Scientific Publishing. IOS

Publishing, Amsterdam, NL– Abel, R.E. & Newlin, L.W. (eds). 2002. Scholarly Publishing: Books,

journals, publishers, and libraries in the twentieth century. John Wiley & Sons, New York



Key resources (7)

• Wikipedia (English, all accessed 2 May 13):– http://en.wikipedia.org/wiki/Peer_review

– http://en.wikipedia.org/wiki/Open_peer_review

– http://en.wikipedia.org/wiki/Peer-reviewed_scientific_journal

– http://en.wikipedia.org/wiki/Academic_journal

– http://en.wikipedia.org/wiki/Open_access_journal

– http://en.wikipedia.org/wiki/Academic_authorship

– http://en.wikipedia.org/wiki/Academic_publishing• See also Scholarly paper and Peer review sections

– http://en.wikipedia.org/wiki/Open_access_(publishing)

– http://en.wikipedia.org/wiki/Peer_review_failure

– http://en.wikipedia.org/wiki/Plagiarism



46

Exercise for listener

• “Beer versus science” as example of open reviewing:– Grim, T. 2008. A Possible Role of Social Activity to Explain Differences

in Publication Output Among Ecologists. Oikos, 177, 484-7. http://www.zoologie.upol.cz/osoby/Grim/Grim_Oikos_2008.pdf. *

• Inverse linear relation between beer input and publications output

– Mack, C.A. 2008. In Defense of Beer-Drinking Scientists. (March 21). http://life.lithoguru.com/index.php?itemid=119. *

– Mack, C.A. 2008. In Defense of my Defense of Beer-Drinking Scientists. (March 27). http://life.lithoguru.com/index.php?itemid=120. *

– Mack, C.A. 2008. More on Beer-Drinking Scientists: A response to Dr Grim. (April 9). http://life.lithoguru.com/index.php?itemid=121. *

– Van Noorden, R. 2010. Make Mine a Double. (Sep 15). http://blogs.nature.com/news/2010/09/make_mine_a_double.html

• Let’s do field research!



Any questions?

Tim GrantRetired But Active Researcher (R-BAR)

[email protected]

+31 (0)638 193 749

With thanks to:

Julie Dugdale, Simon French, Mark S Pfaff, Murray Turoff, Arien vd Wal

Tutorial on reviewing your own paper (and those of others)

Business