ARC Discovery Grant Scheme Peter Lovibond School of Psychology, UNSW Member, Social, Behavioural, and Economic Sciences panel of ARC.

ARC Discovery Grant Scheme

Peter Lovibond

School of Psychology, UNSW

Member, Social, Behavioural, andEconomic Sciences panel of ARC

Thanks to –• Phyllis Tharenou• Cynthia Fisher

Key Resources

• ARC Website: www.arc.gov.au– Funding rules– Instructions to applicants– Sample application form

• Your University’s Research Office– Expertise on rules, help with budgets, use of RMS– Assistance with editing, internal review processes– Actual submission of proposals

How Proposals are Evaluated

Three tiers of assessors for Discovery

1. College of Experts– There are six discipline panels in the CoE

• Biological Sciences and Biotechnology (BSB)

• Engineering and Environmental Science (EE)

• Humanities and Creative Arts (HCA)

• Mathematics, Information & Communication Sciences (MIC)

• Physics, Chemistry and Geosciences (PCG)

• Social, Behavioural and Economic Sciences (SBE)

– Each panel has an Executive Director; currently Phyllis Tharenou for SBE

– CoE members are paid, hold 3-year terms, read 160-200 applications per year, go to Canberra for selection panels twice/year

– Each proposal is read by two CoE members, one of whom has primary responsibility

– CoE members DO NOT provide written comments to authors

Adrian Wilkinson

HR Management Griffith

Mark WesternSociology

UQ

Karen ThorpePsychology

QUT

Kaye StaceyEducationMelbourne

John PiggottEconomics

UNSW

Victor MinichielloSociology/Health

UNE

Peter Lovibond Psychology

UNSW

Bill GriffithsEconometrics

Melbourne

Graeme GillGovernment

Sydney

Cynthia Fisher Management

Bond

Stephen Crain Cognitive Science

Macquarie

Barbara Comber Education

USA

Harry Bloch Economics

Curtin

Martin Bell Geography

UQ

Marian Pitts Psychology

La Trobe

1. Australian readers (OzReaders)– paid (a little), read 5-15 proposals per round,

assigned by Executive Director

2. Specialist readers (IntReaders) – internationally recognised experts in area of

application (Australian or overseas)– unpaid, read 1-5 proposals, assigned by COE1

Who you are writing for

• Executive Director reads only summaries/ keywords/ FOR codes to assign CoE members and Ozreaders

• CoE members read widely, including proposals outside their discipline. They use abstracts and keywords to assign IntReaders.

• OzReaders read in or close to their discipline

• IntReaders are specialists in your area/topic

• Investigators 0.4 – track record relative to opportunity– capacity to undertake the proposed research

• Significance & Innovation 0.3– how will the project advance knowledge?– what novel hypotheses are being tested?

• Approach 0.2– conceptual framework, design, methods and analyses– support aims of proposal

• National Benefit 0.1– economic / environmental / social benefits

Assessment criteria

Review and Selection Process

1. Simultaneous assessment by Int, Oz, and CoE readers

2. Applicants are sent Int and OzReader narrative feedback and have a chance to write a rejoinder.

3. Weighted Average Percentile Rank (WAPR)

a)Ratings on each criterion are multiplied by weights to create overall score for each reader

b)Scores converted to percentile rankseg reader who has 20 applications:1st 100; 2nd 95; 3rd 90 . . . 20th 0

c) Ranks from different readers are weighted by the number of proposals they read, and averaged to create a draft WAPR (CoE weight limited to 25)

4. CoE members get draft WAPRs, comments, ratings, and ranks from other readers, and rejoinders from CIs.

5. CoE members then confer and may recommend slight changes in WAPRs.

6. Selection meeting in Canberra. Top down funding on revised WAPR scores.

trade-off between number of applications funded (20-25%) and % budget awarded (average 65%)

7. Discussion of budgets for proposals to be funded, based on CoE1 recommendations.

unjustified requests cut

higher ranked proposals receive higher % of requested budget

• ARC has a 15% target for ECR-only Discovery Projects

• All CIs on proposal must be ECR - PhD awarded within past 5 yrs at time of application

• Extra consideration for ECR-only proposals on the borderline of being fundable

– Still need a strong project and good track record (relative to opportunity)

– Reviewers have to be convinced you can do the work

– Don’t be too ambitious with scope/budget

– Might be better to work with more experienced person instead

Early career researchers

How to get funded

1. very good track record

and

2. very good proposal

Improving track record

1. apply when your track record is strongif your recent track record is not strong, may be better to write papers / collect pilot data / write Linkage application

2. work on quality as well as quantityjournals rather than books / chapters / conferences

3. choose good collaboratorsstrong track record and commitment to project

don’t include CI with weak track record

PI track records now included

Writing a Good Proposal

1. good research ideainteresting, novel, important, theoretically based

2. sound methodologyarticulate an approach that tests your ideas with appropriate procedures and measures so as to allow valid conclusions about theory and application

3. clear presentationreadable, well-organised, understandable by both general and specialist readers

Critical features of a good proposal

Summary

Must be clear enough to assign your proposal to the right readers

Grab the attention of readers

• What is the research problem and justification?

• Why is it important/urgent?

• What are project’s specific research aims?

• What will you do?

• How will you do it?

• What are the expected outcomes?

National/Community Benefit• Spell out the benefits and intended outcomes

• Don’t promise the world – needs to be plausible

• Write for educated lay people

• Give info additional to that in Summary of Project

• Make it free-standing, comprehensible alone

Classifications

Field of Research and Socio-Economic Objective codes – Select codes and percentages for each – determines which

CoE panel gets the proposal

Keywords – Used to assign proposal to assessors, so should match

keywords assessors are likely to use in their self-descriptions.

– Avoid overly narrow terms that won’t match anyone.

– Avoid overly broad terms that have different meanings in multiple fields (e.g., conflict, culture, bank, utility, justice).

– Include at least one medium-broad term, like “social development” or “mathematics education” or “speech disorders”.

Track Record Relative to Opportunities

1. Statement of your most significant contributions to this research field

2. Refereed publications in last 5 years

3. Ten career best publications

4. Other evidence of impact and contribution to the field

5. Other aspects we should consider

Most Significant Contributions

• Organise around your major line(s) of research or impacts (programmatic research looks good)

• Give objective evidence of quality of output, impact, and international reputation IN THIS FIELD

– Give impact factors, ranks or tiers of journals, evidence for book publisher quality

– Give your citation count, H factor, etc.

– Mention published reviews of your books or articles if relevant

• Leave out teaching, consulting, and practitioner activities

Refereed publications in last 5 years

• Group by books, chapters, articles, proceedings

• List most recent first in each category

• Include in-press pieces with date of acceptance

• *Asterisk those relevant to proposal

• Explain authorship conventions in your field especially if non-standard

Ten career best publications

Evidence of quality

– Showcase your best papers (especially if >5 years ago)

– Give impact factors and ERA ranking for journals(draft rankings at http://lamp.infosys.deakin.edu.au/era/)

– Give citation counts if impressive

– Leave out textbooks, reports, working papers, conference presentations

– Try to include at least 50% first/senior authored papers, even if they are lower quality

Other evidence of impact & contributionFocus on international/high profile contributions

– Invited keynote addresses especially at major international conferences

– Editorial roles especially at international/top journals

– Roles in learned societies, prestigious academic committees

– Academic awards and prizes, fellowships

– Prior competitive grant funding

– Number of PhD completions supervised

– Government or professional association blue-ribbon panels/reports

Other aspects

• Factors that have affected your productivity

– time off academic work, industry job, maternity leave, carer responsibilities, illness

• Special case for why you should be considered an ECR longer than 5 years post PhD due to above.

• Only include convincing and special circumstanceseg not teaching load

Reasons for lower Track Record scores

• no A or A* publications

• no/few first authorships

• lots of chapters in self-edited books or books with little-known publishers

• large number of low level publications

• claims for impact or major contribution that do not seem justified

Advice on the TeamReaders will generally combine their appraisal of the track records of all investigators, weighted by order or % contribution.

However the overall balance of the team is also relevant:

• experienced and junior researchers (ECRs, APDs)

• track record of working together

• complementary skills

• all CIs/PIs have a clear role and the time available to do it

• avoid huge teams unless for a very good reason

Budget

• Requested items should be clearly linked to the proposed work

• Be sure you ask for everything you need, but avoid obvious padding

• Get your Research Office to check calculations

• Clearly explain why you need each item to do the research

• What will the Research Assistants do exactly, and how did you arrive at the hours or % time you asked for?

• For Research Fellows, who do you have in mind and why can’t you do the work yourself?

• Why can’t your university provide the equipment requested?

• Why do you need to travel, and how was the travel budget calculated?

• Why is teaching relief essential?

• Don’t attempt to farm out core responsibilities (eg project management, data analysis) to consultants

Justification of Budget

Project Description: General Advice

• Write and rewrite as carefully and thoughtfully as you would for a top journal article.

• Remember your audience – mixed expertise.

• Embed your work in the literature but show how it goes beyond what is known/has been done.

• Don’t trash or ignore prior work; acknowledge other traditions

• Be specific about what you will do, how, and why.

• Stick to 10-page limit and don’t cheat!

• Use sub-headings to make life easier for readers

Aims and Background

• Make Aims and Background short (1 page)(no official score but will influence reader’s appraisal)

• What’s the problem and why does it matter?

• How is the proposed research embedded in current international research and theory?

• What are your specific research aims and intended outcomes (dot points)?

• How does this build on what we already know?

Significance and Innovation• Worth 30%

• Selective, up-to-date and critical literature review, arguing from theory and prior empirical research to proposed work.

• Clear theory-driven model and/or hypotheses with good rationales, clear statement of IVs and DVs

• What’s new, how will research advance understanding in this field, what new methods or approaches are employed?

• Figure may help

• 3-4 pages

Things to avoid in Significance/Innovation• Repetition of material on national benefit (focus should

be on theoretical significance, not practical importance)

• Insufficient literature review

• No clear hypotheses

• No clear rationale for hypotheses

• Not clear what the constructs/variables are

• Vague, muddled thinking

• Too much jargon without explanation

Approach• Like the Method section of a journal article.

• Method must allow hypotheses to be tested.

• Clearly describe studies, designs, measures, procedures, participants, sensible timeline, analyses if unusual.

• Be sure Method is consistent with the budget and with the hypotheses and research questions.

• Use of mixed/multiple methods can be a strength. But need to articulate how they will be combined to address research questions.

• Present pilot data if possible (or link to your previous published / in press research)

• Lack of detail on what will be done, how data will be collected, how new constructs will be operationalised etc

• Research plan where later components depend on a particular outcome of earlier component (parallel structure is safer)

• The impression that you’ll make it up as you go along,or the doctoral student will – “wishful thinking”

• Stating that the first thing you will do if funded is a literature review – you should have done this already!

• Ditto for developing new measures, interventions, questionnaires - more impressive to cite pilot data

Things to avoid in Approach

National Benefit• May be the critical 10% that gets you over the line

• Avoid glib comments that are not convincing

• What is the potential of the research to result in economic and/or social benefits for Australia?

• Who may be affected, how, and by how much?

• Pure research:– enhance knowledge base– strengthen Australia’s international research position & build

international links– create research training opportunities, build capability and

capacity

Communication of Results

• What academic papers do you plan to write throughout the project?

• What specific journals and other outlets will you target? - be realistic

• If you will develop practical tools or knowledge, how will the profession/ industry/public be informed about it?

Role of Personnel

• Explain who does what exactly

• Be sure all key activities are assigned to a person who is qualified to do them.

• Every CI, PI, APA, APAI, APDI, Fellow, and Research Associate/Assistant needs a clear role.

• No one should just be a research manager.

• Talk up the qualities of the team

Reference List• Demonstrates your credibility, knowledge

• Limit to approx 1 page – you need the other 9 pages for substance

• Give high quality references from top international journals

• don’t over-cite yourself

Research Support

• List all current, past, and requested grants for all CIs and PIs for the years specified.

• Provide a short progress report on all current ARC- funded work.

• Explain delays

• Explain progress (relative to plan)

• List outputs/achievements

Increasing your chances

• Start writing early. Preparing a good proposal takes time.

• Make sure the writing style and organisation of the proposal are clear - use headings and transitions.

• Get feedback from experts in your research area as well as scholars who are NOT experts in your field.

• Give readers the whole submission so they can catch inconsistencies between parts.

• Read successful proposals written by others

• Review for ARC/journals

The rejoinder

The rejoinder typically has little impact on the final WAPR:

• only read by CoE members

• CoE members have little time, and any changes made must be checked against proposals with similar scores

• Rejoinders based on OzReader and IntReader comments

However, it is still important to write a good rejoinder:

• looks bad not to!

• the rejoinder could be critical if there is a discrepancy between CoE panel member scores

• occasionally there may be a factual issue or misunderstanding that you can clear up

• worth writing no matter how good or bad your assessments appear to be

Writing a rejoinder

• Address most important points, not necessarily all

• Take the comments seriously and respond to them rather than dismissing them

• Provide any requested information

• Be reasonable and logical rather than aggressive(ask colleagues for feedback!)

• OK to accept criticism and explain how it will be dealt with

• Take opportunity to update achievements, pilot work

• Don’t be overly self-congratulatory

• Make rejoinder easy to read, not too dense

What if you don’t get funded?

• Don’t give up! – many applications are not funded the first time

• Use internal funds to conduct pilot work

• Revise it for another funding agency or scheme

Proposed new system for peer review at ARC

Consultation Paper

The ARC has released a consultation paper that provides some information on the way they are thinking

Suggested structure: 3 levels

•Level 1 Peer Reviewers

•Level 2 Panel Reviewers

•Level 3 Interdisciplinary Leaders

Goals

• reduce workload of assessors

• assign proposals to assessors with most relevant expertise

• improve quality of outcomes

• better support inter-disciplinary research

• improve feedback to applicants

Level 1 Peer Reviewers

• specialist assessors matched to proposals at 6-digit FOR code level

• 5-10 proposals per round

• provide written evaluations and scores

Level 2 Panel Reviewers

• generalist assessors matched to proposals at 4-digit FOR code level

• grouped into Panel Review Committees

• responsible for 20 proposals per round

• assign proposals to Peer Reviewers

• moderate information from Peer Reviewers plus rejoinder

• make funding recommendations

• develop feedback on uncompetitive proposals

Level 3 Interdisciplinary leaders

• generalist assessors matched to proposals at 2-digit FOR code level

• serve as Chairs of Panel Review Committees, and as members of Interdisciplinary Selection Advisory Committees

• provide broad discipline expertise “to facilitate a comparative overview of recommendations”

• consider committee recommendations & finalise rankings

• make budget recommendations

• provide feedback on unsuccessful competitive proposals

Suggested process for Discovery scheme

• 3-5 Peer Reviewers per proposal

• approximately 25 Panel Review Committees

• Panel Committees work by email or teleconference

• approximately 5 Interdisciplinary Selection Advisory Committees, each made up of 12-20 Interdisciplinary Leaders

Other issues ARC is seeking feedback on

• payment of Level 1 and 2 assessors• methods to encourage participation in peer review by

recipients of ARC funding• matching by FOR codes• selection criteria and weightings• clarification of track record relative to opportunities• how to deal with ECR proposals• introduction of confidence scores for assessors• banding instead of scores• who determines budgets• feedback on proposals• restrictions on submitting proposals

Comments from CoE members 2007

College of Experts’ Views: 2007Good DP Poor DPOverall

Clearly defined & designed project to answer a question worth asking. Elegant writing, sufficient detail to assess feasibility.

Sharp proposals, highly focused & explicitly written to selection criteria

Aims & background clearly specify a problem to be addressed. Describe problem and make need for study obvious.

Gives ½ page statement at start summarising project’s main purpose & key clear aims.

Proposal based on arguments and evidence-based claims, not moral claims. Applies to research design & sections on national benefit etc.

OverallBadly written, weakly structured proposal without a clear research question, an identifiable analytical framework, coherent methodology & clear statement of literature stream to which research would contribute to provide credibility to claims concerning future publications.

Murky, fails to nail & keep track of the key issue – need to see the significant issue flowing through the rationale, method & outcomes.

Skimpy on design detail, unclear about sampling. Spends 3 pages setting out background & then fizzles out in the details of how to do it.

Assumes reader an expert in field; so does not fully describe issues & how they are to be addressed.

College of Experts’ Views: 2007 Good DP Poor DPTeam

Strong track record: Give citation rates & journal impact factors

Publish in high quality outlets

Strong academic pubs - books & chapters by reputable publishers & refereed journals. Publish prior projects in academic literature.

Explains team’s strengths, why mix of people can make a significant contribution & realise project outcomes. Prior work together important.Demonstrated relevant track record & capacity to do research, shown by previous publications, grants, & experience with relevant research methodologies

TeamMediocre - No “A” hits; lots of book chapters in own books;

Many low level publications.

Average publication outlets.

Pubs distantly connected to proposal.

Exaggerated claims about international standing.

Publishing thru endless conference papers. Conference presentations not same as other peer-reviewed publications. CIs with no skills/experience in methods in their proposal, or saying they will hire junior research staff to cover the area.

College of Experts’ Views: 2007 Good DP Poor DPSignificance & Innovation

Engaging; Nails the big issue thru’out and for a general audience, shows why we should care rather than asserting why.

Sells it to any audience, not just an expert

Explicitly argue why research shows theoretical, substantive or applied significance

Clear and expert use of theory, theories used in a novel way to make insightful predictions.

Clear directional hypotheses or research questions.

Significance & Innovation

Addressing a trivial issue Murky, fails to nail & keep track of the key issue – i.e. need to see the significant issue flowing through the rationale, method & outcomes

Raves on about how important question is – by assertion not by demonstration

Gesturing to abstract grand theories when topic calls for reference to specific relevant literature at the "middle range“

No clear hypotheses firmly & coherently grounded in literature.

No clear research questions

Never defines theories & terms or makes clear how they relate to each other.  

College of Experts’ Views: 2007 Good DP Poor DPSignificance & Innovation

Focused on serious issues widely recognised as persistent problems. Might only address one dimension of this.

Explicit about what study is trying to tackle, contribution it will make & justification for this work in terms of social significance

Explicitly argue why research shows theoretical, substantive or applied significance

Explicitly outline claims about innovativeness of project, rather than embed in the text.

Pushing back frontiers in a "scientific" way; if subjective, embracing it in theoretical and empirical approaches.

Significance & InnovationAddresses S&I by moral claims or descriptive accounts not reviewing academic literature.Asserts research is significant or innovative without arguing for why

Figures that include everything but explain nothing.

College of Experts’ Views: 2007 Good DP Poor DPSignificance & Innovation

Argued case for S&I based on existing literature. Locate research in relevant field of research & show how project will add to what is already known.

Strong links to relevant research literature - direct relevance of literature which has received good coverage.

Conveys passion for the problem. Not assume you have expertise but will fully describe the problem and illustrate it.

Significance & InnovationNot just a continuation of something already done or a description.

Does not have a research problem well specified so cannot go on to demonstrate innovation and significance.

Not truly analytical.

Clusters of applications on same vogue issue.

College of Experts’ Views: 2007Good DP Poor DPApproach

Clearly explained approach: Overall research design, sampling strategy, & data collection methods, sources & analysis methods.

Link approach to specific research questions, and justify why approach is appropriate.

Gives detail of method, shows clearly what likely to happen and what means for analysis.

Explain project approach as if to a stranger outside of discipline.

Evidence of pilot work, previous studies & use of proposed methods.

ApproachGrab-bag of different methods, with no clear links to research questions and no justification for why methods are appropriate

Lack of adequate information on which to judge method. Short on detail.

Not being able to tell what IVs and DVs are.

Lack clarity about which level variables are measured at or how they will be measured. Mix levels of analysis in concepts or measures but not know doing so  

College of Experts’ Views:2007Good DP Poor DPApproach

Locate methodologically.

When falls outside conventional disciplines, make case for the kind of research and some indication of what this means for the research design.

In interdisciplinary work, locate work in relation to established and recognised intellectual traditions.

Go beyond conventional approaches when unsuitable for answering a key research question.

Approach“Run regressions on data" without carefully developed set of hypotheses and discussion of methodological problems.

Not enough to make general gestures about sorts of things that might come out of a study.

Say data analysis will be done using SPSS/NVIVO (substitute your software preference) with no reference to actual techniques of analysis.

Too much on how to do qualitative research or why statistical technique used, unless different & unique to proposal.

College of Experts’ Views: 2007Good DP Poor DPApproach

Clearly delineate different roles of all members of research team, not just CIs

Includes information about likely appointees and their records if available.

Specifically identifies PhD topic and how it fits into overall project.

Well described, has contingencies if plan A not work.

Modest & lean budget, not inflated but will do the critical research necessary to complete the project.

Give evidence of university contribution other than CI salaries.

ApproachNo specific delineation of roles of CIs and other research personnel

Ask for too many research assistants /associates, CIs not doing much of the work.

Don’t specify a clear role/distinct project for PhD or postdoc.

Asking for things in budget ARC doesn’t support, minor things like batteries & phones, or stats programs uni should provide.

College of Experts’ Views: 2007Good DP Poor DPNational Benefit

Say specifically how results will contribute to relevant policy areas, & some indication of why policy area is significant/important

Give specific other benefits from research eg contribution to Oz's research reputation, dissemination of new data, methods, concepts, theories etc to research community, research training opportunities

WritingGood clear writing in plain English

Concise writing in active not passive voice

Conveys passion for the problem, Not assume you have expertise but fully describe problem & illustrate it

National BenefitIs asserted and not being argued.Says research will contribute to policy development in area X, without saying why

WritingMuddled writing, grammatical errors.

Verbose prose and saying simple things in complex ways

Jargon & really dense prose

Ambiguous language to deal with complexity or using technical or obscure language.