Best Fit Journal Ratings Paper (Social Change Sales Pitch Mix)

Dr Stephen Dann

The Australian National University

Best Journals versus Best Fit Journals

A Strategic Orientation to

Research Quality

Knight to RQF’s Bishop 3

• "What we're looking to do is distribute funding according to research that's of high quality and high impact."

So what’s quality?

• Quality is a nebulous entity– Eye of Beholder Model

• I know what quality looks like when I see it

– Groucho Marx Model• I wouldn’t want to be published in a journal that

would publish me

– Quality is Greener on the Other Side• Whatever they did is better than what we did.

RQF on Metrics

• Recommendation:– Metrics should be used to inform the peer review

process rather than replace it.– This recommendation was elaborated to make the

process for the interaction of metrics and the Discipline Panel assessment process more explicit:

• Metrics should not be used in isolation, such as in the ‘shadow exercise’ proposed for the 2008 UK RAE.

• The data should be made available to Assessment Panels at the beginning of the assessment process.

• No attempt should be made to aggregate indicators to produce a single ‘quality score’.

RQF on Ranked Outputs• Ranked outputs: Distribution of selected category(ies) of research

output listed in the Research Group’s ‘body of work’ into ranked tiers, based on predetermined discipline specific rankings for the selected forms of output.

• The basis of the rankings will vary by discipline (they could be derived for – refereed journals, – professional journals, – book publishers, – conferences, or – performance venues

• No weightings should be applied to the different tiers

RQF on research output

• 2.3 Ranked outputs– the main thrust of this indicator is to

encourage researchers to publish in the most prestigious outlets for their discipline

• We set the rules on what’s prestigious.– So it’s time to set the rules in our favour

RQF on impact• The RQFDAG recommends that verifiable impact indicators

• The Impact Statement, of no more than ten pages, must be verifiable and auditable, and include a statement of claims against the impact criteria. The broad criteria will require a demonstration of:– the engagement with end-users recognising the importance of the

research to address a defined social, economic, environmental and/or cultural issue;

– the adoption of the research by the relevant end-user community to generate new policies, products, , processes, attitudes, behaviours and/ or outlooks;

– how the research has produced social, economic, environmental and/or cultural benefits for end-users regionally, nationally and/or internationally; and

– the extent of the benefit arising from the research.

Background to the Model

• Over 20 years, over 20 different ranking scales

• Once the reader looks outside of the top three or five journals, the inconsistency of these lists stand as a tribute to "one size fits nobody" production orientations– Polonsky and Whitelaw (2004, 2005, 2006)

• If only the only consistency was change– Top 3 unshakeable

Ageing of the (sample) Population

Rank Name D.O.B.

(1) Journal of Marketing (JM) 1937

(2) Journal of Consumer Research (JCR) 1974

(3) Journal of Marketing Research (JMR) 1964

(4) Marketing Science (MS) 1982

(5) Journal of the Academy of Marketing Science (JAMS) 1973

(6) Journal of Retailing (JR) 1925

(7) International Journal of Research in Marketing (IJRM) 1959

(8) Journal of Business Research (JBR) 1978

(9) Journal of Advertising (JA) 1972

(10) Journal of Advertising Research (JAR) 1961

Quadrant Model

Break the old rules to make the new rules

Shifting from the League Table

• the use of a one-dimensional scale of "best" is inconsistent with the fundamental principles of the discipline and the practice of the industry – (Corfman, 1991)

Assumption of the Model

• Research output is assumed to have two core components– the goal of developing knowledge within a

specific discipline (academic freedom)

– the business strategy component which represents the organisational research goals of the University (academic pragmatism)

Discipline Research Goals

• Development of the marketing discipline– Pure marketing– Applied marketing (impact)– Industry focused research. (impact)

• Two forms– Academy driven goals

• Eg Vargo and Lusch, services dominant logic

– Academic research freedom• Personal pursuit of research interests

University Research Goals

• Organisational research goals / objectives– University / Funding body– Faculty / School / Marketing Unit

• AACSB (2006)– intellectual contributions should be consistent

with the school’s mission and strategic management processes

Four categories

1. Developing the discipline of marketing

2. Addressing Institutional Research Goals

3. Research Agenda Based Objectives

4. Independent Research Output

• First draft mistakenly presented these as tiers of objectives, and naturally, it was seen as a best to worst priority list

Discipline and University Research Goals

AcademyDriven

Research(Tier 1)

Marketing UnitResearchAgenda(Tier 3)

AcademyDriven

Research(Tier 1)

Marketing UnitResearchAgenda(Tier 3)

University ResearchObjectives

(Tier 2)

Non aligned Research

(Tier 4

University ResearchObjectives

(Tier 2)

Non aligned Research

(Tier 4

Discipline Research Goals

University Research Goals

Marketing Academy Journals

(Quadrant 1)

Marketing Group

Agenda (Quadrant 3)

Research Objectives

(Quadrant 2)

Non aligned Research

(Quadrant 4)

Discipline Research Goals

University Research Goals

Discipline and University Research Goals

AcademyDriven

Research(Tier 1)

Marketing UnitResearchAgenda(Tier 3)

AcademyDriven

Research(Tier 1)

Marketing UnitResearchAgenda(Tier 3)

University ResearchObjectives

(Tier 2)

Non aligned Research

(Tier 4

University ResearchObjectives

(Tier 2)

Non aligned Research

(Tier 4

Discipline Research Goals

University Research Goals

Marketing Academy Journals

(Quadrant 1)

Marketing Group

Agenda (Quadrant 3)

Research Objectives

(Quadrant 2)

Non aligned Research

(Quadrant 4)

Discipline Research Goals

University Research Goals

engagement with end-users

adoption of the research by the

relevant end-user community

adoption and benefits for relevant end-users

adoption of the research by the relevant end-user

community

Quadrant 1Developing the discipline of marketing

• Research Agenda: – pursuit of the enhancement and development of the discipline

• Theoretical developments• Methodological developments• Pure science approaches• RQF End user is the marketing academy.

• Which journals are listed?– Constitutes a list of the journals published by the respective

academies of marketing across the major geographic areas• Australasia, Europe and America in current list

– Same nature, same cluster

Only list that should remain constant between universities

Quadrant 1 List

• Academy of Marketing Science Review• Australasian Marketing Journal• International Journal of Research in Marketing• Journal of International Marketing• Journal of Marketing• Journal of Marketing Management• Journal of Marketing Research• Journal of Public Policy & Marketing• Journal of the Academy of Marketing Science

Quadrant 2Addressing Institutional Research Goals

• Research Agenda: – framework to encourage academics to pursue the research

interest of the university / funding agency– End user: University stakeholders, university

• Which journals are listed?– Journal inclusion is based on providing a matching the research

strength of the organisation against the market demand for the research output product

• Requires research and hack work

• Should be relatively similar for organisations with shared/common goals– Opportunity to pursue diversity as sought by Bishop & DEST

Sample Quadrant 2

• Sample Institutional Goal– Advance knowledge of Australian research

• Journals selected based on the extent to which they publish Australian research– Method 1: Journal keyword search

• search by university title• Search by “australian/australia”

– Method 2: Michael Polonsky– Polonsky, Garma and Mittelstaedt (2006) “An examination of

the globalisation of authorship in publishing in 20 leading marketing journals” European Business Review 18 (6)

– Method 3: Cross institutional dataset

Sample Quadrant 2

• Key journals– Asia-Pacific Journal of Marketing and Logistics– European Journal of Marketing*– Industrial Marketing Management– Journal of Brand Management (The)– Journal of Business Research– Journal of Consumer Behaviour– Journal of Consumer Marketing (The)– Journal of Product and Brand Management– Marketing Intelligence and Planning

• *Side note: I was once reprimanded by the editorial board of EJM for describing them on ELMAR as “The European Journal of Australians Publishing in Marketing”

Quadrant 3: Research Agenda

• Research Agenda– Diversification based on collective group research agenda

• Collective agenda of is based on the aggregated areas of research specialisations of the staff

• Self reported identification of research interest

– RQF End user: Business, students, academics

• Which journals are listed?– Discipline specific journals– Sub-disciplinary specific journals – Journals with a specific research concentration in an sub

discipline / sub set of marketing

• Should have a wide range of variation between universities

Create the Research Agenda

• Method: Aggregated Research Agenda– Step 1: Have the research staff of the unit self identify their

research areas against the ANZMAC track stream lists• (eg primary research, secondary research, PhD topic, grants

research)

– Step 2: Research streams are tallied to form an aggregate list of research areas, which is used to form a rank order list.

– Step 3: Cluster track streams against either Polonsky and Whitelaw’s (2006) “areas of research interest” or Lehmann (2006) journal categories

Convert the Research Agenda

• Method: Convert Research Agenda into Target Journals– Step 1: Identify Research Agenda Output Journals

• Option 1: Using Lehmann (2006) “Journal Evolution and the Development of Marketing” Journal of Public Policy & Marketing 24(1) pp138-140

• Option 2: Independent research to select journals based on the number of articles containing the relevant research keyword, and the stated research goal of the journal.

• Option 3: Cross institutional dataset of categorised journals

– Step 2: Prioritise research agenda journals according to the match between the research project and the needs of the target market journal

• Is the paper the best match for the objectives of the journal?

Sample Research Agenda

Sum P&W(2006) ANZMAC Track Areas

12 • Marketing & Society

• Corporate Responsibility, Social Issues and Social Responsibility Marketing and Society Non profit marketing Political Marketing Social Marketing

11 • Marketing Strategy

• Entrepreneurship, Firms in Networks, Marketing Strategy, New Product Development, Supply Chain Management, Financial & Value-Based Marketing

10 • Marketing Communications

• Advertising / Marketing Communication / IMC, Branding

10 •Marketing Communications

•Advertising / Marketing Communication / IMC, Branding

• Sample Quadrant based on Lehmann 2006– Journal of Advertising Research – Journal of Advertising – Journal of Current Issues and Research in Advertising – International Journal of Advertising– Journal of Marketing Communications – Journal of Promotion Management

Quadrant 4Independent Research Output

• Research Agenda– Academic freedom.– Personal research agenda that is not constrained by

the discipline, organisation or unit goals– RQF End user: Other academics/practitioners within

the research area

• Which journals are listed?– The rest of the journals

• Should vary between individuals within a group

Quadrant 4• recognition that it is possible for a quality research outlet to not be

relevant to the research agenda of an institution. – Inclusion in Quadrant 4 is not a criticism of the quality of the journal or

the research.

• used to defend the ability of the individual researcher to pursue research beyond the goal of the university, academy or marketing unit agenda.

• Can easily be destroyed if it is used for sniping between marketing academics

• This is not the Fourth Tier (Group of Death) of the UQBS /AoU lists

Supporting materials

Cross Institution Journal Verification

• Cross verification of journals– Potential ANZMAC Collaboration

• Database shared between marketing groups to identify the match between

– common objectives and journals– Research interest clusters and journals

• Creative Commons Licensed (Attribution Share-Alike) model

– allows for institutions to contribute to the dataset without seeking prior permission, but prevents any single entity from capturing the dataset.

Supporting Behaviours• Publicise the research clusters

– Know what is being published where

• Lay down covering fire– Cite each other, Cite early, cite often– Cite ANZMAC papers

• Create an ANZMAC marketing research output index– shows who’s publishing what and where to accelerate the process of cross-citing

• Specify RQF target end user community.– Target academics. Target teaching. Cross cite, and use in the classrooms.

• Cheat to Win: RQF is one of the largest prisoner’s dilemma games invented – so let’s cheat and communicate with each other.

Questions?