1 ABOLaureen Maines 2008 Choosing Participants for Financial Accounting Experiments October 10, 2008 ABO Doctoral Consortium.

1ABO Laureen Maines 2008

Choosing Participants for Financial Accounting Experiments

October 10, 2008

ABO Doctoral Consortium


Objectives

• Identify participants in financial accounting experiments

• Identify characteristics of participants

• Discuss factors to consider when choosing participants

• Provide advice for finding participants


Financial Reporting Process

Standard Setters

Preparers Financial Reports

Analysts Stock Market Reaction

Auditors Investors

Regulators Creditors


Research Process

Research Question (Theory)

Experimental Design and Task

Participants (Theory)


Variables Influencing Judgments and Decisions

• Decision-Maker Characteristics

• Task Characteristics

• Environmental Characteristics

(Sarah Bonner, Judgment and Decision Making in Accounting, Prentice-Hall, 2007)


Participant (Decision-Maker) Characteristics

• External / Easily observable or measured• Gender, Age, Professional Certifications, Work Experience

• Internal / Need to be measured via test or possibly manipulated

• Abilities

• Knowledge

• Preferences


Participant Characteristics - Abilities

• Abilities• Verbal abilities – reading comprehension

• Reasoning abilities - interpret data, reason from analogy, reason analytically

• Spatial abilities – identify and comprehend patterns

• Measurement• Typically through standardized college admission tests – SAT and GMAT

• Manipulation • Difficult in adults


Participant Characteristics - Knowledge

• Knowledge• Semantic

• General world knowledge, domain-specific knowledge, sub-specialty knowledge

• Episodic knowledge/beliefs based on personal experience

• Incentives, reactions of other parties, empirical regularities

• Measurement• Knowledge-specific tests (e.g., CPA exam questions)

• Post-experimental questions about beliefs

• Manipulation • Possible, but need to be careful


Participant Characteristics - Preferences

• Preferences (Utility)• Money (Risk Aversion)

• Effort

• Honesty, ethical behavior, welfare of others

• Measurement• Validated tests from economics, psychology, sociology

• Manipulation• Induced preferences for money (Berg et al. QJE 1986)


Role of Participant Characteristics

• Independent Variable• Decision-maker characteristic directly affects dependent variable

• McDaniel et al. (TAR 2002)

• Moderating Variable• Decision-maker characteristic changes the relation between a task or environmental variable and the dependent variable

• Elliott (TAR 2006)

• Neither of above – just choosing participants for your task

• Think in terms of a moderating role


Participant Characteristics as Independent Variable (McDaniel et al.)

X Y

Experience(Accounting)(Business)

Judgment(Reporting Issue Identified – recurring vs. nonrecurring)

Causal relation


Participant Characteristics as Independent Variable

• Not typical in financial accounting research

• Typically requires measuring participant characteristics – external/internal

• May be hard to avoid confounding • External characteristics (e.g., years of experience) typically represent a number of internal characteristics (e.g., knowledge, ability)

• For example, knowledge and ability are rarely independent (Link 2 in Libby and Luft AOS 1993)

• A problem if you want to establish causality for internal characteristics (need manipulation + random assignment, not measurement)


Participant Characteristics as Moderating Variables (Elliott 2006)

Y

U

GAAP – ProForma Reconciliation(Yes/No)

Decision

(Investment Decision)

Knowledge

X


Participant Characteristics as Moderating Variables

• Use this approach even if you aren’t planning on a moderating variable approach

• Does theory suggest that judgments and decisions will depend on characteristics (abilities, knowledge, preferences) of participants? Will the JDM process or outcomes change?

• Does the level of relevant characteristics vary among relevant decision-makers or participant groups?


Participant Characteristics as Moderating Variables

• If answers to prior questions are yes, how do I proceed?

• Embed theory in paper and test multiple groups that should differ based on theory – true moderating study

• Examine only one group

• What group is most important in terms of the research question?

• External validity limitations


Participant Characteristics – Professionals versus Students

• Some generalizations

• Basic reasoning abilities typically don’t change much after adulthood

• College students likely have similar abilities to most adults

• Be careful – there can be cross-sectional variation - top people in any field have survived tournaments and are likely to have higher ability




• Knowledge is a big factor in accounting experiments

• Semantic knowledge – world knowledge and sub-specialty knowledge increases with experience

• Episodic Knowledge / Beliefs also change with experience – personal experiences with incentives, behavior of other people




• Preferences

• Reaction to economic incentives is basic human reaction

• Experimental research based on economic theory typically can use abstract tasks and college students


Issues – Working with Professional Participants

• Negative externalities of using professionals when not needed: limited resource (Libby, Bloomfield & Nelson AOS 2002)

• Places high demands on researcher – good research question requiring professionals, good design, good instrument

• Personal experiences of professionals lead to expectations that will create problems if not met

• Need to run materials by professionals and pilot test


Issues – Working with Professional Participants

• Selection and non-response bias may be stronger with professionals than with students (Peecher and Solomon 2001)

• End up using one firm – how general are your results?

• Professionals are busy people which leads to non-response

• Need to get them interested and committed

• Work around their schedule


Source of Participants

• Preparers

• Alumni directory – controllers, CFO’s

• Financial Executive Institute

• Institute of Management Accountants



• Investors - Nonprofessional

• College Students

• Think carefully about research question and task

• Insights on using MBA students as proxies for investors – Elliott et al. (TAR 2007)

• National Association of Investors Corporation – 400,000 members

• See Hodge (AH 2003)



• Financial Analysts (sell and buy side)

• Alumni Directory

• Individual investment banks, financial management firms, pensions funds, etc.

• Association for Investment Management and Research

• Nelson’s Directory of Investment Managers



• Standard setters and regulators

• Good luck!


Conclusions

• Appropriate choice of participants requires a good research question, a good design and task, and careful consideration of how personal characteristics affect JDM

• How does theory suggest different levels of a characteristic would change JDM?


Conclusions

• Be enthusiastic and able to explain why people should care about your research

• Practice describing your research to your accountant and non-accountant friends

• BE PERSISTENT !!! (but not annoying!)


References

• Berg, J.E., Daley, L.A., Dickhaut, J.W., and J. O’Brien. 1986. Controlling preferences for lotteries on units of experimental exchange. Quarterly Journal of Economics 101 (May): 281-306.

• Elliott, W. B. 2006. Are investors influenced by pro forma emphasis and reconciliations in earnings announcements? The Accounting Review 81 (January): 113-133.

• Elliott, W. B., F. Hodge, J. Kennedy, and M. Pronk. 2007. Are MBA students a good proxy for nonprofessional investors? The Accounting Review (January):139-168.

• Hodge, F. D. 2003. Investors’ perceptions of earnings quality, auditor independence, and the usefulness of audited financial statements. Accounting Horizons 17(Supplement): 37-48.

• Libby, R., Bloomfield, R., and M. Nelson. 2002. Experimental research in financial accounting. Accounting, Organizations and Society 27 (November): 775-810.

• Libby, R. and J. Luft. 1993. Determinants of judgment performance in accounting settings: Ability, knowledge, motivation, and environment. Accounting Organizations and Society 18 (July):425-450.

• McDaniel, L.S., Martin, R. D., and Maines, L.A. (2002). Evaluating financial reporting quality: The effects of financial expertise vs. financial literacy. The Accounting Review 77 (Supplement): 139-167.

• Peecher, M. and I. Solomon. (2001). Theory and experimentation in studies of audit judgments and decisions: Avoiding common research traps. International Journal of Auditing (November): 193-203.

1 ABOLaureen Maines 2008 Choosing Participants for Financial Accounting Experiments October 10, 2008 ABO Doctoral Consortium.

Documents

abolaureen maines

characteristics abilities

external characteristics

measurement slide

careful slide

moderating role slide

task participants theory

generalizations knowledge