School of Education University of Tampere, Finland Issues in Study Design Petri Nokelainen [email protected]
Mar 29, 2015
CONTENTS
Issues in Study DesignWriting Scientific Reports
Ten Questions About YOUR Research
AND OR
AND
Scientific research
Theoretical
Empirical
T1: Research body T2: Innovation E1: Numerical data E2: Textual / nominal data
ANDOR
Issues in Study Design
Qual: {T1,T2,E2} {T1,T2}
Quan: {E1} !
Theoretical research
Empirical research
A: {T1,T2}
B: {T1,T2,E1}C: {T1,T2,E2}D: {T1,T2,E1,E2}
Issues in Study Design
B: {T1,T2,E2}
C: {T1,T2,E1}
Research question
Description of person’s attitudes, feelings, meanings, knowledge, etc. about S.
How many cases in the data D have certain attribution x?
How many cases in the data D have certain attribution y?
Is there a relationship between x and y?
Is that relationship causal?
Issues in Study Design
Issues in Study Design
(Nokelainen, 2008, p. 119)
Issues in Study Design
• The left-hand side of the figure shows two main categories of data collection:– Probability sample (PS) and – Non-probability sample (NPS).
• Both methods aim to produce a scientific, representative sample from the target population.
Issues in Study Design
• According to Jackson (2006), a representative sample is “like” the population. – Thus, we can be confident that the results we find based on
the sample also hold for the population. • This is not a problem with PS, which is based on
random, stratified or cluster sampling. – In random sampling each member of the population has an
equal likelihood of being selected into the sample. – Stratified random sampling allows taking into account
different subgroups in the population. – If the population is too large for random sampling of any sort,
cluster sampling is applied.
Issues in Study Design
• Problems arise with NPS as the individual members of the population do not have an equal likelihood of being selected to be a member of the sample.
• The most commonly applied NPS technique is convenience sampling (CS) in which participants are obtained wherever they can be found and wherever is convenient for the researcher (Hair, Anderson, Tatham & Black, 1998).
Issues in Study Design
• Why, then, educational scientists use NPS, typically CS? – Simply because it “tends to be less expensive [than RS] and it
is easier to generate samples using this technique” (Jackson, 2006, p. 84).
Issues in Study Design
• However, on the lower left-hand part of the figure, it is shown that when researcher ensures that the CS is like the population on certain characteristics (location and dispersion descriptive statistics about, for example, age and job title), it becomes a quota sample (QS). – A quota sample is better than a
CS as it allows us to ensure that the results we find based on the sample also hold for the population.
Issues in Study Design
• The upper part of the figure contains two sections, namely “parametric” and “non-parametric” divided into eight sub-sections (“DNIMMOCS OLD”).
• Parametric approach is viable only if – 1) Both the phenomenon modeled
and the sample follow normal distribution.
– 2) Sample size is large enough (at least 30 observations).
– 3) Continuous indicators are used.– 4) Dependencies between the
observed variables are linear. • Otherwise non-parametric
techniques should be applied.
Issues in Study Design
• First, study design (D) is made on the basis of the research question and major goal.
• According to de Vaus (2004, p. 9), “… research design is to ensure that the evidence obtained enables us to answer the initial question as unambiguously as possible.”
Issues in Study Design
• In order to obtain relevant evidence, we need to specify the type of evidence needed to answer the research question.
• More specifically, we need to ask: Given this research question, what type of evidence (data) is needed to answer the question in a convincing way?
Issues in Study Design
• Sometimes we proceed with the so-called qualitative designs, sometimes a quantitative orientation is more appropriate, and sometimes we work both qualitatively and quantitatively (mixed-methods research, for a thorough discussion, see Brannen, 2004).
• Methodological, conceptual etc. triangulation.• Design research is quite new approach, see Bannan-
Ritland (2003).
Issues in Study Design
• Experimental design (a.k.a. ‘pretest post-test randomized experiment’) is the most recommended approach, but only possible with a random sample (a.k.a ‘probability sample’) and random assignment (participants are randomly selected for the experimental and control groups).
• Research is conducted in a controlled environment (e.g., laboratory) with experiment and control groups (threat to external validity due to artificial environment).
• Using experimental design, both reliability and validity are maximized via random sampling and control in the given experiment (de Vaus, 2004).
Issues in Study Design
Random sample
Exp.
Contr.
Pre
Pre
I
-
Post
Post
Random assignment
Issues in Study Design
Random assignment to
groupsPretest Intervention Post-test
Experimentalgroup
Measurement (X) Treatment Measurement (Y)
Control group
Measurement (X) No treatment Measurement (Y)
Issues in Study Design
• Quasi-experimental design (a.k.a. non-equivalent groups design) resembles experimental design but lacks random assignment (sometimes also random sampling) and controlled research environment.
• This type of design is sometimes the only way to do research in certain populations as it minimizes the threats to external validity (natural environments instead of artificial ones).
Random / convenience
sample
Exp.
Contr.
Pre
Pre
I
-
Post
Post
Issues in Study Design
• The most popular quantitative approach in educational research, correlational design (a.k.a. ‘descriptive study’ or ‘observational study’), allows the use of non-probability sample (a.k.a ‘convenience sample’).
• Most correlational designs are missing control, and thus loose some of their scientific power (Jackson, 2006). – Some research journals accept factorial analysis (main and interaction
effects, e.g., MANOVA) based on quasi-experimental design.
Convenience
sample Exp.
Pre
IPos
t
Issues in Study Design
– Observational studies can further be classified into cross-sectional and longitudinal studies (see Caskie & Willis, 2006). • Longitudinal design includes series of measurements over
time.– Change over time, age effect.
• Cross-sectional study involves usually one measurement and is thus considerably cheaper and faster to conduct (although producing less controllable and less powerful results).
– If there are several measurements, individual participants answers are not connected over time (e.g., due to anonymity).
– Causal conclusions are usually out of scope of this research type (ibid.).
Issues in Study Design
• Longitudinal design– One sample that remains the same throughout the study.– Longitudinal study produces more convincing results as it allows the
understanding of change in a construct over time and variability and predictors of such change over time (ibid.).
– However, it takes naturally more time to carry out and suffers from participant drop-out.
Sample Pretest Intervention Post-test
Random sample
Measurement (X) Treatment Measurement (Y)
Issues in Study Design
• Cross-sectional design– Measurement is conducted once (or several times) and the sample varies
throughout the study.
Sample Pretest Intervention Post-test
Convenience or random sample
Treatment Measurement (Y)
Convenience orrandom sample
No treatment Measurement (Y)
Issues in Study Design
RS
RS
TEST
TEST Pre
CONTROL
Pre
Pre
I
I
-
Post
Post
Post
Pre
-Pos
t
CS TEST Pre
IPos
t
CONTROL
RANDOMSAMPLING
RANDOMSELECTION
PRETEST-POSTTEST RANDOMIZED EXPERIMENT
NON-EQUIVALENT GROUPS DESIGN
CORRELATIONAL DESIGN
Issues in Study Design
• Why do, then, educational scholars use correlational designs over controlled experiments?
• The first answer is simple: Correlational designs are far easier, faster and inexpensive to conduct than experimental designs.
• The second answer is more complex as we need to ask if the controlled experiment approach is at all viable method to study educational research questions.
Issues in Study Design
• In science and psychology, most areas of interest are quite easily quantifiable and replicable (like, for example, freezing point of chocolate or systolic blood pressure).
• However, in educational research we study, for example, topics like ‘pedagogical aspects of digital learning material’ (Nokelainen, 2006) or compare pre-existing characteristics of interest (e.g., gender, age, educational level). – In such situations researchers do apply correlational designs,
but still aim to employ different types of data in the analysis with a complementary way (quasi-experimental study).
Issues in Study Design
• Case study design is applied in qualitative research. – The aim is to collect information from one or more cases and stydy,
describe and explain them through how and why questions.– Cases are represented, for example, by individuals, their communication
and experiences. (For thorough discussion, see Flyvbjerg, 2004.)
Issues in Study Design
– As a conclusion, Abelson’s (1995) concept of statistics as principled argument becomes useful: • Data analysis should not be pointlessly formal, but instead
“ ... it should make an interesting claim; it should tell a story that an informed audience will care about and it should do so by intelligent interpretation of appropriate evidence from empirical measurements or observations” (p. 2).
Issues in Study Design
• Second, optimal sample size (N) is divided into two sections in the figure: – Samples that operate in the
optimal area (n 30 – 250) for traditional parametric frequentistic techniques (Black, 1993; Tabachnick & Fidell, 1996), such as t-test or exploratory factor analysis, and the samples that fail to do so (n < 30 or n > 250).
Issues in Study Design
Estimation of sample size
• N– Population size.
• n– Estimated sample size.
• Sampling error (e)– Difference between the true
(unknown) value and observed values, if the survey were repeated (=sample collected) numerous times.
• Confidence interval– Spread of the observed values that
would be seen if the survey were repeated numerous times.
• Confidence level– How often the observed values
would be within sampling error of the true value if the survey were repeated numerous times.
(Murphy & Myors, 1998)
• Traditional non-parametric techniques, such as Mann-Whitney U-test, are considered to operate robustly, also with small samples (-> lack of power?). – Bayesian approach, however, is free of such restrictions.
Issues in Study Design
• Third, independent observations (IO) are always expected, also in time series analysis.
Issues in Study Design
• Controlled experiment designs, when conducted properly, rule out IO violations quite effectively (Martin, 2004), but correlational designs usually lack such control (e.g., to rule out employee’s co-operation when they respond to the survey questions). – On the other hand, some
qualitative techniques, like focus group analysis (Macnaghten & Myers, 2004), are heavily based on non-independent observations as informants are asked to talk to each other as an important part of the data collection.
Issues in Study Design
• Fourth, parametric techniques assume continuous (c) measurement level (ML) of indicators (i.e., so called ‘quantitative’ variables).
Issues in Study Design
Discrete
0 1 2, ..
PHENOMENON
OBSERVATION
Discrete
0 1 2, ..
Issues in Study Design
Discrete
0 1 2, ..
PHENOMENON
OBSERVATION
Continuous
0 ∞
Continuous
0 ∞
Issues in Study Design
Measurements
QuantitativeQualitative
Discrete
Nominal
Continuous
Ordinal Ordinal Interval Ratio
Issues in Study Design
• Non-parametric analysis is based on ordering of values and thus discrete (d) or, when applicable, nominal (n) values are expected (i.e., so called ‘qualitative’ variables). – A respondent’s income level
(euros) or age (years or months) is a representative example of the first indicator type.
– A Likert scale from 1 to 5 is an example of the second indicator type (ordered discrete values).
– Respondent’s gender is an example of the third indicator type (nominal discrete values).
Issues in Study Design
– It is important to note that the central limit theorem, discovered by Pierre-Simon Laplace (1749 - 1827), assures an approximate normal distribution for practically all sums of independent random variables. • For example, it allows the use of parametric t-test with
binomial or ordinal indicators (as the sample of normally distributed group means are compared, not the indicator values themselves).
– Bayesian analysis is based on discrete values, and thus, continuous values must be disceticized (automatically or manually) before the analysis.
Issues in Study Design
• Fifth, parametric techniques are technically based on the assumption of the multivariate distribution (MD) that is normal (n) by nature.
• Non-parametric techniques expect any shaped similar distributions (s). – This is a great news to anyone
who has collected real-life educational science empirical data and checked both univariate and multivariate variable distributions as usually almost all variables violate quite heavily against the normal distribution assumption with small sample sizes (e.g., below n = 100).
Issues in Study Design
• Some researchers try to force their indicators to follow multivariate normal distribution by applying various transformation techniques (e.g., logarithmic, square), but with varying success. – The motivation for transformations lies behind the fact that in
order to enable parametric analysis (i.e., based on, e.g., normal distribution) the bivariate or multivariate statistical dependencies (S) must be linear (l).
– It is important to note that this assumption does not hold for the Bayesian techniques.
Issues in Study Design
Non-parametric statisticsChi-square 2
Multiway Frequency Analysis 2
Spearman Rank Order Correlation rS
Mann-Whitney UWilcoxon Signed Rank
Kruskal-Wallis HFriedman
Bayesian dependency modeling (B-Course)
Logit analysis, Logistic regressionBayesian classification modeling (B-
Course)Categorical variable modeling (Mplus)
Parametric statistics
Pearson Product Moment Correlation rP
Independent-samples tPaired-samples tOne-way between-groups ANOVA FTwo-way repeated-measures ANOVA FANCOVA, MANOVARegression analysis RExploratory factor analysis Principal component analysis Cluster analysisDiscriminant analysis Classification analysisConfirmatory factor analysis
Issues in Study Design
• Sixth, extreme values, outliers (O), affect the results and, thus, the conclusions, of some parametric techniques severely (e.g., regression and discriminant analysis) and should be recognized and removed (see, e.g., Tabachnick & Fidell, 1996). – Non-parametric analysis
techniques are not affected by such values as their analysis is not based on multivariate normal assumption (i.e., linear dependencies between variables).
Issues in Study Design
• Seventh, when calculating correlations (C), Pearson product moment correlation (rP) should be applied with continuous indicators, and Spearman rank-order correlation (rS) with ordinal indicators. – Both techniques are valid
to detect linear dependencies.
Issues in Study Design
• The last point is to discuss about the two types of statistical dependencies (S) among the variables under analysis, namely linear (l) and non-linear (nl).
Issues in Study Design
• It is natural to assume, that both parametric and non-parametric techniques designed to detect linear dependencies work best with samples that contain linear dependencies. – However, there are non-
linear techniques, such as Bayesian analysis and neural networks that also allow the investigation of both dependency types.
Issues in Study Design
• The figure contains a reference to the qualitative analysis techniques, referring here to the empirical textual evidence based approach (e.g., individual or focus group interviews, narrative stories).
Issues in Study Design
• Firstly, it is obvious that qualitative research operates with small samples (usually n < 30).– There is nothing suspicious working with small samples:
Bartlett, Pavlov, Piaget and Skinner did that too!
Issues in Study Design
• Secondly, probability samples could also be used by qualitative researchers (as stated in the figure), but not as the only way to produce scientifically important findings.
• Gobo (2004) illustrates this by listing important qualitative research studies based solely on non-probability samples: – Alvin Gouldner (1920-1980)– Howard Becker (1928-)– Ernest De Martino (1908-1965)– David Sudnow (1938-2007)– Aaron Cicourel (1928-).
Issues in Study Design
• Gobo (2004) defines a new concept of generalizability for qualitative research by arguing that the concept of generalizability is based on the idea of social representativeness, which allows the generalizability to become a function of the invariance (regularities) of the phenomenon. – Thus, “The ethnographer does not generalize one case or event
… but its main structural aspects that can be noticed in other cases or events of the same kind or class.” (id., p. 453.)
Issues in Study Design
• Thirdly, both qualitative and Bayesian analysis techniques allow researcher to apply a priori input to the modeling process and update the model on the basis of increased level of knowledge.
Issues in Study Design
Issues in Study DesignWriting Scientific Reports
Ten Questions About YOUR Research
http://www.uta.fi/aktkk/lectures/sw
NO TURNING BACK!
Writing scientific reports
Literature review
Research questions / hypotheses
Design of the study
Methodology Link between RQ’s
and statistical analyses
Data collectionData analysisWriting scientific reportPeer Review
Conclusions
Theory
Discussion
RQ’sMethodology
Results
SampleMeasurements
Database of scientific
knowledge
Publication of the report
Original idea for the
research
Writing scientific reports
1. Title2. Author(s) name(s) and affiliation(s)3. Abstract and keywords4. Introduction / Goals or aims of the study (periodicals: research
questions)5. Theoretical framework / literature review (periodicals: research
questions)6. Research questions (dissertation)7. Method
7.1 Sample, participants7.2 Measures / instruments7.3 Procedure7.4 Statistical analyses
8. Results9. Conclusion(s) and/or Summary10. Discussion11. Acknowledgements / credits12. References13. Appendix(es)
• 1. Introduction• School leadership is currently one of the most widely studied and
published areas in social sciences. However, leadership as a social process, affecting both end products and personnel emotions, is seldom studied (Nokelainen & Ruohotie, 2006). In this sense one interesting direction to look at is Emotional Intelligence (EI) research that has recently become one of the most important constructs in modern psychological research. EI refers to “the competence to identify, express and understand emotions, assimilate emotions in thought, and regulate both positive and negative emotions in one and others” (Matthews, Zeidner, & Roberts, 2002, p. 123).
Writing scientific reports
• 2. Theoretical Framework • The theory as formulated by Salovey and Mayer (1990; Mayer &
Salovey, 1997) framed EI within a model of intelligence. Goleman’s model formulates EI in terms of a theory of performance (1998b). Goleman argues (2001) that an EI-based theory of performance has direct applicability to the domain of work and organizational effectiveness, particularly predicting excellence in jobs of all kinds, from sales to leadership. Goleman, Boyatzis and McKee further state (2002, p. 38) that EI characteristics are not innate talents, but learned abilities.
Writing scientific reports
• 2. Theoretical Framework • Theoretical framework is summarized in Figure 1. Figure
represents self-regulation (Zimmerman, 1998, 2000) as a system concept (Boekaerts & Niemivirta, 2000) managing leadership behavior through interactive processes between motivation, volition, emotion, attention, metacognition and action control systems. As Markku Hannula (2006) points out, self-regulation should be seen to be much more than mere metacognition.
Writing scientific reports
Figure 1. Self-regulation as a system concept managing leadership competence through interactive processes between different control
systems. (Adapted from Zimmerman, 2000, p. 15-16.)
Writing scientific reports
• 2. Theoretical Framework • Daniel Goleman popularized the term emotional intelligence and
claimed that EI was “as powerful and at times more powerful than IQ” in predicting life success (1995, p. 34). He aimed to show in his studies that emotional and social factors are important (1995, 1998a), but his “views on EI often went far beyond the evidence available” (Brackett et al., 2004). A recent study showed that most popular EI and ability measures are only related at r < .22, i.e. about five per cent of common variance (Brackett & Mayer, 2003).
Brackett, M. A., Lopes, P., Ivcevic, Z., Pizarro, D., Mayer, J. D., & Salovey, P. (2004). Integrating emotion and cognition: The role of emotional intelligence. In D. Dai, & R. J. Sternberg (Eds.), Motivation, emotion, and cognition: Integrating perspectives on intellectual functioning (pp. 175-194). Mahawah, NJ: Lawrence Erlbaum Associates.
Writing scientific reports
• 3. Method• 3.1 Sample• The non-probability sample consists of 124 Finnish teachers from
four comprehensive (n = 84) and two upper secondary (n = 40) schools. All the schools were located in Helsinki, capital of Finland (about 560 000 inhabitants, 9.3% of total population 5 223 442). Each respondent was personally invited to complete a paper and pencil version of the questionnaire. Participants were asked to evaluate their attitude towards the statements measuring emotional leadership.
Writing scientific reports
• 3. Method• 3.1 Sample• The respondents’ age was classified into four categories: (1) 21 to
30 years old (n = 18, 14.5%); (2) 31 to 40 years old (n = 25, 20.2%); (3) 41 to 50 years old (n = 34, 27.4%); (4) over 50 years old (n = 39, 31.5%). Seventy per cent of the respondents were females (n = 87, 70.2%), the rest were males (n = 29, 23.4%).
Writing scientific reports
• 3. Method• 3.2 Instrument• Emotional Leadership Questionnaire operationalises Goleman and
his colleagues (2002) four domains of emotional intelligence characteristics with 51 items: (1) self-awareness, (2) self-management, (3) social awareness and (4) relationship management. Table 1 depicts four EL domains and the eighteen associated characteristics (see Appendix for item level details).
Writing scientific reports
• 4. Results• Next, we examine with descriptive statistics how subordinates’
evaluated their superior’s emotional leadership. Table 1 shows that the school principals had quite strong self-awareness (M = 3.7 – 3.8, SD = 0.8 – 1.0). This finding is natural, as especially self-confidence is an important characteristic of a good leader. On the other hand, we suspect that this result is partly a self-fulfilling prophecy as teachers expect to see those atypical Finnish mentality characteristics strongly present in their leaders.
Writing scientific reports
• 4. Results
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between the three age groups on the IRSSQ dimensions.
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between the three age groups on the IRSSQ dimensions.
Writing scientific reports
Figure 3. Bayesian network of Finnish school principals Emotional Leadership competencies.
Writing scientific reports
Figure 3. Bayesian network of Finnish school principals Emotional Leadership competencies.
Writing scientific reports
Writing scientific reports
Writing scientific reports
Box-plot
Writing scientific reports
• 5. Conclusions• In this paper, we presented a 51 item self-rating Likert-scale
Emotional Leadership Questionnaire (ELQ) that operationalises Goleman’s et al. (2002) four domains of emotional intelligence. Our goal in this paper was to study with an empirical sample the construct validity of the four-domain model (Goleman et al., 2002) of EL. The non-probability sample consisted of 124 Finnish school teachers from six different capital area schools.
Writing scientific reports
• 6. Discussion• We asked teacher’s to evaluate their superiors according to our
fixed, person-related questions. In the next version of the ELQ, we will add an additional scale measuring the importance of each question in a five-point Likert scale. This allows us to compare personal level EL factors to other measures, for example, the Multiple Intelligences Profiling Questionnaire (MIPQ), an operationalization of Howard Gardner’s’ MI theory, (Tirri, K., Komulainen, Nokelainen, & Tirri, H., 2002).
Writing scientific reports
• References (APA style, http://www.apa.org)• Bar-On, R., Tranel, D., Denburg, N. L., & Bechara, A. (2003). Exploring the
neurological substrate of emotional and social intelligence, Brain, 126(3), 1790-1800.
• Boekaerts, M., & Niemivirta, M. (2000). Self-regulation in learning: finding a balance between learning and ego-protective goals. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of Self-regulation (pp. 417-450). San Diego, CA: Academic Press.
• Carmines, E. G., & Zeller, R. A. (1979). Reliability and Validity. Beverly Hills, CA: Sage Publications.
• EQ Symposium (2004). About Reuven BarOn’s Involvement in Emotional Intelligence. Retrieved April 13, 2007, from http://www.cgrowth.com/rb_biolrg.html
Writing scientific reports
Test your APA knowledge: What is WRONG with the following reference list?
Cohen, J. (1988) Statistical power analysis for the behavioral sciences. Second Edition. Hillsdale, NJ, Lawrence Erlbaum Associates.
Hair, J. F., Anderson, R. E., Tatham R. L. & Black, W. C. (1995). Multivariate data analysis. Fourth edition. Englewood Cliffs: Prentice Hall.
Howell, David (1997). Statistical Methods for Psychology. Belmont, CA: Wadsworth Publishing Company.
Nokelainen, P., & Tirri, H. (2007). The Essential Benefits of Using Bayesian Modeling in Professional Growth Research. In S. Saari & T. Varis (Eds.), Professional Growth, 413-423). Hämeenlinna, FI: RCVE.
Kerlinger, F. 1986. Foundations of Behavioral Research. Third Edition. New York: CBS College Publishing.
Tirri, K., and Nokelainen, P. (2008). Identification of multiple intelligences with the Multiple Intelligence Profiling Questionnaire III. Psychology Science Quarterly, 50(2), 206-221.
Writing scientific reports
Issues in Study DesignWriting Scientific Reports
Ten Questions About YOUR Research
Ten questions about YOUR research
Critical Appraisal Skills Programme (CASP) http://www.phru.nhs.uk/casp/casp.htm
• Rigor• Credibility• Relevance
(Abelson, 1995.)
Screening1. Was there a clear statement of the aims of the study?2. Is a qualitative/quantitative methodology appropriate?Research Design3. Was the research design appropriate to address the aims of the
research?4. Was the sampling technique/recruitment strategy appropriate
to the aims of the research?
Ten questions about YOUR research
Data collection5. Were the data collected in a way that addressed the research
issue?• Justification of the setting for data collection.• Clarification how data were collected (e.g., questionnaire, focus group,
semi-structured interview,..).• Justification of the chosen methods.• Explicit report on methods (e.g., how the instruments were delivered,
what were the instructions, how interviews were conducted, was there a topic guide, how data was stored, ..).
• Explicit report on modifications to the methods during the study.• Discussion of the sample size (effect size, power)/saturation of data.
Ten questions about YOUR research
Reflexivity6. Has the relationship between researcher and participants been
adequately considered?• Critical examination of researchers own role, potential bias and
influence during• formulation of research questions• data collection, including sample recruitment and choice of location.
• How researcher responded to events during the study.
Ten questions about YOUR research
Ethical issues7. Have ethical issues been taken into consideration?
• Detailed description how the research was explained to participants – so that reader is able to assess whether ethical standards were maintained.
• Discussion of the issues raised by the study.
Ten questions about YOUR research
Data analysis8. Was the data analysis sufficiently rigorous?
• In-depth description of the analysis process.• Selection of the statistical techniques/use of thematic analysis (e.g., how
the categories/themes were derived from the data?)• Qualitative: How the data presented was selected from the original
sample to demonstrate the analysis process?• Is a sufficient data presented to support the findings?• To what extent contradictory data are taken into account?• Whether the researcher critically examined their own role, potential bias
and influence during analysis (qualitative: selection of data for presentation.
Ten questions about YOUR research
Findings9. Is there a clear statement of findings?
• Explicit findings.• Adequate discussion of the evidence both for and against the
researchers arguments.• Discussion of the credibility of the findings (triangulation, respondent
validation, more than one analyst,..).• Discussion of the findings in relation to the original research questions.
Ten questions about YOUR research
Value of the research10. How valuable is the research?
• Contribution to existing knowledge and understanding.• Identification of new areas where research is necessary.• Transferability (generalizability/representativeness) of findings to other
populations.• Consideration of other ways how the research may be used.
Ten questions about YOUR research
References
• Abelson, R. P. (1995). Statistics as Principled Argument. Hillsdale, NJ: Lawrence Erlbaum Associates.
• Anderson, J. (1995). Cognitive Psychology and Its Implications. Freeman: New York.
• Bannan-Ritland, B. (2003). The Role of Design in Research: The Integrative Learning Design Framework. Educational Researcher, 32(1), 21-24.
• Brannen, J. (2004). Working qualitatively and quantitatively. In C. Seale, G. Gobo, J. Gubrium, & D. Silverman (Eds.), Qualitative Research Practice (pp. 312-326). London: Sage.
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Second edition. Hillsdale, NJ: Lawrence Erlbaum Associates.
• Fisher, R. (1935). The design of experiments. Edinburgh: Oliver & Boyd.• Flyvbjerg, B. (2004). Five misunderstandings about case-study research. In C.
Seale, J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research Practice (pp. 420-434). London: Sage.
• Gigerenzer, G. (2000). Adaptive thinking. New York: Oxford University Press.• Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual: What you
always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp. 391-408). Thousand Oaks: Sage.
• Gobo, G. (2004). Sampling, representativeness and generalizability. In C. Seale, J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research Practice (pp. 435-456). London: Sage.
• Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate Data Analysis. Fifth edition. Englewood Cliffs, NJ: Prentice Hall.
References
• Jackson, S. (2006). Research Methods and Statistics. A Critical Thinking Approach. Second edition. Belmont, CS: Thomson.
• Lavine, M. L. (1999). What is Bayesian Statistics and Why Everything Else is Wrong. The Journal of Undergraduate Mathematics and Its Applications, 20, 165-174.
• Luoma, M., Nokelainen, P., & Ruohotie, P. (2003, April). Learning Strategies for Police Organization - Modeling Organizational Learning Prerequisites. Paper presented at the Annual Meeting of American Educational Research Association (AERA 2002). New Orleans, USA.
• Nokelainen, P. (2006). An Empirical Assessment of Pedagogical Usability Criteria for Digital Learning Material with Elementary School Students. Journal of Educational Technology & Society, 9(2), 178-197.
References
• Nokelainen, P. (2008). Modeling of Professional Growth and Learning: Bayesian Approach. Tampere: Tampere University Press.
• Nokelainen, P., & Ruohotie, P. (2005). Investigating the Construct Validity of the Leadership Competence and Characteristics Scale. In the Proceedings of International Research on Work and Learning 2005 Conference, Sydney, Australia.
• Nokelainen, P., & Ruohotie, P. (2009). Non-linear Modeling of Growth Prerequisites in a Finnish Polytechnic Institution of Higher Education. Journal of Workplace Learning, 21(1), 36-57.
• Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54(4), 837-847.
• de Vaus, D. A. (2004). Research Design in Social Research. Third edition. London: Sage.
References