Contemporary Research Practices Carlo Magno, PhD. 2nd NATIONAL CONFERENCE ON COMPUTERIZED STATISTICAL MODELING AND RESEARCH METHODS Tagbilaran City, BOHOL May 6-8, 2015 5th NATIONAL SEMINAR-WORKSHOP ON ACTION RESEARCH AND COMPUTERIZED ITEM/TEST ANALYSIS AND STATISTICS
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Contemporary Research Practices Carlo Magno, PhD.
2nd NATIONAL CONFERENCE ON COMPUTERIZED STATISTICAL MODELING AND RESEARCH
METHODS
Tagbilaran City, BOHOL May 6-8, 2015
5th NATIONAL SEMINAR-WORKSHOP ON ACTION RESEARCH AND
COMPUTERIZED ITEM/TEST ANALYSIS AND STATISTICS
Contemporary Trends on 10 Perspectives
1. Research is written for an international audience 2. Research is becoming an interdisciplinary process 3. Research designs are based on the needs of the
researchers 4. Identify sample size based on power analysis 5. Adapting the instrument necessary for the sample 6. Instruments validity and reliability computed using the
sample data 7. Computation of the effect size with the NHST 8. Thesis and dissertation beyond the IPO 9. Publishing research reports in international abstracted
and indexed journals 10. Concern on author impact factor
Research written for an international audience
– Initiatives supporting ASEAN Integration
– Research databases easily accessible in the internet
– Implication:
• Specifying whether events are emic (part of the social group) or etic (perspective of an observer)
• Is there a need to separate local and foreign reviews?
• Using more of published studies than books…
• Country of origin in describing the participants
Research becoming an interdisciplinary process
• Looking at the complementary across and within fields: – education and psychology
– biology and education
– education and computer science
• “Communities of practice” generating knowledge
• New shared practices are discovered
• Apprenticeships, participations, and worldviews are developed.
Research designs based on the needs of the researchers
• Nonexperimental designs according to Johnson (2001)
Identifying sample size based on power analysis
No real effect Real Effect
Reject H0 Type 1 error α (.01, .05)
Ho not rejected Type 2 error β (small as possible)
1-β Statistical
power
Slim chance of concluding that the treatment is
effective, despite the fact that it is
Statistical Power
• β=.20 (the error of rejecting a true Ho is 4x more serious than the error of not rejecting a false Ho)
• .80=acceptable power
Statistical Power
• The probability of rejecting a false null hypothesis.
• The likelihood that a study will detect an effect when there is an effect to be detected.
• If statistical power is high, the probability of making a Type II error, (or concluding there is no effect when, in fact, there is one) goes down.
Statistical Power
• The power of any test of statistical significance will be affected by four main parameters:
– the effect size
– the sample size (N)
– the alpha significance criterion (α)
– statistical power, or the chosen or implied beta (β)
• Accuracy of the instrument is supported by using IRT and polytomous IRT models for scales
Computation of effect size with the NHST
• Issues on NHST • 1. NHST does not provide the information which the
researcher wants to obtain • 2. Logical problems derived from the probabilistic
nature of NHST. • 3. NHST does not enable psychological theories to be
tested. • 4. The fallacy of replication. • 5. NHST fails to provide useful information because
H0 is always false. • 6. Problems associated with the dichotomous
decision to reject/not reject the H0. • 7. NHST impedes the advance of knowledge.
1X
2X
n df SD1 SD2 t
p
value
30 28 4.6 4.1 2.3 2.3 0.60 0.56
60 58 4.6 4.1 2.3 2.3 0.84 0.4
100 98 4.6 4.1 2.3 2.3 1.09 0.28
500 498 4.6 4.1 2.3 2.3 2.43 0.02*
1000 998 4.6 4.1 2.3 2.3 3.44 0.00*
A researcher wanted to look at the effect of behavior modification technique on the aggression of clients. A group of participants in the experimental group were given behavior modification technique and no treatment in the control. The aggression of the two groups were measured after.
Effect Size
• Cohen (1988) defines the effect size as the extent to which the phenomenon is found within the population or, in the context of statistical significance testing, the degree to which the H0 is false.
• Snyder and Lawson (1993) argue that the effect size indicates the extent to which the dependent variable can be controlled, predicted and explained by the independent variable(s).
Effect Size Measures • Effect size measures of Two In/dependent Groups
– Cohen’s d – Hedges g – Glass Delta
• Correlation Measure of Effect Size – r – χ2 ►Φ; t ► r; F ► r; d ► r
• Effect size for Analysis of Variance – Eta Squared – Omega Square Index of Strength – Intercalss correlation
22
21
2
21
ss
MMd
2
22
1
12
21
ns
ns
MMt
Cohen’s d Formula t-test for independent Means Formula
Cohen's Standard Effect Size Percentile Standing Percent of
Nonoverlap
2 97.7 81.10%
1.9 97.1 79.40%
1.8 96.4 77.40%
1.7 95.5 75.40%
1.6 94.5 73.10%
1.5 93.3 70.70%
1.4 91.9 68.10%
1.3 90 65.30%
1.2 88 62.20%
1.1 86 58.90%
1 84 55.40%
0.9 82 51.60%
LARGE 0.8 79 47.40%
0.7 76 43.00%
0.6 73 38.20%
MEDIUM 0.5 69 33.00%
0.4 66 27.40%
0.3 62 21.30%
SMALL 0.2 58 14.70%
0.1 54 7.70%
0 50 0%
Thesis and dissertation beyond weighted means, difference, and
relationships
• Bad practices
– All students uses the INPUT-PROCESS-OUTPUT in their theoretical framework
– Research questions #1 determined the demographic profile of participants (gender, SES, age, family structure, residence, etc)
– The factors of the instrument are physical, emotional, social, spiritual …
Publishing research in abstracted and indexed journals
• An indexing and abstracting service is a service that provides shortening or summarizing of documents and assigning of descriptors for referencing documents.
Studies that will be published should have Significant contributions
• New argument or
conjecture
• New definition
• Clarification
• illustration or exemplar
• Elaboration
• refutation or rebuttal
• rephrasing
• Rebuttal of question
• Recasting of question
• Evaluation of an earlier
assertion
• New or alternative
interpretation
• Supportive evidence
• Contrary evidence
Concern on the authors impact factor
• The h-index quantifies the actual scientific productivity and the apparent impact of the scientist. The h-index is based on the author’s most cited papers and the number of citations they have received from other articles.
• "A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) papers have no more than h citations each." [For details in calculation, see Hirsch, 2005]
• An h-index of 16 means, for example, that a researcher has published 16 papers that each had at least 16 citations. Therefore, the h-index reflects both the number of articles as well as the number of citations per article.
• Google scholar • Web of science • Harzing’s Publish or Perish