Michael Lamport Commons Harvard Medical School Eva Yuja Li Dare Institute

Do The Ordinal Orders of Hierarchical Complexity Produce Significant Gaps

Between Stages and Are the Stages Equally Spaced?

Michael Lamport Commons Harvard Medical School

Eva Yuja Li Dare Institute

Presented at Piaget Society, Saturday, 9:00-10:30, June 2, 2012SY21 Symposium Session 21, Symposium: Theory And Application Generated By The Model Of Hierarchical Complexity Conference Room B, Sheraton Centre, Toronto, CA

2

Background

• Model of Hierarchical Complexity is applied to many domains to measure the complexity of tasks – Tasks are assigned integer numbers called Orders of

Hierarchical Complexity • As a measurement scale, its property is investigated in this

paper– Is the Order of Hierarchical Complexity an ordinal scale?

• Only an ordinal scale is meaningful– Are orders of hierarchical complexity linear and equally spaced?

• A linear and equally spaced scale would indicate that there is equal amount of difficulty to move from one stage to the next

3

Background

• Is the Order of Hierarchical Complexity an ordinal scale?– As shown in the Rasch Variable Map, the stages of performance of

items followed the same sequence of their orders of Hierarchical Complexity

– In addition, ordinality might show up as gaps been the stages of performance on those items

• Are orders of hierarchical complexity linear and equally spaced? – Equally spaced orders would indicate that moving from one stage to

the next is always the same difficulty– If the performance measured by Rasch Analysis on the items are

equally spaced, then the orders of Hierarchical Complexity would also be equally spaced

4

MethodInstrument• This study used the laundry instrument that was based on the Inhelder and

Piaget’s (1958) pendulum task. • 111 items ranging from Primary Order 7 to Systematic Order 11

Procedure• Instrument was presented in a survey online• The tasks were presented in a sequence from easy to hard• The items were coded as correct or incorrect with missing answers being

assumed incorrect• Data was analyzed using the Rasch Model

5

Method• Participants:

– 113 participants– 47 (41.6%) men and 66 (58.4%) women– Age 18 to 100 (M = 35.8, (S.D. = 16.1). – Education

• 35 high school graduates• 57 Bachelor’s degree holders• 8 master’s level degree holders and • 13 doctoral level degree holders• M = Bachelors degree

6

Results

• Rasch Analysis yielded two scales– Person Stage of Performance

• Stands for how well the person performs on the set of tasks • Based solely on whether or not a given order of hierarchical Complexity is

correctly carried out– Rasch Scaled Item Difficulty

• How difficult items were empirically• This is the focus of this study

7

Participants MAP OF RANKS <more>|<rare> 7 + | | # T| 6 + | ## | | 5 .## + .## |T .# | ### | 4 .###### + sy #### | S| sy sy ## | sy sy sy 3 ### + ## | sy sy sy sy sy # | sy sy sy sy sy sy sy (Systemtatic) .## |S sy sy sy sy sy sy sy sy 2 ### + sy sy sy sy ######## | ### | f1 f1 f1 f1 ### | f1 f1 f1 f1 f2 f2 f2 f2 f2 f2 f2 f2 (formal) 1 ## + f1 f1 f2 f2 # | M| ######## | 0 .##### +M ab ab ## | ab ab ab ab ab ab ab ab ab (abstract) # | ab ab ab ab ab ab .## | ab ab ab -1 . + | .####### | .# | cr cr cr cr cr -2 . + cr cr (concrete) . S|S cr cr cr cr . | cr .######### | -3 .#### + pr .## | pr pr pr pr .## | pr pr pr pr pr pr pr pr pr pr (primary) # | pr pr -4 # + pr # | | pr pr . |T -5 . T+ . | | | -6 . + <less>|<frequ>

Rasch Variable Map

• Rasch Scaled Item Difficulty was ranged from -4.56 (Primary 7) to 3.94 (Systematic 11)• The higher an item is on

the scale, the more difficult it is

• P-Primary 7, C - Concrete 8, A- Abstract 9, F - Formal 10, S - systematic 11

• Rasch Scaled Item Difficulty of items sequenced in the same order as their Orders of Hierarchical Complexity • No item was out of order

8

• Are there “gaps” between Orders of Hierarchical Complexity?– Gaps are defined as “jumps” in Rasch Scaled Item

Difficulty from one order to the next– Gaps may indicate that demands of tasks between

adjacent orders have significant difference– Every other order adds a level of coordination– The order demands are supposed to be

qualitatively different

• Hypothesis: “Gaps” are significantly larger than the difference of Rasch Scaled Item Difficulty between items within each order

9

Model to Test for Gaps

• Let i = the observation number, which goes from 1 to 102. • DRi = β + a7 I 7i + a8 I8i + a9 I9i + a10 I10i + a11 I11i + εi

where– DRi = the difference of Rasch Scaled Item Difficulty between item

I and item (i-1)– β = the average of Gaps– an = the difference between the average of Item Break at order n

and the average of Gap β – Ini = {1,0} {is, is not} a difference in Rasch scores for Hierarchical

order or group n– εi is a random variable fulfilling the Gauss Markov conditions

10

Result

• DR = 0.65500 - 0.57447 I7i - 0.58864 I8i -0.60553 I9i - 0.62237 I10i - 0.58397 I11i

• This equation shows that the average of Gaps was 0.655. The average item break at each stage was smaller than the average gap size as shown by the an being negative.

• There are 5 null hypotheses: an = 0, n = 7, 8, 9, 10, and 11• There are 5 alternative hypotheses: an ≠ 0, n = 7, 8, 9, 10, 11.

11

Result

• Five t-tests were conducted

– t7 (97) = -10.014, p < 2-16 ≈ 0.00000– t8 (97) = -9.667, p < 2-16 ≈ 0.00000– t9 (97) = -10.555, p < 2-16 ≈ 0.00000 – t10 (97) = -10.848, p < 2-16 ≈ 0.00000 – t11 (97) = -10.499, p <2-16 ≈ 0.00000

• All the null hypotheses were rejected • Average Item Breaks are significantly smaller than the average

Gaps• Therefore, we have shown that Gaps exist

12

Test for Linearity and Equal Spacing

• This section investigates whether the Order of Hierarchical Complexity was a linear and equally spaced scale

• Four models were used– Simple regression model – Lack of fit test – Model on the spacing between Rasch Scaled Item Difficulty– Perturb the linear Order of Hierarchical Complexity

13

Test for LinearitySimple linear regression

• Simple linear regression between Rasch Scaled Item Difficulty y and Order of Hierarchical Complexity x

• y = a + b*x• x = 7, 8, 9, 10, 11 - a linear scale• r = .983, r2 = .975• The result shows that Item Order

of Hierarchical Complexity predicts Rasch Scaled Item Difficulty with r of .983

14

Test for LinearitySimple linear regression

How to know whether a linear relationship is the best option to describe the data?– By comparing the variance explained by the linear

regression model to variance explained by another model. The model that explains more variance is better

• A Lack of Fit test compares the Linear Regression Model with the Separate Means Model

• H0: Linear Regression Model explains significantly less variance than the Separate Means Model

• H1: Linear Regression Model and the Separate Means Model explains equal amount of variance in the data

15

Test for LinearitySimple linear regression

• The lack of fit test shows that F(3) = 1.944, p = 0.128 • The separate means model does not explain significantly more

variance than the linear regression model• The null hypothesis is that the spacing is unequal is not

rejected • The result indicates that the linear relationship between the

Task Order of Hierarchical Complexity and the Rasch Scaled Item Difficulty is not rejected by this analysis

• The linearity assumption can still be held

16

Test for Equal Spacing

• Using a t-test, this analysis tests whether there are equal spacing between adjacent Orders of Hierarchical Complexity

• Spacing is defined as the increment from the average of Rasch Scaled Item Difficulties of a lower order to the average of Rasch Scaled Item Difficulty of the next higher order– There are four spacings as there are five Orders of Hierarchical

Complexity• A statistical Model is constructed to account for the

differences of Rasch Scaled Item Difficulty between items

17

Test for Equal Spacing

• RD = Rasch Scaled Item Difficulty = β7 + γ8 I8i + γ9 I9i + γ10 I10i + γ11 I11i + εi – RD = Rasch Scaled Item Difficulty;– Ini = {1, 0} when the item {is, is not} at the Order of Hierarchical

Complexity denoted by n. n = {7, 8, 9, 10, 11}; – β7 = is the average value of the Rasch Scaled Item Difficulty for

items in order 7– γ8 = the estimate of the difference between the average Rasch

Scaled Item Difficulty at order 8 score and average Rasch Scaled Item Difficulty at order 7 score

18

Test for Equal Spacing

• β7 + γ8 estimates the average Rasch Scaled Item Difficulty at order 8

• {β7 + γ9, β7 + γ10, β7 + γ11} estimates the average Rasch Scaled Item Difficulty at order 9, 10 and 11

• H01: The spacing between order 9 and 8 is the same as the spacing between order 8 and 7. Or γ9 - 2γ8 = 0.

• H02: The spacing between order 10 and 9 is the same as the spacing between order 9 and 8. Or, γ10 - 2 γ9 + γ8 = 0

• H03: The spacing between order 11 and 10 is the same as the spacing between order 10 and 9. Or, γ11 - 2γ10 + γ9 = 0

• One sample t-tests were used to test these hypotheses

19

Test for Equal Spacing

• The result shows that we cannot reject any of these null hypotheses:

• For H01, t(97) = 0.240, p = 0.595• For H02, t(97) = 0.0526, p = 0.479• For H03, t(97) = 0.7949, p = 0.214• Therefore, we cannot reject the null hypotheses that all the

spacing between the orders is the same. This result is consistent the result of lack of fit test, which cannot reject linearity of the Orders of Hierarchical Complexity.

20

Test for Equal SpacingPerturbation Test

• Tests above supported that Order of Hierarchical Complexity as linear and equally spaced scale

• However, it was due to the lack of evidence to reject null hypotheses, which does not prove the alternative hypotheses

• This section of the paper takes an alternative route– We add noise to the Orders of Hierarchical Complexity scale– We test how much noise added to the scale would reject the linearity

hypothesis– It will show the upper limit to the deviation from a linear scale

21

Perturbation TestProcedure

1. Take the Orders of Hierarchical Complexity {7, 8, 9, 10, 11}2. Randomly add or subtract 0.05 (randomly selected by computer)3. Run a linear regression of the Rasch Scaled Item Difficulty on the newly defined order scale, obtain r of the model4. Repeat step 2 three more times 5. Average four r’s, obtain the average r when OHC was perturbed with noise of 0.056. Repeat step 1-5 with noise level 0.1, 0.15, 0.2, … 0.45

– Stop at 0.45 because noise of 0.5 may subvert the sequence of OHC, thus violate ordinality of the scale

22

Perturbation TestResult

• This scatter plot is the size for perturbation versus predictability r

• It shows that as the scale deviates from the original linear scale, predictability decreases steadily

0.940.9450.95

0.9550.96

0.9650.97

0.9750.98

0.9850.99

0 0.1 0.2 0.3 0.4 0.5 0.6

R

Perturbation

23

Perturbation TestResult

• Using the Fisher r-to-z transformation, the significance of the the r found in the original linear regression model and the r's found in the new models when the Order of Hierarchical Complexity is perturbed were assessed

• It is found that perturbing the Order of Hierarchical Complexity by more than 0.25 produces a significant difference in the predictability of the scale

• When the noise was 0.25, the difference was significant at the 0.1 level, with z = 1.68, p = 0.093. When noise = 0.35, the difference was significant at 0.05 level, with z = 2.74, p = 0.006.

24

Summary

• Rasch Analysis showed that the Order of Hierarchical Complexity is an ordinal scale, where Orders predicted the relative difficulty of items

• Simple linear regression showed that Orders of Hierarchical Complexity predicted Rasch Scaled Item Difficulty with an r of

• lack of fit test showed that the linearity of the scale could not be rejected

• Testing for Equal Spacing showed that the equal spacing assumption could not be rejected

• Perturbing the scale by 0.25 led to a significant difference in the predictability of the scale

25

Discussion• The existence of gaps shows that the ordinal nature of the

scale is not just an assumption• MHC is an equally spaced ordinal scale

– It is not an interval scale because it does not have additively or any cancellation property

• Equal Spacing indicates that going from one order to the next produces equal difficulty between stages – This allows one to treat orders as real numbers, and not just indication

of relative position• It might mean the order of Hierarchical Complexity, n, is a

measure the quantity of hierarchical information – The minimum number of order n task may be 2n

• Given that tasks at order n + 1 are defined by and coordinate 2 or more tasks at order n