Factor Analysis.pdf

8/9/2019 Factor Analysis.pdf

1/57

Factor Analysis


2/57

Factor Analysis

• Factor analysis is a general name denoting a class of procedures primarily

used for data reduction and summarization.

• Factor analysis is an interdependence technique in that an entire set of

interdependent relationships is examined without making the distinction

between dependent and independent variables.

• Factor analysis is used in the following circumstances:

– To identify underlying dimensions, or factors, that explain the

correlations among a set of variables.

– To identify a new, smaller, set of uncorrelated variables to replace the

original set of correlated variables in subsequent multivariate analysis(regression or discriminant analysis).

– To identify a smaller set of salient variables from a larger set for use in

subsequent multivariate analysis.


3/57

Factor Analysis Model

Mathematically, each variable is expressed as a linear combinationof underlying factors. The covariation among the variables isdescribed in terms of a small number of common factors plus aunique factor for each variable. If the variables are standardized,the factor model may be represented as:

X i = Ai 1F 1 + Ai 2F 2 + Ai 3F 3 + . . . + AimF m + V i Ui

where

X i = i th standardized variable Aij = standardized multiple regression coefficient of

variable i on common factor j F = common factorV i = standardized regression coefficient of variable i on

unique factor i Ui = the unique factor for variable i m = number of common factors


4/57

The unique factors are uncorrelated with each other and with the common

factors. The common factors themselves can be expressed as linear

combinations of the observed variables.

Fi = Wi1X1 + Wi2X2 + Wi3X3 + . . . + WikXk

where

Fi = estimate of i th factor

Wi = weight or factor score coefficient

k = number of variables



5/57

• It is possible to select weights or factor score coefficients so

that the first factor explains the largest portion of the total

variance.

• Then a second set of weights can be selected, so that the

second factor accounts for most of the residual variance,

subject to being uncorrelated with the first factor.

• This same principle could be applied to selecting additional

weights for the additional factors.



6/57

Statistics Associated with Factor Analysis

• Bartlett's test of sphericity. This- test statistic used to examine

the hypothesis that the variables are uncorrelated in the

population.

– In other words, the population correlation matrix is an identity matrix;

each variable correlates perfectly with itself (r = 1) but has no

correlation with the other variables (r = 0).

• Correlation matrix. It is a lower triangle matrix showing the

simple correlations, r , between all possible pairs of variables

included in the analysis. The diagonal elements, which are all

1, are usually omitted.


7/57

• Communality. Communality is the amount of variance a variableshares with all the other variables being considered.

– This is also the proportion of variance explained by the common factors.

• Eigenvalue. The eigenvalue represents the total varianceexplained by each factor.

• Factor loadings. Factor loadings are simple correlations betweenthe variables and the factors.

• Factor loading plot. A factor loading plot is a plot of the originalvariables using the factor loadings as coordinates.

•

Factor matrix. A factor matrix contains the factor loadings of allthe variables on all the factors extracted.



8/57

• Factor scores. Factor scores are composite scores estimated for eachrespondent on the derived factors.

• Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is an index used to examinethe appropriateness of factor analysis. High values (between 0.5 and 1.0)indicate factor analysis is appropriate. Values below 0.5 imply that factor

analysis may not be appropriate.

• Percentage of variance. The percentage of the total variance attributed toeach factor.

• Residuals are the differences between the observed correlations, as given in

the input correlation matrix, and the reproduced correlations, as estimatedfrom the factor matrix.

• Scree plot. A scree plot is a plot of the Eigenvalues against the number offactors in order of extraction.



9/57

Conducting Factor Analysis

Construction of the Correlation Matrix

Method of Factor Analysis

Determination of Number of Factors

Determination of Model Fit

Problem formulation

Calculation of

Factor Scores

Interpretation of Factors

Rotation of Factors

Selection of

Surrogate Variables


10/57

Conducting Factor Analysis- Formulate the Problem

• The objectives of factor analysis should be identified.

• The variables to be included in the factor analysis should be

specified based on

– past research, theory, and judgment of the researcher.

– It is important that the variables be appropriately measured on an interval

or ratio scale.

• An appropriate sample size should be used.

– As a rough guideline, there should be at least four or five times as many

observations (sample size) as there are variables.

Note: if sample size is small you need to be cautious in interpreting the result


11/57

Example: benefits consumer speak from purchase of tooth paste

A multi item scale – Likert scale

Items S

D

A

S

A

It is important to buy a tooth paste that prevents cavity 1 2 3 4 5 6 7

I like a tooth paste that gives shiny teeth 1 2 3 4 5 6 7

A toothpaste should strengthen my gums 1 2 3 4 5 6 7

I prefer the tooth paste that freshens breath 1 2 3 4 5 6 7

Prevention of tooth decay is not an important benefit offered by

the tooth paste

1 2 3 4 5 6 7

The most important consideration in buying a tooth paste is

attractive teeth

1 2 3 4 5 6 7

Express your degree of agreement


12/57

Conducting Factor Analysis-DataRESPONDENT

NUMBER V1 V2 V3

V4 V5 V6

1 7.00 3.00 6.00 4.00 2.00 4.00

2 1.00 3.00 2.00 4.00 5.00 4.003 6.00 2.00 7.00 4.00 1.00 3.00

4 4.00 5.00 4.00 6.00 2.00 5.00

5 1.00 2.00 2.00 3.00 6.00 2.00

6 6.00 3.00 6.00 4.00 2.00 4.00

7 5.00 3.00 6.00 3.00 4.00 3.00

8 6.00 4.00 7.00 4.00 1.00 4.00

9 3.00 4.00 2.00 3.00 6.00 3.00

10 2.00 6.00 2.00 6.00 7.00 6.00

11 6.00 4.00 7.00 3.00 2.00 3.00

12 2.00 3.00 1.00 4.00 5.00 4.00

13 7.00 2.00 6.00 4.00 1.00 3.00

14 4.00 6.00 4.00 5.00 3.00 6.00

15 1.00 3.00 2.00 2.00 6.00 4.00

16 6.00 4.00 6.00 3.00 3.00 4.00

17 5.00 3.00 6.00 3.00 3.00 4.00

18 7.00 3.00 7.00 4.00 1.00 4.00

19 2.00 4.00 3.00 3.00 6.00 3.00

20 3.00 5.00 3.00 6.00 4.00 6.00

21 1.00 3.00 2.00 3.00 5.00 3.00

22 5.00 4.00 5.00 4.00 2.00 4.00

23 2.00 2.00 1.00 5.00 4.00 4.00

24 4.00 6.00 4.00 6.00 4.00 7.00

25 6.00 5.00 4.00 2.00 1.00 4.00

26 3.00 5.00 4.00 6.00 4.00 7.00

27 4.00 4.00 7.00 2.00 2.00 5.00

28 3.00 7.00 2.00 6.00 4.00 3.00

29 4.00 6.00 3.00 7.00 2.00 7.00

30 2.00 3.00 2.00 4.00 7.00 2.00


13/57

•

Using the data given in the slide 12 followinganalysis is presented in the following slides

(You can also try using the same data)


14/57


15/57


Construct the Correlation Matrix

• Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy.

– It compares magnitude of observed correlation coefficients to themagnitude of the partial correlation coefficient

– Value greater than 0.5 is desirable

– Note: Small values of the KMO statistic indicate that the correlationsbetween pairs of variables cannot be explained by other variables and thatfactor analysis may not be appropriate.


16/57

Correlation Matrix

Variables V1 V2 V3 V4 V5 V6

V1 1.000 0.530 0.873 -0.086 0.858 0.004

V2 -0.530 1.000 -0.155 -0.572 0.020 0.640 V3 0.873 -0.155 1.000 -0.248 --0.778 -0.018

V4 -0.086 0.572 -0.248 1.000 -0.007 0.640

V5 -0.858 0.020 -0.778 -0.007 1.000 0.136

V6 0.004 0.640 -0.018 0.640 -0.136 1.000


17/57

Correlation Matrix


V1 1.000

V2 -0.530 1.000

V3 0.873 -0.155 1.000

V4 -0.086 0.572 -0.248 1.000

V5 -0.858 0.020 -0.778 -0.007 1.000

V6 0.004 0.640 -0.018 0.640 -0.136 1.000


18/57


19/57

• In principal components analysis, the total variance in the data is

considered. – The diagonal of the correlation matrix consists of unities, and full variance

is brought into the factor matrix.

– PCA is recommended when the primary concern is to determine theminimum number of factors that will account for maximum variance in the

data for use in subsequent multivariate analysis. – The factors are called principal components.

• In common factor analysis, the factors are estimated based only on thecommon variance.

– Communalities are inserted in the diagonal of the correlation matrix.

– This method is appropriate when the primary concern is to identify the underlyingdimensions and the common variance is of interest.

– This method is also known as principal axis factoring.


Determine the Method of Factor Analysis


20/57

What is meant by “total variance” in the data set? To understand the meaning of

“total

variance” as it is used in a principal component analysis, remember that the observed

variables are standardized in the course of the analysis. This means that each variable istransformed so that it has a mean of zero and a variance of one.

The “total variance” in the data set is simply the sum of the variances of these

observed variables. Because they have been standardized to have a variance of one,

each observed variable contributes one unit of variance to the “total variance” in the

data set. Because of this, the total variance in a principal component analysis will

always be equal to the number of observed variables being analyzed.

For example:

if seven variables are being analyzed, the total variance will equal seven. The

components that are extracted in the analysis will partition this variance: perhaps the

first component will account for 3.2 units of total variance; perhaps the secondcomponent will account for 2.1 units. The analysis continues in this way until all of the

variance in the data set has been accounted for.


21/57

What is a communality?

A communality refers to the percent of variance in anobserved variable that is accounted for by the retained

components (or factors).

A given variable will display a large communality if it

loads heavily on at least one of the study’s retainedcomponents.

Although communalities are computed in both

procedures, the concept of variable communality is

more relevant in a factor analysis than in principal

component analysis.


22/57

Results of Principal Components Analysis

Communalities

Variables Initial Extraction

V1 1.000 0.926

V2 1.000 0.723

V3 1.000 0.894

V4 1.000 0.739

V5 1.000 0.878

V6 1.000 0.790

Initial Eigen values

Factor Eigen value % of variance Cumulat. %1 2.731 45.520 45.520

2 2.218 36.969 82.4883 0.442 7.360 89.8484 0.341 5.688 95.5365 0.183 3.044 98.5806 0.085 1.420 100.000

decreasing

6.0

(2.731/6)100=45.52

All are above 0.5


23/57

Results of Principal Components Analysis- how many factors to

extract?

Extraction Sums of Squared Loadings

Factor Eigen value % of variance Cumulat. %1 2.731 45.520 45.5202 2.218 36.969 82.488

Factor Matrix

Variables Factor 1 Factor 2 V1 0.928 0.253 V2 -0.301 0.795 V3 0.936 0.131 V4 -0.342 0.789 V5 -0.869 -0.351 V6 -0.177 0.871

Rotation Sums of Squared Loadings

Factor Eigenvalue % of variance Cumulat. %

1 2.688 44.802 44.802

2 2.261 37.687 82.488


24/57


Rotated Factor Matrix

Variables Factor 1 Factor 2

V1 0.962 -0.027

V2 -0.057 0.848

V3 0.934 -0.146

V4 -0.098 0.845

V5 -0.933 -0.084 V6 0.083 0.885

Factor Score Coefficient Matrix

Variables Factor 1 Factor 2

V1 0.358 0.011 V2 -0.001 0.375

V3 0.345 -0.043

V4 -0.017 0.377

V5 -0.350 -0.059

V6 0.052 0.395

How are the above factor scores

for each case calculated?

The answer is that an equation

is used where the dependent variable

is the predicted factor score

and the independent variables are

the observed variables.

We can check this but to do this we need

two more pieces of information the factorscore coefficient matrix and the

standardized scores for the

observed variables.


25/57

Factor Score Coefficient Matrix


V1 0.926 0.024 -0.029 0.031 0.038 -0.053 V2 -0.078 0.723 0.022 -0.158 0.038 -0.105

V3 0.902 -0.177 0.894 -0.031 0.081 0.033

V4 -0.117 0.730 -0.217 0.739 -0.027 -0.107

V5 -0.895 -0.018 -0.859 0.020 0.878 0.016

V6 0.057 0.746 -0.051 0.748 -0.152 0.790

The lower left triangle contains the reproduced correlation matrix;

the diagonal, the communalities; the upper right triangle, the residualsbetween the observed correlations and the reproduced correlations.



26/57

• A Priori Determination.

– Because of prior knowledge, the researcher may knows how many factorsto expect and thus can specify the number of factors to be extractedbeforehand. ( review of literature /judgment/ focus group )

• Determination Based on Eigenvalues.

– In this approach, only factors with Eigen values greater than 1.0 areretained.

• An Eigen value represents the amount of variance associated with the factor.

• Hence, only factors with a variance greater than 1.0 are included.

• Factors with variance less than 1.0 are no better than a single variable, since, due to

standardization, each variable has a variance of 1.0. – If the number of variables is less than 20, this approach will result in a

conservative number of factors.

Conducting Factor Analysis -Determine the Number of Factors

- how many factors to consider


27/57

• Determination Based on Scree Plot.

• It is a plot of the Eigenvalues against the number of factors inorder of extraction.

– Experimental evidence indicates that the point at which the scree beginsdenotes the true number of factors.

– Generally, the number of factors determined by a scree plot will be one ora few more than that determined by the Eigenvalue criterion.

• Determination Based on Percentage of Variance.

• In this approach the number of factors extracted is determined

so that the cumulative percentage of variance extracted by thefactors reaches a satisfactory level.

– It is recommended that the factors extracted should account for at least60% of the variance.


Determine the Number of Factors


28/57

Scree Plot

0.5

2 543 6

Component Number

0.0

2.0

3.0

E i g e n v a l u e

1.0

1.5

2.5

1


29/57

• Determination Based on Split-Half Reliability.

• The sample is split in half and factor analysis is performed oneach half.

• Only factors with high correspondence of factor loadings acrossthe two subsamples are retained.

• Determination Based on Significance Tests.

• It is possible to determine the statistical significance of theseparate Eigenvalues & retain only those factors that are

statistically significant. – A drawback is that with large samples (size greater than 200), many

factors are likely to be statistically significant, although from a practicalviewpoint many of these account for only a small proportion of the totalvariance.


Determine the Number of Factors


30/57

Out putImportant out put of the factor analysis is Factor Matrix, also called

as factor pattern matrix

it contains the coefficients used to express the standardized

variables in terms of the factors

these coefficients and factor loadings represents the

correlations between the factors and variables

Coefficients with a large absolute value indicates that the

factor & variable are closely related

coefficients of factor matrix is used to interpret the factor


31/57

• Although the initial / un-rotated factor matrix indicates the

relationship between the factors and individual variables, itseldom results in factors that can be interpreted

– because the factors are correlated with many variables.

– Therefore, through rotation the factor matrix is transformed into asimpler one that is easier to interpret.

Conducting Factor Analysis- Rotate Factors

Variable 1 2

1 X

2 X X

3 X

4 X X

5 X X

6 X

Variable 1 2

1 X

2 X

3 X

4 X

5 X

6 X

High loadings before rotation High loadings after rotation


32/57

• In rotating the factors, we would like each factor to havenonzero, or significant, loadings or coefficients for only some ofthe variables.

• Likewise, we would like each variable to have nonzero orsignificant loadings with only a few factors, if possible with onlyone.


Rotate Factors


33/57

1.The rotation is called orthogonal rotation if the axes are

maintained at right angles.• The most commonly used method for rotation is the varimax procedure.

• This is an orthogonal method of rotation that minimizes the number of variables with

high loadings on a factor, thereby enhancing the interpretability of the factors.

• Orthogonal rotation results in factors that are uncorrelated.

2. The rotation is called oblique rotation when the axes are not

maintained at right angles, and the factors are correlated.

Sometimes, allowing for correlations among factors can simplify the factor pattern matrix.

Oblique rotation should be used when factors in the population are likely to be strongly

correlated.

Conducting Factor Analysis - Rotate Factors


34/57

Orthogonal Factor Rotation


35/57

Oblique Factor Rotation


36/57

Conducting Factor Analysis - Rotate Factors

• Rotation achieves simplicity & enhances interpretability:

– Though the rotation does not affect the communalities &

percentage of total variance explained

• Loading of variable get restructured

• Variance explained by the individual factor is redistributed by

rotation

• Percentage of variance accounted for by each factor does not

change

• Variables do not correlates highly on many factors


37/57

• A factor can then be interpreted in terms of the variables that

load high on it.

• Another useful aid in interpretation is to plot the variables, using

the factor loadings as coordinates.

• Variables at the end of an axis are those that have high loadings

on only that factor, and hence describe the factor.

Conducting Factor Analysis-Interpret Factors


38/57

Factor Loading Plot

1.0

0.5

0.0

-0.5

-1.0

C o m p o n e

n t 2

Component 1

ComponentVariable 1 2

V1 0.962 -2.66E-02

V2 -5.72E-02 0.848V3 0.934 -0.146

V4 -9.83E-02 0.854

V5 -0.933 -8.40E-02

V6 8.337E-02 0.885

Component Plot in Rotated Space

1.0 0.5 0.0 -0.5 -1.0

V1

V3

V6

V2

V5

V4

Rotated Component Matrix


39/57

It is essential to calculate factor score for each respondent, if

researchers likes to consider composite variable for

multivariate analysis.

The factor scores for the I th factor may be estimated

as follows:

F i = W i1 X 1 + W i2 X 2 + W i3 X 3 + . . . + W ik X k


Calculate Factor Scores


40/57

• By examining the factor matrix, one could select for each factor

the variable with the highest loading on that factor.

That variable could then be used as a surrogate variable for the

associated factor.

•

However, the choice is not as easy if two or more variables havesimilarly high loadings.

– In such a case, the choice between these variables should be based on

theoretical and measurement considerations.

Conducting Factor Analysis- Select Surrogate Variables


41/57

• Determination of model fit is the final step in factor analysis• Assumption: observed correlation between variables can be

attributed to common factors

• The correlations between the variables can be deduced or

reproduced from the estimated correlations between thevariables and the factors.

• The differences between the observed correlations (as given in

the input correlation matrix) and the reproduced correlations

(as estimated from the factor matrix) can be examined todetermine model fit.

• These differences are called residuals.

Conducting Factor Analysis- Determine the Model Fit


42/57

SPSS Windows

To select this procedures using SPSS for Windows click:

Analyze>Data Reduction>Factor …


43/57

Factor Analysis Result – (Data -

Response to SPSS and computer )


44/57

Factor Analysis-result

• Result of un rotated factor analysis

– Data is - anxiety about SPSS


45/57


• Result of un rotated factor analysis

KMO and Bartlett's Test

Kaiser-Meyer-Olkin Measure of Sampling Adequacy. 0.9302

Bartlett's Test of Sphericity

Approx. Chi-

Square 19334

df 253

Sig. 0

Factor Analysis result un rotated


46/57

Factor Analysis-result- un rotatedTotal Variance Explained

Component Initial Eigenvalues Extraction Sums of Squared Loadings

Total

% of

Variance Cumulative % Total % of Variance Cumulative %

1 7.29 31.696 31.696 7.29 31.6958568 31.6958568

2 1.739 7.5601 39.256 1.739 7.560124986 39.25598178

3 1.317 5.725 44.981 1.317 5.725006643 44.98098843

4 1.227 5.3356 50.317 1.227 5.335644146 50.31663257

5 0.988 4.2951 54.612

6 0.895 3.8927 58.504

7 0.806 3.5024 62.007

8 0.783 3.4036 65.41

9 0.751 3.2651 68.676

10 0.717 3.1172 71.793

11 0.684 2.9721 74.765

12 0.67 2.9109 77.676

13 0.612 2.6609 80.337

14 0.578 2.5119 82.84915 0.549 2.3878 85.236

16 0.523 2.2746 87.511

17 0.508 2.2104 89.721

18 0.456 1.9823 91.704

19 0.424 1.8426 93.546

20 0.408 1.773 95.319

21 0.379 1.6499 96.96922 0.364 1.5827 98.552


47/57

Factor analysis output

Factor-1 Items Loadings

I have little experience of computers 0.80

All computers hate me 0.64

Computers are useful only for playing games 0.55

I worry that I will cause irreparable damage

because of my in-competenece with computers 0.65

Computers have minds of their own and

deliberately go wrong whenever I use them 0.58

Computers are out to get me 0.46

SPSS always crashes when I try to use it 0.68

Reliability Statistics

Cronbach's Alpha

.674


48/57


Statistics makes me cry 0.50

Standard deviations excite me 0.57

I dream that Pearson is attacking me withcorrelation coefficients 0.52

I don't understand statistics 0.43

People try to tell you that SPSS makes

statistics easier to understand but it doesn't 0.52I weep openly at the mention of central

tendency 0.51

I can't sleep for thoughts of eigen vectors 0.68

I wake up under my duvet thinking that I am

trapped under a normal distribution 0.66


Cronbach's Alpha

.605


49/57


50/57


My friends will think I'm stupid for not being able to copewith SPSS 0.54

My friends are better at statistics than me 0.65

Everybody looks at me when I use SPSS 0.43

My friends are better at SPSS than I am 0.65

If I'm good at statistics my friends will think I'm a nerd 0.59


Cronbach's Alpha

.570


51/57


• Result of rotated factor analysis

• Run the factor analysis with out considering the

variables having extraction value less than .4

Note: Items 5,10 15 and 19 are deleted and rotatedfactor analysis result is reported below

You can observe that variance explained improved ,and factor membership also changed


52/57

KMO and Bartlett's Test

Kaiser-Meyer-Olkin Measure of Sampling Adequacy. .918

Bartlett's Test of Sphericity Approx. Chi-Square 16263.271

df 171

Sig. .000


53/57

l


54/57

Factor analysis output


I have little experience of computers 0.80

All computers hate me 0.68

My friends are better at statistics than me 0.07

People try to tell you that SPSS makes statistics easier to

understand but it doesn't 0.53I worry that I will cause irreparable damage because of

my incompetenece with computers 0.68

Computers have minds of their own and deliberately go

wrong whenever I use them 0.62

SPSS always crashes when I try to use it 0.74


Cronbach's Alpha = .710

F 2 I L di


55/57


Statiscs makes me cry 0.44

Standard deviations excite me 0.58

I dream that Pearson is attacking me with correlation

coefficients 0.46

I weep openly at the mention of central tendency 0.51

I can't sleep for thoughts of eigen vectors 0.70

I wake up under my duvet thinking that I am trapped

under a normal distribtion 0.62




56/57


I have never been good at mathematics 0.85

I did badly at mathematics at school 0.76

I slip into a coma whenever I see an

equation 0.76




57/57


My friends will think I'm stupid for not being able to copewith SPSS 0.51

My friends are better at SPSS than I am 0.67

If I'm good at statistics my friends will think I'm a nerd 0.64



Note: Cronbach’s Alpha Is less than .6

Factor Analysis.pdf

Documents