Factor Analysis

Jaime Frade December 5, 2007

width 16cm university-logo3.png

Part I

9.4: Factor Rotation

1 Review:

1.1 Covariance Structure for the Orthogonal Factor Model

1. Cov(X) = LL^{t}+

Var(X_{i}) = l_{i1}^{2} + ⋯ + l_{im}^{2} + Ψ

(1)

Cov(X_{i}) = l_{i1}l_{k1} + ⋯ + l_{im}l_{km} + Ψ

2. Cov(X, F) = L or

Cov(X_{i}, F_{i}) = l_{ij}

It has been shown (9-8) pg 487, that all factor loadings obtained from the initial loadings by an orthogonal transformation have the same ability to reproduce the covariance (or correlation) matrix. An orthogonal transformation is a rotation or reflection of the coordinate axis.

Factor analysis is a generic term for a family of statistical techniques concerned with the reduction of a set of observable variables in terms of a small number of latent factors. It has been developed primarily for analyzing relationships among a number of measurable entities (such as survey items or test scores). The underlying assumption of factor analysis is that there exists a number of unobserved latent variables (or ’factors’) that account for the correlations among observed variables, such that if the latent variables are partialled out or held constant, the partial correlations among observed variables all become zero. In other words, the latent factors determine the values of the observed variables.

The primary purpose of factor analysis is data reduction and summarization

2 Definition:

Factor Rotation: An orthogonal transformation of the \underline{factor loadings}, as well as the implied orthogonal transformation of the \underline{factors}. Let T_{m×m} be an orthogonal matrix, (T T^{t} = T^{t}T = I_{m×m}).

If L̂ is the p × m matrix of estimated factor loading obtained by any method (PCP, ML, etc.) then

L̂^{∗} = L̂T, where T T^{t} = T^{t}T = I  (2)

is the p × m matrix of ’rotated’ loadings. L̂^{∗} is an orthogonal transformation of the factor loading matrix. The estimated covariance (or correlation) matrix remains unchanged.

L̂L̂^{t} + Ψ̂ = L̂T T^{t}L̂^{t} + Ψ̂
= L̂^{∗}(L̂^{∗})^{t} + Ψ̂  (3)

indicates that the residual matrix remains unchanged.

Also, the estimated specific variances and communalities, Ψ̂_{i}, and hence the ĥ_{i}^{2}, are unchanged by orthogonal transformations of L̂.

Therefore, can still use either L̂ or L̂^{∗}.

The examples illustrate graphical and analytical methods for determining an orthogonal rotation to a simple structure.

3 Examples

3.1 Example 9.8 (First look at factor rotation.) Oblique transformation.

width 7cm factor1.png Presented with the sample correlation matrix of examination scores in p = 6 subject areas for n = 220

male students. The correlation matrix is

R = [ \begin{matrix}1.0 & 0.439 & 0.410 & 0.288 & 0.329 & 0.248 \\ & 1.0 & 0.351 & 0.354 & 0.320 & 0.329 \\ & & 1.0 & 0.164 & 0.190 & 0.181 \\ & & & 1.0 & 0.595 & 0.470 \\ & & & & 1.0 & 0.464 \\ & & & & & 1.0 \end{matrix} ]  (4)

Using a Maximum likelihood solution for m = 2 factors yields the

RESULTS

http://www.stat.fsu.edu/~jfrade/PAPERS/Statistics/example9_8.htm

CODE

http://www.stat.fsu.edu/~jfrade/PAPERS/Statistics/example9_8_pg505.sas

4 Varimax Criterion

Define the ’scaled’ loading, l̃_{ij}^{∗}, of the i-th variable on the j-th factor \underline{after} rotation. Rotated coffecients scaled by the square root of the communalities.

l̃_{ij}^{∗} = \frac{l̂_{ij}^{∗}}{ĥ_{i}}

Then the varimax procedure selects the orthogonal transformation that maximizes

V = \frac{1}{p} ∑_{j=1}^{m} [∑_{j=1}^{p}(l̃_{ij}^{∗})^{4} − \frac{1}{p} (∑_{i=1}^{p} \binom{}{l_{ij}^{∗}}^{2})^{2}]  (5)

Interpretation of (5),

V ∝ ∑_{j=1}^{m} (Variance of squares of \underline{scaled} loading for the j-th factor)  (6)

Notes: Scaling gives the variables with smaller communalities more influence. Overall, maximizing V (5) corresponds to ’spreading out’ the squares of the loadings on each factor as much as possible. Therefore, we hope to find groups of large and negligible coefficients in any column of the rotated ladings matrix, l̃_{ij}^{∗}.

4.1 Example 9.9(2) (Rotated loadings for stock-price data.)

Returning to the data discussed in example 9.3 pg 493. The original factor loadings (obtained by PCP), the communalities, and the (varimax) rotated factor loadings are shown in Table below. SAS code given as well.

 http://www.stat.fsu.edu/~jfrade/PAPERS/Statistics/example9-10.htm

It is clear that the bank stocks (JP Morgan, Citibank, and Wells Fargo) load highly on the first factor, while the oil stocks, (Shell and Mobil) load highly on the second factor. Factor 1 may indicate those unique factors that case bank stocks to move together and factor 2 represents conditions affecting oil stocks.

 http://www.stat.fsu.edu/~jfrade/PAPERS/Statistics/example9_10_complete.htm
 http://www.stat.fsu.edu/~jfrade/PAPERS/Statistics/example9_10_pg510.sas

The factor loadings for the variables are pictured below with respect to the original and (varimax) rotated factor axes in below figure. After rotation each of the p variables should have a high loading on only one factor, however, this is not always possible.

The general market factor was destroyed by rotation. Analyze the relationship to pattern of correlations.

Part II

9.5: Factor Scores

Factor analysis centers the focus on the parameters in the factor model, however the estimated values of the common factors, factor scores, may also be of interest. These quantities are often used for diagonstic purposes, as well as inputs to a subsequent analysis.

Factor scores are estimates of values for the unobserved random factor vectors, F_{j}, j = 1, 2, ldots, n.

f̂_{j} = estimate of the values f_{j} attained by F_{j} (j th case)

However, estimation is complicated by the fact that unobserved quantities, f_{j} and ϵ_{j}, outnumber the observed x_{j}. The following two methods, Weighted Least Squares and Regression Methods, describe ways to overcome this complication.

5 Estimation of Factor Scores for both models

Suppose that the mean vector, μ, the loading factors, L, and the specific variance Ψ are known for the factor model,

(X_{j} − μ)_{(p×1)} = L_{(p×m}F_{(m×1)} + ϵ_{(p×1}  (7)

regard the specific factors ϵ′ = [ϵ_{1}, ϵ_{2}, …, ϵ_{p}] as errors. Since the Var(ϵ_{i}) = Ψ_{i}, i = 1, 2, …, p need not be equal, thus

Var(ϵ_{i}) = Ψ = [ \begin{matrix}Ψ_{1} & 0 & 0 & … & … \\ 0 & Ψ_{2} & 0 & … & … \\ 0 & 0 & Ψ_{3} & … & … \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & 0 & … & Ψ_{p} \end{matrix} ]  (8)

will use the weighted least squares to estimate the common factor values.

6 Weighted Least Squares Method

WLS estimation of f̂_{j} of f_{j} is

f_{j} = (L′Ψ^{−1}L)^{−1} L′Ψ^{−1} (x − μ)  (9)

Using (9), take estimates L̂, Ψ̂, &μ̂ = x̄ as true values to obtain the factor scores for the j -th case:

f̂_{j} = (L̂′Ψ̂^{−1}L̂)^{−1} L̂′Ψ̂^{−1} (x_{j} − x̄)  (10)

In (10), the estimates L̂, Ψ̂ are obtained by the maximum likelihood method and must satisfy the uniqueness condition, L̂′Ψ̂^{−1}L̂ = Δ̂, a diagonal matrix.

Note: Ordinary (unweighted) least squares estimation is sometimes used when factor loadings are obtained from the principal component method, since specific variances tend to be more nearly equal. (i.e. Ψ̂_{1} = ⋯ = Ψ̂_{p})

The factor scores are then,

f̂_{j} = (L̃′L̃)^{−1} L̃′ (x_{j} − x̄)  (11)
= (L̃_{z}′L̃_{z})^{−1} L̃_{z}′z_{j}  (12)

where, (12), is for the standardized data. Since, (see (9-15) pg. 490

L̃ = [ \sqrt{λ̂_{1}}ê_{1} ; \sqrt{λ̂_{2}}ê_{2} ; ⋮ ; \sqrt{λ̂_{m}}ê_{m} ]  (13)

Using (13), into (11) will obtain

f̂_{j} = [ \frac{1}{\sqrt{λ̂_{1}}} ê′_{1} (x_{j} − x̄) \frac{1}{\sqrt{λ̂_{2}}} ê′_{2} (x_{j} − x̄) ⋮ \frac{1}{\sqrt{λ̂_{m}}} ê′_{m} (x_{j} − x̄) ]  (14)

Comparing, from princpal component analysis, (see (8-21), pg 443), can obtain the ith principal component ŷ_{ji} = ê′_{i}(x − x̄) for any observation vector, x.

Comparing (14) with above (8-21), f̂_{j} is the first m (scaled) principal components evaluated at x_{j}.

7 Regression Method

Starting from original factor model in (7), treat the loadings matrix, L and Ψ as known values.

Consider the joint distribution of (x_{j} − μ) and F_{j}. Assume the multivariate normality as in the maximum likelihood approach to factor analysis.

[ \begin{matrix}x_{j} − μ \\ F_{j} \end{matrix} ] ∼ N_{p+m} (0, ∑^{∗})  (15)

7.1 Example 9.12 (Computing Factor Scores by WLS and Regression.)

Using the MLE solution from R for stock data, the estimated rotated loadings and specific variances, the following equations should hold, where the standardized observations, z′

f̂ = (L̂_{z}^{∗}′Ψ̂_{z}^{−1}L̂_{z}^{∗})^{−1} L̂_{z}^{∗}′Ψ̂_{z}^{−1}z ≈ L̂_{z}^{∗}R^{−1}z  (16)

Part III

9.6: Perspectives and a Strategy for Factor Analysis

8 Suggestible steps for factor analysis

1. Perform PCP factor analysis a. Look for observations that are suspicious behavior b. Try a varimax rotation

2. Perform a maximum likelihood factor analysis, including a varimax rotation

3. Compare the solutions obtained from the two factor analysis a. Do the loadings group in the same manner? b. Plot factor scores using PCP and MLE

4. Repeat the first three steps for other numbers of common factors m.

5. For large data sets, split them in half and perform a factor analysis on each part. width 10cm factor2.png

8.1 Example 9.14 (chicken-bone data)