EXAMINING THE RELATIONSHIP BETWEEN PERCEPTIONS.
. .

__Student Categorical
Proposal (Step 1):__

Data
Sources: I will be drawing information on opinions about economic mobility from
the General Social Survey (a large-scale demographic and attitude survey
conducted at least biennially since 1972) and the Gini
coefficient, a measure of income inequality, from the U.S. Census.

Questions
of interest: Do people's impressions of their/others' economic mobility
potential reflect the true economic mobility potential (as measured by income
inequality)? Do these impressions change over time as income inequality changes
over time? If not, are there factors that mediate the relationship (such as an
individual's own income level, how different an individual's success is from
his/her parents', etc.)?

Analysis:
The data from the GSS will be multinomial, ordinal or continuous. The Gini coefficient will be continuous, and the two will be
merged on time. I will likely be doing a series of multicategory
logit models as are appropriate for my hypotheses
once they are further nailed down.

Concerns:
I believe I'm going to have to contend with missing data. Also, I'm currently
trying to determine whether the GSS is a repeated measure of some of the same
respondents over time as one of the variables has me confused. If they are
repeated measures, then I'll have to look at the methods of analysis for
non-independent data.

__Professor Comments on
Proposal: __

You
have identified a data set that will give you the potential to test many
hypotheses and potentially use many different types of analyses. The difficulty
with such data is frequently narrowing down the possibilities to a very
specific set of testable hypotheses. It looks like you are already forming an
interesting line of questioning, although refinement of your hypotheses and matching
of specific methods will be important. Given the large range of possibilities,
you might consider avoiding multilevel data if possible; once you find out more
about the structure of the data, and whether people were assessed at multiple
time points, please be sure to talk with Luke and me so that you can best plan
your analyses if you do have non-independent observations. Dealing with
multilevel data and missing data can both be large things to address. For the
draft I would suggest that you aim at selecting analyses that will tackle one
of these problems --- either address the missing data and avoid multilevel
questions for now, or ask some multilevel questions but handle the missing data
in a simple (but inadequate) way such as listwise
deletion. Once you get one of these major halves addressed, you can work on the
other for the final paper; but might be a lot unless you have previous
experience with those topics. Luke can give you a mini introduction to multiple
imputation or other approaches you might use to handle the missing data, if you
decide to tackle that.

__Retrospective
Reflection:__

This proposal highlighted the pieces I would
hope to see in a proposal --- the data set, the questions of interest, and a
beginning of a plan for analysis. Particularly given the size of the data set,
I was concerned that the analysis plan was not narrower at this point, as the
student had about three weeks to produce a paper draft. Given the goal was a
draft, at this point I aimed to orientate the student more towards implementing
her questions well, and tackling one of the two major methodological issues
that were going to confront this paper (missing data & the multilevel
structure of the data). The fact that these issues were listed as concerns in
the proposal, however, only helped indicate that this student was already on
track for an excellent paper. Even more impressive, this work was being put
forth by one of the undergraduates completing a minor in Social and Behavioral
Sciences Methodology.

__Student Paper Draft
Excerpts (Step 2):__

Discussions
of economic mobility have a place in both popular and academic discourse, and
in light of the recent economic downturn and social movements (e.g., the Occupy
Wall Street Movement), such discussions have brought a particular salience to
perceptions of economic mobility in the public consciousness. Perceptions of
economic mobility can be broken into three categories: beliefs in meritocracy
(that hard work and perseverance will bring economic rewards), beliefs in
systemic factors (that nepotism and arbitrary factors will bring economic
rewards) and beliefs in some combination of the two. In American society,
beliefs in meritocracy are dominant and considered an integral part of the
American Dream and American psyche.

[. . .]

H1: Meritocratic beliefs increase as economic
inequality increases.

System Justification
Theory states that beliefs in meritocracy should increase as inequality
increases if inequality is perceived as a threat to the societal status quo
notion of the system as meritocratic.

H2: Beliefs in systemic
factors increase as economic inequality decreases.

As
threat presented by economic inequality subsides, beliefs should be less reactionarily meritocratic and should come to somewhat more
closely mirror the economic reality in America of limited hard work- and
merit-induced mobility.

H3: Meritocratic beliefs
are not affected by socioeconomic status.

Previous
research has indicated that people at all levels of the socioeconomic hierarchy
endorse meritocratic beliefs; however, System Justification Theory suggests
that this factor is unimportant in the face of a threat to the societal status
quo.

__Methods__

[. . .]

Missing
Data

[. . .]

Clustering

Much
of the data collected by the GSS are questions asked of the same people each
year for several years. It is presumed that individuals have much in common
with themselves and thus constitute a cluster. This
clustering was addressed in the analysis using the mixed logistic model within
the Zelig package for R[1] which allows for a random intercept in the analysis and for
the imputations necessitated by the missing data.

Analyses

H1:
Meritocratic beliefs increase as economic inequality increases.

This
hypothesis was operationalized as HARDWORK increases as GINI increases and was
tested using the Zelig package with a "logit.mixed" model and a random intercept term to
predict HARDWORK from GINI. Again, only one imputation was used so no
collapsing was necessary. This will be corrected in my final draft.

H2:
Beliefs in systemic factors increase as economic inequality decreases.

This
hypothesis was operationalized as LUCKHELP increases as GINI decreases and was
tested in the same method as the first hypothesis, predicting LUCKHELP from
GINI and a random intercept term.

H3:
Meritocratic beliefs are not affected by socioeconomic status.

My final hypothesis was operationalized as
HARDWORK not affected by SEI and was tested via a comparison of Akaike Information Criteria (AIC) from a model predicting
HARDWORK from just GINI and a random intercept term and a model predicting
HARDWORK from GINI, SEI and a random intercept term. A likelihood ratio test
would have been preferred; however, by using the mixed effects logistic
regression, not all of the necessary pieces of information are stored, and as a
result, the test cannot be preformed.

[. . .]

H3:
Meritocratic beliefs are not affected by socioeconomic status.

A
model predicting HARDWORK from GINI and SEI revealed a random intercept of
1.0561 (Wald p=2.21e-10), a coefficient of –1.0903 for GINI (p=0.00394)
and a coefficient of –0.0016 for SEI (p=0.00102). The model had an AIC of
62316, which is smaller than that of the model predicting HARDWORK from GINI
described above, indicating that a model containing SEI fits better than a
model without it, supporting the hypothesis. For each 1/100 unit increase in
socioeconomic status, as measured by SEI, the odds of an individual endorsing
meritocratic beliefs decrease by 0.16%, holding economic inequality constant.
Holding socioeconomic status constant, for each 1/100 unit
increase in economic inequality, the odds of an individual endorsing
meritocratic beliefs decrease by 197.5%.

[. . .]

Future
study

Previous
research has indicated that demographic factors such as race, gender and
education level have a significant influence on perceptions of meritocracy. A
model containing these factors as well as socioeconomic status and economic
inequality should be tested to observe how they interact. It would also be
interesting to include measures of an individual's experience with unemployment
and poverty in the model. Both the significance of a socioeconomic status term
described above and previous literature indicate that such experiences may
influence people to believe that systemic factors are more influential in
economic success than hard work because individuals with such experiences are
well aware of how hard they worked and where it did (or did not) get them
economically. Finally, only the endorsement of meritocratic or systemic attributions
for economic success in isolation were investigated here, but analysis of
individuals who endorse both categories may further elucidate the relationships
between a variety of factors.

__Professor Comments on Proposal: __

[. . .]

In this analysis you have done a good job of
considering several methodological factors that need to be accounted for with
these data --- including the issues with clustering, missing data, and how the
magnitude of the data set could make very small effects significant. While consideration
of these factors certainly complicates analysis, I am glad to see you taking
them into account.

I have two primary concerns that I would like to see
addressed. The first is to encourage you to be careful about your wording in
the paper. At times I ran into phrases that were either unclear or incorrect.
For example: "Éwas
generated with a random intercept of 0.1241É." Is this the mean estimate?
People fit models, not generate them. As another example: "Holding
socioeconomic status constant, for each 1/100 unit increase in economic
inequality, the odds of an individual endorsing meritocratic beliefs decrease
by 197.5%." I'm not sure what to make of the odds decreasing by more than
100%. Did they go negative?

[. . .]

The following suggestions and questions are
regarding less significant issues, but you also consider these as you write
your final paper:

You
found evidence contrary to your hypothesis, such that meritocratic beliefs
decreased as economic inequality increased. Could this be due to not
controlling for an important variable? Or is it possible that the relationship
is such that it does not form an S-shaped curve as economic inequality
increases, but that rather it follow a U-shaped or inversed U-shaped curve? On
thing to consider is whether there anything that would
be controlled for by system justification theory.

[. . .]

__Retrospective
Reflection:__

In the end, the student decided to try to take
on both of the significant methodological challenges (missing data and
clustering) in analyzing these data, so from a technical standpoint this was
moving towards an excellent paper. The weaker part of the paper was the writing
style, which was less practiced in the presentation of results. As can be seen
on the rubric, I do expect students to be able to state results accurately,
correctly use statistical terminology, and try to express their ideas clearly.
The weaknesses in this paper were understandable, however, as this was written
by an undergraduate; naturally she has less practice writing research reports
than even a student one or two years into graduates school. So for this student
most of my comments were aimed at improving the discussion and presentation of
the results, rather than the analysis.

__Student Final Paper
Excerpts --- Key Changes (Step 3):__

[. . .]

Analyses

H1:
Meritocratic beliefs (compared to systemic beliefs) increase as economic
inequality increases.

This hypothesis was operationalized as GETAHEAD
increases as GINI increases and was tested using the Zelig
package with a "logit.mixed" model and a
random intercept term to predict GETAHEAD from GINI. Twenty imputations were
used, and the results were collapsed using Ruban's
Rules as they were extended by Schafer and Olsen
(1998).

H2:
Meritocratic beliefs (compared to systemic beliefs) are not affected by
socioeconomic status.

This hypothesis was operationalized as GETAHEAD
not affected by SEI and was tested via a comparison of Akaike
Information Criteria (AIC) from a model predicting GETAHEAD from just GINI and
a random intercept term and a model predicting GETAHEAD from GINI, SEI and a
random intercept term. The results of the model containing SEI were also
collapsed across imputations using Ruban's Rules
(Schafer & Olsen, 1998). The AICs were computed as an average of the AICs
of the 20 models associated with the imputations. Which it is recognized that
this may not produce the optimal estimate of AIC, it is the best that I can do
with my base of theoretical knowledge.

Hypothesis
Testing

H1:
Meritocratic beliefs (compared to systemic beliefs) increase as economic
inequality increases.

A
model predicting GETAHEAD from GINI was fit with a random intercept of
–0.090 with a variance of 0.004 (df=35,
p=0.7706) and a coefficient of 4.30 for GINI. A t-test of whether this coefficient
was significantly different than zero (df=19.7[2])
with a non-significant Wald p-value of 0.173. In this case, a Wald p-value was
used because the estimates of standard error given by Ruban's
Rules are more accurate than simply creating a pooled measure for a likelihood
ratio test. Additionally, due to the large sample size, the Wald and likelihood
ratio tests would yield a similar p-value. The model had an AIC of 27695.

These results indicate that the hypothesis is
not supported and that neither meritocratic nor systemic beliefs are affected
by economic inequality.

H2:
Meritocratic beliefs (compared to systemic beliefs) are not affected by
socioeconomic status.

A
model predicting GETAHEAD from GINI and SEI revealed a random intercept of -0.100
with a variance of 0.004 (df=36.6, p=0.792), a
coefficient of 4.13 for GINI (df= 1036.5, p=2.09x10^{-10})
and a coefficient of 0.002 for SEI (df=1453,
p=0.526). The model had an AIC of 27691, which is smaller than that of the
model predicting GETAHEAD from GINI described above (AIC=27695), indicating
that a model containing SEI fits better than a model without it, even though
the SEI term in itself is not significant, which does not support the
hypothesis. For each one unit increase in socioeconomic status, as measured by
SEI, the odds of an individual endorsing meritocratic beliefs over systemic
beliefs are multiplied by 1.002, holding economic inequality constant. Holding
socioeconomic status constant, for each 1/100 unit
increase in economic inequality, the odds of an individual endorsing
meritocratic beliefs over systemic beliefs are multiplied by 62.2.

Conclusions

These
results seem to show a mixed picture of System Justification Theory. Consistent
with the theory, individuals increasingly endorse meritocratic beliefs over
systemic beliefs (luck and help) as economic inequality increases, but as is
demonstrated by the non-significance of the first model, this effect is only
evidenced when socioeconomic status is included in the model, though the term
is not significant itself, indicating that there is a relationship between
meritocratic vs. systemic beliefs and socioeconomic status. It is logical that
there might be an interaction between the two factors, as individuals of
different socioeconomic status levels tend to experience changes in economic
inequality to differing degrees – those at the top are typically more
insulated from negative economic changes than those at the bottom. However, in
System Justification Theory, the threat posed by increasing economic inequality
is supposed to be salient enough that other factors are unimportant. As this
part of the theory is not holding true, other demographic factors may also
mediate the relationship between meritocratic/systemic beliefs and economic inequality.

The
variances of the random effects in both models were found to be
non-significant, indicating that individuals are enough unlike themselves in
their responses from year to year that they do not constitute a cluster and
multiple observations from a single individual can be treated as though they
were from multiple individuals (e.g., as though they are independent), at least
from a statistical analysis perspective. This would seem to indicate that,
contrary to assertions in the literature, meritocratic and systemic beliefs are
not stable personality characteristics and instead vary from year to year in
response to a variety of changing factors (Ledgerwood
et al., 2011).

[. . .]

__Retrospective Reflection:__

In the revised paper, multiple changes were made
to the methods, results and discussion sections. The methods section provided
more details about the analyses that were run. The result primarily added
information about steps taken to further address the problem of missing data,
although small changes in language helped with the presentation of the results.
Finally, the discussion of the results was more extensive, and suggested the
student was spending more time trying to think through the implications of the
analysis and results. While at somewhat of a disadvantage compared to some of
the more senior graduate students, some of whom had
taken many more statistics courses, this paper was certainly above
expectations.

[1] Zelig version 3.5.3, Delia Bailey and Ferdinand Alimadhi. 2007. "logit.mixed: Mixed effects logistic model" in Kosuke Imai, Gary King, and Olivia Lau, "Zelig: Everyone's Statistical Software," http://gking.harvard.edu/zelig

[2] Due to how Ruban's Rules dictate degrees of freedom are calculated, partial degrees are possible.