EXAMINING THE RELATIONSHIP BETWEEN PERCEPTIONS. . .

 

Student Categorical Proposal (Step 1):

 

Data Sources: I will be drawing information on opinions about economic mobility from the General Social Survey (a large-scale demographic and attitude survey conducted at least biennially since 1972) and the Gini coefficient, a measure of income inequality, from the U.S. Census.

 

Questions of interest: Do people's impressions of their/others' economic mobility potential reflect the true economic mobility potential (as measured by income inequality)? Do these impressions change over time as income inequality changes over time? If not, are there factors that mediate the relationship (such as an individual's own income level, how different an individual's success is from his/her parents', etc.)?

 

Analysis: The data from the GSS will be multinomial, ordinal or continuous. The Gini coefficient will be continuous, and the two will be merged on time. I will likely be doing a series of multicategory logit models as are appropriate for my hypotheses once they are further nailed down.

 

Concerns: I believe I'm going to have to contend with missing data. Also, I'm currently trying to determine whether the GSS is a repeated measure of some of the same respondents over time as one of the variables has me confused. If they are repeated measures, then I'll have to look at the methods of analysis for non-independent data.

 

Professor Comments on Proposal:

 

You have identified a data set that will give you the potential to test many hypotheses and potentially use many different types of analyses. The difficulty with such data is frequently narrowing down the possibilities to a very specific set of testable hypotheses. It looks like you are already forming an interesting line of questioning, although refinement of your hypotheses and matching of specific methods will be important. Given the large range of possibilities, you might consider avoiding multilevel data if possible; once you find out more about the structure of the data, and whether people were assessed at multiple time points, please be sure to talk with Luke and me so that you can best plan your analyses if you do have non-independent observations. Dealing with multilevel data and missing data can both be large things to address. For the draft I would suggest that you aim at selecting analyses that will tackle one of these problems --- either address the missing data and avoid multilevel questions for now, or ask some multilevel questions but handle the missing data in a simple (but inadequate) way such as listwise deletion. Once you get one of these major halves addressed, you can work on the other for the final paper; but might be a lot unless you have previous experience with those topics. Luke can give you a mini introduction to multiple imputation or other approaches you might use to handle the missing data, if you decide to tackle that.

 

Retrospective Reflection:

 

This proposal highlighted the pieces I would hope to see in a proposal --- the data set, the questions of interest, and a beginning of a plan for analysis. Particularly given the size of the data set, I was concerned that the analysis plan was not narrower at this point, as the student had about three weeks to produce a paper draft. Given the goal was a draft, at this point I aimed to orientate the student more towards implementing her questions well, and tackling one of the two major methodological issues that were going to confront this paper (missing data & the multilevel structure of the data). The fact that these issues were listed as concerns in the proposal, however, only helped indicate that this student was already on track for an excellent paper. Even more impressive, this work was being put forth by one of the undergraduates completing a minor in Social and Behavioral Sciences Methodology.

 


Student Paper Draft Excerpts (Step 2):

 

            Discussions of economic mobility have a place in both popular and academic discourse, and in light of the recent economic downturn and social movements (e.g., the Occupy Wall Street Movement), such discussions have brought a particular salience to perceptions of economic mobility in the public consciousness. Perceptions of economic mobility can be broken into three categories: beliefs in meritocracy (that hard work and perseverance will bring economic rewards), beliefs in systemic factors (that nepotism and arbitrary factors will bring economic rewards) and beliefs in some combination of the two. In American society, beliefs in meritocracy are dominant and considered an integral part of the American Dream and American psyche.

[. . .]

Hypotheses

H1: Meritocratic beliefs increase as economic inequality increases.

System Justification Theory states that beliefs in meritocracy should increase as inequality increases if inequality is perceived as a threat to the societal status quo notion of the system as meritocratic.

H2: Beliefs in systemic factors increase as economic inequality decreases.

            As threat presented by economic inequality subsides, beliefs should be less reactionarily meritocratic and should come to somewhat more closely mirror the economic reality in America of limited hard work- and merit-induced mobility.

H3: Meritocratic beliefs are not affected by socioeconomic status.

            Previous research has indicated that people at all levels of the socioeconomic hierarchy endorse meritocratic beliefs; however, System Justification Theory suggests that this factor is unimportant in the face of a threat to the societal status quo.

 

Methods

[. . .]

Missing Data

[. . .]

 

Clustering

            Much of the data collected by the GSS are questions asked of the same people each year for several years. It is presumed that individuals have much in common with themselves and thus constitute a cluster. This clustering was addressed in the analysis using the mixed logistic model within the Zelig package for R[1] which allows for a random intercept in the analysis and for the imputations necessitated by the missing data.

 

Analyses

H1: Meritocratic beliefs increase as economic inequality increases.

            This hypothesis was operationalized as HARDWORK increases as GINI increases and was tested using the Zelig package with a "logit.mixed" model and a random intercept term to predict HARDWORK from GINI. Again, only one imputation was used so no collapsing was necessary. This will be corrected in my final draft.

 

H2: Beliefs in systemic factors increase as economic inequality decreases.

            This hypothesis was operationalized as LUCKHELP increases as GINI decreases and was tested in the same method as the first hypothesis, predicting LUCKHELP from GINI and a random intercept term.

 

H3: Meritocratic beliefs are not affected by socioeconomic status.

My final hypothesis was operationalized as HARDWORK not affected by SEI and was tested via a comparison of Akaike Information Criteria (AIC) from a model predicting HARDWORK from just GINI and a random intercept term and a model predicting HARDWORK from GINI, SEI and a random intercept term. A likelihood ratio test would have been preferred; however, by using the mixed effects logistic regression, not all of the necessary pieces of information are stored, and as a result, the test cannot be preformed.

 

Results

[. . .]

H3: Meritocratic beliefs are not affected by socioeconomic status.

            A model predicting HARDWORK from GINI and SEI revealed a random intercept of 1.0561 (Wald p=2.21e-10), a coefficient of –1.0903 for GINI (p=0.00394) and a coefficient of –0.0016 for SEI (p=0.00102). The model had an AIC of 62316, which is smaller than that of the model predicting HARDWORK from GINI described above, indicating that a model containing SEI fits better than a model without it, supporting the hypothesis. For each 1/100 unit increase in socioeconomic status, as measured by SEI, the odds of an individual endorsing meritocratic beliefs decrease by 0.16%, holding economic inequality constant. Holding socioeconomic status constant, for each 1/100 unit increase in economic inequality, the odds of an individual endorsing meritocratic beliefs decrease by 197.5%.

 

Discussion

[. . .]

Future study

            Previous research has indicated that demographic factors such as race, gender and education level have a significant influence on perceptions of meritocracy. A model containing these factors as well as socioeconomic status and economic inequality should be tested to observe how they interact. It would also be interesting to include measures of an individual's experience with unemployment and poverty in the model. Both the significance of a socioeconomic status term described above and previous literature indicate that such experiences may influence people to believe that systemic factors are more influential in economic success than hard work because individuals with such experiences are well aware of how hard they worked and where it did (or did not) get them economically. Finally, only the endorsement of meritocratic or systemic attributions for economic success in isolation were investigated here, but analysis of individuals who endorse both categories may further elucidate the relationships between a variety of factors.

 

Professor Comments on Proposal:

 

[. . .]

 

In this analysis you have done a good job of considering several methodological factors that need to be accounted for with these data --- including the issues with clustering, missing data, and how the magnitude of the data set could make very small effects significant. While consideration of these factors certainly complicates analysis, I am glad to see you taking them into account.

 

I have two primary concerns that I would like to see addressed. The first is to encourage you to be careful about your wording in the paper. At times I ran into phrases that were either unclear or incorrect. For example: "…was generated with a random intercept of 0.1241…." Is this the mean estimate? People fit models, not generate them. As another example: "Holding socioeconomic status constant, for each 1/100 unit increase in economic inequality, the odds of an individual endorsing meritocratic beliefs decrease by 197.5%." I'm not sure what to make of the odds decreasing by more than 100%. Did they go negative?

 

[. . .]

 

The following suggestions and questions are regarding less significant issues, but you also consider these as you write your final paper:

 

You found evidence contrary to your hypothesis, such that meritocratic beliefs decreased as economic inequality increased. Could this be due to not controlling for an important variable? Or is it possible that the relationship is such that it does not form an S-shaped curve as economic inequality increases, but that rather it follow a U-shaped or inversed U-shaped curve? On thing to consider is whether there anything that would be controlled for by system justification theory.

[. . .]

 

Retrospective Reflection:

 

In the end, the student decided to try to take on both of the significant methodological challenges (missing data and clustering) in analyzing these data, so from a technical standpoint this was moving towards an excellent paper. The weaker part of the paper was the writing style, which was less practiced in the presentation of results. As can be seen on the rubric, I do expect students to be able to state results accurately, correctly use statistical terminology, and try to express their ideas clearly. The weaknesses in this paper were understandable, however, as this was written by an undergraduate; naturally she has less practice writing research reports than even a student one or two years into graduates school. So for this student most of my comments were aimed at improving the discussion and presentation of the results, rather than the analysis.


Student Final Paper Excerpts --- Key Changes (Step 3):

 

[. . .]

Analyses

H1: Meritocratic beliefs (compared to systemic beliefs) increase as economic inequality increases.

This hypothesis was operationalized as GETAHEAD increases as GINI increases and was tested using the Zelig package with a "logit.mixed" model and a random intercept term to predict GETAHEAD from GINI. Twenty imputations were used, and the results were collapsed using Ruban's Rules as they were extended by Schafer and Olsen (1998).

 

H2: Meritocratic beliefs (compared to systemic beliefs) are not affected by socioeconomic status.

This hypothesis was operationalized as GETAHEAD not affected by SEI and was tested via a comparison of Akaike Information Criteria (AIC) from a model predicting GETAHEAD from just GINI and a random intercept term and a model predicting GETAHEAD from GINI, SEI and a random intercept term. The results of the model containing SEI were also collapsed across imputations using Ruban's Rules (Schafer & Olsen, 1998). The AICs were computed as an average of the AICs of the 20 models associated with the imputations. Which it is recognized that this may not produce the optimal estimate of AIC, it is the best that I can do with my base of theoretical knowledge.

 

Results

Hypothesis Testing

H1: Meritocratic beliefs (compared to systemic beliefs) increase as economic inequality increases.

            A model predicting GETAHEAD from GINI was fit with a random intercept of –0.090 with a variance of 0.004 (df=35, p=0.7706) and a coefficient of 4.30 for GINI. A t-test of whether this coefficient was significantly different than zero (df=19.7[2]) with a non-significant Wald p-value of 0.173. In this case, a Wald p-value was used because the estimates of standard error given by Ruban's Rules are more accurate than simply creating a pooled measure for a likelihood ratio test. Additionally, due to the large sample size, the Wald and likelihood ratio tests would yield a similar p-value. The model had an AIC of 27695.

These results indicate that the hypothesis is not supported and that neither meritocratic nor systemic beliefs are affected by economic inequality.

 

H2: Meritocratic beliefs (compared to systemic beliefs) are not affected by socioeconomic status.

            A model predicting GETAHEAD from GINI and SEI revealed a random intercept of -0.100 with a variance of 0.004 (df=36.6, p=0.792), a coefficient of 4.13 for GINI (df= 1036.5, p=2.09x10-10) and a coefficient of 0.002 for SEI (df=1453, p=0.526). The model had an AIC of 27691, which is smaller than that of the model predicting GETAHEAD from GINI described above (AIC=27695), indicating that a model containing SEI fits better than a model without it, even though the SEI term in itself is not significant, which does not support the hypothesis. For each one unit increase in socioeconomic status, as measured by SEI, the odds of an individual endorsing meritocratic beliefs over systemic beliefs are multiplied by 1.002, holding economic inequality constant. Holding socioeconomic status constant, for each 1/100 unit increase in economic inequality, the odds of an individual endorsing meritocratic beliefs over systemic beliefs are multiplied by 62.2.

 

Discussion

Conclusions

            These results seem to show a mixed picture of System Justification Theory. Consistent with the theory, individuals increasingly endorse meritocratic beliefs over systemic beliefs (luck and help) as economic inequality increases, but as is demonstrated by the non-significance of the first model, this effect is only evidenced when socioeconomic status is included in the model, though the term is not significant itself, indicating that there is a relationship between meritocratic vs. systemic beliefs and socioeconomic status. It is logical that there might be an interaction between the two factors, as individuals of different socioeconomic status levels tend to experience changes in economic inequality to differing degrees – those at the top are typically more insulated from negative economic changes than those at the bottom. However, in System Justification Theory, the threat posed by increasing economic inequality is supposed to be salient enough that other factors are unimportant. As this part of the theory is not holding true, other demographic factors may also mediate the relationship between meritocratic/systemic beliefs and economic inequality.

            The variances of the random effects in both models were found to be non-significant, indicating that individuals are enough unlike themselves in their responses from year to year that they do not constitute a cluster and multiple observations from a single individual can be treated as though they were from multiple individuals (e.g., as though they are independent), at least from a statistical analysis perspective. This would seem to indicate that, contrary to assertions in the literature, meritocratic and systemic beliefs are not stable personality characteristics and instead vary from year to year in response to a variety of changing factors (Ledgerwood et al., 2011).

 

[. . .]

 

Retrospective Reflection:

 

In the revised paper, multiple changes were made to the methods, results and discussion sections. The methods section provided more details about the analyses that were run. The result primarily added information about steps taken to further address the problem of missing data, although small changes in language helped with the presentation of the results. Finally, the discussion of the results was more extensive, and suggested the student was spending more time trying to think through the implications of the analysis and results. While at somewhat of a disadvantage compared to some of the more senior graduate students, some of whom had taken many more statistics courses, this paper was certainly above expectations.



[1] Zelig version 3.5.3, Delia Bailey and Ferdinand Alimadhi. 2007. "logit.mixed: Mixed effects logistic model" in Kosuke Imai, Gary King, and Olivia Lau, "Zelig: Everyone's Statistical Software," http://gking.harvard.edu/zelig

[2] Due to how Ruban's Rules dictate degrees of freedom are calculated, partial degrees are possible.