The two-year collaborative research project with Mission Measurement started with approximately 90 scales, filtered by the following criteria:
- Created using samples of adults aged 18 years and older.
- Produced after the year 2000.
- Has at least two measurable items within the scale.
NEFE, with assistance from a panel of seven financial literacy experts, narrowed down the 90 scales to 11 final scales, based on the presence of some validity testing, rigorous development, or those that are pervasive and commonly used. The project’s data originated from the 11 chosen scales and demographic questions, with the sample attempting to capture the diversity of the U.S. adult population to ensure sufficient counts of 12 demographic sub-groups to support the planned analyses.
Two methods were used in the assessment of these existing scales. Confirmatory factor analysis examined how well the hypothesized factor structure (i.e. the set of items for each scale or subscale and the factor loadings for each item) fits the new data collected. Scale reliability (assessed using Cronbach’s Alpha) was utilized to measure whether the scale could be expected to provide consistent measures over time. The results showed that, of the 11 scales, five exceeded all fit statistics and were labeled “favorable”; three were labeled “reasonable” because they exceeded most fit statistics; and three were labeled “problematic” because they failed to exceed the thresholds for several fit statistics.
Following those assessments, invariance was calculated through multigroup confirmatory factor analysis, which helped indicate whether the questions are interpreted similarly by sub-groups of the sample. Scales that perform well in the confirmatory factor analysis and reliability analyses may not provide valid and reliable measures of the construct for all demographic subgroups, especially those traditionally marginalized. Invariance testing helps us understand if a given measure is being interpreted in the same way across different cultural backgrounds and other types of demographics, which allows researchers to make more accurate comparisons across groups.
Researchers have a responsibility to ensure reliability with the construct being tested to study the populations and sub-groups in their study, and it is especially important when investigating traditionally marginalized groups who were not likely to have been represented in the original scale development process. This study is a great step for assessment of existing scales, but it does require additional research so that more groups can be included to support scale development and selection. The next steps could include further demographic characteristics for these scales and an evaluation of further scales not included in the study with confirmatory factor analysis and measurement invariance to specify their issues and identify solutions. Additionally, NEFE is in the process of collaborating with the panel to produce a consensus paper sharing recommendations emerging from this project and a set of methodological studies to address specific topics raised during the project.