Scale Validation

For the past two years, NEFE has led a collaboration on a scale validation project with the intent of addressing a major gap in the literature: the lack of rigorous testing and validation of one of the most utilized survey assessments in the financial services industry. It is important to understand which scales work best and for whom, especially when survey results are used to make decisions about individuals and communities. Some studies in personal finance use confirmatory factor analysis to assess the measure fit of scales, but little work has been done to assess measure invariance, meaning there are little to no differences in interpretation across sub-groups. This makes it difficult to interpret study findings or to discern what a study is contributing to the body of knowledge. Invariance testing is crucial to studying how different scales apply across demographics and other consumer subsets. Researchers have faced barriers in making this happen due to lack of training, funding or other reasons. This motivated NEFE to explore solutions that will circumvent this obstacle and help move the field forward.

The two-year collaborative research project with Mission Measurement started with approximately 90 scales, filtered by the following criteria:

• Created using samples of adults aged 18 years and older.
• Produced after the year 2000.
• Has at least two measurable items within the scale.

NEFE, with assistance from a panel of seven financial literacy experts, narrowed down the 90 scales to 11 final scales, based on the presence of some validity testing, rigorous development, or those that are pervasive and commonly used. The project’s data originated from the 11 chosen scales and demographic questions, with the sample attempting to capture the diversity of the U.S. adult population to ensure sufficient counts of 12 demographic sub-groups to support the planned analyses.

Two methods were used in the assessment of these existing scales. Confirmatory factor analysis examined how well the hypothesized factor structure (i.e. the set of items for each scale or subscale and the factor loadings for each item) fits the new data collected. Scale reliability (assessed using Cronbach’s Alpha) was utilized to measure whether the scale could be expected to provide consistent measures over time. The results showed that, of the 11 scales, five exceeded all fit statistics and were labeled “favorable”; three were labeled “reasonable” because they exceeded most fit statistics; and three were labeled “problematic” because they failed to exceed the thresholds for several fit statistics.