Psychometric Analysis of an Instrument to Study Retention in Engineering

Abstract

Although engineering programs admit highly qualified students with strong academic credentials, retention in engineering remains lower than most other programs of study. Addressing retention by modeling student success shows promise. Instruments incorporating noncognitive attributes have proven to be more accurate than those using only cognitive variables in predicting student success. The Student Attitudinal Success Instrument (SASI-I), a survey assessing nine specific noncognitive constructs, was developed based largely on existing, validated instruments. It was designed to collect data on affective (noncognitive) characteristics for incoming engineering students (a) that can be collected prior to the first year and (b) for which higher education institutions may have an influence during students’ first year of study. This chapter will focus on the psychometric analysis of this instrument. Three years of data from incoming first-year engineering students were collected and analyzed. This work was conducted toward investigating the following research questions: Do the scale scores of the instrument demonstrate evidence of reliability and validity, and what is the normative taxonomy of the scale scores of first-year engineering students across multiple years? Further, to what extent did the overall affective characteristics change over the first year of study?

1. Introduction

Engineering programs tend to admit students who are academically talented, defined by strong grade point averages and standardized exam scores. Unfortunately, many of these students leave engineering, often at the end of their first year of study. Further, the structure of engineering plans of study makes it difficult for students to transfer into engineering, meaning that engineering programs show a significantly lower percentage of retained students when compared to other disciplines.

A plethora of publications regarding undergraduate engineering student retention have been written. Efforts to reform undergraduate engineering vary from the introduction to first-year engineering programs, to pedagogical improvements, to curricula focused on design, mentorship programs, etc. Studies have attempted to develop predictive models of student success and retention. Studies have demonstrated evidence of strong predictive power of noncognitive attitudes over purely cognitive measures of students in retention and future academic performance [1, 2, 3, 4, 5, 6, 7, 8, 9].

This paper focuses on the psychometric analysis of the initial version of the Student Attitudinal Success Inventory (SASI-I), which was shown to be valid and reliable, and further discusses the use of normative taxonomy of first-year engineering students across multiple years to assess engineering students’ multifaceted noncognitive attributes. Further, studies of shifts in noncognitive attributes over the first year of study have repeatedly shown trends in an unfavorable direction [8, 10, 11]. The scale was used at the end of the year to examine trends in student normative taxonomy (as operationalized by cluster membership) over the course of an academic year. This chapter will introduce this analysis technique to the study of noncognitive characteristics of first year engineering students.

2. Driving research questions

Specific research questions which led to this analysis include:

1. What is the evidence of reliability and validity for the SASI-I instrument consisting of a number of affective / attitudinal factors related to student success, based on incoming first-year engineering student responses?

2. What evidence supports the use of normative taxonomies (clusters) to establish SASI-I factor stability over different cohorts of incoming first-year engineering students?

3. How do the normative taxonomies of affective student characteristics change over the first year? To what extent do students’ group memberships change over time?

3. Instrumentation

Data were based on separate cohorts of undergraduate engineering students enrolling in a large Midwestern university over a three-year period of 2004 (cohort 1; N = 1,605), 2005 (cohort 2; N = 1,777), and 2006 (cohort 3; N = 1,779). Table 1shows cohort demographics. Each cohort consisted of all entering first-year students who were admitted to the college of engineering but had not yet started their first semester, a small number of whom did not subsequently enroll or attend classes at the institution.

	Preyear			Postyear
	Number	Male	Female	Number	Male	Female
Cohort 1	1,605	1,297; (80.8%)	308; (19.2%)	722	563; (78.0%)	159; (22.0%)
Cohort 2	1,777	1,502; (84.2%)	275; (15.5%)	627	516; (82.3%)	111; (17.7%)
Cohort 3	1,779	1,482; (83.3%)	297; (16.7%)

Table 1.

Demographics of student cohort groups, pre- and postsurveys.

The SASI-I was used to collect data from entering engineering students and was used to assess their affective / attitudinal characteristics. Data were used to identify the normative taxonomy of cohorts of incoming students [12].

Students completed the 161 item SASI-I instrument online as part of a required set of activities at orientation, indicating their responses on a Likert scale (from 1 = Strongly Agree to 5 = Strongly Disagree). The SASI-I was administered along with other tests (e.g., math placement, chemistry) prior to the first semester and again at the end of the academic year. Students who did not complete each of the three parts of the SASI-I assessment were excluded from analysis.

The self-report measures sought to assess students’ affective / attitudinal beliefs across the following constructs, each theorized within the literature to be critical to academic success [1, 12].

• Academic Motivation: consisting of 25 items in four subfactors: Control, Challenge, Curiosity and Career.

• Metacognition: comprised of 20 items in four subfactors: Planning, Self-monitoring/Self-Checking, Cognitive Strategy and Awareness.

• Deep Learning: consisting of 10 items in two subfactors, Motive and Strategy.

• Surface Learning: consisting of 10 items, originally with identical subfactors to Deep Learning. Factor analysis showed two different subfactors, Memorization and Studying.

• Academic Self-Efficacy: consisting of ten individual items that do not form specific subfactors.

• Leadership: consisting of 20 items with four subfactors, Motivation, Planning, Self-Assessment and Teammates.

• Team vs. Individual Orientation: consisting of 10 items in two subfactors, Individual and Team Dynamic.

• Expectancy-Value: consisting of 32 items in five subfactors: Academic Resources, Community Involvement, Employment Opportunities, Persistence and Social Engagement.

• Major Decision: consisting of 21 items in four subfactors: Certainty of Decision, Difficulty in Decision, Personal Issues, and Urgency. One question was shown not to load to any of the subfactors and is assessed on its own (Independence).

The multilevel structure, where each item loads to a superordinate construct or general factor (for example, Major Decision) and one subfactor or subordinate factor within the domain of the construct (for example, the Certainty of Decision subfactor under Major Decision) supports analysis at multiple levels.

Table 2 shows a summary of constructs, subconstructs, and number of items in each construct.

Construct	Origin	NS	Nt	Subconstruct	Ni
Intrinsic motivation	Academic Intrinsic Motivation Scale (AIMS) [13]	4	25	Career; Challenge; Control; Curiosity	5; 6; 7; 7
Academic self-efficacy	Based on [14]	1	10	Self-efficacy	10
Expectancy-value	Based on [15]	5	32	Expected use, acad resources; Community involvement; Employment opportunities; Persistence; Social engagement	5; 4; 8; 7; 8
Deep learning approach	Revised two-factor Study; Process Questionnaire; (R-SPQ-2F) [16]	2	10	Motive; Strategy	5; 5
Surface learning approach		2	10	Memorization; Studying	3; 7
Metacognition	Based on State Metacognitive Inventory [17]	4	20	Awareness; Cognitive strategy; Planning; Self-checking	5; 5; 5; 5
Leadership	Based on [18]	4	23	Motivation; Planning; Self-assessment; Teammates	5; 4; 7; 7
Team vs. individual orientation	Based on [19]	2	10	Individual dynamic; Team dynamic	5; 5
Major decision	Based on [20]	5	21	Certainty of decision; Difficulty of decision; Personal issues; Urgency; Independence	3; 10; 4; 3; 1
Total	9 constructs	29	161		161

Table 2.

Constructs, subconstructs, and number of items in each construct.

4. Psychometric analysis

4.1 Internal consistency of scales and subscales

Internal consistency of scale scores was investigated using Cronbach’s coefficient alpha for each scale and subscale, for each cohort. Values for Cronbach’s coefficient alpha exceeding 0.80 are desired [21, 22]. As Cronbach’s coefficient alpha is sensitive to the number of items within a construct, the Spearman-Brown formula [23, 24] was used to estimate Cronbach’s alpha in any subscale containing less than 10 items.

4.2 Construct and subscale structure

Factor analytic procedures were used to test the scales’ multidimensional structures. Each item loads to a construct or general factor (for example, Surface Learning) and one subscale or factor within the domain of the construct (for example, the Studying subscale under Surface Learning) since the scales were based on multidimensional constructs.

Subscale definitions were examined through confirmatory factor analysis (CFA) for those constructs with an a priori structure. For those constructs developed specifically for this instrument, Exploratory Factor Analysis (EFA) was used to establish the subscale structure.

SAS (version 9.1.3) proc factor with a promax rotation was used for EFA. Promax rotation allows for the rotation of the axes to a position allowing optimal loadings for a set of items. The ideal number of factors was determined using both the Kaiser criterion, in which the number of factors is indicated by factors whose eigenvalues are greater than 1, and examination of the scree plot of eigenvalues.

Confirmatory Factor Analysis was performed to test the factor structure of each construct. Fit was assessed based on the 2004 cohort of students using LISREL™. Each construct (with the exception of self-efficacy) was specified using a path diagram showing each latent variable loading to the overall construct and one individual subscale. The null hypothesis (H₀) is that the data will adequately fit the proposed structure for each construct at the item level. In cases where EFA was used to specify the subscale structure (constructs developed for this instrument), a randomly selected subset of the data (n = 500) was used in the EFA procedure; a mutually exclusive subset of the data (n = 1000) was then used to verify the structure using CFA.

Confirmatory Factor Analysis fit was assessed using a number of criteria, including the chi-square statistic, the Comparative Fit Index (CFI), the Goodness of Fit Index (GFI) and the Root Mean Square Error of Approximation (RMSEA). The chi-square statistic is reported, but its sensitivity to sample size means that it is rarely used as the sole criterion to judge model fit [25]; with a sample size in excess of 1500 students, rejection of the null hypothesis is expected. Instead, Hu and Bentler [25] suggest that acceptable model fit (no Type I or Type II errors) is indicated when CFI > 0.95 and RMSEA <0.08, with an excellent fit indicated when RMSEA <0.05. Tanguma [26] demonstrated that the CFI and GFI were relatively unaffected by sample size for sufficiently large samples (n > 500), with acceptable model fit indicated by values of GFI > 0.90.

4.3 Cohort group normative taxonomy

To show factor stability over time, McDermott’s [27] three stage cluster analysis was used on each cohort to determine normative taxonomies of students, clustering students with similar response patterns. Each cohort was compared to each other cohort to measure the similarity from year to year. Further, results from each postsurvey sample (cohorts 1 and 2, 2004 and 2005) are established and compared to each other and to the pre-first year normative taxonomy.

The first step of the analysis was converting raw scores to normalized z-scores to equally weight each of the nine affective/attitudinal constructs with respect to each other. Subsequently, mutually exclusive groups of approximately equal size partitions (B = number of blocks) of the data using Ward’s minimum variance method [28]. For example, data for the 2005 cohort consists of nine blocks: seven with 197 and two with 199 students, with a normalized z-score for each affective / attitudinal construct for each student. For each random block, criteria for determining the optimal number of clusters (K) were: R² statistic (indicating the proportion of the variance accounted for by the clusters), the pseudo-F statistic over the pseudo-t² statistic [29] and Mojina’s first stopping rule [30].

The second stage involved formation of a (B∙K) x (B∙K) similarity matrix reflecting the consequence of merging any two clusters. Each resulting cluster was considered as input to the cluster analysis procedure, resulting in the final number of clusters indicated for the complete data set. The resulting homogeneity coefficients indicate, in this case, the consistency from year to year for each cohort group.

Finally, the third stage applied k-means iterative (nonhierarchical) partitioning to relocate potentially misassigned individual cases to improve homogeneity coefficients. The profile of each individual is examined to ensure membership in the ideal cluster: misassigned profiles are reassigned to a profile more closely matching the individual. All analysis was done in SAS with modifications to code developed by Paul McDermott [27].

Final clusters were expected to satisfy an average within-cluster homogeneity coefficient $\overline{H}$ > 0.6 [31]. Cattell’s Cluster Similarity Coefficient, r_p, [32] was calculated to demonstrate cluster similarity between clusters within and between cohort groups. Higher coefficient values demonstrate better congruence: excellent similarity is shown with values greater than 0.95 while values between −0.7 and +0.7 show poor factor similarity [33].

4.4 Differences in taxonomy over the course of an academic year

Analysis of the response data showed that an assumption of normality was not valid; therefore, nonparametric tests were used to analyze the data for differences over the course of the academic year. Comparisons for statistically significant differences were done using Mann-Whitney nonparametric tests of comparison. Mann-Whitney nonparametric test results were found using SAS for Windows (version 9) proc. npar1way with the Wilcoxon and Monte Carlo (MC) options. The MC option produces Monte Carlo estimates of exact p values and is used specifically for large data sets. Specifying the Monte Carlo estimate results in an estimated value of p as well as upper and lower bounds of the confidence interval of the actual p value (using alpha = 0.01) with a significant savings in computational time. In addition to nonparametric tests, standard two tailed t-tests were also computed and results compared with those of nonparametric tests.

4.4.1 Effect size: Cohen’s d

Statistical significance of differences is influenced by large sample sizes, and a statistically significant difference does not necessarily imply a meaningful or important difference – only that a true difference of means most likely exists. As the size of the population increases, even very small differences tend to become significant. The effect size, or Cohen’s d, is a measure of the magnitude of the effect or the importance of the difference [33, 34, 35]. Cohen’s d is found by:

$$d=\frac{{\text{M}}_1-{\text{M}}_2}{{\sigma }_{pooled}}$$E1

where M₁ and M₂ are the means of the male and female population. The pooled standard deviation, σ_pooled, is the root-mean-square of the standard deviations of the two populations [33]. That is:

$${\sigma }_{pooled}=\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{2}}$$E2

When the two standard deviations are similar (as is typically the case), the root mean square differs very little from the simple average of the two variances.

Hyde [36, 37] defined ranges for effect sizes as part of the Gender Similarity Hypothesis as: near-zero, d ≤ 0.10; small, 0.11 < d ≤ 0.35; moderate, 0.36 < d ≤ 0.65; large, 0.66 < d ≤ 1.0; and very large, d > 1.0.

5. Results

5.1 Structure of the overall instrument

For those constructs with a predefined structure (Motivation, Metacognition, Deep Learning and Surface Learning), Exploratory Factor Analysis (EFA) was performed to verify that the items loaded to the constructs as specified in the literature. EFA results agreed in all but one case, Surface Learning. In the constructs without a predefined structure (Academic Self-efficacy, Leadership, Team vs. Individual Orientation, Expectancy-Value and Major Decision), EFA was used to define the multidimensional structure. Confirmatory Factor Analysis was used to assess the fit of the multidimensional structure of the constructs. Fit indices including chi-square, CFI, GFI and RMSEA for each construct are shown in Table 3.

	χ2	χ2 df	pr > χ2	GFI	RMSEA	CFI
Motivation	902.6	240	<.0001	0.99	0.053	1.00
Metacognition	767.53	141	<.0001	0.99	0.067	1.00
Leadership	522.16	197	<.0001	0.99	0.041	1.00
Team vs. Individual orientation	63.45	22	<.0001	1.00	0.012	1.00
Expectancy value	1410.1	417	<.0001	0.98	0.049	1.00
Major decision	805.16	153	<.0001	0.99	0.065	1.00
Deep learning	49.73	22	.00064	1.00	0.036	1.00
Surface learning (original subscales)	152.5	25	<.0001	0.97	0.072	0.942
Surface learning (revised)	31.34	24	0.14	0.99	0.013	1.00
Academic self-efficacy	n/a

Table 3.

Confirmatory Factor Analysis results for each construct.

Surface Learning original and revised are listed to show improvement with revision of subscale structure.

χ2 = Chi-squared; χ2 df = Chi-square degrees of freedom; GFI = Goodness of Fit Index; RMSEA = Root Mean Square Error of Approximation estimate; CFI = Bentley’s Comparative Fit Index

Motivation, Metacognition and Deep Learning constructs: Subscales that were unchanged from those originally presented in their literature included those under motivation, metacognition and deep learning. Factor analysis supported the subscales as originally specified. CFA results show an acceptable fit for each of these constructs with values for GFI > 0.9, CFI > 0.9 and RMSEA <0.08.

Surface Learning: The subscales of surface learning were originally defined the same as those of deep learning: Motive and Strategy. However, EFA results indicated that the subscale Strategy itself loaded into two separate factors, which is typically not indicative of a homogeneous construct. EFA results on the entire Surface Learning construct showed individual items clearly loading into one of two factors, which were redefined as Memorization and Studying based on context of the questions. CFA results indicate a significant improvement in fit. The redefined structure with modified subscales resulted in a value of chi-square that was not significant, meaning that the data did indeed fit the theoretical structure, even with a very large sample size.

Academic Self-Efficacy: EFA performed on the academic self-efficacy construct indicated no subscales.

Leadership, Team vs. Individual Orientation, and Expectancy-Value: Subscales for each of these constructs were defined based on exploratory factor analysis: results were validated using CFA on a mutually exclusive subset of the data. Table 3 shows the results of the CFA for each construct, with values for GFI > 0.9, CFI > 0.9 and RMSEA <0.05, verifying the validity of the structure for each construct.

Major Decision: Results of the EFA showed the items in the Major Decision construct loaded to five subscales and CFA results showed an acceptable fit for this structure (GFI = 0.99, CFI > 0.99 and RMSEA = 0.065). The Major Decision scale contained one question which was shown not to load on any particular subscale and is presented independent of the subscales (Independence). Three items were shown to negatively correlate to the remainder of the scale, and were reverse scored during the analysis.

5.2 Internal consistency of scales and subscales

Reliability of the instrument is demonstrated with acceptable values of Cronbach’s alpha for each construct and subscale for each cohort [21, 22, 38]. Complete results are shown in Table 4. Values of Cronbach’s alpha are shown after reverse-scoring two of the items in the Major Decision scale.

Number of items		Cohort 1	Cohort 2	Cohort 3
Motivation	25	0.93	0.93	0.93
Career	5	0.82	0.81	0.83
Challenge	6	0.80	0.79	0.80
Control	7	0.79	0.80	0.79
Curiosity	7	0.81	0.81	0.82
Metacognition	20	0.92	0.92	0.93
Awareness	5	0.77	0.79	0.79
Cognitive strategy	5	0.78	0.79	0.79
Planning	5	0.76	0.78	0.79
Self-checking	5	0.72	0.74	0.76
Deep learning	10	0.81	0.83	0.82
Motive	5	0.69	0.70	0.69
Strategy	5	0.67	0.71	0.71
Surface learning	10	0.79	0.79	0.79
Memorization	3	0.60	0.59*	0.62
Studying	7	0.80	0.79	0.80
Leadership	23	0.89	0.89	0.89
Motivation	5	0.69	0.69	0.69
Planning	4	0.70	0.70	0.67
Self-assessment	7	0.78	0.77	0.77
Teammates	7	0.68	0.70	0.68
Team vs. Individual orientation	10	0.75	0.79	0.76
Individual	5	0.61	0.63	0.59*
Team dynamic	5	0.69	0.74	0.73
Expectancy value	32	0.90	0.91	0.91
Academic resources	5	0.75	0.76	0.76
Community involvement	4	0.71	0.72	0.72
Employment opportunities	8	0.81	0.82	0.82
Persistence	7	0.80	0.81	0.81
Social engagement	8	0.81	0.80	0.80
Major decision	21	0.85	0.85	0.85
Certainty of decision	3	0.49*	0.49*	0.53*
Difficulty in decision	10	0.92	0.92	0.92
Personal issues	4	0.60	0.61	0.56*
Urgency	3	0.57*	0.58*	0.59*
Independence	1	—	—	—
Academic Self-efficacy	10	0.90	0.90	0.90

Table 4.

Values of Cronbach’s coefficient alpha for each construct and subscale.

*

indicates alpha values less than 0.6

Cronbach’s coefficient alpha values for all scales exceed 0.8 with two exceptions: Surface learning (α = 0.79) and Team vs. Individual Orientation (α ≥ 0.75), demonstrating the homogeneous nature of each construct [22, 39]. The lack of variation in values of Cronbach’s coefficient alpha for different student cohort groups is one indication of stability and repeatability of the scales over time. Using the Spearman-Brown formula [24, 40] results in an estimate for Cronbach’s coefficient alpha for a construct when interpolated to a specified number of items. In cases where there were fewer than 10 items in a subscale, the Spearman-Brown formula was used to assess values of Cronbach’s coefficient alpha for a consistent number of items. Some subscales with very few items were outside of this range; the small number of items within each subscale certainly contributed to low values of alpha. In each case, values of alpha were very consistent from cohort to cohort, further demonstrating the internal consistency of the constructs and subscales.

5.3 Cluster analysis: Pre-year survey

McDermott’s three stage cluster analysis was used to derive the core profiles for each of the three years of cohort data from cohort 1 to cohort 3. The primary goal of this analysis was to identify normative taxonomies of individuals who exhibit similar profiles within a given cluster, yet dissimilar profiles across clusters. Similar taxonomies are indicated by a consistent number of core profiles, and profiles of similar magnitude and shape as determined by the cluster homogeneity coefficient $\overline{H}$ > 0.6 [31] and Cattell’s similarity coefficient [32] (similar: r_p > 0.95, dissimilar: |r_p| < 0.7) [41].

Cluster analysis resulted in three core profiles for each cohort. The shape and pattern of each profile was consistent from cohort to cohort, showing strong repeatability and stability.

Figure 1 shows an overlay of plots of the means of each construct for each cluster of students. Specific constructs are shown on the x-axis, with center values (normalized z-scores) for each construct for each cluster on the y-axis. There is no significance to the order of the constructs on the x-axis. Center means for each core cluster within each cohort are shown in Table 5. The identification of three distinct clusters and the shape of each cluster of students are significant. Students who tended to rate themselves at least one standard deviation stronger than other students tended to do so across the board, except for their propensity toward surface learning: the sharp spike seen in the plots indicates these students view their learning style as deep (developing an understanding and appreciation of material) as opposed to surface (memorization). As might be expected, those students rating themselves below the affective / attitudinal norms indicated a propensity toward surface learning. Students in cluster 1 responded approximately one standard deviation below the norm (with the exception of Surface Learning) while students in cluster 3 responded one standard deviation above the average (again, with the exception of Surface Learning). Students in cluster 2 clustered about the norms for each construct. There is no significance to the cluster numbers; they are used only to distinguish between groups.

Figure 1.

Overlay of plots of normalized center means for each construct, shown for each cohort of students. Data points are shown as connected to illustrate similarly / dissimilarity of each cluster, and are not meant to imply a relationship between constructs.

Cluster	Cohort	Zex-val	Zlead	Zmaj dec	Zteam	Zmeta	Zdeep	Zsurf	Zmotiv	Zself	n
1 (lower)	2004	−1.03	−1.06	−0.52	−0.83	−0.96	−0.94	0.67	−1.12	−0.96	400
	2005	−0.85	−0.85	−0.50	−0.71	−0.78	−0.80	0.63	−0.94	−0.83	611
	2006	−0.87	−0.89	−0.49	−0.73	−0.79	−0.87	0.61	−0.96	−0.79	570
2 (middle)	2004	−0.02	−0.02	−0.03	−0.06	−0.08	−0.03	0.02	−0.02	−0.06	804
	2005	0.09	0.09	0.12	0.05	0.01	0.06	−0.13	0.14	0.11	840
	2006	0.09	0.08	0.08	0.06	−0.01	0.09	−0.12	0.13	0.05	892
3 (upper)	2004	1.07	1.09	0.58	0.94	1.12	1.01	−0.69	1.16	1.09	401
	2005	1.35	1.35	0.62	1.21	1.43	1.33	−0.82	1.37	1.25	326
	2006	1.33	1.39	0.65	1.19	1.45	1.28	−0.78	1.37	1.30	317

Table 5.

Center means for each construct for core cluster.

Table 6 shows the number of students within each profile. Final clusters satisfied an average within cluster homogeneity coefficient $\overline{H}$ > 0.99 (Table 6), demonstrating that the clusters were indeed homogeneous. The plots in Figure 2 show year to year consistency in shape with minimal variation in values for each construct. Stability is demonstrated with minimal variability in the center values for each cohort over multiple years.

	n (2004)	$\overline{H}$ (2004)	n (2005)	$\overline{H}$ (2005)	n (2006)	$\overline{H}$ (2006)
Cluster 1 (lower)	400 (25%)	0.997	.11 (35%)	0.997	570 (32%)	0.997
Cluster 2 (middle)	804 (50%)	0.998	840 (47%)	0.998	892 (50%)	0.998
Cluster 3 (upper)	401 (30%)	0.997	326 (18%)	0.997	317 (18%)	0.997

Table 6.

Number of students (n) and average within cluster homogeneity coefficient ($\overline{H}$) in each cluster for cohorts 1–3 (2004–2006).

Figure 2.

Overlaid plots of center means for each subscale for clusters 1 and 2 for presurvey data.

Cattell’s Cluster Similarity Coefficients were calculated to objectively determine the similarity between each cluster for each year. Excellent cluster similarity is demonstrated with values of Cattell’s coefficient r_p > 0.95; values of Cattell’s coefficient comparing clusters expected to be highly similar are consistently 0.94 < r_p < 1.00 (see Table 7), demonstrating these clusters to be similar and stable over this time period. Clusters expected to be dissimilar show comparison values well below values indicating similarity, demonstrating these clusters to be dissimilar to each other. Strong coefficients of similarity and similar percentages of students within each cluster show that student responses tend to remain stable from year to year.

		2005					2006					2006
cluster		1	2	3	cluster		1	2	3	cluster		1	2	3
2004	1	0.98	0.54	0.08	2004	1	0.99	0.53	0.08	2005	1	1.00	0.41	0.18
	2	0.29	0.98	−.42		2	0.30	0.98	−.42		2	0.44	1.00	−.36
	3	0.39	−.25	0.94		3	0.37	−.26	0.94		3	0.17	−.37	1.00

Table 7.

Cattell’s Cluster Similarity Coefficient for each cluster 2004–2006.

Values show the coefficient for the cluster in the year indicated for each row / column. Bold values show comparison of clusters theoretically similar.

Interestingly, the smallest spread among cohorts was found in Major Decision, indicating that students in each profile appear to differ least in their initial intent to pursue an engineering degree. The widest disparities between clusters appear to be among motivation and metacognition; this may be expected given the strong academic backgrounds of incoming engineering students.

5.4 Clusters in the postsurvey sample population

Cluster analysis on the postsurvey data shows that the sample population surveyed at the end of the first year clustered into four distinct groups; a significant finding as it differs from the three groups found in the presurvey. The sample population was divided into mutually exclusive groups as input to McDermott’s three-stage cluster analysis; during this process, the ideal number of clusters is assessed for each subgroup, then carried through the analysis to arrive at a final answer. The criteria include multiple measures to establish the ideal number of clusters taken in concert with each other. In most cases, four clusters were indicated by Mojina’s first stopping rule while the Cubic Clustering Criteria (CCC) and the pseudo-F statistic over the pseudo-t² statistic [29] indicated between 4 and 6 clusters. However, Milligan and Cooper [42] found that the CCC often indicates too many clusters. In each subgroup, a four cluster solution was indicated, leading to a final 4-cluster solution for the sample population in cohort 1 and 2 postsurvey data. Figure 2 shows an overlay plot of the clusters in the postsurvey data for 2004 and 2005.

An examination of the four clusters shows a clearly “upper” cluster and “lower” cluster, where students tended to respond to the SASI either higher or lower (respectively) than their peers (except for Surface Learning as expected). Two clusters emerge near the middle of the responses. As seen in Figure 2, these clusters are similar with the exception of Deep and Surface Learning and Major Decision. As a group, students who responded near the average of their peers and tended toward surface learning tended to be significantly lower in the decision to continue in engineering – this group is designated “middle (low)”. Conversely, students who tended away from surface learning tended to indicate decisiveness toward their major [“middle (high)”]. Further examination of these two groups showed that, although the sample population was heavily skewed toward those remaining in engineering, retention in the “lower” and “middle (low)” groups was lower than retention in the “upper” and “middle (high)” group, 95% vs. 98% within the sample population. However, the low numbers of students who did not continue in engineering in the program under study do not allow a definitive conclusion to be drawn from these percentages.

Table 8 shows values of Cattell’s similarity coefficients, indicating similarity between clusters which are presumed to be similar, and dissimilarity between all other clusters. Notably, the “upper” profiles did not demonstrate excellent similarity (with r_p = 0.75), although they are acceptably similar (with r_p > 0.7). The “middle (high)” and “middle (low)” clusters do show a degree of similarity, as expected, but are dissimilar enough to justify two distinct clusters of students by Cattell’s similarity coefficient (r_p < 0.6) (Figure 3).

	$\overline{H}$ (2004)	$\overline{H}$ (2005)			2005
Cluster 1	0.997	0.997	cluster		1	2	3
Cluster 2	0.998	0.998	2004	1	0.98	0.54	0.08
Cluster 3	0.997	0.997		2	0.29	0.98	−0.42
				3	0.39	−0.25	0.94

Table 8.

Average within cluster homogeneity coefficient ($\overline{H})$ and Cattell’s similarity coefficient for cohorts 1 & 2, presurvey data.

Cluster 1 = “lower”; cluster 2 = “middle”; cluster 3 = “upper”.

Figure 3.

Overlay plot of 2004 and 2005 postsurvey clusters.

Mid (1) indicates a cluster about the average with a higher value of Major Decision. Mid (2) indicates a cluster about the average with a lower value of Major Decision.

5.5 Cluster analysis of the aggregate postsurvey population

Because of the similarity of the cluster solution between cohorts 1 and 2, the data will be taken in aggregate for much of the analysis. McDermott’s cluster analysis was repeated on the aggregate population by combining the data, sorting the students randomly and forming mutually exclusive datasets as previously described. Eight blocks of 184 students were used as input and, as expected, a four cluster solution was indicated once again. Cattell’s similarity coefficient showed the resultant cluster solution was similar to the solution based on cohorts 1 and 2 where expected, with values of r_p > 0.84 (Table 9).

		Cohort 2
population	cluster	Post high	Post mid (high)	Post mid (low)	Post low
Cohort 1	Post High	0.75	0.54	0.27	−0.24
	Post Mid (high)	−0.01	0.94	0.59	0.48
	Post Mid (low)	0.07	0.48	0.99	0.22
	Post Low	−0.49	0.08	0.10	0.94

Table 9.

Cattell’s similarity coefficients, cohorts 1 and 2, postsurvey clusters.

Bold text indicates clusters which are presumed similar. Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

5.6 Differences in constructs, pre- to postsurvey

Ideally, students should improve not only in their cognitive abilities through the first year, but also in their desirable noncognitive characteristics. Of the nine constructs in the SASI-I, a desirable outcome would be an increase in the students’ self-perception in eight of the constructs (Motivation, Metacognition, Deep Learning, Academic Self-Efficacy, Leadership, Team vs. Individual Orientation, Expectancy-Value, and Major Decision) with a lower propensity toward Surface Learning. An examination of the means of student responses from the presurvey to the postsurvey shows us that postsurvey responses went down significantly (except for Surface Learning, which increased) over the first year of study. Table 10 shows the mean values and effect sizes for all differences. While there was a statistically significant movement (p < 0.001) in the nondesired direction for each construct, only Surface Learning and Expectancy-Value also showed a large effect size, or a large ‘importance’ of shift in mean values.

		Cohort 1 and 2 aggregate postsurvey
population	cluster	Post High	Post Mid (high)	Post Mid (low)	Post Low
Cohort 1	Post High	0.84	0.49	0.22	−0.29
	Post Mid (high)	0.06	0.97	0.62	0.40
	Post Mid (low)	0.13	0.48	0.98	0.16
	Post Low	−0.46	0.12	0.17	0.98
Cohort 2	Post High	0.97	0.11	0.05	−0.45
	Post Mid (high)	0.23	1.00	0.57	0.19
	Post Mid (low)	0.17	0.56	0.99	0.18
	Post Low	−0.37	0.32	0.31	0.99
Aggregate	Post High	1.00	0.19	0.11	−0.41
	Post Mid (high)	0.19	1.00	0.57	0.24
	Post Mid (low)	0.11	0.57	1.00	0.25
	Post Low	−0.41	0.24	0.25	1.00

Table 10.

Cattell’s similarity coefficients, comparing aggregate population postsurvey data to each of cohorts 1 and 2 and aggregate postsurvey data.

Bold text indicates clusters which are presumed similar. Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

Comparing the presurvey cluster analysis results with the postsurvey cluster analysis results gives an indication of the similarity of the profiles prior to the first year and at the end of the first year. The fact that the postsurvey data results in a four cluster solution indicates that shifts have certainly occurred with the emergence of an additional cluster, or the further division of student responses.

Figure 4 shows an overlay plot of presurvey clusters from 2004 and postsurvey aggregate results from 2004 to 2005. There is a clear similarity in appearance in the “upper” and “lower” clusters from the presurvey and postsurvey. The middle clusters show a clear deviation for Surface Learning and Major Decision. Table 11 shows values of Cattell’s similarity coefficients, indicating that the “upper” and “lower” clusters are indeed similar (r_p = 0.84 and r_p = 0.93, respectively). The presurvey “middle” group shows acceptable similarity to the two middle clusters of the postsurvey data (r_p = 0.84 and r_p = 0.79). Clusters presumed to be dissimilar are indeed shown to be dissimilar (|r_p| < 0.53).

Figure 4.

Presurvey clusters (cohort 1 shown) and postsurvey clusters (cohorts 1 and 2 aggregate).

	Presurvey		Postsurvey
Construct	Mean	σ, pre	Mean	σ, post	Post-pre	Cohen’s d	Effect size
Surface Learning	2.487	0.493	3.028	0.628	0.542	0.959	Large
Expectancy-Value	3.915	0.359	3.582	0.500	−0.333	−0.765	Large
Motivation	4.111	0.407	3.816	0.526	−0.295	−0.628	Moderate
Major Decision	3.515	0.441	3.222	0.521	−0.293	−0.608	Moderate
Self-Efficacy	4.232	0.471	3.950	0.589	−0.281	−0.528	Moderate
Deep Learning	3.663	0.477	3.422	0.606	−0.241	−0.442	Moderate
Team vs. Individual *	3.940	0.389	3.793	0.510	−0.147	−0.323	Small
Metacognition *	3.923	0.408	3.803	0.523	−0.120	−0.255	Small
Leadership *	3.926	0.372	3.818	0.486	−0.108	−0.249	Small

Table 11.

Differences in mean student responses, cohort 1, pre- and postsurvey, Effect Size shown.

* = Statistically significant difference, small to near-zero effect size.

While these results are important, an investigation based on students tending to shift from one cluster to another through the course of the year should shed additional light on these trends.

5.7 Cluster drift: changes in cluster membership over the first year

While the discovery of a three-cluster presurvey and four-cluster postsurvey solution is significant, the movement of students from a precluster to a postcluster is also of interest.

Table 12 is a frequency table showing the number of students moving from each pre-survey cluster to each postsurvey cluster, including the number of male and female students within each group. One indication of cluster stability from the presurvey to the postsurvey is that 55% of students remained within their original cluster: this assumes students from the presurvey “middle” cluster remain in one of the two middle clusters in the postsurvey group.

		Cohorts 1 and 2 aggregate data, postsurvey
Population	Cluster	Post High	Post Mid (high)	Post Mid (low)	Post Low
Cohort 1	Pre High	0.84	0.52	0.29	−0.26
	Pre Middle	0.01	0.84	0.79	0.53
	Pre Low	−0.45	0.12	0.21	0.93

Table 12.

Cattell’s similarity coefficients, 2004 presurvey and 2004–2005 aggregate postsurvey clusters.

Bold text indicates clusters which are presumed similar.

Italics indicate clusters from the middle regions which are expected to be somewhat similar.

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

Some other observations can be made immediately:

• Most students who were in the presurvey “upper” cluster moved to either the postsurvey “upper” cluster (n = 81) or the middle cluster with high Major Decision [“middle (high)”] (n = 88).

• Most students who were in the presurvey “lower” cluster moved to the postsurvey “lower” cluster (n = 189), although a large percentage went to the middle cluster with high Major Decision [“middle (high)”] (n = 132 or 31% of students from the presurvey “low” cluster), a favorable shift in student noncognitive characteristics.

• Most students in the postsurvey “upper” cluster came from the presurvey “upper” cluster (n = 81).

• Most students in the postsurvey “lower” cluster came from the presurvey “lower” cluster (n = 189), although a large percentage came from the presurvey “middle” cluster (n = 155, or 41% of the postsurvey low cluster).

• The fewest number of students (n = 6) took the transition from the “lower” presurvey cluster to the “upper” postsurvey cluster. Second to this was the least favorable transition, from presurvey “upper” to postsurvey “lower” (n = 36).

• The largest number of students transitioning was from the presurvey “middle” cluster to the middle cluster with a high Major Decision score [“middle (high)”]: n = 276, or nearly 21% of the population.

If an unfavorable shift is defined as one where a student downgrades their original cluster membership, for example, from presurvey “middle” to postsurvey “lower”, or remains in the postsurvey “lower” cluster, 515 students, or 39%, shifted unfavorably in their noncognitive characteristics. Defining a favorable shift as one where students’ trajectories are from a lower presurvey cluster to a higher postsurvey cluster or remain in the postsurvey “upper” cluster results in 354 students (or 27%) shifting in a favorable direction. It should be noted that, because the means of each cluster decreased from the presurvey to the postsurvey, students classified as having a positive shift may in fact have lower scores in their self-perception of their noncognitive attributes; if this indeed constitutes a positive shift remains to be explored.

A visual representation of the shift from presurvey cluster membership to postsurvey cluster membership is shown in Figure 5. Line weight represents the number of students transitioning from a presurvey to a postsurvey cluster. Postsurvey clusters have been labeled to illustrate their separation from one another.

Figure 5.

Visual representation of student trajectories from presurvey cluster membership to postsurvey cluster membership.

Line weight represents number of students transitioning cluster membership.

5.8 Cluster membership and indicators of student success

Table 13 shows progress toward degree (operationalized by credits at the end of semester 4). Neither membership in presurvey cluster nor membership in postsurvey cluster was indicative of more successful progress toward degree. No significant difference was seen and no trends seen from one cluster to another, from either presurvey cluster membership or postcluster membership.

			Cohorts 1 & 2 aggregate, postsurvey
Cohorts 1 & 2 aggregate, presurvey	cluster		Post High	Post Mid (high)	Post Mid (low)	Post Low
	Pre High	»	81; (61 M, 20 F)	88; (66 M, 22 F)	47; (42 M, 5 F)	36; (32 M, 4 F)
	Pre Middle	»	44; (41 M, 3 F)	276; (187 M, 89 F)	181; (148 M, 33 F)	155; (127 M, 28 F)
	Pre Low	»	6; (3 M, 3 F)	132; (103 M, 29 F)	91; (77 M, 14 F)	189; (169 M, 20 F)

Table 13.

Number of students shifting from each presurvey cluster to each postsurvey cluster (cohorts 1 and 2 aggregate data).

Number of male and female students indicated in parentheses.

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

Table 14 shows retention to the end of the second year. In this case, no significant difference in indicated by presurvey cluster membership; however, students in the “upper” or “middle (high)” cluster were significantly more likely to remain in engineering with 93.6% retention vs. students in the “middle (low)” or “lower” postsurvey cluster, with 87.6% retention (z = 3.67). This trend is visible regardless of student presurvey cluster. While this data set is biased in that overall first year retention was very high compared to the overall student population in engineering, the emergence of a significant difference in second year retention based on postsurvey cluster membership indicates the potential for postsurvey cluster membership as an indicator of retention.

		Cohort 1 & 2 aggregate data, postsurvey
Population	Cluster	Post High	Post Mid (high)	Post Mid (low)	Post Low
Cohort 1	Pre High	61.1	60.2	60.5	58.2	60.3
	Pre Middle	57.8	60.7	60.1	59.3	60.0
	Pre Low	63.4*	62.3	57.5	60.6	60.5
		60.1	61.0	59.4	59.9

Table 14.

Progress toward degree as measured by credits at the end of year 2.

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

* indicates very small sample size, n = 6.

Grade point averages at the end of the first and second year are shown in Tables 15 and 16 respectively. Neither GPA shows any significant difference based on presurvey cluster membership, when taken in aggregate or taken within individual postsurvey cluster memberships. In other words, there is no evidence that presurvey cluster memberships in indicative of improved GPA after one or two years.

		Cohort 1 & 2 aggregate data, postsurvey
Population	Cluster	Post High	Post Mid (high)	Post Mid (low)	Post Low
Cohort 1	Pre High	97.5%	94.2%	87.2%	85.3%	90.4%
	Pre Middle	88.6%	91.2%	87.7%	89.0%	89.5%
	Pre Low	66.7%*	98.5%	85.6%	87.8%	90.4%
		93.1%	93.7%	87.0%	88.0%
		93.6%		87.6%

Table 15.

Retention: registration for fourth semester.

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

* indicates very small sample size, n = 6.

		Cohort 1 & 2 aggregate data, postsurvey
Population	Cluster	Post high	Post mid (high)	Post mid (low)	Post low
Cohort 1	Pre High	3.07	3.11	2.80	2.69	2.98
	Pre Middle	3.14	3.03	2.90	2.90	2.97
	Pre Low	3.15*	3.18	2.91	2.87	2.98
		3.10	3.08	2.89	2.86
		3.09		2.87

Table 16.

End of first year GPAs (4.0 scale).

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

* indicates very small sample size, n = 6.

Postsurvey cluster membership does appear to be indicative of GPA to an extent. Improvement in GPA is seen as students progress from the “lower” to the “upper” postsurvey cluster membership when data are taken in aggregate, and within individual presurvey clusters, for both one year and two year GPAs. Using postsurvey cluster membership, a significant difference and small (but near moderate) effect size was found when students in the “upper” and “middle (high)” cluster were combined and compared to students in the “middle (low)” and “lower” clusters: one year GPAs were 3.09 and 2.87 respectively (p < 0.001, d = 0.334); two year GPAs were 3.00 and 2.79 respectively (p < 0.001, d = 0.286). Therefore, it appears that postsurvey cluster membership is indicative of improved student success while presurvey cluster membership is not indicative of improved student success as operationalized by GPA.

Examination of the postsurvey clusters shows that one construct, Major Decision, distinguishes the “upper” and “middle (high)” clusters from the “middle (low)” and “lower” clusters, which is where significant differences emerge. Further examination of differences in indicators based on this construct show that a statistically significant difference is found in two-year retention rate, end of first year GPA and end of second year GPA based on Z score of Major Decision as a single construct (Table 17). The effect size of the difference is small for GPAs from year one (d = 0.267) and year two (d = 0.190). No difference was found in student progress toward degree (Table 18).

		Cohort 1 & 2 aggregate data, postsurvey
Population	Cluster	Post high	Post mid (high)	Post mid (low)	Post low
Cohort 1	Pre High	3.07	2.93	2.73	2.52	2.88
	Pre Middle	3.04	2.92	2.80	2.80	2.87
	Pre Low	3.14*	3.15	2.84	2.82	2.93
		3.06	2.99	2.81	2.78
		3.00		2.79

Table 17.

End of second year GPA (4.0 scale).

Mid (high) indicates a cluster about the average with a higher value of Major Decision. Mid (low) indicates a cluster about the average with a lower value of Major Decision.

* indicates very small sample size, n = 6.

	Progress toward degree	2-year retention*	GPA: 1 year*	GPA: 2 year*
Zmaj > 0	60.7 credits	93.1%	3.06	2.96
Zmaj < 0	59.7 credits	86.0%	2.89	2.82
significance, effect size	p = 0.097; d = 0.09	z = 4.23	p < 0.001; d = 0.267	p = 0.097; d = 0.19

Table 18.

Student success indicators based on postsurvey Major Decision construct.

* indicates a statistically significant difference.

6. Conclusion

The Student Attitudinal Success Instrument I, an instrument to assess nine affective / attitudinal characteristics (and their associated subscales) of incoming students prior to the beginning of their program of study, was evaluated using data collected from large cohorts of incoming engineering students.

The SASI-I is shown to be a psychometrically sound instrument for the population of first-year engineering students at a large institution in the Midwest (United States). Internal consistency of scale scores was investigated. Factor analysis was used to establish and verify the structure of the factors and subfactors. Cronbach’s coefficient alpha values for all scales exceed 0.8 (with two exceptions, 0.75 and 0.79); confirmatory factor analysis results verify the theoretical factor structure of each. McDermott’s three-stage cluster analysis was used to define the normative taxonomies of three years of student data. Cluster analysis results in a stable, repeatable 3-cluster solution over multiple years: clusters expected to be highly similar were shown to have values of Cattell’s coefficient > 0.94, providing evidence of stability.

This instrument is a tool to collect data prior to beginning classes in the first year of engineering, thus, it can provide valuable input to any model predicting retention past the first year (when most attrition in engineering occurs). In addition, this tool provides information on characteristics for which intervention methods may be developed, thus increasing likelihood of student success. Unlike other assessment instruments which rely on data collected during or after the first year or data for which a school may not have an influence, this tool provides necessary input for the creation and adoption of first-year programs at the earliest possible time.

Inputs to model(s) to be developed include these affective / attitudinal constructs in addition to cognitive data; such models will allow for guidance for individual students or small groups who may particularly benefit from specific intervention programs [6, 43, 44, 45]. Cluster membership offers a potential model input that has previously not been presented in the literature within engineering. Additionally, students who may not experience a benefit from these interventions may be able to opt out of some first-year programs, thus increasing the value of the course / program content they will experience during their first year.

Finally, while this instrument is effective, additional affective / attitudinal characteristics which could prove to be predictive of retention have been incorporated into the SASI-2 [4]. Ideally, the size of the existing instrument could be reduced to allow for inclusion of additional constructs without increasing the number of items. Additional affective / attitudinal constructs to be investigated and eventually proposed for inclusion should include only those constructs for which first-year intervention programs can have an effect.

Acknowledgments

Many people contributed to this effort. The original work is based on a dissertation by the author, with exceptional contributions from Dr. P.K. Imbrie and Dr. Teri Reed (University of Cincinnati), Dr. Alice Pawley and Dr. David Radcliffe (Purdue University), and Dr. Joe J. J. Lin.