5/18/2015 What is the Impact of Smartphone Optimization on Long Surveys? Introduction & Purpose Shimon Sarraf Jennifer Brooks James Cole Xiaolin Wang Widespread adoption of mobile technologies has dramatically impacted the landscape for survey researchers (Buskirk & Andrus, 2012), and those focusing on college student populations are no exception. National Survey of Student Engagement Indiana University Bloomington AAPOR 70th Annual Conference AAPOR 70 A lC f Optimizing surveys for smartphones is of interest to many but ideal formats are still being developed. May, 2015 This study investigated the impact that one smartphone optimization approach had on a long college student survey. National Survey of Student Engagement NSSE aims to understand the curricular and co‐curricular engagement of first‐year and senior college students using over 100 survey items. NSSE Smartphone Respondents 100% Smartphone Users (%) Proportional Increase 63% Since 2000, ~ 5 million Si 2000 5 illi students from about 1,500 US and Canadian institutions participated. 50% 38% 27% Formatted for “computer” though increasing numbers use smartphones to complete. 18% 13% 8% 4% 2011 2012 2013 2014 2015 Research Questions Are there differences in respondent characteristics by smartphone optimization status. How does optimization impact: a) early abandonment, b) completion, c) item nonresponse, d) duration, e) straight‐lining, f) subjective evaluations, and g) measurement invariance for scales? Administration Year 1 5/18/2015 Study Details NSSE Desktop View • NSSE 2015 winter/spring administration • 10 US colleges/universities • Sample: 38,245 first‐year & senior students • Sample divided equally by smartphone optimization availability • 7,735 respondents; 7,347 included in study Smartphone View Optimized – Vertical Position Unoptimized – Vertical Position Results 2 5/18/2015 Respondent Characteristics Early Abandonment Optimized group less likely to abandon the survey upon viewing the very first page of survey items. • Optimized respondents looked very similar to unoptimized and desktop groups: 26% 22% – Gender, Age, Race/Ethnicity, Parental education, Cumulative grades, Part‐time enrollment, Academic major 12% • Statistically significant differences found but not very large 5% Optimized 10% 4% Unoptimized Desktop Optimized First‐Year Students Missing Data 4 3 50% About 18% decrease in duration compared to unoptimized group—even lower than desktop. 15.0 FY Unoptimized FY Desktop 40% Desktop Seniors Duration Optimization appears to reduce missing data though variation exists between first‐year and senior populations. FY Optimized Unoptimized 12.2 2 13.0 14.6 12.9 12.2 SR Optimized SR Unoptimized SR Desktop 30% 20% 10% Optimized Unoptimized First‐Year Students Desktop Optimized Unoptimized Desktop Seniors 106 97 103 94 100 91 88 85 82 79 76 73 70 67 64 61 58 55 52 49 46 43 40 37 34 31 28 25 22 19 16 7 13 4 10 1 0% 3 5/18/2015 Straight‐lining Subjective Evaluations Optimized straight‐lined less than unoptimized group Page 1 scales (out of 6 scales) Page 2 scales (out of 5 scales) Page 3 scales (out of 3 scales) 1.8 1.4 1.7 1.9 Ease of use: "Very easy" 1.9 1.6 13 1.3 1.1 1.1 58% 55% 41% 1.1 Visual Design: "Excellent" 61% 59% 13 1.3 1.0 Optimization betters ease of use and visual design. 52% 46% 34% 40% 35% 37% 27% 0.5 0.3 0.3 Optimized Unoptimized First‐Year Students Desktop 0.3 0.4 Optimized Unoptimized 0.3 Desktop Seniors Optimized Unoptimized Desktop Optimized First‐Year Students Unoptimized Desktop Seniors Measurement Invariance Conclusions Across the three groups, all first‐year and senior scales met scalar invariance criteria, except for Learning Strategies • Optimization can improve data quality even for a long survey, while also maintaining scale properties. • Smartphone optimized respondent data quality rivals that of desktop respondents. • Some Some measures indicate differences between younger measures indicate differences between younger and older smartphone respondents in the sample. What does this mean for ongoing optimization efforts? • College student survey developers should focus on optimizaztion as smartphone usage continues to increase. 4 5/18/2015 Thank you! Copy of this and past presentations can be found at: nsse.iub.edu/html/publications_presentations.cfm Additional NSSE information can be found at: Additional NSSE information can be found at: nsse.indiana.edu Feel free to contact us with any questions regarding this study or NSSE. ssarraf@, brooksjl@, and [email protected] 5
© Copyright 2024