March 8, 2016

Strengthening Information on Community Engagement Project:

Follow Up Data on the 2015 Annual Survey from National Campus Compact

Project Coordinator: Anne Weiss, Director of Assessment, Indiana Campus Compact, 317-274-5512, haweiss {at} iupui(.)edu


We are imperfect beings who make imperfect tools. As information seekers, our work is to take reasonable steps toward creating and seeking improvement, so that the data collected from our measurement tools are reliable, accurate, meaningful, and useful. What is known about the hard work of community engagement programs, initiatives, and pedagogies in higher education is due, in large part, to the efforts of National Campus Compact’s Annual Survey (i.e. “the tool”).

The fact is, a lot is still not known about this hard work because “the tool” is not completed by nearly two thirds of the 1,100 higher education institutions which make up the National Campus Compact coalition. Unpacking the (important) differences between those who did and those who did/could/would not complete the Annual Survey in 2015 will provide valuable data and information regarding the quality of representation tied to the Annual Survey.

As it is proposed here, this follow-up project is not intended, nor able, to assess the measurement quality (e.g., construct validity, measurement error), but rather can and will focus on the quality of representation across those who did and did not complete the Annual Survey in 2015. In this follow-up project we seek answers to questions such as:

• What are the important differences between those who did and those who did not complete the Annual Survey in 2015?
• What are the important differences within the group of those who did not complete the Annual Survey in 2015?
o Can we better understand issues surrounding contactability, inability to participate, refusals, and interest in the topic, which may impact ability or willingness to complete the Annual Survey in 2015?
• How strong (i.e., reliable, accurate) is the data provided from those who did complete the Annual Survey in 2015?
• What did it take to complete the Annual Survey in 2015?
o How many people?
o What definitions/understanding of terms?
o What support, professional development or incentives?

Purpose and Goal(s)

The purpose of this project is to conduct two different follow-up questionnaires in order to better understand the quality (or error properties) surrounding the National Campus Compact (nCC) Annual Survey.

The goals of collecting and analyzing data regarding the quality of representation are:
• Discover differences between nonrespondents and respondents;
• Unpack assumptions about respondents that are inherent in the survey instrument (e.g., confidence and ability to accurately track community engaged activities);
• Aggregate information about what support or resources are needed by non-respondents, so that state Compacts can support their members in being able, willing, and ready to complete the Annual Survey in the future;
• Provide suggestions regarding quality of representation, timing, and recruitment for future Annual Surveys to the nCC Annual Survey Committee.

Population & Sampling

It is understood that the intended population of the Annual Survey is all Campus Compact partner institutions. For this project, however, in order to keep the project manageable, Indiana Campus Compact approached a select, few State Directors to garner their support and participation in this follow-up project. Therefore, the population for this questionnaire can be defined as all institutions who are members of a participating state compact. Those state directors who have signed on to support this project are, in alphabetical order:

• Alexis Bucknam, Utah Campus Compact
• Matt Farley, Connecticut Campus Compact and Interim-Director of Rhode Island Campus Compact
• Leslie Gavin, North Carolina Campus Compact
• Elaine Ikeda, California Campus Compact
• J.R. Jamison, Indiana Campus Compact
• Gavin Luter, Wisconsin Campus Compact
• Josh Todd, Oregon Campus Compact

Sampling and Sampling Frame

State Directors were approached based on personal conversations with the Project Coordinator or because their state has strong involvement from members in the newly formed Best Practices in Assessing Community Engagement (BPACE) group, spearheaded by the Project Coordinator on behalf of Indiana Campus Compact. The BPACE group is a company of Compact staff and faculty who participate in bi-monthly, facilitated, online learning opportunities, in order to improve their practices and form a community of support surrounding the assessment of outcomes related to community engaged activities in our higher education institutions.

Across these states the sampling frames that will be in place for this project are:
• Respondents to the 2015 Annual Survey: those who submitted a completed survey.
• Nonrespondents to the 2015 Annual Survey: Community Service Directors (CSDs) who were invited to the survey AND whose institution did not complete the survey.

Collaboration with National Campus Compact

This project has been reviewed by Dr. Matt Hartley, Chair of National Campus Compact’s Annual Survey Committee, Dr. Andrew Seligsohn, President and Maggie Grove, Vice President for Strategy and Operations of National Campus Compact. With their assistance we have created a close working relationship to inform the format of the project and how useful the data will be to make decisions around supporting members’ efforts to track and report on community engagement activities.

Design and Methods

More information regarding the plans, devices, and procedures inherent in this project.
o 3/8: Questionnaire Open
o 4/8: Questionnaires CLOSE
o Project Coordinator compiles results and constructs drafts of report(s).
• MAY:
o Final reports completed and shared with appropriate stakeholders.
Instruments & Platform
The questionnaires are administered through the Survey Monkey account of Indiana Campus Compact.
Reliable data can only be obtained with sound and accurate processes, which is why many reputable institutions (U.S. Census, National Survey on Student Engagement, etc.) conduct follow-up projects. These follow-up projects typical focus on obviating and measuring nonresponse bias. Given the nature of Campus Compact’s Annual Survey (which is more census than survey given its nature and goal to collect data from all partner institutions, versus a random sample) statistical means cannot be used to calculate the nonresponse bias or error. Therefore, it is necessary that- in order to better unpack the quality of representation- exploratory, non-statistical methods be taken to understand differences between those who did and did not complete the Annual Survey in 2015. Unpacking these differences can lead to improvements in our future efforts to:
• Provide support- through workshops or other professional development opportunities- to partner institutions so that they can complete the Annual Survey;
• Invite and/or incentivize the best (i.e., correct) contact(s) from each institution to complete the Annual Survey;
• Nurture current or future relationships between community engagement professionals and institutional research units/personnel;
• Improve the data collection tool; and,
• Tell an influential and powerful story of community engagement in higher education.
Questions related to this project should be sent to the Project Coordinator, Anne Weiss, Director of Assessment, Indiana Campus Compact, 317.274-5512 or haweiss {at} iupui(.)edu

Leave a Reply

Your email address will not be published. Required fields are marked *