Evaluation

Evaluation Reports:

At the close of Year 2 of the CCC SMHP (June 2014), PIRE developed a Final Evaluation Report detailing the Program’s resources, services, and outreach efforts.

Capacity Survey of Campus Based Mental Health Services

The California Community Colleges Student Mental Health Program (CCC SMHP) Capacity Survey of Campus Based Mental Health Services was implemented to track change in capacity for MH services across all California Community College campuses. The baseline survey had been conducted previously (in Spring 2013). The Fall 2013 capacity report included descriptive results for the college campuses that responded to the second wave of our survey. A final survey was concluded in Fall 2014.

HSACCC-ACHA-NCHA Consortium Project: CCC-NCHA 2013 Data

A nationally recognized research survey, the ACHA-NCHA contains 300 questions that includes multi-topic areas relevant to a student's health and wellness, and includes questions regarding alcohol, tobacco, and other drug use, sexual health, weight, nutrition and exercise, mental health, personal safety, and violence. The ACHA-NCHA also ties mental health measures with measures of academic success, such as academic performance and student retention. In 2013, the HSACCC facilitated a statewide consortium of California community colleges to administer the American College Health Association National College Health Assessment (ACHA-NCHA II). With, 17,231 respondents, it is the largest study of community college students in the nation. Preliminary reports were presented at the 2014 HSACCC Annual Conference. Results from prior years are available at www.hsaccc.org/mmental_health.pdf.

CalMHSA Student Mental Health Campus-Wide Survey 2013 Summary Report

As part of the California Mental Health Services Authority (CalMHSA) Statewide Evaluation of the Prevention and Early Intervention (PEI) Initiatives, RAND conducted campus-wide online surveys of California college and university faculty, staff, and students during the spring and fall of 2013. The purpose of these surveys was to understand (1) experiences and attitudes related to student mental health, (2) perceptions of how campuses are serving students' mental health needs, and (3) perceptions of the overall campus climate toward student mental health and well-being. UC and CSU invited all of their campuses to participate. CCCs invited a subset (30 campus-based grantees [CBGs] with CalMHSA funding and 30 campuses without campus-based grants) of their campuses to participate.

CCC SMHP Evaluation Overview of Phase I Activities (2012-2014):


All evaluation work for the California Community Colleges Student Mental Health Program (CCC SMHP) is being done by the Pacific Institute for Research and Evaluation (PIRE). They are one of the nation's preeminent independent, nonprofit organizations focusing on individual and social problems associated with the use of alcohol and other drugs. PIRE is dedicated to merging scientific knowledge and proven practice to create solutions that improve the health, safety, and well-being of individuals, communities, nations and the world. The Institute has a significant national presence in the area of prevention, with funded research projects at its ten research centers located around the country.

PIRE scientists and practitioners continually focus on the design and implementation of complex program evaluation strategies and the conduct of research related to health and social issues, including criminal justice. PIRE provides training and technical assistance in many health-related areas to states and communities that are attempting to mitigate the effects of alcohol and other drug abuse.

In all its work, PIRE maintains a commitment to the application of research findings in solution of real-world problems, achieved, in part, by translating scientific knowledge into practical guidance for policymakers and practitioners.

The PIRE evaluation team is collaborating with the CalMHSA statewide evaluation team led by the RAND Corporation (RAND). The CCC SMHP is part of the higher education component of the statewide prevention and early intervention evaluation. This partnership is effective in leveraging resources and data sharing to enhance the capacity of both evaluations.

The focus of the CCC SMHP is to build mental health capacity across the CCC system. PIRE's evaluation is designed to develop a baseline measurement of mental health services across the CCC system (re: Capacity Survey of Mental Health Services) and monitor change over the time of the project. In addition, we will measure the overall climate of student mental health services (re: CalMHSA Higher/Education Student Survey) and the faculty / staff's experience and attitudes related to student mental health issues (re: CalMHSA Higher Education Faculty/Staff Survey).

As the CCC SMHP evaluator, PIRE has the responsibility to design and implement a comprehensive and integrated evaluation of CCC mental health services capacity with a major focus on the three key components of the CCC SMHP.

  • Statewide training and technical assistance (TTA) services that are provided to all CCC campuses by the Center for Applied Research Solutions (CARS). CARS provides training and technical assistance services to support the implementation of campus‐based grants; provides regional training conferences on priority student mental health topics; collaborates and coordinates with community partners including county mental health; and develops tools and resources that can be used throughout the state in support of student mental health, including online resources.
  • Suicide Prevention for Faculty / Staff training: Kognito Interactive provides suicide prevention training (SPT) for all interested faculty and staff using evidence‐based programs that have been developed specifically for use by colleges and universities. Campus Based Grants (CBGs) will be required to utilize the services offered by the SPT contractor, but they may choose to augment these efforts with additional services. In addition to faculty/staff training, SPT services have expanded to include students and veterans. SPT services are offered to all CCC campuses that request the online training.
  • Campus Based Grants: The cross-site evaluation focuses on the expansion and enhancement of CBGs' mental health services capacity and the integration of TTA and SPT to achieve this objective.

As can be seen in figure 1, through training and technical assistance provided by the SPT and the TTA initiatives, CBGs and non-CBG campuses will have opportunities to expand their mental health capacity and to improve student mental health outcomes.

As the CalMHSA statewide evaluator, the RAND team has the responsibility to design and implement a comprehensive and integrated statewide evaluation plan for the CalMHSA Suicide Prevention, Stigma and Discrimination Reduction, and Student Mental Health (SMH) Statewide Prevention and Early Intervention (PEI) Initiatives. Statewide evaluation aims include the following:

  • Evaluate PEI program partners' progress toward improving student mental health;
  • Assess the activities implemented and resources created by PEI program partners;
  • Evaluate program outcomes as appropriate. Examples include:
    • Targeted program capacities and their reach (e.g., provision of services, social marketing, workforce training);
    • Short-term outcomes (e.g., attitudes and knowledge about mental illness, behavior toward people with mental illness); and
    • Longer term outcomes (e.g., reduced suicide, reduced discrimination, improved student performance).

Data Collection Components

There are three major data collection components for the CCC SMHP evaluation:

Training Surveys

  • Kognito Online Training Survey: Pre, post, and 90-day follow-up online surveys completed by participants of mental health "Avatar" trainings for faculty/staff, students and veterans.
  • Training and Technical Assistance (TTA) Survey: Evaluation survey completed online or on paper by participants of TTA services (CBGs and non-CBGs.)
  • CCC Campus Training Survey: Post survey completed by participants in all CalMHSA-supported SMH trainings offered at CCC campuses (except Kognito or ASIST trainings) via cell phone or website.
  • Regional Strategizing Forum (RSF) Survey and materials: Post (paper) survey completed by participants of RSFs. CBGs administer surveys, document RSF logistics and participant information, and submit completed forms to the Foundation for California Community Colleges (FCCC) for data entry.

Pre/Post Comparison Surveys

  • Capacity Survey of Mental Health Services: Survey designed to measure the mental health capacity for referral and services across the CCC campuses completed by one key informant per campus in spring 2013 and fall and spring 2014.
  • CalMHSA Higher Education Student Survey: Online survey administered to students at all CBG campuses and a random selection of non-CBG campuses to learn about their experiences and attitudes related to mental health and well-being, and perceptions of overall campus climate toward supporting student mental health.
  • CalMHSA Higher Education Faculty/Staff Survey: Online survey administered to faculty and staff at all CBG campuses and a random selection of non-CBG campuses to learn about faculty and staff's experiences and attitudes related to student mental health and well-being, and perceptions of overall campus climate toward supporting student mental health.

Regular Reporting (CBGs only)

  • Monthly Progress Report and invoices sent directly to the FCCC;
  • Quarterly Report capturing capacity building and activities conducted by the CBGs that are not captured in the Monthly Progress Report (to avoid unnecessary duplication.) The Quarterly Report is an online report divided into modules to allow for reporting activities as they take place.

CCC SMHP Evaluation Contact Information

Richard P. McGaffigan
Program Director
Prevention Research Center
6386 Alvarado Court, Suite 201
San Diego, CA 92120
Office: (619) 583-5678 ext. 2489
Home Office: (760) 448-5275
Cell: (619) 977-7245
Fax: (619) 287-2040
rmcgaffigan@prev.org

Robert F. Saltz, Ph.D.
Senior Scientist
Prevention Research Center
1995 University Avenue, Suite 450
Berkeley, CA 94704
Office: (510) 883-5733
Fax: (510) 644-0594
saltz@prevention.org

The Center for Applied Research Solutions (CARS) is responsible for the content of the page, including but not limited to:
Content accuracy, initial vetting of external links, photographs and graphic images (copyright), files, program, and downloadables, libel and defamation.
For questions or comments concerning this website, please contact CARS at (855) 304-1647 or email kdecelle@cars-rp.org.