Mental Health Treatment Study Final Report July 2011 Prepared for:
Mental Health Treatment Study Final Report July 2011 Prepared for: Social Security Administration 7111 Security Boulevard Baltimore, MD 21244 Contract: SS00-05-60072 Prepared by: Westat 1600 Research Boulevard Rockville, Maryland Investigators: William D. Frey, Ph.D. Robert E. Drake, M.D., Ph.D. Gary R. Bond, Ph.D. Alexander L. Miller, M.D. Howard H. Goldman, M.D., Ph.D. David S. Salkever, Ph.D. Steven Holsenbeck, M.D. Authors and Contributors Listed below are the authors and contributors of the Final Report. Authors William D. Frey, Ph.D. (Co-Principal Investigator) Westat Mustafa Karakus, Ph.D. Westat Robert E. Drake, M.D., Ph.D. (Co-Principal Investigator) Dartmouth Medical School Roline Milfort, Ph.D. Westat Gary R. Bond, Ph.D. Dartmouth Medical School Jarnee Riley, M.S. Westat Alexander L. Miller, M.D. University of Texas Health Science Center at San Antonio Cheryl Reidy, B.S.N., M.Ed. Westat Howard H. Goldman, M.D., Ph.D. University of Maryland School of Medicine David S. Salkever, Ph.D. University of Maryland Baltimore County Julie Bollmer, Ph.D. Westat Megan Collins, M.B.A. Westat Contributors Susan T. Azrin, Ph.D. National Institute of Mental Health (formerly at Westat) David Judkins, M.A. Westat Deborah Becker, M.Ed., C.R.C. Dartmouth Medical School Hilary Kruger, M.A. Westat James Bethel, Ph.D. Westat Janice Machado, M.B.A. Westat Michael Cohen, M.A., C.A.G.S. New Hampshire Alliance on Mental Illness Troy Moore, Pharm.D., M.S.Pharm., B.C.P.P. South Texas Veterans Health Care System 119 David Garner Westat Brent Gibbons, M.P.P. University of Maryland Baltimore County Rene Gonin, Ph.D. Westat Linda Gowen, B.S. Westat Sigurd Hermansen, M.A. Westat Sheila O’Neill, M.S.W., L.C.S.W. Thresholds Jon Ratner, Ph.D. Westat Sandy Reese, B.A. LifeWorks Northwest Carina Tornow, M.A. Westat Acknowledgements Successful completion of a study as large and complex as the Mental Health Treatment Study (MHTS) requires extraordinary efforts from many individuals beyond those directly involved in the day-to-day operations. The MHTS needed people who believed in it, supported it, and worked behind the scenes to ensure its success. The study had many such individuals. The investigators appreciate the efforts of these many people who contributed to the study’s success. We are particularly appreciative of the support of the Social Security Administration, especially Thomas Hale, Ph.D., the current Project Officer; Susan Kalasunas, the original Project Officer; and Soosan Shahrokh, Ph.D., the Alternate Project Officer. Each one worked extremely hard in many ways to support the study. Whether it required getting last minute approvals, following through with individual beneficiary concerns with care, or asking probing questions to be sure that the research team understood the nuances of the Social Security Disability Insurance (SSDI) program, Tom, Susan, and Soosan were always ready to step in and provide guidance to facilitate seamless study operations. We also wish to thank the directors of the 23 study sites, their boards, and their staff whose efforts and commitments to support the study and provide the best services possible to SSDI beneficiaries—not their typical clientele—were invaluable. Research sites often do not fully recover the actual costs of participating in studies such as the MHTS. For this reason alone, it takes an extremely dedicated director to commit center staff and resources to participate in large research studies of this nature knowing that there is an unrecoverable cost. The MHTS was no different. The directors and staff of the community mental health centers (listed at the end of this report) deserve a large share of the credit for helping to make the MHTS a successful study. We greatly appreciate the efforts of ValueOptions/Colorado Health Networks and Hirsch Financial Services for their support of the study through provision of health benefits to study participants. This important feature of the intervention literally defined the term “access” to services because on-time payments to providers are keys to continued services. Dedicated staff at these organizations made certain that participant health care coverage met the standards set for the study. We also thank the many computer systems, statistical, operations, and data management staff at Westat who worked tirelessly to develop and maintain the study’s infrastructure which facilitated the successful coordination of daily activities across 23 locations. Finally, we are most grateful to the 2,238 SSDI beneficiaries who entrusted us with their care by enrolling in the MHTS. No small amount of courage is required to commit to 24 months of participation in a research effort—knowing that assignment to the treatment group is not a given. Case-control studies similar to the MHTS usually experience large numbers of complaints and exodus of participants assigned to the control group. It was just the opposite for the MHTS. Participants in both the treatment and control groups often commented on how grateful they were to be a part of the study, and how much they appreciated that the Social Security Administration was studying ways to improve services to beneficiaries in the SSDI program. Table of Contents Chapter Page Executive Summary .............................................................................................................. EX-1 Chapter 1: Study Overview Background and Understanding ....................................................................... Design of the Mental Health Treatment Study .............................................. Outcome Measures............................................................................................. MHTS Investigators ........................................................................................... 1-5 1-12 1-14 1-15 Chapter 2: Study Design Key Scientific Features....................................................................................... Study Site Selection ............................................................................................ Sampling Methodology ...................................................................................... Sample Size .......................................................................................................... Overview of the Enrollment and Randomization Process........................... Intervention Benefits and Services................................................................... Control Condition .............................................................................................. Study Site Staffing ............................................................................................... Fidelity Measures ................................................................................................ Outcome Measures............................................................................................. Data Collection Procedures .............................................................................. Interview Completion Rates ............................................................................. Data Preparation and Adjustments .................................................................. Statistical Procedures.......................................................................................... Data Limitations ................................................................................................. 2-1 2-3 2-6 2-9 2-10 2-12 2-15 2-16 2-17 2-18 2-21 2-23 2-24 2-28 2-30 Chapter 3: Enrollment and Participation Are There Differences Between SSDI Beneficiaries Who Enrolled and Those Who Did Not Enroll in the MHTS? .............. Characteristics of the Beneficiaries Who Enrolled in the Study .................. Why Did Eligible Beneficiaries Refuse to Enroll? ......................................... Discussion ............................................................................................................ 3-2 3-10 3-17 3-18 Chapter 4: Outcomes Employment Outcomes .................................................................................... Earnings and Income ......................................................................................... Health and Quality of Life ................................................................................ Discussion ............................................................................................................ 4-3 4-28 4-45 4-53 Table of Contents (continued) Chapter Page Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services Background.......................................................................................................... Methods ............................................................................................................... Results ................................................................................................................. MHTS Site-Level Fidelity-Outcome Analysis ................................................ Discussion ............................................................................................................ 5-2 5-4 5-11 5-21 5-22 Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management Background.......................................................................................................... Methods ............................................................................................................... Results ................................................................................................................. Discussion ............................................................................................................ 6-1 6-6 6-11 6-21 Chapter 7: Health Care and Supported Employment Financing Health Care and SE Financing Principles ....................................................... Health Care Coverage and Financing Decisions............................................ Implementation of the HBP ............................................................................. SE Financing ....................................................................................................... HBP Expenditures.............................................................................................. Discussion ............................................................................................................ 7-1 7-3 7-8 7-16 7-22 7-26 Chapter 8: Utilization of Services Dependent Variables .......................................................................................... Covariates............................................................................................................. Estimation Models.............................................................................................. Results ................................................................................................................. Discussion ............................................................................................................ 8-2 8-4 8-4 8-5 8-12 Chapter 9: Study Limitations, Key Findings, and Policy Implications Study Limitations ................................................................................................ Key Findings ....................................................................................................... Policy Implications ............................................................................................. 9-2 9-5 9-8 References ................................................................................................................... Glossary of Terms ...................................................................................................... List of Acronyms ........................................................................................................ Acknowledgement of Study Site Staff .................................................................. References-1 Glossary-1 Acronyms-1 Acknowledgement-1 Table of Contents (continued) Table List of Tables Page 1-1 Number of SSDI beneficiaries under age 50 who had a primary diagnosis of Other Mental Disorder: 1996-2009 ........................................... 1-7 2-1 MHTS sample by key stratifications ................................................................ 2-8 2-2 Content domains for the Baseline, Quarterly, and final Followup interviews ............................................................................................................. 2-23 2-3 Interview completion rates by quarter and study group ............................... 2-24 3-1 Results of logistic regressions on MHTS participation (Enrolled = 1) ...... 3-7 3-2 Mean beneficiary characteristics for quintiles based on predicted enrollment probability of potential enrollees in the validation sample (N=7,933) ............................................................................................................ 3-11 3-3 Demographic and program characteristics of enrolled beneficiaries by study condition.................................................................................................... 3-13 3-4 Recent employment history and health status comparisons ........................ 3-14 3-5 Physical health conditions identified in CAPI screener ................................ 3-15 3-6 Total individual income ..................................................................................... 3-17 3-7 Refusal frequencies ............................................................................................. 3-17 4-1 Employment rates by treatment and control group ...................................... 4-3 4-2 Employment rates by treatment and control group, stratified by age, gender, diagnosis, and education ...................................................................... 4-6 4-3 Other employment outcomes for all participants, participants who worked at least one job, and participants who worked at least one competitive job by treatment and control group ........................................... 4-7 Table of Contents (continued) Table 4-4 List of Tables (continued) Page Statistically significant subgroup comparisons for other employment outcomes including all participants, participants who worked at least one job, and participants who worked at least one competitive job by age, gender, diagnosis, and education .............................................................. 4-10 4-5 Classification of worker types by treatment and control group .................. 4-12 4-6 Characteristics of steady workers by treatment and control group ............. 4-13 4-7 Types of occupations held by participants who reported working in a job for each study-month by treatment and control group.......................... 4-15 4-8 Types of occupations held by participants who reported working in a job by diagnosis................................................................................................... 4-16 4-9 Types of occupations held by participants who reported working in a job by education.................................................................................................. 4-17 4-10 Potential predictor variables included in employment-related regression models ............................................................................................... 4-18 4-11 Estimates of marginal effects on obtained employment, steady worker, and number of months employed (for all participants).................. 4-20 4-12 Estimates of marginal effects on obtained employment, steady worker, and number of months employed (for treatment group participants only) ................................................................................................ 4-22 4-13 Reasons for working part-time ......................................................................... 4-23 4-14 Barriers to competitive employment for non-working treatment group participants ............................................................................................... 4-25 4-15 Agreement with statements about Social Security Disability benefits programs by treatment and control group ...................................................... 4-27 4-16 Past month’s earnings (averaged over 8-post baseline interviews), past three months’ earnings (at study exit) and earnings above SGA (averaged over 8-post baseline interviews) by treatment and control group..................................................................................................................... 4-29 Table of Contents (continued) Table List of Tables (continued) Page 4-17 Number of quarters with past month’s earnings above, near, or below SGA by treatment and control group .................................................. 4-31 4-18 Past month’s SSDI and SSI income (averaged over 8 post-baseline interviews) by treatment and control group ................................................... 4-32 4-19 Past month’s income from other public programs (averaged over the 8 post-baseline interviews) by treatment and control group ........................ 4-33 4-20 Past month’s total individual income (averaged over 8 post-baseline interviews) and past month’s total household income (at study exit) by treatment and control group ........................................................................ 4-34 4-21 Past month’s earnings regression slope (averaged over 8 post-baseline interviews) by treatment and control group ................................................... 4-35 4-22 Statistically significant earnings and income subgroup comparisons for age, gender, diagnosis, and education for unconditional mean comparisons, conditional mean comparisons, and percent reporting non-zero amounts comparisons ....................................................................... 4-36 Descriptive statistics for dependent measures included in multivariate analyses of the MHTS impact on earnings and income by treatment and control group ............................................................................................... 4-40 4-24 Estimates of average marginal MHTS effects on earnings........................... 4-44 4-25 Estimates of average marginal MHTS effects on income ............................ 4-45 4-26 Mental and physical health status at baseline and study exit by treatment and control group ............................................................................. 4-46 4-27 Linear regression for mental health status at study exit (for all participants) ......................................................................................................... 4-47 4-28 Linear regression for mental health status at study exit (for treatment group participants only) ..................................................................................... 4-48 4-29 Satisfaction with life at baseline and study exit by treatment and control group ....................................................................................................... 4-48 4-23 Table of Contents (continued) Table List of Tables (continued) Page 4-30 Ordered logistic regression estimates for quality of life (all participants) ......................................................................................................... 4-50 4-31 Ordered logistic regression estimates for quality of life (treatment group participants only) ..................................................................................... 4-50 4-32 Alcohol and drug use at baseline and study exit by treatment and control group ....................................................................................................... 4-52 4-33 Any drug use during the past 30 days at baseline and study exit by treatment and control group ............................................................................. 4-52 5-1 IPS service engagement rates for employed and not employed treatment group participants in the last two reporting periods ................... 5-14 5-2 Percentage of treatment group participants who received IPS service elements while enrolled in the MHTS ............................................................. 5-14 5-3 IPS employment service contact rates of engaged but not employed treatment group participants during the reporting period ............................ 5-16 5-4 Rates of IPS contact with employed treatment group participants by reporting period .................................................................................................. 5-17 5-5 Percentage of treatment group participants who received OBH or related services while enrolled in the MHTS1 ................................................ 5-18 5-6 Frequency and percentage of high and low access sites for other behavioral health services .................................................................................. 5-19 5-7 Frequency and percentage of MHTS study sites that received various overall quality of site-level IPS implementation ratings based on the QMPD Site Reports: 2008 and 2009 ............................................................... 5-21 Frequency and percentage of MHTS study sites with various sitelevel barriers to implementation based on QMPD qualitative assessments .......................................................................................................... 5-21 Site-level fidelity-outcome correlations in the MHTS ................................... 5-22 5-8 5-9 Table of Contents (continued) Table List of Tables (continued) Page 6-1 Co-occurring psychiatric conditions for participants in the treatment condition .............................................................................................................. 6-14 6-2 Frequency of co-occurring psychiatric conditions......................................... 6-14 6-3 Physical health conditions by source (treatment beneficiaries) (N=981) ............................................................................................................... 6-16 6-4 NCC ratings of beneficiary engagement in SMM .......................................... 6-18 6-5 NCC ratings of prescriber engagement by prescriber location.................... 6-18 6-6 Percent of beneficiaries seen by prescribers (at final study visit by beneficiary) grouped by NCC ratings of prescriber engagement and by prescriber location ......................................................................................... 6-19 6-7 SMM implementation (% of participants) ...................................................... 6-20 6-8 Summary of medication-related self-reports in final Followup interview ............................................................................................................... 6-22 7-1 Insurance plans and estimated costs to Westat and participants ................. 7-7 7-2 Number of participants enrolled in a new health plan by the WIP, by year and by type of coverage ............................................................................. 7-13 7-3 Medicare Part B premium changes .................................................................. 7-14 7-4 Distribution of HBP expenditures among all participants and 24month participants.............................................................................................. 7-23 8-1 Weighted mean values and N's for treatment and control groups and total: Overall utilization measures .................................................................... 8-6 8-2 Weighted mean values and N's for treatment and control groups and total: ER and inpatient mental health utilization measures .......................... 8-7 8-3 Estimated average marginal treatment effects ................................................ 8-8 Table of Contents (continued) Table List of Tables (continued) Page 8-4 Estimated average marginal treatment effects for emergency room visits and overnight hospital use for mental health problems ..................... 8-11 8-5 Regression results for average main treatment effects with late dropout dummy and dummy interaction with treatment* ........................... 8-12 Figure List of Figures Page 2-1 Map of MHTS study sites.................................................................................. 2-5 3-1 Logic chart for assigning beneficiaries to enrollee group classifications ....................................................................................................... 3-4 4-1 Monthly employment rates for any job by treatment and control group over the 24-month study period ........................................................... 4-4 4-2 Employment rates for any employment by treatment group and control group and by study site ........................................................................ 4-5 5-1 IPS fidelity for MHTS sites (based on 15-item IPS Fidelity Scale) ............. 5-12 5-2 Site integration of IPS and behavioral treatment (IPS Fidelity Scale item) ...................................................................................................................... 5-13 6-1 Factors influencing prescriber medication decisions..................................... 6-4 6-2 Factors influencing prescriber medication decisions in SMM ..................... 6-4 6-3 Role-based functions in SMM .......................................................................... 6-5 6-4 Concordance of SSA diagnostic group with SCID diagnosis (includes dual diagnosis) ..................................................................................................... 6-12 6-5 Concordance of SSA diagnostic group with SCID diagnosis (excludes dual diagnosis) ..................................................................................................... 6-13 6-6 Frequency of co-morbid physical health conditions for treatment group beneficiaries.............................................................................................. 6-15 Table of Contents (continued) Figure List of Figures (continued) Page 6-7 Frequencies of beneficiaries with one or more physical health conditions (N=981) ............................................................................................ 6-17 6-8 BMI distribution in treatment group by gender............................................. 6-17 7-1 Distribution of total expenditures by type of claim including all participants (n=1,121) (based on date of service) .......................................... 7-24 7-2 Distribution of cumulative expenditures by type of claim including all 24-month participants (n=982) (based on date of service) .......................... 7-25 7-3 Average HBP expenditures per person-per quarter (n=1,121).................... 7-25 7-4 Average HBP expenditures per participant month (n=1,121)..................... 7-26 Appendix List of Appendices Page 2A SF-12 Questionnaire and Scoring Algorithm ................................................. A-2A-1 2B Imputation Procedures ...................................................................................... A-2B-1 3A Variable Definitions for Enrollment and Participation Analyses ................ A-3A-1 3B Descriptive Statistics for Participation Analysis Test and Validation Samples................................................................................................................. A-3B-1 3C Probabilities of Enrollment for Test and Validation Samples ..................... A-3C-1 4A Percent Obtained Employment (Any Job) by Site ........................................ A-4A-1 4B Subgroup Comparisons for Employment Outcomes ................................... A-4B-1 4C Subgroup Comparisons for Earnings and Income Outcomes .................... A-4C-1 4D Regression Estimates for Formal Earnings .................................................... A-4D-1 4E Regression Estimates for Formal and Informal Earnings ............................ A-4E-1 4F Regression Estimates for Income Variables ................................................... A-4F-1 Table of Contents (continued) List of Appendices (continued) Appendix Page 4G Subgroup Comparisons for Health and Quality of Life Outcomes ............ A-4G-1 5A IPS Fidelity Scale ................................................................................................ A-5A-1 6A Listing of Medications and the Corresponding Physical Condition ........... A-6A-1 6B SCID Diagnostic Frequencies .......................................................................... A-6B-1 6C SMM QM Summary ........................................................................................... A-6C-1 6D NCC Survey Results ........................................................................................... A-6D-1 6E NCC Survey Administrator Version Results .................................................. A-6E-1 6F SMM Implementation Findings by Site........................................................... A-6F-1 6G SMM QA Findings by Site ................................................................................ A-6G-1 7A Health Insurance Questionnaire....................................................................... A-7A-1 7B MHTS Monthly Encounter Form .................................................................... A-7B-1 7C Original and New Supported Employment and Related Services Payment Schedules ............................................................................................. A-7C-1 Supplemental Appendix List of Supplemental Appendices Page A Study Implementation ........................................................................................ SA-A1 B CAPI Screener..................................................................................................... SA-B1 C Baseline Questionnaire ...................................................................................... SA-C1 D Treatment Quarterly Questionnaire ............................................................... SA-D1 E Control Quarterly Questionnaire .................................................................... SA-E1 F Final Followup Questionnaire ......................................................................... SA-F1 Note to Reader This report aims to provide enough information to the reader to understand the purpose, design, general procedures, outcomes, and conclusions of the Mental Health Treatment Study. To aid understanding, the reader will find lists of acronyms and terminology used throughout the chapters at the back of this report. It was not possible to include all of the details of the study in one volume. Therefore, a number of documents appear in the Appendices or in the Supplemental Appendices. The Appendices are a valuable complement to this Final Report, and readers may wish to refer to them when reading the report. The Appendices comprise a separate document and include expanded analyses, frequencies, distributions, scales, forms, etc. referenced in the chapters. While the Supplemental Appendices (also a separate document) provide interesting and detailed information about study implementation, they are not required to comprehend this Final Report. The Supplemental Appendices include details of the study implementation process and procedures, and copies of the study questionnaires. Executive Summary The Mental Health Treatment Study (MHTS) provides a test of the hypothesis that access to supported employment (SE) services and systematic medication management (SMM) services, coupled with the removal of some known programmatic disincentives, will enable Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or an affective disorder to return to work. Fielded by the Social Security Administration (SSA) between November 2006 and July 2010, the test included 2,238 SSDI beneficiaries in 23 study sites throughout the United States. Beneficiaries volunteering to participate in the study received a random assignment to either a treatment group or a control group following completion of a comprehensive Baseline interview. Beneficiaries participated for 24 months. The study collected self-report data quarterly on the primary outcome measures of employment (including earnings), health status, and quality of life. These data provided the means to test the hypothesis that access to evidence-based behavioral health treatments and employment supports would result in improved outcomes. Background SSDI is a long-term disability insurance program designed to provide income to people who are no longer able to work because of a disability. Age at the time of onset of disability determines the number of quarters a person must have previously worked in order to qualify for the program. The SSDI program addresses the income support and health care needs of workers who can no longer engage in substantial gainful activity (SGA) due to a medically determinable physical or mental impairment. Beneficiaries accepted into the program receive cash benefits based on their past job earnings and, after a 2-year waiting period, are eligible for Medicare. A high priority at SSA is to support SSDI beneficiaries in their efforts to return to work. In doing so, the agency provides a broad range of programs and supports to facilitate a beneficiary’s pursuit of employment goals. For example, the Ticket to Work program assists beneficiaries to gain access to services needed to get and keep a job. The Trial Work Period (TWP) allows beneficiaries to test their ability to work for up to 9 months over a rolling five-year period without worry of losing cash benefits. The Expedited Reinstatement rule serves as a safety net for those beneficiaries who return to work, get off the program completely, but then find at a later date that they can no longer work. The Extended Period of Eligibility (EPE) begins immediately following the TWP and extends for 36 months. The program allows the beneficiary to resume receiving benefits if earnings fall below SGA Executive Summary EX-1 in a given month. Programs and supports like these assist SSDI beneficiaries to achieve their employment goals. The MHTS is just one in a series of demonstrations at SSA to explore new and improved employment supports to beneficiaries in the SSDI program. Other efforts include the Benefit Offset National Demonstration (BOND) project and the Youth Transition Demonstration (YTD). The MHTS developed out of three pressing realities. First, beneficiaries with mental disorders in the SSDI program, specifically those with psychiatric impairments, have been and continue to be a significant policy concern. The percentage of beneficiaries under the age of 50 who receive SSDI benefits due to a psychiatric impairment has steadily increased over the past 10 years and continues to grow. Second, evidence-based employment supports exist with demonstrated success in helping people with mental illness return to work. There is also a large body of evidence demonstrating that antipsychotic medications, mood stabilizers, and antidepressants provide symptom relief for persons with schizophrenia, bipolar disorder, or depression. There is virtually universal use of medication to treat these conditions in public and private mental health facilities in the United States. Third, surveys of individuals with severe mental illness consistently indicate that they want to work. Yet they have one of the lowest employment rates of any subpopulation. Taken together, the concerns about supporting beneficiaries in the SSDI program, the promise of evidence-based mental health treatments and employment supports, and lack of understanding how these treatments and supports might work with SSDI beneficiaries with psychiatric impairments, prompted SSA to test the effectiveness of the treatment intervention. The policy concerns discussed above drove the following research questions, which the study was expected to answer: 1. To what extent does delivering appropriate mental health treatment and employment supports lead to better employment, health status, and quality of life among SSDI beneficiaries with schizophrenia or an affective disorder? 2. What programmatic disincentives exist that create barriers for Title II beneficiaries with schizophrenia or an affective disorder to return to work? 3. What specific programmatic changes would support the efforts of SSDI beneficiaries with schizophrenia or an affective disorder to sustain competitive employment? Executive Summary EX-2 Study Design The MHTS was a randomized controlled trial of SSDI beneficiaries with schizophrenia or an affective disorder. This feature involved randomly assigning beneficiaries enrolling into the study to either a treatment group or a control group. The treatment group received a comprehensive package of services and benefits, including evidence-based SE, SMM, behavioral health and related services, and comprehensive insurance to pay for needed services and out-of-pocket expenses. In addition, SSA suspended each beneficiary’s medical Continuing Disability Review (CDR) for a period of 3 years from the date of study enrollment. The control group received a resource manual that listed available local and national services and resources for persons with mental illness, and a nominal payment of $100 for participating in quarterly interviews. The control group was still subject to the medical CDR. The MHTS followed the intent-to-treat (ITT) model so that outcomes for all study participants would be included in the analysis, regardless of the extent to which they participated in the treatment or completed study activities. The ITT approach is most appropriate for answering the study questions within the context of a policy concern about expected outcomes of individuals offered access to services, and not simply expected outcomes of individuals who use services at a particular level of engagement and intensity. The ITT analysis is conservative and works against finding positive effects, but it is the most appropriate design for evaluating the impact of a program that would offer—but not require—engagement in services. Twenty-three study sites located in each region of the U.S., except the Southwest, participated in the study. Two criteria dictated site selection, including: (1) the ability to deliver behavioral health and SE services and (2) documented fidelity to the Individual Placement and Support (IPS) model for SE. All but two study sites were community mental health agencies that provided an array of mental health services. The remaining two sites included a vocational center and a housing center for homeless people with mental illness that also provided vocational services. The vocational center contracted mental health services with the county mental health facility. The homeless program brokered mental health services in the nearby community. SSDI beneficiaries eligible to participate in the MHTS included those with a primary diagnosis of schizophrenia or an affective disorder, between the ages of 18 and 55 (inclusive), and residing within a 30-mile radius of one of the study sites. Beneficiaries excluded from consideration for the study were those living in a nursing home or other custodial institution, had a legal guardian, had a lifethreatening or terminal physical health condition, were already receiving SE services from their Executive Summary EX-3 participating demonstration site within 30 days of recruitment, or were currently working in a competitive job within 6 months of recruitment. Data sources for the MHTS included information collected during the study and data retrieved from external sources. The primary data source used to test the stated hypothesis were self-report measures collected directly from enrolled study participants in the treatment and control groups through computer-assisted interviews. Research staff interviewed each treatment and control group beneficiary at enrollment (i.e., baseline) and then every 3 months following baseline, resulting in seven Quarterly interviews and a final Followup interview over the 24-month period of participation. The nine interviews collected information about the beneficiary’s demographic characteristics, work history and employment, income, health status, alcohol and drug use, health care coverage, health care service utilization, understanding of SSA’s return-to-work programs, and quality of life. Additional information collected on the treatment group only included data related to pre-treatment activities, receipt of SE services, SMM services, and other behavioral health services, as well as expenditures related to service utilization. Staff at each study site included a dedicated Nurse Care Coordinator (NCC) and Research Assistant (RA), funded by the study. The NCC served in dual roles supporting both treatment (particularly the SMM component) and research (completing various reports and data-entry forms). The RA served primarily in a research role and was responsible for recruitment, enrollment, completing the interviews, and other research tasks. In addition to the NCC and RA, each site required an adequate number of SE specialists to serve the treatment group participants. Standard procedures guided study recruitment activities, including introductory mailings, recruitment telephone calls, and Recruitment Information Group (RIG) meetings. Once beneficiaries attended two required RIG meetings, they enrolled in the study by completing a screener (to assess competence to give consent, confirm eligibility, and then provide consent), and completing the Baseline interview. A computer-generated call-in program randomized each beneficiary to either the treatment or control group following completion of the Baseline interview. For beneficiaries randomized to the treatment group, pre-treatment activities included completion of the Axis 1 Disorders section of the Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders, fourth edition (referred to as the SCID), the Brief Assessment of Cognition in Schizophrenia, and a General Medical Exam (GME). Additionally, beneficiaries randomized to the treatment group were assigned treatment providers (e.g., prescriber, SE specialist, therapist, or case manager) as needed. Executive Summary EX-4 Throughout the study, treatment group participants had access to evidence-based mental health treatments and services, as well as integrated mental health services and employment support services. The treatments and services offered to participants included SE, SMM, and other behavioral health (OBH) and related services; regular meetings with the NCC; benefits counseling (where possible); and payments for out-of-pocket expenses for behavioral health care (health insurance coverage, debit card for payment of prescriptions, co-pay reimbursement, etc.). All treatment group participants who remained in the study at 20 months participated in the development of a personal transition plan. The 24th month was the end point for completing the transition plan. The primary goal of the transition plan was to maximize the potential for the participant to maintain all positive employment, behavioral health, and other outcomes by ensuring continued access to care and services after exiting the study. The study design included a range of quality assurance procedures built into the various components of the study to ensure that the treatment group received high quality SE, OBH, and related services. Specifically, three Quality Management Program Directors (QMPDs) worked closely with staff at each of the 23 study sites to ensure participants received evidence-based treatments. In addition, Nurse Care Coordinators (NCCs) completed the SE/OBH and SMM quality management reports which collected data on receipt of services. The QMPDs and SMM experts often consulted directly with site staff to address issues as they arose. Other quality assurance activities ensured appropriate execution and use of study benefits (e.g., examinations of MHTS debit card use and the provision of procedural manuals and periodic staff trainings throughout the course of the study). To ensure protection of study participants, the MHTS included a number of safeguards. First, voluntary withdrawal meant termination of all study benefits for treatment group participants. A standard withdrawal protocol assisted this process and ensured that no harm came to participants electing to withdraw from the study. A standard “administrative drop” process guided staff in removing treatment group participants from the study who did not obtain the required GME or failed it. Finally, event-generated reports of any adverse event or unanticipated problem were submitted to the MHTS Institutional Review Board (IRB) Administrator and SSA as needed. Assessment of the Randomization Procedure. A critical aspect of the participation analyses was an assessment of the success of the randomization procedure in creating two equal groups in the treatment and control arms of the study. The results of tests revealed that the procedure successfully created two equal groups on the key measures at baseline. These tests found no differences between Executive Summary EX-5 the treatment and control groups on the (1) amount of work in the past two years, (2) mental health as measured by the SF-12, (3) physical health as measured by the SF-12, or (4) total individual income in the past month. Findings The following sections address the study findings. Section A presents findings about Enrollment and Participation, including an estimate of how many SSDI beneficiaries might take advantage of an MHTS-like program, and the reasons beneficiaries refused to enroll in the study when offered the opportunity. Section B addresses the Primary Study Outcomes, including employment rate, health status, and quality of life. Section C presents findings associated with the Secondary EmploymentRelated Outcomes. Section D presents findings associated with the Secondary Earnings and Income Outcomes. Section E addresses study implementation of SE and SMM. Section F presents findings from the Health Benefits Plan (HBP). Finally, section G presents findings related to Utilization of Services. A. Enrollment and Participation 1. Based on the sample of potential enrollees, the findings suggest that nearly 14 percent of SSDI beneficiaries with schizophrenia or an affective disorder (who can be reached by telephone or letter) would take advantage of an MHTS-like program. 2. Some of the strongest predictors of enrollment were those related to prior work activity reported in the SSA administrative records, including beneficiaries who reported earnings within the past six months (prior to enrollment), had assigned their Ticket (in the Ticket to Work program), or had a Trial Work Period End Date within the past three years. 3. Beneficiaries enrolling in the study tended to be younger, on the SSDI rolls a shorter period, only on SSDI (as opposed to concurrent SSI beneficiaries), and did not have a Representative Payee. 4. The most commonly recorded single reason for beneficiaries not enrolling in the study was general disinterest (37%). However, nearly 40 percent of all beneficiaries for whom a reason was recorded, indicated work-related or physical health issues as the reason they did not enroll in the study. Executive Summary EX-6 B. Primary Study Outcomes: Employment rate, physical health status, mental health status, and quality of life 5. The 24-month employment rate for the treatment group was 61 percent compared to 40 percent for the control group. The difference between these percentages was significant (p-value < 0.001). 6. Measured at baseline and again at study exit, the treatment group showed a significant improvement over the control group in mental health status (p-value < 0.001). 7. Measured at baseline and again at study exit, both the treatment and control groups showed a slight decline in physical health status. The changes between the groups were not significant (p-value = 0.924). 8. Measured at baseline and again at study exit, the treatment group showed a significant improvement over the control group in quality of life (p-value < 0.001). C. Secondary Outcomes: Employment-related 9. Treatment group participants had significantly better outcomes than did control group participants on total months employed (d = 2.58, p-value < 0.001), consecutive months of employment at study exit (d = 1.43, p-value < 0.001), average weekly earnings at main job (d = $40.54, p-value < 0.001), average hours per week at main job (d = 4.29, pvalue < 0.001), and highest hourly wage (d = $2.56, p-value < 0.001). 10. When the treatment and control participant groups were narrowed to participants who worked at least one job, the findings revealed significant differences between the treatment and control groups on total months employed (d = 0.69, p-value = 0.016), consecutive months of employment at study exit (d = 0.87, p-value = 0.001), and average weekly earnings at main job (d = $4.07, p-value = 0.080). Neither average hours per week at main job (d = 0.76, p-value = 0.173) nor highest hourly wage (d = $0.02, pvalue < 0.443) were significant. 11. When the participant groups were narrowed to participants who worked at least one competitive job, the findings were not significant, with one exception. Only total months employed (d = 0.91, p-value = 0.017) was significant, favoring the treatment group. 12. The investigators classified study participants into three groups: steady workers (those who worked 10 or more months), erratic workers (those who worked between 3 and 9 months), and minimal workers (those who worked less than 3 months including those who did not work at all). A Chi-square test comparing differences in classifications between the treatment and control groups found that the treatment group had a significantly greater number of study participants in the steady worker and erratic worker classifications (and consequently lower numbers in the minimal worker classification) than did the control group (p-value < 0.001). 13. Logistic regression predictions of obtained employment and steady worker status identified five common predictors, including the treatment dummy (ME = 0.226, p- Executive Summary EX-7 value < 0.001; ME = 0.139, p-value < 0.001 respectively). Additional predictors included baseline physical health status (p-values < 0.001, = 0.021 respectively), worked in the last 2 years (p-values < 0.001, < 0.001 respectively), unemployment rate (p-values = 0.017, = 0.002 respectively), and months on rolls (p-values = 0.005, < 0.001 respectively). A zero inflated negative binomial regression on number of months employed identified the same five significant predictors, including the treatment dummy (ME = 2.57, p-value < 0.001), baseline physical health status (p-value < 0.001), worked in the last 2 years (p-value < 0.001), unemployment rate (p-value < 0.001), and months on rolls (p-value < 0.001). This model identified three additional predictors, including ever had a ticket (p-value = 0.031), baseline mental health status (p-value = 0.020), and number of hospital stays at baseline (p-value = 0.016). 14. Logistic regression predictions of obtained employment and steady worker status for the treatment group only identified the same predictors of employment-related outcomes. However, the models revealed that SE engagement (p-value = 0.002, p-value < 0.001 respectively) and receipt of case management services (p-value = 0.004, p-value = 0.004 respectively) were also significant. The zero inflated negative binomial regression prediction of number of months employed also identified SE engagement (p-value < 0.001) and receipt of case management services (p-value = 0.031) as significant predictors of number of months employed. This model identified three additional predictors, including ever had a ticket (p-value = 0.031), baseline mental health status (p-value = 0.020), and number of hospital stays at baseline (p-value = 0.016). 15. Among the five general industry job type categories, service occupations, and sales and office occupations accounted for more than 70 percent of all jobs obtained by study participants. This figure remained steady at baseline, during the study, and at the end of the study. Looking at the overall occupational data, treatment group and control group participants entered service occupations (37%, 37% respectively), and sales and office occupations (36%, 34% respectively) in roughly equal percentages. Neither difference between the treatment and control groups was statistically significant. In roughly equal percentages, both the treatment and control groups entered into management, business, science and arts occupations (13%, 14% respectively), natural resources, construction, and maintenance occupations (3%, 2% respectively), and production, transportation, and material moving occupations (11%, 13% respectively). D. Secondary Outcomes: Earnings and income 16. Analyses of average earnings (past month earnings averaged over eight post-baseline interviews) show that treatment group participants earned significantly more than did the control group participants. The first analysis concerns comparisons between the unconditional means (including all 2,238 study participants) of the treatment and control groups ($148 vs. $97, p-value < 0.001). A second analysis concerns the comparison between conditional means (i.e., including only those participants with nonzero earnings) of the treatment and control groups ($251 vs. $228, p-value < 0.001). The third analysis concerns the percentages of participants in the treatment and control groups with earnings (59% vs. 43%, p-value < 0.001). Executive Summary EX-8 17. Analyses of past three months earnings (from the final Followup interview only) show that treatment group participants earned significantly more than did the control group in two of three analyses. The first analysis concerns comparisons between the unconditional means (all study participants) of the treatment and control groups ($859 vs. $479, p-value < 0.001). The second analysis concerns the comparison between conditional means (i.e., including only those participants with non-zero earnings) of the treatment and control groups ($2,538 vs. $2,739, p-value = 0.607). The third analysis concerns the percentages of participants in the treatment and control groups with earnings (34% vs. 17%, p-value < 0.001). 18. Eight percent of the study participants showed average earnings over the 24-month study period that exceeded the current level of SGA. Beneficiaries in the treatment group did not experience an increase in work that SSA considers SGA when compared to participants in the control group. Neither did participants in the treatment group experience a reduction in benefit payments when compared to participants in the control group. 19. Analyses of past month’s individual income (averaged over eight post-baseline interviews) and past month’s household income (at study exit only) show significant treatment effects. The first analysis shows unconditional means (d = $59.56, p-value < 0.001; d = $16.74, p-value = 0.063 respectively); the second analysis shows conditional means (d = $59.56, p-value < 0.001; d = $23.39, p-value = 0.043 respectively. E. Implementation of SE and SMM 20. Eighty percent or more of the study sites achieved a high level of IPS program implementation (i.e., met the documented standard for high fidelity). This high level of implementation persisted across the entire study period. 21. The level of unengagement in employment services among treatment group participants was relatively low overall (~10 %). 22. Concordance between the SSA diagnostic category and the SCID diagnosis was greater than 80 percent. 23. More than 87 percent of treatment group participants had at least one physical health condition, and 69 percent had two or more. More than half of all beneficiaries in the treatment group had a Body Mass Index (BMI) in the obese range. 24. The quality of SMM varied considerably across study sites. The off-site location of many prescribers affected this measure. For purposes of conducting the MHTS, the decision to allow participants to remain with their off-site prescribers was reasonable. However, the goal of MHTS was to deliver an integrated package of services to participants. Having off-site prescribers presented great difficulties in integrating the SMM components with other treatments. Executive Summary EX-9 F. Health Benefits Plan 25. Spending through the Health Benefits Plan averaged $6,986 per study participant per year. Spending was less ($6,342 per year) for the overall treatment group population (which includes those participating for less than 24 months). These figures do not include spending for services related to the NCC role. 26. The overall spending distribution included over 70 percent for SE services, 11 percent for health insurance premiums, 8 percent for medication prescriptions, 7 percent for behavioral health services, and less than 3 percent for employment-related work expenses, transportation, and other miscellaneous expenses. SE services were the only services completely paid by the study. G. Utilization of Services 27. The treatment intervention had significant positive impacts in reducing inpatient hospital use (for both admissions and number of days) and psychiatric crisis visits. 28. As expected with the intervention package, treatment group participants showed significantly higher use of regularly scheduled clinic or mental health visits. 29. The average reduction in hospital days was 0.9 days per year which translated into approximately $1,800 per year per person. Sustaining these costs over longer periods would increase their magnitude. SSDI beneficiaries with schizophrenia or an affective disorder who indicated that they wanted to return to work and who enrolled in the study and then received a random assignment to the treatment intervention, as a group, ended the study with significantly better employment rates, better mental health, and a higher quality of life. Further, they ended the study with many additional significantly better results than the control group, including higher earnings and income, more hours worked, a greater number of months worked, and greater satisfaction with their main job. The treatment intervention essentially delivered to SSA everything it asked for in the study design. For whatever reason(s), the intervention package, which comprised a rich mix of services and benefits, was successful in getting a large portion of the beneficiaries who enrolled in the study into jobs—the primary goal of participation. The next section explores the implications of these findings. Policy Implications The original research question concerns the extent to which the treatment intervention resulted in better outcomes. The findings presented above easily answer this question. The remaining two questions about programmatic barriers and specific programmatic changes do not come easily or directly from the study data. Instead, they are qualitative in their nature. The following questions and Executive Summary EX-10 answers attempt to provide insight that will provide the agency with the information it needs to move the MHTS result forward. 1. Is there significant interest among SSDI beneficiaries with schizophrenia or an affective disorder to access an MHTS-like program? Answer. Yes. Today there are nearly 2.25 million SSDI beneficiaries with a psychiatric impairment. The largest portion of this group includes those with schizophrenia or an affective disorder. The study findings suggest that a reasonable take-up rate would be nearly 14 percent or 306,000 of the 2.25 million SSDI beneficiaries with a psychiatric impairment. In addition, if SSA targeted enrollment to only those with recent work activity (e.g., recent TTW activation), the rate would range between 26 and 32 percent. 2. What essential services or features of services are required to achieve MHTS-like results? Answer. The following services and features comprise the unique characteristics of the MHTS: 3. a. Essential services: Evidence-based SE (the IPS model), SMM, OBH services (such as therapy, counseling, substance abuse counseling, etc.), benefits counseling, and modest monetary support to pay for services that are otherwise inaccessible. b. Community mental health centers (CMHC): CMHCs offer both the comprehensive range of services needed to treat mental illness and provide integrated vocational services in the community mental health center. The experience of the MHTS suggests that the community mental health center was ideal for delivery of these essential services. c. Care coordinator: Use of a care coordinator to facilitate SMM was a key feature of the mental health services provided in the MHTS. The SMM program in the MHTS used the NCC to facilitate and promote prescriber use of evidence-based guidelines and recommendations for medication management of severe and persistent mental illnesses. d. Out-of-pocket expenses: Payment for out-of-pocket mental health and essential workrelated expenses was an important feature that improved access to services and jobs for study participants. How much did the study spend on each treatment group participant to achieve the study results? Answer. Overall, the MHTS spent an average of $6,986 per year per participant. These expenditures do not reflect all of the participant costs for services needed during participation in the study. NCC services, for example, were paid through the study and not as part of the HBP. Medicare or Medicaid paid for many of the mental health and general health services required by participants. Executive Summary EX-11 Expenditures through the HBP made it clear that obtaining payment for all behavioral health care costs, along with payment of insurance premiums, removed some of the putative barriers to participation in active efforts to return to work for individuals in the treatment group. These payments represented less than 10 percent of the total HBP expenditures, suggesting that across all participants in the treatment group, gaining access to needed health care amounted to approximately $53 per month. This is a remarkably low cost for improving access to needed health care. It is noteworthy that fewer hospital stays and fewer psychiatric emergency or crisis visits may offset some of the HBP expenditures. Further exploration of the MHTS data may clarify the potential for health care. 4. What programmatic disincentives exist that create barriers for Title II beneficiaries with schizophrenia or an affective disorder to return to work? Answer. Three general barriers describe the kinds of problems SSDI beneficiaries faced in gaining access to needed health care and employment programs and supports. Under current conditions, SSDI beneficiaries have insufficient access to health care programs, services, and treatments. Each of the problems presented below create distinct barriers to effective treatment, better functioning and, ultimately, to employment. No one barrier by itself is particularly troublesome. However, as a group, they represent formidable obstacles to return to work. a. i. The study revealed many instances where insufficient access to programs, services, and treatments was a problem. Approximately 7 percent (74) of treatment group participants required enrollment in one or more of Parts A, B, or D of Medicare. ii. Reports from participants in the treatment group suggested that in the past, the cost of insurance co-pays (for health care visits, prescription medications, etc.) kept some participants from seeking treatment. In general, it is clear that such decisions to seek treatment were complicated and highly individual. For example, a participant related a story that prior to the study she chose not to refill a prescription for her psychiatric medication because it was too expensive given her financial condition. However, upon further discussion, it became clear that the decision was more complicated than that. She felt the medication was not effective (had unpleasant side effects, was not sufficiently reducing symptoms) and, therefore, was not worth the cost of the co-pay to refill it. Many previous trips to the doctor to find a better medication proved expensive and ineffective. It was no longer worth the effort required to seek a solution. iii. One ongoing concern throughout the study was the role that high-cost psychiatric medications played in finding effective treatments for participants in the treatment group. Several situations were problematic. One clear problem was the effect of the Executive Summary EX-12 so-called “donut hole” on medication use. NCCs reported that prior to the MHTS when participants came up against the “donut hole,” they would stop refilling their prescription(s) for psychiatric (and other) medication(s) because the high costs did not fit within their budget. Another problem reported by some participants was that prior to the study, prescribers were reluctant to prescribe some (potentially more effective) medications due to the high cost to the patient. iv. Another problem was finding a Part D plan that covered the preferred medications. During the study, this always presented a challenge, and required more efforts from the Westat Insurance Planner than any other insurance issue. A change in the Part D prescription medication plan required waiting until the open enrollment period. The findings would not likely be the same without the support of the study to pay for medications when they were not covered or to pay for high co-pays. v. The study paid special attention to receipt of mental health case management services because case management is a key for achieving adequate mental health treatment. The fact that only 54 percent of participants received mental health case management, and of these, 28 percent received their case management services offsite, is far below the expected rate in high fidelity IPS programs serving clients with severe mental illness. b. Under current conditions, SSDI beneficiaries lack access to evidence-based SE services in community mental health centers. Analysis of the health services utilization data reveals that the treatment group used significantly more vocational services than the control group. This was expected, given the intervention package included the provision of evidence-based IPS SE. These services, which emphasized competitive employment, clearly contributed to the positive employment results attained in the study. However, as discussed elsewhere in this report, community mental health centers do not generally provide SE services. Payment for these services is limited to Medicaid. In fact, even Medicaid does not cover these services in every state. c. Many SSDI beneficiaries with schizophrenia or an affective disorder have complex co-morbid physical conditions that impede efforts to return to work. Physical impairments created by a wide variety of health conditions were serious deterrents to employment efforts made by study participants, NCCs, and employment specialists. As reported above, 87 percent of the participants in the treatment group had at least one co-morbid physical health condition, 69 percent had two or more, and more than half were obese. Frequent comments on conference calls throughout the study noted that the SSDI beneficiary population in the study was much less healthy and had more problematic health conditions than did the populations normally served by the mental health centers. These comments corroborated the data collected from treatment group participants and formal documentation of why some beneficiaries do not return to work. In fact, good physical health (as measured by the SF-12) frequently came up as a predictor of employment (obtained employment, steady worker, or number of months employed), suggesting that there was enough variability in physical health to pick it up in multivariate analyses. Executive Summary EX-13 Chapter 1 Study Overview The Mental Health Treatment Study (MHTS) provides a test of the hypothesis that access to supported employment (SE) services and systematic medication management (SMM) services, coupled with the removal of programmatic disincentives, will enable Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or an affective disorder to return to work. Following nearly 6 years of conceptual development, panel reviews and preparation, the Social Security Administration (SSA) fielded the study between November 2006 and July 2010. The study design was a randomized controlled trial that included 2,238 SSDI beneficiaries between the ages of 18 and 55 with a primary impairment of schizophrenia or an affective disorder. Following their election to enroll in the study (giving consent and then completing an extensive Baseline interview), study participants received a random assignment to either a treatment or control group. Participants in both groups subsequently completed seven Quarterly interviews and a final Followup interview, designed to collect self-reported data on the outcome of interest. The treatment group received a comprehensive package of benefits that included SE services, SMM, behavioral health and related services (and comprehensive insurance to pay for them), reimbursement for out-of-pocket behavioral health expenses, and a guarantee that the SSA would suspend their medical Continuing Disability Reviews (CDRs) for 3 years from the date of study enrollment. The study paid all costs associated with obtaining services and prescription medications associated with behavioral health care that were not paid for by other sources. The control group received “services as usual” with minimal enhancements, which included a comprehensive manual of services that were available in their community and a modest payment amounting to $100 over the study period for completing the interviews. Medical CDRs remained in effect for these beneficiaries. Beneficiaries in the control group were not limited or restricted in any way from seeking community services to achieve their employment goals. In fact, they were encouraged to use the manual provided to them to seek assistance in obtaining employment. Services as usual refer to the fact that beneficiaries in the control group receive no special or unique services from the study. Any services they seek or receive depend entirely upon their own efforts. Chapter 1: Study Overview 1-1 Beginning on the day of randomization, enrollees remained in the study for a period of 24 months. At the 20th month of study participation, beneficiaries in the treatment group participated in a formal transition process designed to ease them back into “services as usual,” without the enhancements provided by the intervention. A core component of the treatment intervention was the evidence-based Individual Placement and Support (IPS) model of SE services. The seven core principles that form the foundation of the IPS model include the following: 1. Consumer choice. Individuals who are interested in work are eligible for IPS without exception. 2. Integrated services. Vocational and mental health services together are a part of the overall treatment approach. 3. Competitive employment in regular work settings. No pre-employment training or placement in sheltered or segregated work settings is required before placement in competitive work settings. 4. Rapid job search. IPS employment specialists help clients begin the job search soon after enrollment in IPS. 5. Personalized follow-on support. After placement, individuals and their employers (if desired) receive ongoing support for as long as they need it. 6. Person-centered services. Clients’ personal preferences, experiences, strengths, and choices drive the job search and follow-on supports rather than the judgment of the employment specialist. 7. Benefits counseling. Clients receive benefits counseling to ensure successful navigation of any impact of employment on government entitlements such as Medicaid or Social Security benefits. Notable among this list are the principles of consumer choice and rapid job search. Contrary to many employment programs for persons with mental illness, IPS places no limitations or restrictions on who is eligible to participate. For example, there is no programmatic requirement to meet a particular therapeutic goal prior to work placement, as is the case in many mental health treatment programs. Any individual interested in work is eligible for the program. With the principle of rapid job search, individuals enter the labor force as soon as they feel ready. There is no special preparation through training or career exploration, as is the case with some vocational programs. When the individual feels ready, they begin work. Any program adopting the evidence-based IPS model must Chapter 1: Study Overview 1-2 demonstrate fidelity to the seven core principles if it expects to achieve the measured outcomes (Bond, 2004; Loveland, Driscoll, & Boyle, 2007). Another core component of the treatment intervention was the evidence-based SMM services. While psychiatric symptom relief by itself typically may not lead to return to employment for those with a severe mental illness, it is clear that symptoms of depression, mania, and psychosis can interfere with functionality and interpersonal relations. Hence, symptom amelioration and stability may contribute to the success of behavioral interventions, such as SE, and to greater productivity and satisfaction with life (Awad et al., 1999; Leidy et al., 1998; Skevington et al., 2001). While both IPS SE services and SMM are widely recognized as having a strong evidence base, investigators have never fully tested these treatment modalities as a combined intervention package. Additionally, prior studies of the effectiveness of IPS SE or SMM include samples of people with severe mental illness who were already engaged in treatment when approached to participate in the research. By contrast, people recruited to participate in the MHTS came from the rolls of SSDI beneficiaries and not all of them were currently engaged in treatment. Hence, SSA wants to know if it is effective to offer this package of services in community mental health centers across the United States, and whether providing SSDI beneficiaries with access to these services improves employment, health, and quality of life. SSA’s Request for Proposals (SSA-RFP-05-1044) posed three research questions for the MHTS. They were as follows: 1. To what extent does delivering appropriate mental health treatment and employment supports lead to better employment, health status, and quality of life among SSDI beneficiaries with schizophrenia or an affective disorder? 2. What programmatic disincentives exist that create barriers for Title II (SSDI) beneficiaries with schizophrenia or an affective disorder to return to work? 3. What specific programmatic changes would support the efforts of SSDI beneficiaries with schizophrenia or an affective disorder to sustain competitive employment? (SSA, 2005, p. 6) In consultation with SSA, MHTS investigators interpreted the research questions in more operational terms. The first question concerns the primary outcomes of the study, including the extent to which the intervention results in better employment, health, and quality of life outcomes for SSDI beneficiaries. Answers to this question appear mainly in Chapter 4 (Outcomes) and Chapter 8 (Utilization of Services). The second question requests an explanation of the programmatic barriers to work, including a focus on why beneficiaries elected not to enroll in the Chapter 1: Study Overview 1-3 MHTS when given the chance, why beneficiaries who did enroll did not engage in its services, and why some beneficiaries who participated in the intervention were unsuccessful in their efforts to return to work. The answers are complex and explored in many different analyses conducted following data collection. Chapters 3-8 each offer answers to various parts of this question. The third question focuses on interpretations of the primary outcomes of the study and the potential for programmatic changes to support beneficiaries with schizophrenia or an affective disorder in their efforts to return to work. The answers to this question focus on policy implications of the study results, which appear in Chapter 9. This final report includes a comprehensive description of the study, including the study design, implementation of key intervention components, outcomes, and policy implications. The report includes the nine chapters listed below. Chapter 1: Background and Understanding, acquaints the reader with the significant policy concerns that SSA has for this population, relates recent research that motivated the study, and presents the key research questions. Chapter 2: Study Design, presents the design features that drive the scientific and policy objectives of the study. Chapter 3: Enrollment and Participation presents analyses of the characteristics of SSDI beneficiaries who enrolled in the study. The primary analyses include identification of distinguishing characteristics of SSDI beneficiaries who enrolled compared to the larger SSDI population who were eligible to participate in the study, as well as baseline comparisons between beneficiaries randomized to the treatment and control groups. Chapter 4: Outcomes, presents analyses of the key outcomes of the study and includes a comprehensive analysis of the intervention’s impact on SSDI beneficiaries’ employment and earnings (including months employed during the study, types of jobs obtained, wages, etc.), health status, and quality of life. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services, presents a description of the implementation strategy and fidelity of the SE and OBH components of the intervention, and includes data on the actual implementation parameters. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management (SMM), presents a description of the role of the NCC and SMM intervention components, and includes data on the actual implementation parameters. Chapter 7: Health Care and Supported Employment Financing, describes the approach used to provide participants in the treatment group with access to medical, behavioral, and SE services, and the resulting expenditures associated with providing those services. Chapter 1: Study Overview 1-4 Chapter 8: Utilization of Services, presents a comparison of the use of behavioral health care services (including emergency room visits and hospital overnight stays) between participants in the treatment and control groups at baseline and during the study. Chapter 9: Study Limitations, Key Findings, and Policy Implications, revisits the original research questions posed by SSA, presents study limitations, concludes the findings of the previous chapters, and offers a broad range of policy implications for SSA and other federal agencies affecting provision of evidence-based services to SSDI beneficiaries with severe mental illness. Background and Understanding SSDI is a long-term disability insurance program, managed by SSA, designed to provide income to people who are no longer able to work because of a disability. Age at the time of onset of disability determines the number of quarters a person must have previously worked in order to qualify for the SSDI program. After a two-year waiting period, SSDI beneficiaries are eligible for Medicare benefits provided by the Center for Medicare and Medicaid Services. The SSDI program addresses the income support and health care needs of workers who can no longer engage in substantial gainful activity (SGA) due to a medically determinable physical or mental impairment. SGA is a formal term from the statute authorizing SSDI. SGA is set in regulations and defines the monthly earnings level below which a beneficiary is eligible for cash benefits (for 2011 the amount is $1,000/month or $1,640 for individuals who are legally blind). However, any work activity can trigger a work CDR. It is possible that a beneficiary can lose benefits because of either of these processes. Monthly earnings consistently above SGA will eventually result in loss of cash benefits following both a grace period and a trial work period. Regardless of work activity and level of earnings, the program requires a periodic medical CDR to reestablish the existence of the underlying impairment that was the basis for disability or to document medical improvement that can lead to program ineligibility. In cases where improvement is expected, the next medical CDR occurs within 1 year. In cases where improvement is possible, the next medical CDR occurs between 1 and 3 years. In cases where improvement is not expected, the next medical CDR occurs within 5 to 7 years (SSA, 2010). In most cases of psychiatric impairment, the CDR occurs within the 1 to 3 year range. A high priority at SSA is supporting SSDI beneficiaries in their efforts to return to work. The agency offers numerous programs and supports that facilitate return to work and pursuit of personal employment goals. For example, the Ticket to Work program assists beneficiaries to get and keep a Chapter 1: Study Overview 1-5 job. The Trial Work Period (TWP) allows beneficiaries to test their ability to work for weeks or months at a time without worry of losing cash benefits. The Expedited Reinstatement rule serves as a safety net for those beneficiaries who return to work, get off the program completely, but then find at a later date that they can no longer work. Programs and supports like these exist to assist beneficiaries in achieving their employment goals. The MHTS is one more in a series of ongoing demonstrations at SSA to explore new and improved employment supports to beneficiaries in the SSDI program. Other efforts include the Benefit Offset National Demonstration project and the Youth Transition Demonstration (SSA, 2009). SSDI Beneficiaries with Psychiatric Impairments SSA categorizes individuals with psychiatric impairments as having a mental disorder in the SSDI program. This classification also includes individuals with mental retardation. However, the majority of SSDI beneficiaries with mental disorders are those with psychiatric impairments, including schizophrenia, paranoia, and other psychotic disorders; affective disorders; anxiety-related disorders; and personality disorders. According to the SSA Listings for Mental Disorders, “The evaluation of disability on the basis of mental disorders requires documentation of a medically determinable impairment(s), consideration of the degree of limitation such impairment(s) may impose on the individual’s ability to work, and consideration of whether these limitations have lasted or are expected to last for a continuous period of at least 12 months” (SSA Blue Book, 2008, Section 12.00). Accordingly, consideration for SSDI benefits first requires there be medical findings that substantiate the presence of a mental disorder. Second, an assessment of the level of severity examines the impairment-related functional limitations that are incompatible with the ability to engage in SGA. These include limitations related to activities of daily living; social functioning; concentration, persistence, or pace; and episodes of decompensation. Beneficiaries with psychiatric impairments have been and continue to be a significant policy concern for SSA. Since the early 1980s, the agency has struggled with regulations governing disability determinations for this population (Grob & Goldman, 2006) and the reliability and validity of standards for diagnosis of psychiatric impairments (Pincus et al., 1991). During this period, there has been a steady, if not dramatic, growth in the number of SSDI awards for this population. In 1970, program awards for psychiatric impairment were 2 percent of all new awards (Danziger, Frank, and Meara, 2009). However, in 2006, the year MHTS implementation began new awards for this Chapter 1: Study Overview 1-6 population that had increased to nearly 22 percent of all new awards (SSA, 2005). In addition, the number of beneficiaries under the age of 50 continues to increase at a rate of nearly 3 percent per year. Table 1-1 shows the number of SSDI beneficiaries under age 50 with “Other Mental Disorders” (i.e., mental disorders other than mental retardation) between 1996 and 2009, along with the number and percent change each year. In only one year (in 1997) did the number of SSDI beneficiaries drop. The number of SSDI beneficiaries with Other Mental Disorders increased by 268,004 over the period from 1996 to 2009. This represented an increase of 38 percent (SSA, 2009). Individuals classified as having a Mental Disorder are more costly than other SSDI beneficiary populations, primarily because they are younger (in their mid 40s to mid 50s on average), have been on the rolls on average for nearly a decade, and stay on the rolls longer (Hennessey & Dykacz, 1989; McAlpine & Warner, 2000). In addition, one in four SSDI beneficiaries have income and resources low enough that they also receive Supplemental Security Income (SSI) benefits from SSA (Berkowitz, 2003; Newcomb, Payne, & Waid, 2003). Table 1-1. Number of SSDI beneficiaries under age 50 who had a primary diagnosis of Other Mental Disorder1: 1996-2009 Year 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 1 Number 701,887 685,227 703,537 719,237 737,289 777,823 814,543 847,508 875,735 898,200 909,980 919,886 940,168 969,891 Change Percent change -16,660 18,310 15,700 18,052 40,534 36,720 32,965 28,227 22,465 11,780 9,906 20,282 29,723 -2.4 2.7 2.2 2.5 5.5 4.7 4.1 3.3 2.6 1.3 1.1 2.2 3.2 “Other Mental Disorders” includes schizophrenia and affective disorders, as well as organic mental disorders, anxiety related disorders, somatoform disorders, personality disorders, substance addiction disorders, and autistic/other pervasive developmental disorders. The category does not include mental retardation. SOURCE: Annual Statistical Report on the Social Security Disability Insurance Program, 2009, Social Security Administration, Office of Retirement and Disability Policy, Office of Research, Evaluation, and Statistics. Once on the rolls, SSDI beneficiaries in general, and beneficiaries with Other Mental Disorders more specifically, do not return to substantial work activity in any significant numbers. Using crosssectional statistics published by SSA in 2009 (Tables #6 and #52, SSA 2009), approximately eighttenths of one percent (.78%) of beneficiaries age 18 to 49 with Other Mental Disorders had their Chapter 1: Study Overview 1-7 cash benefits withheld due to substantial work activity. An almost identical number (.76%) had their benefits terminated in 2009 due to successful return to work (Tables #6 and #53, SSA, 2009). Recent analyses conducted by Stapleton (2010), who used administrative records to follow a longitudinal cohort of SSDI beneficiaries (with any diagnosis), suggested that benefit termination due to work activity may be much higher, at around 4 percent. Regardless of whether one uses a cross-sectional or longitudinal cohort, both percentages are remarkably low, suggesting that many individuals who obtain SSDI benefits do not seek work and likely consider themselves as having left the work force. Use of SE to Improve Employment Outcomes Research conducted in the past 20 years shows that many individuals with disabling mental illness want to work and believe they could work if they had access to employment services and supports (Frounfelker et al., in press; McQuilken et al., 2003; Mueser, Salyers, & Mueser, 2001; Uttaro & Mechanic, 1994; Van Dongen, 1996). SE services, particularly the IPS model, have demonstrated success in getting people with severe mental illness into competitive jobs. IPS is a well-defined form of SE, and is an evidence-based practice specifically designed to serve individuals with severe mental illness (Becker & Drake, 2003). The body of evidence supporting IPS effectiveness began to develop in the early 1990s with quasiexperimental studies comparing day treatment programs to SE programs. When combined across all of the early studies, comparisons between people receiving IPS modeled SE services to those receiving day treatment showed that IPS was significantly more effective at increasing competitive employment rates than the day treatment model.1 While 38 percent of the SE group achieved competitive employment, the comparison group remained static at 15 percent competitive employment (Bond, 2004). Subsequent research employing the IPS model shifted to experimental trials with the goal of establishing a causal relationship between IPS and employment. An empirical review of 11 randomized controlled trials of IPS programs serving individuals with severe mental illness concluded that employment outcomes were consistently higher than the alternative control group programs (Bond, Drake, & Becker, 2008). In these studies, participants in the control group received 1 Competitive employment is a job that pays at least minimum wage; is “owned” by the employer rather than a mental health center or rehabilitation agency; not set aside for people with disabilities; and is carried out with some degree of regularity in terms of work schedule. Chapter 1: Study Overview 1-8 either vocational services as usual, or a specific non-IPS vocational rehabilitation service. The length of followup across the studies ranged from 6 to 24 months. The overall rate of competitive employment was 61 percent for the participants receiving IPS services and 23 percent for participants in the control groups. A subsequent meta-analysis (Bond, Xie, & Drake, 2007) confirmed these findings in a similar sample of 681 individuals with a diagnosis of severe mental illness that also varied in their receipt of SSI and SSDI.2 Despite this strong body of supporting evidence, the IPS model of SE is not widely implemented across the United States, and is not available to many individuals with severe mental illness who could potentially benefit from it. According to the Substance Abuse and Mental Health Services Administration (SAMHSA, 2009), only about 2 percent of the population with severe mental illness has access to SE services, and far fewer to the evidence-based IPS model. The reason is primarily due to difficulties in financing these services (Drake et al., 2009). Local mental health service providers cannot recover the costs associated with hiring, training, or retaining employment specialists. In many states, Medicaid will pay for some portion of these employment services for beneficiaries receiving SSI, but the limitations and complexity of accessing such coverage appears to discourage even the most ardent supporters of these services. Similarly, state vocational rehabilitation services cover some aspects of IPS services, but not others. Without reliable and substantive financial support, it is not plausible for community mental health centers to provide SE services to their clients. In a recent report to the Assistant Secretary for Planning and Evaluation in the Department of Health and Human Services (DHHS), Karakus, Frey, Goldman, Fields, & Drake (2010) addressed in detail the lack of access to evidence-based SE services for persons with severe mental illness and the potential financing solutions to making those services more widely accessible. Use of Medications to Improve Psychiatric Symptoms Medications such as antipsychotics, mood stabilizers, and antidepressants provide symptom relief for persons with schizophrenia, bipolar disorder, or depression (Hales et al., 2010). There is virtually universal use of medication to treat these conditions in public and private mental health facilities in the United States. In addition, prescribing antidepressants is extremely common among primary care providers. Relatively recent studies, however, have identified significant problems with the ways in which medications are used and with the adequacy of patient-based information needed to guide 2 It should be noted that all of these studies of the effectiveness of supported employment recruited participants from the rolls of community mental health programs and not from the general rolls of beneficiaries on SSI or SSDI. Chapter 1: Study Overview 1-9 individualized medication selection. The Schizophrenia Patient Outcomes Research Team (PORT) study, for example, showed that both medication and psychosocial treatments for schizophrenia in community settings are frequently not in accord with expert, research-based recommendations (Lehman & Steinwachs, 1998). A number of studies identified underdosing of antidepressants as a major issue in primary care settings. Chart reviews in public mental health clinics also found poor documentation of schizophrenia symptoms more than half the time and an inadequate description of side effects 85 percent of the time (Cradock et al., 2001). Evidence from primary care and public mental health settings has demonstrated the feasibility and effectiveness of a collaborative care model, in which a trained non-physician clinician works in tandem with prescribers to document symptoms and side effects. Trained to recognize and evaluate the disorders, symptoms, and side effects of particular medications, this non-physician clinician prompts the prescriber with evidence-based recommendations based on the patient’s history and characteristics, and the features of available medication treatments (Katon et al., 1995; Miller et al., 2004; Suppes et al., 2003; Trivedi et al., 2004). Nurses with training in the use and side effects of medications, and who regularly work directly with physicians, have the skill set necessary for effective implementation of these collaborative models. Disincentives for Returning to Work Employment rates and the amount of work among people with severe mental illness are very low; lower than those of people with other types of disabilities, and remarkably lower than those of people without disabilities. Recent employment statistics released by the U.S. Bureau of Labor Statistics (2011) indicate that in 2010 the labor force participation rate for people without disabilities (ages 16 to 64) was 77 percent, while the employment rate was 70 percent. Comparable statistics in 2010 for people with disabilities in general suggest a participation rate of 34 percent and an employment rate of 29 percent, both less than half the rates for the population without disabilities. Research conducted by Mechanic, Bilder, McAlpine (2002) determined that the employment rate for persons with severe mental illness is approximately 22 percent during any given period. Further, they note that a substantial portion of the remaining population receives SSDI or SSI and does not participate in the labor force. The U.S. Bureau of Labor Statistics (2010) reported that approximately 80 percent of workers without disabilities worked full time. The Mechanic, Bilder, and McAlpine article noted that only about 12 percent of persons with severe mental illness work full time. Chapter 1: Study Overview 1-10 For the majority of people with a severe mental illness, getting a job and working can be a struggle. These individuals face a complex set of barriers that introduce doubt and confusion about their ability to work. The barriers include the stigma and discrimination attached to mental illness; disincentives unintentionally produced by the SSDI and SSI programs; and, as described above, the inaccessibility of evidence-based behavioral health services and employment supports. In addition, people with psychiatric impairments often have a relapsing-remitting course of illness that can negatively influence work consistency and social relationships. While all of these barriers are important, the barriers associated with the disincentives created by SSDI and SSI program participation make the decision to return to work particularly complicated. In a qualitative study of 16 employed and 16 not employed people with severe mental illness who also received SSDI benefits, O’Day and Killeen (2002), found that the current federal policies and practices associated with the program created disincentives for beneficiaries wanting to return to work. Initially, the individuals studied found the SSDI application process to be difficult, timeconsuming, unpleasant, and even humiliating. Because of these difficulties in qualifying for benefits and the ordeal they experienced during the application process, they were fearful of risking their benefits, once obtained, to try to return to work. Even if they recovered and felt well enough to try to return to work, there was fear of a relapse, resulting in anxiety about the prospect of having to reapply for benefits if symptoms returned. While this study included only small numbers of SSDI beneficiaries, its rich detail provides important information not normally brought to light in studies with larger populations. The fear of losing Medicare or Medicaid that comes with SSDI or SSI benefits further creates concerns about returning to work (MacDonald-Wilson et al., 2003; McQuilken et al., 2003; O’Day & Killeen, 2002). MacDonald-Wilson and colleagues conducted a study of Social Security work incentives and barriers to work. They surveyed 539 people with mental illness, 120 service providers, and 174 family members. All three groups rated the potential loss of health insurance as the number one barrier to returning to work. In another survey of consumers with mental illness receiving services at an urban mental health center, McQuilken and colleagues (2003) reported a number of reasons why SSDI beneficiaries did not return to work. Two key points that stood out were (1) the fear of losing benefits if they returned to work, and (2) the difficulty of getting back on benefits if they returned to work, and then lost their benefits. Three groups of beneficiaries consistently made these points, including those who indicated that they did not want to work, those who indicated they wanted to work but were not currently looking for work, and those who were looking for work. Chapter 1: Study Overview 1-11 Even with successful return to work, health care receipt is an issue among individuals with severe mental illness. Once on private health insurance, former beneficiaries will likely face limitations in the amount of mental health treatment and type of medications that their new insurer will cover. This prospect has improved with the passage of the “parity” legislation, removing differential costsharing and other insurance benefit limits between behavioral health care and general medical services. Nonetheless, private health insurance does not cover some of the essential services that support a return to work, in particular, SE services. The potential for trading Medicare benefits for inadequate (or no) health care coverage only causes more hesitation in returning to work. SSDI earnings limits are another major disincentive to returning to work. Beneficiaries report feeling trapped in part-time, low-paying jobs even though they feel they are capable of earning more and working in higher paying jobs. Some beneficiaries express a desire to work full-time, but know they would lose their benefits under earning limits (Livermore, Goodman, & Wright, 2007; O’Day and Killeen, 2002). Because of their education and skills, some of these beneficiaries are unlikely to earn enough to make up for the SSDI benefits (including health insurance) they would lose by returning to work full-time. The situation is particularly dire for SSDI beneficiaries who are also on SSI. For SSI beneficiaries, after the first $85 of earned income, their SSI check is reduced by $1 for every $2 they earn. Many beneficiaries elect to forego jobs that place them in this position. Summary of Background and Understanding Many SSDI beneficiaries with severe mental illness want to work, and they can work, but they struggle against difficult odds to get and keep jobs. As previously noted, it is relatively rare for these beneficiaries to have access to the evidence-based treatments and supports necessary to improve their functioning needed for work. While effective SE and mental health services exist, they are not readily available to more than a small percentage of SSDI beneficiaries. However, research suggests that, when available, these services improve employment rates. Whether or not access to needed treatments and supports can overcome the strong disincentives resulting from SSDI program participation is an unanswered question. Design of the Mental Health Treatment Study In response to the seemingly surmountable barriers to effective treatment and return to work, the SSA designed the MHTS. The agency had high expectations that the study could demonstrate Chapter 1: Study Overview 1-12 successful SSDI beneficiary access to evidence-based treatments and employment supports across a representative sample of community mental health settings; and further, that such access would result in improved employment rates, health, and quality of life. Formal efforts to design the MHTS began more than a decade ago with an SSA Task Order request to design an experimental study to test the effects of providing SSDI beneficiaries with affective disorders access to SMM and SE services. In 1999, SSA announced the study as a historic demonstration project that could help beneficiaries with affective disorders overcome the disabling effects of their illness, return to the workforce, and lead more productive lives. At that time, 11 percent of the 4.7 million SSDI beneficiaries receiving disability payments had a primary diagnosis of an affective disorder. Research during the previous decade strongly supported the need to test the hypothesis that such access would lead to improvements in daily functioning and ultimately employment. Conducted by The Lewin Group, the Task Order resulted in the release of a Request for Proposals by SSA in June 2001, titled Affective Disorders Treatment Demonstration Project (RFP-010031, 2001). SSA accepted proposals, but later cancelled the procurement. After several attempts to field the study, a second Task Order request, released in July 2003, called for a Technical Advisory Panel to assist in the conceptual design of the MHTS (SSA-RFTOP-03-2006). The Task Order, conducted by the Urban Institute, released its final report in February 2005 (Aron, Burt, & Wittenberg, 2005). The report recommended expansion of the study to include individuals with schizophrenia. In May 2005, after more than 5 years of development, the SSA released a request for proposal (RFP) to conduct the MHTS (SSA-RFP-05-1044). SSA’s design for an effective test of access to SE and SMM services, in the face of an emerging reality that many SSDI beneficiaries want to work and can work, is a reflection of the complexity of SSA disability policies. At present, being on the rolls and working are effectively (though not technically) mutually exclusive. Beneficiaries that want to work are encouraged to work by SSA. In fact, SSA offers a variety of programs to encourage work (SSA, 2009; Livermore, Goodman, & Wright, 2007; MacDonald-Wilson et al., 2003). On the other hand, SSDI beneficiaries who do work face reduced cash payments if they work “too much.” If they work at or above SGA for lengthy periods, they also risk losing their medical benefits due to program ineligibility. The medical benefits, while not necessarily sufficient, are still an important safety net for those who fear recurrence of mental illness symptoms. Chapter 1: Study Overview 1-13 Outcome Measures SSA specifically targeted three outcome areas of study for beneficiaries participating in the MHTS. These areas included employment, health, and quality of life. While there were many other areas of secondary interest, these three areas were the primary focus of the agency in its attempt to understand the impact of the intervention on SSDI beneficiaries with schizophrenia or an affective disorder. Employment. The primary employment measure was employment rate, as measured by beneficiary self-report of work at any point during the study period. Beneficiaries responded to standardized questions about employment (Current Population Survey, sponsored by the Bureau of Labor Statistics, 1994-present) during each of nine interviews spaced approximately every quarter throughout the 24 months of study participation. Health. The health measures were physical and mental health status based on beneficiary selfreports of perceptions of their health during the previous month. Assessed at baseline and again at study exit, the key measures were the SF-12 component scores that summarize overall physical and mental health (Ware, Kosinkski, Turner-Bowker, and Gandek, 2002). Quality of Life. The measure of quality of life was a single self-report scale item from the Modified Lehman Quality of Life Inventory (QOLI-M). The QOLI-M is a shortened version of the wellknown QOLI (Lehman, 1988). Similar to the health status measures described above, the quality of life measure was assessed at baseline and study exit. Secondary Outcomes of Interest. The four measures mentioned above—employment rate, physical health status, mental health status, and quality of life—form the heart of the outcome analysis presented in Chapter 4. However, the study offered the opportunity to assess the effectiveness of the intervention on a number of additional outcomes of secondary interest to SSA. One set of these outcomes concerned variables related to employment, including earnings, job type, hours worked, highest hourly wage, and satisfaction with job, among others. Additional outcomes of secondary interest were income from a variety of sources (e.g., other public programs, investments, savings, alimony, etc.), alcohol and substance abuse, and health care (including mental health care) service utilization levels and patterns. Lastly, there was interest in the outcomes of some specific subgroups of beneficiaries participating in the study. During discussions about sample size for the study and statistical power, SSA noted Chapter 1: Study Overview 1-14 potential interest in diagnosis, age, and gender, and education. Accordingly, these variables also received attention in the outcome analysis, though they were secondary to the overall impact of the intervention on all study participants. MHTS Investigators Westat was the prime contractor for the MHTS. Supporting Westat were researchers from Dartmouth Medical School, the University of Texas Health Sciences Center at San Antonio, and the University of Maryland Baltimore County. Co-Principal Investigators William Frey, Ph.D., from Westat, and Robert Drake, M.D., Ph.D., from Dartmouth Medical School served as Co-Principal Investigators of the study. Dr. Frey, a vice president and study area director in the Health Studies sector at Westat, provided overall management for the study. Dr. Drake, who directs the New Hampshire-Dartmouth Psychiatric Research Center, provided support for treatment implementation at the study sites. Westat Led by Dr. Frey, the Westat team developed the study protocols, oversaw the day-to-day operations, and served as the primary point of contact for the 23 study sites. The Westat team also oversaw the management of the MHTS beneficiary sample, designed and implemented the study management information systems and computer-assisted personal interviews (CAPI) used for the study, and led the data cleaning and analytic file preparation efforts. Staff also played a key role in the data analysis. Howard Goldman, M.D., Ph.D., a Westat consultant and Professor of Psychiatry at the University of Maryland, School of Medicine, served as mental health policy expert to the study. An expert on mental health disability policy, Dr. Goldman provided the study team with ongoing consultation regarding all aspects of the MHTS. Chapter 1: Study Overview 1-15 Dartmouth Medical School Robert Drake, M.D., Ph.D., Gary Bond, Ph.D., and Deborah Becker, M.S., provided implementation support, fidelity measurement, and quality assurance associated with the provision of SE and OBH services to study participants at the 23 study sites. They also provided oversight through a group called the Quality Management Program Directors who provided ongoing technical assistance to the study sites and conducted on-site SE fidelity visits. The University of Texas Health Science Center at San Antonio Alexander Miller, M.D. and Troy Moore, Pharm.D, M.S. Pharm., BCCP with the University of Texas Health Science Center at San Antonio provided implementation support, fidelity measurement, and quality assurance associated with the provision of SMM services to study participants at the 23 study sites. Drs. Miller and Moore also conducted site visits to the 23 study sites to ensure that NCCs followed the SMM intervention protocol for treatment group participants who were prescribed psychiatric medication. The University of Maryland, Baltimore County David Salkever, Ph.D., a senior health economist and leading expert in the evaluation of mental health treatment and economics of disability, led the impact analysis of the study outcomes. Chapter 1: Study Overview 1-16 Chapter 2 Study Design The Social Security Administration (SSA) developed the research agenda for the Mental Health Treatment Study (MHTS) with guidance from its Technical Advisory Panel (TAP) (Aron et al., 2005). The final study design is the focus of this chapter. It incorporates the priorities of SSA, recommendations of the TAP, as well as insight and contributions from the investigators. The chapter begins with an overview of the two predominant scientific features of the study design—the experimental nature of the design and the intent-to-treat (ITT) approach to data analysis. These features were central to obtaining a clear assessment of the impact of the intervention on study participants’ employment, health, and functioning. The next section addresses the target population, and includes descriptions of study site selection, sampling methodology, and the randomization process. The next sections describe the intervention approach, including the intervention benefits and services provided to the treatment group, the control condition, and the quality control (fidelity) measures used to monitor implementation of the intervention. The concluding sections describe the outcome measures, data collection procedures, data preparation, statistical procedures for analysis of the results, and data limitations. Key Scientific Features The study design provides SSA with an objective and unbiased assessment of the impact of providing the intervention package to its SSDI beneficiaries who enrolled in the study. The two predominant features of the study design were its randomized controlled trial (RCT) structure and the ITT principle of the data analysis. Both are features of a strong experimental study design that contribute to arriving at answers to the primary research question about the impact of the intervention on employment, health, and quality of life. RCT Design The primary intervention components of the study—systematic medication management (SMM) and the Individual Placement and Support (IPS) model of supported employment (SE) services—have strong evidence to suggest their efficacy. Researchers have demonstrated success Chapter 2: Study Design 2-1 with each component in previous studies (including randomized trials) with a variety of populations of individuals with mental illness in a variety of clinical settings. However, their effectiveness remained untested with the SSDI population exclusively, in combination with one another, and across the diverse mental health treatment settings that reflect the existing service structures in the United States. The MHTS tested the intervention components under these circumstances with a two-arm RCT design. The intervention components (treatment group) comprised one arm of the design, and a services as usual (control group) comprised the second arm. The primary value of the randomized design was reduction in allocation bias provided by random assignment of SSDI beneficiaries to either the treatment group or the control group. Doing this improved the chances that treatment and control group participants were as similar as possible at the outset of the study, especially with regard to baseline measures of employment, health, and functioning. If participants proved to be different on any of these baseline measures, then additional design or analysis procedures would have been necessary. Following random assignment, tests of equivalence between the treatment and control groups on these primary baseline measures confirmed the success of the randomization process. Details of the randomization process appear below. The test results appear in Chapter 3 Enrollment and Participation. ITT Principle Another study design feature of the MHTS was the ITT principle. ITT is fundamental to RCTs. Data analysis includes all randomized subjects according to the group to which they were allocated (Lachin, 2000; Hollis & Campbell, 1999). Without this principle, even randomized trials may be subject to bias. For example, in the MHTS, participants in the treatment group could have withdrawn for any number of reasons, leaving only those who were best able to look for and obtain a job. If the study were to evaluate only the outcomes of those who remained in the study from beginning to end, and ignore the withdrawals, then the outcome would not represent the population of all beneficiaries enrolling into the study. For these reasons, the MHTS followed the ITT principle so that the outcomes for all study participants would be included in the analysis regardless of whether they withdrew from the study, the extent of adherence to the treatment intervention, or their level of engagement. SSDI beneficiaries randomized to the treatment group were not required to participate in all study activities in order to remain in the study. For example, if they elected not to take prescription medications, then they were less likely to participate in the SMM component of the intervention. In Chapter 2: Study Design 2-2 addition, they were not required to see a SE specialist at the study site. Designers of the study anticipated that some SSDI beneficiaries with mental illness would be unable or unwilling to participate in certain study activities, would become hospitalized, or refuse certain intervention components. Other beneficiaries might completely disengage from the study after enrollment, no longer participating in any study activities. The study did not “drop” these partially engaged or unengaged participants. Instead, study procedures required local study site staff to attempt to reengage these beneficiaries throughout the period in which they were participants. Study participants in either the treatment group or control group could voluntarily withdraw from the study in which case they received no further study contacts. However, the analysis included data available before such participants voluntarily withdrew from the study. The study design included two strategies to account for beneficiaries who withdrew or otherwise had missing data at study end. If an enrollee did not complete at least two post-baseline self-report interviews, then these cases were non-respondents. Weights were calculated and applied to the remaining study population to adjust for this type of non-response. Some enrollees withdrew later or did not complete all research interviews. In these cases, imputed data replaced the missing data. A more detailed discussion of each strategy appears later in this chapter in the section titled Data Preparation and Adjustments. Study Site Selection Two of SSA’s priorities were to implement the study in real world mental health settings and to do so in all regions of the United States. These priorities presented the investigators with particularly difficult challenges, given research clearly indicated that the core services were not readily available in most communities throughout the United States. Thus, it was a challenge for the investigators to identify community mental health centers with the capacity to provide the primary intervention components and then prepare them for delivering the necessary services. The TAP convened by the Urban Institute also noted that the MHTS should cover multiple states supporting the goal that the demonstration be representative of the types of community mental health centers found in the United States. One potential strategy for achieving representation was random selection through a nationally representative probability sample of study sites. However, the investigators decided against this strategy for two reasons. First, the study schedule required a rapid start-up with sites fully functioning within 6 months. It would have been impractical to randomly select study sites, ensure the necessary services were in place, and have the key components of the Chapter 2: Study Design 2-3 intervention—IPS SE services and SMM services—fully operating at high fidelity to the evidencebased practice within this time frame. Second, the number of study sites to be selected (originally 20) constituted a rather small number for a national probability sample. In addition, with the random selection of study sites approach, the level of fidelity would likely have been highly variable, leading to potentially ambiguous results. For example, it was possible that the results would not be clear as to whether the intervention was ineffective, or the ability of the sites to deliver high fidelity services was limited. SSA and the investigators concluded that the sites needed to deliver high fidelity services in order to establish the effectiveness of the treatment intervention with the target population. The investigators selected the study sites from among the more than fifty IPS programs that were already in operation, while taking additional steps to ensure the selection of sites was as diverse as possible and reflected various types of community mental health centers in the United States. In selecting the sites, considerations included U.S. census region, urbanicity, population composition, and center organization. Most of the selected sites had a history of delivering SE services, although not necessarily at high fidelity to IPS. With one exception (described below), they all delivered medication management services and were willing to participate in SMM. Initially, the selection process resulted in a recommendation of 20 study sites to SSA. SSA approved all 20 sites but requested recommendations for two additional study sites to ensure Hispanic population representation in the study. The sites identified were two mental health centers with strong interest in SE services in New York City, New York and San Antonio, Texas. Both sites served large Hispanic populations. While neither had existing IPS SE, both centers had experience with the provision of other forms of SE services and were willing to hire staff and undergo rapid training in order to provide IPS services. Upon approval, both of these additional sites received extra resources and training in advance of study startup to enhance their capacity to implement IPS SE. The study added an additional (twenty-third) study site approximately one-third of the way through the recruitment period. The inability of the original south Chicago site to handle potential study participants in the northern part of the catchment area necessitated the addition of a new site on the north side of Chicago. Operated by the same community mental health agency, this site received training and startup support from the south Chicago site. The map in Figure 2-1 shows the location of the twenty-three sites across the United States. One of the selected study sites (Spring Lake Park, Minnesota) was chosen to increase geographic diversity by Chapter 2: Study Design 2-4 its location in the northern Midwest region. Known for its excellent vocational services, this agency delivered SE services directly to enrollees, but not behavioral health and related services. This site had an existing contractual relationship with a county agency to provide these services. Discussions that occurred during the site selection process resulted in the understanding that such situations were a natural part of the continuum of community mental health service arrangements available across the country, and therefore, were an appropriate addition to the study. Figure 2-1. Map of MHTS study sites Two study sites faced internal operational issues within the first year of the study, which negatively influenced their ability to continue with the study. As a result, these sites ceased recruitment and enrollment activities within the first year (in June and September 2007). One site reorganized with involvement and oversight from their state office of mental health. The upheaval created by the reorganization was too disruptive to study operations to continue enrollment. The second site reported that their indirect costs were too high to continue participation in the study, and thus requested to withdraw. Both study sites terminated the two research positions of Nurse Care Coordinator (NCC) and Research Assistant (RA) but agreed to continue to provide treatment and SE services to the small number of beneficiaries already enrolled in the study. Treatment and control Chapter 2: Study Design 2-5 group participants at these sites also continued to complete the post-baseline self-report interviews throughout their 24-month period of participation. Their study data were included in the analysis along with that of participants from all of the study sites. Most of the other sites faced severe financial stress due to the economic recession and state funding cutbacks, but were, nevertheless, able to sustain the treatment intervention. Sampling Methodology The investigators identified a target population of SSDI beneficiaries residing within the catchment areas of the study sites, organized the beneficiaries into relevant strata, and then randomly assigned beneficiaries (proportional to each stratum) to recruitment release groups with a size of 25 beneficiaries per release group. Research staff at each study site then proceeded to recruit SSDI beneficiaries from their release groups. The target population for the study included SSDI beneficiaries between the ages of 18 and 55 inclusive with a primary diagnosis of schizophrenia or an affective disorder. With the few exceptions noted below, SSDI beneficiaries living within one of the study site catchment areas who met these criteria comprised the study target population. The exceptions were those beneficiaries living within the study site catchment area with one or more of the following situations: Beneficiaries living in a nursing home or other custodial institution; Beneficiaries designated legally incompetent to manage their own affairs (i.e., had a legal guardian); Beneficiaries with a life-threatening or a terminal condition (e.g., terminal cancer, AIDS, or end-stage renal disease 1); Beneficiaries receiving SE services from their participating study site within six months of study enrollment; and Beneficiaries with a competitive job less than thirty days prior to study enrollment. Define primary and backup catchment areas. The catchment area refers to the specific postal zip codes included in the communities, cities, or counties that the study site typically serves. With 1 Prior to completing the Baseline interview, beneficiaries were asked if they had any diseases, disorders, or physical impairments that would prevent them from working, receiving SE services, or participating in any other study activities. If a beneficiary reported having terminal cancer, AIDS, or end-stage renal disease, he or she would have been excluded from the MHTS; however, no beneficiaries met this exclusion criteria (i.e., no beneficiaries reported having terminal cancer, AIDS, or end-stage renal disease prior to completing the Baseline interview). Chapter 2: Study Design 2-6 the relevant zip codes identified, the administrative records from the SSA Master Beneficiary Record (MBR) file permitted identification of potentially eligible beneficiaries. The investigators held meetings with each study site to define their primary catchment area by geography (community, city, or county) and zip code. When appropriate, the primary catchment area was based on a combination of the service area from which the site was legally obligated (allowed) to draw clientele, as well as the estimated distance potential beneficiaries might be willing to travel to receive services from the site. The defined catchment area included an approximate 30-mile radius around the study site; however, the service area of the site took precedence over distance from the mental health center. Each study site also identified zip codes for a “backup” catchment area from which to recruit if a sufficient number of beneficiaries did not enroll from the primary catchment area. Since the study sites defined the backup catchment areas, the composition of these areas varied but included a combination of (1) beneficiaries who lived outside of the approximate 30-mile radius around the study site and (2) beneficiaries who lived in an adjoining county. Identify sample for target population. SSA provided the investigators with an electronic file from the MBR containing the target population with eligible SSDI beneficiaries. The electronic file contained a number of relevant variables needed for sampling, such as name and contact information, Supplemental Security Income (SSI) status, date of birth, and diagnosis. Table 2-1 presents a summary of the sample for the target population for each study site by psychiatric impairment type (diagnosis) and gender. The table contains two categories of SSDI beneficiaries— beneficiaries receiving SSDI only and beneficiaries receiving both SSDI and SSI (i.e., dual eligible beneficiaries). The ratio of beneficiaries with an affective disorder to schizophrenia is 2 to 1 (68% to 32%). While the overall ratio of males to females is slightly smaller than 1 to 1 (48% to 52%), the percentage of males to females among beneficiaries with schizophrenia is about 2 to 1 (67% to 33%). However, substantially more beneficiaries with an affective disorder are female; the ratio of females to males among beneficiaries with an affective disorder is approximately 1.6 to 1 (61% to 39%). Apply pre-recruitment exclusion criteria. It was possible to exclude some SSDI beneficiaries living within the catchment areas prior to release of the sample to the study site. The MBR contained variables that identified individuals as having a legal guardian, or living in a nursing home or other custodial institution. Thus, the final sample of eligible beneficiaries did not include those with such designations. In order to eliminate beneficiaries whose MBR data were not up to date, the screener that preceded the Baseline interview also included questions to identify these individuals and remove them from the pool of eligible beneficiaries. Chapter 2: Study Design 2-7 Table 2-1. MHTS sample by key stratifications Interview SSDI Male 18-35 36-55 Female 18-35 36-55 <24 months on rolls ≥24 months on rolls SSDI with concurrent SSI Male 18-35 36-55 Female 18-35 36-55 Schizophrenia (N=19,822) freq % 14,055 70.9 9,537 48.1 1,749 8.8 7,788 39.3 4,518 22.8 588 3.0 3,930 19.8 690 3.5 13,365 67.4 5,767 3,728 1,081 2,647 2,039 404 1,635 29.1 18.8 5.5 13.4 10.3 2.0 8.2 Affective Disorder (N=41,708) freq % 32,909 78.9 13,200 31.6 1,641 3.9 11,559 27.7 19,709 47.3 2,506 6.0 17,203 41.2 2,911 7.0 29,998 71.9 8,799 2,882 794 2,088 5,917 1,345 4,572 21.1 6.9 1.9 5.0 14.2 3.2 11.0 Total (N=61,530) freq % 46,964 76.3 22,737 37.0 3,390 5.5 19,347 31.4 24,227 39.4 3,094 5.0 21,133 34.3 3,601 5.9 43,363 70.5 14,566 6,610 1,875 4,735 7,956 1,749 6,207 23.7 10.7 3.0 7.7 12.9 2.8 10.1 Three additional exclusion criteria could not be determined from the administrative data. One criterion, current competitive employment, was determined from a series of employment questions asked during the recruitment process. Beneficiaries competitively employed within the past thirty days were not eligible for the study. A second criterion, existence of a physical health condition that precluded study participation, was also determined from the screener, or through the General Medical Exam (GME) administered to treatment group participants following random assignment. During the screener, beneficiaries responded to a question that asked whether they had any diseases, disorders, or physical impairments that would prevent them from working, receiving SE services, or participating in any other study activities. The beneficiary was ineligible if he or she responded “yes” and then reported having terminal cancer, AIDS (being HIV positive was not on its own grounds for ineligibility), or end-stage renal disease. For participants randomized to the treatment group, their primary care provider assessed all other identified conditions to determine study eligibility. Beneficiaries who had received SE services from a study site within the past six months were not eligible for the study. As the study site began recruitment efforts, a comparison between the list of potentially eligible beneficiaries and the client list of the study site yielded a few beneficiaries who received services within the past six months. Consequently, these beneficiaries were declared ineligible and removed from the recruitment list. Chapter 2: Study Design 2-8 Finally, the initial sample of eligible beneficiaries did not include those who were in the SSDI program for less than 24 months. This group of beneficiaries was of interest to SSA. However, they did not have access to Medicare until their 24th month of program eligibility, and there was concern that they would pose a financial risk to the study. Once it was clear that study funding could accommodate these individuals, the study sites received their names and contact information. The inclusion of these individuals occurred near the end of the first year of recruitment. Treatment group participants residing in Connecticut and Maryland who were in the 24-month waiting period were recruited from the beginning of the study since these states offered access to health insurance without precondition (see Chapter 7 Health Care and Supported Employment Financing for further details). Release sample for recruitment. The final step of the sample specification process involved formatting and loading the eligible target population of SSDI beneficiaries into an electronic Study Management System (SMS). Organization of the sample for each study site included multiple release groups in blocks of 25. Each release group contained the names and contact information of 25 beneficiaries selected from a stratified sort of the final list of eligible beneficiaries. The study sites received access to two release groups at a time through the SMS. The release groups were activated (i.e., made available to the study site) on a weekly basis or as needed by the site. The number of groups released each week varied over time based on the site’s recruitment progress. The primary reason for segmenting the release groups was to ensure that sites would “work” the sample of beneficiaries and not “speed-call” all of the beneficiaries in the sample to identify those who would readily enroll in the study. The philosophy was to make a good faith attempt to offer the study to all eligible beneficiaries in the study site catchment area. A detailed description of the recruitment process and information provided to potential beneficiaries appears in Supplemental Appendix A, Study Implementation (see Recruitment section). Sample Size The initial sample size was 3,000 participating beneficiaries, with a target of 1,500 in each study arm. However, a preliminary review of early study data prompted a downward revision of this initial target. Estimates for both overall and subgroup comparisons based on data collected from the first three months of the study indicated that a sample of 2,000 participants would provide sufficient power to address the primary questions of interest for overall comparisons between treatment and Chapter 2: Study Design 2-9 control groups, as well as many of the subgroup analyses. Specifically, this sample size would be more than adequate (80% power or greater) to achieve the following: 1. Find a $0.50 per hour wage difference in the overall sample; 2. Show an increased proportion of persons working in the treatment group for beneficiaries with schizophrenia, affective disorder, or aged 36 to 55; 3. Determine whether the treatment had a beneficial effect on total monthly earnings; and 4. Detect differences in the number of hours worked among persons with affective disorder and persons aged 36 or older. While the original sample size of 3,000 would provide enough power to detect differences in all of the above variables and subgroups, it would not provide enough power to evaluate most outcomes in the smaller subgroups, e.g., younger beneficiaries, beneficiaries with schizophrenia, or beneficiaries with the shortest time receiving SSDI benefits. Based on the power calculations and extensive discussions with SSA, the target population size decreased from 3,000 to 2,000. The overall final sample size for the study was 2,238 SSDI beneficiaries, with 1,121 allocated to the treatment arm of the study and 1,117 to the control arm of the study. The increase occurred for two reasons. As the end of recruitment neared, SSA requested an additional 100 beneficiaries for the treatment group to ensure that the numbers would be large enough for planned comparisons of subgroups. Thus, approximately 200 additional beneficiaries were required in order to meet this request due to the randomization procedures. Second, all study sites active in enrolling beneficiaries at the end of the recruitment period had an end date for enrollment of July 31, 2008. As the final days neared, it was not possible to stop enrollment precisely at 2,200 for a variety of reasons. Some sites had prior commitments to beneficiaries, and in other cases, appointment dates for enrollment already existed. Overview of the Enrollment and Randomization Process Enrollment into the study and random assignment to the treatment or control group followed a structured and lengthy process. Research staff at each study site received extensive training on a standard set of recruitment procedures designed specifically for the study. A detailed description of these enrollment procedures appears in Supplemental Appendix A, Study Implementation. Chapter 2: Study Design 2-10 Several steps preceded beneficiary enrollment into the study. Beneficiaries in each release group received a letter introducing the study. Study staff subsequently contacted the beneficiaries by telephone to invite them to attend a Recruitment Information Group (RIG) meeting. The RIG meetings provided beneficiaries with complete information about the study as well as the opportunity to have their questions answered. The use of RIG meetings is now standard practice in RCTs of IPS and justification for its use in the psychiatric population well described (Drake et al., 1994). Two RIG meetings were required before a beneficiary could enroll in the study. When a beneficiary decided to enroll in the study, he or she first completed a competency screener (see Supplemental Appendix B, CAPI Screener) to ensure the ability to provide informed consent. 2 Following the competency determination, the enrollment process involved obtaining written consent to participate in the study, and participating in a Baseline interview. At the end of the Baseline interview, the site RA completing the interview called into Westat’s WesTrax™ system, an automated clinical trial management tool, to obtain the randomization assignment. The RA entered identifying information for him/herself and the beneficiary, and the system responded with a treatment or control group assignment for the beneficiary. The RA informed the beneficiary of the decision and proceeded to close out the interview and discuss next steps with the beneficiary depending upon the randomization assignment. The randomization scheme developed for the study made assignments in equal proportions (1 to 1) using a stratified permuted block randomization with a fixed size of four. The latter ensured balance between the two study arms within the four identified strata, including (1) beneficiary on SSDI < 24 months and not receiving SSI, (2) beneficiary on SSDI ≥ 24 months and not receiving SSI, (3) beneficiary receiving SSI, and (4) study site. For example, with the two treatment arms, a permuted block randomization with block size of four could yield the following allocations: ABAB; ABBA; BAAB; or BABA. 2 No beneficiaries failed the competency screener. Chapter 2: Study Design 2-11 Intervention Benefits and Services The TAP identified seven primary characteristics of the MHTS intervention. Listed in the technical report (p. 14, Aron, et al., 2005), these characteristics included the following: 1. The primary focus of all aspects of the intervention (including clinical services and supports) should be on promoting work. 2. Great care should be taken to (a) establish an accurate medical diagnosis (including secondary and/or confounding physical/mental medical conditions), (b) follow wellestablished treatment guidelines for the given diagnosis, (c) attend to the effects of symptoms/impairments on executive and cognitive functioning rather than medical conditions/diagnoses. 3. The MHTS must adhere to established supported employment principles, specifically as exemplified in the IPS approach. 4. Clinical and employment support must be fully integrated with one another (one set cannot be offered without the other). 5. The intervention should rely primarily on established “evidence-based practices.” 6. All communities participating in the MHTS must serve the same target populations and adopt the same general approach (in other words, they cannot limit their intervention to only one diagnosis, to SSDI applicants only, etc.). 7. Given the challenges likely to be involved in recruiting participants, an important part of the intervention will be an education and training component during an orientation meeting. This meeting could also be used to obtain informed consent from participants and include any additional screening for the target population. The seven characteristics identified above effectively define what SSA meant when it referred to providing SSDI beneficiaries with “access” to evidence-based services. When made operational, the study defined access to services by the study benefits and services described in the paragraphs below. Beneficiaries assigned to the treatment group in each of the study sites received these benefits and services to the extent possible. Diagnostic psychiatric assessment. As recommended by the TAP and required by SSA, the first step in providing full access to care consisted of establishing an accurate and current psychiatric diagnosis for each beneficiary. The diagnostic assessment, typically conducted within the first month of study participation, used the Structured Clinical Interview for DSM-IV (SCID) Axis I Disorders. The 4th edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) organizes all psychiatric disorders and other problems into five different categories (or axes), however as Chapter 2: Study Design 2-12 mentioned, the SCID was used to assess the presence of Axis I disorders only for MHTS participants randomized to the treatment group. Examples of Axis I disorders include: Anxiety disorders (e.g., panic disorder, social anxiety disorder, posttraumatic stress disorder); Mood disorders (e.g., major depression, bipolar disorder); Eating disorders (e.g., anorexia nervosa, bulimia nervosa); Psychotic disorders (e.g. schizophrenia and schizoaffective disorder); Dissociative disorders; and Substance use disorders. Clinical psychologists in each study site received special training on conducting the SCID. General Medical Exam (GME). The GME was an SSA requirement for participation in the treatment group. The GME confirmed the physical fitness of the beneficiary for work. Specifically, the GME policy required a review of recent and past health history and a brief physical examination to ensure that working would not pose harm to the beneficiary. The GME requirement was described during RIG meetings, as a part of the consent process, and at enrollment. Additionally, once a participant was randomized to the treatment group, the GME requirement was again explained during the post-randomization meetings. Failure of the GME or refusal 3 to complete a GME prompted dismissal from the study. The study treated these beneficiaries as non-respondents. These individuals were included in the analysis through weights created to represent their participation in the study. Forty-two beneficiaries in the treatment group became administrative drops due to a failed GME (8) or refusal to complete the GME (34). SMM services. SMM for participants in the treatment intervention consisted of two parts. First, a nurse experienced in working with mentally ill clients (referred to as the NCC) provided a systematic assessment of participant symptoms and functioning to the physician prior to each prescriber visit, or quarterly (whichever period was shorter). These assessments provided a structured view of the participant’s overall response to current medication and cognitive functioning. The second component included an algorithmic approach to prescribing psychiatric medications. The SMM component of the intervention included the Texas Medication Algorithm Project 3 With minimal exception, those treatment group participants refusing to complete the GME were beneficiaries who had also failed to engage in the SMM or IPS services and tended to be non-responsive to outreach attempts by site staff. Chapter 2: Study Design 2-13 (TMAP) medication algorithms regarding schizophrenia and mood disorders. The TAP recommended TMAP as a “rigorous demonstration of the implementation of step-wise decision trees for specific psychiatric diagnoses in public mental health treatment centers” (Aron et al., 2005). In addition to the algorithms, TMAP includes recommendations about dosage, length of treatment, use of concomitant medications, evaluation of non-responders, use of rating scales, and other medication issues. SE services. In line with the TAP recommendation for SE services, the lead criterion for selecting a study site was its demonstrated provision of high fidelity IPS services. Each study site was required to have the infrastructure in place to provide IPS SE services with a ratio of one SE specialist to every 25 beneficiaries. Other behavioral health (OBH) and related services. The intervention package also provided access to evidenced-based behavioral health services over the course of a beneficiary’s enrollment. OBH and related services included but were not limited to psychosocial services, medical services, substance abuse services, housing, legal services, family intervention, financial services, case management, and benefits counseling. While desirable, the ability to provide all of these services was not a requirement for study sites. However, the expectation remained that the study sites provide most or all of these services. Health insurance coverage. The study reimbursed treatment group participants for health insurance premiums over the course of their participation in the study. The majority of MHTS participants had Medicare, Medicaid, private insurance, military insurance, or some form of statelevel insurance. Specifically, the study paid each participant’s portion of his or her monthly health insurance premium as well as the participant’s co-pays for behavioral health-related services. The study provided an insurance coordinator to assist with any changes in Medicare Part D prescription coverage plans to control costs and to ensure the most appropriate plan for each participant’s medication profile. Supplemental health insurance. A goal of the treatment intervention was to ensure that all participants had some form of health insurance coverage that was comparable to Medicare Parts A, B, and D. When supplemental health insurance was required, the study paid the premiums. There were two primary reasons why a participant required supplemental health insurance—either the participant was uninsured at enrollment, or the participant had Medicare, but only Part A. Those participants without health insurance at enrollment were in the 24-month waiting period for Medicare. However, not all participants in the 24-month waiting period were without health Chapter 2: Study Design 2-14 insurance. Some had coverage under plans of other family members. Further discussion of supplemental health insurance appears in Chapter 7 Health Care and Supported Employment Financing. Coordination and payment of participant out-of-pocket and other non-covered expenses. At times, treatment group participants had to pay for services that the study could not pre-pay. In these cases, the participant was required to submit expenses and request reimbursement. In addition, there were expenses not covered by health insurance or by other means. Some examples were studyapproved work-related expenses (e.g., license fees, special work clothes, tuition for special classes), and the cost of transportation to and from providers for treatment participants who would otherwise not have access to care. Criterion number four of the TAP report states that “clinical and employment services must be fully integrated with one another.” Implementation of this criterion was a goal of the study. However, the extent to which this requirement was possible is subject to question. Early in the enrollment process, it became clear that many beneficiaries who wanted to work and enroll in the study also wanted to keep their current (off-site) clinical provider. SSA emphatically supported the views of these beneficiaries. Thus, the study promulgated a policy to that effect and enacted procedures to improve coordination and service integration between on-site and off-site providers. While co-location is not a requirement for full integration of clinical and SE services, it clearly improves the probability for such integration. Further discussions of the issue of on-site versus off-site providers appear in the study implementation chapters (Chapters 5 and 6). Suspend medical Continuing Disability Reviews (CDR). Participation in the study included suspension of the beneficiary’s medical CDR for 36 months from the date of enrollment into the MHTS. The goal of the CDR suspension was to encourage work activity and to remove beneficiary doubt that the study would lead to their removal from the SSDI program for medical reasons. Control Condition Enrolled beneficiaries randomized to the control condition did not receive any study-provided intervention services or benefits, including suspension of the medical CDR. Instead, these participants received a resource manual that listed services and resources for persons with mental illness available both locally and nationally. The manual, copies of which are available on SSA’s website (http://socialsecurity.gov/disabilityresearch/mentalhealth.htm), also included the study Chapter 2: Study Design 2-15 sites. However, in most cases, the participants were not aware of the study site’s name as recruitment activities took place outside the study site offices. The types of services listed in each resource manual included the following: Public and private mental health clinics that serve individuals on a sliding scale; Self-help and consumer support groups; State, county, and local assistance programs that provide payment for medications for uninsured indigent patients; Pharmaceutical company medication assistance programs (sometimes called “pharmaceutical programs for the indigent”); Public health insurance options, including state Medicaid buy-in programs, the Qualified Medicare Beneficiary program, and other federal, state, county, or local programs that provide medical insurance to beneficiaries; and Information on SSA work incentive provisions (such as Ticket to Work), and SSA initiatives designed to help beneficiaries return to work. Participants in the control group also received a modest payment to compensate for their participation in completing interviews over their 24 months of study participation. Each participant received a total of $100 for completion of a Baseline, seven Quarterly, and a final Followup interview. Partial payments accompanied completion of each of the nine interviews. Study Site Staffing To fulfill required research tasks and fully implement the service interventions of the MHTS, each study site employed an RA, NCC, and SE specialist. Each position provided a unique contribution to the study to ensure that all sites achieved all research protocol and evidence-based SMM and IPS requisites. As described in detail in Chapter 5 Implementation of Supported Employment and Other Behavioral Health and Related Services the role of the SE specialist was to ensure the comprehensive implementation of all aspects of IPS SE at the beneficiary level. The role of the NCC, described in Chapter 6 Implementation of the Nurse Care Coordinator Role and Systematic Medication Management, was split between supporting participant treatment, particularly the SMM component, coordinating medical and psychiatric care, and completing research activities related to the treatment intervention. Chapter 2: Study Design 2-16 The primary responsibility for the RA role was to recruit and enroll beneficiaries into the study, and then subsequently administer the eight post-baseline interviews to all treatment and control participants enrolled at that site. The RA was the staff anchor for both treatment and control participants throughout their 24-month MHTS participation. Other RA tasks included compiling, updating, and documenting all participant insurance-related data, including issues related to participant healthcare co-pays and debit card utilization. RAs’ meticulous attention to detail is reflected in much of the data provided in Chapter 3 Enrollment and Participation and Chapter 4 Outcomes. Fidelity Measures The investigators created measures of fidelity for the two key intervention components of IPS SE and SMM. 4 Fidelity refers to the degree of adherence to the evidence-based practice. It answers the question, “To what extent does the implementation match the model protocol?” The fidelity measures used in the MHTS met several needs, including documentation of the nature and degree of implementation, documentation of variations across study sites, and information needed to promote study site improvements. Assessment of fidelity to IPS. A designated team assessed the quality of implementation at each study site using a 15-item measure known as the IPS Fidelity Scale (Bond, Becker, Drake, & Vogler, 1997; see Appendix 5A). The format and assessment procedures for the IPS Fidelity Scale follow the conventions formalized in the National Implementing Evidence-Based Practice Project (McHugo et al., 2007). Each item on this scale reflects a specific element in the practice. A 5-point behaviorally anchored scale provides the rating range. A rating of “5” indicates close adherence to the model. A rating of “1” represents a substantial lack of model adherence. For quality improvement purposes, the study employed Quality Management Program Directors (QMPD) to provide ongoing technical assistance and feedback to sites about their relative attainment of a core element in the IPS model using the item-level fidelity ratings. In addition, the average of the item ratings yielded a total fidelity score that expressed a global picture of overall fidelity. The total fidelity score ranged from one (1) to five (5), with higher scores indicating more faithful implementation. 4 The study included many other quality assurance measures and procedures for various aspects of the study, such as participant use of debit cards for co-pays. A detailed discussion of these measures and procedures appears in Supplemental Appendix A, Study Implementation. Chapter 2: Study Design 2-17 Assessment of fidelity to SMM. The rationale for the inclusion of SMM as part of the MHTS treatment intervention was that it could contribute to improved outcomes by decreasing the deleterious effects of inadequately treated illness symptoms and or medication side effects. Study site staff received training in the principles and materials designed for TMAP, an evidenced-based approach to medication management. Standardized assessments and documentation are used to quantify illness symptoms, explicitly identify medication side effects, and record these observations so as to be accessible to present and future providers. Two MHTS investigators (Drs. Bond and Miller) developed a rating scale to measure quality of prescriber medication management of schizophrenia in a prior project (Taylor et al., 2009). In addition, in consultation with experts in treatment of depression and bipolar disorder, the investigators adapted TMAP scales for assessment of quality of medication management of bipolar disorder and major depressive disorder. Medical records served as the basis for quality assurance rating reviews. The scales were somewhat lengthy and required the rater to search through the medical record for evidence that the prescriber had documented intent and rationale for medication decisions and had attended to patient symptoms and side effects. Trained RAs completed the scales on a 10 percent sample of participant records in each quarter at each site. Westat staff randomly selected site records for review by way of an automated process using the SMS. Outcome Measures The main research question addressed by the MHTS is to what extent delivering appropriate mental health treatment and employment supports leads to better employment, health status, and quality of life among SSDI beneficiaries with schizophrenia or an affective disorder. The primary analyses conducted to answer this question involve a direct head-to-head test of treatment versus control beneficiaries on the most important outcomes of interest in the study, namely employment, health, and quality of life. However, employment was the central focus of the intervention effort, and thus it was the focal point for measuring impact of the intervention. Employment. The primary employment measure of interest was employment rate, specifically the percentage of participants who reported working in a job at any point during the study period. Since competitive employment in regular work settings is one of the seven core principles of the IPS model of SE services, the study also measured the rate at which participants obtained competitive jobs. Chapter 2: Study Design 2-18 Although the primary outcome of interest to the MHTS was employment rate, there were a number of additional characteristics of participant employment experiences of interest to the study. In addition to analyzing outcome measures that considered all jobs a participant reported, the study also analyzed measures that focused only on the main job 5 worked during a study period. Definitions of these additional employment measures, analyzed for both any type of job as well as for competitive jobs only, appear below. 5 Average weekly earnings at main job – Derived from reported weekly earnings averaged over all reported main jobs. Average hours per week at main job – Derived from reported hours worked per week averaged over all reported main jobs. Job satisfaction at main job at study exit – Derived from the level of agreement with 23 statements about personal satisfaction with a job, the work environment, and coworkers. Examples include “I feel good about this job,” “my supervisor is fair,” “my coworkers are easy to get along with,” “working conditions are good.” Number of months to first job – Derived from all jobs reported and indicates the number of whole months elapsed from study start (enrollment date) to the month in which a job was first obtained. Total months employed – Since the study collected longitudinal information directly from participants on their employment activities since enrollment, a derived measure assessed for each month of the study (over the 2-year period) whether participants did not work during the month, worked part of the month, or worked for all of the month. These monthly measures of employment status created the opportunity to construct a measure of total months employed during the study period. Consecutive months of employment at study exit – Derived from the endpoint of the study, this measure computes the number of months of employment counting backwards from the final month of study participation to the month when the participant did not report employment. For example, if a participant did not report employment at any time during the final month of study participation, then he received 0 months credit. If a participant reported no employment during month 17, but did report employment during month 18, and worked in every month until transition out of the study, then he received 7 months credit for this variable. Highest hourly wage – This measure is the highest hourly wage reported across all jobs held during the study. The main job was defined as the job at which a participant worked the longest or worked the most hours during the reporting period for a respective interview. For most participants, interviews were spaced approximately every three months, however, if the previous interview was skipped, questionnaire items related to employment outcomes were asked since the date of the last completed interview (i.e., past six months). Chapter 2: Study Design 2-19 Health status. While participation in employment was the primary outcome of interest for the study, SSA was also interested in knowing whether access to the intervention would result in improved health status. The SF-12 provided the measures of health status for the MHTS. The SF-12 is a well-known self-report health survey empirically derived from the longer SF-36. Designed for use in clinical research, health policy evaluations, and general population surveys, the SF-36 is the “gold standard” in self-reported health assessment (Ware, Kosinski, & Keller, 1996). The SF-12 serves to reduce respondent burden while maintaining acceptable precision. The eight health concepts measured by the SF-12 are: 1. Limitations in physical activities due to a health problem; 2. Limitations in social activities due to a health problem; 3. Limitations in usual role activities due to a physical health problem; 4. Limitations in usual role activities due to an emotional problem; 5. Pain; 6. General mental health; 7. Vitality; and 8. General health perceptions. The SF-12 yields two summary measures—a Mental Component Score (MCS) and a Physical Component Score (PCS). Both the MCS and PCS were assessed at baseline and again at study exit to assess improvements in health and functioning over time, as well as determine if treatment group participants showed more improvement than control group participants. Refer to Appendix 2A for a copy of the SF-12 items and instructions on how to derive the MCS and PCS summary scores (Ware et al., 2002). Quality of life. Previous research suggests that individuals who maintain competitive employment over time report higher quality of life (Bond et al., 2001; Mueser et al., 1997). The MHTS measured quality of life with a single scale item from the Modified Lehman Quality of Life Inventory (QOLIM). The QOLI-M is a shortened version of the well-known QOLI (Lehman, 1988), and has been used in four IPS studies (Bond, 2007; Drake et al., 1999; Drake, McHugo et al., 1996; Mueser et al., 2004). Participants were asked to rate how they felt about their life in general using a 7-point scale: (1) terrible, (2) unhappy, (3) mostly dissatisfied, (4) mixed, (5) mostly satisfied, (6) pleased, and Chapter 2: Study Design 2-20 (7) delighted. Similar to the health status measures described above, the quality of life measure was assessed at baseline and study exit. Secondary outcomes of interest. The primary outcomes of interest to the MHTS to determine the effectiveness of the intervention include employment rate, mental and physical health status, and quality of life. However, the scope of the planned outcomes analyses expanded to include additional secondary outcomes including employment-related measures (e.g., hours worked per week, wages); earnings and income measures; and healthcare service utilization measures. Impacts of the MHTS intervention on health and mental health service utilization levels and patterns are potentially important to SSA from a budgetary and social cost standpoint. There is a substantially reduced net cost to government if the intervention reduces overall costs of these services. In order to estimate the intervention impacts on service use in the MHTS, secondary outcome measures were constructed that compared participant reports of service utilization for several different kinds of healthcare services during the two-year study period. The measures of service utilization studied include number of emergency room visits, number of hospital admissions, number of nights spent in the hospital, number of visits to receive outpatient psychiatric emergency/crisis services, and number of visits to other mental health providers (e.g., clinics, therapists, etc.). Data Collection Procedures The principal source of data used to test the study hypotheses were self-reported measures collected directly from enrolled beneficiaries in the treatment and control groups through computer-assisted personal interviews (CAPI). Beneficiaries participated in nine interviews evenly distributed over the 2-year period of study participation. The initial Baseline interview occurred at enrollment, before randomization to a study condition. All Baseline interviews were conducted in person. Each participant subsequently completed seven Quarterly interviews covering the interim months of study participation. These interviews were shorter than the Baseline interview and collected only the most pertinent information needed to assess the study outcomes. These interviews were either in person or over the telephone, depending upon the circumstances of the particular participant. At the end of the two-year study period, participants participated in a final Followup interview. Conducted either over the telephone or in person, this interview mirrored the more extensive Baseline interview. Chapter 2: Study Design 2-21 The Baseline interview included questions about the participant’s demographic information, work history, benefit status, attitudes toward work, health care coverage, health care utilization, and prescription medication use. A final section of the Baseline interview included administration of a paper-pencil version of the Digit Symbol Test. The test is a reliable and efficient measure of information processing efficiency (Dickinson, Ramsey, and Gold, 2007). It served as a brief measure of enrolled beneficiaries’ cognitive functioning. The seven Quarterly interviews (conducted every three months) included questions about each participant’s contact information, demographics, health status, alcohol and substance use, employment outcomes and income, job seeking behavior, health care service utilization, and quality of life. Quarterly interviews were not the same across the treatment and control groups. In order to reduce respondent burden and minimize non-response, the seven Quarterly interviews for participants in the control group included only the same battery of items covering employment and healthcare service utilization. Additional questions asked of treatment group participants included questions related to health and functioning, and alcohol and drug abuse. The final Followup interview collected once again particular demographic information about the participant, and information about employment outcomes and income, health status, alcohol and substance abuse, attitudes toward work and SSA benefits programs, health care service utilization, and quality of life. Interviews with the treatment group also included questions about participant satisfaction with the services they received during the study. The Baseline and final Followup interviews were essentially the same with the exceptions that the Baseline interview collected information on work history prior to study enrollment, and the final Followup interview collected information on employment outcomes since the last completed interview. Table 2-2 provides an overview of the content and timing of these self-report interviews. The Supplemental Appendix includes a copy of the questionnaires that were developed for the Baseline, Quarterly, and final Followup interviews. Chapter 2: Study Design 2-22 Table 2-2. Content domains for the Baseline, Quarterly, and final Followup interviews Measures Demographic Information Baseline Quarterly1 Followup Work History Employment Outcomes Income Review Health Status (SF-12) Quality of Life (Item from Lehman QoL Inventory) Alcohol and Drug Use (Addiction Severity Index) Health Care Coverage Health Care Service Utilization Attitudes Toward Work 1 2 2 Control condition participants received only the questions about Employment Outcomes and Health Care Service Utilization in the Quarterly interviews. Only control condition participants received questions about Health Care Coverage. Interview Completion Rates Table 2-3 shows the interview completion rates by month and study group. The overall interview completion rate for the followup interviews was 84 percent. The overall completion rate for the treatment group was 81.6 percent, ranging from a low of 78 percent in Quarter 7 to a high of 86 percent in Quarter 1. The overall completion rate for the control group was 86 percent, with a range of 84 percent to 89 percent. There were fewer completions by the treatment group than by the control group at every quarter after baseline. The difference in attrition rates between arms grew steadily from Quarter 1 to followup and was statistically significant for every quarter after Quarter 5. As discussed previously in this chapter, participants in the control group were paid a nominal fee for the Baseline interview (after being assigned to the control group) and small payments per interview beyond the baseline. Due to the rich nature of the intervention, treatment group participants did not receive payments for any interviews. Chapter 2: Study Design 2-23 Table 2-3. Interview completion rates by quarter and study group Interview Baseline Treatment (N=1,121) freq % 1,121 100.0 Control (N=1,117) freq % 1,117 100.0 Quarter #1 Quarter #2 Quarter #3 Quarter #4 Quarter #5 Quarter #6 Quarter #7 Followup Total** 962 942 920 918 909 890 876 902* 7,319 988 964 956 947 956 942 933 991* 7,677 85.8 84.0 82.1 81.9 81.1 79.4 78.1 80.5 81.6 88.5 86.3 85.6 84.8 85.6 84.3 83.5 88.7 85.9 Total (N=2,238) freq % 2,238 100.0 1,950 1,906 1,876 1,865 1,865 1,832 1,809 1,893 14,996 87.1 85.2 83.8 83.3 83.3 81.9 80.8 84.6 83.8 p-value 0.0627 0.1308 0.0239 0.0667 0.0043 0.0024 0.0012 0.0000 The figures on completion rates include five interviews (2 treatment group and 3 control group) found to be unusable due to a laptop computer malfunction. ** Excludes baseline. * Data Preparation and Adjustments As is typical of large controlled trials, events occur that result in participants failing to complete all phases of a study. Some participants drop out of the study, others simply do not return for various activities, and some participate only sporadically. Beneficiaries in the MHTS were no different. The study experienced participants who withdrew for a variety of reasons (including they left the country, became disillusioned or lost interest, etc.) or participants who died. The study dropped some in the treatment group because they failed their GME or refused to complete it. All of these situations give rise to concerns about attrition bias and reduced precision and power. The Baseline interviews and every Quarterly interview prior to attrition provide substantial information about the types of people who stay in the study or left the study before the end of 24 months. The follow-along interview patterns also provide information about those beneficiaries who participated only sporadically. Of course, it is unknown what events following the last interview of a beneficiary who stopped participating led him or her to the decision to stop participating. There is concern that participants who dropped out of the treatment group were less motivated to find employment than those who remained in the study. If true, comparisons between treatment and control groups may be exaggerated relative to an ITT analysis without attrition. Two data adjustment methods addressed this issue. First, participants who did not complete at least two post-baseline interviews were “non-respondents” and these participants were not included in Chapter 2: Study Design 2-24 the analysis. Instead, weights were created to adjust for their removal. Second, imputations provided a solution for other participants with substantial missing data. Weighting adjustments for severe non-response and imputation for mild non-response were employed for specific reasons. Weighting is better than imputation at avoiding biases in complex multivariate relationships, but it makes very light use of the collected data on cases that are classified as non-respondents. Imputation makes very strong use of partial data, and therefore yields better estimates of marginal means and differences between pre-identified subgroups than can be obtained through weighting. Thus, the rationale for weighting was that the limited participation of non-respondents resulted in insufficient partial data to justify imputation-induced biases in complex multivariate analyses. The rationale for imputation was so much partial data existed with the respondents that using weighting would result in unacceptable variance increases. Below is a more detailed description of the methods for these two procedures. Weighting Adjustments As indicated in the preceding section, non-respondents were participants who did not complete at least two interviews beyond the Baseline interview. There were 154 “non-respondents” in the study with less than two post-baseline interviews. (These 154 non-respondents include those treatment cases that were dropped from the study on account of a failed or incomplete GME.) However, the non-respondents group also included another five participants who completed two or more interviews, but whose data were corrupted due to a malfunctioning interviewer laptop computer. The weights for these 159 participants were set to zero and the remainder of the study sample (N=2,079) was weighted to reflect these individuals. An additional 24 participants (11 treatment group participants and 13 control group participants) who died during the study completed more than two interviews after baseline. Although not considered to be “non-respondents,” these individuals were excluded from the analyses included in this report. Thus, there were a total of 2,055 participants included in the analytic sample used for this report. The weighting process involved two steps. First, demographic characteristics and baseline measures potentially associated with likelihood of study withdrawal defined 20 subgroups, across which studywithdrawal rates widely varied. Five of these 20 had perfect response rates. Response rates in the other 15 dipped as low as 64 percent. The 20 subgroups also varied in size from 31 to 447 participants. Separate subgroups were defined for the treatment and control groups, so as to avoid Chapter 2: Study Design 2-25 transfers of weight across the two groups. Some of the baseline variables related to attrition included: Receipt of Temporary Assistance for Needy Families, Desired number of work hours per week, Desired pay, Perceived quality of life, Number of children in the household, Number of adults in the household, Hospital utilization, Outpatient mental health utilization, Use of psychotropic drugs, Current work status (at baseline), and Severity of substance abuse. Second, calculations with the following formula provided non-response adjustment factors within each subgroup: Adjustment factor = Number of enrolled participants Number of responding participants All enrolled beneficiaries received an initial weight of 1.0. The adjustment factors were applied to the initial weights of respondents to increase their weight upward for the non-responding beneficiaries. Imputation There are two key reasons why imputing missing values is critical in this study. Most statistical procedures drop entire cases where any of the variables required for a particular analysis contain missing values. For simple marginal means, two-way tables, and bivariate correlations, this is sometimes an acceptable practice, but for analyses that draw on many variables simultaneously, a handful of cases with one variable missing and another handful of cases with a different variable missing soon add up to large numbers of dropped cases. For the MHTS, many of the outcome Chapter 2: Study Design 2-26 variables require data across all the waves of data collection, and so drop-case rules would result in the loss of a large quantity of data. This loss of data would have resulted in reduced precision and power, and potential attrition bias. Imputation helps to correct both of these problems. Many records deleted because of missing data actually have substantial partial data that can be used to impute the values that are missing. Using this partial information helps to improve power and precision. The issue of attrition bias is particularly important in studies with an ITT model (such as the MHTS) with the expectation that the analysis accounts for all cases. While it is generally impossible to understand the full details of the missing data mechanism, the study did gather some valuable clues. As is discussed in the final section (Imputation Procedures) of Appendix 2B, subjects who missed one or more waves but participated in the final Followup interview had many more hospitalizations, and subjects who dropped out of the survey for good at some point after baseline had poorer job-seeking results than those who persisted in the survey. Since the intervention tends to reduce hospitalization and increase the probability of finding a job, a simple complete-case analysis would probably exaggerate treatment effects. A well-done imputation procedure can reduce the risk of such biases due to informative non-response. Weighting can also reduce the risk of bias, but discards much more partial data and is thus far less efficient. Imputation inflates some differences and deflates others. There is no way to determine whether the differences based on imputed values are better or worse for any particular analysis than those obtained with only reported data. They will certainly be different. At the same time, it is most likely that variances on estimated differences will be smaller with imputation than without. For some analyses, the reduction in variance will be substantial, while for others it will be modest. There is a tendency to weaken associations between variables with imputation. This would suggest that it is more likely for imputation in this case to have deflated treatment effect estimates than to have inflated them. Even where this happens, the reduction in variance may be enough to offset the depression of the estimated effect so that one still obtains a more significant test statistic for the treatment effect. The methodology used in the imputations is described in Krenzke and Judkins (2008). The methodology combined traditional methods for hot deck imputation with modern model-dependent chained parametric procedures. Several research papers have shown performance better to the best alternative software systems available. The post-baseline interviews (of which there are eight—seven Quarterly and a final Followup) are the source of all outcome data for the study. The most Chapter 2: Study Design 2-27 important imputations are for those participants who did not complete a final Followup interview but did complete the Baseline interview plus at least two post-baseline interviews. For these participants, imputations included all of the information that should have been collected after the last interview that they completed. In addition, there were imputations for a small number of missing values for a limited set of items for participants who missed one or more quarters but who returned to complete the final Followup interview. For these participants, the imputation included income components for the missed quarters, attributes of the main jobs held in the missing quarters (provided that the catch-up interviews 6 reported employment during the period), and (for the treatment group only) updated demographics and substance abuse information. Fifty-eight percent of the study sample required no imputation at all. Nine percent required imputing the final Followup round and possibly additional quarterly rounds. Thirty-three percent required imputation of only scattered items not subject to extended recall. These figures are typical for a complex study with many moving pieces in the data collection system. Marker, Judkins, and Winglee (2001) report on the imputation of costs and payments for medical events in the Medicare Current Beneficiary Survey. In that survey, 96 percent of persons in the best-reporting domain of interest had to have at least one cost or payment amount for at least one medical event over the course of a year imputed. Without the imputation, it would have been impossible to assemble reasonable statistics about person-level costs and payments. Appendix 2A contains a complete description of the imputation procedure and more information on the likely impact of the imputation for a few analyses. While some analyses were conducted separately for cases with nearly complete records and cases requiring major imputation, not much time was devoted to this effort. The results definitely are sensitive to the imputation procedure, but given that non-respondents appeared to be sicker and less successful in their job hunts than were respondents, there is little reason to pay attention to analyses based only on good respondents. Statistical Procedures Data analyses presented in this report use the weights described the previous section, unless otherwise specified. Statistical Analysis Software (SAS) procedures for analyzing data were used to analyze the weighted data, notably SURVEYFREQ, SURVEYREG, and SURVEYLOGISTIC. 6 During each interview, participants were asked if they worked since the date of the last completed interview allowing for the collection of complete data on employment outcomes even if interviews were skipped. For example, if they had skipped the previous interview (three months ago), they were asked about work in the past six months as opposed to the normal three months. Chapter 2: Study Design 2-28 These procedures adapt standard statistical procedures to incorporate unequal weighting adjustments for non-response. Chi-square tests served to test for differences between treatment and control groups for categorical data. In cases where the outcome measure resulted in continuous data, an analogue of the Wilcoxon test served as the test for differences between treatment and control groups. Specifically, data were ranked in the combined study groups then weighted t-tests (computed in SURVEYREG) were used to test for differences between ranks in treatment and control groups. Since most measures considered in many analyses were non-negative and some had clear outliers, normality was not a reasonable assumption. The Wilcoxon test is a well-known solution to overcome the need to assume normality in the distribution of outcome scores (Conover and Iman, 1981; Zimmerman, 1992). The study had the potential for exploring outcomes in substantial depth, given the rich nature of the data collected over the 2-year study period. Among the many classes of variables that could account for particular outcomes were clinical variables (e.g., diagnosis or cognitive functioning), demographic variables (e.g., age and education), baseline characteristics of participants (such as health status and recent work history), among others. In an attempt to provide a better understanding of the outcomes, the study required use of multivariate techniques to explore relations among large numbers of potential predictor variables. Several multivariate analyses used multiple logistic regression technique with binary dependent variables, such as obtained employment (yes vs. no), to explore and improve understanding of the outcomes. The study employed the zero-inflated negative binomial regression technique to analyze the outcome number-of-months employed (in Chapter 4) and the count of healthcare service utilization variables (in Chapter 8). The rationale for using this estimation method was based primarily on the distributional characteristics of the dependent variables, in particular, the facts that they are all count (integer) data and that they are characterized by a large mass of observations at zero. The zeroinflated negative binomial regression model assumes that the data are a mixture of two separate data generation processes—one generates only zeros and the other is a negative binomial data-generating process. For the quality of life outcome measure, an ordered logistic regression model was run. This model estimates categorical outcomes and expresses probabilities of each outcome as subtraction of cumulative probability. All bivariable models included a forced variable (called the “treatment dummy”), indicating whether individual cases referred to a treatment intervention case or a control case, along with each of 15 independent (predictor) variables. Candidate variables for the multivariable model were those that Chapter 2: Study Design 2-29 showed a marginal association with the dependent variable at the 0.25 level of significance after adjustment for the treatment dummy. The multivariable, multiple logistic regression procedures included both backwards and stepwise regression to ensure that disparate models did not occur. Discussion of final models focused on any independent variables showing a 0.05 level of statistical significance. In an attempt to provide an even better understanding of the outcomes, investigators also used multivariate techniques to explore relations among an additional 13 potential predictor variables that were applicable for treatment group participants only. Additional predictor variables included for treatment group participants only included dummy variables indicating use of some other behavioral health or related service (e.g., substance abuse treatment, case management, social skills training, etc.) at some point during the 24-month study period; variables measuring the extent to which a participants was engaged with the IPS and SMM components of the intervention; size of study site (measured by the number of clients served at the site); and the site fidelity scores. Data Limitations The study results rely almost exclusively on the self-report data collected in the eight post-baseline interviews. The study RAs, NCCs, and the QMPDs provided additional data on participants in the treatment group that served a number of treatment group only analyses, and some analyses relied upon administrative data provided by SSA. However, the self-report data collected in interviews with the treatment and control groups served to answer the primary research question concerning the impact of the intervention on employment, health, and quality of life. Data provided from all of these sources have particular limitations that require acknowledgement. The study results requiring use of SSA administrative data presented in this report are limited to the accuracy of that data. SSA’s administrative data did in fact contain errors of accuracy, much of it due to the currency of the data. For example, during and after the enrollment process, a few beneficiaries revealed that they had legal guardians, thus making them ineligible for the study. Yet the administrative data did not indicate this fact, more than likely because the data were at best several months old when recruitment efforts began. This example serves to remind the reader that the administrative data contain some inaccuracies. Chapter 2: Study Design 2-30 The study development process resulted in the production of 14 manuals (e.g., recruitment and enrollment, conducting RIG meetings, documenting MHTS insurance coverage) explicating precise procedures for documenting and carrying out study activities. In addition to the manuals, the study provided training, ongoing discussion groups, and periodic monitoring to check on and improve reliability. However, despite these extensive efforts and precautions, RA, NCC, and QMPD errors in accurate recording or omission inevitably occurred. Limitations associated with the interview data are likely the greatest concern to SSA and others interested in the results of the MHTS. Construction of the CAPI instrument used for the self-report interviews followed standard procedures for developing content, translating the interview content into programming specifications (including appropriate skip patterns), testing the CAPI, and deploying the instrument into the field. Each study site received a laptop computer, assigned to its RA(s), preloaded with the interview and customized with a limited amount of beneficiary information for individuals residing in the respective study site catchment areas (study ID and name). A CAPI interviewer manual and a two-day training session provided RAs with the preparation and practice necessary to conduct the interviews. However, because of a 5-month delay in the study start, RAs received refresher training and practice closer to the start of recruitment. RAs also participated in telephone conference calls to receive ongoing support and technical assistance related to administering and completing the CAPI interviews. RAs transmitted interview data from their CAPI laptops to Westat’s home office daily. Chapter 2: Study Design 2-31 Chapter 3 Enrollment and Participation The intervention package provided to treatment group participants in the Mental Health Treatment Study (MHTS) combined a number of interventions, three of which included (1) the Individual Placement and Support (IPS) model of supported employment (SE); (2) systematic medication management (SMM); and (3) enhanced insurance coverage for behavioral health services. Based on prior evidence with these interventions, one might expect them to yield positive employment and health outcomes for Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or an affective disorder. However, no tests exist with these interventions, either individually or in combination, on a target population primarily comprised of this group of beneficiaries. Thus, the focus of the MHTS was to carry out such a test to determine the effectiveness of this intervention package and thus the likely consequences of a Social Security Administration (SSA) policy initiative to make this package of interventions available on a national basis. Of course, to understand the consequences of making the MHTS interventions available, the initial question that needed addressing was, if offered, who will take advantage of these services? This question was critical to understanding the generalizability of findings from the MHTS. It was also critical to understanding the policy implications for SSA and other agencies serving individuals with severe mental impairments, and it emphasized the concern that SSA had for providing SSDI beneficiaries with access to quality mental health services. This chapter addresses the extent of participation in the MHTS from the SSDI rolls. It is worth noting that not everyone eligible to participate in the MHTS agreed to enroll in the study, so it is not appropriate to draw inferences about all SSDI beneficiaries with schizophrenia or an affective disorder from the results presented in this report. However, the characteristics of those SSDI beneficiaries that did agree to participate and those who did not needs to be studied to help design strategies to increase participation in the event that the SSA or another agency intends to implement the treatment intervention on a wider scale. This section presents extensive comparisons between the characteristics of beneficiaries who enrolled in the MHTS to those of beneficiaries who also had the opportunity but chose not to enroll. These analyses provide SSA with an understanding of the universe to which one might appropriately generalize the findings of the MHTS. In addition, this understanding provides clues as to who might agree to participate in a broader implementation of the intervention. Chapter 3: Enrollment and Participation 3-1 A second set of analyses characterizes the beneficiaries who enrolled in the study, and examines the similarities and differences between participants assigned to each arm of the study. A fundamental expectation of the random assignment process was equivalence between the treatment and control groups on variables important to the investigation. Using data collected at baseline (which was prior to randomization), an assessment was made of the success in creating two equal groups of participants. This assessment included a comparison on demographic variables, SSDI program variables, and key measures of employment, health status, and income. Are There Differences Between SSDI Beneficiaries Who Enrolled and Those Who Did Not Enroll in the MHTS? Since the MHTS was a randomized controlled trial, eligible beneficiaries only had a 50 percent chance of receiving the intervention package, whereas an ongoing program of services would not have this feature of a randomized assignment. Even when interested in participating, some eligible beneficiaries may have declined knowing they only had a 50 percent chance of receiving the intervention services offered to the treatment group. The approach to answering this important policy question consisted of the following steps: 1. Assign the 61,530 beneficiaries residing in the catchment areas of the 23 study sites to potential enrollment groups based on knowledge gleaned from the recruitment effort. 2. Prepare a test and validation sample. 3. Estimate predictors of enrollment using logistic functional form and maximum likelihood estimation on the test and validation samples. Assignment of Beneficiaries to Enrollment Groups As described in Chapter 2, the recruitment process involved several attempts by the study Research Assistant (RA) in each study site to contact beneficiaries in their catchment area. First, a letter was sent to the beneficiary’s address of record. Following the letter, one or more telephone calls were made to make verbal contact with the beneficiary. Interested beneficiaries were asked to attend a Research Information Group (RIG) meeting in person to obtain further information about the study. A beneficiary could refuse the study at any of the three steps. However, to join the study, a beneficiary was required to attend at least two RIG meetings. Chapter 3: Enrollment and Participation 3-2 The RA at the sites following release of beneficiaries’ names and contact information documented all key recruitment activities in the web-based Study Management System (SMS). These data served as the basis for classifying beneficiaries into one of three potential enrollment groups—potential enrollee, possibly potential enrollee, or not a potential enrollee. Figure 3-1 presents the logic model used to assign beneficiaries across all 23 study sites to a potential enrollment group. Using the information recorded in the SMS that documented the RAs recruitment efforts, MHTS investigators were able to follow a beneficiary’s path throughout the recruitment process. The total eligible population identified from SSA administrative records as meeting the inclusion criteria and available to the study sites for recruitment into the study was 61,530. Of the 61,530 SSDI beneficiaries that were included in the list sample received from SSA, RAs spoke with only 17,642 beneficiaries. Of those 17,642 beneficiaries contacted via telephone to determine their interest in attending a RIG meeting to learn more about study participation, only 3,971 agreed to attend a RIG meeting, with 2,238 beneficiaries subsequently enrolling into the MHTS after attending at least two RIG meetings. The figure shows the final disposition of those who did not attend a RIG meeting, as well as the final disposition for those who attended a RIG meeting but did not enroll into the study. Figure 3-1 also presents the final disposition at the end of all recruitment efforts for beneficiaries who never had an opportunity to hear about the MHTS, for a variety of reasons. In some cases, RAs screened out beneficiaries who were found to be ineligible before recruitment efforts began (N=3,614). In other cases, RAs were never able to reach beneficiaries over the telephone to invite them to a RIG meeting, in spite of various call attempts (i.e., labeled as “max call attempts”). RAs were not able to locate almost one-quarter of beneficiaries included in the sample (N=14,397). Although SSDI beneficiaries receive monthly payments from SSA, the majority of beneficiaries receive their cash benefit in some electronic form (e.g., direct deposit), as opposed to receiving an actual check in the mail; so in many cases, beneficiary mailing addresses were out-of-date. Toward the end of the recruitment period, some beneficiaries were not contacted via telephone even though an introductory letter was mailed to them (i.e., labeled as “letter sent, no followup”) because the site’s recruitment target had been met. Some RAs preferred to use hard-copy materials to manage their recruitment tasks, and then subsequently document the results in the SMS. However, not all RAs were diligent about documenting their efforts in the SMS; at the end of the recruitment period there was no electronic call record in the SMS for 10,171 beneficiaries. Chapter 3: Enrollment and Participation 3-3 Figure 3-1. Logic chart for assigning beneficiaries to enrollee group classifications All beneficiaries (N=61,530) Letter mailed (N=57,364) No letter mailed (N=4,166) Did not speak to beneficiary (N=39,722) Spoke to beneficiary (N=17,642) Attended RIG meeting (N=3,971) Did not attend RIG meeting (N=13,671) Ineligible (N=1,499) Enrolled Refused Ineligible Other2 (N=2,238) (N=684) (N=161) (N=892) Potential Enrollees (N=15,982) Test (N=7,991) Validation (N=7,991) No call record (N=10,171) Max call attempts (N=3,592) Max call attempts (N=4,022) Ineligible1 (N=3,614) Refused No followup (N=7,419) (N=728) Possibly Potential Enrollees (N=13,763) Test (N=6,882) Letter sent, no followup (N=7,948) Could not locate (N=14,397) Not Potential Enrollees (N=31,785) Validation (N=6,881) 1 These beneficiaries were found to be ineligible in a variety of ways, including reviews of study site records revealed the beneficiary was receiving SE services at the site; a proxy answered the recruitment call and indicated the beneficiary has a legal guardian, etc. 2 The “Other” group who attended a RIG meeting includes beneficiaries who fall into one of three result codes: No longer locatable (N=74), Max calls (N=731), and Unknown (N=87). Using the information recorded in the SMS indicating the final disposition of each recruitment case, MHTS investigators were able to classify all 61,530 beneficiaries included in the original sample that was received from SSA into one of three potential enrollment groups. The criteria for classifying a beneficiary as a not potential enrollee were (1) absence of a contact date (most likely because release of the beneficiary information never occurred) or (2) presence of a result code that indicated that the beneficiary was “ineligible,” “activated, not recruited,” or “not locatable.” The criteria for classifying the beneficiary as a potential enrollee were (1) presence of a contact date or (2) the result of a contact attempt coded as either “completed” or “refused,” or (3) there was a recorded “spoke to beneficiary date.” The criteria for classifying a beneficiary as a possibly potential enrollee were (1) presence of a contact date and (2) recorded result was “unknown” or “max calls.” Beneficiaries contacted and spoken to by study site staff comprised the 15,982 potential enrollees. These individuals had at least one opportunity to hear about the study, even though they may or may not have attended a RIG meeting. However, nearly a quarter of them (24.8%) attended a RIG Chapter 3: Enrollment and Participation 3-4 meeting before making a decision to enroll in the study. The 31,785 not potential enrollees included beneficiaries who had no opportunity to participate in the study or (presumably) even know it existed. Those beneficiaries considered to be out of the pool of potential enrollees included individuals for whom the SMS indicated they were “ineligible” (beneficiary was deceased, moved out of the catchment area, was already receiving services at the study site in their area, etc.), or not recruited because the study site “could not locate” the beneficiary. The 13,763 possibly potential enrollees were a unique group of beneficiaries for whom no concrete information about their interest in the study existed. The largest beneficiary group in this category (10,171) had a final enrollment result code of “no call record,” suggesting that there was no indication of a date on which a recruiter actually spoke to the beneficiary. The other group comprising the possibly potential enrollees included 3,592 beneficiaries with a final enrollment result code of “max call attempts” (again with no indication of a “spoke to beneficiary” date). In these cases, the RA was able to identify a phone number that applied to the beneficiary, and in some of these cases, recruiters indicated that a telephone message was left, but there was no indication that anybody spoke to the person or that the beneficiary ever received the message. However, it was not technically correct to classify the beneficiary as ineligible or not locatable. Thus, it was appropriate to create a third category, namely possibly potential enrollee. Preparation of Test and Validation Samples The goal of the enrollment analysis was to identify predictors that could convey what characteristics of beneficiaries were indicative of study enrollment. To achieve this end, the analysis included only two of the three groups of beneficiaries, those classified as potential enrollees, and those classified as possibly potential enrollees. Potential enrollees were those beneficiaries for whom there was clear evidence that they had received some personal contact about joining the study and either enrolled or did not. At least they knew that the study existed and that they were eligible for it. Possibly potential enrollees included beneficiaries for whom such clear evidence of personal contact was lacking, but for whom clear evidence of not receiving a personal contact was also lacking. Of course, none of the beneficiaries in this second group in fact enrolled in the study. However, the investigative team could not ensure that these beneficiaries never received an opportunity to know about the study. The not potential enrollee group was set aside for this analysis because the vast majority of them had no opportunity to know about the study. The others were not eligible. Chapter 3: Enrollment and Participation 3-5 The groups of potential enrollees and possibly potential enrollees were randomly split into two roughly equal halves. One-half became a test sample, and the other a validation sample (See Figure 3-1). The subsequent analyses used the two test samples to explore predictors of enrollment. The validation sample corroborated the results. Recognizing that the results between the two sets of data would likely vary, only those predictors with a significant relationship to the dependent variable (actual enrollment into the study) were retained in the validation sample results. Estimation of Regression Models Predicting Enrollment Regression analysis was used to identify the factors that predicted who enrolled in the study (and who did not). Exploratory logistic regressions were estimated with the test samples (of potential enrollees and of potential plus possibly potential enrollees) using a variety of estimation methods, including maximum likelihood with separate site intercepts, conditional (fixed-effects) maximum likelihood, maximum-likelihood with clustering of errors (by site), and random-effects logistic regressions (with random intercepts for each site). A variety of explanatory variables were included in the test regressions, including (but not limited to) those shown in Table 3-1. (See Appendix 3A for a list and description of all variables included in the regression models. See Appendix 3B for descriptive statistics from the regression analysis cases.) A number of different variables were included in preliminary versions of the exploratory regressions on the test sample but were dropped from the final exploratory regressions, including the dummy variables for year of recruitment, the variables pertaining to serious mental illness diagnosis from SSA’s Master Beneficiary Record (MBR), and county unemployment rates. Other variables tested, but not included in the final exploratory regressions (reported below), included additional proxy socioeconomic variables from (1) the Census 2000 SF3 files and the American Communities Survey files, (2) alternative definitions for Disability Control File earnings variables, and (3) alternative functional forms for variables such as age and distance from the study site. Note also that the participation regressions did not include any of the variables from the Baseline interview since these were only available on beneficiaries who did in fact decide to enroll in the study. Chapter 3: Enrollment and Participation 3-6 Table 3-1. Results of logistic regressions on MHTS participation (Enrolled = 1)1 Test sample Validation sample Potential plus possibly potential enrollees (n=14,513) p-value Potential enrollees (n=7,933) Coeff p-value Potential plus possibly potential enrollees (n=14,637) Coeff p-value -0.002 <0.001 -0.002 <0.001 -0.002 <0.001 -0.002 <0.001 0.276 0.026 0.376 0.002 -0.039 0.749 0.006 0.957 -0.103 0.009 -0.126 <0.001 0.004 0.922 -0.002 0.960 Gender (Male) 0.221 0.001 0.124 0.060 0.135 0.054 0.091 0.176 SSI -0.194 0.033 -0.182 0.037 -0.194 0.039 -0.216 0.017 Predictor variable Months on Rolls Age (days) x (Age)2 x 10-3 10-7 Rep Payee Potential enrollees (n=7,815) Coeff p-value Coeff -0.487 <0.001 -0.760 <0.001 -0.573 <0.001 -0.806 <0.001 0.038 <0.001 0.053 <0.001 0.007 0.457 0.027 0.004 -0.045 0.002 -0.055 <0.001 0.003 0.860 -0.014 0.327 Some College (census tract) 0.542 0.023 0.447 0.049 -0.130 0.597 -0.108 0.648 Median Income (census tract) -0.374 0.005 -0.285 0.025 -0.062 0.651 0.023 0.863 Race (Black) 0.286 0.001 0.223 0.007 0.274 0.002 0.292 <0.001 Ln of Distance from Site -0.114 0.005 -0.141 <0.001 -0.111 0.008 -0.119 0.003 Had Active Ticket (w/in 90 days) 0.290 0.193 0.216 0.135 0.536 0.118 0.575 Had Active Ticket (ever) 0.839 <0.001 0.857 <0.001 0.725 <0.001 0.841 <0.001 Trial Work End Date (10 yrs.+ ago) 0.398 0.004 0.367 0.006 0.523 <0.001 0.517 <0.001 Trial Work End Date (5-10 yrs. ago) 0.336 0.014 0.296 0.024 0.139 0.352 0.130 0.367 Trial Work End Date (0-5 yrs. ago) 0.309 0.062 0.334 0.029 0.325 0.072 0.229 0.181 Trial Work End Date (0-3 yrs. post-recruitment date) 1.183 <0.001 1.081 <0.001 0.726 0.013 0.839 0.002 Sq. Root of Reported Earnings (1-6 mos. prerecruitment) -0.014 <0.001 -0.019 <0.001 -0.020 <0.001 -0.022 <0.001 Sq. Root of Reported Earnings (7-23 mos. pre-recruitment) x 10-1 0.038 0.014 0.029 0.036 0.013 0.002 No Earnings Report (1-6 mos. pre-recruitment) -0.657 0.002 -0.653 0.001 -0.914 <0.001 -0.931 <0.001 Constant 0.129 0.937 -2.443 0.112 0.040 -2.246 Time to Recruit x (Time to 10-1 Recruit)2 x 10-4 1 The 0.294 0.435 0.981 0.922 0.158 reduced Ns are a result of the regression analysis excluding beneficiaries with missing or undefined values for variables included in the regressions, or with negative values that were clearly erroneous. Such exclusions constituted less than 10 percent of all cases in each of the groups shown above. Chapter 3: Enrollment and Participation 3-7 Variables were retained in the exploratory regressions based on their p-values and (in select cases) the consistency of the regression results with prior expectations, evidence, or theory. Since results for individual explanatory variables were not always very similar between the potential enrollee exploratory regressions for the test sample and those for the potential enrollees plus possibly potential enrollees, the general approach was to retain variables that met the criteria in at least one of the two regressions for testing in the validation sample. Table 3-1 shows the final versions of the exploratory regressions on the test sample. Reported regression results include the coefficients for each explanatory variable. 1 All of the explanatory variables except one have two-tailed p-values < 0.1 for both potential enrollees and potential enrollees plus possibly potential enrollees. The one exception is Had Active Ticket (within 90 days) 2, which has the expected positive sign but only achieves significance at the 0.19 level based on a two-tailed test in the potential enrollee group. Note, however, that these results for the test sample overstate true significance levels since they do not correct for the many other exploratory regressions run with alternative sets of explanatory variables included in them. In contrast, p-values for the results in the validation sample, shown on the right side of Table 3-1, are not biased downward since exploratory regressions did not precede tests on the validation sample. Not surprisingly, these p-values are not consistently as low as those in the test sample are. The variables that are least significant (i.e., have the highest p-values) in the validation sample are the socioeconomic variables from the census tract data (Some College and Median Income), age (and age2), and the variables reflecting exposure to the recruitment variables (Time to Recruit and Time to Recruit2) particularly in the regression for potential enrollees in the validation sample. Three nonsignificant variables relating to prior work-related activities include the following: (1) the variable for an active Ticket to Work (TTW) in the 90 days prior to the beneficiary’s recruitment date (Had Active Ticket w/in 90 days), (2) the variable for a trial work period ending between 5 and 10 years prior to the recruitment date (Trial Work End Date 5-10 yrs. ago), and (3) the square root of gross earnings plus net self-employment earnings reported to SSA summed over months 7 to 23 prior to the recruitment date (Sq. root of Reported Earnings 7-23 mos. pre-recruitment). 1 One cannot directly interpret each reported coefficient as a marginal (ceteris paribus) effect on the probability of enrolling in the MHTS with the logistic functional form. However, a good approximation to this marginal effect, MEi for the ith X variable, is computed from the expression MEi=bi x q x (1-q), where bi is the reported coefficient and q is the fraction of the beneficiaries in the regression sample who in fact enrolled. 2 Had Active Ticket (within 90 days) refers to beneficiaries who activated their Ticket in the Ticket To Work (TTW) program, see http://ssa.gov/work/aboutticket.html. Chapter 3: Enrollment and Participation 3-8 Apart from the differences mentioned above, the general pattern of results for the validation samples is very similar to that for the test sample. Demographic variables such as gender and race were very strong predictors of enrollment outcome. Beneficiaries on the SSDI rolls for a longer time (Months on Rolls), beneficiaries receiving both SSDI and Supplemental Security Income (SSI), and those having a representative payee (Repayee) were much less likely to enroll. As expected, distance from the beneficiary’s residence to the MHTS site (Ln of Distance from Site) was a strong deterrent to enrollment. The general pattern of results for the work and work-related activity variables indicated that holding other factors constant, beneficiaries with a very recent history of work activity (e.g., Trial Work End Date 0-3 yrs. post-recruitment date) were less likely to enroll; this was also true for those with no prior history of an active TTW (Had Active Ticket ever) or a trial work period (Trial Work End Date 0-3 yrs. post-recruitment date). Probabilities of enrollment. Table 3-2 shows comparisons of the characteristics of beneficiaries who were more likely to enroll compared to those who were less likely to enroll according to the regression model for the validation sample of potential enrollees. The basis for the table is the predicted enrollment probability derived from the regression model for each beneficiary. Beneficiaries were grouped into quintiles (and the top decile) based on their predicted enrollment probabilities, and then mean values of beneficiary characteristics and other explanatory variables are displayed for each of the quintiles and the top decile. The results for the top decile, in particular, show the characteristics for those beneficiaries who are estimated to be most likely to enroll in the study. The mean predicted enrollment probability (in the top row of Table 3-2) ranged from 0.06 for the lowest quintile to 0.26 for the highest quintile and 0.32 for the highest decile. Comparing mean characteristics across the quintiles, those in the highest quintile tended to be about 4 years younger than those in the lowest quintile, were more likely to be male (50% vs. 43%), and more likely to be black (37% vs. 14%). Those in the highest quintile were also much less likely to have a representative payee and tended to live closer to the study site than is true for those in the lower quintiles. With regard to the time pattern of enrollment, beneficiaries in the highest quintile tended to be those with a longer duration for possible enrollment (Time to Recruit) and those whose contact date was in 2006 or 2007 (rather than 2008). With regard to SSDI or SSI program history, beneficiaries in the highest quintile were less likely to be concurrently receiving SSI, and were on the SSDI rolls (Months on Rolls) about 50 months less time than those in the lowest quintile. Among other recipient and program-related characteristics, a clear pattern was not observed showing variation among the quintiles among beneficiaries. Chapter 3: Enrollment and Participation 3-9 However, there did appear to be some tendency for the fraction of beneficiaries not yet enrolled in Medicare (No Medicare at enrollment) to decrease in the higher quintiles, though these characteristics were not significant predictors in the exploratory regressions (not shown here). The socioeconomic variables associated with census tract statistics that were included in the analyses suggested a trend toward lower education levels and lower median earnings levels for the higher quintiles. All the work-related activity history variables tended to show increasing work or related activities as the probability of enrollment increased. Finally, the diagnostic indicators derived from the SSA administrative data did not show a systematic pattern of variation in relation to enrollment probability. Corresponding results for the potential enrollees in the test sample (see Appendix 3C) showed many of the same patterns seen in Table 3-2, with a few observed differences. The negative trend of education level for the socioeconomic variables associated with the census tract statistics was reversed in test sample, as was the negative relationship between enrollment probability and fraction of the quintile not enrolled in Medicare. The results for the potential enrollees plus possibly potential enrollees in both the test and validation samples also appear in Appendix 3C. It is also of interest to compare the means between the beneficiaries in the top decile with beneficiaries in the top quintile. The most striking differences appear to be in recent work-related activities in TTW and trial work period experiences. For example, the fraction of beneficiaries with an active ticket in the 90 days prior to recruitment (Had Active Ticket w/in 90 days) was more than one-third higher for the highest decile than it was for the highest quintile. This suggests that targeting MHTS recruitment efforts toward beneficiaries with recent TTW activity would tend to yield higher enrollment rates. Characteristics of the Beneficiaries Who Enrolled in the Study The 23 study sites enrolled 2,238 beneficiaries, with 1,121 randomized to the treatment group and 1,117 randomized to the control group. One of the goals of the randomization process was to ensure equivalence between the study groups. Thus, a test of the effectiveness of the randomization process included comparisons on the key demographic and program variables as well as on the baseline measures of employment, health status, and income. All of the data for these analyses came from either the administrative records from SSA’s MBR or the MHTS Baseline interview. Chapter 3: Enrollment and Participation 3-10 Table 3-2. Mean beneficiary characteristics for quintiles based on predicted enrollment probability of potential enrollees in the validation sample (N=7,933) Variable Enrolled (yes/no) Age (years) Gender (Male) Race (Black) Rep Payee (yes/no) Distance from Site (miles) Months on Rolls SSI (yes/no) SSDI<24 mos. (no SSI) Primary Insured Amount No Medicare (at enrollment) Time to Recruit 2006 (Recruitment Yr.) 2007 (Recruitment Yr.) 2008 (Recruitment Yr.) Some College Median Income (census tract) Had Active Ticket (w/in 90 days) Had Active Ticket (ever) Trial Work End Date (10 yrs.+ ago) Trial Work End Date (5-10 yrs. ago) Trial Work End Date (0-5 yrs. ago) Trial Work End Date (0-3 yrs. post- recruitment date) Reported Earnings (1-6 mos. pre-recruitment) Reported Earnings (7-23 mos. pre-recruitment) No Earnings Report (1-6 mos. pre-recruitment) Diagnosis (Affective Disorder) Primary Diagnosis of Serious Mental Illness (SMI) % with Secondary Diagnosis SMI % with SMI Lowest 1st Quintile (n=2,530) 0.06 47.86 0.43 0.14 0.32 13.46 148.04 0.25 0.02 8,092.08 0.06 232.10 0.03 0.43 0.54 0.27 26,531.91 2nd Quintile (n=1,444) 0.09 46.37 0. 43 0.22 0.16 11.74 116.75 0.18 0.03 8,674.24 0.05 266.45 0.06 0.48 0.46 0.26 25,874.18 3rd Quintile (n=1,194) 0.11 46.31 0.43 0.25 0.09 10.90 107.83 0.15 0.02 8,782.19 0.04 279.85 0.07 0.52 0.41 0.26 25,306.46 4th Quintile (n=1,254) 0.16 45.35 0.45 0.33 0.05 11.01 97.70 0.15 0.05 8,771.65 0.06 308.35 0.11 0.59 0.30 0.27 25,183.33 Highest 5th Quintile (n=1,511) 0.26 43.47 0.50 0.37 0.05 9.37 97.93 0.14 0.03 8,566.14 0.05 348.67 0.19 0.59 0.22 0.25 24,609.61 Top 10% (n=842) 0.32 43.31 0.49 0.35 0.06 9.53 99.77 0.14 0.02 8,405.18 0.04 377.98 0.23 0.59 0.18 0.23 24,427.10 0.001 0.01 0.02 0.04 0.22 0.29 0.003 0.02 0.04 0.08 0.28 0.37 0.03 0.04 0.07 0.08 0.13 0.12 0.05 0.05 0.04 0.06 0.08 0.08 0.02 0.03 0.04 0.06 0.10 0.12 0.002 0.003 0.003 0.005 0.04 0.05 358.74 193.83 141.20 174.89 275.37 321.75 833.95 529.44 668.16 867.88 1,696.57 2,007.42 0.96 0.95 0.95 0.94 0.83 0.79 0.62 0.70 0.72 0.71 0.68 0.69 0.80 0.72 0.70 0.66 0.71 0.74 0.36 0.34 0.34 0.33 0.35 0.37 0.80 0.72 0.70 0.66 0.71 0.74 Chapter 3: Enrollment and Participation 3-11 Comparisons on demographic and SSDI program variables. Table 3-3 provides a summary of the demographic and program characteristics of beneficiaries enrolled in the study. The overall mean age of participants was 47.4 years. The majority of enrolled participants come from the older age range of 36 to 55 (84.8%). Thus, only a small percentage (15.2%) of the participants was younger (ages 18 to 35). A brief review of the table shows that there were more females than males overall (52.7%) as well as within each study group. In both groups and overall, 12 percent or fewer were Hispanic. The majority of the enrollees were white (60.0%) or black (26.3%). In both study groups, approximately 12 percent had less than a high school education. However, the largest group included participants with some college or some technical school (34.9%), followed by participants with high school education (26.3%) or a bachelor’s degree (11.6%). Nearly half the enrolled participants (46.2%) were never married, with 27.4 percent divorced and 16.9 percent married. Among these key demographic variables, race was the only one to show a significant difference between the treatment and control groups (p-value = 0.024). On closer inspection, it appears that the difference was associated with the number of participants indicating “Other” as their race. Of the 209 (9.3%) participants in this category, 123 (11% of the total) were in the control group and 86 (7.7% of the total) were in the treatment group. The SSDI program characteristics of study participants included diagnosis associated with the primary impairment, insurance status, and length of time on the SSDI rolls. Participants with an affective disorder comprised approximately 70 percent of the study population, including 68.2 percent of the treatment group and 72.5 percent of the control group. Conversely, 29.7 percent of the study participants had a diagnosis of schizophrenia, comprising 31.8 percent of the treatment group and 27.5 percent of the control group. As noted in the table, these differences were statistically significant (p-value = .024). As a result, analyses reported in later chapters use statistical control for diagnosis as a covariate. More than three-quarters of enrollees (76.3%) had been SSDI beneficiaries for more than 2 years (SSDI > 24 months), indicating that they were eligible for Medicare. Nearly 5 percent were on SSDI less than 24 months, indicating that they were not yet eligible for Medicare at the time of enrollment (unless they had a reduced waiting period as SSI recipients). Approximately 19 percent were also on SSI and likely dual-eligible for Medicare and Medicaid. The average time on the rolls was 103.3 months, with participants in the treatment group having been on the SSDI rolls an average of 106.3 months and those in the control group 103.3 months. Chapter 3: Enrollment and Participation 3-12 Table 3-3. Demographic and program characteristics of enrolled beneficiaries by study condition Variable DEMOGRAPHIC CHARACTERISTICS Treatment (n=1,121) Age Mean Median Control (n=1,117) years 47.2 49 Full sample (N=2,238) years 47.5 49 p-value years 47.4 49 0.815 Age Group 18-35 36-55 n 176 945 % 15.7 84.3 n 165 952 % 14.8 85.2 N 341 1897 % 15.2 84.8 0.541 Gender Male Female n 519 602 % 46.3 53.7 n 539 578 % 48.3 51.7 N 1058 1180 % 47.3 52.7 0.354 Ethnicity Hispanic Not Hispanic Refused or don’t know n 121 1000 0 % 10.8 89.2 0.0 n 134 980 3 % 12.0 87.7 0.3 N 255 1980 3 % 11.4 88.5 0.1 0.359 Race White Black Asian Two or more races Other Refused or don’t know n 674 308 16 33 86 4 % 60.1 27.5 1.4 2.9 7.7 0.4 n 669 281 11 32 123 1 % 59.9 25.2 1.0 2.9 11.0 0.1 N 1343 589 27 65 209 5 % 60.0 26.3 1.2 2.9 9.3 0.2 0.034 Marital Status Never married Married Living as married Separated Divorced Widowed Refused or don’t know n 519 196 5 59 312 27 3 % 46.3 17.5 0.4 5.3 27.8 2.4 0.3 n 514 183 19 69 302 29 1 % 46.0 16.4 1.7 6.2 27.0 2.6 0.1 N 1033 379 24 128 614 56 4 % 46.2 16.9 1.1 5.7 27.4 2.5 0.2 0.959 Education Less than high school High school or GED Some college or technical Associate’s degree Bachelor’s degree Some graduate school Master’s degree Doctoral degree Other, refused, or don’t know n 136 285 402 81 140 20 49 6 2 % 12.1 25.4 35.9 7.2 12.5 1.8 4.4 0.5 0.2 n 135 303 378 99 120 32 35 13 2 % 12.1 27.1 33.8 8.9 10.7 2.9 3.1 1.2 0.2 N 271 588 780 180 260 52 84 19 4 % 12.1 26.3 34.9 8.0 11.6 2.3 3.8 0.8 0.2 0.248 Chapter 3: Enrollment and Participation 3-13 Table 3-3. Demographic and program characteristics of enrolled beneficiaries by study condition (continued) Variable PROGRAM CHARACTERISTICS Treatment (n=1,121) Control (n=1,117) Full sample (N=2,238) p-value Diagnosis Schizophrenia Affective disorder n 357 764 % 31.8 68.2 n 307 810 % 27.5 72.5 N 664 1574 % 29.7 70.3 0.024 Insurance Status SSDI > = 24mos. SSDI < 24mos. SSI n 857 53 211 % 76.4 4.7 18.8 n 851 54 212 % 76.2 4.8 19.0 N 1708 107 423 % 76.3 4.8 18.9 0.897 Months on SSDI Mean 106.3 110.3 103.3 0.184 Comparisons on recent employment and health status measures. Beneficiaries enrolled in the study reported during their Baseline interview if they had worked at any time during the past 2 years. As shown in Table 3-4, 665 of the 2,238 (29.7%) study participants indicated that they had worked at some time during the past 2 years. Of these individuals, 316 (28.2%) received random assignments to the treatment group and the remaining 349 (31.2%) to the control group after completing the Baseline interview. A Chi-square test indicates there is no significant difference between the treatment and control groups on this variable. Table 3-4. Recent employment history and health status comparisons Variable Employment History Worked in past 2 years Control (n=1,117) N % Full sample (N=2,238) N % p-value 0.114 316 28.2 349 31.2 665 29.7 Mean Std Dev Mean Std Dev Mean Std Dev Physical Health (PCS) 44.2 12.0 44.0 11.9 44.1 11.9 0.673 Mental Health (MCS) 36.3 13.1 36.0 13.0 36.1 13.1 0.597 SF-121 1 N= Treatment (n=1,121) N % 2,233. Five beneficiaries answered “don’t know” or refused individual SF-12 items. Those scores were dropped from this analysis. Beneficiaries enrolling in the study also answered questions about their health. Table 3-4 presents descriptive data for all enrollees including comparisons between the treatment and control groups. The two groups had almost identical mean scores, thus differences between the groups proved not to be statistically significant based on a t-test statistic. Chapter 3: Enrollment and Participation 3-14 Comparison on physical health conditions measures. The screener that preceded the Baseline interview included items about current medications and physical conditions that might interfere with employment. The investigators suspect that data from these sources (Table 3-5) are underestimates of physical health conditions, because at the time of the interview most beneficiaries had limited documentation of their current medications to report during the interview. However, information that is more detailed came from the treatment group later during the implementation of the SMM. Chapter 6 provides more details on physical health conditions from multiple sources for the treatment group only. Table 3-5. Anemia Physical health conditions identified in CAPI screener Physical health condition Autoimmune Disorders Treatment (N=1,121) freq % 14 1.2 Control (N=1,117) freq % 19 1.7 p-value 0.375 11 1.0 13 1.2 0.675 Blindness 0 0.0 1 0.1 0.499 Brain Damage 5 0.4 5 0.4 1.000 Cancer 11 1.0 9 0.8 0.659 Cardiovascular Diseases 23 2.1 26 2.3 0.656 4 0.4 8 0.7 0.265 84 7.5 119 10.7 0.009 167 14.9 198 17.7 0.070 97 8.7 113 10.1 0.235 143 12.8 186 16.7 0.009 19 1.7 21 1.9 0.741 1 0.1 5 0.4 0.124 Hyperlipidemia 138 12.3 170 15.2 0.046 Hypertension 224 20.0 271 24.3 0.015 Liver Disease 9 0.8 8 0.7 0.813 Migraines 13 1.2 32 2.9 0.004 Narcolepsy 1 0.1 2 0.2 0.624 Neuromuscular/Degenerative Disorders 42 3.7 40 3.6 0.835 Neuropathy 19 1.7 21 1.9 0.741 Renal Disease 19 1.7 30 2.7 0.109 Seizure Disorder 26 2.3 27 2.4 0.879 Thyroid Disorders 85 7.6 88 7.9 0.375 4 0.4 2 0.2 0.675 Chronic Fatigue Syndrome Chronic Lung Disorder Chronic Pain Conditions Diabetes Gastrointestinal Disorders HIV Hearing Loss Other Chapter 3: Enrollment and Participation 3-15 The relative frequencies of conditions were similar across groups. On average, the control group had somewhat higher rates of physical conditions. The t-test revealed that the differences between treatment and control group beneficiaries were significant for Chronic Lung Disorder (p-value = 0.0092); Gastrointestinal Disorders (p-value = 0.0093); Hyperlipidemia (p-value = 0.0458); Hypertension (p-value = 0.0147); and Migraines (p-value = 0.0041). Comparisons on income measures. Both the Baseline and Followup interviews collected data about various sources of income. The sources assessed included the following: Paid employment; SSDI; SSI; Social Security Retirement or Survivor’s Benefits; VA or other armed services disability benefits; State or county social welfare benefits (e.g., general assistance or public aid, food stamps or assistance from the Temporary Assistance for Needy Families program); Vocational programs (e.g., vocational rehabilitation, Job Training Partnership Act, Easter Seals); Unemployment benefits; “Regular payments” from retirement, pension, investing or savings income, alimony and child support; Money (including loans, gifts, or bill payments) from family members; “Other income” (from informal sector jobs); and Total household income (not asked if living alone or if in residential housing with staff or other clients). In general, the analyses of each measure found little difference (not shown) between the treatment and control groups on most of the variables studied, and no statistically significant differences. Table 3-6 presents the average levels of individual income of beneficiaries enrolled in the study reported by treatment and control participants during the Baseline interview. There were no Chapter 3: Enrollment and Participation 3-16 statistically significant differences between the treatment and control groups on the “total individual income” outcome variable. Table 3-6. Total individual income1 Variable All responses Treatment Mean SE N 1055.5 18.5 1121 Control Mean SE N 1015.0 14.7 1117 Total sample Mean SE N 1035.3 11.8 2238 All non-missing 1064.7 18.7 1090 1032.6 14.8 1076 1048.7 11.9 2166 0.15 Non-HH income 1058.4 32.2 19.7 1026.6 18.7 918 0.11 451 996.0 467 p-value2 1 This is the sum of all individual income items. The figures for “All responses” recodes “Don’t know” or “Refused” responses for individual items as zeros. The figures for “All non-missing” only include responses for persons who did not report any “DK” or “Refused” responses for any individual items. The “Non-HH income” figures are for all persons who (by virtue of their living situations) were not asked to report household income. 2 The reported p-value is the two-tailed p-value of the coefficient of the treatment dummy from a bivariate regression assuming clustering of errors by site. Why Did Eligible Beneficiaries Refuse to Enroll? The study collected some information about the reason a beneficiary refused to engage in a RIG meeting or enroll after attending one or more RIG meetings. For each beneficiary who refused participation, the RA recorded in the SMS (with varying levels of detail) the reason for the refusal. The investigative team developed a categorical list of refusal reasons, with assistance from RAs from across the study sites. The resulting reasons in their order of magnitude appear in Table 3-7. The table shows both all beneficiary refusals, as well as refusals from those who attended one or more RIG meetings (a subgroup of all beneficiary refusals). Table 3-7. Refusal frequencies All Refusal category General disinterest Frequency (n=8103) 3029 Percent of total % 37.4 Attended RIG Percent of Frequency total (n=684) % 330 48.3 Work-related (i.e., cannot work) 1728 21.3 87 12.7 Physical health 1242 15.3 51 7.5 Other 1049 12.9 77 11.3 Life issues or other obligations 586 7.2 93 13.6 Symptom-related 392 4.8 22 3.2 Unknown 63 0.8 10 1.5 Missing 14 0.2 14 2.1 Chapter 3: Enrollment and Participation 3-17 0.07 As presented in Table 3-7, the majority of refusals were due to general disinterest for all beneficiaries (37.4%) and for those who attended one or more RIG meetings (48.3%). For all beneficiaries, other common reasons for refusing to enroll in the MHTS included work-related reasons (21.3%), physical health (15.3%), and other reasons (e.g., the beneficiary was moving, or someone else such as a spouse or parent refused on behalf of the beneficiary, 12.9%). Less common reasons included life issues or obligations (e.g., taking care of a sick parent or family member, 7.2%) and symptom-related reasons (4.8%). For those beneficiaries who attended one or more RIG meetings, other common reasons for refusing to enroll in the MHTS included life issues or obligations (13.6%), work-related reasons (12.7%), and other reasons (11.3%). Less common reasons included physical health (7.5%) and symptom-related reasons (3.2%). Discussion This chapter provides answers to the many questions about enrollment. However, in doing so, it has also identified a number of important issues for the study. First, it is important to know that the randomization process was successful with regard to the key measures at baseline. None of them— employment, income, or health status—showed a significant difference between the treatment and control groups at baseline. There were, however, some differences between the treatment and control groups on other variables. The research found significant differences between the treatment and control participants on race, diagnosis, and five of the self-reported health conditions (including hypertension, chronic lung disorder, gastrointestinal disorders, hyperlipidemia, and migraines—all higher among the control group participants). The treatment group included a slightly higher percentage of blacks, while the control group included a higher percentage of participants in the “Other” race category. With regard to diagnosis, the treatment group showed a higher percentage of participants with schizophrenia, while the control group showed a higher percentage of participants with affective disorder. Prior to recruitment, the investigators decided not to stratify using diagnosis due to the large number of beneficiaries with these disorders. The important stratification variable was insurance status, as the need to ensure equal distribution between the two groups of the low numbers of beneficiaries with less than 24 months on the rolls, and dual-eligible beneficiaries. With 61,530 beneficiaries and a ratio of approximately 3 to 1 beneficiaries with affective disorders relative to those with schizophrenia, there did not seem to be a need to stratify on the diagnosis variable. Chapter 3: Enrollment and Participation 3-18 The difference between the groups in the number of participants reporting health conditions was not necessarily problematic. The overall numbers were not high relative to the large numbers of participants in the treatment and control groups. In addition, there were some concerns about the quality of self-reported health conditions in the Baseline interview. Many of the beneficiaries did not bring their medications with them to the interview as directed. Those who did bring their medications were able to provide better information on their health conditions. In addition, there were other indicators of health collected on treatment group participants during the study. As reported in Chapter 6, there were many more health conditions than originally reported in the Baseline interview by these beneficiaries. Thus, the baseline reporting is likely an underestimate of health conditions. The most interesting conclusion suggested from the analysis of participation was that data routinely available in SSA’s administrative records were predictive of participation in the MHTS intervention. These variables included presence of a representative payee, distance from study site, months on SSDI, and recent TTW activity. The differences in the validation sample, between the predicted probability for the highest quintile (26% for potential enrollees) and the lower quintiles in Table 3-2, highlight this conclusion. More generally, the figures in Tables 3-2 and the corresponding tables in Appendix 3C suggest that a participation rate in excess of 25 percent might be a reasonable expectation with recruitment efforts focused on a targeted group (e.g., beneficiaries with a recent active TTW). However, without targeting particular beneficiaries, a reasonable rate of enrollment might be closer to 14 percent. The analyses found that 26 percent of the top 20 percent of the potential enrollee group were predicted to enroll, resulting in 5.2 percent of the group. For the remaining 80 percent of potential enrollees (i.e., the lower four quartiles), the corresponding figures were as follows: 1.2 percent (.06 x .20), 1.8 percent (.09 x .20), 2.2 percent (.11 x .20), and 3.2 percent (.16 x .20). Therefore, a total predicted enrollment rate would be 13.6 percent. By implication, one might expect an enrollment of 306,000 SSDI beneficiaries out of the 2.25 million who have a psychiatric impairment. One other qualification to these projections should also be borne in mind. As noted earlier, these analyses were for a randomized trial rather than for an expanded implementation effort (where researchers assign all enrollees to the “treatment” group). It seems plausible that the random assignment in MHTS trial may in fact have led to a lower participation rate than would have been observed in an expanded implementation process; however, this is unknown. Chapter 3: Enrollment and Participation 3-19 Beneficiaries’ concerns about returning to work or physical health issues accounted for nearly 40 percent of the documented reasons for refusal. However, many of the enrolled treatment group participants experienced anxieties about work. In some cases, working with the SE specialist alleviated these fears. In addition, the Nurse Care Coordinator spent considerable time working with beneficiaries to address physical health challenges. Thus, it is possible that with further education and knowledge of how the intervention could address personal challenges, the uptake rate would have been higher. Chapter 3: Enrollment and Participation 3-20 Chapter 4 Outcomes This chapter presents analyses and results that answer the fundamental research question asked of the Mental Health Treatment Study (MHTS): To what extent does delivering appropriate mental health treatment and employment supports lead to better employment, health status, and quality of life among Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or an affective disorder (SSA, 2005, p. 6)? The answer to this question comes from a direct head-to-head test between the outcomes attained by the treatment group and the control group. This chapter is divided into three sections, including employment outcomes, earnings and income, and health and quality of life. The Employment Outcomes section begins with an assessment of the primary study outcome—employment rate. This section also contains a large number of additional, secondary analyses of other employment-related measures (e.g., months employed, average weekly earnings, average hours at main job, highest hourly wage, satisfaction with job at study exit). All of these analyses include tests of differences between key subgroups of interest to the Social Security Administration (SSA). Summary tables indicating the subgroup comparisons for which the results were statistically significant appear in the chapter; however, to save space and facilitate reading of the chapter, the descriptive statistics and test data appear in an appendix. The Employment Outcomes section also includes analyses associated with the concept of a steady worker (referring to participants who reported employment in 10 or more months during the study) and types of jobs obtained by study participants. A key aspect of these secondary analyses includes several multivariate analyses that attempt to predict employment. The section ends with presentations of data suggesting why some study participants worked part-time and others did not work at all. The section on Earnings and Income includes a large number of secondary outcome analyses. The first analyses concern reports of past month‘s earnings (averaged over the eight post-baseline interviews), and comparisons between the reported earnings and Substantial Gainful Activity (SGA). The section also includes comparisons between treatment and control group participants‘ income from sources other than earnings, including individual and household income and trends in earnings differentials. This section also reports the estimates of MHTS impacts on earnings and income. The section on Health and Quality of Life presents assessments of the remaining primary study outcomes. These non-vocational outcomes include mental health status, physical health status, and Chapter 4: Outcomes 4-1 quality of life. The section includes a number of secondary tests of differences between key subgroups, including age, gender, diagnosis (according to SSA administrative files), and education. Additionally, the section includes several multivariate analyses that attempt to predict mental health status and quality of life. The final section includes a secondary outcome analysis of the study impact on alcohol and drug abuse as measured by the Addiction Severity Index. Multiple comparisons. This chapter includes a large number of statistical tests, including many ttests and other comparisons. In most cases, these multiple tests are all different ways of looking at the basic hypothesis that the treatment intervention improved the primary outcomes of employment, health, and quality of life. The employment outcomes, particularly, are measured in different ways, such as whether employed, amount of earnings, hours worked, and income, all of which are related. There are many points of view on the issue of multiple comparisons (e.g., Proschan and Waclawiw 2000, Savitz and Oshan 1998, Thompson 1998, Thompson 1997, Savitz and Olshan 1995, Rothman 1990, and O‘Brien 1983). In his excellent article on the subject, Schochet (2008) reviews a number of issues related to this question, such as whether analyses are confirmatory or exploratory, whether the study can focus on a few rather than many outcomes, and whether outcomes can be divided into domains. In the sense of Schochet (2008), this complex study involves both confirmatory and exploratory hypotheses. The general confirmatory hypothesis is that the intervention improves the primary outcomes. However, the question of how the intervention improves outcomes is complex, involving many different measures, and is largely, exploratory. In this study, it is not appropriate to focus on a few measures, which is one of the strategies suggested by Schochet, since each measure shows a slightly different picture of one of the main overall study endpoints of ―improved employment outcomes.‖ In the classic multiple comparisons scenario, one conducts a series of independent hypothesis tests; in this situation, the Type I error is indeed quite inflated. However, the approach here is to look at these measures as a whole, as opposed to looking at individual measures for the primary outcome of employment. While it is true that the Type I error for any given individual test is inflated, and that individual tests are discussed, the emphasis is on interpreting results across measures. Moreover, the fact remains that nearly all measures examined indicate improved outcomes for the treatment group, generally with highly significant p-values. The reader is encouraged to assess the results presented here in the broader context of the four primary outcomes defined at the outset of this study by SSA—employment, health status (physical and mental), and quality of life. Chapter 4: Outcomes 4-2 Employment Outcomes Employment rate. Employment rate was the primary outcome for the study. The policy interest is whether the treatment intervention offered participants a better chance of obtaining employment than did the current system of ―services as usual‖ (as represented by the control group). Table 4-1 shows the employment rates for any job and competitive jobs by treatment and control group. The treatment group attained significantly better employment rates than the control group both in terms of obtaining any type of job and for competitive jobs. The data represent participants who reported having a job at any time during their 24 months of study participation. These unduplicated numbers count participants only once, including those who worked more than one month, or more than one job. The differences for both types of jobs of approximately 20 percentage points were statistically significant and substantively meaningful from both a clinical and social policy perspective. Table 4-1. Employment rates by treatment and control group Variable Employment rate (any) Employment rate (competitive) Treatment (N=1,004) n % 605 60.5 526 52.6 Control (N=1,051) n % 423 40.3 347 33.0 p-value <0.001 <0.001 NOTE: Weighted percents may not be consistent with unweighted counts. Figure 4-1 shows a comparison of the monthly employment rates (any employment) for the treatment and control groups. The first 6 months of the study shows a sharp rise in employment rates across both groups, ranging from approximately 5 to 6 percent in month 1 to between 14 and 18 percent in month 6. The average employment rate over the 6-month period was 14 percent for the treatment group and 11 percent for the control group. Between months 7 and 12, these rates increased to 26 percent and 16 percent for the treatment and control groups respectively. The employment rates for the control group remained steady at 16 percent (on average) for the remainder of the study. However, the employment rates for the treatment group continued to rise to 30 percent and remained there throughout the second half of the study. As shown by the standard error bars on the graph, the differences between the treatment and control groups were not significant throughout the first three months of the study. At month 4, the differences became significant and continued to widen at a rate of 1 to 2 percent in any given month until reaching a plateau of over 15 percent at month 16. From then to the end of the study, the treatment group continued its advantage over the control group by 13 to 16 percentage points. Chapter 4: Outcomes 4-3 Figure 4-1. Monthly employment rates for any job by treatment and control group over the 24month study period 50% 45% Treatment 40 % Control 35% 30% 25% 20% 15% 10 % 5% 0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Months Figure 4-2 presents employment rates by study site (see Appendix 4A for statistics). Note that this graph presents overall employment rates, by study site, over the 24-month study period (not monthly). Thus, the employment rates appearing in Figure 4-2 are much higher than the monthly rates that appear in Figure 4-1. The graph shows substantial variation in the employment rates by site. Over the 24-month study period, employment rates for the treatment group ranged from a low of 34 percent (site 21) to a high of 77 percent (site 15) with a standard deviation of 11 percent. Commensurate rates for the control group were 23 percent (site 21) and 60 percent (site 3) with a standard deviation of 10 percent. Thus, the amount of variability is similar across both treatment and control groups. Chapter 4: Outcomes 4-4 Figure 4-2. Employment rates for any employment by treatment group and control group and by study site 80% Treatment Control 70% 60% 50% 40% 30% 20% 10% 0% 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Site Table 4-2 presents employment rates for both any job and competitive jobs by age, gender, diagnosis, and education.1 All of these stratifications showed significant and meaningful differences. In general, participants with different demographic and clinical characteristics benefitted significantly and meaningfully from the treatment. For example, among participants diagnosed with an affective disorder, the treatment group demonstrated significantly better employment rates for both any job and for competitive jobs than control group. While the differences in employment rates for any job and competitive jobs were smaller for participants with schizophrenia, they too were significant and meaningful. 1 Note that competitive employment is a subgroup of any employment, so participants included in statistics for competitive employment are also counted in statistics for any employment. Chapter 4: Outcomes 4-5 Table 4-2. Employment rates by treatment and control group, stratified by age, gender, diagnosis, and education Variable Employment rate (any) Age: 18 to 34 Age: 35+ Gender: Male Gender: Female Diagnosis: Affective Disorder Diagnosis: Schizophrenia Education: Less than HS Education: HS graduate Education: More than HS Treatment (N=1,004) n % 605 60.5 61 70.9 544 59.5 285 61.6 320 59.5 417 60.3 188 60.9 78 65.1 151 59.2 376 60.2 Control (N=1,051) n % 423 40.3 40 47.2 383 39.7 199 39.6 224 41.0 305 39.5 118 42.5 49 38.2 110 38.1 264 41.7 p-value <0.001 0.002 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 Employment rate (competitive) Age: 18 to 34 Age: 35+ Gender: Male Gender: Female Diagnosis: Affective Disorder Diagnosis: Schizophrenia Education: Less than HS Education: HS graduate Education: More than HS 526 59 467 251 275 371 155 67 140 319 347 33 314 158 189 253 94 37 86 224 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 52.6 68.6 51.1 54.3 51.1 53.7 50.2 56.3 55.1 50.9 33.0 38.9 32.5 31.2 34.6 32.7 33.8 28.7 29.9 35.3 NOTE: Weighted percents may not be consistent with unweighted counts. Other employment outcomes. Table 4-3 presents a summary of other employment outcomes. The table shows employment outcomes for three groups of participants: all participants in the study, participants who worked at least one job (any job), and participants who worked at least one competitive job. The third group, participants who worked a competitive job, is a subset of the second group who worked at least one job. Further, the second group is a subset of all participants in the study. The distinctions among the three groups warrant a brief explanation. First, many people believe that type of employment is important. The Individual Placement and Support (IPS) model for supported employment (SE) services gives priority to competitive jobs as important aspects of community integration and patient-centeredness, but other vocational programs are more likely to use sheltered or non-competitive jobs. People with psychiatric illness themselves clearly prefer competitive jobs, which by definition are in integrated rather than segregated work settings, because they value community integration (Bedell, Draving, Parrish, Gervey, & Guastadisegni, 1998). The U.S. Rehabilitation Act of 1973 (Pub. L. No. 93-112, 87 Stat. 394 [September 26, 1973]) Chapter 4: Outcomes 4-6 also emphasizes employment in integrated work settings as part of the American commitment to social inclusion of people with disabilities. Table 4-3. Other employment outcomes for all participants, participants who worked at least one job, and participants who worked at least one competitive job by treatment and control group Variables All participants Months to 1st job Total months employed Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Participants who worked at least 1 job Months to 1st job Total months employed Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Job satisfaction with main job at study end1 Participants who worked at least 1 competitive job Months to 1st job Total number of months employed Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Job satisfaction with main job at study end2 Treatment M SD N=1,004 † † 6.23 7.48 3.22 6.55 116.58 11.93 7.65 139.30 12.47 8.77 n=605 7.37 6.17 10.30 7.09 5.32 7.72 192.69 19.72 12.65 38.77 131.70 10.07 7.98 8.34 n=526 Control M SD N=1,051 † † 3.65 6.44 1.79 5.27 d p-value † 2.58 1.43 † <0.001 <0.001 140.59 11.20 8.12 40.54 4.29 2.56 <0.001 <0.001 <0.001 n=423 6.68 6.31 9.04 7.33 4.45 7.55 0.69 1.26 0.87 0.016 0.001 0.005 4.07 0.76 0.02 1.49 0.080 0.173 0.443 0.023 76.04 7.64 5.09 188.62 18.96 12.63 37.28 166.91 9.82 8.25 8.44 n=347 7.74 9.26 4.27 6.04 6.66 6.94 7.22 8.35 3.72 6.16 6.82 6.86 0.52 0.91 0.55 0.107 0.017 0.113 201.11 20.49 11.36 38.40 143.61 10.69 5.90 8.27 193.32 19.40 11.54 37.43 176.16 10.14 7.02 8.95 7.79 1.09 -0.18 0.97 0.060 0.097 0.645 0.347 † Not applicable. 1The Ns for job satisfaction are n=335 for the treatment group and n=178 for the control group because this measure includes only participants who reported working in a job during the final Followup interview (whereas the other measures include participants who reported working in a job during any of the eight post-baseline interviews). 2The Ns for job satisfaction are n=251 for the treatment group and n=138 for the control group because this measure includes only participants who reported working in a competitive job during the final Followup interview (whereas the other measures include participants who reported working in a competitive job during any of the eight post-baseline interviews). Second, separate analyses restricted to those who obtain employment only must be interpreted in recognition of the fact that interventions that obtain higher rates of employment are including as Chapter 4: Outcomes 4-7 workers people who have less motivation, readiness, background, skills, or other characteristics as workers. Other things being equal, they should therefore have relatively poor ―other employment outcomes‖ compared to programs that focus on clients with greater motivation, readiness, background, skills, or other characteristics as workers. Thus, it is important to measure these other employment outcomes (such as total months employed, average weekly earnings, and consecutive months employed) to further assess the effectiveness of the treatment intervention. In the context of the entire study sample, the other employment outcomes for the treatment group were statistically and meaningfully significant over those of the control group. As noted in Table 4-3, the treatment group (as a whole) was employed for more months over the study period (6.23 vs. 3.25, p-value < 0.001), employed longer as measured by consecutive months employed at study exit (3.22 vs. 1.79, p-value < 0.001), attained higher average weekly earnings at their main job ($116.58 vs. $76.04, p-value < 0.001), worked more hours per week (11.93 vs. 7.64, p-value < 0.001), and had higher hourly wages ($7.65 vs. $5.09, p-value < 0.001). In the more specific contexts of participants who obtained at least one job or participants who obtained at least one competitive job, the treatment group demonstrated advantages on some but not all of the other employment outcomes. Time to first job, which of course applies only to those who obtained a job, showed an advantage for the control group participants, in relation to any job but not competitive jobs. This finding suggests that some control group participants went to work rapidly in non-competitive jobs. The treatment group exceeded the control group on total months employed in any job (10.30 vs. 9.04, p-value = 0.001) and in competitive employment (9.26 vs. 8.35, p-value = 0.017); these differences were also statistically significant when analyses were restricted to workers only. Consecutive months of employment among workers at study exit favored the treatment group for both any employment (5.32 vs. 4.45, p-value = 0.005) and competitive employment (4.27 vs. 3.72, p-value = 0.113). These results were significant for any employment, but not significant for competitive employment. The 24-month end date censored this outcome, which quite possibly could become more significant over time. Previous IPS followup studies have shown greatly extended job tenure over time (Becker, Whitley, Bailey, & Drake [2007]; Salyers, Becker, Drake, Torrey, & Wyzik [2004]). Average weekly earnings among workers favored the treatment group for both types of employment, but results were only marginally significant. Most of these participants were in the early Chapter 4: Outcomes 4-8 phases of returning to work, perhaps anticipating increased earnings in the future, as they expanded their work hours. Average hours of work per week at the main job were non-significantly greater for the treatment group for both any employment and competitive employment. These workers were in the early phase of returning to work and may have felt constrained by fears regarding stress, Social Security regulations, or other factors. Highest hourly wage did not differ between groups for any employment or competitive employment. Again, participants in both conditions were in early phases of returning to work. Satisfaction with main job at study exit significantly favored the treatment group among participants with any job, and was non-significant among those with competitive jobs. Other employment outcomes by subgroups. For each of the seven other employment outcomes, Table 4-4 presents a summary of the results of t-tests that were conducted to determine whether differences in mean values between treatment and control group participants were statistically significant, for the following subgroups: age, gender, diagnosis, and education. These comparisons are presented for the three participant groups described in the previous section—all participants in the study, participants who worked at least one (any) job, and participants who worked at least one competitive job. Summary descriptive statistics and the results of the significance tests are presented in Appendix 4B for all measures and all subgroups shown in Table 4-4. In general, the pattern of results is similar to that described in the previous section, when comparing treatment group to control group participants overall. Participants with different demographic and clinical characteristics benefitted from the treatment intervention in the context of the entire study sample (i.e., all participants), regardless of age, gender, diagnosis, or education status. In other words, the other employment outcomes for the treatment group were statistically and meaningfully significant over those of the control group. More specifically, treatment group participants were employed for more months; had more consecutive months of employment at study exit; averaged higher weekly earnings; averaged more hours per week; and had higher hourly wages compared to control group participants for all subgroups, with one relatively minor exception. Although younger treatment group participants between the ages of 18 and 34 had more consecutive months of employment at study exit than control group participants in the same age group (3.30 vs. 2.38, p-value = 0.9305) this difference was not statistically significant, more than likely because of the small N for this younger age group. Chapter 4: Outcomes 4-9 Table 4-4. Statistically significant subgroup comparisons for other employment outcomes including all participants, participants who worked at least one job, and participants who worked at least one competitive job by age, gender, diagnosis, and education Variables All participants Total months employed Age 18 to 35 35+ Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Participants who worked at least 1 job Months to 1st job Total months employed Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Job satisfaction with main job at study end * Diagnosis A S Education < HS HS grad > HS * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Participants who worked at least 1 competitive job Months to 1st job Total number of months employed * Consecutive months of employment at study exit Average weekly earnings at main job Average hours per week at main job Highest hourly wage Job satisfaction with main job at study end Gender M F * * * * * * * * * * * * * * * * * * * * * * p-value < 0.05 When analyzing these same employment outcomes for participants who obtained at least one job or participants who obtained at least one competitive job, the treatment group demonstrated advantages on some, but not all, outcomes for the various subgroups presented. As shown in Chapter 4: Outcomes 4-10 Table 4-4, there were statistically significant differences between treatment and control for months to first job; however, examination of Appendix 4B reveals that the direction of the test favored control group participants. Participants randomized to the control group went to work more rapidly than participants randomized to the treatment group for participants aged 35 or older, females, and those diagnosed with schizophrenia. When restricting months to first job even further, to participants who worked at least one competitive job, observed differences between the treatment and control group for all subgroups presented were not statistically significant. When reviewing the entire sample, the treatment group reported working more months (i.e., total months employed) during the 24-month study period than the control group for all subgroups, and these differences were statistically significant. However, for participants who worked at least one job, the treatment group showed an advantage over the control group for only older participants, males, those diagnosed with an affective disorder, and participants with more than a high school diploma. These same subgroups also showed a statistically significant advantage for the treatment group when restricting the sample to participants who worked at least one competitive job. Average weekly earnings were statistically significant for all subgroups when comparing differences between treatment and control group participants for all participants (regardless of their employment status). However, fewer significant differences were observed between treatment and control participants when restricting the comparison to participants who worked at least one job (for both any employment and competitive employment). This pattern of fewer significant observed differences when comparing treatment and control participants among working participants only holds true for average hours worked per week as well. In fact, for average hours worked per week, when restricting the comparison to participants who worked at least one competitive job, statistically significant differences occur only for females and participants with an affective disorder. While statistically significant differences emerge for highest hourly wage among all subgroups in the context of all participants, there were no significant differences between the treatment and control groups when analyzing workers. Satisfaction with main job at study exit, for those who worked at least one job, significantly favored the treatment group for older participants, females, those with an affective disorder, and those with higher education. When looking at competitive jobs only, there were no statistically significant differences between treatment and control participants for any of the subgroups analyzed. Steady workers. Classification of study participants into three work categories allowed for examination of the characteristics of participants who successfully returned to work. After careful Chapter 4: Outcomes 4-11 examination of the MHTS data and the long-term employment followup literature, the criterion for a ―steady worker‖ was set at a minimum of 10 months of employment during the 24-month study period. This criterion is conceptually similar, if not more stringent, than the definition of steady worker of ―employed at least 50 percent of the time,‖ which has been used in the literature (Becker, Whitley, Bailey & Drake, 2007; Salyers, Becker, Drake, Torrey & Wyzik, 2004).2 Additional cut points were set at 3 months and less than 10 months to identify the ―erratic worker,‖ and 3 months or less to identify the ―minimal worker (including no work).‖ Table 4-5 presents the distribution of work status of participants by study arm. Findings indicate that there are more participants in the steady worker category among the treatment group participants (30% vs. 16%) and there were more participants in the minimal worker category among the control group participants (53% vs. 72%). These differences between treatment and control groups were statistically significant (p-value < 0.001). Table 4-5. Classification of worker types by treatment and control group Variable Steady worker Erratic worker Minimal worker (including no work) Treatment (N=1,004) n % 295 29.5 175 17.5 534 53.0 Control (N=1,051) n % 166 15.8 132 12.6 753 71.6 p-value <0.001 NOTE: Weighted percents may not be consistent with unweighted counts. Significantly more treatment group participants comprised the steady worker category within each of the key demographic categories of age, gender, diagnosis, education, and the indicator of having worked in the past two years (at baseline). Table 4-6 presents the distributions by treatment and control for the two study age groups of interest, gender, diagnosis, and education. 2 As seen in Table 4-3, workers needed an average of seven months to obtain their first job. Assuming it took a working participant on average seven months to obtain a first job, he or she would then have 17 months left remaining in the 24-month study period. In order to meet the definition of steady worker (i.e., employed at least 50% of the time), a participant would have to report working at least half of that 17 months (i.e., 8.5 months). So the MHTS criterion of 10 months of employment during the 24-month study period is conceptually similar to that described in the literature, taking into account the amount of time it takes for an MHTS beneficiary to obtain a first job. Chapter 4: Outcomes 4-12 Table 4-6. Characteristics of steady workers by treatment and control group Variable Age: 18 to 34 Age: 35+ Gender: Male Gender: Female Diagnosis: Affective Disorder Diagnosis: Schizophrenia Education: Less than HS Education: HS graduate Education: More than HS Worked in past 2 years (baseline) Not worked in past 2 years (baseline) Treatment n/N 25/86 270/918 142/465 153/539 213/693 82/311 35/120 68/256 192/628 123/285 172/719 % 29.6 29.5 30.7 28.5 30.9 26.5 28.7 26.7 30.8 43.1 24.2 Control n/N 13/85 153/966 67/503 99/548 115/772 51/279 17/127 46/287 103/637 98/330 68/721 % 15.5 15.8 13.2 18.2 15.0 18.2 13.1 16.0 16.3 29.9 9.4 p-value 0.031 <0.001 <0.001 0.001 <0.001 0.017 0.003 0.003 <0.001 <0.001 <0.001 NOTE: Weighted percents may not be consistent with unweighted counts. Types of occupations obtained. The interviewer recorded job titles verbatim during each followup interview. This section examines those data and the types of occupations that participants held during the study. Classification of the occupational types followed the Standard Occupational Classification (SOC) System maintained by the U.S. Bureau of Labor Statistics (BLS) (http://www.bls.gov/soc/), which includes an online search engine matching a large database of job titles to 6-digit SOC codes. The 2010 SOC System is a standard method of classifying workers‘ jobs. It is widely used across many Federal statistical agencies to document the types of jobs (occupations) and numbers of workers doing them. The six digit code level includes 840 detailed occupations. These are often aggregated into smaller groupings for particular users. The standard grouping includes 461 broad occupations, 97 minor groups, and 23 major groups. For users who wish to present occupational data at an even broader level, the 2010 SOC User Guide (http://www.bls.gov/soc/soc_2010_user_guide.pdf) has recommended a high-level aggregation of the 23 major groups into 6 categories. These six aggregated occupational categories include: Management, business, science, and arts occupations; Service occupations; Sales and office occupations; Natural resources, construction, and maintenance occupations; Production, transportation, and material moving occupations; and Military specific occupations. Chapter 4: Outcomes 4-13 Table 4-7 shows a comparison of the monthly frequency of jobs held in each of these SOC categories between the treatment and control groups, with the exception of military specific occupations; none of the MHTS participants obtained military specific jobs during the study. Therefore, all tables related to type of job report on the remaining five high-level aggregated categories. Summarized at the bottom of Table 4-7 are the mean frequencies for jobs reported during the Baseline interview (i.e., jobs worked the past 2 years prior to baseline), Year 2 (the second 12 months of the 24-month study period), and 2-Year Total (months 1-253). The meaningful statistic is the percentage of participants within each SOC category. The process involved summing the numbers within each group, then calculating the percentage from these totals. At baseline, the distributions of new study participants across the five occupational categories were remarkably similar between treatment and control groups. A Chi-square test of differences between treatment and control resulted in no statistically significant differences. Service occupations and sales and office occupations represented nearly 70 percent of all jobs held by either treatment (67.2%) or control (68.4%) participants. About half of those percentages were represented by management, business, science, and arts occupations and production, transportation, and material moving occupations, evenly spread across treatment (30.1%) and control (28.3%) group participants. Less than three percent of treatment or control group participants held occupations in the natural resources, construction, and maintenance category. The results for Year 2 were similar to those for the 2-Year Total. The Year 2 results may be more credible in that the employment rates were low during the first several months after study enrollment. In any case, the percentages with SOC categories for any given month generally appear to mirror both the Year 2 and 2-Year averages. The Year 2 findings indicate that of the jobs held by participants: 13 percent of both treatment and control participants were in management, business, science, or arts; 37 percent of treatment participants compared to 41 percent of controls held service jobs; 37 percent of treatment participants compared to 34 percent of controls held sales and office jobs; 2 percent of both treatment and control participants worked in construction or maintenance; and 10 percent of treatment participants compared to 11 percent of controls worked in production or transportation. 3 For each job reported, participants also reported the month and year the job started and stopped. Since a participant could start a job immediately after study enrollment, the month in which a participant enrolled was the first calendar month for which employment outcome data were analyzed (Month 1). Counting out 24 months from Month 1 yields a total of 25 calendar months for which MHTS employment outcomes were assessed. Chapter 4: Outcomes 4-14 Table 4-7. Study month 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Baseline freq % Year 2 M % Total M % Types of occupations held by participants who reported working in a job for each study-month by treatment and control group Management, business, science, and arts occupations Trmt Ctl 10 11 14 17 18 20 26 21 30 17 31 16 34 26 40 22 34 19 42 26 31 26 35 26 41 25 39 21 38 21 50 26 38 24 35 19 39 27 35 21 33 17 43 22 39 25 39 21 40 18 Service occupations Trmt Ctl 16 18 29 32 49 32 62 52 65 41 81 53 96 60 92 51 87 53 109 58 102 53 106 55 106 65 103 67 109 64 112 69 112 67 113 70 122 66 120 63 113 68 120 72 113 71 102 75 98 74 Sales and office occupations Trmt Ctl 12 20 21 28 44 40 67 46 70 39 83 49 95 63 93 60 85 59 95 64 93 56 94 52 112 57 107 54 102 49 115 58 112 55 120 53 120 59 101 55 96 54 120 62 117 55 118 56 111 58 Natural resources, construction, and maintenance occupations Trmt Ctl 1 2 2 2 7 4 6 5 11 4 7 5 9 4 8 4 7 3 8 6 6 4 6 6 10 7 6 6 8 5 10 2 5 2 5 1 6 1 4 1 3 2 8 1 7 1 5 2 6 2 Production, transportation, and material moving occupations Trmt Ctl 9 9 10 18 15 21 25 27 21 22 22 23 33 26 24 25 26 21 29 24 33 24 32 21 35 21 29 17 30 13 37 17 34 16 38 19 39 24 35 22 26 21 29 18 27 17 24 13 23 12 71 15.8 74 13.6 119 26.6 170 31.3 182 40.6 202 37.1 12 2.7 18 3.3 64 14.3 80 14.7 39.2 13.1 22.1 13.3 111.0 37.1 68.5 41.1 111.6 37.3 55.8 33.5 6.4 2.1 2.5 1.5 31.2 10.4 17.7 10.6 33.5 13.6 21.1 14.0 90.5 36.8 56.4 37.4 89.4 36.3 50.9 33.8 6.2 2.5 3.2 2.1 26.6 10.8 19.2 12.7 Chapter 4: Outcomes 4-15 Tables 4-8 and 4-9 show the distributions of occupational types by diagnosis and by education. These tables demonstrate the expected pattern of a higher percentage of managerial or business jobs among participants with an affective disorder, as compared to participants with schizophrenia, and a gradient of increasing proportions of participants achieving managerial jobs as a function of level of education. In each case, there were correspondingly higher percentages of participants with schizophrenia and with less education in service jobs. Table 4-8. Types of occupations held by participants who reported working in a job by diagnosis Treatment Affective disorder % M* Management, business, science, and arts occupations Service occupations Sales and office occupations Natural resources, construction, and maintenance occupations Production, transportation, and material moving occupations *The Control Schizophrenia M* % Affective disorder M* % Schizophrenia M* % 26.2 15.2 7.3 10.0 18 16.9 3.2 7.3 53.1 70.5 30.7 40.8 36.9 19 50.6 26.1 37.9 38.2 35.5 35.9 18.8 12.3 42.5 27.8 4.3 2.5 2 27.8 1 0.9 2.2 5.0 18.7 10.8 7.8 10.7 11.5 10.8 7.7 17.4 reported mean is the average proportion across the 24-month study period. Predicting employment. An important secondary question in the MHTS was whether it was possible to predict which participants work, or under what circumstances they work. Three variables were particularly useful for explorations of work and work behavior: obtained employment, total months employed, and steady worker. These three variables offered the opportunity to explore increasing levels of work behavior in participants who obtained jobs during the study. ―Obtained employment‖ includes the largest number of participants in the study, counting any participant who reported having a job in any month during the study. ―Total months employed‖ offers a different dimension of obtained work, including the amount of work measured by the number of months in which a participant reported having a job (up to 25). Third, the ―steady worker‖ variable defines a special subgroup of participants in the MHTS. These individuals reported having a job for at least 10 months during the study. The goal of these analyses was to identify predictor variables using a combination of logistic and zero-inflated negative binomial regression techniques to explain the characteristics of working participants, events associated with working, or conditions that predict work. The analyses involving categorical outcome variables (obtained employment and steady Chapter 4: Outcomes 4-16 worker) used logistic regression. The analyses involving the numerical variable (total months employed) used zero-inflated negative binomial regression. Table 4-9. Types of occupations held by participants who reported working in a job by education Less than HS M* % Treatment Management, business, science, and arts occupations Service occupations Sales and office occupations Natural resources, construction, and maintenance occupations Production, transportation, and material moving occupations Control Management, business, science, and arts occupations Service occupations Sales and office occupations Natural resources, construction, and maintenance occupations Production, transportation, and material moving occupations *The HS graduate M* % College graduate M* % Graduate school M* % 0.6 1.9 13.9 10.0 12.4 20.4 6.6 42.7 11.8 9.8 39.5 32.9 57.0 50.3 40.9 36.2 17.0 25.2 27.9 41.4 4.3 3.7 27.8 23.8 1.5 5.1 4.2 3.0 0.5 0.9 0.0 0.0 6.1 20.5 13.7 9.9 5.7 9.4 0.9 6.0 0.7 4.2 7.5 8.9 10.8 24.8 2.3 32.9 8.4 5.0 53.4 32.1 33.8 27.2 40.0 32.2 12.5 16.0 28.8 36.6 1.9 2.3 27.9 32.9 0.3 2.2 2.5 3.0 0.3 0.8 0.0 0.0 1.2 7.8 13.6 16.0 3.9 9.0 0.5 6.7 reported mean is the average proportion across the 24-month study period. A potentially large number of variables were available to assess employment across the entire study population, including both treatment and control group participants. The criteria for selecting predictor variables for the entire study population included an indication of importance in previous research or variables of interest to SSA. Table 4-10 lists the demographic, clinical, and site-level potential predictors included in the analysis for all participants. Demographic variables included age, gender, education, and Supplemental Security Income (SSI) status. Clinical variables included mental or physical health, diagnosis, emergency room visits at baseline, and hospital stays at baseline. Site variables included urbanicity as measured by population density around the study site, and an effective unemployment rate was computed by weighting the local unemployment rate in a given year by the participant‘s length of participation in the study in that particular year. All of these Chapter 4: Outcomes 4-17 variables were available on each participant enrolled in the study, including those in the treatment and control groups. Table 4-10. Potential predictor variables included in employment-related regression models Variable Treatment dummy Baseline mental health (SF-12) Baseline physical health (SF-12) Gender Worked in last 2 years (pre-baseline) Diagnosis Age Less than HS graduate vs. More than HS graduate HS graduate vs. More than HS graduate Unemployment rate Months on rolls Number of ER visits at baseline Number of hospital stays at baseline Population density Not receiving SSI (SSDI only) Ever had active ticket SE engagement score SMM engagement category 0 vs. 2 SMM engagement category 1 vs. 2 Received medication services Received general medical care Received substance abuse treatment Received housing services Received family counseling or legal assistance Received social skills training Received financial assistance training Received case management Average site fidelity score Number of clients served at site Treatment and control X X X X X X X X X X X X X X X X Treatment only X X X X X X X X X X X X X X X X X X X X X X X X X X X X An even larger number of variables were available to assess employment among treatment group participants only. The study collected a rich set of useful data on just treatment group participants, most of which pertain to the treatment intervention, services received, or study site characteristics, that were not particularly relevant to control group participants. Also shown in Table 4-10, the additional potential predictors included in the regression models for treatment group participants only included dummy variables indicating receipt of some other behavioral health or related service (e.g., substance abuse treatment, case management, or social skills training) at some point during the Chapter 4: Outcomes 4-18 24-month study period; variables measuring the extent to which a participant was engaged with the IPS SE and systematic medication management (SMM) components of the intervention; size of study site (measured by the number of clients served at the site); and the site fidelity scores. The general strategy for all regressions (logistic and zero-inflated negative binomial) associated with predictions of employment was to run unadjusted bivariable models using each predictor variable with the outcome variable. Analyses using data on all participants included the treatment dummy (i.e., in treatment group—yes vs. no) as a third variable in the unadjusted bivariable preliminary models. Predictor variables with an estimate having a p-value of .25 or less were included in the final multivariable model. Reporting in the tables that follow only include those models containing variables with a p-value of .05 or less. Table 4-11 presents three panels containing the final regression models for obtained employment, total months employed, and steady worker. Overall the “obtained employment” model yielded an acceptable fit (c=.709) and included 5 of the original 16 predictor variables (including treatment dummy), two of which appear to be relatively important. Being in the treatment group, having better physical health, working in the past two years at baseline, lower unemployment rates, and lower number of months on the SSDI rolls were significant in the final model for “obtained employment.” The MHTS treatment dummy had a positive and highly significant marginal effect estimate. The results show that, on average, being in the treatment group increased the probability of obtaining employment by 23 percent, ceteris paribus. Higher physical health status at baseline and having work experience in the last 2 years (pre-baseline) significantly increased the probability of obtaining employment. A unit increase in the local unemployment rate and number of months on SSDI rolls significantly decreased the probability of obtaining employment. The “steady worker” model repeated these same results with a few minor exceptions. In addition to the same five variables (being in the treatment group, having better physical health, working in the past two years at baseline, number of months on rolls and unemployment rates), being on the SSI rolls and number of hospital stays at baseline figured into the “steady worker” model. Being in the treatment group, on average, increased the probability of becoming a steady worker by 14 percent, ceteris paribus. Being on the SSI rolls and a unit increase in the number of hospital stays at baseline both reduced the probability of reporting steady employment. Chapter 4: Outcomes 4-19 Table 4-11. Estimates of marginal effects on obtained employment, steady worker, and number of months employed (for all participants) Variable Estimate p-value Logistic regression A. Obtained employment1 Treatment Baseline physical health (SF-12) Worked in last 2 years (pre-baseline) Unemployment rate Months on rolls 0.226 0.004 0.308 -0.019 -0.001 <0.001 <0.001 <0.001 0.017 0.005 B. Steady worker2 Treatment Baseline physical health (SF-12) Not receiving SSI (SSDI only) Worked in last 2 years (pre-baseline) Unemployment rate Months on rolls Number of hospital stays at baseline 0.139 0.002 -0.523 0.200 -0.019 -0.001 -0.048 <0.001 0.020 0.017 <0.001 0.001 0.001 <0.001 Zero-inflated negative binomial regression C. Total months employed Treatment Baseline mental health (SF-12) Baseline physical health (SF-12) Number of hospital stays at baseline Months on rolls Worked in last 2 years (pre-baseline) Ever had active ticket Unemployment rate 2.573 0.029 0.042 -0.673 -0.008 3.836 0.849 -0.359 <0.001 0.022 0.002 0.002 <0.001 <0.001 0.054 0.001 1The C-statistic for the Obtained Employment model is 0.709. The C-statistic ranges from.5 to 1 where C=1 for a perfect model and C=.5 for a model no better than random classification. 2The C-statistic for the Steady Worker model is 0.708. The C-statistic ranges from.5 to 1 where C=1 for a perfect model and C=.5 for a model no better than random classification. A zero inflated negative binomial regression model supported examination of the treatment intervention impact on the count of ―total months employed‖ over the study period. The dependent variable was total months employed, which carries a value up to 25 months with a large concentration of ―zero‖ observations (i.e., never employed). Zero-inflated count models provide a powerful way to model this type of situation. This model assumes that the data are a mixture of two separate data generation processes: one generates only zeros and the other is a negative binomial data-generating process. The result of a Bernoulli trial is used to determine which of the two processes generates an observation. Chapter 4: Outcomes 4-20 Table 4-11 presents marginal effect estimates for “total months employed,” for the backward stepwise estimation including only covariates that are significant at.05 (i.e., the reduced model). Marginal effect estimates in the reduced model indicated that the expected impact for each person of participating in the MHTS (versus not participating in the MHTS), averaged across the entire study sample (both control and treatment), was 2.57 additional months of employment. Baseline mental health, baseline physical health, having prior work experience, and having an active ticket were significant predictors with positive effects on the “total months employed” outcome. Number of hospital stays, number of months on SSDI rolls, and the local unemployment rate were significant with negative effects on the “total months employed” outcome. Table 4-12 presents the results of similar regression estimates using treatment group only data. These estimates are important because they offer a greater number of potentially important predictor variables associated only with beneficiaries who participated in the treatment intervention. For participants that obtained employment during the study, eight predictor variables were significant in the final regression model. A unit increase in physical health score (at baseline), engagement in SE, and population density all indicated a positive impact on the probability of ―obtained employment.‖ Participants who worked in the last 2 years at baseline also had a higher likelihood of obtaining employment. A unit increase in local unemployment rate, number of months on rolls, and number of hospital stays at baseline decreased the probability of obtaining employment. Participants who received case management services also were less likely to report employment. The logistic regression results for the steady worker model were similar to the obtained employment model with a few minor exceptions. Population density and physical health at baseline did not appear to be statistically significant in the model for the steady worker. Instead, having an active ticket (from SSA‘s Ticket to Work program) was significant and had a positive marginal effect on the probability of being a steady worker. As with obtained employment, participants reporting that they worked in the past two years (at baseline) were more likely to be steady workers than were participants who did not report such work experience. A unit increase in the local unemployment rate, number of months on rolls, and number of hospital stays all had negative marginal effects on the probability of becoming a steady worker. Participants that received case management services were also less likely to report steady employment. Table 4-12 also presents the results of the zero-inflated negative binomial estimation for total months employed, using data from the treatment group only. Seven predictor variables were significant in the reduced regression model. Number of hospital stays, number of months spent on Chapter 4: Outcomes 4-21 SSDI rolls, local unemployment rate, and receipt of case management all had a significant negative effect; having a work experience in the last two years (at baseline), ever having an active ticket, and level of engagement in SE had a significant positive effect on total months employed. Table 4-12. Estimates of marginal effects on obtained employment, steady worker, and number of months employed (for treatment group participants only) Variable Logistic regression A. Obtained employment1 Baseline physical health (SF-12) SE engagement score Worked in last 2 years (pre-baseline) Unemployment rate Months on rolls Number of hospital stays at baseline Receive case management Population density Estimate p-value 0.004 0.018 0.221 -0.032 -0.001 -0.038 -0.100 0.029 0.006 <0.001 <0.001 0.003 <0.001 0.044 0.003 0.003 B. Steady worker2 Worked in last 2 years (pre-baseline) Unemployment rate Months on rolls Number of hospital stays at baseline Ever had active ticket SE engagement score Receive case management 0.192 -0.029 -0.001 -0.070 0.110 0.014 -0.094 <0.001 0.003 <0.001 0.002 0.017 <0.001 0.001 Zero-inflated negative binomial regression C. Total months employed Number of hospital stays at baseline Months on rolls Worked in last 2 years (pre-baseline) Ever had active ticket Unemployment rate SE engagement score Receive case management -1.091 -0.013 3.675 1.770 -0.495 0.254 -1.401 0.001 <0.001 <0.001 0.010 0.001 <0.001 0.006 1The C-statistic for the Obtained Employment model is 0.709. The C-statistic ranges from.5 to 1 where C=1 for a perfect model and C=.5 for a model no better than random classification. 2The C-statistic for the Steady Worker model is 0.708. The C-statistic ranges from.5 to 1 where C=1 for a perfect model and C=.5 for a model no better than random classification. Reasons for not working full-time. Reasons for not working full-time among all participants reporting part-time work have been examined. During the final Followup interview, participants that did not report working full-time during the previous quarter answered a followup question about why. Table 4-13 shows the responses for those participants responding to the question. For both Chapter 4: Outcomes 4-22 treatment (38.6 percent) and control group (50.6 percent) participants, the most frequent reason is ―Too sick to work full-time.‖ The difference between the treatment and control groups is statistically significant (p-value = 0.009). Other most frequent reasons were ―Couldn’t find full time job‖ (15.7 and 15.9 percent for treatment and control group, respectively) and ―Don’t know‖ (14.5 and 13.2 percent for treatment and control group, respectively). Table 4-13. Reasons for working part-time Reasons Couldn’t find full-time job Too sick to work full-time Don’t want to work more Other demands on time (i.e., pets, child) Make enough money working part-time Didn’t want to lose SSDI or medical benefits In school (full- or part-time) Other reasons Don’t know Treatment (N=332) N % 52 15.7 128 38.6 23 6.9 14 4.2 17 5.1 23 6.9 5 1.5 22 6.6 48 14.5 Control (N=182) N % 29 15.9 92 50.6 9 5.0 0 0.0 2 1.1 12 6.6 1 0.6 13 7.1 24 13.2 p-value 0.936 0.009 0.374 0.005 0.021 0.886 0.334 0.824 0.691 Among treatment group participants, 6.9 percent of respondents stated that they did not want to work more and the same percent reported they did not want to lose SSDI benefits or medical benefits (provided by the Center for Medicare and Medicaid Services). While 6.6 percent of treatment participants reported other reasons for working part-time, 1.5 percent of them were in school on a full- or part-time basis. Among control group participants, 5.0 percent stated that they did not want to work more (than they were currently working) and 6.6 percent reported they did not want to lose benefits. The findings also show that 7.1 percent of treatment participants had other reasons for working part-time and 0.6 percent reported going to school on a full- or part-time basis as the reason for working part-time. Reasons for not working. Many of the treatment group participants achieved some success in working. As expected, some did not. It was just as important to find out as much as possible about those who experienced little or no success in working. The literature on employment and mental illness, reviewed earlier, emphasizes several factors, including uncontrolled symptoms of mental illness, ongoing substance use, poor social skills, cognitive problems, and so forth. From the beginning of the study, it was apparent that a small proportion of participants entered the study with primary motivations other than to work. These participants understandably wanted better Chapter 4: Outcomes 4-23 insurance, better health care, or some other benefit from the study, but they showed minimal or no interest in working throughout their tenures in the study. Some never met with an SE specialist; some engaged superficially but showed no interest in trying to work. Another problem encountered from the beginning of the study was that a number of participants had serious physical health problems that limited their ability to find a job and participate in employment. Apparently, physical health problems were part of their disablement even though the primary listing on Social Security records was mental illness. These two problems differed from the usual barriers to employment that IPS specialists have confronted in previous studies. Of course, the two problems also reflected the unique features of the MHTS. That is, beneficiaries entering this study often did not have adequate health care insurance or were afraid of losing Medicare or Medicaid coverage, were not already engaged in community mental health care, were older, and were more physically ill than the usual participants in IPS because they were all SSDI beneficiaries. After randomization, the study had relatively little contact with the control group, other than to conduct the quarterly interviews, and little health data. Thus, these factors were assessed more fully for the treatment group. It was assumed that these problems were equally distributed by randomization. Their similar SF-12 scores at baseline on the physical scale partially validate this assumption. The study has several ways to investigate lack of motivation, physical health, and other barriers in the treatment group. Later chapters report on assessments of engagement in SE and of service utilization. Reports of baseline predictors of employment appear earlier in this chapter. This section considers barriers to competitive employment through ratings made by the clinical teams at the end of each participant‘s participation in the study. Barriers to competitive employment survey. The nurse care coordinator (NCC), along with the SE specialist and other team members rated (as a group) the reasons for lack of employment among those treatment group participants who worked little or not at all. The definition was less than one month of employment during the 24-month study period. Using a fixed list of 14 common barriers from the literature, the teams rated up to three reasons that a participant did not work, and prioritized the factors as the first, second, and third most important, assuming that a combination of factors, rather than just one factor, might have been important for many participants. Site staff completed 430 rating forms. Table 4-14 shows a summary of the ratings on barriers to competitive employment on the 430 participants with completed rating forms. The number of completed ratings by site ranged from 5 to 41, in correspondence with the number of enrolled treatment participants at the site. Three factors dominated the primary barriers category. For 34 percent of the group, poorly controlled symptoms Chapter 4: Outcomes 4-24 of mental illness were the primary barrier; these participants had unremitting or recurrent and severe symptoms despite SMM (if they were even engaged with SMM). For 26 percent of the group, failure to engage in SE was primary; these clients showed little or no interest in meeting with a SE specialist from the beginning. For another 16 percent, poorly controlled physical health problems were primary; these participants were generally more disabled by their severe emphysema, diabetes, or other physical illnesses than by their mental illnesses. As primary barriers, one of these three problems affected over 75 percent of the participants who were unsuccessful in working. No other factor was rated as primary for even 10 percent of the group. Table 4-14. Barriers to competitive employment for non-working treatment group participants1 Priority 1 Barrier/Factor2 Failure to engage in SE Priority 2 Priority 3 Priority 1, 2, or 3 n 111 % 25.8 n 52 % 12.1 n 26 % 6.1 n 189 % 44.0 Disengagement in SE 24 5.6 28 6.5 16 3.7 68 15.8 Physical health problems not controlled Substance abuse/dependence not well controlled Symptoms of mental illness not well controlled Cognitive problems 70 16.3 48 11.2 23 5.4 141 32.8 37 8.6 31 7.2 11 2.6 79 18.4 144 33.5 75 17.4 17 4.0 236 54.9 5 1.2 15 3.5 11 2.6 31 7.2 10 2.3 48 11.2 23 5.4 81 18.8 Family problems Lack of needed services (e.g., case management) Transportation 3 0.7 10 2.3 11 2.6 24 5.6 2 0.5 7 1.6 7 1.6 16 3.7 Lack of work skills 6 1.4 5 1.2 6 1.4 17 4.0 Lack of social skills 4 0.9 13 3.0 18 4.2 35 8.1 Criminal justice system problems 3 0.7 9 2.1 9 2.1 21 4.9 Housing problems 1 0.2 6 1.4 13 3.0 20 4.7 Behavior problems 11 2.6 17 4.0 20 4.7 48 11.2 1Includes participants who did not experience employment success, defined as no employment or only brief employment of less than one month. Rating forms were completed for 430 participants. 2Up to 3 barriers were rated as a priority for each participant. For some participants only 1 or 2 barriers were rated as priorities. As a percentage of the total treatment group (N=1,004), these figures allow us to estimate the influence of primary barriers for participants who did not experience employment success. For example, 111 of the 1,004 treatment group participants, or 11 percent, never engaged in SE. These individuals may have entered the study for reasons other than pursuing employment. For another 144 (14%) of these 1,004 participants the investigative team judged them to have been unsuccessful in work because their symptoms of mental illness were poorly controlled. In addition, another 70 Chapter 4: Outcomes 4-25 (7%) were unable to work primarily because of physical health problems. These percentages are likely underestimates because the percentages only include participants who worked less than one month during the two years of the study and had a completed rating on barriers to competitive employment. Further inspection of Table 4-14 reveals that for many participants, multiple barriers were pertinent. No single factors were highly prevalent as secondary or tertiary barriers, but the sum of barriers (primary, secondary, and tertiary) indicated that several factors were often involved. Poorly controlled symptoms of mental illness affected 55 percent of the group; 44 percent had difficulties with engagement; and 33 percent had poorly controlled physical health problems. In relation to the entire treatment group, poorly controlled symptoms of mental illness impeded work for 23 percent of participants, lack of engagement for 18 percent, and physical health problems for 13 percent. Of course, the total proportions with these problems were probably much higher because many individuals with similar problems probably worked. In addition to these three major barriers, family problems, poorly controlled substance use disorders, disengagement from SE services, and behavior problems affected greater than 10 percent of the group. Factors with extremely low ratings (less than 10%) indicate that treatment group participants in the study did not apparently encounter them. These factors included cognitive problems, lack of services, transportation difficulties, lack of work skills, lack of social skills, criminal justice system problems, and housing problems. Understanding the impact of work on Social Security benefits. Integrated service provisions at each study site also included some form of benefit counseling for treatment group participants. In order to assess whether there was any significant difference over time in understanding and gaining knowledge of the SSDI program benefits, both treatment and control group participants agreed or disagreed in interviews with several statements about their Social Security benefits. Table 4-15 shows the percent of participants who ―agreed‖ with each statement about their Social Security benefits. The baseline and final followup interviews included the same items. The treatment group had a larger decrease in the percent of participants agreeing with the statement: ―As soon as people start working they stop getting their benefit checks.‖ For the treatment group, participants who agreed with this statement went from 16 percent at baseline to 9 percent at followup. The control group had a decrease from 16 percent to 14 percent (p-value = 0.019). Comparisons between the changes observed for the treatment and control groups yield significant increases in the percent of participants who agree with the following statements: ―I can make more money at a job and still collect my benefit checks‖ (77% to 81% vs. 78% to 69%, p-value < 0.001); ―As soon as people start Chapter 4: Outcomes 4-26 working they lose their medical coverage‖ (15% to 12% vs. 14% to 18%, p-value = 0.004); ―Unless a job offers coverage of mental health and prescriptions, I can’t afford to take it‖ (42% to 39% vs. 40% to 44%, p-value = 0.007). The percent of treatment participants who agree with the statement ―If I go to work, get off benefits and get sick right away, I’ll have a hard time getting back on benefits‖ went down from 50 percent to 40 percent while there was very minimal change in the control group (p-value < 0.001). All these significant changes in beliefs in the direction favoring improved knowledge of the treatment group indicate that the benefit counseling was effective in explaining availability of work trial benefits and other options to the treatment participants. Table 4-15. Agreement with statements about Social Security Disability benefits programs by treatment and control group Statements Treatment (N=1,004) Baseline Exit d Control (N=1,051) Baseline Exit d As soon as people start working they stop getting their benefit checks. n 163 90 167 % 16.4 -7.6 7.6 16.4 142 13.7 p-value -2.7 I can make more money just collecting my benefit check than I can if I go to work while on benefits. n 112 115 107 129 % 11.1 11.5 0.4 10.1 12.2 2.1 0.019 0.315 I can make money at a job and still collect my benefit checks. n 774 815 % 77.2 81.3 4.1 826 78.4 730 69.1 -9.3 <0.001 As soon as people start working they lose their medical coverage. n 150 125 % 15.2 12.5 -2.7 149 14.4 183 17.7 3.3 0.004 4.1 0.007 Unless a job offers coverage of mental health and prescriptions, I can’t afford to take it. n 415 384 416 461 % 41.6 38.5 -3.1 39.8 43.9 If I go to work, get off benefits and get sick right away, I’ll have a hard time getting back on benefits. n 502 398 502 498 % 50.5 39.7 -10.8 48.0 47.5 -0.5 I can’t afford to get training to help me get a better job. n 546 432 % 54.4 43.3 -11.1 567 54.2 456 43.6 <0.001 -10.6 0.695 If I knew that I wouldn’t lose all of my benefits, I would try to get a job or get a better job. 500 734 513 n 732 % 73.1 50.2 -22.9 69.8 48.8 -21.0 0.413 NOTE: Weighted percents may not be consistent with unweighted counts. Chapter 4: Outcomes 4-27 There were no statistically significant differences between the treatment and control groups in terms of a percent change in agreement with the statements ―I can’t afford to get training to help me get a better job‖ and ―If I knew that I wouldn’t lose all of my benefits, I would try to get a job or get a better job.‖ In both cases, the percentage of participants agreeing with the statement dropped in favor of improved knowledge, but the drop was consistent across both treatment and control group, and therefore, not significant. Earnings and Income Results on earnings rely on data from two different questionnaire sources: self-reports on earnings in the past month (collected at each of the 8 post-baseline interviews) and self-reports on wages per hour and average hours per week in each job in the past three months (collected only at the final Followup interview). Past month’s earnings. Past month‘s earnings were averaged for each participant over all 8 postbaseline interviews. These averages are shown in Table 4-16. The mean monthly earnings for the treatment group were $148.16 versus $97.41 for the control group, a difference that is both statistically and substantively significant (p-value < 0.001). A large portion of this difference in means was due to the much higher percentage of the treatment group participants reporting positive earnings in one or more of the eight interviews than did control group participants (58.67% vs. 42.72%, p-value < 0.001). However, if the comparison is restricted to only those participants reporting any earnings, the difference in means ($251.12 vs. $227.93) was also significant (p-value < 0.001). The corresponding difference in median earnings ($165.37 vs. $93.98) was even larger and, as is usually the case with earnings data and as is implied by the extent to which the means exceed the median, the distribution of mean earnings was highly skewed to the right. Relatively large differences (with higher figures for the treatment group) were also observed at the 25th and 75th percentiles of the distributions of average monthly earnings (for those who reported any positive average). Past three months’ earnings. The data on earnings in the prior three months from the final Followup interview (Table 4-16) showed significantly higher treatment group mean earnings ($858.60 vs. $478.87, p-value < 0.001) and significantly higher treatment group percentages for having any earnings (33.8% vs. 17.5%, p-value < 0.001). Chapter 4: Outcomes 4-28 Table 4-16. Past month’s earnings (averaged over 8-post baseline interviews), past three months’ earnings (at study exit) and earnings above SGA (averaged over 8-post baseline interviews) by treatment and control group Unconditional (Includes zero earnings) Variable Treatment Control d Past month’s earnings (averaged) N (n) 1,004 1,051 Mean 148.16 97.41 50.75 SD 257.84 283.34 Median 23.79 0.00 23.79 % with earnings >0 Past 3 months’ earnings (at study exit) N (n) 1,004 1,051 Mean 858.60 478.87 SD 1,752.36 1,698.82 Median 0.00 0.00 % with earnings >0 379.73 p-value <0.001 <0.001 0.00 Earnings above SGA n % Conditional (Excludes zero earnings) Treatment Control d 589 251.12 294.32 165.37 59.00 449 227.93 397.97 93.98 42.73 340 2,538.16 2,203.52 1,935.59 33.83 184 2,739.25 3,233.26 1,910.20 17.48 49 8.18 40 8.76 p-value 23.19 <0.001 71.39 16.27 <0.001 -201.29 0.607 25.39 16.35 <0.001 -0.58 0.7394 NOTE: Weighted percents may not be consistent with unweighted counts. Substantial gainful activity (SGA). Table 4-16 also presents treatment group versus control group differences in the percent of participants reporting past month‘s earnings (averaged across all eight post-baseline interviews) was above the current SGA level of $1,000. The differences were generally fairly small and not significant (p-value = 0.7294). For all participants, the relevant percentages were 8.2 percent (treatment) and 8.8 percent (control). Investigators and policy makers have observed that only very small numbers (~4%) of SSA disability participants leave the rolls by working at jobs with earnings above SGA (Stapleton, 2010). This phenomenon is true in spite of successful SE interventions that assist participants in returning to competitive employment. Few SE participants work above SGA and leave the disability rolls. Some policy analysts have observed that even when participants return to work, they seem to work intentionally below the SGA level to avoid losing their benefits. There is speculation and anecdotal evidence that this occurs because participants fear losing their cash payments and their access to health insurance if they leave the rolls, even if they leave for better paying jobs. This could occur because of a lack of awareness of trial work programs and other ways to return to work and not lose benefits or because participants are afraid that if they relapse from their underlying impairments or they lose their current work arrangement, they will have difficulty returning to benefits in the Chapter 4: Outcomes 4-29 disability program. It took many workers a long time to get on the program, having declared themselves unable to work at any job or perform SGA, and they fear losing those benefits. The phenomenon of working just below the SGA level is called ―parking‖ (Schimmel, Stapleton, & Song, 2010; Porter, Smith, Payette, Tremblay, Burt, 2009), and the MHTS tested the distribution of earnings to determine if MHTS participants exhibited earnings distribution consistent with the ―parking‖ phenomenon. It should be noted that a recent GAO report on the effect of SGA levels on work behavior indicated that very few participants exhibited the ―parking‖ phenomenon. A distribution for each of the 8 quarters of employment reported by participants in the MHTS— both participants in the treatment and control conditions—was examined to see what proportion of individuals had earnings that fell into each of three categories of earnings: above SGA, between SGA and 75 percent of SGA, and below 75 percent of SGA. Ideally, the MHTS would have been successful if many participants were able to work steadily above SGA and leave the disability program entirely—earning higher wages and salaries and obtaining health insurance in the workplace. First, workers had earnings all across the earnings distribution but mostly below SGA indicating that return to work was difficult for MHTS participants and that SGA might be a natural barrier for work capacity for individuals. Second, workers worked many quarters in the range of 75 percent of SGA but below SGA because they were capable of considerable earnings from work but deliberately chose to work below SGA to avoid loss of cash benefits (―parking‖). Table 4-17 shows the number of MHTS participants who have earnings in three different ranges of earnings. Note that some workers will report earnings in all three distributions as the observation is persons by quarter and not an unduplicated count of workers between the three tables reporting on three different ranges of earnings. Across eight quarters, only 13.5 percent of participants in the treatment group reported earnings above SGA in the last month of one or more quarters, and only 5.6 percent did so for two or more quarters. The comparable figures for the control group were 8.2 percent and 3.4 percent, respectively. The vast majority of those who did report a quarter of work above SGA report earnings at that level in only one or two quarters. Only ten workers in both groups combined worked six or more quarters above SGA. Among the participants that reported work between 75 percent and 99 percent of SGA, 15.9 percent of those in the treatment group reported at least one quarter of work and only 6.4 percent did so for two or more quarters. The comparable figures for the control group were 7.9 percent and 2.3 percent, respectively. Again, as observed for work above SGA, the vast majority of those reporting work in this range do so for only one or two quarters. In fact, only five treatment group members Chapter 4: Outcomes 4-30 report six or more quarters in this range, and none in the control group report six quarters or more above 75 percent of SGA but below SGA itself. An assessment of earnings in the latter four quarters of the study did not yield anything different from above results (not shown). Table 4-17. 0 1 2 3 4 5 6 7 8 Q1-Q8 Number of quarters with past month’s earnings above, near, or below SGA by treatment and control group Greater than or equal to SGA Treatment Control N % N % 868 86.5 963 91.6 80 7.9 50 4.8 29 2.8 13 1.2 7 0.7 9 0.8 7 0.7 6 0.6 9 0.9 4 0.4 0 0.0 2 0.2 2 0.2 3 0.3 2 0.2 1 0.1 136 13.5 88 8.2 Between 75% and 99% SGA Treatment Control N % N % 844 84.1 968 92.1 97 9.7 59 5.6 38 3.8 16 1.5 16 1.6 2 0.2 3 0.3 5 0.5 1 0.1 1 0.1 3 0.3 0 0.0 1 0.1 0 0.0 1 0.1 0 0.0 160 16.1 83 7.9 Below 75% SGA Treatment Control N % N % 447 44.5 630 59.9 168 16.7 170 16.2 116 11.6 97 9.2 90 9.0 47 4.5 65 6.5 28 2.7 55 5.5 34 3.2 32 3.2 21 2.0 19 1.9 17 1.6 12 1.2 7 0.7 557 55.8 421 39.9 NOTE: Weighted percents may not be consistent with unweighted counts. Although earnings were significantly higher for participants in the treatment group compared with those in the control group, few people were earning greater than SGA. Furthermore, there was little work yielding earnings in the range of 75 percent to 99 percent of SGA. These data suggest that there is little evidence of ―parking‖ of earnings just below the SGA level to avoid loss of benefits by participants returning to work who ―could‖ work more but do not do so to avoid loss of benefits. MHTS participants may be aware of the SGA level of earnings and set target earnings below this level, but it does not appear to result in a sustained pattern of working to receive earnings just below SGA. Income from sources other than earnings. Each post-baseline interview collected data for a number of other sources of income received in the past month. These sources include SSDI, SSI, SSA retirement or survivors benefits, Veterans Affairs (VA) benefits, other public welfare, food stamps or Temporary Assistance for Needy Families (TANF), vocational programs, and unemployment compensation. For many of these sources, the percentage of participants who participated was very low (5% or less) reporting receipt of any income from that source. The exceptions were SSDI benefits and SSI benefits, as well as income from family members, food stamps or TANF, and a residual ―other income‖ category. Chapter 4: Outcomes 4-31 Results in Table 4-18 indicate that mean SSDI income participants received was almost identical for treatment ($855.49) and controls ($853.00). The SSI income participants received in the past month also was not significantly different between the treatment and control groups. The analysis included an examination of the past month‘s income from other public programs (averaged over the 8 postbaseline interviews) by treatment and control group. Table 4-18. Past month’s SSDI and SSI income (averaged over 8 post-baseline interviews) by treatment and control group Unconditional (Includes zero income) Treatment Control d Variable SSDI N (n) 1,004 Mean 855.49 SD 340.30 Median 787.48 % with income >0 1,051 853.00 337.75 784.22 p-value 2.49 0.695 3.26 Conditional (Excludes zero income) Treatment Control D 1,003 856.34 339.40 787.54 59.00 1,050 853.77 336.97 784.60 99.90 241 93.41 89.52 59.54 24.50 259 105.37 94.71 79.08 24.61 p-value 2.57 0.692 2.94 -40.90 0.945 -11.96 0.175 -19.54 -0.11 0.952 SSI N (n) 1,004 Mean 22.48 SD 60.65 Median 0.00 % with income >0 1,051 25.94 65.40 0.00 -3.46 0.854 -4.75 NOTE: Weighted percents may not be consistent with unweighted counts. Results in Table 4-19 indicate that the mean SSA retirement or survivors benefits were similar for treatment ($117.01) versus control ($115.74) groups. The results were similar for the past month‘s income reported from other public welfare ($79.99 vs. $66.21; p-value = 0.906) and food stamps or TANF ($74.26 vs. $78.96; p-value = 0.269). Mean VA benefits in the past month were higher among the treatment group than the control group, but again, the difference was not statistically significant ($689.81 vs. $351.73; p-value = 0.102). Similarly, income from vocational programs ($58.66 vs. 85.44; p-value = 0.173) and unemployment compensation ($133.25 vs. $169.39; p-value = 0.773) seemed to be lower among participants in the treatment group; however, neither of these differences was statistically significant. There were more participants in the treatment group that reported nonzero income from vocational programs (6.03% vs. 2.07%; p-value < 0.001) and unemployment compensation (3.62% vs. 2.28%; p-value = 0.071). These results indicate that treatment and control groups were very similar in reporting past month‘s income from other public programs. Chapter 4: Outcomes 4-32 Table 4-19. Past month’s income from other public programs (averaged over the 8 postbaseline interviews) by treatment and control group Unconditional (Includes zero income) Variable Treatment Control d SSA retirement or survivors benefits N (n) 1,004 1,051 Mean 4.01 5.44 -1.43 SD 39.16 33.44 Median 0.00 0.00 0.00 % with income >0 p-value 0.139 Conditional (Excludes zero income) Treatment Control d 34 117.01 180.53 40.42 3.42 50 115.74 109.66 84.36 4.70 p-value 1.27 0.211 -43.94 -1.28 0.146 338.08 0.102 40.09 0.51 0.532 13.78 0.906 2.00 0.06 0.962 -4.70 0.269 -5.86 -1.76 0.427 -26.78 0.173 -9.86 3.96 <0.001 -36.14 0.773 -26.46 1.34 0.071 VA benefits N (n) Mean SD Median % with income >0 1,004 1,051 26.05 216.02 0.00 11.52 118.38 0.00 Other public welfare N (n) Mean SD Median % with income >0 1,004 7.54 41.48 0.00 Food stamps or TANF N (n) Mean SD Median % with income >0 1,004 36.79 65.76 0.00 Vocational program N (n) Mean SD Median % with income >0 1,004 3.54 33.62 0.00 Unemployment compensation N (n) 1,004 Mean 4.82 SD 36.80 Median 0.00 % with income >0 1,051 6.20 31.99 0.00 1,051 40.51 72.39 1.18 1,051 1.77 22.59 0.00 1,051 3.85 39.60 0.00 14.53 0.00 1.34 0.959 0.00 -3.72 0.290 -1.18 1.77 <0.001 0.00 0.97 0.00 NOTE: Weighted percents may not be consistent with unweighted counts. Chapter 4: Outcomes 0.522 4-33 0.072 39 35 689.81 917.00 141.15 3.78 351.73 567.37 101.06 3.27 94 79.99 110.56 36.65 9.42 99 66.21 83.96 34.65 9.36 485 74.26 75.88 47.83 49.54 539 78.96 84.88 53.69 51.30 62 58.66 127.45 12.79 6.03 22 85.44 135.06 22.65 2.07 37 133.25 148.74 64.72 3.62 24 169.39 207.68 91.18 2.28 Individual and household income. An examination of total income from all sources for the individual (averaged across the 8 post-baseline interviews) and total household income (at study exit) is presented below. Given the observed differences in past month‘s earnings noted earlier, it is relevant to see whether the increased earnings for the treatment group translated into higher total income or were offset by reductions in income from other sources, and particularly from reductions in SSI or SSDI benefit payments. Table 4-20 shows the total monthly individual income averaged over the study and the total monthly household income at exit for participants in the treatment and control groups. The analysis reveals a highly significant difference in the mean value of average monthly overall individual income for all participants ($1180.52 treatment vs. $1120.96 control; p-value < 0.001). Presumably, this larger mean income observed for treatment group participants is due almost entirely to the larger mean earnings for these same participants relative to controls. Results for comparisons of total household income reported at the final Followup interview between treatment and control group participants are also presented in the table below. The treatment group reported higher monthly total household income ($1,678.30 vs. $1,661.56; p-value = 0.063), though the differential was somewhat smaller than that observed for individual earnings or for total individual income. Table 4-20. Past month’s total individual income (averaged over 8 post-baseline interviews) and past month’s total household income (at study exit) by treatment and control group Unconditional (Includes zero income) Variable Treatment Control d Past month’s total individual income N (n) 1,004 1,051 Mean 1,180.52 1,120.96 59.56 SD 495.14 525.64 Median 1,072.84 987.05 85.79 % with earnings >0 Past month’s total household income N (n) 1,004 1,051 Mean 1,678.30 1,661.56 SD 1,180.98 1,248.01 Median 1,296.63 1,199.85 % with earnings >0 16.74 p-value <0.001 0.063 96.78 NOTE: Weighted percents may not be consistent with unweighted counts. †Not applicable. Chi-square not calculated. Chapter 4: Outcomes 4-34 Conditional (Excludes zero income) Treatment Control d 1,004 1,180.52 495.14 1,072.84 100.00 1,051 1,120.96 525.64 987.05 100.00 997 1,689.90 1,176.95 1,298.11 99.31 1,048 1,666.51 1,246.50 1,201.46 99.70 p-value 59.56 <0.001 85.79 0.00 † 23.39 0.043 96.65 0.213 Trends in earnings differentials. The clearest treatment effects were on earnings and on total income figures that included earnings. However, further exploration assessed the time pattern of emergence of these earnings effects. Given the emphasis in IPS SE on rapid job placement, one might expect that the treatment effect on earnings would be greater earlier in the 24-month followup period and less pronounced later in the period as control group participants found work. To test this hypothesis, a regression model was used to compute the linear time trend of past month‘s earnings for each participant over the eight post-baseline interviews. Averages for the individual time trend coefficients were computed for treatment and control group participants, and then tested for differences in the mean values of these time trend coefficients between the two groups. The results in Table 4-21 indicate that the mean quarterly increase in the past month‘s earnings figure was $13.20 for the treatment group versus $4.25 for the control group; this difference was highly significant (p-value < 0.001). In summary, the results suggest that the effect of the treatment on earnings in fact increased over the 24-month followup study period. Table 4-21. Past month’s earnings regression slope (averaged over 8 post-baseline interviews) by treatment and control group Unconditional (Includes zero earnings) Variable Treatment Control d p-value Past month’s earnings N (n) 1,004 1,051 Mean 13.20 4.25 8.95 <0.001 SD 63.52 55.86 Median -0.04 -0.04 0.00 Conditional (Excludes zero earnings) Treatment Control d p-value 589 22.38 81.29 9.54 449 9.95 85.17 -0.81 12.43 <0.001 10.35 NOTE: Weighted percents may not be consistent with unweighted counts. Earnings and income by subgroups. For many of the earnings and income outcome measures, Table 4-22 presents a summary of the results of significance tests that were conducted to determine whether differences between treatment and control group participants were statistically significant for the following subgroups: age, gender, diagnosis, and education. These subgroup comparisons are presented for the same three summary statistics analyzed for the overall comparisons— unconditional means (i.e., includes reports of zeroes), conditional means (i.e., excludes reports of zeroes), and percent reporting non-zero earnings and income amounts. Summary descriptive statistics and the results of the significance tests are presented in Appendix 4C for all measures and all subgroups shown in the table below. Chapter 4: Outcomes 4-35 Table 4-22. Statistically significant earnings and income subgroup comparisons for age, gender, diagnosis, and education for unconditional mean comparisons, conditional mean comparisons, and percent reporting non-zero amounts comparisons Variables Unconditional mean comparisons Past month’s earnings (averaged) Past 3 months’ earnings (at study exit) Past month’s SSDI Past month’s SSI Past month’s total individual income Past month’s total household income Past month’s earnings regression slope Age 18 to 35 35+ * Conditional mean comparisons Past month’s earnings (averaged) Past 3 months’ earnings (at study exit) Past month’s SSDI Past month’s SSI Past month’s total individual income Past month’s total household income Past month’s earnings regression slope % Reporting non-zero amounts comparisons Past month’s earnings (averaged) * Past 3 months’ earnings (at study exit) Past month’s SSDI Past month’s SSI Past month’s total individual income Past month’s total household income Past month’s earnings regression slope * Gender Diagnosis Education M F A S < HS HS grad > HS * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * p-value < 0.05 Analyses of treatment group versus control group differences in average past month‘s earnings were also reported by age group (18–34 vs. 35+), by gender, by diagnosis (affective disorder vs. schizophrenia), and by education level (less than high school, high school graduate, and more than high school) (see Appendix 4C). The pattern of results is generally very similar to that described for the overall group; however, there were some minor differences. Rates of participants reporting any Chapter 4: Outcomes 4-36 positive earnings, which were higher for younger participants, were still significantly higher for the treatment group, but the difference in means among those with positive earnings was not significant. Since the magnitude of this difference in means was even larger for younger participants, the lack of statistical significance was largely a result of very small numbers. While the percent reporting non-zero past month‘s earnings was greater for females in the treatment group, the conditional mean of past month‘s earnings among females was not significant. Conditional mean past month‘s earnings were also not significantly different between the treatment group and the control group for several other subgroups: participants 18 to 35, participants with schizophrenia, participants with a high school diploma, and participants with less than a high school education. In the case of participants with schizophrenia, the treatment versus control difference in the conditional mean was positive but small; in the case of participants with less than a high school education, this difference was large but the numbers of participants were small. As noted above, the pattern of results for all other subgroups was similar to that for the entire study. With regard to differences in the percentage of participants with average earnings above SGA for subgroups, similar to the comparison for the overall group, no subgroup comparisons were statistically significant, largely due to the very small sample size for this measure. Comparisons of past three months‘ earnings (at study exit) revealed statistically significant differences between treatment and control unconditional mean earnings, as well as significantly higher treatment group percentages for having any earnings, for all subgroups except the small number of participants who were under age 35 (for whom no significant treatment vs. control differences exist). On the other hand, comparisons between conditional means showed little difference between treatment and control for any subgroups of participants. Analysis of total individual income revealed differences in the same direction that were also generally significant for almost all respondent subgroups; the only exception was the non-significant difference for participants under age 35. The only subgroups that showed significantly greater household income for the treatment group were participants under age 35, males and females, participants with schizophrenia, and those participants with more than a high school education. Estimates of MHTS Impacts on Earnings and Income In the previous section, the descriptive data presented used the imputed data for ease of understanding the results. In this section, the analysis focuses on the results from multivariate Chapter 4: Outcomes 4-37 analyses of MHTS effects earnings and income. Heteroscedasticity is a particular concern that complicates the multivariate analyses due to unequal variances between groups. For example, the large number of participants with zero earnings creates skewed distributions in some of the comparisons. Because of these issues with heteroscedasticity (unequal variances), the multivariate analysis used the unimputed data. Data on dependent variables. The dependent variable measures originate from questions asked in each of the eight followup interviews concerning earnings and income from other sources in the immediate past 30 days prior to the interview date. The analysis includes estimates of MHTS impacts for three different followup periods: quarters one through eight (i.e., two years of MHTS), quarters one through four (i.e., first year of the MHTS), and quarters five through eight (i.e., second year of the MHTS). Earnings variables. The analyses used two different sets of dependent variables for earnings. One definition uses the average past month‘s ―earned income or money from all paid employment, including tips or commissions.‖ The second definition uses the same average earned income figure plus the average of the past month‘s informal earnings; the latter figure based on responses pertaining to ―money received by doing odd jobs such as babysitting or yard work, helping in a business, or doing work ‗under the table‘.‖ For brevity, these two types of earnings variables are referred to as ―formal‖ earnings and ―formal plus informal‖ earnings. Descriptive statistics on these dependent variables pertaining to earnings, and their corresponding baseline values, are shown in Table 4-23. Comparing the overall baseline values with the values for the first four quarterly followup interviews (year 1) and the second four quarterly followup interviews (year 2), a clear upward trend is discernable. For formal earnings, the upward trend was from a baseline mean of $18.69 to average monthly figures of $103.25 and $139.73 for year 1 and year 2 respectively. Analogous figures for formal plus informal earnings are $30.76, $110.67, and $146.82, respectively. Comparisons between treatment vs. control groups show somewhat higher baseline earnings figures for the control group, a strong upward trend in earnings for both groups (especially through the first year of followup), substantially higher earnings for the treatment group in followup quarters 1-4, and an even greater excess of treatment group earnings versus controls in year 2. Chapter 4: Outcomes 4-38 Income variables. Five different income measures are examined below: 1. Disability income from Social Security (SSDI + SSI); 2. All public sector income (SSDI + SSI +Social Security retirement or survivors benefits + VA + general assistance + food stamps or TANF + vocational programs + unemployment compensation); 3. Private non-earned income; 4. Total non-earned personal income (public + private); and 5. Total personal income (earned + unearned). Table 4-23 presents the descriptive statistics on average values of each of these five variables along with corresponding baseline values over the 24-month followup period. Average disability income from social security (SSDI + SSI) was almost the same for both treatment and control groups in the baseline as well as the followup period, and was only very slightly higher in the followup period. A similar pattern shows for all public sector income supports, though the baseline value for the control group was slightly lower than for the treatment group, thus the increase in this figure for the followup period was slightly greater for the control group. Private sector non-earned income was small relative to the income from social security (SSDI + SSI figure) and the overall public sector figure, but it is interesting to note that the baseline value for private sector non-earned income was modestly higher for the treatment group. The followup figures were almost identical for the two groups, with the treatment group showing a very small decline from baseline to followup while the control group showed a modest increase. Total non-earned income was about 4 percent higher in the baseline for the treatment group ($1,035) than the controls ($996). Followup values for total non-earned income were almost identical between the two groups, because of a very small increase from baseline for the treatment group and a slightly larger increase for the controls. Finally, note that total personal income was only about 3 percent higher at baseline for the treatment group versus the controls ($1,063 vs. $1,031), while the increase from baseline to followup was somewhat greater for the treatment group, resulting in a differential of about 5 percent between the treatment and controls at followup ($1,194 vs. $1,130). This differential likely occurred because of the differential growth of earnings for the treatment group. Chapter 4: Outcomes 4-39 Table 4-23. Descriptive statistics for dependent measures included in multivariate analyses of the MHTS impact on earnings and income by treatment and control group Variables Dependent variables Avg. formal earnings (Q1–Q8) Avg. formal earnings (Q1–Q4) Avg. formal earnings (Q5–Q8) Avg. formal + informal earnings (Q1–Q8) Avg. formal + informal earnings (Q1–Q4) Avg. formal + informal earnings (Q5–Q8) Avg. SSDI + SSI income (Q1-Q8) Avg. total public income support (Q1–Q8) Avg. private non-earned income (Q1–Q8) Avg. total non-earned income (Q1–Q8) Avg. total personal income N All observations M SD N Treatment only M SD N Control only M SD 2,079 120 281 1,015 146 272 1,064 94 288 2,072 103 282 1,014 119 265 1,058 87 298 1,999 140 351 971 176 355 1,028 103 343 2,079 127 285 1,015 153 274 1,064 101 292 2,072 111 286 1,014 127 268 1,058 94 303 1,999 147 354 971 183 357 1,028 110 347 2,075 885 345 1,011 884 342 1,064 886 348 2,075 961 391 1,011 967 412 1,064 956 369 2,078 72 314 1,015 72 338 1,063 73 288 2,071 2,071 1,035 1,162 528 602 1,010 1,010 1,039 1,194 551 608 1,061 1,061 1,030 1,130 504 595 0.50 0.50 2,074 19 107 1,013 16 96 1,061 21 117 2,068 31 147 1,009 27 117 1,059 35 172 2,011 1,047 548 984 1,063 607 1,027 1,032 482 2,028 947 439 993 962 499 1,035 932 368 2,068 69 264 1,009 75 302 1,059 63 218 2,018 1,016 518 988 1,035 580 1,030 996 446 2,034 870 345 997 874 355 1,037 867 337 Independent variables 2,079 Treatment (yes=1) Baseline dependent variable values Baseline past month’s formal earnings Baseline past month’s formal + informal earnings Baseline past month’s total personal income Baseline past month’s total public income Baseline past month’s private non-earned income Baseline past month’s non-earned personal income Baseline past month’s SSDI + SSI Chapter 4: Outcomes 4-40 Table 4-23. Descriptive statistics for dependent measures included in multivariate analyses of the MHTS impact on earnings and income by treatment and control group (continued) All observations Variables N M SD Baseline socio-demographics 2,077 45.4 7.7 Age at enrollment 2,079 0.47 0.50 Gender (male=1) 2,079 0.39 0.49 Race (non-white=1) 1,015 1,015 1,015 45.3 0.47 0.39 7.7 0.50 0.49 1,062 1,064 1,064 45.6 0.48 0.38 7.7 0.50 0.49 High school graduate (yes=1) Some college (yes=1) N Treatment only M SD N Control only M SD 2,078 2,078 0.69 0.18 0.46 0.39 1,014 1,014 0.69 0.19 0.46 0.39 1,064 1,064 0.70 0.18 0.46 0.38 2,075 0.38 0.82 1,015 0.38 0.85 1,060 0.39 0.78 2,076 2,074 0.46 0.46 0.87 0.50 1,015 1,015 0.46 0.46 0.89 0.50 1,061 1,061 0.46 0.46 0.85 0.50 2,077 0.23 0.42 1,015 0.23 0.42 1,062 0.23 0.42 2,079 0.71 0.45 1,015 0.69 0.46 1,064 0.73 0.44 2,079 2,079 110 0.17 72 0.37 1,015 1,015 108 0.17 73 0.37 1,064 1,064 113 0.17 70 0.38 2,079 8,496 Work history and vocational services 3,458 1,015 8,536 3,438 1,064 8,456 3,479 Baseline health status Fair or poor health (yes=1) Physically limited in daily activities (yes=1) Baseline hospital stays Baseline emergency room visits Diagnosis (affective disorder=1) SSA program variables Months on rolls Concurrent with SSI Primary insurance amount (ln) Sq. root of reported earnings in 23 months pre-baseline No reported earnings in 23 months prebaseline (yes=1) Worked in past 2 years (pre-baseline) Trial work period end date 0-3 years postrecruitment Trial work period end date 0-10 years ago Had active ticket ever 2,078 1,473 4,910 1,015 1,430 4,733 1,063 1,515 5,084 2,079 0.79 0.41 1,015 0.80 0.40 1,064 0.78 0.41 2,074 0.30 0.46 1,012 0.28 0.45 1,062 0.31 0.46 2,078 0.02 0.15 1,015 0.03 0.17 1,063 0.02 0.13 2,078 2,078 0.14 0.14 0.35 0.35 1,015 1,015 0.15 0.14 0.36 0.35 1,063 1,063 0.14 0.14 0.34 0.35 Covariates used in the regression analyses. Regression estimates of the effects of the MHTS intervention on the dependent variables came from regressions that included a selection of baseline covariates as additional statistical controls. As is indicated in Appendix 4D, covariates selected to represent the following categories of explanatory factors include the following: socio-demographic characteristics, baseline health status (including recent use of health or mental health services), receipt of benefits (both status and history), recent labor market history and work status, and longer- Chapter 4: Outcomes 4-41 term work history and vocational services. Covariates pertaining to receipt of benefit history, and some of the covariates relating to history of work and vocational services, were selected because these were significant predictors of enrollment probability in our pervious analysis on participants‘ decision to participate in the MHTS (Discussed in Chapter 3). Functional form and estimation method. The choice of the functional form and estimation methods reflect several considerations about the distribution of the dependent variable. First, the significant number of participants who had no earnings in the relevant followup period resulted in a large concentration of zero values for the dependent variables. Second, the presence of small numbers of very large dependent variable values suggests that heteroscedasticity (i.e., differing variance) is a strong possibility. Due to the expected heteroscedasticity pattern, it is also very likely that there is a positive relationship between the expected value of the dependent variable and the variance of the unobservable random disturbance. Third, in the presence of heteroscedasticity and the right-skewed distribution of observed positive values for the dependent variable, a modeling approach that involves transformation of the dependent variable values (e.g., a two-part model with a logarithmic transformation, or a one-part model with a square root transformation) raises practical difficulties in generating estimates of marginal effects via a retransformation process. Given these considerations, the most straightforward approach for using a regression model for generating MHTS impact estimates is to use the generalized linear model (GLM) technique with a logarithmic link function and a gamma error distribution. This model allows the variance of the unobservable disturbance to increase with the square of the expected value of the dependent variable. This method has previously been applied in several recent studies that analyze earnings for persons with serious mental disorders. One or more regression analyses did not include a small number of participants because of ―don‘t know‖ or ―refused‖ responses to questions about income amounts from specific income sources. A few other cases were dropped because they had missing values for baseline covariates. Specifically for regressions using the 12-month followup periods of earnings, a few other cases were dropped because the participant did not complete any quarterly interviews during the relevant 12-month time period. Since the dependent variables were averaged across all complete interviews for the relevant followup periods, and since some participants did not complete all interviews, it is important to consider variations among participants in the number of quarters over which the average earnings values were computed as a possible source of heteroscedasticity. The investigators expected a correlation Chapter 4: Outcomes 4-42 between the variance of the errors in our regression estimates and the number of interviews over which the average dependent variable values were computed. The number of interviews used to compute the average values for the dependent variables was generally close to the maximum. For example, for formal earnings, for the two-year study (with a maximum of 8 such interviews), the mean was 7.09; for the 12-month (first year) interviews (with a maximum of 4 such interviews), the mean numbers of interviews were 3.59 and 3.65 for the first and second 12-month periods respectively. Mean numbers of interviews for formal-plus-informal earnings were the same. Similar observations pertain to our five dependent variable measures of income over the 24-month study period. The mean numbers of interviews used for averaging were as follows: (1) SSDI-SSI 6.95; (2) All Public-Sector 6.92; (3) Private Sector Non-earned 7.10; (4) Total Not Earned 7.12; (5) Total Personal Income 6.83. Differences between these figures and the maximum of 8 again were relatively small.4 Accordingly, it is reasonable to conclude that variation in numbers of completed interviews was not an important source of heteroscedasticity. Therefore, there was no need for further adjustments for heteroscedasticity beyond the gamma distribution assumption for the error term and the use of robust variance estimates. Regression results. Regression results are reported for 4 different specifications of covariates: regressions with only the treatment group (1 = yes, 0 = no) as an explanatory variable (Model 1), regressions with the treatment group and the baseline value of the dependent variable as the explanatory variables (Model 2), regressions that add the other categories of covariates (Model 3), and regressions that replicate Model 3 but exclude all covariates with a p-value for their estimated average marginal effect that is > 0.2. In presenting the regression results, the emphasis is on the marginal effects of the treatment variable (the MHTS dummy) and the selected covariates defined above. These marginal effects, computed for each participant using the observed covariate values, were then averaged over all participants in 4 As these mean figures suggest, the numbers of beneficiaries with very small numbers of interviews used for computing our dependent variable values were very small. For 24-month followup data, for formal earnings only 2.54 percent of beneficiaries had fewer than 3 interviews with available data; for informal earnings the corresponding figure was 2.64 percent. For the 12-month followups, the percentages of beneficiaries with only one interview with usable earnings data ranged from 2.30 percent to 3.20 percent. For our 5 income variables measured over the 24-month followup, corresponding percentages of beneficiaries with less than 3 usable interviews were as follows: (1) SSDI-SSI 3.63; (2) All Public-Sector 3.69; (3) Private Sector Non-earned 2.54; (4) Total Not Earned 2.31; (5) Total Personal Income 4.03. Of course, excluding from our analyses all beneficiaries who completed no followup interviews beyond the first quarter is consistent with these low percentages. Chapter 4: Outcomes 4-43 the analysis. Thus, the estimated MHTS impact is the average difference between the predicted values of the dependent variables (for values of 0 vs. 1 for the treatment variable) with the average computed over the entire weighted study sample of participants. Variance estimates for all marginal effects were obtained using robust methods and allow for clustering at the site level. Table 4-24 reports the estimated MHTS effects on earnings for Models 1 to 3. In all cases, these estimated effects were positive and significant. Controlling for the baseline dependent variable value and on the values of the other baseline covariates, results in substantial increases in the estimated size of the MHTS effects across all three models. There was also a clear tendency for the estimated effects to be larger for the second year of the study compared to the first year of the study. For example, the results with Model 3 indicate that the estimated MHTS effect in the second year was equal to or more than $100 per month in average earnings compared to estimates of $50 and $63 per month in the first year. Table 4-24. Estimates of average marginal MHTS effects on earnings Dependent variables Avg. formal earnings (Q1–Q8) Avg. formal earnings (Q1–Q4) Avg. formal earnings (Q5–Q8) Avg. formal + informal earnings (Q1–Q8) Avg. formal + informal earnings (Q1–Q4) Avg. formal + informal earnings (Q5–Q8) N Model 1 Avg MEff p-value Model 2 Avg MEff p-value N 2,051 51.17 <0.001 2,051 56.76 <0.001 2,051 72.35 <0.001 2,044 30.87 0.001 2,044 35.32 2,044 50.12 <0.001 1,972 73.16 0.001 1,972 78.00 <0.001 1,972 100.00 <0.001 2,046 52.54 <0.001 2,046 67.31 0.001 2,046 80.70 <0.001 2,039 32.51 0.001 2,039 52.07 0.031 2,039 62.60 <0.001 1,967 74.07 0.001 1,967 81.09 0.001 1,967 102.19 <0.001 0.001 N Model 3 Avg MEff p-value Model 1: GLM model (gamma family, log link) with treatment dummy as only independent variable. Model 2: GLM model (gamma family, log link) with treatment dummy and baseline earnings as only independent variables. Model 3: GLM model (gamma family, log link) with treatment dummy, baseline earnings, and 20 other covariates as independent variables. Table 4-25 shows the estimated MHTS effects on the five income measures. Most of these estimates were not statistically significant; however, the estimates of small negative MHTS impacts on SSDI + SSI income did have p values < 0.1. Estimates for broader measures of non-earned income, however, did not support the hypothesis that MHTS treatment reduces receipts of non-earned income. Positive and significant estimated MHTS effects on total personal income were clearly the result of the positive MHTS effects on earnings; the fact that these estimates were somewhat smaller Chapter 4: Outcomes 4-44 than the estimated earnings effects suggests the possibility of negative MHTS impacts on nonearned income but other results in the table give limited support to that hypothesis. Table 4-25. Estimates of average marginal MHTS effects on income Dependent variables Avg. SSDI + SSI (Q1–Q8) Avg. public income (Q1–Q8) Avg. private non-earned income (Q1–Q8) Avg. non-earned personal income (Q1–Q8) Avg. formal + informal earnings (Q5–Q8) Model 1 Avg N MEff p-value 2,010 -5.88 0.645 Model 2 Avg N MEff p-value 2,010 -11.99 0.071 Model 3 Avg N MEff p-value 2,010 -11.19 0.089 2,004 4.25 0.807 2,004 -17.98 0.096 2,004 -14.51 0.129 2,045 -0.30 0.984 2,045 80.85 0.892 2,045 865.96 0.555 1,992 3.14 0.867 1,992 -14.23 0.406 1,992 -12.55 0.378 1,985 61.13 0.004 1,985 0.006 1,985 52.14 0.002 61.60 Model 1: GLM model (gamma family, log link) with treatment dummy as only independent variable. Model 2: GLM model (gamma family, log link) with treatment dummy and baseline earnings as only independent variables. Model 3: GLM model (gamma family, log link) with treatment dummy, baseline earnings, and 20 other covariates as independent variables. Estimated average marginal effects of all other covariates are reported in Appendix Table 4D (for formal earnings), Appendix Table 4E (for formal + informal earnings,) and in Appendix Table 4F (for the five income dependent variables). These tables also report analogous results for the MHTS treatment dummy that were already shown (for Model 3) in Tables 2 and 3, as well as the MHTS results when insignificant covariates were deleted from Model 3. It is clear that the estimated MHTS effects were quite stable to the exclusion of covariates that were least significant in the full Model 3. Health and Quality of Life While the primary outcomes of interest were vocational, focusing on employment and earnings, the rich nature of the treatment was also expected to affect health and functioning. Access to SMM and other behavioral health services was a significant part of the treatment package. For those participants who took advantage of the services, improvements were expected. Health status. This study reports on two health status scores—the Mental Component Score (MCS), which provides a summary of the participant‘s mental health in the past 4 weeks, and the Physical Component Score (PCS), which provides a summary of the participant‘s physical health in the past 4 weeks. Population norms associated with the MCS and PCS have a mean of 50 with a Chapter 4: Outcomes 4-45 standard deviation of 10. Thus, higher scores indicate better mental or physical health. As scores deviate from the population mean of 50, they indicate better (above 50) or worse (below 50) health than persons in the normal population. The norms were based on the 1998 U.S. population (Ware, Kosinski, Turner-Bowker, & Gandek, 2002). Table 4-26 presents a summary of the MCS and PCS at baseline and study exit for both the treatment and control groups. As can be seen in the table, mental health scores are well below the normal population between 1 and 1 ½ standard deviations, indicating that the mental health of participants in the study was quite poor. However, the results indicate that average MCS improved in the treatment group (from 35.83 to 38.85) during their 24 months in the study, while the average MCS of participants in the control group changed very little (from 35.96 to 35.92). The difference in change scores (3.02 vs. -0.04) was statistically significant (p-value < 0.001). Table 4-26. Mental and physical health status at baseline and study exit by treatment and control group Variable Mental health Mean SD Physical health Mean SD Treatment (N=1,004) Baseline Exit d Control (N=1,051) Baseline Exit d p-value 35.83 13.08 38.85 13.37 3.02 35.96 13.00 35.92 13.27 -0.04 <0.001 44.27 11.90 43.13 11.69 -1.14 43.96 11.86 42.92 12.21 -1.04 0.924 Further analyses of the average MCS scores (see Appendix 4G) of participants reveal that the treatment versus control group differences were also statistically significant (at least at the.05 level) for the following key subgroups: Participants in the age groups 18 to 34 and those 35 and older, Males and females, High school graduates and participants with more than a high school education, and Participants diagnosed with an affective disorder as well as participants diagnosed with schizophrenia. The only subgroup that did not show any significance between treatment and control at a significance level of.05 from baseline to study exit were participants with less than a high school Chapter 4: Outcomes 4-46 education (p-value = 0.060; refer to Appendix 4G for statistics); however, increasing the significance level to.10 reveals marginally significant differences for this subgroup. Average physical health scores were comparatively higher than the mental health scores. While still falling below normal population mean of 50, physical health scores for both the treatment and control groups remained within a single standard deviation of the normal population. The average scores dropped slightly between both the treatment and control groups between the Baseline and final Followup interviews. Thus, the physical health of participants in the study declined slightly in the 24 months of study participation. The difference in change scores between the treatment and control groups (or among any subgroup—see Appendix 4G) was not significant (p-value = 0.924). Predicting mental health status. A stepwise linear regression model provided a means to obtain estimates for the mental health score at study exit. MHTS treatment showed a highly significant positive impact on mental health status (see Table 4-27). Participants with higher baseline mental health scores, those with an affective disorder, and participants with an active ticket also reported better mental health at study exit. While age had a negative effect, participants living in areas with higher population density also reported better mental health at exit. Table 4-27. Linear regression for mental health status at study exit (for all participants) Variable Treatment dummy Age Ever had active ticket Diagnosis Baseline mental health (SF-12) Population density Number of ER visits at baseline Estimate 2.87 -0.44 1.86 3.06 0.46 0.62 -0.58 p-value <0.001 0.209 0.011 <0.001 <0.001 <0.001 0.089 R-squared = 0.2665 Table 4-28 presents stepwise linear regression results for the mental health status at study exit for treatment group participants only. As expected, baseline mental health score was a highly significant predictor of mental health status at study exit. Receiving general medical care and engagement in SMM also had significant and positive effects on mental health at exit. Participants with an affective disorder and those who lived in areas with higher population density reported better mental health at exit. Chapter 4: Outcomes 4-47 Table 4-28. Linear regression for mental health status at study exit (for treatment group participants only) Variable Received general medical care SMM engagement category Diagnosis Baseline mental health (SF-12) Population density Estimate 2.50 1.51 2.25 0.49 0.57 p-value 0.047 0.020 0.008 <0.001 <0.001 R-squared = 0.2910 Quality of life. The Modified Lehman Quality of Life Inventory provided the single item that measured quality of life in the study. As noted in Chapter 2, individuals who obtain competitive employment over time report higher quality of life. In this study, interviewers asked participants the single item on general life satisfaction during the Baseline interview and then again during the final Followup interview. Table 4-29 presents the study results. General life satisfaction reported by participants in the treatment group improved significantly over that reported by participants in the control group (p-value < 0.001). Table 4-29. Satisfaction with life at baseline and study exit by treatment and control group Variable Life satisfaction Mean SD Treatment (N=1,004) Baseline Followup 3.78 1.55 4.22 1.51 Control (N=1,051) Baseline Followup d 0.44 3.82 1.53 4.01 1.58 d p-value 0.19 0.0002 Appendix 4G also presents tabled values of general life satisfaction by relevant subgroups. In summary, the results indicate that significant differences always favored the treatment group. The following subgroups show significant differences (p-value <.05) favoring the treatment group: Participants 35 years of age and older, Males and females, Participants diagnosed with an affective disorder, and Participants with a high school education and those with more than a high school education. No significant findings between the treatment and control groups were noted in any of the remaining subgroups, including participants under the age of 35 or participants with schizophrenia. Chapter 4: Outcomes 4-48 Predicting quality of life. This section presents an analysis of the MHTS effects on the quality of life scores at study exit. An ordered logistic regression technique was used to estimate the categorical outcome quality of life, where some outcomes are better (or valued higher) than others. Participants reported on their level of satisfaction with their life in general using Lehman’s terrible—delighted scale. The scale ranges from a value of 1 (terrible) to 7 (delighted). In order to provide an easier interpretation of the findings, the three lowest scores (terrible, unhappy, mostly unsatisfied) and the three highest scores (mostly satisfied, pleased, delighted) were aggregated. Results were similar to the regression estimates using the seven-point scale. Table 4-30 presents the results of ordered logistic regression estimates. The results suggest that the MHTS treatment had a significant positive impact on quality of life at study exit. MHTS participants were 6.4 percent less likely to report feeling “terrible, unhappy, or mostly unsatisfied” about their life in general, and they were 7.5 percent more likely to report feeling “mostly satisfied, pleased, or delighted.” In addition, baseline mental and physical health status, diagnosis of an affective disorder, and having an active ticket were all statistically significant with positive effects on the quality of life outcome measure. Local unemployment rate and age were significant as well, and had a negative relationship with the quality of life at study exit. As with previous models, being in the treatment group was associated with higher quality of life. Table 4-31 presents results for the ordered logit estimates for the quality of life score at study exit for treatment group participants only. Having better mental and physical health at baseline were both significant with positive impacts on the quality of life score. Age was significant with negative effects on the quality of life score. Engagement with SMM was highly significant with a positive effect on quality of life. Clients receiving social skills training or receiving case management were more likely to report a lower quality of life score. Chapter 4: Outcomes 4-49 Table 4-30. Ordered logistic regression estimates for quality of life (all participants) Dependent variable Treatment dummy Baseline mental health (SF-12) Baseline physical health (SF-12) Gender Worked in last 2 years (prebaseline) Diagnosis Age Less than HS graduate vs. more than HS graduate HS graduate vs. more than HS graduate Unemployment rate Months on rolls Number of ER visits at baseline Number of hospital stays at baseline Population density Not receiving SSI (SSDI only) Ever had active ticket Table 4-31. Terrible, unhappy, and mostly unsatisfied Estimate p-value -0.064 <0.001 -0.013 <0.001 -0.004 <0.001 0.027 0.145 -0.005 0.801 Mixed Estimate p-value -0.011 0.003 -0.002 <0.001 -0.001 0.001 0.004 0.153 -0.001 0.805 Mostly satisfied, pleased, and delighted Estimate p-value 0.075 <0.001 0.015 <0.001 0.004 <0.001 -0.031 0.144 0.006 0.801 -0.053 0.003 0.043 0.015 0.030 0.161 -0.008 0.001 0.004 0.029 0.018 0.021 0.061 -0.004 -0.047 0.015 0.007 0.139 -0.020 0.319 -0.004 0.374 0.024 0.327 0.020 0.001 -0.003 -0.006 0.001 0.597 0.803 0.644 0.003 0.001 0.001 -0.001 0.004 0.599 0.803 0.646 -0.023 0.001 0.003 0.007 0.001 0.597 0.803 0.644 0.005 -0.022 -0.051 0.331 0.364 0.047 0.001 -0.004 -0.008 0.340 0.437 0.065 -0.006 0.026 0.060 0.331 0.376 0.047 Ordered logistic regression estimates for quality of life (treatment group participants only) Dependent variable Treatment dummy Baseline mental health (SF-12) Baseline physical health (SF-12) Gender Worked in last 2 years (prebaseline) Diagnosis Age Less than HS graduate vs. more than HS graduate HS graduate vs. more than HS graduate Unemployment rate Chapter 4: Outcomes Terrible, unhappy, and mostly unsatisfied Estimate p-value Mixed Estimate p-value Mostly satisfied, pleased, and delighted Estimate p-value -0.013 -0.004 0.030 -0.002 0.001 <0.001 0.218 0.941 -0.004 -0.001 0.010 -0.001 0.001 0.001 0.217 0.941 0.017 0.006 -0.040 0.003 <0.001 <0.001 0.215 0.941 -0.023 0.003 0.036 0.420 0.030 0.373 -0.008 0.001 0.009 0.425 0.042 0.242 0.031 -0.005 -0.046 0.420 0.029 0.347 -0.015 0.597 -0.005 0.618 0.020 0.602 0.014 0.067 0.005 0.081 -0.019 0.067 4-50 Table 4-31. Ordered logistic regression estimates for quality of life (treatment group participants only) (continued) Dependent variable Months on rolls Number of ER visits at baseline Number of hospital stays at baseline Population density Not receiving SSI (SSDI only) Ever had active ticket SE engagement score SMM engagement category Received medication services Received general medical care Received substance abuse treatment Received housing services Received family counseling or legal assistance Received social skills training Received financial assistance training Received case management Average site fidelity score Number of clients served at site Treatment dummy Baseline mental health (SF-12) Terrible, unhappy, and mostly unsatisfied Estimate p-value 0.001 0.719 -0.016 0.297 0.008 0.638 Mixed Estimate p-value 0.001 0.719 -0.005 0.304 0.003 0.639 Mostly satisfied, pleased, and delighted Estimate p-value 0.001 0.719 0.021 0.297 -0.010 0.638 -0.004 0.057 -0.022 0.003 -0.071 -0.054 0.028 -0.004 0.557 0.116 0.499 0.177 0.001 0.190 0.477 0.906 -0.001 0.013 -0.008 0.001 -0.024 -0.017 0.009 -0.002 0.559 0.027 0.548 0.192 0.003 0.185 0.476 0.908 0.006 -0.070 0.030 -0.004 0.095 0.072 -0.038 0.006 0.557 0.090 0.512 0.178 0.001 0.186 0.476 0.906 -0.043 -0.040 0.235 0.274 -0.018 -0.016 0.330 0.366 0.061 0.056 0.263 0.301 0.115 -0.019 0.008 0.604 0.018 -0.007 0.002 0.639 -0.133 0.025 0.002 0.614 0.074 -0.001 -0.002 0.052 0.795 0.428 0.020 0.001 -0.001 0.028 0.795 0.433 -0.094 0.002 0.003 0.042 0.795 0.428 -0.013 0.001 -0.004 0.001 0.017 <0.001 Alcohol or drug use. Table 4-32 shows the change in the percentage of study participants meeting the Addiction Severity Index (ASI) cutoff score for alcohol use at baseline and exit. Among the treatment group participants, while 9.1 percent reported an alcohol use problem at baseline, this figure dropped to 6.2 percent at exit. Similarly, while 8.3 percent of the control group participants reported an alcohol use problem at baseline, this figure dropped to 5.2 percent at exit. Both groups had almost a one-third reduction in alcohol use at the time they exited the study. However, the difference between treatment and control groups was not statistically significant (p-value = 0.967). Table 4-32 also shows the change in the percentage of study participants meeting the ASI cutoff score for drug use at baseline and exit. Although the percent of participants reporting drug problems Chapter 4: Outcomes 4-51 increased among the treatment group, there is statistically no significant difference between the two groups in terms of change in drug use problems (p-value = 0.238). Table 4-32. Alcohol and drug use at baseline and study exit by treatment and control group Variable Alcohol use score > 0.17 n % Drug use score > 0.16 n % Treatment (N=1,004) Baseline Exit 91 9.1 62 6.2 39 3.9 53 5.3 Control (N=1,051) Baseline Exit d d p-value -2.9 86 8.3 54 5.2 -3.1 0.967 1.4 47 4.5 47 4.4 -0.1 0.238 NOTE: Weighted percents may not be consistent with unweighted counts. Table 4-33 shows that treatment and control group participants reporting any drug use during the last 30 days went down from 52.0 percent and 51.1 percent to 37.8 percent and 38.4 percent, respectively. While the change from baseline to exit is significant for both groups, there is no statistically significant difference between treatment and control (p-value = 0.5340). Table 4-33. Any drug use during the past 30 days at baseline and study exit by treatment and control group Variable Any drug use past 30 days n % Treatment (N=1,004) Baseline Exit 522 51.95 379 37.75 d 14.20 Control (N=1,051) Baseline Exit 536 51.09 405 38.39 d 12.70 p-value 0.5340 NOTE: Weighted percents may not be consistent with unweighted counts. The MHTS was designed to provide employment supports, SMM, and insurance supports to improve participants‘ employment and mental health outcomes. Most MHTS study sites were not equipped to provide substance use treatment. Thus, consistent with our earlier expectations, change in alcohol or drug use does not indicate statistically significant differences between the treatment and control groups. Chapter 4: Outcomes 4-52 Discussion The MHTS was designed to test the fundamental value of providing evidence-based mental health treatment and employment supports to SSDI beneficiaries to improve their employment, health, and functioning. However, the primary emphasis was on employment. The basic components of the treatment included IPS SE services, SMM, and other behavioral health services—all of which were paid for with a comprehensive health insurance package. Employment The data presented in this chapter clearly specify that the employment rates of participants who received the treatment intervention were significantly and meaningfully improved over participants in the control group. When viewed in the context of all participants in the intent-to-treat analysis, the treatment group participants were more likely to find employment, worked more months, worked more months at study exit, and worked more hours per week than did control group participants. These findings were consistent across different demographic, clinical, Social Security program title (SSDI and SSI), and, with few exceptions, were consistent across sites as well. These findings only partially extended to the subgroups of participants who worked any job or at least one competitive job. Overall employment and competitive employment were generally consistent with previous research on IPS SE showing that SSDI beneficiaries achieve much better than in other vocational interventions or in no intervention at all via a treatment-as-usual design. The fact that participants in the treatment group were employed for more months than were participants in the control group accords with previous IPS research (Bond, Drake, & Becker, 2008). The parsimonious interpretation is that IPS helps people to find jobs and to keep them, and this is a reasonable interpretation of the findings from the MHTS. The findings were, however, surprising in one aspect, namely that the control group participants did remarkably well. Despite the fact that participants in the treatment group did significantly better than those in the control group, the overall paid and competitive employment rates for the control group participants were much higher than those of SSDI beneficiaries in previous IPS studies (Bond, Xie, & Drake, 2007). The SSDI beneficiaries in the MHTS were different from those in previous IPS studies in that most were not attending community mental health centers prior to the study and self- Chapter 4: Outcomes 4-53 selected for participation based on letters from SSA and introduction meetings that clarified that the study was for beneficiaries who wanted to return to work. Therefore, it is reasonable to assume that they were highly motivated to work. Comparison of MHTS employment outcomes with the IPS literature. While the overall findings unambiguously show a strong statistical advantage to the treatment condition over the control condition on employment outcome measures, one might also ask more specifically how strong these findings are, and how they compare to the general literature on the effectiveness of IPS. One comparison would be to the aggregated findings from 11 randomized controlled trials (RCTs) of IPS reviewed by Bond et al. (2008). Several important caveats are in order for comparisons between the MHTS and this earlier IPS literature: the samples differ in important ways, as do the interventions themselves. Regarding the samples, most of the RCTs of IPS reviewed by Bond et al. (2008) enrolled participants who were already clients in the public mental health system at the time of enrollment and only a minority (approximately 40%) were SSDI beneficiaries (Bond, Xie, & Drake, 2007). Regarding the interventions, the participants in the 11 RCTs generally received mental health case management and a range of other behavioral services that were unevenly offered to MHTS treatment study participants. On the other hand, participants in these earlier studies did not receive the full package of benefits the MHTS treatment participants did, including financial assistance for out-of-pocket medical expenses, SMM, assistance from NCCs, and suspended CDRs. Regarding differences in outcomes, one difference between the current study and the IPS literature is the proportion of treatment participants in the current study who obtained noncompetitive paid jobs (7.9%) as their highest level of paid employment. In most of the prior IPS studies the rate is probably lower. For example, in one prior IPS study, 4 percent of treatment participants obtained casual labor and 1 percent were placed in a sheltered workshop as their highest level of paid employment (Bond, Salyers, et al., 2007). In terms of overall employment rates for the treatment group in the current study—61 percent in paid employment and 53 percent in competitive employment—these rates fall with the midrange for the Bond et al. (2007) review, which reported a mean competitive employment rate of 61 percent across 11 studies. In the current study, the average time to the first job for treatment participants who obtained work was 7.7 months, compared to 4.6 months for seven studies in the Bond et al. (2007) review, suggesting a much longer delay to the first job in the current study. Unlike the findings in the Bond et al. (2007) review article, which found IPS participants achieving their first Chapter 4: Outcomes 4-54 job significantly sooner than controls, time to first job did not differ between treatment and controls in the current study. It is reasonable to speculate that the added demands of testing required of treatment participants (e.g., General Medical Exam, cognitive testing) may have contributed to this delay, although it is also possible that the engagement process was more extended in the current study, given that most beneficiaries were new to the study site. In the current study, the total sample of treatment participants averaged 6.2 months of employment over the 2-year period, which exceeds the average of 5.6 months (24.2 weeks) of competitive employment for seven studies in the Bond et al. (2007) review. Similar comparisons were made for participants who obtained work. In the current study, treatment participants who obtained work averaged 10.3 months of employment over the 2-year period, which also exceeds the average of 8.9 months (38.4 weeks) of competitive employment for seven studies in the Bond et al. (2007) review. Finally, treatment participants had greater job satisfaction than controls for their main job (assessed at followup), a difference found in some, but not all, of the previous IPS studies examining job satisfaction. Another key difference favoring the current study over prior IPS studies was the significantly better non-vocational outcomes for treatment participants, as discussed below. Controlled studies consistently fail to show significant improvement in non-vocational domains for IPS clients, compared to clients receiving traditional vocational services (Burns et al., 2009; Drake et al., 1999; Drake, McHugo, Becker, Anthony, & Clark, 1996; Gold et al., 2006; Latimer et al., 2006; Lehman et al., 2002; Mueser et al., 2004; Twamley, Narvaez, Becker, Bartels, & Jeste, 2008). One major difference between the current study and these earlier IPS studies is that none of these earlier studies included SMM or other behavioral health services that were provided solely to the participants in the treatment intervention. Since the difference between the MHTS and other IPS studies was the provision of the non-IPS supports in the MHTS, it seems reasonable that these other supports were primarily responsible for the improvement in non-vocational outcomes. To summarize the comparisons between the outcomes in the current study with the IPS literature, the treatment condition generally attained employment outcomes comparable to those reported in the literature for IPS participants. Time to first job was the one main exception. The comparisons to the IPS literature raise the question about the consequences if the unique features of the MHTS treatment model (e.g., financial assistance for out-of-pocket medical expenses, SMM, assistance from NCCs, and suspended CDRs) had not been provided. The short answer is Chapter 4: Outcomes 4-55 that the design of the MHTS does not permit a statistical answer to the dismantling question, because these features were offered as a package and therefore were completely confounded with IPS. However, the literature suggests that when clients do not receive vocational assistance (i.e., IPS), they do not increase employment outcomes, regardless of the excellence of mental health treatment, so this implies that vocational assistance is a necessary critical ingredient (Bond, 1998). However, it is also true that vocational assistance is much less effective when it is not well integrated with mental health treatment (Cook et al., 2005). Site differences in employment. Employment findings were consistent across most of the sites in the MHTS. Altogether, 21 of 23 showed better employment outcomes for the treatment condition than control. Of the two sites that did not show a clear advantage for the treatment group, one had an unusually high rate of competitive employment in the control group (52%), probably due to extreme treatment drift, and the other had an unusually low rate of competitive employment in the treatment group (29%), almost certainly due to the site leader‘s unwillingness to follow the model, due to disagreement with the model. MHTS investigators considered excluding the latter site from the study at several points, but retained it as an example of extreme model changes. Program planners often believe that they can change the IPS model to do it better (Menear et al., 2011). This case example illustrates the most common outcome when a program leader makes fundamental changes in the IPS model; the outcomes suffer. Predicting employment outcomes from participant characteristics. The predictors of employment among treatment group participants are entirely congruent with the literature on SE. Younger clients with serious mental illness are considered to have better vocational potential, particularly those in the early phases of illness who have not yet been socialized into disability (Rinaldi et al., 2010). Poor physical health status as a predictor has not received extensive attention in studies. The SF-12 measure assesses perceived impairment due to physical illnesses rather than objective measures of physical illness, which was examined in the ―reasons for not working‖ section. This finding regarding inability to obtain employment adds to growing concerns about the physical health status of people with serious mental illness. Recent earnings and recent work history may be proxies for capacity, readiness, motivation, or a complex mixture of these and other constructs. As expected, recent work history was a strong predictor of obtaining employment and of amount of work. The amount of work among participants who obtained at least one job is related to work history, which is a predictor in many studies (Bond & Drake, 2008; Campbell et al., 2010). Work history undoubtedly captures explanatory value by encompassing capacity, motivation, and other constructs. Chapter 4: Outcomes 4-56 More important for policy purposes, engaging in IPS SE services predicted obtaining employment. The study findings clearly support the view that SSDI beneficiaries benefit from high-quality employment services in pursuit of the goal of employment. Engagement in SMM was a significant predictor of the number of months of employment in the treatment group in the regression analysis. This finding can be interpreted to mean that participants who were judged as less engaged with their prescriber may have had less contact with the NCC— and by extension, with other treatment services at the study site—and achieved fewer months in employment. It is unclear whether better medication management per se accounted for this association, because otherwise one would have expected a stronger association between engagement in SMM and change in symptoms. The correlation between engagement in SMM and improvement in self-reported psychiatric symptoms was weak (r =.08). An alternative explanation is that engagement in SMM is a proxy for receipt of integrated services at the study site. If this speculation is correct, these findings would be consistent with research suggesting the critical role of integrated services on employment outcomes (Cook et al., 2005; Drake, Becker, Bond, & Mueser, 2003). Additional analyses of the existing data will likely reveal a clearer understanding of these relationships. Types of occupations. The large preponderance jobs held by study participants (just under 75% of all jobs held at any time) were in two sectors: Service and Sales/Office. There are only modest proportional differences in the occupations held by treatment group participants compared to controls. Although these differences were not tested statistically, the clinical importance of any differences appears negligible; therefore, it is reasonable to conclude that the kinds of jobs obtained by treatment group participants were similar in proportion to those obtained by participants in the control group. The data also reveal that both diagnosis and education affect the types of jobs participants obtain, but there are no obvious differences in attainment of occupational type between the treatment and control groups within diagnosis or within educational level. In terms of the types of jobs obtained by MHTS study participants, the distribution of occupational types mirrors that commonly obtained in studies of SE clients with severe mental illness, as reported in the literature spanning the last two decades, including recent studies (Becker, Bebout, & Drake, 1998; Becker, Drake, Farabaugh, & Bond, 1996; Bond, Drake, Mueser, & Becker, 1997; Bond & Kukla, 2011; Fabian, 1989; Gervey & Kowal, 1994; Heslin et al., 2011; MacDonald-Wilson, Revell, Nguyen, & Peterson, 1991; Mowbray, McCrohan, & Bybee, 1995; Mueser, Becker, & Wolfe, 2001; Resnick & Bond, 2001; Shafer & Huang, 1995). Many of the jobs obtained by MHTS participants Chapter 4: Outcomes 4-57 were entry level and unskilled, reflecting the work experience and skill levels of many participants in the study sample. While the distribution of jobs might suggest that participants obtained unappealing jobs, MHTS investigators interpret the findings differently. From the perspective of the IPS model, the critical question is not occupational type, but job match, as repeatedly documented in the literature (Becker et al., 1998; Becker et al., 1996; Carpenter & Perkins, 1997; Gervey & Kowal, 1994; Huff, Rapp, & Campbell, 2008). SE clients who obtain jobs that suit their preferences (which includes type of occupation, work hours, supervision, location, and many other factors) have greater job satisfaction and job tenure than those who are poorly matched. Another factor in interpreting occupational types is the unavoidable reductionism of any classification system. Within a particular occupational category are both high quality and low quality jobs. Barriers to employment for participants who achieved little or no success. Near the end of the data collection period, the NCC, employment specialist, and other team members at each site identified reasons that they believed were barriers to employment for treatment participants who had little or no success in obtaining work. This analysis helps us to understand the reasons that some participants did not work during MHTS. The most prominent barriers clearly were uncontrolled mental illness, uncontrolled physical health problems, and failure to engage in IPS SE. The factors with extremely low ratings were also of interest because many were discrepant from the literature on barriers to employment. Less than 10 percent of the participants were judged to have barriers related to cognitive problems, lack of services, transportation difficulties, lack of work skills, lack of social skills, criminal justice system problems, and housing problems. The literature clearly notes each of these problems as a barrier to work. The lack of prominence in MHTS could be due to the special characteristics of SSDI beneficiaries (e.g., substantial work histories and other factors discussed earlier), the services they received through MHTS, the teams‘ failure to recognize some problems (e.g., cognitive difficulties may be overlooked), or other factors. It should also be acknowledged that these ratings were based on the perspective of the treatment team, which prior research indicates differs from client perspectives (Crane-Ross, Roth, & Lauber, 2000), as well as other data sources. In addition to these three major barriers, family problems, poorly controlled substance use disorders, disengagement from SE services, and behavior problems affected greater than 10 percent of the group. One interpretation of these barriers is as follows: families often rely on the beneficiaries‘ income for support or on the beneficiaries‘ willingness to work in casual labor for the family. Both issues can lead families to resist the beneficiaries‘ interests in employment because they threaten the status quo. Some families also recall previous experiences when their relative had a negative work Chapter 4: Outcomes 4-58 experience or lost disability benefits due to working temporarily. Co-occurring substance use disorder is very common among people with serious mental illness and may develop or become exacerbated due to joblessness. Some clients clearly lose jobs if they continue to use substances. Clients withdraw from services for a variety of reasons. They may decide not to work when they understand the long-term threat to benefits, when they try to find a job or actually work and find it more stressful than they imagined, or when they have a relapse of any of their illnesses. Behavior problems often indicate co-occurring personality disorders that interfere with working. Earnings and Income Treatment group participants earned more on average than did control group participants (past month‘s earnings 1.5 to 1, past three months‘ earnings at study exit 2 to 1). However, these differences are attributed mostly to the higher employment rate among MHTS treatment group participants. This differential effect on earnings of participation in treatment was expected as part of the general hypothesis that the treatment group would achieve better employment outcomes than controls. The findings are consistent with the overall model of the IPS intervention, which addressed increased employment and improving earnings. The study also examined how often study participants had monthly earnings exceeding SGA. As an exploratory hypothesis, analyses were conducted to examine whether treatment group participants would have a higher frequency of months above SGA than control group participants. However, there were very few quarters in which either treatment group participants or control participants exceeded SGA in the last month of the quarter. In fact, only 6 percent of treatment group participants and only 3 percent of control group participants exceeded SGA in more than one quarter (of 8 quarters examined). Thus, it is reasonable to conclude that while the employment rate for treatment group participants was significantly higher than that for control group participants, the extent of employment as measured against SGA was not significantly different between the two groups. A further exploratory hypothesis was that participants in both groups would ―park‖ just below SGA, that is, work just under the limit, attaining monthly earnings between 75 percent and 99 percent of SGA. The results of this analysis were very similar to the preceding analysis: very few of either group earned between 75 percent and 99 percent of SGA in any month during the study. Only 6 percent of treatment group participants and only 2 percent of control group participants earned between 75 Chapter 4: Outcomes 4-59 percent and 99 percent of SGA in more than one quarter. Parking behavior occurred at a very low rate, if at all, in this study. In summary, economic disincentives (that is, earnings limits) do not appear to have been a potent factor in how many hours per week study participants worked. MHTS investigators infer that beneficiary decisions about intensity of employment (i.e., how many hours to work per week) were dictated by other factors, which might include stamina, lifestyle, and presence of co-occurring physical conditions. Beneficiary attitudes towards SSA benefits were also measured using a series of items included in the Baseline and final Followup interview. The findings suggested the treatment group had a modest reduction in fears about losing benefits as a result of employment. The control group did not show much attitude change. The attitude change in the treatment group was probably attributable to the higher rates of employment and to benefits counseling received as an element of the intervention offered to the treatment group. At the end of the MHTS, treatment group members on average had nearly twice the earnings of members of the control group and more than twice as many were likely to have had earnings in the prior three months. Monthly income was also higher for the treatment group and for their families. Income increased faster for the treatment group, reflecting the philosophy and emphasis of the IPS model of SE on rapid job placement—place and train rather than train and place. While income and earnings differed significantly between the two groups, mean SSDI benefits remained the same for both groups, as did benefits from most other social welfare programs. The only treatment group participants to have an observed significant difference in Social Security benefits were in the subgroup of participants on SSI. Males and participants with schizophrenia were observed to have lower income from SSI. Health and Functioning The assessments of health and functioning outcomes reveal two prominent findings from the study: participants in the treatment group exited the study with statistically significant improved mental health and general satisfaction with life compared to the control group participants. While the commensurate effect sizes suggest these improvements were modest at best, the patterns of improvement and the significance extended throughout the entire study sample, with mean differences always benefitting the treatment group and nearly all subgroups showing a significant Chapter 4: Outcomes 4-60 difference from the control group. Further analyses of the existing data are needed to assess whether or not these improvements can be better understood. The self-reported health status of participants in the study (both treatment and control) is consistent with the literature. While little normative information about the physical or mental health status (using the SF-12) of individuals with schizophrenia exists, there are norms for individuals with depression. Those norms, reported by Ware, Kosinski, Turner-Bowker, and Gandek (2002), suggest that both the physical and mental health of participants in the study with an affective disorder approximate the health status of the population with depression. This fact further substantiates the study findings associated with improved mental health. Earlier studies found that SE has minimal or no systematic impact on non-vocational outcomes (Drake, McHugo, et al., 1996; Becker, Bond, et al., 2001; Bond, 2004). However, the intervention design of the MHTS was substantially different from all of the previous studies examining the relationship between non-vocational outcomes and SE. The MHTS intervention combined SE services with SMM and coverage for other behavioral health services. This study finds that the mental health and general satisfaction with life of the treatment group improved significantly over the course of the study and over the comparison group. Alcohol and other substance abuse measures did not change differentially for treatment and control group participants. It is important to note that substance abuse-related services were not a primary focus of the intervention. Change in self-reported general health status was also not significantly different between the two groups, which is also to be expected, since general medical services were not a differential element for the treatment and control groups. For the most part, both groups had Medicare coverage during the period of the MHTS and had comparable access to general medical services. Treatment participants did have access to counseling regarding which Medicare benefits to obtain and received reimbursement for Parts B and D, but general medical services were not an explicit element of the treatment group services. Chapter 4: Outcomes 4-61 Chapter 5 Implementation of Supported Employment and Other Behavioral Health and Related Services The primary treatment intervention components of the Mental Health Treatment Study (MHTS) offered access to systematic medication management (SMM), Individual Placement and Support (IPS) supported employment (SE), other behavioral health (OBH) and related services, and Nurse Care Coordinator (NCC) services. This chapter describes implementation of the IPS SE, and OBH and related services component of the intervention. The description includes an assessment of several different aspects of implementation. Measurement at the site level assessed the extent to which the study sites implemented SE services as intended. That is, did the study sites develop and provide the kind of SE services that are consistent with the IPS model of demonstrated effectiveness? The relevant term for this level is “program-level fidelity.” Measurement at the individual level assessed the extent to which treatment group participants received the array of SE assistance and OBH and related services expected in a program that is faithful to the IPS model. The relevant term for this level is “beneficiary-level fidelity.” A third aspect of implementation assessed in the study, also at the beneficiary level, was the extent of active beneficiary involvement in the intervention offered by the study site. The relevant term for this is “engagement.” The primary focus of this chapter is on the fidelity of the services implemented in the MHTS to the planned IPS service model. Even though most of the sites were chosen because they were already implementing or seeking to implement IPS, a wealth of field experience suggests that, in routine practice, mental health programs often drift from close adherence to a model, even when program leaders seek to implement the model as intended (Becker et al., 2006). In addition, agency leaders commonly assume they are implementing an evidence-based practice when their actual practice departs from the defined model (Bond et al., 2011). Thus, a primary objective of this chapter is to convey how closely the 23 study sites, individually and as a group, adhered to the principles of the IPS model of SE and offered treatment group participants the wide range of OBH and related services needed to improve their employment, health, and quality of life. Secondary objectives include an understanding of the rates of receipt of services, consistency in delivery of the services, where services occurred, and the relationship between fidelity in the MHTS and employment rates. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-1 Background Fidelity refers to adherence to the principles of an evidence-based program model. A fidelity scale is an instrument used to assess adherence to the specific model (Bond, Evans, Salyers, Williams, & Kim, 2000). In the MHTS, IPS fidelity referred to the degree of adherence to the evidence-based IPS model of SE within and across the multisite study. IPS was a primary focus of the intervention, thereby bringing fidelity to the forefront as an important consideration in the overall assessment of the study implementation. It is possible to have study sites that meet the standards of high programlevel fidelity, but fail to provide all of the required SE or behavioral health and related services. It is also possible to have high program-level fidelity without all treatment group participants fully engaged in the service components of the intervention. High program-level fidelity sites may not obtain optimal outcomes if they do not engage all or most treatment group participants fully or if many participants do not receive the full array of services. For these reasons, it is important to assess program-level fidelity, individual-level fidelity, and engagement to determine the extent of implementation of SE and OBH and related services. There are four main reasons for assessing fidelity in a multisite randomized controlled trial. The first is to facilitate communication to readers of reports about the nature and degree of implementation of an intervention under investigation. The importance of providing a basic description of adherence to model implementation is evident from a review of the history of model dissemination. Within the community mental health field, the need for systematic methods to assess fidelity became clear in the 1970s and 1980s (Drake, Essock, & Bond, 2009). During this period, many demonstration projects produced disappointing results, in part due to poor model specification (Brekke, 1988). The absence of clear, objective program standards—criteria for implementing a practice—interfered with many well-intentioned efforts to disseminate these practices and severely attenuated the accumulation of scientific evidence in support of these program models. In the MHTS, the target audiences include the Social Security Administration, state and federal policymakers, advocacy groups, the research community, Social Security beneficiaries, and the public at large. The report of findings will better specify the intervention provided and will make it possible to replicate with use of a reliable and valid fidelity scale in a multisite randomized controlled trial. A second reason for assessing fidelity is to document variation across sites in a multisite study. The MHTS was the largest demonstration ever of the IPS model, conducted in 23 study sites. Understanding the level of variation in the context of the study outcomes provides policy guidance for future implementations. The MHTS will provide information on the generalizability of the Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-2 overall intervention model across sites. One question is how feasible is it to offer the IPS model with high fidelity? The third reason to assess fidelity is to improve implementation. If collected periodically throughout an intervention, fidelity assessments can provide feedback regarding changes needed to ensure full implementation of the evidence-based practice. In other words, how feasible is it to modify programs when they are not operating at high fidelity (Bond, Drake, McHugo, Rapp, & Whitley, 2009) and to sustain those practices over time (Rapp, Goscha, & Carlson, 2010)? If it is possible to make program improvements through fidelity reviews, then this will greatly enhance the quality of the research and future adaptation. A fourth use of a fidelity measure is to predict outcomes. The general issue is measurement of the extent to which adherence to the program model is associated with better outcomes. Program-level fidelity has been found in prior research to be a strong and consistent predictor of client outcomes (Bond, Becker, & Drake, 2011). Receipt of IPS and OBH and related services is important for similar reasons as outlined for fidelity. To facilitate communication, it is essential to describe services provided. Documenting variation across sites offers guidance in assessing the degree of generalizability. Additionally, receipt of services is also hypothesized to be associated positively with client outcomes (Bond & Kukla, 2011; Ryan, Sherman, & Bogart, 1997). MHTS investigators hypothesized that receiving integrated services (assistance for different needs provided by a single treatment team) rather than fragmented services (e.g., assistance provided by different service agencies or multiple sources) would be more effective, and as such, positively influence the study outcomes. Therefore, it was important to not only examine whether services were provided, but also to examine how services were provided (Corrigan, Mueser, Bond, Drake, & Solomon, 2008). This general finding is well-established for the integration of behavioral health and employment services (Cook et al., 2005; Drake, Becker, Bond, & Mueser, 2003) and for psychiatric and substance abuse treatment (Drake & Bond, 2010). In the MHTS, the location of service (on-site vs. off-site) served as one proxy for the degree of integration. However, we know that providing all services at a single site does not guarantee integration of services, and investigators found that some service systems were able to integrate across settings. The IPS Fidelity Scale also contains a global measure of integration of employment services with mental health treatment based on interviews with staff, observation of meetings, and medical charts. While this item is not an ideal measure of integration, it is the best available measure of integration obtained in the MHTS. A more ideal Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-3 measure of integration would be a psychometrically-validated, comprehensive, multi-item scale using multiple data sources. Such a scale has not yet been developed. The remaining sections in this chapter describe the methods for assessing implementation, data collection procedures, and the results. Methods Overview of Treatment Intervention The treatment intervention had four service components: (1) IPS SE, (2) OBH and related services, (3) SMM, and (4) treatment coordination by the NCC. The topics covered in this chapter include an examination of implementation of IPS SE, and OBH and related services giving emphasis to the quality of service delivery, timeliness of service delivery, service delivery rates, and service integration. Additional topics include a description of the site variation in IPS implementation and in the provision of the various elements of other behavioral services. The chapter concludes with an examination of the relationship between IPS fidelity and site-level employment rates. A review of the implementation of the remaining intervention components, SMM and the NCC role, follows in Chapter 6. Treatment group participants also received financial support for insurance premiums and out-of-pocket costs. Chapter 7 presents a detailed review of the financial support provided to treatment group participants. SE. Chapter 1 of this report describes the literature on the IPS model of SE. In sum, IPS is a welldefined form of SE and is an evidence-based practice specifically designed to serve individuals with severe mental illness (Becker & Drake, 2003). OBH and related services. The service needs of people with severe mental illness have been identified in the literature (Corrigan et al., 2008; Drake et al., 2001; Kreyenbuhl, Buchanan, Dickerson, & Dixon, 2010). In addition to IPS and SMM, the investigators identified the following eight service categories, which they hypothesized to be essential services to which SSDI beneficiaries should have access: General medical care; Mental health case management; Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-4 Social skills training; Financial assistance; Housing assistance; Substance abuse treatment; Family counseling; and Legal assistance. Throughout the study, MHTS investigators tracked the extent to which treatment group participants received these services. Unfortunately, the investigators were not able to assess the intensity or quality of implementation. MHTS implementation of the IPS model. To help ensure fidelity to the IPS model, the investigators used several strategies. First, as described in the section titled Study Site Selection in Chapter 2, investigators purposively selected sites based on their historical commitment to implementing high-fidelity IPS services. Specifically, a majority of the study sites were part of a learning collaborative, which had been developed as a component of the Johnson & Johnson-Dartmouth Community Mental Health Program (Becker et al., 2011; Drake, Becker, Goldman, & Martinez, 2006). Second, three consultants were employed to serve as Quality Management Project Directors (QMPDs). The QMPDs were experienced clinicians and program leaders who were experts in IPS. The QMPD role was that of a trainer-consultant to the sites regarding implementation of the psychosocial components of the intervention, with special attention to IPS implementation. The three QMPDs shared the work of monitoring the 23 study sites in the MHTS. Each monitored approximately one-third of the sites for the entire study period. The QMPD role was modeled on the trainer role in a national multisite implementation study (McHugo et al., 2007). The QMPDs made weekly telephone contact with the NCC and periodically spoke with the program director at the site and IPS team leader. The primary focus of these calls was to review the progress of each participant and help the NCC develop good IPS and clinical intervention plans. Consultations also included discussions about the research protocol (e.g., recordkeeping and form completion); assessments and overall workflow to implement the project; and how the NCC role related to the clinical team, IPS specialists, and implementation of the IPS model to ensure fidelity. Another focus of consultation was on establishing an alliance between clinicians and researchers. QMPDs fielded questions about access to services, service needs of the study participants, care coordination, and emergency treatment interventions. Also addressed were questions regarding the Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-5 authority of the NCC within the organizational structure of the study site. QMPDs noted that for nurses, functioning in the capacity of a coordinator was typically not a funded position within the community mental health center organizational structure. Many study sites hired the NCCs specifically for conducting the unique job function and role specific to the MHTS. As a result, most of the NCCs had limited experience working on a mental health team that provided IPS or employment services. The QMPDs made sure to address employment-related updates on all calls (i.e., any new jobs obtained, follow-along supports provided, barriers to gaining employment, and reasons for job terminations). The QMPDs also made annual site visits and conducted fidelity assessments of the IPS program, using the IPS Fidelity Scale (Bond et al., 2011; Bond, Becker, Drake, & Vogler, 1997) to provide site directors recommendations for improving IPS implementation (Bond et al., 2009) at their respective site. Thus, the expectation was that all 23 MHTS study sites would achieve high fidelity. Data Collection Procedures for Assessing MHTS Implementation Data collection used to assess the implementation of the SE services and OBH and related services of the treatment intervention required multiple strategies. First, to assess IPS fidelity at the programlevel, QMPDs conducted daylong annual site visits and systematically coded their observations using the IPS Fidelity Scale (Appendix 5A). Second, to assess beneficiary-level receipt of services, NCCs completed electronic service utilization reviews using the SE/OBH Quality Management Template. Third, to obtain a global view of site implementation from the perspective of the QMPD, the QMPDs completed a QMPD Site Report on two occasions, once in 2008 and once in 2009. Discussions of these strategies appear in detail throughout the remainder of this chapter. Measurement of IPS Fidelity at Program-Level The quality of IPS implementation was assessed at the program-level using a 15-item measure known as the IPS Fidelity Scale (Bond et al., in press; Bond et al., 1997) (Appendix 5A). The format and assessment procedures for the IPS Fidelity Scale follow the conventions formalized in the National Implementing Evidence-Based Practice Project (McHugo et al., 2007). Each scale item reflects a specific element in the practice. A 5-point behaviorally anchored scale provides the rating range. A rating of “5” indicates close adherence to the model, while a rating of “1” represents a substantial lack of model adherence. For example, a score of five (5) for rapid job search indicates that Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-6 a first contact with an employer is, on average, within one month after program entry, whereas a score of one (1) represents a delay of up to one year after program entry. Ratings of four (4), three (3), and two (2) represent gradations between these two extremes. For quality improvement purposes, the QMPDs provided feedback to sites about their relative attainment of core elements in the IPS model using the item-level fidelity ratings. In addition, the average of the item ratings yielded a total fidelity score, which expressed a global picture of overall program-level fidelity. Thus, the total fidelity score ranged from one (1) to five (5), with higher scores indicating more faithful implementation. Bond et al. (1997) established the following benchmarks: a cutoff score of 4.33 (total score > 65) represents high fidelity, while 3.66 (total score > 55) (but less than 4.33) represents fair fidelity. Although these benchmarks were originally based solely on clinical judgment, these conventions have served the vocational rehabilitation field well by communicating objectively the level of attainment of fidelity. IPS fidelity ratings are used to define higher reimbursement rates for employment services in three state systems (Bond et al., 2011). The utility of the IPS fidelity standards has been examined in 10 studies conducted by different research groups from around the world, though refinement of the IPS fidelity standards continues (Bond et al., 2011). IPS Fidelity Scale assessment procedures. Typically, two trained fidelity assessors assessed the IPS Fidelity Scale over the course of a one- to two-day site visit. One assessor was the QMPD assigned to the site and the other assessor was a member of the MHTS investigative team. Because the MHTS IPS services in many sites were a subcomponent of a larger IPS team, and IPS services directed to MHTS treatment participants sometimes varied from the usual IPS services, the assessors rated IPS services as delivered to MHTS treatment group participants. In normal IPS fidelity assessments, assessors follow a detailed protocol with instructions for preparing sites for the visit, critical elements in the fidelity assessment, and sample interview questions (Becker, Swanson, Bond, & Merrens, 2008). For the MHTS, the assessors typically interviewed the employment program leader and two or more employment specialists; observed clinical team meetings where the NCC and the IPS specialist participated; shadowed employment specialists on community contacts with employers; interviewed treatment group participants; and reviewed beneficiary charts. When two assessors participated in an IPS review, each assessor independently made fidelity ratings. The two assessors then reconciled any discrepancies to arrive at a final set of fidelity ratings. For quality improvement purposes, the QMPDs prepared a fidelity report summarizing the fidelity ratings and provided recommendations concerning any components of the program that were deficient. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-7 Psychometric properties of the IPS Fidelity Scale. The IPS Fidelity Scale has excellent interrater reliability (range of .67 to .99 for individual items and .98 for the total scale) and adequate internal consistency (Cronbach’s alpha = .92). Nine of 10 studies assessing its predictive validity found positive associations with employment outcomes. Its use in quality improvement has been supported by positive reports from seven multisite projects (Bond et al., 2011). In addition to measuring total fidelity, the scale also examines item-level fidelity. One item of particular interest was integration of rehabilitation with mental health treatment, which often is the most difficult component of IPS fidelity to achieve (Campbell et al., 2007). SE/OBH Quality Management (QM) Templates The Supported Employment/Other Behavioral Health (SE/OBH) QM template (form) was designed for the MHTS and implemented at startup. Prior to project startup, MHTS investigators developed a template that the NCC would complete on each treatment group participant. The intent was to describe and track services received during the course of the study. For each participant, the NCC completed a summary report covering the following time intervals: 0-3 months, 3-6 months, 612 months, 12-18 months, and 18-24 months. As part of the report, the template provided the format for assessing elements in the IPS model as well as receipt of OBH and related services. Systems staff programmed the template into the electronic Study Management System (SMS), including automated reminders that triggered the NCC to complete the form at regularly scheduled intervals. The NCCs received training in the completion of the template at the project kickoff in June 2006. Subsequently, the QMPDs provided consultation to the NCCs regarding questions that surfaced about the template completion. Based on the investigators’ review of the early template data, concerns voiced by the NCCs about the template’s redundancy, and the QMPDs assessment of informational shortcomings of the tool, it was determined that the template was not adequately eliciting quantifiable and reliable information. The original template design failed to capture some of the critical data elements needed by the investigators (for example, whether services were being delivered on-site or off-site). Consequently, the investigators met in November 2007 to tailor and simplify the procedures for completion of the template and to capture only the most salient items. The Results section of this chapter report the Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-8 findings for the common items on the original and revised templates, as well as the new items developed for the revised template. The IPS data elements assessed in the revised template included the following: Employment status (was participant employed during period, and if yes, start and end dates); Engagement in MHTS services; Vocational profile completed or updated; Vocational plan completed; Benefits counseling received; Number of employer contacts; Number of job interviews; Number of calendar months in which there was a contact with the employment specialist; and Number of calendar months in which the employment specialist made contact with the employer. OBH and related service elements assessed on the template included whether a service was needed, whether the participant received it, and (after the template revision described above) whether it was received on-site or off-site. Overall, NCCs completed 95 percent of the SE/OBH QM templates with site completion rates ranging from 75 percent to 100 percent. Issues affecting analysis of SE/OBH QM template. The SE/OBH QM templates incorporated the participant engagement question well after the study was underway. Consequently, a sufficient sample to report on engagement for the first two reporting periods (0-3 months and 3-6 months) was not available, and therefore is not included in this report. In addition, three of the IPS variables (vocational profile completed or updated, vocational plan completed, benefits counseling received) were aggregated across the entire 2-year reporting period. The investigators concluded that it was sufficient to count achievement of these milestones. Because initial employer contacts and job interviews apply primarily to people who are not working, these analyses were limited to the not employed subgroup during each reporting period. Most not employed treatment group participants had no employer contacts or interviews, or if they did, by far the modal rate was a single contact. Because of the low frequency of employer contacts and job interviews for each reporting period, the Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-9 investigators decided to dichotomize these variables into yes or no responses. The subgroup examined in these analyses was further limited by excluding treatment group participants who the NCC rated as unengaged during the reporting period. In the OBH section of the template, inspection of the pattern of responses to the items concerning need for services indicated virtually no difference between need and receipt of service. The MHTS investigators concluded that this item might have been difficult for the NCCs to assess and that the responses may not be valid for analysis. Specifically, the data suggested that the NCCs were inferring need from the receipt of services, essentially obviating the distinction between need and receipt. Thus, there was no reliable assessment of need for services and the investigators dropped from the analysis “need for service” items included in the SE/OBH QM template. The SE/OBH QM template also included an item regarding receipt of medication management services. Careful inspection of this item strongly suggested that some NCCs interpreted this question to be indicative of the need for a participant to receive training, intervention, or assistance with taking his or her prescriptive medicines rather than an affirmation of the receipt of a psychotropic medication. Therefore, the investigators dropped this item from this set of analyses. Instead, the investigators assessed medication management through the Systematic Medication Management Quality Management reporting system. That data is included in Chapter 6 of this report. Finally, MHTS investigators studied the temporal patterns for receipt of OBH and related services. The overall rates of receipt of services within each service category were highly stable. Thus, this chapter reports only the mean rates of receipt of OBH services across all reporting periods for the 2-year time period. The original OBH template did not collect on-site or off-site data; therefore, this report contains the mean on-site and off-site rates for the last two reporting periods only. QMPD Site Reports on Site Implementation In January 2008 and again in January 2009 the three QMPDs completed a global report on each of their study sites for the preceding calendar year using a structured checklist labeled the QMPD Site Report. This report supplemented information captured in the IPS Fidelity Scale, adding more detailed description of site level activities and characteristics specific to the implementation of the MHTS. These observations provided additional explanations for site-level performances not captured in the IPS Fidelity Scale. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-10 The checklist contained 73 items consisting of yes or no responses, Likert scale ratings, and estimates of percentages. A comments section for each item permitted the QMPD to provide qualitative responses. The checklist included nine domains, each designed to assess various aspects regarding the quality of implementation of MHTS services: Global Rating of Implementation of IPS Services, Steering Committee and Leadership, Other Behavioral Services, Study Site Director, SE Leader, SE Staffing, Mental Health Treatment Team, NCC Services, and Financing. Results Program-Level Fidelity to the IPS Model Based on the 15-item IPS Fidelity Scale, 77 percent of study sites achieved high fidelity within the first year (see Figure 5-1). An even higher proportion achieved high fidelity in the second and third years (approximately 86% of sites in both years). Conversely, across 65 fidelity reviews, the fidelity assessors rated one active site as having poor fidelity in any annual assessment. Overall, IPS fidelity ratings averaged 67.6 in Year 1, 69.3 in Year 2, and 67.2 in Year 3 (the total possible score is 75). Because of the high level of attainment of fidelity overall, most IPS fidelity items showed little variability. However, the item on Integration of IPS with Behavioral Treatment was generally a more difficult item on which to attain high fidelity. As shown in Figure 5-2, 55 percent of the sites achieved the highest score for full integration in Year 1. By Year 3, two-thirds of the study sites had achieved full integration. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-11 Figure 5-1. IPS fidelity for MHTS sites (based on 15-item IPS Fidelity Scale) 100.0% 4.8% 13.6% 22.7% 9.5% 80.0% 60.0% Low 40.0% 86.4% 77.3% 85.7% Fair High 20.0% 0.0% Year 1 (N=22) Year 2 (N=22) Year 3 (N=21) NOTE: Years 1 and 2 did not include the study site added in the third year of the study. Year 3 did not include sites that completed study participation (i.e., all treatment group participants transitioned from the study) at the time of the fidelity visit. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-12 Figure 5-2. Site integration of IPS and behavioral treatment (IPS Fidelity Scale item) 100% 18% 32% 24% 80% 23% 60% 10% 14% Poor Moderate 40% 55% 59% 67% Full 20% 0% Year 1 (N=22) Year 2 (N=22) Year 3 (N=21) Engagement in IPS Analysis of participant engagement in IPS is an important aspect that is part of the fabric of implementation. NCCs rated engagement on the QM template based on their review of participant records and consultation with the participant and the SE specialist. In this descriptive analysis, the investigators assumed engagement by employed treatment group participants (during a reporting period). Thus, the focus in this analysis was on treatment group participants who were not employed during the specified reporting period (see Table 5-1). According to the SE/OBH QM templates, the unengagement rate was 8 percent in the 12-18 month reporting period and 11 percent in the 18-24 month reporting period. During the 12-18 month reporting period, site-level unengagement rates (not shown) ranged from 0 percent to 19 percent; three sites had unengagement rates of 15 percent or higher, while nine sites had unengagement rates of 5 percent or less. During the 18-24 month reporting period, site-level unengagement rates (not shown) ranged from 0 percent to 30 percent; nine sites had unengagement rates of 15 percent or higher, while eight sites had unengagement rates of 5 percent or less. The engagement rate for treatment group participants is much higher at both 18 Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-13 months and 24 months than found in many studies of mental health treatment. For example, a review by Kreyenbuhl, Nosse, and Dixon (2009) found that “up to one-third of individuals with serious mental illnesses who have had some contact with the mental health service system disengage from care” (p. 696). Table 5-1. IPS service engagement rates for employed and not employed treatment group participants in the last two reporting periods Beneficiary group Engaged and employed Engaged and not employed Not engaged and not employed Missing2 1 12 to 18 months (N=9811) freq % 356 36.3 524 53.4 81 8.3 20 2.0 18 to 24 months (N=981) freq % 354 36.1 478 48.7 111 11.3 38 3.9 This number reflects the total number of treatment group participants who completed 24 months in the MHTS (i.e., excludes administrative drops, withdrawals, and deaths) with the exception of one beneficiary who was miscoded at the time of these analyses. 2 The missing category includes participants for whom the NCC did not complete the QM SE template. Beneficiary-Level Receipt of IPS Three important elements of IPS services are the provision of benefits counseling, conducting a vocational assessment, and formulating a vocational plan. As shown in Table 5-2, 69 percent of the treatment group participants received benefits counseling at some point during the 2-year study period, with wide variation across sites (9% – 98%). Eleven sites (48%) provided benefits counseling to 80 percent or more of the treatment group participants, nine sites (39%) provided benefits counseling to 46 percent to 74 percent of participants, and three sites (13%) provided benefits counseling to 31 percent or fewer of participants. In reporting these statistics, it should be noted that receipt of benefits counseling at any time is a modest criterion of implementation; optimally treatment group participants would receive benefits counseling whenever their benefits are expected to be affected (e.g., when contemplating starting or changing jobs). Table 5-2. Percentage of treatment group participants who received IPS service elements while enrolled in the MHTS IPS elements Benefits counseling % Received service at any time 69 Site-level % (Range) 9% - 98% Vocational profile 90 61% - 100% Vocational plan 94 63% -100% Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-14 As shown in Table 5-2, regarding both the vocational assessments and vocational plans, the overall completion rates exceeded 90 percent, but there was site variation on these indicators as well, with some study site rates below 65 percent. The timing of completing the assessment and vocational plans have not been reported here, although in a high-fidelity IPS program, IPS clients should complete these milestones within the first 3 months. Early completion of these milestones was the modal pattern for treatment group participants in the MHTS. Table 5-3 shows the rates of contact across reporting periods among not employed treatment group participants, excluding the unengaged subgroup. Treatment group participants could meet multiple times with employment specialists. This table reports the mean number of months per beneficiary when there was at least one contact with an employment specialist. During the first reporting period (months 0-3), not employed treatment group participants met with their employment specialist on average 2.2 months out of the three, or 73 percent of the available months. This rate declined for each subsequent reporting period, dropping to a rate of 53 percent in the 18-24 month reporting period. The criterion of one contact per month is a very modest criterion; in high-fidelity programs clients looking for work would be expected to meet weekly or even more often (Leff et al., 2005). As shown in Table 5-3, the rate of contact with employers, by treatment group participants who were not employed during a reporting period or by their employment specialist, ranged from 25 percent to 40 percent across the 5 reporting periods, with the highest rate during the period from 6 to 12 months. Mirroring these rates was the percentage of treatment group participants with at least one. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-15 C S Chapter 5: Implementation off Supported Employment and Other Behavioral Health and Related Services Table 5-3. IPS employment service contact rates of engaged but not employed treatment group participants during the reporting period Months 0-3 3 months Months 3-6 3 months Months 6-12 6 months Months 12-18 6 months Months 18-24 6 months Mean of all reporting periods 4 months Mean # months with at least one IPS contact during interval(s) 2.2 (73%)1 2.1 (70%) 3.7 (62%) 3.4 (57%) 3.2 (53%) 63% N (%) with at least one contact with employer 83 (25%) 102 (28%) 178 (40%) 167 (32%) 142 (31%) 31% 4 sites ≥ 50% 3 sites ≤ 10% 3 sites ≥ 50% 5 sites ≤ 10% 7 sites ≥ 50% 3 sites ≤ 10% 7 sites ≥ 50% 3 sites ≤ 10% 5 sites ≥ 50% 6 sites ≤ 10% 3 sites ≥ 50% 3 sites ≤ 10% 65 (19%) 72 (20%) 119 (27%) 100 (19%) 76 (17%) 20% 3 sites ≥ 50% 5 sites ≤ 10% 1 sites ≥ 50% 5 sites ≤ 10% 3 sites ≥ 50% 3 sites ≤ 10% 1 sites ≥ 50% 4 sites ≤ 10% 0 sites ≥ 50% 9 sites ≤ 10% 0 sites ≥ 50% 4 sites ≤ 10% 335 359 443 519 455 Interval Variation across sites on employer contacts 5-16 N (%) with at least one job interview Variation across sites on job interviews Total N2 1 2 Percentage is expressed in person-months. The total N excludes treatment group participants with missing QM templates for the given reporting period. In addition, this table does not include treatment group participants identified as unengaged (no contacts and not employed) during each of the last two reporting periods (see Table 5-1). job interview during a reporting period, ranging from 17 percent to 27 percent. Using greater than or equal to 50 percent as the criterion for a high-performing site, the number of high-performing sites ranged from three to seven sites on contacts with employers and ranged from zero to three sites on job interviews across reporting periods. Using the criterion of less than or equal to 10 percent as the criterion for a low-performing site, the number of low-performing sites ranged from three to six sites on contacts with employers and ranged from three to nine sites on job interviews across reporting periods. The NCCs reported much higher rates of monthly contact among employed treatment group participants, compared to the rates for not employed treatment group participants, as shown in Table 5-4. Overall, employed treatment group participants met with their employment specialist in 75 percent of the beneficiary months across their 24 months of participation in the study, with the rates ranging from 68 percent to 85 percent across reporting periods. The rates of followup contact with employed treatment group participants declined slightly over time. This decline is consistent with the IPS literature (Bond & Kukla, 2011). Sites varied considerably in rates of followup by employment specialists with treatment group participants. During the first 6 months, 11 to 14 sites had consistent followup with employed treatment group participants (defined as 5 of 6 months with contact). The number of sites with consistent followup with employed treatment group participants tapered off to three sites during the last three reporting periods. Conversely, the number of sites with infrequent contact with employed treatment group participants (defined as contact in one-half or less of the months in the reporting period) increased from one to four sites over the period of participation. Table 5-4. Rates of IPS contact with employed treatment group participants by reporting period Mean # months with an IPS contact Variation across sites on IPS contacts Total N2 1 2 Months 0-3 Months 3-6 Months 6-12 Months 12-18 Months 18-24 2.6 (85%)1 2.5 (83%) 4.3 (72%) 4.1 (68%) 4.1 (69%) 14 sites ≥ 2.5 1 site ≤ 1.5 11 sites ≥ 2.5 1 sites ≤ 1.5 6 sites ≥ 5 2 sites ≤ 3 4 sites ≥ 5 4 sites ≤ 3 3 sites ≥ 5 4 sites ≤ 3 87 164 319 353 347 Percentage is expressed in person-months. The total N excludes treatment group participants with missing QM templates for the given reporting period. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-17 Mean of reporting periods 75% Receipt of OBH and Related Services Rates of receipt of OBH and related services were examined for all treatment group participants who completed 24 months in the study (N=981). Table 5-5 shows a summary of those data. The reported rates are averages across the five reporting periods, which were generally stable. Overall, the study sites provided more than half of the treatment group participants with case management services. This rate does not include any case management services provided by the NCCs. Similarly, 53 percent of treatment group participants received general medical care. The rate of receipt of the other 6 behavioral services ranged from 7 percent to 21 percent. Table 5-5. Percentage of treatment group participants who received OBH or related services while enrolled in the MHTS1 Services Mental health case management General medical care Social skills training Financial assistance Housing assistance Substance abuse treatment Family counseling Legal assistance 1 2 % of treatment group participants who received specific behavioral service 54 53 21 16 15 13 8 7 % of treatment group participants receiving service who received specific behavioral service on-site2 72 12 68 68 69 44 64 37 Rate of receipt of services based on an average across all five reporting periods. The sample size for each reporting period varied based on the number of completed QM templates. The mean sample size for five reporting periods was 732. The denominator for calculating percentages includes unengaged treatment group participants. Rates of receipt of services on-site were based on an average of the last three reporting periods because the question was only included in the revised template. Table 5-5 also shows the percent of treatment group participants who received OBH or related services, and the percentage of that group who received the service on-site at the study site, as opposed to off-site at another location with a different service provider. One might expect general medical care to be primarily an off-site service, with the remaining seven services offered on-site. The data revealed that this was not consistently the case, as rates for these other on-site services ranged from a low of 37 percent to a high of 72 percent. MHTS investigators examined more carefully the site-level variation in the receipt of OBH and related services. For each behavioral service category, sites were classified as “high access,” “moderate access,” or “low access,” based on the percentage of treatment group participants Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-18 receiving each specific service. The criteria for high and low access were based on the distribution of rates of services within the MHTS sample of 23 sites. The summary overview of site variation in the rate of high and low access sites is shown in Table 5-6. Table 5-6. Frequency and percentage of high and low access sites for other behavioral health services Other Behavioral Service Mental health case management Social skills training Financial assistance Housing assistance Substance abuse treatment Family counseling Legal assistance High Access Sites Criterion for high access freq % ≥80% 6 26.1 ≥70% 1 4.3 ≥30% 5 21.7 ≥33% 3 13.0 ≥33% 1 4.3 ≥20% 3 13.0 ≥10% 5 21.7 Low Access Sites Criterion for low access freq % ≤25% 6 26.1 ≤10% 9 39.1 ≤10% 11 47.8 ≤10% 8 34.8 ≤10% 10 43.5 ≤10% 15 65.2 ≤10% 18 78.3 Site-level rates of substance abuse diagnosis and treatment varied across sites. Six sites had substance abuse diagnosis rates of 50 percent or more (based on the Structured Clinical Interview for DSM-IV Axis I Disorders [SCID]), while 5 sites had rates of 31 percent or less. In terms of receipt of substance abuse treatment, one site reported that 33 percent of all its treatment group participants received these services. No other site provided substance abuse treatment to more than 30 percent of all treatment group participants. Ten sites reported less than 10 percent of all treatment group participants received substance abuse treatment services. The correlation between site-level rates of diagnosis of substance abuse and receipt of substance abuse treatment was .11 (non-significant). The lack of correlation suggests that (1) need did not determine service provision and (2) integrated substance abuse treatment was not offered by all the sites. In 5 sites, 75 percent or more of treatment group participants receiving substance abuse treatment received it on-site, while in 6 sites less than 10 percent received it on-site. There was wide variation in the provision of case management services. At six sites 80 percent or more of treatment group participants received case management, while at six other sites 25 percent or fewer of treatment group participants received case management. Most sites provided case management on-site. In 13 sites, 90 percent or more of the treatment group participants receiving case management received it on-site, while 4 sites provided case management on-site to less than 10 percent of treatment group participants who received case management. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-19 One site provided social skills training to 70 percent of treatment group participants. No other site provided social skills training to more than half of treatment group participants, while 9 sites provided social skills training to fewer than 10 percent of treatment group participants. When it was provided, social skills training was typically on-site, with 11 sites providing it to three-fourths or more of treatment group participants who received it. Few sites provided extensive housing services, which was likely due in part to limited need among treatment group participants. Three sites provided housing services to more than one-third of their treatment group participants, while 8 sites provided housing to 10 percent or fewer of treatment group participants. When provided, housing services were most often on-site; 10 sites provided housing on-site in 80 percent or more of cases. Provision of family counseling was very rare, with 15 sites providing these services to less than 10 percent of treatment group participants. Only 3 sites provided families counseling to as many as 20 percent of treatment group participants, and for these 3 sites the family services were on-site. Provision of financial assistance also varied across sites, with 5 sites providing help to 30 percent or more of treatment group participants while 11 sites provided help to 10 percent or less. Financial assistance typically was on-site; fourteen sites provided financial assistance on-site in 75 percent or more of cases. Sites rarely offered legal assistance; only five sites provided assistance to 10 percent or more, with 22 percent being the highest rate overall. Location of legal assistance was quite variable. QMPD Site Reports Table 5-7 shows the QMPD global ratings of IPS implementation. The results show that 86 percent of sites had adequate-to-very strong implementation during 2008, and 77 percent had adequate-tovery strong implementation during 2009. In other words, there were three to five sites with substandard implementation during these periods. These results roughly correspond to the IPS fidelity ratings. As part of an exploratory analysis, the investigators asked the QMPDs to identify up to three barriers to implementation. It should be understood that this was in the context of understanding barriers in the context of a group of sites that generally had achieved the project goals regarding IPS service implementation according to the quantitative data reported earlier in this chapter. As shown in Table 5-8, the most commonly mentioned barriers to implementing the treatment in the 2009 site report were unresponsive leadership, finances, and unavailability of mental Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-20 health services. Regarding the salience of leadership as a barrier to implementation, these findings are consistent with other studies in the literature (e.g., Torrey et al., 2011). Table 5-7. Very strong Strong Adequate Substandard Table 5-8. Frequency and percentage of MHTS study sites that received various overall quality of site-level IPS implementation ratings based on the QMPD Site Reports: 2008 and 2009 Ratings freq 9 4 7 3 2008 (N=23) % 39.1 17.4 30.4 13.0 freq 5 6 6 5 2009 (N=23) % 21.7 26.1 26.1 21.7 Frequency and percentage of MHTS study sites with various site-level barriers to implementation based on QMPD qualitative assessments Barriers Leadership: unresponsive/not committed Finances Mental health services not available/not integrated Staff turnover Failure to follow model (e.g., time in community) Lack of understanding or experience Supervisor: lack of time, unresponsive to feedback State mental health actions Transportation Offices outside the Mental Health Center Vocational rehabilitation access Poor communication Local economy (unemployment rate) Capacity: Not prepared to work with # clients Outside agency provide staff Poorly paid staff Start-up issues freq 10 7 6 5 5 4 4 3 2 2 1 1 1 1 1 1 1 % 45.5 31.8 27.3 22.7 22.7 18.2 18.2 13.6 9.1 9.1 4.5 4.5 4.5 4.5 4.5 4.5 4.5 NOTE: Assessments completed in January 2009 on all 23 MHTS study sites. MHTS Site-Level Fidelity-Outcome Analysis MHTS investigators also examined associations between site-level IPS fidelity and site-level employment rates for the treatment group. The measure of IPS fidelity was the total IPS fidelity score for each site, assessed annually, as described earlier in this chapter. The total site scores for Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-21 Years 1-3 and the mean across the three years served as predictors. The site-level employment rate served as the employment outcome measure. Employment rates were based on beneficiary self-reports (i.e., the responses on the computer assisted personal interviews [CAPI], as described in Chapter 2), which indicate the percentage of treatment group participants employed at any time during the two-year period, as reported in the Quarterly and final Followup interviews (see Data Collection Procedures section of Chapter 2). As shown in Table 5-9, IPS fidelity ratings were not associated with self-reported employment rates. Table 5-9. Site-level fidelity-outcome correlations in the MHTS Fidelity CAPI (self-report) Employment Rates Site employment rate for treatment Year 1 (n = 22) Year 2 (n = 22) Year 3 (n = 21) Mean Years 1-3 (n = 23) -.10 .10 .05 -.02 Discussion This analysis of the implementation of IPS and OBH and related services in the MHTS suggests some areas of excellence as well as other areas falling short on the criteria for the planned model. The most important finding was that 80 percent or more of sites achieved a high level of IPS program implementation and maintained it across the entire study period. Given many instances in the literature in which multisite trials have failed because of inadequate implementation (Bond, 2007; Brekke, 1988; Drake et al., 2009; Michie, Fixsen, Grimshaw, & Eccles, 2009), the high level of fidelity and documentation of that fidelity using a well-validated scale is a noteworthy achievement. Even the National Implementing Evidence-Based Practices Project, which was designed to achieve high fidelity using a comprehensive and standardized training-consultation strategy, achieved only a 55 percent success rate (McHugo et al., 2007). The success in the MHTS was due in part to the judicious site selection along with careful monitoring and fidelity reviews provided by skilled consultants (i.e., the QMPDs). The review of beneficiary-level IPS fidelity was not as successful. Part of the challenge was lack of beneficiary engagement with IPS services. The nominal rate of unengagement was relatively low— less than 10 percent overall, but not employed treatment group participants appeared to have a relatively low rate of employer contact. According to NCC reports, only a small fraction of not Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-22 employed treatment group participants received direct IPS program support for actively seeking employment (in terms of employer contacts and job interviews) during the study period. What these data do not show are the reasons for the lack of employer contact by the IPS program for employed participants. The IPS model assigns responsibility for engagement to the IPS program (which includes integrated behavioral health, such as case management) rather than attributing unengagement to a lack of motivation on the part of IPS participants. In this sense, the IPS programs fell short in employing effective engagement strategies. Investigators speculated that this might have something to do with the particular clientele that included persons from the Social Security Disability Insurance (SSDI) rolls who joined the study but had not been engaged in treatment at any community mental health center. For example, the QMPDs had received reports from NCCs regarding treatment group participants who found the population served at some of the study sites to be too mentally disabled and unsettling to sit in the waiting room while waiting to meet with staff. With respect to other behavioral services, special attention was paid to receipt of mental health case management services (i.e., assistance from mental health center staff to clients with regard to illness management, practical issues such as shopping, cooking, and other activities of daily living, finding housing, and obtaining food stamps and other entitlements), because case management is key for achieving adequate mental health treatment (Rapp & Goscha, 2004). As is true in general throughout the U.S., Medicaid typically was the primary source of funding for mental health case management services to treatment group participants who received case management. The fact that NCCs reported that only 54 percent of treatment group participants received mental health case management is far below the expected rate in IPS programs serving clients with severe mental illness. There are several mitigating factors, however. First, it may be assumed that some treatment group participants did not need case management services. Unfortunately, as previously mentioned, measuring need through QM reporting was not successful, thus negating the reliability of estimating extent of need. Second, some treatment group participants received case management assistance from the NCC. However, no formal data exist on the extent to which NCCs provided case management services. Given their competing responsibilities, it is most likely that NCCs played only a limited case management role—both in number of treatment group participants assisted and in the range of case management activities performed. The QMPD site reports reinforce the conclusion that at least at some sites, case management services were insufficient. Since case management is a vital part of an integrated IPS team and was often unavailable, it may be the missing factor in engaging treatment group participants to IPS. This may have been more of a factor in this study than in typical IPS programs, because many of the treatment group participants were not engaged in any mental health center services at the start of the project. In usual IPS practice, clients are typically Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-23 first engaged in mental health treatment and then referred to IPS. By contrast, in this study, randomization into the treatment intervention occurred prior to the receipt of mental health treatment. Integration of IPS and behavioral health services was unevenly implemented. Converging evidence from the IPS Fidelity Scale, provision of OBH and related services off-site, and QMPD site reports suggest that integration was an ongoing issue for all sites, and an especially serious limitation in some sites. Based on the analyses reported in this chapter, the investigators conclude that it is possible to develop and implement faithfully a program of treatment and SE and to engage reasonably large populations of SSDI beneficiaries. Significantly, these services can be delivered not only to SSDI beneficiaries who are enrolled in public mental health services prior to enrollment in IPS, but also SSDI beneficiaries who have not been receiving mental health treatment in the public mental health system. Many of the MHTS participants were not currently engaged in mental health services— certainly not at the typical IPS clinical settings. Those who were engaged in mental health treatment had a range of options for receiving behavioral health services other than their local community mental health center, as most of them had Medicare as part of their SSDI benefits (and some had Medicaid as well). Because the investigators sought to maximize geographic dispersion in site selection and given that the large majority of sites achieved high IPS fidelity, it is plausible to hypothesize that IPS is transportable to throughout the U.S., though the study was not statistically powered or designed to answer the question of generalizability across geographic regions. This study adds to the substantial literature suggesting that IPS can be successfully implemented in different regions of the U.S. (Becker et al., 2011). While the IPS services at the program-level were available at high fidelity, job-seeking rates among not employed treatment group participants were disappointing. IPS experts may need to make modifications to the IPS model to address the characteristics of this population. This study has not isolated the exact barriers to employment for this population (e.g., concern are about losing SSDI or healthcare benefits, being less physically able to job search given the co-existence of physical illnesses, etc.). Another factor in this study was the limited time horizon; in other words, one barrier might have been the relative brevity of the followup period. This factor, however, was probably of limited impact in that the probability of actively searching for competitive work declines sharply over time for study participants in published studies of IPS (e.g., Bond et al., 2008 – PRJ review Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-24 paper). We conclude that one strategy for overcoming the barriers to employment is for IPS programs, in conjunction with mental health treatment staff at mental health centers, to employ intensive engagement efforts with this population. This study found a surprisingly modest uptake of other behavioral services among treatment participants. As is true in the general mental health services literature, the reasons for nonreceipt of other behavioral services are complex and diverse (Drake & Essock, 2009). The current study was not equipped to quantify the reasons for the unexpectedly modest uptake of other behavioral services by a group of participants likely experiencing a range of untreated psychiatric problems. One challenge in achieving precise estimates was the absence of psychometrically valid measures of need for treatment. The investigators abandoned the assessment of need by NCCs when they determined that their assessments were not credible. Therefore, one unknown factor was the extent of need. Based on the findings from the mental health services literature (Drake & Essock, 2009), another likely factor was participant refusal to accept treatment services. A third factor, which the research team documented anecdotally in weekly consultation calls by the QMPDs to the NCCs and through site visits, was the lack of access to services. Treatment group participants experienced many instances of system and organizational barriers when in need of or receiving OBH services. Drake and Essock (2009) found that the public mental health system is deeply underfunded and access to services is difficult for anyone with mental health problems. Even with the provision of full health insurance coverage, many treatment group participants did not receive the assistance they needed because of limitations in access and supply of behavioral health providers as well as policies and procedures at each site, which in turn were often prompted by state regulations. Overall, investigators observed a wide variation across sites in provision of “related” services. State policies, provider agency policies, agency culture regarding assertive engagement, leadership, and clinical supervision accounted for the wide variance of site differences. Finally, the investigators examined one predictor of employment outcomes based on site-level variation in IPS fidelity. IPS fidelity was not associated with the beneficiary-reported employment rate; this lack of correlation was more than likely a result of restriction of range. While there was some variation among sites in fidelity, it may not have been sufficient to detect differences in employment rates attributable to fidelity of program implementation. Chapter 5: Implementation of Supported Employment and Other Behavioral Health and Related Services 5-25 Chapter 6 Implementation of the Nurse Care Coordinator Role and Systematic Medication Management The primary treatment intervention components of the Mental Health Treatment Study (MHTS) offered access to systematic medication management (SMM), Individual Placement and Support (IPS) supported employment (SE), other behavioral health (OBH) and related services, and Nurse Care Coordinator (NCC) services. While both IPS SE services and SMM are widely recognized as having a strong evidence base, investigators have never fully tested these treatment modalities on the Social Security Disability Insurance (SSDI) population or across a large number of community mental health centers. The Social Security Administration (SSA) wants to know if it is feasible to offer these evidence-based services in community mental health centers across the United States, and whether providing SSDI beneficiaries with access to these services improves employment, health, and quality of life. Chapter 4 presented convincing evidence that the combination of services known as the MHTS treatment intervention indeed improves employment, mental health, and quality of life. Chapter 5 presented evidence of program-level and individual-level fidelity of the IPS SE model tested as a primary component of the treatment intervention. The purpose of this chapter is to address the extent of implementation of the SMM and NCC service components of the treatment intervention. The report covers four topics pertinent to the implementation of SMM. These topics include an overview of the psychiatric diagnoses and health conditions presented by treatment group participants, beneficiary and prescriber levels of engagement in SMM, the NCC role in the study, and a summary of the SMM implementation and its impact. Prior to addressing these topics, it is important to provide a background for understanding SMM and its intended operation in the study. Background SMM services in the MHTS used an NCC to facilitate and promote prescriber use of evidence-based guidelines and recommendations for medication management of severe and persistent mental illnesses. SMM has its roots in two different areas of clinical research into treatment of major psychiatric disorders. First, a number of studies of medication use and chart documentation in treatment of schizophrenia have identified widespread problems among providers with medication selection, medication dosing, medication side effect recognition and treatment, recognition and Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-1 treatment of persistent symptoms, and documentation of symptoms, side effects, and treatment. Second, a large literature on recognition and treatment of depression in primary care settings has demonstrated the utility of a team approach to the illness, in which a physician extender (typically a registered nurse) functions to evaluate symptoms and provide relevant assessment documentation to the treating physician with regard to need for treatment and medication selection. Given these observations, the Technical Advisory Panel and SSA decided that inclusion of SMM in the MHTS treatment intervention could contribute to improved outcomes by decreasing the deleterious effects of inadequately treated illness symptoms and medication side effects. An initial effort to devise a medication management program to remedy these identified deficiencies was the Texas Medication Algorithm Project (TMAP), which developed manuals to guide treatment of depression, bipolar disorder, and schizophrenia. Additionally, the manuals provided clinical coordinators the ability to work with physicians in evaluating patients, providing patient education, and in medication selection (Rush et al., 2003). A key component of TMAP was use of standardized assessments and documentation to quantify illness symptoms, explicit identification of medication side effects, and documentation of these observations so as to be accessible to present and future providers. Subsequent to completion of TMAP, there were a number of efforts in public mental health clinics and systems to implement approaches to medication management based on TMAP. Observations of successes and failures in these efforts supported two conclusions. First, the organization must take the lead in SMM implementation, because it involves changes in medical recordkeeping, patient flow through the system, and training of clinical teams. Second, the critical role of the clinical coordinator as the “glue” that holds together the SMM approach was emphasized by the poor record of implementation in sites that did not provide personnel to fill this role. Simply adding new tasks to already overburdened prescribers usually generated resistance and poor implementation. A corollary observation from TMAP and related projects has been that nurses have the best training to work with prescribers on medication management issues, and that their credibility with prescribers and patients in the medication arena is typically higher than for other non-physician mental health professionals. In the design of the MHTS, the investigators decided to hire an NCC at each study site. The NCC’s primary roles were: (1) to implement the SMM program (described more fully on the next page); (2) to monitor implementation of SE and OBH and related services (e.g., substance abuse treatment, benefits or family counseling) when prompted; and (3) to promote integration of care among participants’ mental and physical health providers. The NCC documented (on the SMM intake form Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-2 and NCC Clinical Reports) physical health conditions and whatever medication treatments the treatment group participants were receiving from their providers, and communicated with the physical health care provider if clinically indicated. The NCC also kept the mental health treatment team informed about participants’ physical health conditions and their treatment. The potential impact of physical illnesses and the treatments for them on functionality is clear, but it is also notable that patients with major mental illnesses, as a group, have significantly earlier mortality from medical conditions, with average life spans up to 25 years shorter than the general population (Parks et al., 2006). The majority of this excess mortality is attributable to medical conditions such as cardiovascular, pulmonary, and infectious diseases. Two realities shaped the implementation of SMM in the MHTS. First, it was impractical for participating sites to change their entire operational systems to accommodate implementation of SMM. Therefore, a system for implementation of SMM just for treatment group participants was developed, which depended heavily on forms and electronic data entry systems (i.e., the Study Management System or SMS) created by the prime contractor. That is, only a subset of patients received SMM, and clinic prescribers otherwise continued to function in their usual roles and duties. This meant that prescribers used SMM for one group of patients, but not for others, which likely affected the degree to which they bought into and adopted the SMM program as their modus operandi. Second, since beneficiary recruitment for the study was from the community at large, it was expected, and ultimately confirmed, that many treatment group participants would have ongoing relationships with prescribers who had no connection with the clinical system within the study site in which the NCC was working, and which was providing SE services. The MHTS investigators considered it quite likely that “outside” prescribers would not fully participate in the SMM program for both logistical and systemic reasons. However, the investigators concluded that it would be unwise and unnecessarily exclusionary to require that participating beneficiaries receive their medication management at the study site. Some participants opted to switch prescribers on their own, but many chose to continue with their original off-site prescribers. The NCCs tried to work with the prescribers at a distance, through e-mail, fax, and telephone contacts. At some sites, NCCs made office visits to outside prescribers to introduce themselves and the SMM program. As will become evident from the data presented later in this chapter, these accommodations to the realities of participant prescriber selection and of existing site medication management systems had large effects on implementation of SMM. Figures 6-1, 6-2, and 6-3 depict decision-making strategies and interactions related to medication selection. Figure 6-1 illustrates the contextual variables that affect prescriber medication decisions, Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-3 organized by those variables that are associated with the patient, those associated with the medication, and those that are associated with the system within which the decisions are occurring. Figure 6-1. Factors influencing prescriber medication decisions PATIENT LEVEL • Medication history • Psychiatric history • Current symptoms/side effects • Adherence/Non-adherence • Concurrent physical illness • Age, race, ethnicity • Preferences MEDICATION LEVEL • Efficacy • Tolerability • Drug - drug interactions • Drug metabolism • Dosing SYSTEM LEVEL • Formulary availability/restrictions • Cost to patient • Cost to 3rd party payors • Access/convenience issues Prescriber Medication Decisions Figure 6-2. Factors influencing prescriber medication decisions in SMM Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-4 Figure 6-3. Role-based functions in SMM The central goal of SMM is to provide prescribers with key, updated information about the patient and medication options at the time of each medication visit with the prescriber. Figure 6-2 depicts the role of the NCC as an intermediary who assimilates relevant information and recommendations and conveys them to the prescriber. Figure 6-3 lists the inputs into the process used by the NCC in the MHTS, and shows the flow of information from prescriber to NCC concerning actual medication decisions and the rationales for them. In the ideal SMM model, NCC and prescriber function in tandem, with the NCC first assessing the patient’s status with regard to medication use, medication effects, and medication-related symptom outcomes. The NCC conveys this information, along with any recommendations for change based on medication management guidelines and expert consultation, to the prescriber. The prescriber evaluates the information, and the patient, and makes final judgments about medication management. The exact division of labor between NCC and prescriber is determined at the local level. In the MHTS, the NCCs assumed responsibility for conducting clinical symptom and side effect ratings, and providing patients with education about their medications and their illnesses. Manuals provided by the investigators guided NCC medication decisions. The SMM experts trained the NCCs on how to conduct clinical rating scales at orientation. Investigators periodically made site Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-5 visits to review and reinforce scale-rating skills of the NCC. Expert consultations were available by phone and e-mail throughout the conduct of the study. The SMM model provided to treatment group participants differs from current medication management practices both in mental health clinics and in private practice settings in several important respects. First, there are few nurses in mental health clinics or psychiatrists’ offices, and their duties typically revolve around “medical” activities such as measuring vital signs and administering injections. Second, monitoring medication-related outcomes and medication decisionmaking are typically under the purview of prescribers, without systematic input from other clinic or office personnel. Third, responsibility for gathering and integrating current and historical information on medication treatments, both psychiatric and non-psychiatric, falls to the prescriber. Thus, in clinic and office settings, implementation of SMM requires systemic changes in personnel roles and in recording of information. Methods This section presents the methods used to obtain psychiatric diagnoses and physical conditions in the treatment group, and describes measures of SMM implementation and impact. Since the SMM component was only available to treatment group participants, most of the relevant measures collected by investigators were for that group only; thus, comparisons with the control group are not possible. Psychiatric Diagnoses After beneficiary randomization into the treatment arm, MHTS investigators immediately began gathering essential baseline data so that the SMM intervention could effectively begin. The investigators needed to learn what the psychiatric diagnoses of the treatment group were and how they related to SSA diagnostic categories. Trained clinicians obtained psychiatric diagnoses using the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID). These trained staff completed the SCID as soon as practicable after randomization of a beneficiary to the treatment group. The SCID interview consists of a series of questions designed to determine current and lifetime Axis I diagnoses (See Chapter 2—Diagnostic Psychiatric Assessment) according to the 4th edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) criteria. The structure of the SCID Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-6 interview is algorithmic, with initial screening questions followed by diagnosis-specific queries in response to positive answers. The NCC and MHTS experts then used the SCID diagnoses as the initial working diagnoses, which may have differed from the SSA diagnostic groups of “schizophrenia” and “affective disorder” in SSA administrative files. The SCID interview also identified other Axis I diagnoses, such as substance abuse or dependence and anxiety disorders, as well as participants with no diagnosable Axis I disorder. SCID lifetime diagnoses are not mutually exclusive from current SCID diagnoses. Therefore, it is possible to fall into both the “schizophrenia” and “affective disorder” categories over the course of a participant’s lifetime. Cooccurring substance abuse is common in these disorders, and the SCID data allowed estimation of the frequency of these and other co-occurring Axis I disorders in the treatment group. Physical Health Conditions An essential element in the study concerned the physical health conditions of participants and if or how health-related issues affected their participation. Information about physical health conditions came from a variety of sources. The eligibility screening process included an item that asked all beneficiaries enrolling in the study about physical conditions that might interfere with work or study participation. In addition, all participants indicated at baseline what medications they were taking. The General Medical Exam (GME) was required of all participants randomized to the treatment group. Since the goal of the GME was to exclude participants who were physically unable to work, rather than to identify all medical conditions, there was considerable variability among providers as to how extensively they documented conditions that were present but would not keep the participant from working. Thus, in many cases the GME report did not list all current and past medical diagnoses. The other source of information about medical conditions was the NCC, who documented current DSM Axis III conditions (physical conditions) and all medications at intake. The NCC also documented participant medications as part of the Comprehensive Transition Review (CTR) for treatment group participants at the end of study participation. MHTS investigators used medications as proxies for physical health conditions. Some medications mapped easily onto putative physical conditions, while others were ambiguous. For example, the team could easily map statins, used almost exclusively for the treatment of hyperlipidemias, while propanolol in low doses has a variety of uses beyond treatment of hypertension. For purposes of inferring physical conditions from medication treatments, the investigators adopted a conservative approach to categorizing the medications taken by enrolled participants. The investigators mapped medicines used for only one type of disorder (e.g., lung disease) to that disorder. In instances where Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-7 the medication had multiple uses, the team made no inference. Anti-epileptic medications, for example, are widely used in the treatment of bipolar disorder, and inferring a seizure disorder from their use would greatly overestimate the frequency of seizure disorders in this psychiatric population. A listing of the medications and their corresponding mapped physical conditions is included in Appendix 6A. If a report of a condition existed in the screener interview, GME, SMM intake form, or through reports of medication use, the investigators determined a participant had a physical condition. The sources for medication inference came from: (1) the Baseline interview, (2) the SMM intake form, or (3) the end of study CTR form. Since these measures were frequently redundant, researchers created an unduplicated count of participants with each condition. Most participants had more than one physical condition. As part of the SMM intake at baseline, NCCs recorded height and weight. The research team used that data to calculate Body Mass Index (BMI) for each participant. Investigators grouped these values into BMI ranges that define underweight, normal, overweight, and four levels of obesity. Engagement in SMM For investigators to examine the degree to which SMM implementation occurred and any subsequent impact of SMM on employment outcomes, they needed to design strategies to monitor quantity, quality and consistency of participant and prescriber engagement in SMM. Each NCC completed a brief quality management (QM) assessment on each treatment group participant for each quarter after SMM intake. One question asked if the participant had been actively engaged in SMM during the previous quarter, based on the NCC’s contacts with the participant. Data from these QM assessments were used to ascertain “ever” engaged (defined as having at least one occasion when the participant was rated as engaged and “always” engaged (defined as the participant being rated as engaged during every reporting period). MHTS investigators introduced the item on SMM engagement six months after the study started; therefore, not every participant had the same number of ratings. The data used in this analysis only included the 981 treatment group participants who completed the two-year study period. This was done to avoid confounding of the definitions by the 140 who were either administratively dropped from the study, formally withdrew, or died. When the NCCs characterized treatment group participants as not engaged, they would choose among a checklist of reasons for lack of engagement of the participant. The NCC could select as many reasons as needed to describe the lack of engagement. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-8 In a separate questionnaire completed by NCCs in late 2009, NCCs were asked to rate on a fourpoint scale each of the prescribers, with whom they had worked, ranging from not at all engaged in SMM to fully engaged in SMM. The questionnaire provided guidance with regard to criteria for each rating. Additionally, the NCCs characterized each prescriber as providing services either “on-site” or “off-site.” “On-site” did not necessarily mean the NCC and prescriber were co-located, but that they operated within the same clinical system, used a common medical record, and could easily communicate with each other in person, by telephone, or by a network e-mail system. Thus, for each site, there were data about numbers of on-site and off-site prescribers and NCC perceptions of their engagement in SMM. At the beneficiary level, many participants had multiple prescribers over time, and some participants moved between on- and off-site prescribers. For analytic purposes, the research team used the site characterization of their last known prescriber in the study to determine if a prescriber was on- or off-site, recognizing that this designation did not accurately capture the prescriber data history on those with multiple prescribers over time. NCC Role At various points in the study, investigators requested that NCCs estimate time spent on different functions, as well as to assess the relative value of the services they provided within SMM. An early survey of NCC percent time and effort in various roles yielded widely divergent estimates. Investigators speculated that non-recurring duties related to study start-up affected the wide variation of responses. Toward the end of the study, investigators requested NCCs to estimate hours per week spent on SMM and to evaluate their SMM role from various perspectives. Clinic administrators provided survey data in the same period. SMM Implementation Each time a participant visited an NCC, it was the NCC’s job to complete an NCC clinical report. Clinical need determined the frequency of these reports, except that investigators required NCCs to complete an NCC clinical report for each participant at least quarterly. Hence, in the 2-year period of study participation, the threshold minimum number of NCC clinical reports was set at eight reports per beneficiary. Investigators considered that SMM implementation was inadequate if there were fewer than eight recorded NCC clinical reports for a participant. Failure to meet this criterion Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-9 was due to various factors including the participant’s lack of engagement in SMM. Some sites were without an NCC for periods ranging from one to five months. In addition, as mentioned in Chapter 2, two sites ceased active enrollment of participants within one year of study onset and discontinued the staffing for both site research positions. Westat was able to provide NCC coverage for one of these two sites allowing for the continued capturing of clinical research data. However, the extended absence of an NCC affected the data on numbers of completed reports and clearly affected SMM implementation during the period of absence. The quality of medication management ratings were dependent on availability of NCC reports, of which there were none during any extended absence of an NCC. Prescriber reports in the SMS served as the comparable measure of prescriber engagement in SMM. Investigators anticipated that prescribers would see each participant at least once every four months, or six times in the course of two years. NCCs had no direct control over frequency of prescriber visits and could only pursue strategies to encourage prescriber cooperation. NCCs typically received completed prescriber reports after the participant’s prescriber visit. In some cases, the NCC completed the prescriber report based on information gleaned from the prescriber and the medical record. Quality of Medication Management Two study investigators (Drs. Bond and Miller) developed a rating scale to measure quality of prescriber medication management of schizophrenia in a prior project (Taylor et al., 2009). For the MHTS, in consultation with experts in treatment of depression and bipolar disorder, the scales were adapted for assessment of quality of medication management of bipolar disorder and major depressive disorder. Medical record reviews served as the basis for these quality assurance (QA) ratings. The scales were somewhat lengthy and required the rater to search through the record looking for evidence that the prescriber had documented intent and rationale for medication decisions and had attended to patient symptoms and side effects. Research Assistants (RAs) completed the scales on a 10 percent sample of participant records at each site each quarter. The prime contractor randomly selected site records for review by way of an automated process of the SMS. The RAs excluded previously selected records. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-10 Medication-Related Self Reports in Final Followup Interview In the final Followup interview, interviewers asked participants if they were taking medication for their psychiatric illness. If the participants responded “yes,” the interviewers asked followup questions about frequency of taking the medication as prescribed, whether they had sufficient information about the medication(s), and whether their attitude about taking the medication(s) was positive, negative, or neutral. These questions were added to the final Followup interview because part of the NCC’s role was to help treatment group participants understand the benefits of regular medication adherence and to educate them about medications. Thus, these questions could potentially help identify NCC effects on these behaviors and attitudes in the treatment group, relative to the control group. Results This section reports descriptive analyses of SMM in the MHTS. The investigators did not emphasize site differences in SMM here, but that information will be of value in multivariate analyses of site differences in beneficiary participant and other outcomes. Psychiatric Diagnoses for Participants in the Treatment Condition SSA disability diagnostic categories of schizophrenia or an affective disorder served as selection criteria for study participants. Current and past psychiatric diagnoses were determined through SCID interviews. Appendix 6B presents a summary of the SCID diagnostic frequencies. The data presented here are for all 1,025 persons assigned to the treatment condition and who completed a SCID interview. This number includes participants who did not complete the two-year study intervention. A comparison between the SCID diagnoses and the SSA diagnostic categories showed 87.92 percent concordance when including those with a dual diagnosis (Figure 6-4) and 80.33 percent concordance when excluding those with a dual diagnosis (Figure 6-5). Ambiguous case classifications affected the degree of concordance. When the investigators rated the participants with a SCID diagnosis of schizoaffective disorder or with diagnoses of both affective disorder and schizophrenia as concordant (Figure 6-4), then concordance was somewhat higher. However, if investigators limited its rating of schizoaffective disorder as only concordant with schizophrenia and those with combined diagnoses were excluded from the analysis, then the level of concordance is Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-11 lower (Figure 6-5). Among the 39 participants without a schizophrenia spectrum or affective disorder diagnosis based on the SCID, most had at least one psychiatric diagnosis. Only eight did not have an Axis I SCID diagnosis. Figure 6-4. Concordance of SSA diagnostic group with SCID diagnosis (includes dual diagnosis) 100.00% Legend: SCID Diagnosis/SSA Group 90.00% 80.00% Percent of Total (N=993) 70.00% Affective or Schizoaffective /Affective 63.75% Schizophrenia or Schizoaffective/Schizophrenia 60.00% Affective/Schizophrenia 50.00% Schizophrenia/Affective 40.00% 30.00% Other/Affective 24.17% Other/Schizophrenia 20.00% 5.94% 10.00% 2.42% 0.00% Concordant (87.92%) Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 2.92% Discordant (12.08%) 6-12 0.81% Figure 6-5. Concordance of SSA diagnostic group with SCID diagnosis (excludes dual diagnosis) 100.00% 90.00% 80.00% Legend: SCID Diagnosis/SSA Group Percent of Total (N=946) 70.00% 60.00% 57.51% Affective/Affective Schizophrenia or Schizoaffective/Schizophrenia Schizophrenia or Schizoaffective/Affective Affective/Schizophrenia 50.00% 40.00% 30.00% 22.73% Other/Affective 20.00% 9.62% 10.00% 0.00% Concordant (80.33%) Other/Schizophrenia 6.24% 3.07% 0.85% Discordant (19.67%) It was common to find co-occurring Axis I psychiatric diagnoses in addition to affective disorders and schizophrenia spectrum diagnoses (Table 6-1). The most common co-occurring condition was substance abuse or dependence. Anxiety disorders were also common, especially among the group with depression diagnoses. Both these observations are consistent with the psychiatric literature on co-occurring disorders (Kessler, 2004; Grant et al., 2004). There were 12 (1.2%) diagnoses of a General Medical Condition (GMC) resulting in a psychiatric condition (i.e., the psychiatric condition was secondary or resulting from the GMC), and 10 of these were for participants who did not have a diagnosable affective disorder or schizophrenia spectrum illness. Eight participants (0.8%) had no SCID psychiatric diagnosis. Table 6-2 illustrates the frequency of the number of psychiatric conditions in the MHTS treatment group. The SCID data refer to lifetime diagnoses. At any point in time an illness might be in remission. However, each of these illnesses was recurrent; having had the condition historically means that the person was considered at high risk for a return of active symptoms. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-13 Table 6-1. Substance abuse disorders Anxiety disorders GMC Other None Total (excluding none) 1 2 3 4 5 Co-occurring psychiatric conditions for participants in the treatment condition affective5 No or Schizophrenia4 diagnosis (n=39) Both affective5 and Schizophrenia4 diagnosis (n=55) Total (N= 1,025) Depressive disorders1 (n=3132) Bipolar3 disorder (n=300) Schizophrenia4 (n=318) 128 79 1 2 140 138 38 0 2 137 147 23 1 0 157 15 12 10 3 8 26 5 0 2 26 454 157 12 9 468 210 178 171 40 33 632 Depressive Disorder or Depressive Disorder with Psychotic Features excluding those who also have Schizophrenia or Schizoaffective. Fourteen (14) participants have both Depressive Disorder and Depressive Disorder with Psychotic Features. The duplication was excluded from the total count for Depressive Disorders. Bipolar Disorder excluding those who also have Schizophrenia or Schizoaffective. Schizophrenia or Schizoaffective Disorders excluding those with an Affective Disorder. Depressive Disorder or Depressive Disorder with Psychotic Features or Bipolar Disorder. Table 6-2. Frequency of co-occurring psychiatric conditions Number of conditions 0 1 2 3 4 5 Total Number of beneficiaries 8 447 481 82 6 1 1,025 Physical Conditions Figure 6-6 shows the frequencies of the most common physical conditions for the 981 treatment group participants who completed the two-year study period. Hypertension was the most common condition, but there was a wide range of common medical conditions. Diabetes affected 207 beneficiaries (21.1%), a percent that was substantially above what one would expect in the general population for this age group (22-56 years old). Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-14 Figure 6-6. Frequency of co-morbid physical health conditions for treatment group beneficiaries The estimations of frequencies of physical conditions came from a variety of sources. Table 6-3 shows frequency of condition by data source. About half the conditions could be inferred from current medication treatments (collected during the Baseline interview, SMM Intake, and Comprehensive Treatment Review process), but this varied quite widely across conditions. Because investigators restricted medication-based inference to medications whose use was largely limited to one category of illness, the estimations likely underestimate actual incidences of physical conditions, and of their medication treatment, to a modest degree. However, the current results support investigators’ conclusions that a substantial number of beneficiaries had untreated physical conditions known to have deleterious long-term health consequences (e.g., hypertension). Figure 6-7 shows a classification of the number and percent of participants grouped by the number of physical health conditions. The graph indicates that on average, there were between two and three physical conditions for each participant in the treatment group and only one eighth of the group had no identifiable physical condition. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-15 Table 6-3. Physical health conditions by source (treatment beneficiaries) (N=981) Condition Anemia Autoimmune Disorders Blindness Brain Damage Cancer Cardiovascular Diseases Chronic Fatigue Syndrome Chronic Lung Disorder Chronic Pain Conditions Diabetes Gastrointestinal Disorders HIV Hearing Loss Hyperlipidemia Hypertension Liver Disease Migraines Narcolepsy Neuromuscular or Degenerative Disorders Neuropathy Renal Disease Seizure Disorder Thyroid Disorders Other 1 2 3 4 5 Eligibility screener1 1 5 0 5 5 5 4 11 42 19 6 1 1 2 8 9 3 1 4 4 0 7 5 4 Baseline interview2 13 8 0 0 6 20 0 80 147 93 139 19 0 137 222 0 12 0 41 15 19 21 84 0 SMM Intake3 22 8 0 0 6 23 0 105 172 124 171 24 0 155 273 0 16 0 48 GME4 15 7 6 99 45 116 10 189 131 170 67 23 8 83 324 69 29 1 7 CTR5 20 7 0 0 8 28 0 106 151 134 175 21 0 203 285 0 9 0 52 Unduplicated beneficiary total 55 18 6 102 54 138 12 251 295 207 259 35 8 280 429 71 46 2 76 18 27 24 98 1 27 42 46 131 62 21 30 30 110 1 50 87 72 171 68 Beneficiary self-reported health conditions during the final eligibility screening process administered prior to enrollment. Beneficiary self-reported medications in the Baseline interview. Systematic Medication Management Intake conducted by the NCC. Based on beneficiary self-report and review of medical records. Beneficiary health information based on the GME findings. CTR of medications during Transition Planning. Weight gain is a common side effect of psychiatric medications and obesity is a known health risk, therefore, investigators tracked BMI. Figure 6-8 shows the findings on BMI by gender. Only 20 percent of participants were not overweight or obese. The majority of participants fell into the obese range, with more females in the more severe obesity categories. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-16 Figure 6-7. Frequencies of beneficiaries with one or more physical health conditions (N=981) 225 200 182 194 Frequency 175 158 150 125 122 125 100 84 75 56 50 28 25 16 16 0 0 1 2 3 4 5 6 7 8 Number of Conditons Figure 6-8. BMI distribution in treatment group by gender 0.0% Underweight <18.5 5.0% 10.0% 15.0% 20.0% 25.0% 35.0% 1.5% 0.7% Normal 18.524.9 18.7% 19.1% Overweight 25.0-29.9 20.8% Obese 30.034.9 21.0% Severely Obese 35.0-39.9 15.7% 14.9% Morbidly Obese 40.0-50 Super Obese >50 30.0% 9.4% 18.2% 4.0% 2.2% Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-17 29.4% 24.3% FEMALE N=523 MALE N=456 ≥9 Engagement in SMM Participant utilization affects the effectiveness of any intervention. Appendix 6C presents a summary of NCC quarterly SMM QM findings. Based on these assessments of treatment group participants, most were engaged in SMM at some point during their study participation, and the majority were engaged throughout (Table 6-4 excludes one site that ceased collecting all SMM QM research data one year into the study). The range across sites in percent of beneficiaries that reported “always” or “ever engaged” was especially pronounced for the “always engaged” category (25 – 88.6%). The “always” category excluded QM assessments completed in the first 6 months of the study because they did not include the engagement question. Table 6-4. NCC ratings of beneficiary engagement in SMM1 Ever (any reporting period) 90.42% (Site Range 75% – 100%) Always2 (all reporting periods) 57.32% (Site Range 25% – 88.6%) 1 Based on completed SMM QM Forms for the 981 treatment beneficiaries enrolled in the MHTS for 24 months. The number of QM forms completed for each beneficiary varied, ranging from 0 to 8. 2 Excludes SMM QM Forms that did not contain the engagement question. When completing the SMM QM templates for beneficiaries not engaged in SMM, NCCs were required to record the reason for unengagement. The reasons most commonly given were “refusing to meet with NCC to perform rating scales” (27%), “not on psychiatric medications” (10%), and “cannot locate beneficiary” (7.5%). Appendix 6C includes a summary of reasons for nonengagement in SMM by site. Toward the end of the study, NCCs completed a questionnaire rating prescriber engagement in SMM, at a time when they could characterize almost all prescribers who saw study participants. The differences in level of engagement between on-site and off-site prescribers strikingly favor on-site prescribers (Table 6-5, p<0.001 by chi square). Multiple prescribers from other mental health systems treated many of the MHTS treatment beneficiaries, accounting for the large number of offsite prescribers reflected in the report. Table 6-5. Relationship On-site Off-site NCC ratings of prescriber engagement by prescriber location Not at all engaged (1) freq % 3 4.7 96 33.8 Minimally engaged (2) freq % 8 12.5 107 37.7 Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-18 Moderately engaged (3) freq % 19 29.7 49 17.3 Fully engaged (4) freq % 34 53.1 32 11.3 Total 64 284 About half of the beneficiaries were seen by on-site prescribers and half by off-site prescribers. Table 6-6 reports the percent of beneficiaries seen by on- and off-site prescribers for each level of prescriber engagement. There were more missing engagement ratings of off-site than of on-site prescribers. Since off-site prescribers were more likely to be minimally or not at all engaged, the data in Table 6-5 likely underestimates the proportion of beneficiaries seen by prescribers in these two categories. Table 6-6. Relationship On-site Off-site Total 1 2 Percent of beneficiaries seen by prescribers (at final study visit by beneficiary) grouped by NCC ratings of prescriber engagement and by prescriber location Not at all engaged freq %1 5 0.7 82 11.0 87 11.6 Minimally engaged freq %1 37 4.9 129 17.2 166 22.2 Moderately engaged freq %1 69 9.2 8.2 61 130 17.4 Fully engaged freq %1 334 44.7 31 4.1 365 48.8 Total2 445 303 748 The denominator for % Total is the Total Non-Missing Beneficiary Ratings N=748. There were a total of 226 beneficiaries with a missing prescriber rating; 47 on-site and 179 off-site prescribers. NCC Roles In the NCC administered survey conducted in 2009, one question was about number of hours per week spent on SMM. More than 83 percent of NCCs estimated 10-30 hours per week, with slightly more than half this group reporting 10-20 hours per week spent on SMM. In the same survey, NCCs identified which of their activities they thought were of greatest benefit to patients. The most frequent choice by far was “providing continuity of care” (ranked first or second most important by 87% of respondents), with “tracking medications,” and “performing psychiatric scales” also being endorsed moderately frequently. Continuity of care relates to the NCCs role of integrating care across providers, including the treatment team, prescribers, and physical health care providers. When asked if NCCs can effectively perform their jobs with on-site prescribers, 76.7 percent reported “most of the time” or “all of the time.” For off-site prescribers, only 16.7 percent of NCCs indicated that they could effectively work with this group “most of the time” or “all of the time.” Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-19 Twenty-one administrators responded to survey questions about the role of the NCC. The large majority endorsed the importance of the function and rated it as having greater value than traditional nurse functions. However, more than 75 percent indicated very little likelihood that there would be funding available for such a position after the study ended. Appendices 6D and 6E present a summary of the findings from the NCC Survey and the NCC Survey Administrator Version. SMM Implementation The measures of implementation assessed frequency of required documentation of participant visits to the NCCs and prescribers. Table 6-7 shows the frequency with which NCCs and prescribers met minimal thresholds for documentation of participant visits. It is clear that NCCs were much more likely to achieve these threshold levels of visit frequencies for participants. Appendix 6F presents a summary of SMM implementation findings by site. Table 6-7. SMM implementation (% of participants1) Across all sites Site range NCC form number above threshold2 80.8 8.3 – 100.0 Prescriber form number above threshold3 37.2 0.0 – 93.5 1 2 3 Based on the 981 participants enrolled in the intervention for two years. The minimum standard for NCC Clinical Reports was eight completed forms during the 2-year study period (i.e., one form at least once every three months). The minimum standard for Prescriber Reports was six completed forms during the 2-year study period (i.e., one form at least once every four months). Quality of Medication Management The record review to evaluate quality of medication management (SMM QA) contained 23 items, each rated on a 5-point Likert-type scale. Not all items are applicable to all participants (e.g., treatment of refractory symptoms). The item mean scores ranged from 2.0 to 4.9. Above or equal to 4 is good, 3-3.99 may be problematic, depending on the measure, and 1-2.99 indicates a definite need for improvement. Appendix 6G shows a summary of the SMM QA findings by site. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-20 The grand mean of the item score means was 3.76 for the 265 treatment group participants who had ratings of their charts. Items with poor scores, on average, were documentation of side effects, description of side effects of current medications, evidence that outcome measures were used to guide treatment, and evidence of review of side effect treatments. These are all items that depend on prescriber documentation in the medical record. Mean scores for the study sites on the scale as a whole ranged from 2.9 to 4.5. Since each site had its own rater, there could have been some degree of systematic scoring differences across sites. However, investigators did spot checks of ratings of quality of medication management during site visits, and provided feedback to raters to reduce variability across sites. Medication-Related Self Reports in Final Followup Interview Results from the medication-related items included in the final Followup interview (Table 6-8) revealed that both groups had very high rates (>93%) of reporting that they took their medication as prescribed most of the time and that they had sufficient information about their medication(s). There were no differences between the treatment and control groups on these two questions. With regard to attitude toward the use of psychotropic medications to control symptoms, nearly two-thirds of participants in the treatment and control groups reported positive attitudes. There were no group differences. The groups did differ significantly in the proportion of respondents who reported currently taking psychotropic medications, with the treatment group reporting a higher current medication use (4.3%). This difference was statistically significant (p-value = 0.01). Since this was at the end of the study, this difference might be attributable to participation in SMM having prompted more participants to take medication. Discussion The results presented in this chapter point to a number of conclusions of importance to SSA. The discussion follows the same order as the presentation of the results. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-21 Table 6-8. Summary of medication-related self-reports in final Followup interview Interview Question Are you currently taking any prescription medications for an emotional or mental problem, or a problem with your nerves? Yes No Missing Treatment (N=902) freq % Control (N=991) freq % Total (N=1,893) freq % 778 122 2 86.3 13.5 0.2 813 177 1 82.0 17.9 0.1 1,591 299 3 84.0 15.8 0.2 739 28 11 0 95.0 3.6 1.4 0.0 773 31 7 2 95.1 3.8 0.9 0.2 1,512 59 18 2 95.0 3.7 1.1 0.1 724 47 7 93.1 6.0 0.9 776 34 3 95.4 4.2 0.4 1,500 81 10 94.3 5.1 0.6 503 98 171 6 64.7 12.6 22.0 0.8 545 109 147 12 67.0 13.4 18.1 1.5 1,048 207 318 18 65.9 13.0 20.0 1.1 How often do you take your psychiatric medications as prescribed by the doctor or as directed on the label? Most of the time Some of the time Less than half the time Missing Do you have all the information you need about your psychiatric medications? Yes No Missing In general, how do you feel about taking psychiatric medications? Positive Negative Neither Missing Concordance between SSA diagnostic category and SCID diagnosis was at least 80 percent. While this observation helps validate the diagnostic basis for disability, there was only a weak correlation between diagnosis and functional impairment. Use of the SCID in psychiatric disability evaluations would undoubtedly improve diagnostic precision and might improve treatment selection. However, the contribution of improved diagnostic capabilities to decisions about level of disability is arguable. Many studies have shown that co-occurring psychiatric conditions, especially substance abuse, worsen the course of illness of affective and schizophrenia spectrum disorders. Concurrent Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-22 treatment of both disorders is considered to be far preferable to sequential treatment (Drake & Bond, in press). With 44 percent of the treatment group having a lifetime diagnosis of substance abuse or dependence, the value of co-location of mental health and substance abuse treatment programs was evident. In the MHTS, treatment of substance abuse problems was left to the sites. Results did not indicate significant improvements in this area (Chapter 4), but the frequency of treatment for substance abuse was only 13 percent (Chapter 5). Future work should examine the effects on employment with greater focus on substance abuse, using evidence-based treatment strategies. The added burden of chronic physical health conditions affects functional abilities directly through the physical health problem, as well as via side effects of treatments for these problems. More than 87 percent of the treatment group had at least 1 physical health condition, according to estimates, and 69 percent had 2 or more. Moreover, more than half the group had a BMI in the obese range. All of these observations are in accord with recent reports that persons with serious mental illnesses have life spans that average up to 25 years shorter than the rest of the population and that this premature mortality is mostly attributable to physical health conditions (Parks et al., 2006). The NCCs reported that for a number of participants impairment from physical health conditions overshadowed the mental health condition that had originally been the basis for their disability. Efforts to increase employability of the population with mental health disabilities should attend to medical as well as mental health issues. The MHTS did not directly measure physical disability (the SF-12 physical component assesses patient self-perception of physical health) and did not systematically record level of control and severity of most physical conditions during the course of the study. Hence, there are no data on the impact of SMM on direct measures and treatment of physical conditions. Field reports from the NCCs and from the Quality Management Project Directors suggest that the role of the NCCs in identifying physical conditions, in recording the medications used to treat them, and in contacting physical health care providers was highly valued as an important part of the treatment program. In principle, the prescribers of psychotropic medications for study participants would be fully aware of their physical conditions and the treatments for them. The reality of our fragmented care system, however, is that care for mental and physical health care conditions is typically not integrated, often resulting in the neglect of physical health care for the mentally ill. In the MHTS, the NCC’s role was essential to the integration of mental and physical health care, and in helping participants get their physical health care needs met. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-23 Engagement of prescribers and of participants in SMM was quite variable, as was degree of implementation. The data are clear that working with off-site prescribers is extremely challenging in implementing SMM. For purposes of conducting the MHTS, the decision to allow participants to remain with their outside prescribers was perfectly reasonable. However, the goal of the MHTS was to deliver an integrated package of interventions to participants; having an off-site prescriber presented great difficulties in integrating the SMM component with other treatments. Chart reviews showed that quality of medication management varied considerably across sites and items. The poor documentation related to side effects is very consistent with literature reports (Cradock et al., 2001). Again, the off-site location of many prescribers affected this measure, since these prescribers were much less likely to use the documentation forms recommended in the MHTS. Results indicate areas of poor performance that could be addressed directly by the NCC (e.g., assessment and documentation of side effects), or indirectly through more aggressive prompting of prescribers (attending more to outcome measure ratings in making medication decisions). The final Followup interview did not find group differences in reported adherence to, knowledge about, and attitudes toward psychotropic medications among those taking them. With both groups reporting more than 90 percent good adherence and good knowledge, there may have been a ceiling effect on these items. It is worth noting that in similar patient populations, actual “good” adherence rates are typically 60-70 percent, and that patients usually overestimate their own adherence (Velligan et al., 2007). Direct measures of adherence and of knowledge would have been helpful, but were beyond the scope of the interviews. Positive attitudes toward medications also did not differ across groups. The interviewer only asked this question as a followup of those who indicated they were taking psychotropic medication(s). It might be more revealing in future surveys to evaluate these attitudes in those not taking medications, of which there were more in the control group. Finally, there is the question of feasibility of SMM implementation in existing mental health systems. Nurses are relatively expensive staff. There may well be cost savings from the NCC contributing to greater prescriber efficiency, but it is not clear that the position can pay for itself under current reimbursement schemes. While SSA does not pay directly for mental health services, implementation of the service package used in MHTS for persons with schizophrenia or an affective disorder will require other governmental agencies to develop policies and procedures that promote and pay for these services. Chapter 6: Implementation of the Nurse Care Coordinator Role and Systematic Medication Management 6-24 Chapter 7 Health Care and Supported Employment Financing A primary goal of the Mental Health Treatment Study (MHTS) was to ensure treatment participants had access to needed health care and supported employment (SE) services and supports that facilitated their return to competitive employment—without fear of incurring burdensome out-ofpocket expenditures for health insurance premiums, co-pays, work-related expenses, and other services. Toward this goal, Westat set out to identify and develop individualized benefit packages that optimized the coverage of health care, SE, and other service needs for each participant while minimizing participants’ out-of-pocket expenditures. One primary challenge for the study was ensuring availability of needed evidence-based services; a second was providing the wherewithal needed to access those services. Chapters 5 and 6 addressed the level of success in making the services available to participants. This chapter describes the strategy for gaining access to those services. The team first identified the categories of health care, SE, and other services that participants in the treatment group would require when participating in the intervention. The team then sought additional information to determine the expected spending for these services and related health care coverage and identified the relevant financing mechanisms. This chapter presents a summary, in broad scope, of the financing principles and key decisions that cumulatively formed the financing framework for providing treatment intervention participants complete access to the services they needed. Health care and SE services that constitute the benefit package referred to as the Health Benefits Plan (HBP) by the Social Security Administration (SSA). Health Care and SE Financing Principles Westat developed the following eight principles that guided the development of health care and SE financing for individual participants and related payment decisions. SSA study design requirements, recommendations from the Technical Panel, and information gathered by Westat served as the foundation for these principles. 1. Provide all participants a comprehensive health care coverage package. The Introduction and a section in Chapter 2 describe the MHTS intervention components. Chapter 7: Health Care and Supported Employment Financing 7-1 Study Design and Methodology necessitated that participants had access to traditional health care services (i.e., medications, outpatient and hospital or inpatient services for both behavioral health and general medical conditions). In addition, participants needed access to SE and related support services. Consequently, all participants in the treatment group needed comprehensive health benefit packages that, at a minimum, provided coverage for medications, outpatient care, inpatient care, and SE. 2. Ensure that participants receive needed care. Not all insurance plans cover behavioral health treatments recommended in a participant’s MHTS treatment plan. For example, Medicare did not cover certain types of rehabilitative services. When the treatment plan called for a particular treatment—whether a service, medication, or otherwise—lack of insurance coverage for the treatment was not to preclude the study covering the needed treatment. 3. Ensure that participants receive evidence-based treatments. All participants were to receive evidence-based SE services via the Individual Placement and Support (IPS) approach. Likewise, participants were only to receive evidence-based behavioral health treatments, including systematic medication management (SMM). 4. Cover SE in full. Because SE represented an essential MHTS intervention for all treatment participants and health insurance did not necessarily cover this service, the study covered the cost of SE in full. SE was not a covered benefit under Medicare and was almost never (if ever) covered by private health plans. Medicaid sometimes covered SE services in full or in part, depending on the state. 5. Cover all other approved MHTS behavioral health care expenditures in full. Participants should never become liable for any part of the expenditures of a MHTSapproved treatment for a behavioral health condition. (MHTS-approved treatments were services indicated in the participant’s MHTS treatment plan or medications prescribed by the participant’s prescriber.) 6. Utilize scarce resources wisely. Westat was mindful that both within the MHTS and in the broader health care market, health care resources were scarce. Furthermore, SSA limited MHTS health care and coverage funds to an average of $10,000 per participant for each year they were in the study (24 months). Thus, it was imperative that the study employ the limited resources as efficiently as possible. To this end, the study simplified the process of coordination of benefits, payment of claims, and reimbursement of outof-pocket expenditures. 7. Minimize participant up-front, out-of-pocket expenditures. The reimbursement process developed by the investigators minimized participants’ up-front out-of-pocket spending. In addition, the study later reimbursed these expenditures. Westat recognized that guaranteeing reimbursement for approved out-of-pocket expenditures might not have overcome the financial barriers to treatment. Many MHTS participants had limited financial resources and thus were unable to produce the credit or cash on-hand necessary to purchase needed medications or services, even when they knew that the study would reimburse them. Such participants simply could not pay the required out- Chapter 7: Health Care and Supported Employment Financing 7-2 of-pocket expenditures up front. Westat staff worked with the site staff to develop service payment solutions for such situations. 8. Collect research data on encounters and expenditures. Encounter and service expenditure data were essential in understanding the overall spending during the implementation of the MHTS intervention components across sites. Westat implemented payment mechanisms to support the timely collection of accurate encounter and service expenditure data in order to inform policy on this population and to provide the services participants needed to recover, achieve independence, and work competitively. Westat drew upon various sources to estimate the expenditures associated with the needed services and supports for MHTS participants and the price of a standard health care benefits package. The MHTS Health Benefits Plan team obtained information that included actuarial estimates of insurance premiums, the availability of comprehensive health insurance policies for the uninsured, historical spending on SE, and Medicare Part D policy (for prescription drug coverage). Members of the team of investigators conducted focus groups with MHTS site directors to learn how the sites customarily finance the SE and behavioral health services they deliver. Health Care Coverage and Financing Decisions This section chronicles the fact finding process and summarizes the history, the decisions made, methodology used, and the payment mechanism used to ensure appropriate health insurance coverage and premium reimbursements for treatment participants during the course of their participation in the treatment intervention. Participants’ Expected Insurance Coverage at Time of MHTS Enrollment While the prevalence of participants with private plans was unclear, based on findings from previous studies of adult populations with disabling mental disorders, the team anticipated that: Approximately one-third of MHTS participants would be ―dual-eligible‖ (i.e., have both Medicaid and Medicare coverage), having qualified first on Social Security Disability Insurance (SSDI) but later having qualified for Supplemental Security Income (SSI); Nearly two-thirds would have Medicare only; Chapter 7: Health Care and Supported Employment Financing 7-3 Approximately one percent would have neither coverage (i.e., be in the 24-month Medicare waiting period); and One percent would be in the waiting period for Medicare, but have Medicaid. Insurance premium expenditures. The Hay Group (an MHTS subcontractor) provided Westat an actuarial estimate of the premium cost for commercial health insurance that provided individual coverage equivalent to the benefits provided under Medicare Parts A, B, and D; the amount was $3,780 per year or $315 per month. Westat used this estimate of the cost of comprehensive coverage as a guide in determining the MHTS premium reimbursement amounts for participants with various insurance coverage arrangements. Availability of health insurance policies for the uninsured. If the study were to enroll participants possibly with no health care coverage (e.g., those in the 24-month Medicare waiting period), Westat needed to purchase coverage equivalent to Medicare Parts A, B, and D for these participants as soon as possible after enrollment. Consequently, a Westat team of experts in health economics and mental health service policy researched the availability of comprehensive health insurance policies for participants in each of the states in which study sites were located. In all states except Maryland, the available insurance plans contained stringent pre-existing condition clauses. These policies did not cover treatment for any condition that the participant had at the time he or she enrolled in the insurance plan until a waiting period ranging from 3 months to 24 months had passed. Thus, the plans precluded coverage for participants’ (well-documented) mental disorders, as well as for any general medical conditions they had when they enrolled in the study. The two state-operated sites (both in Connecticut) proved an exception. Since the state itself administered these two sites, they agreed to provide behavioral health care, both outpatient (including SMM) and inpatient, to uninsured MHTS participants at the state-run facilities at no cost. As with other sites, the study compensated these two sites for SE and related services. Medicare Part D policy. Members of the team of investigators, Westat health economists, and University of Texas pharmacy experts studied the policies and regulations of the Medicare Part D, as well as costs and cost sharing for a sample of Part D plans. Study results found that Part D plan premiums ranged from $15 to more than $100 monthly, depending on many factors, including the plan’s cost-sharing requirements, the specific medications on the plan’s formulary, whether coverage was provided in the plan’s so-called ―donut hole,‖ and other factors. The ―donut hole‖ refers to a gap in Part D coverage that occurred when the participant reached the plan’s initial annual coverage limit, generally $2,400 in 2007. Such a participant has ―fallen into the donut hole‖ or coverage gap Chapter 7: Health Care and Supported Employment Financing 7-4 and becomes responsible for the total spending on all medications until total out-of-pocket expenses on formulary medications for the year, including the deductible and initial co-insurance, reaches $3,850. When total out-of-pocket costs reached $3,850, Part D coverage resumed.1 Some Part D plans offered coverage ―in the gap,‖ while others did not. It became apparent that participants with many or high-cost medications—whether for behavioral health or general medical conditions— would represent a sizable expenditure to the MHTS when they hit the donut hole. Based on the foregoing review and substantial discussion with experts, the following decisions became the guiding blueprint for providing participants access to the intervention treatments and services. Decision 1: Provide comprehensive coverage equivalent to Medicare Parts A, B, and D. To maximize access to effective treatments, all treatment participants received insurance coverage equivalent to Medicare Parts A, B, and D upon enrollment in the MHTS. This level of coverage is ―comprehensive coverage.‖ Medicare Part A provides inpatient coverage, while Part B covers office visits, lab work, and other office-based care. Part D provides prescription drug coverage. Thus, Medicare Parts A, B, and D represent a comprehensive health insurance package and cover the major services needed by MHTS participants—with the exception of SE, which the study covered in full. The MHTS provided participants who had not yet completed the Medicare 24-month waiting period with comprehensive coverage (i.e., coverage equivalent to Medicare Parts A, B, and D). As discussed earlier, because of the issues related to obtaining comprehensive coverage for the uninsured participant, the MHTS sample initially included participants in the 24-month Medicare waiting period only from the state of Maryland and the two state-operated sites.2 In the case of individuals who had Medicaid, private, or other insurance at the time of enrollment, Westat tailored their coverage to ensure its equivalence to Medicare Parts A, B, and D. For participants who chose to remain on their existing private insurance policies, Westat met the MHTS comprehensive coverage standard without compromising the individual’s preference, whenever possible. In short, Westat brought participants’ insurance coverage up to the study standard regardless of their coverage at the time of study enrollment for the duration of the 24-month intervention period. 1 Equivalently, Part D coverage resumed when the combined annual amount of the participant’s total out-of-pocket costs and Medicare payments exceeded $5,451. 2 Enrollment for persons in the 24-month Medicare waiting period at other sites were postponed for 6 months until completion of a review of enrollment data and further investigation into health care coverage options in the remaining study sites. Chapter 7: Health Care and Supported Employment Financing 7-5 For participants that entered the study with Medicare Part A (which requires no premium) but without Part B or D (which require monthly premiums), Westat obtained Part B and Part D coverage for them at the earliest opportunity (as needed), which often required waiting until the respective open enrollment period. Part B open enrollment is from January 1 through March 31 of each year, with coverage taking effect July 1 of that year; Part D open enrollment is from November 15 through December 31 of each year, with coverage taking effect the first day of the new year. Regardless of any waiting periods, the participant did not incur a cost for MHTS-approved behavioral health care treatments or medications. The study paid the full expenditures related to such care when no other coverage existed. Table 7-1 shows the insurance coverage that Westat obtained for each participant, depending on the type of coverage the participant initially brought to the study, as well as the expected costs to the study and to the participant for premiums, behavioral health care, and general medical care. As this table shows, participants using MHTS-approved providers for MHTS-approved services incurred no costs for their behavioral health care. Decision 2: Reimburse premium payments for coverage equivalent to Medicare Parts A, B, and D. Westat paid the insurance premiums for Medicare Parts B and D (Part A has no premium) or equivalent coverage (i.e., comprehensive coverage). Whenever possible, Westat paid the insurance premiums directly to the insurer to minimize out-of-pocket expenditures to participants. When this was not feasible, Westat reimbursed the participants for their insurance premiums in advance of premium due dates. For example, when participants had Medicare Part B coverage, the insurer is the Centers for Medicare & Medicaid Services (CMS). CMS deducts premiums directly from Social Security benefit checks. To avoid a lag between this premium deduction and participants’ receipt of premium reimbursement checks, Westat paid premium reimbursements to these participants prior to receipt of their Social Security benefits checks. Further detail to the approach to premium payments is presented in greater detail in the next section covering Implementation of the HBP. Chapter 7: Health Care and Supported Employment Financing 7-6 Table 7-1. Participant group1 General medical care Behavioral health care2 medium $0 None Medicare premiums medium low $0 None $0 very low very low $0 Medicaid only None $0 very low very low $0 51 No insurance Coverage equivalent to Medicare Parts A, B, and D4 Higher than Medicareequivalent premiums high medium $0 61 Private insurance None Higher than Medicareequivalent premiums high low $0 41 4 Westat behavioral health care reimbursement2 high 3 3 Participant-paid out-of-pocket costs Medicare premiums 2 2 Participant health care coverage at study entry Medicare Parts A, B, and D only Medicare Parts A, B, and D Medigap Medicare Parts A, B, and D Medicaid Basic health care coverage Westat obtained following Westat basic premium coverage cost3 None 1 1 Insurance plans and estimated costs to Westat and participants Participants in groups 4, 5, and 6 are in the 24-month waiting period for Medicare. The MHTS paid for all behavioral health care out-of-pocket costs incurred by participants using MHTS-approved providers for MHTSapproved services. The MHTS paid premiums for basic health care coverage, Medicare Parts A, B, and D or the equivalent coverage. If a participant with Medicare did not have either Part B or Part D coverage at study entry, the MHTS obtained Part B and the most appropriate Part D policy for the participant and paid the premiums. Decision 3: Delay enrollment of uninsured participants until costs are analyzed. After analyzing the health insurance landscape for uninsured participants in states with MHTS study sites, Westat concluded that the available health insurance policies would afford MHTS participants insufficient coverage due to the pre-existing condition exclusions.3 Moreover, Westat had particular concern about the lack of control over cost containment for long-term behavioral health care 3 Although the health plan may accept enrollment, SSDI beneficiaries who certainly have pre-existing conditions may not have coverage for any care or services related to the pre-existing condition. Depending on the policy and the state’s insurance regulations, this exclusion period can range from 3 to 24 months. Chapter 7: Health Care and Supported Employment Financing 7-7 inpatient costs. A relatively small number of uninsured participants incurring lengthy inpatient stays could potentially drain study intervention resources. As noted earlier, preliminary research indicated that only in one state (Maryland) could the study purchase comprehensive health care coverage—without pre-existing condition exclusions—for uninsured participants. In addition, Connecticut’s two state-run MHTS study sites agreed to provide behavioral health care to uninsured participants at no cost. Prior to enrollment, uninsured beneficiaries who would be served at the state-operated study sites were informed that behavioral health services, including inpatient care, could only be obtained through the MHTS study site. If participants chose to receive behavioral health care outside of these facilities, they incurred the costs. Finally, for uninsured participants from these state-operated sites, the study purchased insurance coverage from the state’s high-risk pool to provide coverage for their general medical care. For other than the exceptional accommodations mentioned above, Westat and SSA decided to delay enrollment of participants who are in the 24-month waiting period for Medicare for 6 months until completion of a review of enrollment data and further investigation into health care coverage options for uninsured participants in the remaining study sites. After reviewing the collected data, Westat and SSA decided to enroll all beneficiaries across all the study sites, regardless of the length of time they had been receiving SSDI benefits. This addition became effective August 1, 2007, with the provision of a spending cap on study expenditures for uninsured treatment group participants from the remaining study sites. Implementation of the HBP Westat created a dedicated full-time position, the Westat Insurance Planner (WIP), to assess and verify required insurance premiums, investigate additional insurance needs, facilitate obtaining any additional insurance needed to attain a comprehensive level of coverage, and coordinate the timely payment of insurance premiums. Participants’ insurance issues typically required immediate assessment and response to ensure adequate coverage at all times during their study enrollment period. The WIP responded to and resolved participant insurance premium and other coverage matters in a manner that would not have been possible were the premium payment function housed outside of Westat. Chapter 7: Health Care and Supported Employment Financing 7-8 Assessment and Verification of Existing Coverage The Research Assistant (RA) at each study site collected detailed insurance coverage and premium information on each newly enrolled treatment participant during the post-randomization meeting (discussed in the Chapter 2). The Health Insurance Questionnaire (completed by the RA during the postrandomization meeting; Appendix 7A) showed the insurance information collected. The RA used the secure MHTS fax line to send the WIP a copy of the completed Insurance Questionnaire and copies of the participant’s insurance card(s). The WIP reviewed this information, collected additional information as necessary, and established the extent of the participant’s current coverage; the WIP then determined whether the participant needed additional coverage. During this process, the main goal was to make sure that each participant had comprehensive health care coverage (i.e., Medicare Parts A, B, and D or equivalent coverage from another source). For example, beneficiaries needed Medicare Part B coverage if they did not have Part B or coverage from another source that paid for physician services. Similarly, a beneficiary needed Part D coverage if he or she did not have Medicare Part D or any other coverage for prescription medications. To ensure the appropriate use of study funds for insurance premium reimbursement, the WIP verified each participant’s insurance coverage to determine the level of reimbursement. In the course of the study and especially during the enrollment phase, treatment participants often did not have their insurance cards or Proof of Income Letter from SSA to verify their monthly premium payments for Medicare Parts B and D. The Proof of Income Letter lists SSA benefits and any deductions such as Medicare Part B or D premiums (if applicable). The letter would indicate no premium deductions for participants who were dually eligible (Medicare and Medicaid) since the premiums were paid by Medicaid. Westat, with significant support from the site RAs, relied on the participants to supply the needed documentation. Trained site RAs helped participants request necessary information from the SSA and Medicare websites. The WIP routinely visited the Medicare.gov website to verify participants’ enrollment in Medicare Part D. Though sometimes dated, the Medicare.gov website provided valuable information on participants’ Medicare Part D or HMO coverage. In addition, the site provided premium cost information coverage limitations. Third party queries (TPQY) were another verification tool used when allowed access by the state in which a treatment participant resided. The TPQY provided a listing of all insurance premium deductions along with effective dates. To verify private insurance premiums costs, a copy of the policyholder’s paystub was requested (most often the participant was not the primary on the policy). In the event that an employer covered the full cost of an insurance premium, the primary insured obtained a letter from his or her human resources department. Once verified, the WIP entered this information Chapter 7: Health Care and Supported Employment Financing 7-9 in the insurance section of the Study Management System (SMS). The investigators were not provided access to the CMS system for insurance verification. The process therefore was longer than originally anticipated, taking on average four to six weeks per beneficiary. Final notices were sent to participants via certified mail after six months where attempts to gather the verification documentation failed. The letter outlined the failed attempts to collect this information and listed the information needed. Some participants immediately responded by providing the missing information. In these instances, the WIP updated the SMS and set up a payment schedule. The WIP closed the insurance verification file and made a notation in the SMS for participants who did not respond to the certified letter. The investigative team concluded that most of these cases were individuals who were on Medicaid and did not pay for insurance premiums. The study did reimburse any participant who subsequently produced proof of premium payments. Meeting insurance needs. Most participants entering the study had coverage through Medicare only, Medicare and private insurance, or some other form of insurance (e.g., VA benefits or other government benefits). However, some participants were within the 24-month waiting period to be eligible for Medicare coverage and were thus uninsured. Other participants had Medicare Part A only. Still others had Part A and D coverage or Part A and B only. The following address the steps taken to enroll participants in Medicare Part B or D or another form of government insurance. Lack of Medicare Part B coverage. Open enrollment for Medicare Part B occurs annually from January 1 through March 31, with coverage becoming effective on July 1. In January of each year, the SMS generated a report that listed the participant ID numbers for those who did not have Medicare Part B coverage. The site RA met with each participant without Medicare coverage to discuss insurance enrollment. The WIP provided each RA with the proper forms and instructions on how to enroll any participant interested in enrollment. The RA documented the participants’ decisions about opting to enroll or not and communicated this information to Westat. Some participants did not wish to enroll because they received coverage under a spouse or parent’s insurance. Others simply refused. Participants who chose to enroll received confirmation, premium cost information, and an updated Medicare card from CMS. Westat staff monitored the progress and status for all affected participants throughout the open enrollment period and documented any changes in the SMS system. Once SSA began to deduct premium costs from a participant’s SSDI monthly payment, Westat generated a payment schedule in the SMS, to reimburse the participant for the cost of that premium. The process is explained in detail below. Chapter 7: Health Care and Supported Employment Financing 7-10 Lack of Medicare Part D coverage. Open enrollment for Medicare Part D is November 15 through December 31 annually. In November of each year, the SMS generated a report listing the participant ID numbers for those without Medicare Part D coverage. As with Part B enrollment efforts, Westat provided each site RA with a list of the IDs for their participants without a prescription plan. The site RA and the WIP assisted participants opting for enrollment or those wanting to switch from their current plan utilizing the Medicare.gov website. The criteria for choosing the Optimal Medicare Part D plans were as follows: 1. Participant had neither medication history information nor Part D plan. When no medication information was available at the time of Part D enrollment, the strategy was to use the list of most commonly prescribed (brand-name) medications for the following: – Schizophrenia: Seroquel, Zyprexa, Risperdal; – Bipolar: Depakote ER, Trileptal, Geodon; and – Depression: Zoloft, Lexapro, Effexor XR. By using this method, there was consideration for drug expense and most commonly prescribed behavioral health drugs (i.e., second generation antipsychotics, selective serotonin reuptake inhibitors, and mood stabilizers) for the study population. Given this medication list, the Medicare Drug Plan Finder (MDPF) produced a list of plans with expected annual spending. Considerations for an optimal plan choice included: 2. – Expected total annual spending, – Coverage in gap, – Monthly premium and annual deductible, – Network of pharmacy, and – Rating (two or more stars in all plan quality dimensions). Participant had medication history information but no current Part D plan. Strategy was based on: – Current medication information; or – Predicted future medications (strategy 1) to find an optimal plan for the participant with respect to the five items listed above. Chapter 7: Health Care and Supported Employment Financing 7-11 3. Participant had a Part D plan but requested a switch of plans. Strategy was based on: 4. – Current medication information; along with – Predicted future medications (strategy 1), to find an optimal plan for the participant. The WIP compared the annual total spending of the optimal plan with the participant’s current plan. To recommend switching plans the optimal plan expected spending had to be at least 10 percent lower than the current plan. Participant had private insurance plan that covered medication. In this case, the study did not enroll the participant in a Medicare Part D plan. To avoid a participant incurring unwanted Part D premium plan payments during the calendar year in which their study participation ended, the MHTS made the decision to continue Medicare Part D reimbursement for the entire calendar year that transition occurred. This continued reimbursement for the remainder of the calendar year ensured that the participant incurred no financial hardship in relation to the Medicare Part D plan due to participation in the MHTS.4 Lapse of coverage. Two participants lost Medicare coverage during the study because of a medical Continuing Disability Review (CDR) that commenced prior to study enrollment. They were able to enroll in the study because they were appealing the decision to have their coverage revoked and therefore were still allowed to have Medicare until the final decision was made. When these participants lost their coverage, the study enrolled them in state-funded high-risk insurance programs. Enrolling Uninsured Participants Westat assisted uninsured participants to enroll in high-risk insurance plans in their state, when one was available. The WIP sent applications to the site RA via email, which in turn assisted the participant with the form completion. The WIP then reviewed and submitted completed applications to the insurance company along with any initial enrollment fees. When the WIP received enrollment confirmation, the information was documented in the SMS and a payment schedule was set up. 4 The Westat Institutional Review Board required the study to take precautions to ensure that participants incurred no harm as a result of participation in the MHTS. Participants were to leave the study no worse off than they were at the start of the study. SSA concurred with this view. Chapter 7: Health Care and Supported Employment Financing 7-12 Enrolling Participants in a New Health Plan Table 7-2 shows the number of participants that Westat staff assisted with enrollment into a health plan for the first time. The table breaks this information out by year and by type of coverage. The HBP implementation resulted in new enrollments of 21 participants in Medicare Part B, 50 participants in a Medicare Part D plan, and 3 participants in a private insurance plan (e.g., state highrisk pool). Thus, the WIP assisted enrollment of 74 participants in a new health plan that would otherwise lack adequate coverage. In addition to acquiring coverage, the WIP assisted a number of participants switching Part D plans, comparing private, HMO, and Part B plan options, and in general, making sure they received and maintained adequate health coverage throughout the study. Table 7-2. Number of participants enrolled in a new health plan by the WIP, by year and by type of coverage Insurance coverage Medicare Part B 2007 0 2008 13 Medicare Part D 23 25 2 50 Private Insurance 0 2 1 3 23 40 11 74 Total 2009 8 Total 21 Determining Premium Payment Amounts to Participants As noted earlier, the standard (without late penalties) monthly Part B premium was $93.50 in 2007, and the monthly premium for a typical Part D plan was approximately $50. Thus, the MHTS average expected premium payment for a participant who had both Medicare Part B and D was roughly $143.50 per month. Table 7-3 summarizes the changes in Medicare Part B premiums during the course of the study. The MHTS paid the entire premium, up to a predetermined maximum amount (see the following page) for a participant who had private insurance that covered only him or her. However, when a participant was not the only individual covered on an insurance plan policy, the study paid only that portion of the monthly premium that represented the participant’s share. Furthermore, Westat reimbursed the person identified as the primary insured, (not necessarily the participant), for the premium or premium portion. Chapter 7: Health Care and Supported Employment Financing 7-13 Table 7-3. 1 Medicare Part B premium changes Year 2006 Medicare Part B premium $88.50 Minimum premium reimbursement amount1 $138.50 2007 $93.50 $143.50 2008 $96.40 $146.40 2009 $96.40 $146.40 2010 $96.50 $146.50 The minimum premium reimbursement amount was always the Medicare Part B premium plus $50 (typical Medicare Part D premium) for each year. As a result, Westat set the minimum premium reimbursement amount at $143.50 per month ($93.50 for Part B equivalent coverage and $50.00 for Part D equivalent coverage) for participants with private insurance coverage for 2007. The minimum premium reimbursement amount increased as the Medicare Part B premium increased. A maximum premium reimbursement amount of $350 per month was also set. Actual premium reimbursement amounts for private insurance coverage were as follows: If a plan covered the participant and one other person, the study paid 50 percent of the monthly premium up to the maximum premium reimbursement limit. If a plan covered the participant and two or more other persons, the study paid 40 percent of the monthly premium up to the maximum premium reimbursement limit. If the total monthly premium payment for the private insurance was less than $146.50 (in 2010), the study paid the total premium amount. If the participant’s portion of the monthly premium for the private insurance was less than $146.50, the study paid $146.50. A $350 per month, per participant cap was set so that no participant received any reimbursement above that amount. For participants with Veterans Affairs (VA) benefits who also had Part B and Part D coverage, the study reimbursed the premiums associated with those plans. MHTS offered participants covered by VA benefits but without Part B and Part D coverage enrollment in those plans. The site RA informed participants about the necessity of Part B and Part D Coverage if they wanted to receive health services from providers outside the VA system. If a VA-insured participant asked to be enrolled in Part B and Part D, as with a participant on Medicaid, he or she would be considered to have ―creditable coverage‖ and therefore could be assisted with enrollment in Parts B and D plans outside the open enrollment window. Chapter 7: Health Care and Supported Employment Financing 7-14 Treatment group participants who were dual-eligible (had both Medicare and Medicaid coverage) were enrolled in Medicare Part B only if they specifically requested such enrollment. As previously discussed, Medicaid paid for Medicare premiums for dual-eligible participants. In addition, the study obtained Part B and Part D coverage and started paying the associated monthly premiums for participants who lost Medicaid coverage during the period of study participation. Paying Prescription Medication Co-Pays One of the study requirements was to cover participant out-of-pocket expenses associated with study-related services. As discussed elsewhere, a participant could file a claim for any expense they believed related to their efforts to return to work. Some of these included making co-payments for behavioral health visits to an off-site provider, transportation to or from a medical appointment, etc. SSA expected that the study also would cover out-of-pocket expenses associated with the purchase of prescription medications for the treatment of mental disorders. Even with the promise of eventual reimbursement through the HBP, some participants would find the initial out-of-pocket expenses a substantial financial barrier. SSA requested that the investigators explore ways to overcome this potentially expensive problem for participants seeking treatment. The solution—at least for prescription co-payments—was a debit card. Subsequently, Westat issued debit cards to all participants for purposes of making prescription co-payments associated with behavioral health medications. A more complete description of the debit card plan and implementation appears in the Supplemental Appendix. Paying Essential Work-Related Expenses An additional responsibility of the study was covering essential work-related expenses (EWRE).5 It is often the case with low-income people with disabilities that the difference between getting a job and not getting a job is the expense of some incidental item. Some examples are the cost associated with retaking the nursing licensing exam to update a license, and obtaining appropriate work attire (special uniforms or interview clothes), and obtaining dental work or prescription lenses. The study always considered requests for payment for these types of expenses. The Nurse Care Coordinator was required to complete a special form and submit it to an identified member of the investigative 5 EWRE should not be confused with the SSA support program known as Impairment-Related Work Expenses (IRWE). The EWRE was designed specifically for the HBP. Chapter 7: Health Care and Supported Employment Financing 7-15 team for review and approval. These types of expenses were rare in the study. Approvals of these types of expenses occurred when a clear and direct link between the item and a specific potential job was imminent. An example of expenses not approved was general software training (such as Microsoft suite of general business use software). A few exceptions occurred when an item clearly improved the participant’s opportunity to find employment. An example that occurred more than once in the study was the purchase of dentures for participants without teeth. SE Financing The MHTS investigators made many decisions in the course of designing the HBP and implementing its functions. This section details some of the team’s key decisions made with regard to the provision of SE services. Decision 1: Replicate the fee-for-service financing environment in SE payment mechanisms. During the past decade state mental health authorities have moved away from funding psychiatric rehabilitation services using grants and have moved toward fee-for-service financing using Medicaid funds. To generate SE expenditure data that are relevant to policymakers, the study initially proposed to replicate this fee-for-service environment. Likewise, the MHTS required accurate data on the number and types of participants’ SE and related encounters. To achieve these objectives, Westat initially required sites to complete and submit Monthly Encounter Forms for each treatment group participant in order to receive payment for SE services delivered to treatment participants. Sites submitted forms monthly to ValueOptions, the claims processor for the study. ValueOptions extracted encounter data from these forms, which allows the study team to construct expenditures estimates of the SE services delivered. Appendix 7B shows the Monthly Encounter Form. Decision 2: Recognize the cost to study sites of carrying inactive participants on their SE caseloads. Westat and the study sites realized that treatment group participants represented a different type of client than many of the community mental health centers serving as study sites were accustomed to serving. Typically, clients enrolled in other services at the agency, and approached the site with a stated desire to work and receive SE services. These clients often remain on a wait list until an SE ―slot‖ becomes available. In the study, however, sites first reached out to the participant, who may not have considered the possibility of returning to work. The participant therefore, may have enrolled in the study with some uncertainty about working or because of the many incentives treatment group participation offered (e.g., payment of insurance premiums and out-of-pocket behavioral health care expenditures). In the focus groups with the study site directors, Chapter 7: Health Care and Supported Employment Financing 7-16 sites voiced this concern. Site directors were also concerned that participants might fill SE slots without actively participating in the treatment intervention. If this occurred, sites would be unable to generate income by billing for services while still having to pay the full salary of an SE specialist (a fixed cost). Furthermore, as an intent-to-treat study design, the rules forbid dropping or replacing inactive participants. To cushion any such revenue losses that sites incurred due to participants not participating in the intervention—and in recognition of the fixed cost component of carrying MHTS participants on the SE specialist’s caseload—Westat developed an SE payment schedule that paid sites a minimum monthly amount per participant, regardless of the participant’s level of study participation. Decision 3: Base sites’ SE payments on participant insurance coverage, site billing capabilities, and the number of face-to-face participant contacts. The MHTS initially paid sites for delivering SE based on three factors: 1. Whether the site billed Medicaid for SE; 2. Whether the site billed a third party for other behavioral health services delivered to the beneficiary; and 3. The number of face-to-face SE and related encounters in the billing month. The study paid less to a study site that was able to collect revenue through another payer for other behavioral health services. The rationale was that the first site collected revenue from both a thirdparty insurer and the study, while the second site depended entirely on the study for payment of the SE and other behavioral services. A site’s ability to bill Medicaid—the only insurer that pays any part of SE—depended not only on the participant having Medicaid coverage, but also on the site’s billing capabilities. As noted earlier, many study sites were accustomed to receiving grants or capitated payments for their services and consequently lacked the administrative capacity to bill insurers such as Medicaid or Medicare on a fee-for-service basis. The MHTS did not penalize sites for their lack of third-party billing expertise. Instead, the study paid these study sites for SE at the highest rate, as they had no other means of generating revenue for MHTS participants. Finally, the evidence-based IPS approach to SE demanded frequent face-to-face client contact. To encourage this contact, SE payments to sites also depended on the number of face-to-face contacts between the participant and SE specialist (or case manager providing SE-related case management). In general, the study SE payment rates were to cover the cost of additional non-face-to-face contacts that were also a part of SE. Appendix 7C, Original and New Supported Employment and Related Services Payment Schedules, shows the SE Chapter 7: Health Care and Supported Employment Financing 7-17 payment schedule used at the start of the study in the columns marked ―Original Schedule.‖ Almost from the beginning of the study, Westat negotiated alternate payment schedules for 5 of the 22 sites—Maryland, Minnesota, New York, and two sites in Connecticut—because of unique service and billing circumstances in each of these sites. Decision 4: Modify the original payment schedule to increase the payment amounts due to lower than expected revenues to support SE services. The original schedule outlined payments for participants on Medicaid (Schedule C), participants for whom there was some reimbursement for other behavioral health services but not for supported employment-related services (Schedule B), and participants for whom there was no reimbursement for SE-related services or other behavioral health services (Schedule A). The original schedule paid a minimum of $100 per month for each participant, even if there was no face-to-face contact. Sites had raised a variety of concerns about the schedule from its inception, and Westat made a commitment to the sites to conduct a review of the sites’ experience with the schedule after 6 months. Thus, in July 2007 the MHTS investigators conducted a mid-course review of the study payment schedule for SE and related services. Overall, Westat needed to determine if the schedule was serving the study objective of adequately compensating sites for delivering evidence-based services. Five related questions emerged as salient for both Westat and the sites: Is the basic structure of the schedule adequate? Are the payments adequate to cover study sites’ staff costs? Does the schedule properly compensate for non-face-to-face work associated with SE and related services? Case management often is critical to SE—can the schedule cover the unreimbursed costs of case management? Are the rules of the schedule clear to sites? Westat conducted a fact-finding mission, questioning study site administrators directly to learn more about these five issues. Westat also interviewed the three Quality Management Project Directors (QMPDs), experts in the service delivery of IPS SE and evidence-based behavioral health treatments. The three QMPDs (described in Chapter 2) worked closely with the study sites to ensure the delivery of high-quality services within the study sites. Therefore, they were familiar with the Chapter 7: Health Care and Supported Employment Financing 7-18 structural and financial barriers that sites faced in delivering these services. Westat also reviewed ValueOptions billing records for SE and other behavioral services in order to document the sites’ actual billing practices in the study’s first 6 months of enrollment. Finally, the investigators compared the actual participant enrollment data with expected enrollment over this period. This investigation led to a number of findings, summarized below. Difficulty Phasing in the SE Specialist Many of the sites eased into the project by using existing SE specialists while the MHTS participant caseload grew, rather than hiring new full-time employment specialists to serve a small caseload. Until the site has nearly 20 participants, this phase-in seemed to be the most efficient strategy. However, not all sites used this gradual approach to increasing existing employment specialists’ caseloads. Furthermore, some sites had little or no experience with fee-for-service payment schedules. Lack of billing. At the time of the review, sites had billed ValueOptions for only slightly more than half of all enrolled treatment group participants, suggesting that more revenue could come to some sites if they billed fully. Most of the bills processed by ValueOptions were for only one or two faceto-face contacts each month. Need for case management. The QMPDs reported that more case management services were needed for some participants. However, some sites were hesitant to provide these services because of concern that they were not compensated (for example, case management is not a covered service under Medicare). However, the Monthly Encounter Form explicitly allowed for the billing of case management, an opportunity that many sites were missing. Improper use of the payment schedule. In general, some sites were not using the schedule as intended and could have generated more revenue had they billed for all participants and for all eligible face-to-face encounters, including case management. Unfair compensation for one versus no contacts. QMPDs reported that some sites found it unreasonable that Schedule B paid the same rate for one contact as for no contacts. Lack of compensation for non-face-to-face contacts. Many sites were providing considerable non-face-to-face services involving both employment specialists and case managers. For example, Chapter 7: Health Care and Supported Employment Financing 7-19 some participants needed assistance with job development, benefits coordination, housing, and integration of services. In addition, there were increasing requirements for treatment planning among clinical staff who did not require face-to-face contact with the participant. These required services were non-reimbursable. Low enrollment. Enrollments were lower than expected across all sites. The study paid sites on a per-participant basis, therefore, lower enrollments limited sites’ revenue-collecting opportunities. The study originally based study site contracts on an overall enrollment of 3,000 study participants (1,500 in the treatment group). As discussed in Chapter 2 (Sampling Methodology), the sample was reduced to a target of 2,200 (1,100 in the treatment group). Collectively, these obstacles meant that some sites were financially unable to deliver needed evidence-based services within the MHTS. With respect to the five specific questions articulated above, the investigators concluded the following: While the basic structure of the payment schedule remained adequate, there was a need to pay for one face-to-face contact at a higher rate than for zero face-to-face contacts. Few sites billed for participants at the top of the schedule (i.e., four or more contacts per month), but many sites had not implemented SE-related case management services and when they did, the maximum number of contacts in the schedule (four) may not have been adequate. The investigators increased face-to-face encounters from five to six encounters per month to create a proper incentive to deliver these services. Payment amounts were not adequate within the Supported Employment and Related Service Payment Schedule to cover study sites’ staff costs due to increased service demands at each site, particularly involving non-face-to-face activity. This situation suggested consideration for raising current payment rates substantially. The schedule did not properly compensate sites for the non-face-to-face SE-related contacts. Sites should receive a higher payment when there are no face-to-face contacts with a participant in a given payment month. Case management is often critical to SE success, and should be encouraged. When performed face-to-face, the staff should count it as SE-related contacts within the schedule on the Monthly Encounter Form. The rules of the payment schedule were not clear to all sites, particularly regarding the inclusion of case management on the Monthly Encounter Form. Sites required more technical assistance on billing and its structure and rationale. Based on the findings from this review, Westat decided to increase the payment amounts substantially in the schedule because of lower than expected revenues from participant billings, some Chapter 7: Health Care and Supported Employment Financing 7-20 of which was due to low enrollment and some to problems in engaging participants in face-to-face encounters. Appendix 7C compares the original and new payment rates for all three schedules A, B, and C. All payment schedule changes took effect on August 1, 2007. Though payment amounts increased dramatically, the same basic structure and reimbursement rules remained in place. The team considered moving to a grant or cost-based contracting arrangement, but the investigators decided to maintain the payment structure already in place allowing the study to continue to yield information about fee-for-service payment schedules. The specific payment schedule changes were as follows: Add a payment step to Schedules A and B to differentiate payments for no contacts and one contact. The team increased payment for no contacts to $200 in all three schedules. The team further revised the schedule to increase payments at all levels. Schedule C increased to $200 for all contact levels, as Medicaid billing for all face-to-face services occurred. Provided clarification on the rules for coverage of case management in the schedules, where some outside payment system did not reimburse case management. Identified payment schedule coverage for case management, which promotes better SE and integrated care. Provided more technical assistance to the sites about use of the schedules to pay for SE and case management. Decision 5: Modify the payment schedule to a “negotiated fixed rate” per participant month of enrollment. As mentioned, during the review that resulted in Decision 4, the team considered moving to a grant or cost-based contracting arrangement. However, the team decided to maintain the existing fee-for-service payment structure. The team was reluctant to abandon the payment system because it was closest to the usual Medicaid mechanisms for paying for SE. In large part, this reluctance was due to the expectation that the MHTS experience might influence other health insurance arrangements to consider covering SE on a fee-for-service basis. The team hoped that the reporting of specific services as a part of the request for payment would improve the accuracy of data on service utilization for analysis in the study. However, a second review conducted a few months after implementing the fee schedule described in Decision 4, confirmed that it was necessary to modify the schedule, yet again. The decision was to move to a negotiated fixed rate per participant month of enrollment regardless of the number of face-to-face contacts. In December 2007, Westat began the process of establishing a negotiated fixed rate. The feedback from the site directors was that the modified schedule (Decision 4) did not sufficiently provide revenue to fund evidence-based SE and other behavioral health services as required by the study. In addition to the Chapter 7: Health Care and Supported Employment Financing 7-21 findings summarized for the rationale for Decision 4, the following factors provided the rationale to move forward with a negotiated fixed-rate payment schedule: The provision of evidence-based SE services required ongoing supervision, planning meetings, and other support for the employment specialists. Treatment group participants required case management and other non-billable services. Enrollment rates continued to be significantly less than expected, which limited revenue but still required full-time employment specialists and case managers. There continued to be a reasonable number of unengaged participants who did not meet with the employment specialists face to face. Nonetheless, the investigators expected the employment specialists to actively attempt to engage or re-engage these participants while the site received the lowest payment schedule for these efforts. The continued financial challenges that the sites faced in providing evidence-based services, led to the site-specific negotiated fixed monthly rate per participant per month enrolled. The final negotiated flat rates were $400 (12 sites), $450 (6 sites), and $500 (4 sites) per participant per month enrolled. HBP Expenditures This section presents the expenditures associated with the implementation of the HBP. The HBP spent $14,219,241 for the 1,121 treatment participants in the MHTS, representing $6,342 per person per year. Table 7-4 shows the distribution of expenditures for all the treatment participants (n=1,121) and for 24-month treatment participants (n=982). The total expenditure for the 9826 treatment participants enrolled for 24 months was $13,719,519 (representing $6,986 per person per year), a difference of $499,722. The HBP spent $10,000 or less on about one quarter of the participants, accounting for $1,749,595 for all treatment participants and $1,335,466 for just the treatment participants enrolled for 24 months. The HBP spent between $10,001 and $15,000 per person for about half of the participants during the 24 months of their participation. That accounted for $6,898,139 for all treatment participants and $6,849,130 for just the treatment participants enrolled for 24 months. Spending for participants who had expenses between $15,001 and $20,000 accounted for $3,458,374 for all treatment beneficiaries and $3,442,355 for just the treatment 6Excluding withdrawals, administrative drops, and deaths. Chapter 7: Health Care and Supported Employment Financing 7-22 participants enrolled for 24 months. The original contract placed a $10,000 per person annual limit on expenditures, but the MHTS investigators renegotiated this limitation as an average annual expenditure. The modal expenditure was considerably less than the original $10,000 annual limit. However, 78 individuals had expenditures of $20,001 or more, accounting for $2,113,133 among all treatment participants, and for the 77 individuals of the 78 of them in this range (among those enrolled for 24 months) accounted for $2,092,568. Table 7-4. Distribution of HBP expenditures among all participants and 24-month participants All participants Total expenditures ≤ $10,000 24-month participants (n=1,121) Total Frequency expenditures 285 $1,749,595 (n=982) Total Frequency expenditures 152 $1,335,466 Difference $414,129 $10,001-$15,000 554 $6,898,139 550 $6,849,130 $49,009 $15,001-$20,000 204 $3,458,374 203 $3,442,355 $16,019 78 $2,113,133 77 $2,092,568 $20,565 1,121 $14,219,241 982 $13,719,519 $499,722 ≥ $20,001 Total Figure 7-1 illustrates the components of the total of $14,219,241 in HBP expenditures. SE was farand-away the largest area of spending for treatment participants, accounting for approximately 72 percent of expenditures or $10,227,400. Of course, most often there were no other payment sources for SE services other than the HBP. The next largest areas of expenditure were for health insurance premiums (11%), followed by behavioral health spending (7%), and medication expenditures from the debit card arrangement (8%). Spending was one percent each for employment-related work expenses and transportation, miscellaneous expenditures to participants for research participation, and general medical expenses. Chapter 7: Health Care and Supported Employment Financing 7-23 Figure 7-1. Distribution of total expenditures by type of claim including all participants (n=1,121) (based on date of service) Unclassified 1% Supported employment claims 72% Behavioral health services 7% Prescriptions EWRE and 8% transport 1% General medical <1% Premium 11% Figure 7-2 illustrates the components of the total of $13,719,518 in HBP expenditures for the 24month participants. SE was again the largest area of spending for 24-month treatment participants, accounting for approximately 71 percent of expenditures or $9,800,225. The next largest area of expenditure was again health insurance premiums (12%), followed by prescription medications (8%), and behavioral health spending (7%). Spending was one percent or less each for employment-related work expenses and transportation, miscellaneous (unclassified) expenditures to participants for research participation, and general medical services. Figures 7-3 and 7-4 illustrate the implementation expenditures of the HBP over time. Figure 7-3 shows the implementation in the life of the project and demonstrates that per person per quarter expenditures reached a steady state of approximately $600 per month by March 2008. Spending started out at approximately $400 per person per month and increased slowly over time until reaching the steady state. The rise coincides with the two changes in payment for SE described in the earlier sections of this report. Since SE spending accounts for 72 percent of the total, these changes in payment rates drive the implementation spending observations. There is one additional peak in expenditures to approximately $650 per person per month in the quarter from April – June 2008, coinciding with the renewal payments of annual insurance premiums. Chapter 7: Health Care and Supported Employment Financing 7-24 Figure 7-2. Distribution of cumulative expenditures by type of claim including all 24-month participants (n=982) (based on date of service) Unclassified 1% Behavioral health services 7% Prescriptions EWRE and 8% transport 1% General medical <1% Premium 12% Supported employment claims 71% Average HBP expenditures per person-per quarter (n=1,121) Average spending per participant Figure 7-3. Chapter 7: Health Care and Supported Employment Financing 7-25 Average HBP expenditures per participant month (n=1,121) Average payment per FTE Figure 7-4. Figure 7-4 illustrates the spending pattern for each month of individual participation in the 24 months of the intervention. It presents the monthly payments on behalf of treatment participants for their 24 months of potential tenure in MHTS. As would be expected, the amount of spending is virtually identical to the per person-per month spending shown in Figure 7-3—with a steady state of approximately $600 per person per month. The figure also shows the initial spending for insurance premiums in addition to the other spending, spread out throughout the period. Per person-per month spending is almost $700 in the first month, and the peak to approximately $650 in month 8-9 is for the renewal of the insurance premiums for the second year of participation. Discussion For the most part, expenditures within the HBP conformed to expectations about financing MHTS services. SE dominated the intervention and accounted for slightly more than 70 percent of HBP expenditures. While the demonstration initially intended to pay for SE on a fee-for-service basis, the project decided to negotiate a monthly case rate per person enrolled to cover the cost of employment specialists, as well as some case manager time and supervision by senior clinical staff. As a result, the SE payments ranged from approximately $4,800 to $6,000 per enrolled participant per year. Financing with case rates provides steady payments to sites, as reflected in Tables 7-3 and 7-4, above, once the final payment rates were negotiated. Payment for health insurance premiums accounted for slightly more than 10 percent of spending and out-of-pocket health expenditures, paid Chapter 7: Health Care and Supported Employment Financing 7-26 directly or using the debit card for medications, represented together about 15 percent of expenditures. Covering health insurance premiums was a unique element of the MHTS that made participation in the demonstration particularly advantageous for treatment group participants. Reimbursing for out-of-pocket expenses meant that treatment participants had virtually no cost to themselves for participation in the MHTS. This was one of the objectives of the HBP, and it was implemented successfully. Lessons from implementing the HBP. In the sections of this chapter prior to the discussion of expenditures, the lessons from the HBP experience are outlined as a series of critical decisions in implementation. These decisions could guide any future effort to finance a complex set of services, similar to those involved in the MHTS, as well as to reimburse participants for the health insurance premiums and out-of-pocket costs. The MHTS experience suggests that most individuals attracted to a demonstration like the MHTS have health insurance and do not need much assistance in obtaining appropriate coverage, but an important minority do need assistance. The MHTS developed a HBP that successfully matched participant needs with appropriate Medicare coverage, particularly with the then-new Part D prescription drug plans. The HBP also successfully implemented a debit card arrangement to prevent out-of-pocket costs for medications rather than retrospectively reimburse them, another benefit of MHTS participation. The HBP developed and tailored an approach to financing SE that could be emulated in any future demonstration. The experience suggests that each site will have its own special needs for financing and that a standard fee-for-service arrangement, even with complex payment rates, is unlikely to work well. Furthermore, it is important to recognize that sites that participate in such a demonstration will have upfront costs to start up service, will have costs associated with outreach to all individuals who are enrolled even if they do not use SE services, and that case management and supervision are ―hidden costs‖ of participation for any site. The design of the MHTS does not permit an exact assessment of how essential the HBP was to enrollment or to subsequent success of the overall intervention. There were many elements to the MHTS, and no single one can be isolated and assessed for its specific impact. It is clear, however, that obtaining payment for all behavioral health care costs and SE, along with payment of insurance premiums, removed some of the putative barriers to participation in active efforts to return to work for individuals with a disabling mental disorder on the SSDI rolls. The experience of the MHTS suggests that it is feasible to implement a payment system to cover these services without any outof-pocket expenditure for SE or related behavioral health services, including medications and psychosocial services. Though the negotiated payment rate per site, per participant was between $400 and $500, the expenditures for all but 78 of 1,121 participants did not exceed the original Chapter 7: Health Care and Supported Employment Financing 7-27 expected limit of $10,000 annually, and the modal payment was far less than that amount. These SErelated expenditures accounted for slightly more than 70 percent of all HBP expenditures. If an agency such as SSA wanted to implement the MHTS on a broader scale, the MHTS experience describes the expected spending of a HBP to pay for SE and also to insulate participants from other out-of-pocket expenditures associated with IPS SE, behavioral health treatment, and related efforts to return to work. Access to evidence-based services was a hallmark of the MHTS. The HBP made it possible to provide such access with virtually no out-of-pocket costs to participants. Chapter 7: Health Care and Supported Employment Financing 7-28 Chapter 8 Utilization of Services This chapter reports on the utilization of healthcare services of MHTS study participants in the treatment and control groups. Impacts of the MHTS intervention on physical and mental health service utilization levels and patterns are potentially important from a budgetary and social cost standpoint. There is a substantially reduced net cost to government if the intervention reduces overall utilization of these services, given that other public programs finance the bulk of these services. Funds for these services come primarily from Medicare and Medicaid, but also from other public programs such as care from the Department of Veterans Affairs (VA), state vocational rehabilitation, or other agencies. Service utilization data collected from treatment and control group participants during their 24months of participation in the MHTS provide the basis for constructing estimates of the impact of the treatment intervention on service use. The types and measures of utilization studied include number of emergency room (ER) visits, number of hospital admissions, number of nights spent in the hospital, number of visits to receive outpatient psychiatric emergency or crisis services, and number of visits to other mental health providers (e.g., clinics or therapists). With the intent-to-treat model, the primary focus is on unconditional mean values for these measures comparing participant service use in the treatment group to that in the control group. The primary source of data for these analyses of participant service utilization in the MHTS came from the eight post-baseline computer-assisted interviews with each study participant. The interviews included a section on health care service utilization, asking participants to recall details about ER visits, ER visits that resulted in an inpatient (overnight) hospital admission, other overnight hospital stays, psychiatric emergency or crisis (outpatient) visits, and other clinic or mental health provider visits. “Other” clinic or health care provider visits included ongoing (regularly scheduled) visits to mental health providers such as prescribers and therapists. During each postbaseline interview, participants recalled these events since the date of the last completed interview, which allowed for collection of complete data on service utilization even if the participant skipped a post-baseline interview. For instance, if participants skipped the previous interview (i.e., 3 months ago), they were asked about service utilization in the past 6 months, as opposed to the typical recall period of 3 months. Chapter 8: Utilization of Services 8-1 Dependent Variables The data collected in the computer-assisted interviews required manipulation to create useful analytic variables to assess the extent to which participants used various healthcare services during the 24-month study period. The descriptions below include each of the specific service utilization variables included in this chapter. ER visits. During each interview, participants responded to questions about visits to an emergency room (since the date of the last completed interview). Interviewers requested the participant enumerate each ER visit, and then for each ER visit, report the reason for the visit and if the visit resulted in a hospital admission. A constructed group of variables included the number of ER visits a participant made for any given month of the 24-month study period, specifically for a mental health problem, or for some other problem. Outpatient visits for psychiatric emergency or crisis. Interviewers asked participants if they received help for a psychiatric emergency or crisis from some source other than an emergency room or hospital. When appropriate, they enumerated each event and listed all providers associated with the event. In addition, interviewers requested information about the outcome of the visit. These data allowed for construction of a series of variables indicating the number of outpatient psychiatric emergency or crisis visits since the date of the last completed interview. When combined, the interviews provide a complete picture of outpatient psychiatric emergency or crisis visits over the entire length of study participation. Participants also provided an outcome for each visit, including (1) obtained a prescription or consult on medication, (2) mental health counseling, or (3) some other outcome (i.e., vocational counseling, spiritual or religious counseling, or peer support). Other mental health provider visits. After asking about ER visits, hospital admissions, and outpatient visits for psychiatric emergencies or crises, the interviewer requested information about care received from any other clinic or mental health provider (since the date of the last interview). For all clinics or providers visited, participants responded to questions about the reason for the visit (i.e., physical problem, mental health problem, alcohol problem, drug problem, or some other problem), how many times they went to the facility, and the names of all providers seen at the facility. Finally, the interviewer requested information about the outcome of each visit. Each visit included an appropriate outcome code as noted above. Hospital admissions. For each ER visit reported, participants were asked if the ER visit resulted in a hospital admission. If so, participants responded to questions about the reason for the visit (i.e., Chapter 8: Utilization of Services 8-2 physical problem, mental health problem, alcohol problem, drug problem, or some other problem) and the number of nights stayed in the hospital. In addition to hospital stays following an ER visit, participants reported any other hospital stays since the date of the last completed interview (i.e., any other hospital stays not already mentioned during the interview). Again, for each hospital stay reported, participants responded to questions about the reason for the visit and the length of the stay. These data allowed for the construction of a series of variables indicating the number of hospital admissions, the number of hospital nights, and the reason (mental health problem or for some other problem, including physical health) for the hospitalization. The resulting statistics included the mean number of hospital admissions for each participant during the study period for mental health problems and for other problems (not related to mental health), and the mean number of nights stayed in the hospital (for mental health problems or for other problems not related to mental health). The service utilization data from the interviews required a small number of minor adjustments to account for reporting anomalies or missing data. In a very small number of cases, respondents who reported an overnight stay in the hospital also reported either (a) that the number of nights stayed was zero or (b) that they did not know the number of nights they stayed. In the former case (i.e., zero nights stayed), the hospital stay was excluded from our analyses. In the latter case, the number of nights stayed was imputed either from data on other hospital stays for the same respondent or from data on average nights stayed as reported by other respondents. On rare occasions, the participant responded, “don’t know” or refused to answer some of the service utilization items. Such recorded responses were set to missing and dropped from the analysis. In other instances (also rare), the respondents indicated that a service use had occurred but when asked how many times the service was used, the recorded response was either “don’t know” or “refused.” In these cases, imputations filled in the number from data on other respondents.1 In summary, this chapter provides estimates of the MHTS effects on the following principal measures of service utilization: 1 Number of overnight hospital stays, Number of days of overnight hospital stays, Note that this imputation step applied only to numbers of visits to psychiatric crisis or emergency centers (excluding hospitals or emergency rooms) and to numbers of visits to other clinics or mental health providers. This imputation step was not relevant for numbers of overnight hospital stays or ER visits since each hospital stay was reported separately; so no participant was asked how many stays he or she had (though the participant was asked for each stay how many nights he or she stayed), and each ER visit was also reported separately. Chapter 8: Utilization of Services 8-3 Number of ER visits, Number of visits to psychiatric crisis or emergency centers (excluding hospitals or emergency rooms), and Number of visits to other clinics or mental health providers. The analysis includes estimation of MHTS effects for the full two-year followup period, for the last year of the followup period, and for the first year of the followup period. Each analysis uses the five principal service use measures (regardless of the reasons for the service use), and for three additional measures of hospital stays, hospital days, and ER visits for reasons relating to mental health problems. For respondents who did not complete the fourth quarterly followup interview, service use data for the first year were computed by allocating service use in the next subsequent completed followup evenly across the quarters of the recall period for that followup interview. Covariates The number of potentially relevant covariates in the data is very large, and even with over 2,000 observations, to include all of these potentially relevant covariates in regression models of MHTS effects could present major problems of over-fitting and multicollinearity. For these reasons, a limited number of measures and patterns of service use during the baseline period served as useful controls for: (1) multiple dimensions of baseline health status, (2) the presence of baseline health problems, and (3) the risk of future (post-baseline) problems that result in mental health or somatic service use. Controlling for the volume and pattern of baseline service use suggests that the omission of other covariates would not bias the results and would not substantially reduce the efficiency of the MHTS effect coefficient estimates. Estimation Models The rationale for the selected estimation method and models was based primarily on the distributional characteristics of the dependent variables, in particular, the fact that (1) they are all count (integer) data, and that (2) they are characterized by a large mass of observations at zero (i.e., no visits for the particular variable). A regression model developed specifically for analyzing such data is the zero-inflated negative binomial regression. Estimation of this regression model (using Stata) also provides straightforward tests of two related models, namely, a zero-inflated Poisson regression and a standard negative binomial regression (without inflation for excess zeros). Specific Chapter 8: Utilization of Services 8-4 coefficient values were obtained via maximum pseudo-likelihood estimation. All models, except those with site-specific dummy variables, used an estimated variance-covariance matrix for random errors that allowed for clustering (i.e., correlation) of such errors across individual respondents from the same site.2 In the models reported below, estimates of MHTS effects include four different specifications of covariates: (1) no covariates, (2) the baseline value of the relevant dependent variable as the only covariate, (3) baseline values for all five primary service use variables, and (4) baseline values for all five primary service use variables plus site-specific “dummy” variables. Results Descriptive statistics. Table 8-1 gives the mean values for all dependent variables and all covariates based on overall service utilization measures for the total sample and for the treatment and control groups separately. Note that the computed means used the post-stratification weights and assigned zero weights to persons designated as non-respondents (defined in Chapter 2). Differences in baseline values of service utilization between treatment and control groups were generally very small. For two of the baseline measures (psychiatric crisis or emergency center visits and days of overnight hospital stays) the mean values for the treatment group were about 20 percent below those for the control group; however, for the three remaining baseline measures, the treatment and control group means were almost equal. It is interesting to note that, after accounting for the differences in duration between the 24-month followup period and the baseline recall periods, mean rates of service use were not dramatically different between the Baseline and Followup interviews. The exceptions to this were: (1) the decline in rates of visits to psychiatric crisis or emergency centers in the followup period, (2) the modest increase in rates of overnight hospital stays in the followup period, and (3) the large increase in rates of visits to other clinics or mental health providers in the followup period. 2 When site-specific dummies were included instead of error clustering by site, coefficient estimates were obtained via maximum likelihood. Chapter 8: Utilization of Services 8-5 Table 8-1. Weighted mean values and N's for treatment and control groups and total: Overall utilization measures Variable Definition DEPENDENT VARIABLES Overnight hospital stays Overnight hospital stays – year 1 Overnight hospital stays – year 2 Days of overnight hospital stays Days of overnight hospital stays – year 1 Days of overnight hospital stays – year 2 Emergency room visits Emergency room visits – year 1 Emergency room visits – year 2 Outpatient psychiatric crisis or emergency center visits Outpatient psychiatric crisis or emergency center visits – year 1 Outpatient psychiatric crisis or emergency center visits – year 2 Other clinic or mental health provider visits Other clinic or mental health provider visits – year 1 Other clinic or mental health provider visits – year 2 BASELINE COVARIATES Overnight hospital stays – past 12 months Days of overnight hospital stays – past 12 months Emergency room visits – past 6 months Outpatient psychiatric crisis or emergency center visits – past 6 months Other clinic or mental health provider visits – past 3 months Treatment Weighted Mean N* Control Weighted Mean N* Total Weighted Mean N* 0.86 0.45 0.42 5.74 3.21 892 1,006 897 892 1,006 0.97 0.51 0.47 7.37 4.16 973 1,057 977 973 1,057 0.92 0.48 0.45 6.57 3.69 1,865 2,063 1,874 1,865 2,063 2.60 897 3.25 977 2.93 1,874 1.91 0.95 0.95 0.52 890 1,008 895 896 1.94 0.97 0.99 0.90 968 1,056 974 978 1.92 0.96 0.97 0.72 1,858 2,064 1,869 1,874 0.41 1,009 0.56 1,062 0.49 2,071 0.15 899 0.31 980 0.23 1,879 51.54 892 35.34 970 43.29 1,862 24.87 1,010 17.90 1,059 21.39 2,069 26.24 895 17.14 975 21.60 1,870 0.38 1,015 0.39 1,060 0.38 2,075 2.87 1,015 3.53 1,060 3.20 2,075 0.46 1,015 0.46 1,061 0.46 2,076 0.31 1,015 0.39 1,063 0.35 2,078 5.26 1,014 5.28 1,064 5.27 2,078 * N is the unweighted number of respondents with a weight > or = 1. It excludes respondents with weight = 0 and other respondents whose variable values were missing. Table 8-2 shows the analogous mean values for dependent variables and baseline covariates based on additional mental-health-specific service utilization measures. Treatment versus control group baseline differences are somewhat larger here, with the treatment group having lower values of Chapter 8: Utilization of Services 8-6 hospital stays (0.16 vs. 0.21; 25% lower), hospital days (1.67 vs. 2.62; 36% lower), and ER visits (0.12 vs. 0.13; 8% lower). Table 8-2. Weighted mean values and N's for treatment and control groups and total: ER and inpatient mental health utilization measures Variable Definition DEPENDENT VARIABLES Mental health overnight hospital stays Mental health overnight hospital stays – year 1 Mental health overnight hospital stays – year 2 Days of mental health overnight hospital stays Days of mental health overnight hospital stays – year 1 Days of mental health overnight hospital stays – year 2 Mental health emergency room visits Mental health emergency room visits – year 1 Mental health emergency room visits – year 2 BASELINE COVARIATES Mental health overnight hospital stays – past 12 months Days of mental health overnight hospital stays – past 12 months Mental health emergency room visits – past 6 months Treatment Weighted Mean N* Control Weighted Mean N* Total Weighted Mean N* 0.40 892 0.45 973 0.42 1,865 0.21 1,006 0.26 1,057 0.23 2,063 0.19 897 0.20 977 0.20 1,874 3.86 892 4.92 973 4.40 1,865 2.19 1,006 2.98 1,057 2.58 2,063 1.70 897 2.01 977 1.86 1,874 0.39 890 0.52 968 0.45 1,858 0.20 1,008 0.27 1,056 0.23 2,064 0.19 895 0.26 974 0.22 1,869 0.16 1,015 0.21 1,060 0.19 2,075 1.67 1,015 2.62 1,060 2.14 2,075 0.12 1,015 0.13 1,061 0.12 2,076 * N is the unweighted number of respondents with a weight > or = 1. It excludes respondents with weight = 0 and other respondents whose variable values were missing. Regression results for estimated MHTS effects. Table 8-3 shows the estimated treatment effects, based on the zero-inflated negative binomial regression coefficients, for the MHTS treatment dummy variable. Note that these treatment effects are the average differences between predicted values of the dependent variables (for values of zero vs. one for the treatment dummy) where the average is computed over the entire weighted study sample of participants. Chapter 8: Utilization of Services 8-7 Table 8-3. Estimated average marginal treatment effects Dependent Variable A. Emergency room visits None Baseline emergency room visits All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies** B. Outpatient psychiatric crisis visits None Baseline emergency room visits All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies** C. Other clinic or mental health provider visits None Baseline emergency room visits All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies** D. Overnight stays None Baseline emergency room visits All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies** E. Overnight days None Baseline emergency room visits All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies** Year 1 Est. Effect p-value Year 2 Est. Effect p-value 2-Year Estimate Est. Effect p-value -0.02 -0.07 -0.06 -0.06 0.788 0.326 0.404 0.379 -0.04 -0.06 -0.06 -0.05 0.464 0.247 0.248 0.430 -0.04 -0.06 -0.06 -0.05 0.464 0.247 0.248 0.430 -0.15 -0.15 -0.14 -0.21 0.164 0.146 0.203 0.166 -0.17 -0.24 -0.25 † 0.311 0.043 0.096 † -0.39 -0.45 -0.48 -0.74 0.019 0.004 0.037 0.006 6.97 8.88 9.09 8.40 <0.001 <0.001 <0.001 <0.001 9.10 9.88 9.99 8.79 0.004 0.003 0.002 <0.001 16.20 19.57 19.87 17.56 0.001 0.001 <0.001 <0.001 -0.06 -0.05 -0.05 -0.05 0.130 0.194 0.200 0.268 -0.05 -0.06 -0.07 -0.09 0.233 0.161 0.13 0.038 -0.12 -0.13 -0.14 -0.16 0.092 0.041 0.035 0.024 -0.95 -0.71 -0.78 -0.87 0.016 0.064 0.049 0.061 -0.65 -0.75 -0.87 -1.14 0.064 0.082 0.035 0.012 -1.63 -1.63 -1.75 -2.29 0.008 0.025 0.006 0.003 *Baseline value of outpatient psychiatric crisis visits is excluded from the model for Year 1. **Other clinic or mental health visits baseline and outpatient psychiatric crisis visits baseline are excluded for Year 1; outpatient psychiatric crisis visits baseline are excluded for Year 2. †Not applicable. Maximum likelihood estimation did not converge. The following analytic reports for each of the overall service utilization measures include estimates of MHTS effects for (1) the full 24-month period, (2) the first 12 months post-baseline, and (3) the second 12 months post-baseline. Four different specifications for the regression covariates comprise the estimates, including: 1. Regressions using only the MHTS treatment dummy and no baseline covariates, 2. Regressions using the MHTS dummy and the baseline value for the dependent variable, Chapter 8: Utilization of Services 8-8 3. Regressions that add the baseline values for the four additional overall utilization measures to specification (2), and 4. Regressions that use specification (3) but also include site-specific variables. Reported p-values for MHTS effects based on specifications (1), (2), and (3) also take into account clustering of error terms among observations for participants in the same site; specification (4) does not account for site-specific clustering since it includes site-specific variables. Looking first at the estimates of MHTS effect on use of ER visits (Table 8-3, panel A), only weak and clearly non-significant negative effects were observed. While the inclusion of baseline ER visits tends to increase the absolute size of the estimated impact (relative to specification (1) with no covariates), results are in general qualitatively and quantitatively similar across all four specifications. In contrast, estimated MHTS effects on numbers of psychiatric crisis or emergency center visits (Table 8-3, panel B) for the full 24-month followup are strongly negative in all specifications. It is also interesting to note that the magnitudes of these effects (approximately 0.5 visits less) are large relative to the mean observed rates of 0.72 visits for the 24-month followup. Analogous results for the first 12 months of followup show MHTS effects that are consistently negative, clearly not as significant as the 24-month results, and a decline in magnitude that is more than proportionate to the difference in followup periods. Comparing results for the second 12 months of followup with the first 12 months reveals somewhat stronger and more significant negative MHTS effects in the former period, especially when baseline covariates are included in the model. This trend suggests that the impact of the intervention was increasing over time. Note also that for all three followup periods, as in the case of the ER visit regressions, controlling for baseline covariates and including site-specific variables increased the magnitude and significance of the estimated MHTS effects. Panel C of Table 8-3 shows estimates of MHTS effects on visits to other clinics and outpatient mental health providers. The results are in sharp contrast to the negative and sometimes insignificant estimates for ER visits, and the negative and significant impacts on psychiatric emergency visits. Instead, this panel reveals relatively large and highly significant positive impacts of the MHTS on numbers of visits. It is reasonable to attribute most of these positive impacts to the design of the treatment intervention itself. The increases in insurance coverage for behavioral health and the provision of systematic medication management services presumably encouraged treatment group participants to make ongoing use of behavioral health services. (This is also consistent with a decline in the need for psychiatric emergency care.) Individual Placement and Support (IPS) supported employment services in the MHTS were most often provided by a program that was Chapter 8: Utilization of Services 8-9 integrated with a provider organization that also provided a range of non-vocational behavioral health services. It is also interesting to note that the magnitude of the MHTS effect on these visits did not substantially increase from the first 12 months of followup to the second 12 months. Table 8-3, panels D and E, report estimates of MHTS average effects on overnight hospital use. Regressions on numbers of overnight stays show consistently negative and significant (p-value < 0.1) estimates of MHTS effects for the full 24-month followup period. Estimates for the first 12-month and second 12-month followup periods are also negative but are not significant except for the latter period when site-specific dummies are included. There is only weak evidence of a tendency for the negative MHTS effect to increase over time. In addition, the pattern of larger negative impacts when baseline covariates and site-specific dummies are controlled for is once again observed, but only for the 24-month and second 12-month followup periods. Table 8-4 provides estimates of MHTS effects on the additional mental-health-specific utilization measures (for ER visits, overnight stays, and days in overnight stays). As was true for the estimates for all ER visits, all overnight stays, and days in all overnight stays, MHTS estimated impacts are all negative; the main differences in Table 8-4 compared to the results in Table 8-3 pertain to p-values and magnitudes of the estimated effects. For mental health overnight stays, estimated impacts are smaller than for all stays (in Table 8-4) and are uniformly not significant. For mental health overnight days, impact estimates are also smaller and less significant than the corresponding estimates for all days in Table 8-3, but the impact estimates for two years and for the first year of the followup has p-values that in several cases are significant or approach significance. Finally, while the impact estimates in Table 8-3 for all ER visits were clearly and uniformly not significant, the corresponding estimates in Table 8-4 are all clearly significant and generally larger in magnitude. Evidence of negative MHTS effects on days of inpatient care in panel B of Table 8-4 is fairly strong and consistent. Clearly, the tendency for the size of this effect increases with inclusion of baseline covariates and site dummies but the pattern of increasingly large MHTS effects over time is not clear. The magnitude of the estimates is also worth noting; they imply that the MHTS effects (approximately 0.8 to 1 day less of use per year) are just slightly less than one-third the size of the mean 12-month use figures for baseline and for the first 12-month followup. Chapter 8: Utilization of Services 8-10 Table 8-4. Estimated average marginal treatment effects for emergency room visits and overnight hospital use for mental health problems Dependent Variable A. Overnight mental health stays None Baseline overnight mental health stays All 5 baseline overall use measures* All 5 baseline overall use measures, site dummies* B. Overnight mental health days None Baseline overnight mental health stays All 5 baseline overall use measures * All 5 baseline overall use measures, site dummies* C. Emergency room visits for mental health problems None Baseline overnight mental health stays All 5 baseline overall use measures * All 5 baseline overall use measures, site dummies* Year 1 Est. Effect p-value Year 2 Est. Effect p-value 2-Year Estimate Est. Effect p-value -0.04 -0.02 0.242 0.475 -0.01 -0.003 0.728 0.913 -0.05 -0.03 0.316 0.546 -0.01 -0.01 0.610 0.841 -0.01 † 0.656 † -0.03 -0.01 0.593 0.789 -0.79 -0.44 0.037 0.171 -0.31 -0.19 0.360 0.570 -1.05 -0.60 0.101 0.301 -0.59 -0.56 0.103 0.182 -0.44 † 0.222 † -0.93 -1.33 0.144 0.045 -0.07 -0.06 0.048 0.026 -0.07 -0.08 0.032 0.004 -0.13 -0.12 0.026 0.006 -0.07 -0.07 0.004 0.014 -0.08 † 0.004 † -0.15 † <0.001 † *Baseline value of the dependent variable is also included as a covariate. †Not applicable. Maximum likelihood estimation did not converge. Results of specification tests. Further analyses examined evidence of two different types of specification tests. First, evidence testing the null hypothesis that the correct specification is actually Poisson rather than negative binomial is provided by the estimated confidence intervals on the overdispersion parameter (“alpha”) provided in the regression output (using Stata). In all cases of the analyses, the results permit rejection of the Poisson specification. Second, the Vuong test provides a test for the null hypothesis that the correct specification is the negative binomial model rather than the zero-inflated negative binomial model. Since an algorithm for performing this test is not available when there is clustering of errors, it was applied only for those models including site dummies to control for such clustering (rather than using a “vce [cluster]” command in Stata). Once again, the results of the test consistently rejected the null hypothesis of no zero inflation, thereby supporting use of the zero-inflated model. Examining attrition bias: Comparing late dropouts versus non-dropouts. A more limited analytic approach to assessing the effects of attrition bias compared (a) participants with non- Chapter 8: Utilization of Services 8-11 missing values for the dependent variables in both the first and second 12-month followup periods to (b) participants with non-missing values for the first 12-month followup period but missing values for the second followup period. The number of participants in the second group, the late dropouts, was approximately 200 (ranging from 196 to 207) and varied slightly depending on the examination of whichever (of the five) main dependent variable. The analysis replicated previous regressions on the values of the five main dependent variables for the first 12-month followup but included a dummy variable for the late dropouts and an interaction of this dummy variable with the MHTS treatment effects. The results (in Table 8-5) indicate significantly positive average marginal effects for the late dropout dummy for two of the five dependent variables (ER visits and psychiatric crisis or emergency visits) but only one (positive) significant interaction average marginal effect (for hospital stays). Results for the main MHTS effects were very similar to those reported for “Year 1” in Table 8-3.3 Table 8-5. Regression results for average main treatment effects with late dropout dummy and dummy interaction with treatment* Dependent Variable Emergency room visits – year 1** Outpatient psychiatric crisis or emergency center visits – year 1 Overnight stays – year 1 Overnight days – year 1 Other clinic or mental health provider visits – year 1 MHTS Est. Effect p-value -0.028 0.732 Late Dropout Est. Effect p-value 1.535 <0.001 Dummy Interaction Est. Effect p-value -0.193 0.392 -0.247 0.114 -0.528 0.293 0.959 0.140 -0.054 -0.847 0.213 0.057 0.046 0.078 0.618 0.946 3.579 0.624 0.002 0.678 9.017 <0.001 -6.790 0.004 2.712 0.503 *Included covariates are the five baseline use measures except as noted below. **Covariates exclude baseline other clinic or mental health provider visits and outpatient psychiatric crisis or emergency center visits due to convergence problems. Discussion The findings indicate that the treatment intervention had significant impacts in reducing inpatient hospital use, and in reducing crisis-oriented mental health outpatient services such as ER visits for mental health reasons and visits to other psychiatric emergency providers. In contrast, estimated 3 It should be noted that the test of bias from late dropouts does raise some concerns about endogeneity, since dropping out after four quarters of followup may be due to unobserved severity that is correlated with the (unobservable) levels of services use in Year 2. Other strategies for comparing late dropouts and non-dropouts, such as using service use data from Medicare and Medicaid claims for the followup period, would provide a firmer basis for assessing late dropout bias. Chapter 8: Utilization of Services 8-12 impacts on hospital stays for mental health problems were negative but not significant. The significance of the negative estimated MHTS impacts on hospital inpatient days for mental health treatment varied with the choice of included covariates; they were significant or nearly significant in some models but not in others. None of the negative estimated impacts on ER visits for all reasons ever approached significance. Estimated MHTS effects on visits to other clinics or mental health providers were strongly positive. Analyses of data on types of services provided by individual providers at these visits show positive impacts of the MHTS on all types of services. This view is consistent with the nature of the treatment intervention, which included vocational services using the IPS model as well as systematic medication management and increased insurance coverage for all types of behavioral health service. This pattern of results (i.e., negative impacts on hospital use and crisis-oriented mental health services) provides support to the notion that the costs of implementing the MHTS intervention were offset (at least in part) by reductions in costs for other services from non-MHTS providers. In some cases, these cost reductions could have been substantial. For example, the average reduction in hospital days, due to the MHTS, of approximately 0.9 days per year implies cost savings of $1,800 per year per person.4 Sustaining these annual cost savings over a longer period of time would, of course, increase their magnitude. While the MHTS data do not allow for projecting savings over a longer time period, it is interesting to note that the magnitude of the MHTS negative impacts on days of inpatient hospital use appear to be roughly the same in the first and second years of the MHTS followup period. 4 This calculation assumes a cost per day of hospital care of $2,000, which is approximately equal to the 2009 figure reported by the Agency for Healthcare Research and Quality. Chapter 8: Utilization of Services 8-13 Chapter 9 Study Limitations, Key Findings, and Policy Implications The Social Security Administration (SSA) initiated the Mental Health Treatment Study (MHTS) to assess the impact of access to evidence-based mental health treatment and supported employment (SE) services on the employment, health, and quality of life of Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or an affective disorder. The study design included input from an expert panel to enhance the validity and utility of the results. Westat and its subcontractors accepted the design, and with a few negotiated modifications, implemented the design. The earlier chapters described the elements of the study with each contributing key pieces of information to the overall mosaic. They provide important details needed to interpret the study results and understand the implications for new policy initiatives. This chapter revisits the original research questions posed by SSA, identifies study limitations, presents a summary of the key findings, and consolidates the findings and information from the previous chapters into a set of policy implications. No specific policies are identified nor are any recommended for change. Instead, the policy implications section intends to give a deeper understanding of what the study has to say about return to work of SSDI beneficiaries with schizophrenia or an affective disorder. Motivation for the MHTS came from a variety of sources. However, what appears to be paramount was the availability of evidence-based treatments and employment services that held the promise of support for the return to work efforts of SSDI beneficiaries with psychiatric impairments. In 2005, SSA released a request for proposals (RFP) to conduct the MHTS. The RFP described the following three research questions, which the study would answer: 1. To what extent does delivering appropriate mental health treatment and employment supports lead to better employment, health status, and quality of life among SSDI beneficiaries with schizophrenia or an affective disorder? 2. What programmatic disincentives exist that create barriers for title II beneficiaries with schizophrenia or an affective disorder to return to work? 3. What specific programmatic changes would support the efforts of SSDI beneficiaries with schizophrenia or an affective disorder to sustain competitive employment? Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-1 It is important to remember these questions as they drove the development of the study design, as well as the operational and analysis plans. These questions now serve to focus communication of the findings and policy implications. The following sections attempt to answer these three questions, beginning with a brief overview of the study limitations. Study Limitations SSA has a keen interest in external validity and the extent to which the MHTS findings generalize to the specific population of SSDI beneficiaries with schizophrenia or an affective disorder, and to the existing system of community mental health services in the United States. SSA is concerned with internal validity of the study and the extent to which the findings can support inferences about the effectiveness of the intervention, and the likelihood that changes in policy or programmatic guidance would produce similar results. A brief discussion of each concern follows. External Validity There are two main topics of concern about external validity. One topic is selection of the study sites and their representativeness of existing community mental health centers in the United States. The other is the selection of SSDI beneficiaries and their representativeness of the SSDI rolls. This latter question concerns the extent to which beneficiaries who participated in the MHTS represent the universe of beneficiaries with schizophrenia or an affective disorder in the SSDI program. Site representativeness. Community mental health centers recruited for the study were already familiar with Individual Placement and Support (IPS), and capable of having the complex interventions up and running within six months of the contract start date. Further, they were willing to establish new services to meet the needs of the MHTS treatment intervention (rather than simply redirecting services already in place). As a result, the study sites were not typical of mental health service organizations across the country, but neither were they atypical, such as the academic community mental health centers offering highly specialized services. None of the study sites were affiliated with a major university hospital or clinic. At least two of the sites did not provide mental health services directly. Instead, they negotiated provision of the services with other agencies. Even with the intent of selecting sites that could implement the intervention quickly, as described below, there was variability in implementation and ability to achieve fidelity in the delivery of evidence- Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-2 based services. This variability may indicate that the sites were not so carefully handpicked as to threaten generalizability of the findings. The study sites did not come from a randomly selected list of community mental health service providers across all regions of the United States. Such a selection process might have increased the external validity of the study, but the MHTS investigative team concluded (based on considerable research evidence) that it was not feasible, given the short startup period permitted by SSA. In addition, there was fear that this process would result in an unknown number of sites failing completely, as not all sites would have the capability to provide access to the designated treatment intervention services. As it was, two sites ended recruitment (in essence, failed) within the first year of the study. The purposive site selection process meant that the study did not have as much regional diversity as could have been achieved with stratified random site selection. In fact, finding sites with any form of IPS SE in some states was a serious challenge. Nevertheless, there were study sites in almost every region of the country, except for the southwest, and the beneficiary sample did achieve a diverse set of demographic characteristics in terms of race, education, and marital status. For example, 40 percent of the sample comprised individuals who were non-white or Hispanic. Beneficiary representativeness. It is important to understand who was interested in participating in the MHTS, who eventually agreed to join the study, and who remained engaged throughout. Several selection points in the recruitment process introduced potential sources of bias and threats to external validity. The investigators made an effort to understand these potential sources of bias and correct for them where possible. Study site staff received training to implement extensive standardized recruitment and enrollment procedures. The procedures described how each beneficiary should hear about the study and how he or she should be offered the opportunity to participate in it. Thus, to the extent possible, all beneficiaries had equal opportunity to learn about the study and to enroll in it if desired. The standardized procedures were important for documenting and estimating the “take-up rate” in the study. This information about MHTS participation is useful for understanding potential enrollment in a future program. Internal Validity As discussed in Chapter 2, the MHTS employed random assignment of beneficiaries within sites to the treatment or control group. This procedure alone increased the internal validity of the observations and the ability to make inferences about the effectiveness of the treatment intervention. One key characteristic of the study design was that the treatment intervention was Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-3 complex with many moving parts. Primarily by design, and for cost reasons, it was not possible to determine which component(s) of the treatment intervention might predict outcomes. The MHTS informs SSA about the effects of access to the multi-component intervention as a whole. Thus, future replications expecting to achieve the same results will need to implement all of the components. The intent-to-treat (ITT) design feature focuses the interpretation of the results on the entire sample of study participants—including all beneficiaries who had access to the treatment intervention services without regard to the extent to which they might have participated in those specific services. This feature of the design had potential to reduce the perceived impact of the intervention. As discussed below, these potential limitations may bias the MHTS results in one direction or another. The study recruited sites that were likely to be better than the average community mental health service organization but not as good as the early adopters of IPS who participated in the first demonstrations of the effectiveness of IPS SE or systematic medication management (SMM). Selecting participants from the SSDI rolls and not from individuals already in mental health services and vocational rehabilitation (who were the subjects in previous studies of IPS SE or SMM) introduced other potential influences on the results. SSDI beneficiaries had more work experience than many of the participants in earlier studies of IPS SE, thus it was reasonable to expect that both the control group and the treatment group would demonstrate better employment outcomes than those observed in earlier studies. Yet recruiting individuals with less connection to the mental health service system might mean worse outcomes than expected from prior studies of IPS SE. In reality, the extremely high percentages of SSDI beneficiaries in the study with serious physical health problems and medical conditions (such as morbid obesity) made clear what may be the most important difference between the SSDI population and populations comprising previous demonstrations of IPS and SMM effectiveness. Due primarily to their co-occurring physical health conditions, SSDI beneficiaries in the MHTS presented far more complex and serious disabilities to the study sites when compared to subjects with severe mental illness in previous demonstrations, or other clients at participating study sites. This theme became an ongoing concern throughout as the study sites attempted to work with the SSDI population recruited and randomized to the treatment group. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-4 Key Findings Findings presented below represent analyses conducted on data collected from all sources during the study. Enrollment and participation. The findings from this chapter focus on the extent of participation in the MHTS from the SSDI rolls, and reasons why beneficiaries chose not to enroll in the study. The key findings are presented below. 1. Based on the sample of potential enrollees, the findings suggest that SSA could expect 14 percent of the SSDI beneficiaries with schizophrenia or an affective disorder might enroll in an MHTS-like program. That percentage could jump to 26 percent if SSA were to target a specific subpopulation of beneficiaries. 2. The strongest predictors of enrollment were those related to prior work activity reported in SSA’s administrative records, including beneficiaries who report earnings within the past 6 months (prior to enrollment), had assigned their Ticket (in the Ticket to Work program), or had a Trial Work End Date within the past 3 years. 3. Beneficiaries enrolling in the study were younger, had been on the SSDI rolls a shorter period, were only on SSDI (as opposed to concurrent SSI beneficiaries), and did not have a RepPayee. 4. The most commonly recorded single reason for beneficiaries not enrolling in the study was general disinterest (37%). However, nearly 40 percent of all beneficiaries with a recorded reason, indicated work-related or physical health issues were reasons they did not enroll in the study. 5. The randomization procedure was successful, as no statistical differences existed between the treatment and control groups at baseline on key outcome measures, including recent employment (in the 2 years prior to enrollment), physical health status, and mental health status. Outcomes. The following findings represent analyses of interview data on the primary outcomes of interest in the study, including employment rate and earnings; mental health status; physical health status; and quality of life. For purposes of brevity, no secondary outcome analyses appear here. 6. The 24-month employment rate for the treatment group was 61 percent compared to 40 percent for the control group. The difference between these percentages was significant (p-value < 0.001). 7. Analyses of average earnings (in the past month averaged over eight interviews) show that treatment group participants earned significantly more than did control group participants. One analysis concerns comparisons between the unconditional means of Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-5 the treatment and control groups ($148 vs. $97, p-value < 0.001). A second analysis concerns the comparison between conditional means (i.e., including only those participants with non-zero earnings) of the treatment and control groups ($251 vs. $228, p-value < 0.001). A third analysis concerns the percentages of participants in the treatment and control groups with earnings (59% vs. 43%, p-value < 0.001). 8. Eight percent of the study participants (both treatment and control) showed average earnings over the 24-month study period that exceeded the current level of substantial gainful activity (SGA). Participants in the treatment group did not experience an increase in work that SSA considers SGA when compared to participants in the control group. Neither did participants in the treatment group experience a reduction in benefit payments when compared to participants in the control group. 9. Measured at baseline and again at study exit, the treatment group showed a significant improvement over the control group in mental health status (p-value < 0.001). 10. Measured at baseline and again at study exit, both the treatment and control groups showed a slight decline in physical health status, however, the changes between the groups were not statistically significant (p-value = 0.924). 11. Measured at baseline and again at study exit, the treatment group showed a significant improvement over the control group in quality of life (p-value < 0.001). Implementation of SE. The key findings are: 12. Eighty percent or more of the study sites achieved a high level of IPS program implementation (i.e., met the documented standard for high fidelity). This high level of implementation persisted across the entire study period. 13. The level of unengagement in employment services among treatment group participants was relatively low overall (~10%). Implementation of SMM. The key findings are: 14. Concordance between the SSA diagnostic category and the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID) diagnosis was greater than 80 percent. 15. More than 87 percent of treatment group participants had at least one physical health condition, and 69 percent had two or more. More than half of all participants in the treatment group had a body mass index (BMI) in the obese range. 16. The quality of SMM varied considerably across study sites. The off-site location of many prescribers affected this measure. For purposes of conducting the MHTS, the decision to allow participants to remain with their outside prescribers was reasonable. However, the goal of MHTS was to deliver an integrated package of services to participants. Having off-site prescribers presented great difficulties in integrating the SMM components with other treatment components. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-6 Health care and SE financing. The Health Benefits Plan (HBP) paid for behavioral health and SE services, out-of-pocket expenses, and health insurance premiums. The goal was to leave the treatment group participant with no costs associated with services provided with the intervention package. The key findings are: 17. Spending through the HBP averaged $6,986 per study participant per year for those treatment group participants who completed the study. Spending was less ($6,342) for the overall treatment group population (which includes those participating for less than 24 months). These figures do not include cost of services related to the Nurse Care Coordinator (NCC) role. 18. The overall spending distribution included over 70 percent for SE services; 11 percent for health insurance premiums; 8 percent for medication prescriptions; 7 percent for behavioral health services; and less than 3 percent for essential work-related expenses, transportation, and other miscellaneous expenses. SE services were the only services completely paid by the study. Utilization of services. Self-report interviews also asked about hospital admissions, emergency room (ER) visits, outpatient psychiatric emergency or crisis visits, and regular provider visits. The key findings associated with service use include the following: 19. The treatment intervention had significant positive impacts in reducing inpatient hospital use (for both admissions and number of days) and outpatient psychiatric emergency or crisis visits. 20. As expected with the intervention package, treatment group participants showed significantly higher use of regularly scheduled clinic or mental health visits. 21. The average reduction in hospital days was 0.9 days per year, which translated into approximately $1,800 per year per person. Sustaining these costs over longer periods would increase their magnitude. These findings answer SSA’s first question whether the treatment intervention leads to better employment, health, and quality of life. The answer is, it did. The findings presented above are clear and unambiguous about the impact of the treatment intervention on SSDI beneficiaries with schizophrenia or an affective disorder who express an interest in working. The treatment group attained significantly better outcomes compared to the control group, and they did it repeatedly with nearly every outcome variable. With no demonstrated differences at baseline between the treatment and control groups on key outcome measures, at study exit the treatment group demonstrated statistically significant improvements or differences over the control group. Rarely did the control group outperform the treatment group, and when it did, it was an isolated finding. In Chapter 4, these differences between the treatment and control groups also held in many, if not most, Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-7 secondary analyses of the employment-related outcomes (such as total months employed, average hours per week at main job, etc.), as well as the analyses of relevant subgroups (such as diagnosis, gender, age, and education). The findings also point to the extent to which the treatment intervention would affect the target SSDI population. The enrollment and participation findings make clear that a significant segment of the SSDI population with schizophrenia or an affective disorder would enroll in an MHTS-like program. These results offer SSA a significant opportunity to develop policies to assist these beneficiaries to return to work. What effects or influences these findings may have on the SSDI rolls, the Trust Fund, or cost offsets for providing MHTS-like services are unclear and outside the purview of this report. However, it is abundantly clear that significant improvement in employment, mental health, and quality of life is possible with improvements in access to the package of services delivered through the treatment intervention. Policy Implications The findings from the study have implications for policies related to the SSDI program, as well as for employment policies of other federal agencies as these pertain to individuals with severe mental illness. It is clear from the research that SSA cannot solve all of the problems of providing access to all of the services tested in the MHTS with changes to its SSDI program policies only. The services implemented in the study are relevant and should be of interest to many federal agencies that serve individuals with serious mental illness, including schizophrenia or an affective disorder. The purpose of the following comments is to inform policy discussions at SSA and other relevant agencies. The intent of the discussion is to provide answers to the remaining SSA research questions about programmatic disincentives and programmatic changes needed to support SSDI beneficiaries return to work. The remaining sections offer a question-and-answer format to make the information more accessible. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-8 Is there significant interest among SSDI beneficiaries with schizophrenia or an affective disorder to access an MHTS-like program? The study findings presented above, and further detailed in Chapter 3, make it clear that a substantial percentage of beneficiaries with schizophrenia or an affective disorder would consider participating in an MHTS-like program. The numbers may approach a quarter or more of the targeted beneficiaries on the SSDI rolls. Today there are nearly 2.25 million SSDI beneficiaries with a psychiatric disorder. The largest portion of them includes beneficiaries with schizophrenia or an affective disorder. The MHTS finding that 14 percent of beneficiaries could be expected to take advantage of MHTS-like services (Chapter 3) suggests that perhaps 306,000 SSDI beneficiaries could be affected. Taking the finding further, the data suggest that more than 60 percent of those who go on to participate in such services would find a job within a 24-month period. While this is hypothetical, the study findings point out that there is a significant number of SSDI beneficiaries who want to work and can work if given access to an MHTS-like intervention package. What essential services or features of services are required to achieve MHTS-like results? The intervention package in the MHTS was rich in services and benefits. The following list and descriptions provide details of the services and benefits, as well as the features considered essential to achieve the results attained in this study: Essential services. The primary services included evidence-based SE (the IPS model), SMM, other behavioral health (OBH) services (such as therapy, counseling, substance abuse counseling, etc.), benefits counseling, and modest monetary support to pay for inaccessible services. Community mental health centers. The usual mental health care setting that can offer both the comprehensive range of services needed to treat mental illness and provide integrated vocational services is the community mental health center. The experience of the MHTS suggests that the community mental health center is ideal for delivery of these essential services. As noted above, community mental health centers hosted the study in every site with two exceptions. One of the two exceptions was a vocational services program with a negotiated contract for mental health services with the county mental health board. The other site was a housing program for the homeless and mentally ill. This program offered employment services and negotiated mental health services in the Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-9 community. All of the community mental health centers offered a wide range of mental health services, case management, and employment services. With only one exception, all of the community mental health centers had experience providing IPS SE services. However, not all of them had the documentation to support that they were performing at high fidelity at the time the study began. Need for a care coordinator. Use of a care coordinator to facilitate SMM was a key feature of the mental health services provided in the study. The SMM program in the MHTS used the NCC to facilitate and promote prescriber use of evidence-based guidelines and recommendations for medication management of severe and persistent mental illnesses. As noted in Chapter 6, the central goal of SMM was to provide prescribers with pertinent, updated information about the patient and medication options at the time of each medication visit with the prescriber. In the ideal SMM model, the NCC and prescriber function in tandem with the NCC first assessing the patient’s status with regard to medication use, medication effects, and medication-related symptom outcomes. The prescriber received this information, along with any recommendations for change based on medication management guidelines and expert consultation. The prescriber evaluated the information and the patient, and made final judgments about medication management. The exact division of labor between NCC and prescriber was determined at the local level. In clinic and office settings, implementation of SMM requires systemic changes in personnel roles and in the recording of information obtained from patients. The study findings note that 90.4 percent of participants were engaged in SMM at some point in the study. Further, the study found that 53 percent of on-site prescribers engaged the NCC compared to only 11 percent of off-site prescribers. This latter finding suggests that on-site prescribers, while not a requirement, are more likely to be fully engaged in the SMM process. Out-of-pocket expenses. Payment for out-of-pocket mental health and essential work-related expenses is an important feature that improved access to services and jobs for study participants. Unfortunately, it is not a simple decision for some SSDI beneficiaries (as described below in the section on disincentives to return to work). The decision to make a co-payment is often complex and difficult for these beneficiaries. In the MHTS, payments for out-of-pocket expenses for participants who were trying to return to work created opportunities that, otherwise, would not have existed. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-10 How much did the study spend on each treatment group participant to achieve the study results? Understanding expenditures associated with the intervention helps to put into perspective the impact of the treatment intervention on the study outcomes. The HBP provided all payments for participants in the study. Eight principles guided the development of health care and SE financing for individual participants and related payment decisions. These principles included the following: 1. Provide all participants a comprehensive health care coverage package. 2. Ensure that participants receive needed care. 3. Ensure that participants receive evidence-based treatments. 4. Cover SE in full. 5. Cover all other approved MHTS behavioral health care expenditures in full. 6. Utilize scarce resources wisely. 7. Minimize participant up-front out-of-pocket expenditures. 8. Collect research data on encounters and expenditures. Overall, participants spent an average of $6,986 per year. Of course, these expenditures do not reflect all of the participant costs for services needed during participation in the study. NCC services, for example, were paid through the study, and not as part of the HBP. Medicare or Medicaid paid for many of the mental health and general health services required by participants. However, it is not clear how important those costs are to a discussion of how to provide MHTS-like services to the broader population of participants with psychiatric impairments reflected on the rolls. That is, except for participants in the 24-month waiting period for Medicare. It is noteworthy that fewer hospital stays and fewer psychiatric emergency or crisis visits may offset some of the HBP expenditures. Further exploration of the MHTS data may clarify the potential for health care. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-11 What programmatic disincentives exist that create barriers for title II participants with schizophrenia or an affective disorder to return to work? The design of the MHTS did not permit any direct tests of potential programmatic barriers to work. Given the many components of the intervention package, it was not possible to parse them into discrete testable components. Each major element of the intervention package was a solution to a known barrier to return to work. The design contract for the MHTS (Aron, Burt, & Wittenburg, 2005) specifically built the study to test the impact of removing these barriers—as a whole. Thus, it was not possible to isolate the individual contributions of any particular feature of the intervention to the study results, including access to comprehensive health insurance support, access to SE services, SMM, or waiver of the medical Continuing Disability Review. Three general barriers describe the kinds of problems SSDI beneficiaries faced in gaining access to needed health care and employment programs and supports. They include access to health care, access to evidence-based employment services, and complex co-morbid physical conditions. They are not disincentives in the sense of the well-known disincentives associated with fear of loss of benefits—or more likely fear of not being able to get back on benefits once they are lost. Instead, they represent barriers, or hurdles, that participants cannot negotiate alone without some sort of programmatic intervention. Below we provide a discussion of each set of barriers as we have come to understand them. Under current conditions, SSDI beneficiaries have insufficient access to health care programs, services, and treatments. The study revealed many instances where insufficient access to programs, services, and treatments was a problem. Expenditures through the HBP made it clear that obtaining payment for all behavioral health care costs, along with payment of insurance premiums, removed some of the putative barriers to participation in active efforts to return to work for individuals in the treatment group. These payments represented less than 10 percent of the total HBP expenditures, suggesting that across all participants in the treatment group, gaining access to needed health care amounted to approximately $53 per month. A remarkably low cost for improving access to needed health care. The vast majority of participants in the treatment group at the time of study entry had comprehensive health care coverage through Medicare or Medicaid, while a small fraction did not. Approximately 7 percent (74) participants required enrollment in one or more Parts (A, B, or D) of Medicare (See Chapter 7). Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-12 Reports from participants in the treatment group suggested that in the past, the cost of insurance co-pays (for health care visits, prescription medications, etc.) kept some participants from seeking treatment. In general, it is clear that such decisions to seek treatment were complicated and highly individual. For example, a participant related a story that prior to the study she chose not to refill a prescription for her psychiatric medication because it was too expensive given her financial condition. However, upon further discussion, it became clear that the decision was more complicated than that. She felt the medication was not effective (had unpleasant side effects and was not sufficiently reducing symptoms) and, therefore, was not worth the cost of the co-pay to refill it. Many previous trips to the doctor to find a better medication proved expensive and ineffective. It was no longer worth the effort required to seek a solution. It was cheaper and easier to do nothing. In sum, the participant decided that refilling the prescription was not worth the investment of scarce dollars. This example reveals the importance and roles of various MHTS elements, such as covering co-pays, assisting with travel to treatment visits when indicated, and SMM in the overall scheme of return to work. No single MHTS benefit could have solved the problem presented by this participant. One ongoing concern throughout the study was the role that high-cost psychiatric medications played in finding effective treatments for participants in the treatment group. Several situations were problematic. One clear problem was the effect of the so-called “donut hole” on medication use. NCCs reported that prior to the MHTS when beneficiaries came up against the “donut hole,” they would stop refilling their prescription(s) for psychiatric (and other) medication(s) because the high costs did not fit within their budget. Another problem reported by some beneficiaries was that prior to the study, prescribers were reluctant to prescribe some (potentially more effective) medications due to the high cost to the patient. A third problem was finding a Part D plan that covered the preferred medications. During the study, this always presented a challenge, and required more efforts from the Westat Insurance Planner (WIP) than any other insurance issue. A change in the Part D prescription medication plan required waiting until the open enrollment period. This situation limited prescribers in their options to find an optimal treatment for their patient. Fortunately, the MHTS paid for a prescription medication whenever a participant’s plan did not cover it. Then, during the open enrollment period, the WIP would work with the NCC and the participant to find a more acceptable plan. The findings would not likely be the same without the support of the study to pay for medications when they were not covered, or high co-pays. The study paid special attention to receipt of mental health case management services, because case management is a key for achieving adequate mental health treatment (Rapp & Goscha, 2004). The fact that only 54 percent of participants received mental health case management, and of these, 28 Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-13 percent received their case management services off-site, is far below the expected rate in high fidelity IPS programs serving clients with severe mental illness. Several factors may account for this finding. First, some participants did not need case management services. Unfortunately, study attempts to measure need were not successful, so there was no way of reliably estimating need. Second, some participants received case management assistance from the NCC. The study did not include a direct measure of NCC case management activity, so no data exists on the extent of NCC case management. Nevertheless, given the competing responsibilities, NCCs must have played a limited role in case management—both in number of participants assisted and in the range of case management activities performed. The Quality Management Program Director site reports reinforce the conclusion that at least at some study sites, case management services were insufficient. Since case management is a vital part of an integrated IPS team and was often unavailable, it may be a missing link in the engagement process of participants receiving IPS services. This may have been more of a factor in this study than in past IPS interventions, because many of the participants were not engaged in community mental health services when they entered the study. In usual IPS practice, clients are typically first engaged in mental health treatment and then referred to IPS. As has been noted many times, randomization into the treatment intervention occurred without participants’ first receiving mental health treatment at the study site. Some were not receiving any treatment, and many were being treated elsewhere. Each of the problems presented above create distinct barriers to effective treatment, better functioning and, ultimately, to employment. No one barrier by itself is particularly troublesome. However, as a group, they represent formidable obstacles to return to work. Under current conditions, SSDI beneficiaries lack access to evidence-based SE services in community mental health centers. Analysis of the health services utilization data reveals that the treatment group used more vocational services than the control group. This was expected, given the intervention package included the provision of evidence-based IPS SE. These services, which emphasized competitive employment, clearly contributed to the positive employment results attained in the study. However, as discussed elsewhere in this report, community mental health centers do not generally provide SE services. Payment for these services is limited to Medicaid. In fact, even Medicaid does not cover these services in every state. A recent report to the Assistant Secretary for Planning and Evaluation by Karakus, Frey, Goldman, Fields, & Drake (2011) describes both the difficulty of funding these services in general, and more specifically, the need for a federal financing strategy to provide them in community mental health centers. The report points out that Medicaid funding, as designed, is inadequate for this purpose. The result is most community mental Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-14 health centers cannot afford to employ the staff (employment specialists, supervisors, etc.) necessary to provide SE services. Many SSDI beneficiaries with schizophrenia or an affective disorder have complex comorbid physical conditions that impede efforts to return to work. Physical impairments created by a wide variety of health conditions were serious deterrents to employment efforts made by study participants, NCCs, and employment specialists. As reported above, 87 percent of the participants in the treatment group had at least one co-morbid physical health condition, 69 percent had two or more, and more than half were obese. Frequent comments on conference calls throughout the study noted that the SSDI beneficiary population in the study was much less healthy and had more problematic health conditions than did the populations normally served by the mental health centers. These comments corroborated the data collected from treatment group participants and formal documentation of why some beneficiaries do not return to work (discussed in Chapters 4, 5, and 6). In fact, good physical health (as measured by the SF-12) frequently came up as a predictor of employment (obtained employment, steady worker, or number of months employed), suggesting that there was enough variability in physical health to pick it up in multivariate analyses. The implications of this fact are twofold. First, this situation reinforces the need for improved health care access. As noted above, these beneficiaries sometimes weigh the cost of seeking health care with the expectation of a good or poor outcome. As already mentioned, sometimes the decision is made to forego refilling a prescription or doctor visit, when it is not likely in the best interest of physical health. Second, working on physical health, mental health, and employment all at once likely slows the return to work process. This was the case for a number of beneficiaries in the study. Chapter 9: Study Limitations, Key Findings, and Policy Implications 9-15 References Aron, L., Burt, M., & Wittenburg, D. (2005). Recommendations to the Social Security Administration on the design of the Mental Health Treatment Study (MHTS). Washington, DC: Urban Institute. Becker, D.R., Bebout, R.R., & Drake, R.E. (1998). Job preferences of people with severe mental illness: A replication. Psychiatric Rehabilitation Journal, 22(1), 46-50. Becker, D.R., Bond, G.R., McCarthy, D., Thompson, D., Xie, H., McHugo, G.J., & Drake, R.E. (2001). Converting day treatment centers to supported employment programs in Rhode Island. Psychiatric Services, 52(3), 351-357. Becker, D.R. & Drake, R.E. (2003). A working life for people with severe mental illness. New York: Oxford University Press. Becker, D.R., Drake, R.E., Bond, G.R., Nawaz, S., Haslett, W.R., & Martinez, R.A. (2011). A mental health learning collaborative on supported employment. Psychiatric Services, 62, 704-706. Becker, D.R., Drake, R.E., Farabaugh, A., & Bond, G.R. (1996). Job preferences of clients with severe psychiatric disorders participating in supported employment programs. Psychiatric Services, 47, 1223-1226. 22(1), 24-29. Becker, D.R., Swanson, S., Bond, G.R., & Merrens, M.R. (2008). Evidence-based supported employment fidelity review manual. Lebanon, NH: Dartmouth Psychiatric Research Center. Becker, D.R., Whitley, R., Bailey, E.L., & Drake, R.E. (2007). Long-term employment outcomes of supported employment for people with severe mental illness. Psychiatric Services, 58, 922-928. Becker, D.R., Xie, H., McHugo, G.J., Halliday, J., & Martinez, R.A. (2006). What predicts supported employment program outcomes? Community Mental Health Journal, 42, 303-313. Bedell, J.R., Draving, D., Parrish, A., Gervey, R., & Guastadisegni, P. (1998). A description and comparison of experiences of people with mental disorders in supported employment and paid prevocational training. Psychiatric Rehabilitation Journal, 21(3), 279-283. Berkowitz, M. (2003). The Ticket to Work Program: The complicated evolution of a simple idea. In K. Rupp & S.H. Bell (Eds). Vocational rehabilitation: Will provider incentives work for Ticket to Work. Washington, DC: Urban Institute. Bond, G.R. (2007). Modest implementation efforts, modest fidelity, and modest outcomes. Psychiatric Services, 58, 334. Bond, G.R. (2004). Supported employment: Evidence for an evidence-based practice. Psychiatric Rehabilitation Journal, 27, 345-359. Bond, G.R. (1998). Principles of the Individual Placement and Support model: Empirical support. Psychiatric Rehabilitation Journal, 22(1), 11-23. References-1 Bond, G.R., Becker, D.R., & Drake, R.E. (2011). Measurement of fidelity of implementation of evidence-based practices: Case example of the IPS Fidelity Scale. Clinical Psychology: Science and Practice, 18, 126-141. Bond, G.R., Becker, D.R., Drake, R.E., & Vogler, K.M. (1997). A fidelity scale for the Individual Placement and Support model of supported employment. Rehabilitation Counseling Bulletin, 40, 265-284. Bond, G.R., & Drake, R.E. (2008). Predictors of competitive employment among patients with schizophrenia. Current Opinion in Psychiatry, 21, 362-369. Bond, G.R., Drake, R.E., & Becker, D.R. (2008). An update on randomized controlled trials of evidence-based supported employment. Psychiatric Rehabilitation Journal, 31, 280-290. Bond, G.R., Drake, R.E., McHugo, G.J., Rapp, C.A., & Whitley, R. (2009). Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice, 19, 569-581. Bond, G.R., Drake, R.E., Mueser, K.T., & Becker, D.R. (1997). An update on supported employment for people with severe mental illness. Psychiatric Services, 48, 335-346. Bond, G.R., Evans, L., Salyers, M.P., Williams, J., & Kim, H.K. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75-87. Bond, G.R., Xie, H., & Drake, R.E. (2007). Can SSDI and SSI beneficiaries with mental illness benefit from evidence-based supported employment? Psychiatric Services, 58, 1412-1420. Brekke, J.S. (1988). What do we really know about community support programs? Strategies for better monitoring. Hospital and Community Psychiatry, 39, 946-952. Burns, T., Catty, J., White, S., Becker, T., Koletsi, M., Fioritti, A., Rössler, W., Tomov, T., van Busschbach, J., Wiersma, D., & Lauber, C. (2009). The impact of supported employment and working on clinical and social functioning: Results of an international study of Individual Placement and Support. Schizophrenia Bulletin, 35, 949-958. Campbell, K., Bond, G.R., Drake, R.E., McHugo, G.J., & Xie, H. (2010). Client predictors of employment outcomes in high-fidelity supported employment: A regression analysis. Journal of Nervous and Mental Disease, 198, 556-563. Campbell, K., Bond, G.R., Gervey, R., Pascaris, A., Tice, S., & Revell, G. (2007). Does type of provider organization affect fidelity to evidence-based supported employment? Journal of Vocational Rehabilitation, 27, 3-11. Carpenter, V., & Perkins, D.V. (1997, November 8). Supported employment as a transition from state hospital to community. Paper presented at the American Public Health Association, Indianapolis, IN. Conover, W.J., & Iman, R.L. (1981). Rank transformation as a bridge between parametric and nonparametric statistics. The American Statistician, 35, 125-129. References-2 Cook, J.A., Lehman, A.F., Drake, R., McFarlane, W.R., Gold, P.B., Leff, H. S., Blyler, C., Toprac, M.G., Razzano, L.A., Burke-Miller, J.K., Blankertz, L., Shafer, M., Pickett-Schenk, S.A., & Grey, D.D. (2005). Integration of psychiatric and vocational services: A multisite randomized, controlled trial of supported employment. American Journal of Psychiatry, 162, 1948-1956. Corrigan, P.W., Mueser, K.T., Bond, G.R., Drake, R.E., & Solomon, P. (2008). Principles and practice of psychiatric rehabilitation: An empirical approach. New York: Guilford Press. Cradock, J., Young, A.S., et al. (2001). The accuracy of medical record documentation in schizophrenia. Journal of Behavioral Health Services and Research, 28, 456-465. Crane-Ross, D., Roth, D., & Lauber, B.G. (2000). Consumers' and case managers' perceptions of mental health and community support service needs. Community Mental Health Journal, 36, 161-178. Danziger, S., Frank, R., & Meara, E. (2009). Mental illness, work and income support programs. American Journal of Psychiatry, 166(4), 398-404. Dickinson, D., Ramsey, M., & Gold, J. (2007). Overlooking the obvious: A meta-analytic comparison of digit symbol coding tasks and other cognitive measures in schizophrenia. Archives of General Psychiatry, 64, 532-542. Drake, R.E., Becker, D.R., Bond, G.R., & Mueser, K.T. (2003). A process analysis of integrated and non-integrated approaches to supported employment. Journal of Vocational Rehabilitation, 18, 51-58. Drake, R.E., Becker, D.R., Goldman, H.H., & Martinez, R.A. (2006). Best practices: The Johnson & Johnson—Dartmouth Community Mental Health Program: Disseminating evidence-based practice. Psychiatric Services, 57, 302-304. Drake, R.E., & Bond, G.R. (2010). Implementing integrated mental health and substance abuse services. Journal of Dual Diagnosis, 6, 251-262. Drake, R.E., & Essock, S.M. (2009). The science-to-service gap in real-world schizophrenia treatment: The 95% problem. Schizophrenia Bulletin, 35, 677-678. Drake, R.E., Essock, S.M., & Bond, G.R. (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35, 704-713. Drake, R.E., McHugo, G.J., Bebout, R.R., Becker, D.R., Harris, M., Bond, G.R., & Quimby, E. (1999). A randomized clinical trial of supported employment for inner-city patients with severe mental illness. Archives of General Psychiatry, 56, 627-633. Fabian, E.S. (1989). Supported employment for severely psychiatrically disabled persons: A descriptive study. Psychosocial Rehabilitation Journal, 13(2), 53-60. References-3 Frounfelker, R.L., Wilkniss, S., Bond, G.R., Drake, R.E., & Devitt, T.S. (in press). Interest, access, and outcomes of supported employment services for clients with co-occurring disorders. Psychiatric Services. Gervey, R., & Kowal, H. (1994, May). A description of a model for placing youth and young adults with psychiatric disabilities in competitive employment. Paper presented at the International Association of Psychosocial Rehabilitation Services Conference, Albuquerque, NM. Gold, P.B., Meisler, N., Santos, A.B., Carnemolla, M.A., Williams, O.H., & Kelleher, J. (2006). Randomized trial of supported employment integrated with assertive community treatment for rural adults with severe mental illness. Schizophrenia Bulletin, 32, 378-395. Grant, B.F., Stinson, F.S., Dawson, D.A, Chou, S.P., Dufour, M.C., Compton, W., et al. (2004). Prevalence and co-occurrence of substance use disorders and iindependent mood and anxiety disorders: Results from the National Epidemiologic Survey on Alcohol and Related Conditions. Archives of General Psychiatry, 61, 807-816. Grob, G.N., & Goldman, H.H. (2006). The dilemma of federal mental health policy: Radical reform or incremental change? New Brunswick, NJ: Rutgers University Press. Hales, R.E, Yudofsky, S.C., Gabbard, G.O. (Eds.). (2010). Essentials of psychiatry (3rd ed). Arlington, VA: American Psychiatric Publishing, Inc. Hennessey, J.C., & Dykacz, J.M. (1989). Projected outcomes and length of time in disability insurance program. Social Security Bulletin, 52(9), 2-41. Heslin, M., Howard, L., Leese, M., McCrone, P., Rice, C., Jarrett , M., Spokes, T., Huxley, P., & Thornicroft, G. (2011). Randomized controlled trial of supported employment in England: 2 year follow-up of the Supported Work and Needs Study. World Psychiatry, 10, 132-137. Hollis S. & Campbell F. (1999). What is meant by intention to treat analysis? Survey of published randomised controlled trials. British Medical Journal, 319, 670-674. Huff, S.W., Rapp, C.A., & Campbell, S.R. (2008). Every day is not always Jell-O: A qualitative study of factors affecting job tenure. Psychiatric Rehabilitation Journal, 31, 211-218. Karakus M., Frey W., Goldman H., Fields S., & Drake R.E. (2010). Federal financing of supported employment & customized employment for people with mental illness. (Prepared under contract to the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services). Rockville, MD: Westat. Katon, W., Von Korff, M., Lin, E., Walker, E., Simon, G.E., Bush, T., et al. (1995). Collaborative management to achieve treatment guidelines: Impact on depression in primary care. Journal of the American Medical Association, 273, 1026-1031. Kessler, R.C. (2004). The epidemiology of dual diagnosis. Biological Psychiatry, 56, 730-737. References-4 Krenzke, T., & Judkins, D. (2008). Filling in blanks: Some guesses are better than others – Illustrating the impact of covariate selection when imputing complex survey items. Chance, 21(3), 7-13. Kreyenbuhl, J., Buchanan, R.W., Dickerson, F.B., & Dixon, L.B. (2010). The Schizophrenia Patient Outcomes Research Team (PORT): Updated treatment recommendations 2009. Schizophrenia Bulletin, 36, 94-103. Kreyenbuhl, J., Nosse, I.R., & Dixon, L.B. (2009). Disengagement from mental health treatment among individuals with schizophrenia and strategies for facilitating connections to care: A review of the literature. Schizophrenia Bulletin, 35, 696-703. Lachin, J. (2000). Biostatistical methods: The assessment of relative risks (Wiley Series in Probability and Statistics). New York: Wiley. Latimer, E., Lecomte, T., Becker, D., Drake, R., Duclos, I., Piat, M., Lahaie, N., St. Pierre, M.S., Therrien, C., & Xie, H. (2006). Generalisability of the individual placement and support model of supported employment: Results of a Canadian randomised controlled trial. British Journal of Psychiatry, 189, 65-73. Leff, H.S., Cook, J.A., Gold, P.B., Toprac, M., Blyler, C., Goldberg, R.W., McFarlane, W., Shafer, M., Allen, I.E., Camacho-Gonsalves, T., & Raab, B. (2005). Effects of job development and job support on competitive employment of persons with severe mental illness. Psychiatric Services, 56, 1237-1244. Lehman, A.F. (1988). A Quality of Life Interview for the chronically mentally ill. Evaluation and Program Planning, 11, 51-62. Lehman, A.F., Goldberg, R.W., Dixon, L.B., McNary, S., Postrado, L., Hackman, A., & McDonnell, K. (2002). Improving employment outcomes for persons with severe mental illness. Archives of General Psychiatry, 59, 165-172. Lehman, A.F., Steinwachs, D.M., & the co-investigators of the PORT. (1998). Project patterns of usual care for schizophrenia: Initial results from the Schizophrenia Patient Outcomes Research Team client survey. Schizophrenia Bulletin, 24, 11-19. Livermore, G.A., Goodman, N., & Wright, D. (2007). Social Security disability beneficiaries: Characteristics, work activity, and use of services. Journal of Vocational Rehabilitation, 27, 85-93. Loveland, D., Driscoll, H., & Boyle, M. (2007). Enhancing supported employment services for individuals with a serious mental illness: A review of the literature. Journal of Vocational Rehabilitation, 27, 177-189. MacDonald-Wilson, K.L., Revell, W.G., Nguyen, N.H., & Peterson, M.E. (1991). Supported employment outcomes for people with psychiatric disability: A comparative analysis. Journal of Vocational Rehabilitation, 1, 30-44. References-5 MacDonald-Wilson, K.L., Rogers, E.S., Ellison, M.L., & Lyass, A. (2003). A study of the Social Security work incentives and their relation to perceived barriers to work among persons with psychiatric disability. Rehabilitation Psychology, 48, 301-309. Marker, D.A., Judkins, D.R., & Winglee, M. (2001). Imputation in large-scale complex surveys. In B. Groves et al. (Eds.), Survey nonresponse (pp. 329-341). New York, NY: John Wiley. McAlpine, D.D., & Warner, L. (2000). Barriers to employment among persons with mental illness: A review of the literature. New Brunswick, NJ: Center for Research on the Organization and Financing of Care for the Severely Mentally Ill Institute for Health, Health Care Policy and Aging Research, Rutgers, the State University. McHugo, G.J., Drake, R.E., Whitley, R., Bond, G.R., Campbell, K., Rapp, C.A., Goldman, H.H., Lutz, W.J., & Finnerty, M.T. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services, 58, 1279-1284. McQuilken, M., Zahniser, J.H., Novak, J., Starks, R.D., Olmos, A., & Bond, G.R. (2003). The Work Project Survey: Consumer perspectives on work. Journal of Vocational Rehabilitation, 18, 59-68. Mechanic, D., Bilder, S., & McAlpine, D.D. (2002). Employing persons with serious mental illness. Health Affairs, 21(5), 242-253. Menear, M., Reinharz, D., Corbière, M., Houle, N., Lanctôt, N., Goering, P., Goldner, E.M., Kirsh, B., & Lecomte, T. (2011). Organizational analysis of Canadian supported employment programs for people with psychiatric disabilities. Social Science and Medicine, 72, 1028-1035. Michie, S., Fixsen, D., Grimshaw, J.M., & Eccles, M.P. (2009). Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implementation Science, 4(40), doi:10.1186/1748-5908-1184-1140. Miller, A.L., Crismon, M.L, Rush, A.J. Chiles, J., Kashner, T.M., Toprac, M. et al. (2004). The Texas Medication Algorithm Project: Clinical results for schizophrenia. Schizophrenia Bulletin, 30(3), 627-647. Mowbray, C.T., McCrohan, N.M., & Bybee, D. (1995). Integrating vocational services into case management: Implementation analysis of Project WINS. Journal of Vocational Rehabilitation, 5, 89-102. Mueser, K.T., Becker, D.R., & Wolfe, R.S. (2001). Supported employment, job preferences, job tenure and satisfaction. Journal of Mental Health, 10, 411-417. Mueser, K.T., Becker, D.R., Torrey, W.C., Xie, H., Bond, G.R., Drake, R.E., & Dain, B.J. (1997). Work and nonvocational domains of functioning in persons with severe mental illness: A longitudinal analysis. Journal of Nervous and Mental Disease, 185, 419-426. Mueser, K.T., Clark, R.E., Haines, M., Drake, R.E., McHugo, G.J., Bond, G.R., Essock, S.M., Becker, D.R., Wolfe, R., & Swain, K. (2004). The Hartford study of supported employment for persons with severe mental illness. Journal of Consulting and Clinical Psychology, 72, 479-490. References-6 Mueser, K.T., Salyers, M.P., & Mueser, P.R. (2001). A prospective analysis of work in schizophrenia. Schizophrenia Bulletin, 27, 281-296. Newcomb, C., Payne, S., & Waid, M.D. (2003). What do we know about disability beneficiaries’ work and use of work incentives prior to Ticket? In K. Rupp & S.H. Bell (Eds). Vocational rehabilitation: Will provide incentives work for Ticket to Work. Washington, DC: Urban Institute. O’Day, B., & Killeen, M. (2002). Does U.S. federal policy support employment and recovery for people with psychiatric disabilities? Behavioral Sciences and the Law, 20, 559-583. O'Brien, P.C. (1983). The appropriateness of analysis of variance and multiple-comparison procedures. Biometrics, 39, 787-788. Parks, J., Svendsen, D., Singer, P., & Foti, M. (Eds.) (2006). Morbidity and mortality in people with serious mental illness. Technical report. Alexandria, VA: National Association of State Mental Health Program Directors (NASMHPD) Medical Directors Council. Pincus, H.A., Kennedy, C., Simmens, S.J., Goldman, H.H., Sirovatka, P., & Sharfstein, S.S. (1991). Evaluating disabilities based on mental impairment: APA’s formative evaluation. American Journal of Psychiatry, 148, 1037-1043. Porter, A., Smith, J., Payette, A., Tremblay, T., & Burt, R (2009). SSDI $1-for-2 benefit offset pilot demonstration: Vermont pilot final report. Contractor's report to the Social Security Administration (Contract Number SS00-05-60009). Waterbury, VT: Vermont Division of Vocational Rehabilitation. Proschan, M.A., & Waclawiw, M.A. (2000). Practical guidelines for multiplicity adjustment in clinical trials. Controlled Clinical Trials, 21(6), 527-539. Rapp, C.A., & Goscha, R.J. (2004). The principles of effective case management of mental health services. Psychiatric Rehabilitation Journal, 27, 319-333. Rapp, C.A., Goscha, R.J., & Carlson, L.S. (2010). Evidence-based practice implementation in Kansas. Community Mental Health Journal, 46, 461-465. Resnick, S.G., & Bond, G.R. (2001). The Indiana Job Satisfaction Scale: Job satisfaction in vocational rehabilitation for people with severe mental illness. Psychiatric Rehabilitation Journal, 25, 12-19. Rinaldi, M., Killackey, E., Smith, J., Shepherd, G., Singh, S., & Craig, T. (2010). First episode psychosis and employment: A review. International Review of Psychiatry, 22, 148-162. Rothman, K.J. (1990). No adjustments are needed for multiple comparisons. Epidemiology, 1, 43-46. Rush A.J., Crismon, M.L., Kashner, T.M., Toprac, M.G., Carmody, T.J., Trivedi, M.H., et al. (2003). Texas Medication Algorithm Project, Phase 3 (TMAP-3): Rationale and study design. Journal of Clinical Psychiatry, 64, 357-369. References-7 Ryan, C.S., Sherman, P.S., & Bogart, L.M. (1997). Patterns of services and consumer outcome in an intensive case management program. Journal of Consulting and Clinical Psychology, 65(3), 485-493. Salyers, M.P., Becker, D.R., Drake, R.E., Torrey, W.C., & Wyzik, P.F. (2004). Ten-year followup of clients in a supported employment program. Psychiatric Services, 55, 302-308. SAMHSA. (2009). 2009 CMHS uniform reporting system http://www.samhsa.gov/dataoutcomes/urs/urs2009.aspx. output tables. Available at: Schimmel, J., Stapleton, D.C., Song, J. How common is "parking" among Social Security Disability Insurance (SSDI) beneficiaries? Evidence from the 1999 Change in the Level of Substantial Gainful Activity (SGA). Working Paper 220. University of Michigan, Michigan Retirement Research Center. http://www.mrrc.isr.umich.edu/publications/Papers/pdf/wp220.pdf. Schochet, P.Z. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations. NCEE 2008-4018. U.S. Department Of Education. Shafer, M.S., & Huang, H.W. (1995). The utilization of survival analysis to evaluate supported employment services. Journal of Vocational Rehabilitation, 5, 103-113. Social Security Administration. (2002-2009). Annual statistical report on SSDI. Retrieved 11/29/10. http://www.socialsecurity.gov/policy/docs/statcomps/. Social Security Administration. (2008). Disability evaluation under Social Security. http://www.ssa.gov/disability/professionals/bluebook/. Retrieved November 29, 2009. Social Security Administration. (2009). 2009 RED BOOK: A summary guide to employment supports for individuals with disabilities under the Social Security Disability Insurance and Supplemental Security Income Programs. Stapleton, D., Liu, S., Phelps, D., & Prenovitz, S. (2010). Work activity and use of employment supports under the original Ticket to Work regulations: Longitudinal statistics for new Social Security Disability Insurance beneficiaries, final report. Washington DC: Mathematica Policy Research. Suppes, T., Rush, A.J., Dennehy, E.B., Crismon, M.L., Kashner, T.M., Toprax, M.G., Carmody, T.J., et. al. (2003). Texas Medication Algorithms Project, phase 3 (TMAP-3): Clinical results for patients with a history of mania. Journal of Clinical Psychiatry, 64, 370-382. Taylor, A.C., Bond, G.R., Tsai, J., Howard, P.B., El-Mallakh, P., Finnerty, M., et al. (2009). Scales to evaluate quality of medication management: Development and psychometric properties. Administration and Policy in Mental Health and Mental Health Services Research, 36, 247-254. Thompson, J.R. (1998). Comment on “Multiple comparisons and related issues in the interpretation of epidemiologic data” (1995V142 p904-908). American Journal of Epidemiology, 147, 801-806. Thompson, J.R. (1998). Reply to comments on “Multiple comparisons and related issues in the interpretation of epidemiologic data”' (1998V147 p807-814). American Journal of Epidemiology, 147, 815-815. References-8 Torrey, W.C., Bond, G.R., McHugo, G.J., & Swain, K. (2011). Evidence-based practice implementation in community mental health settings: The relative importance of key domains of implementation activity. Administration and Policy in Mental Health and Mental Health Services Research, DOI: 10.1007/s10488-011-0357-9. Trivedi, M.H., Rush, A.J., Crismon, M.L., Kashner, T.M., Toprac, M.H., Carmody, T.J., et. al. (2004). Clinical results for patients with major depressive disorder in the Texas Medication Algorithm Project. Archives of General Psychiatry, 61, 669-680. Twamley, E.W., Narvaez, J.M., Becker, D.R., Bartels, S.J., & Jeste, D.V. (2008). Supported employment for middle-aged and older people with schizophrenia. American Journal of Psychiatric Rehabilitation, 11, 76-89. U.S. Bureau of Labor Statistics, Department of Labor. (2011). Unpublished tables from the Current Population Survey, 2010 annual averages. U.S. Bureau of Labor Statistics, Department of Labor (2010). Persons with a disability: Labor force characteristics – 2009. http://www.bls.gov/news.release/disabl.nr0.htm. Released August 25, 2010. U.S. General Accountability Office. (2002, January). SGA levels appear to affect the work behavior of relatively few beneficiaries, but more data needed. Report to the Committee on Finance, U.S. Senate and the Committee on Ways and Means, House of Representatives. Uttaro, T., & Mechanic, D. (1994). The NAMI consumer survey analysis of unmet needs. Hospital and Community Psychiatry, 45, 372-374. Van Dongen, C.J. (1996). Quality of life and self-esteem in working and nonworking persons with mental illness. Community Mental Health Journal, 32, 535-548. Velligan, D.I., Diamond, P., Wang, M., Glahn, D.C., Castillo, D., Bendle, S., et al. (2007). Relationships among subjective and objective measures of adherence to oral antipsychotic medications. Psychiatric Services, 58, 1187-1192. Ware, J.E., Jr., Kosinski, M., Turner-Bowker, D.M., & Gandek, B. (2002). How to score version 2 of the SF-12® Health Survey (with a supplement documenting version 1). Lincoln, RI: QualityMetric Inc. Ware, J., Kosinkski, M., and Keller, S. (1996). SF-12: A 12-item short-form health survey. Medical Care, 34(3), 220-233. Zimmerman, D.W. An extension of the Rank Transformation Concept (1992). Experimental Education, 61(1), 73-80. References-9 Journal of Glossary of Terms Term Definition Administrative Drop Treatment group participants who were removed from the study for failing to complete the General Medical Exam (GME), failure of the GME, or inability to meet an initial study requirement for participation in the treatment condition (e.g., moved out of the country). Affective Disorder A psychiatric disorder characterized by a disturbance of mood, accompanied by a full or partial manic or depressive syndrome (i.e., bipolar or depression). Benefits Counseling Personalized information about the impact of work (earnings) on Social Security benefits, other entitlements, or income-based assistance program benefits provided to a beneficiary by a trained benefits counselor. Case Management A clinical advocacy service designed in collaboration with the participant and treatment staff that provides for coordination of and access to a wide range of care needed to improve physical, social, and mental functioning. Catchment Area Refers to the specific postal zip codes included in the communities, cities, or counties that the study site typically served; used to define the geographic area from which SSDI beneficiaries were recruited into the study. Collaborative Care Model A model of care in which a trained non-physician clinician works in tandem with prescribers. Competitive Employment Jobs that pay at least a minimum wage are "owned" by the employer (rather than a mental health center or rehabilitation agency), and are not set aside positions for people with disabilities. Comprehensive Coverage In the MHTS, refers to health insurance coverage equivalent to Medicare Parts A, B, and D. Continuing Disability Review (CDR) A periodic review (by the Social Security Administration) of complete current information about a beneficiary’s condition to determine continued eligibility to receive SSDI and/or SSI benefits. Glossary-1 Term Definition Co-occurring Condition (Psychiatric) An active drug and/or alcohol diagnosis with simultaneous presentation of a thought or affective disorder. Often referred to as a dual diagnosis. Creditable Coverage Health coverage you have had in the past (such as a group health plan [including COBRA continuation coverage], an HMO, an individual health insurance policy, Medicare or Medicaid), and this prior coverage was not interrupted by a significant break in coverage. The time period of this prior coverage must be applied toward any pre-existing condition exclusion imposed by a new health plan. Proof of creditable coverage may be shown by a certificate of creditable coverage or by other documents showing an individual had health coverage, such as a health insurance ID card. Donut Hole A Medicare Part D annual coverage gap that occurs when a beneficiary has spent a certain amount of money for covered drugs through his or her plan. During this coverage gap, the beneficiary must pay all costs out-of-pocket for his or her prescription medications (up to a limit). The yearly deductible, coinsurance or copayments, and payments in the coverage gap all count toward this out-of-pocket limit. The limit doesn’t include the drug plan’s premium. Dual-eligible A beneficiary who is eligible for disability benefits under both the Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) programs. Employment Specialist Specially trained staff who provide employment services (such as engagement, vocational profiling, job search, job development, and individualized follow-up support) to individuals with serious mental illness wanting to return to work. Engagement Beneficiary level of participation in the treatment intervention components of the MHTS. Components measured include supported employment (SE), other behavioral health (OBH) services, and systematic medication management (SMM). Essential Work-Related Expense (WRE) An item or expense that a provider team considered to be essential to the participant’s return to work, needed immediately, and without which work could not commence. Glossary-2 Term Definition Evidenced-Based Practice A practice which, based on experimental research findings, or based on expert consensus opinion about available evidence, is expected to produce a specific clinical outcome in the tested population. Fidelity (Model) Adherence to the principles of an evidence-based program model. • Program Level Fidelity In the MHTS, the extent to which a study site provided supported employment services consistent with the Individual Placement and Support model. For the MHTS, this was referred to as “Study Site Fidelity.” • Individual Level Fidelity The extent at which an individual received SE and other behavioral health and related services that were expected to be received. For the MHTS, this was referred to as “Beneficiary-level Fidelity.” Health Benefits Plan (HBP) Refers to the benefits package provided to MHTS treatment group participants, consisting of the provision of healthcare (including health insurance coverage) and supported employment services. Individual Placement and Support Model (IPS) An evidence-based model of supported employment that adheres to core principles, specifically designed to serve individuals with serious mental illness. Intent to Treat (ITT) A fundamental principle in conducting randomized trials where all randomized participants are analyzed according to the group to which they were randomly assigned regardless of extent of adherence to the treatment intervention, level of engagement, or whether they withdrew from the study. Medicaid Federal and state (jointly-funded) health insurance program for low-income and needy individuals. It covers certain children, some or all of the aged, blind, and/or disabled individuals in states eligible to receive federally assisted income maintenance payments. Title XIX of the Social Security Act authorizes Medicaid. The law gives the states options regarding eligibility under Medicaid. Medicaid is operated under rules set forth by the U.S. Centers for Medicare and Medicaid Services. Glossary-3 Term Definition Medicare A federal health insurance program for people age 65 or older and people with disabilities under age 65 who meet specific requirements. SSDI beneficiaries are eligible for Medicare after a 24-month waiting period. Medicare is operated by the U.S. Centers for Medicare and Medicaid Services. Title XVIII of the Social Security Act authorizes Medicare. Medicare consists of hospital insurance (Part A), supplemental medical insurance (Part B), Medicare Advantage (Part C), and voluntary prescription drug benefits (Part D). Other Behavioral Health (OBH) and Related Services Refers to the range of services that were provided to MHTS treatment group participants in addition to SMM and SE services for the treatment, rehabilitation, or habilitation of mental disorders. These services include therapy, case management, general medical care, social skills training, financial assistance, housing assistance, substance abuse treatment, family counseling, and legal assistance. Psychiatric Impairment Illness based on signs or symptoms of distress and/or dysfunction relating to thought, emotion, or behavior. Psychosocial Interventions Treatment aimed at improving an individual’s interactions with others and the environment. Examples include Assertive Community Treatment, Cognitive Behavioral Therapy, Dual Diagnosis Treatment of Mental Illness and a Substance Abuse Disorder, Family Education and Support, illness selfmanagement, and supported employment. Quality Assurance (QA) For the MHTS, QA is a periodic individual chart review to assess the quality of systematic medication management by the NCC and the prescriber. SMM QA consists of 23 items scored from 1 to 5. Not all items are applicable to all charts. The summary score is the sum of all scores divided by the number of scored items multiplied by five. Quality Management (QM) For the MHTS, QM is a regularly scheduled individual chart review to a) monitor needs, b) monitor services received, and c) identify participants in need of improved supported employment, behavioral health, and/or systematic medication management services. Glossary-4 Term Definition Repayee A person, agency, organization or institution selected by SSA to manage a beneficiary’s funds when it has been determined that the beneficiary is unable to do so on his/her own. Schizophrenia A chronic mental disorder characterized by disintegration of thought processes and of emotional responsiveness. It most commonly manifests as auditory hallucinations, paranoid or bizarre delusions, or disorganized speech or thinking and it is accompanied by significant social or occupational dysfunction. Severe Mental Illness A mental disorder associated with a significant degree of distress and dysfunction in at least one area of life functioning (e.g., work, school, daily care, etc.). Social Security Disability Insurance (SSDI) A disability insurance entitlement program administered by the Social Security Administration authorized under Title II of the Social Security Act. SSDI provides benefits (income and health insurance) to disabled or blind individuals who are “insured” by workers’ contributions to the Social Security trust fund. These contributions are based on earnings (individual, spouse, or parents) as required by the Federal Insurance Contributions Act (FICA). Eligibility for SSDI is based on SSA’s disability criteria, which includes inability to engage in any substantial gainful activity (SGA) because of a medically-determinable physical or mental impairment. SSA Mental Disorders The nine categories of mental disorders by which SSA considers an individual for disability eligibility. The categories include the following: Organic mental disorders (12.02); schizophrenic, paranoid, and other psychotic disorders (12.03); affective disorders (12.04); mental retardation (12.05); anxiety-related disorders (12.06); somatoform disorders (12.07); personality disorders (12.08); substance addiction disorders (12.09); and autistic disorder and other pervasive developmental disorders (12.10). Substance Use Disorder A maladaptive pattern of drug or alcohol use and/or dependence with adverse physical and/or emotional consequences. Glossary-5 Term Definition Substantial Gainful Activity (SGA) A term used by SSA to describe a level of work activity and earnings. Work is “substantial” if it involves doing significant physical or mental activities or a combination of both. For work activity to be substantial, it does not need to be performed on a full-time basis. Work activity performed on a parttime basis may also be substantial gainful activity. Work is “gainful” if a) it is performed for pay or profit, b) it is of a nature generally performed for pay or profit, or c) it is intended for profit, whether or not profit is realized. At the start of the MHTS in 2006, earnings averaging more than $860 for nonblind beneficiaries demonstrated SGA. At the end of the MHTS in 2010, earnings averaging more than $1,000 for non-blind beneficiaries demonstrated SGA. SGA is used as a factor to determine eligibility for benefits. Supplemental Security Income (SSI) An income supplement program administered by the Social Security Administration authorized under Title XVI of the Social Security Act. The SSI program makes cash assistance payments to aged, blind, and disabled individuals (including children) who have limited income and resources. The federal government funds SSI from tax revenues. Many states pay a supplemental benefit to individuals in addition to their federal benefits. Supported Employment (SE) Services intended to assist people with disabilities participate as much as possible in the competitive labor market, working in jobs they prefer, and with the level of professional help they need. Systematic Medication Management (SMM) An algorithmic method to facilitate and promote prescriber use of evidence-based guidelines and recommendations for medication management for individuals with schizophrenia or an affective disorder. Ticket to Work (TTW) Program administered by the Social Security Administration (SSA) that issues tickets to eligible beneficiaries who, in turn, may choose to assign those tickets to an Employment Network of their choice to obtain employment services, vocational rehabilitation services, or other support services needed to obtain or keep a job. It is a free and voluntary service. There is no penalty for not using an assigned ticket. Glossary-6 Term Definition Title II Beneficiary An individual who meets the eligibility criteria set forth under Title II of the Social Security Act to receive SSDI benefits. Vocational Plan Individualized set of written procedures aimed at assisting an MHTS treatment group participant to reach his or her vocational goal. Vocational Profile A template to document a beneficiary’s work history, job preferences, and other factors that would influence the vocational plan. Vocational Rehabilitation (VR) Any service or program that has the goal of assisting individuals with disabilities’ return to work or attain self-sufficiency. Westat Insurance Planner (WIP) A Westat staff member who is responsible to oversee the MHTS insurance services, including plan eligibilities and access, and insurance-related disbursements provided to or on behalf of participants randomized to the treatment arm of the study. Withdrawal Any participant who chose to formally end his or her participation in the MHTS prior to his or her 24th month of enrollment in the study. Glossary-7 List of Acronyms Acronyms used by the Mental Health Treatment Study Acronym ASI BACS BMI BPAO CAPI CDR CHAMP-VA CMS DHHS DSM-IV EBP EIR EROC FICA GED GME HBP HFS IPS IRB ITT IVR JTPA MBR MDPF MEF NCC NHIS OBH PIA PORT QA QM QMPD QOLI-M RA RFP Full Term Addiction Severity Index Brief Assessment of Cognition in Schizophrenia Body Mass Index Benefits Planning, Assistance, and Outreach Program Computer-Assisted Personal Interview (Baseline, Quarterly, Final Followup) Continuing Disability Review Civilian Health and Medical Program of the Department of Veterans Affairs Centers for Medicare and Medicaid Services Department of Health and Human Services Diagnostic and Statistical Manual of Mental Disorders, 4th Edition Evidenced-Based Practice Employment and Income Review Electronic Record of Contact Federal Insurance Contributions Act tax General Equivalency Diploma General Medical Exam Health Benefits Plan Hirsch Financial Services Individual Placement and Support Institutional Review Board Intent to Treat Interactive Voice Response (Westrax) Job Training Partnership Act Master Beneficiary Record Medicare Drug Plan Finder Monthly Encounter Form Nurse Care Coordinator National Health Interview Survey Other Behavioral Health Primary Insurance Amount (Schizophrenia) Patient Outcomes Research Team Quality Assurance Quality Management Quality Management Project Directors Lehman Quality of Life Inventory - Modified Research Assistant Request for Proposal Acronyms-1 List of Acronyms (continued) Acronym RIG Meeting RSI SAMHSA SCAP SCID SCHIP SE SE Specialist SGA SMM SMS SSA SSDI SSI SURF TANF TMAP TPQY TRC TTW UCDI UMBC UTHSC VA VO VR WIP WIPA WRE Full Term Recruitment Information Group Meeting Retirement and Survivors Insurance Substance Abuse and Mental Health Services Administration The Schizophrenia Care and Assessment Program Health Questionnaire Structured Clinical Interview for DSM Disorders (DSM-IV) State Children’s Health Insurance Program Supported Employment Supported Employment Specialist/Employment Specialist Substantial Gainful Activity Systematic Medication Management Study Management System Social Security Administration Social Security Disability Insurance Supplemental Security Income Services Utilization and Resources Form Temporary Assistance for Needy Families Texas Medication Algorithm Project Third Party Queries Telephone Research Center (Westat) Ticket to Work Uniform Client Data Inventory University of Maryland Baltimore County University of Texas Health Science Center at San Antonio Veterans Affairs ValueOptions Vocational Rehabilitation Westat Insurance Planner Work Incentives Planning and Assistance (Essential) Work-Related Expense Acronyms-2 Acknowledgement of Study Site Staff The MHTS investigators would like to thank the Supported Employment staff at each of our 23 MHTS study sites. In addition, we would like to recognize and thank the following administrators, staff, Nurse Care Coordinators (NCCs), Research Assistants (RAs), and other researchers at each site: Aiken Barnwell Mental Health Center Aiken, SC John Young* Rick Acton Tamara Smith Julia Ellis (NCC) Shannon Britt (RA) Adult & Child Mental Health Center, Inc. Indianapolis, IN Michael McKasson Susan Reul (NCC) Kimberly Dreher (RA)* Abigail Mook (RA) Donna Hofmann (NCC Assistant)* Boley Centers St. Petersburg, FL Gary MacMath Miriam Williams Curtis Anderson Junior Bentham (NCC) Christine Pickens (NCC) Michael Haines (RA)* The Center for Health Care Services San Antonio, TX Mary Bohne* James Castro Linda Lopez Sheryl Pavlick (NCC)* Nancy Howland (NCC) Michael Gambuti (RA)* Steven Parra (RA) Freda Facey (NCC Assistant) Cobb/Douglass Community Services Board Smyrna, GA Tod Citron Doug Crandell Michelle Robison Christine Wright (NCC)* Kelly Smith (NCC) Jennie Keleher (RA) Camille McDonald (NCC Assistant) Columbia River Mental Health Services Vancouver, WA Gregory Robinson* Nancy Parker Lisa Shannon Caroline Wrightman (NCC)* Laryn Dole (RA)* Sarah Friel (RA)* Community Connections Washington, DC Maxine Harris Helen Bergman Kate Boucek Cheryl Reidy (NCC)* Susan Cummins (RA)* Fayette Companies Human Service Center Peoria, IL Michael Boyle Becky Weaver (NCC) Julie Sprouse (RA) *Former site director or former MHTS study site research staff. Acknowledgement-1 Henderson Mental Health Center Ft. Lauderdale, FL Pamela Galan* Steven Ronick Mayra Reimer Linda Ciani (NCC) Elise Ward (NCC) Dacia Stone (RA)* Kara Williams (RA) Options for Southern Oregon Grants Pass, OR Jeffrey Krolick Judith Klein (NCC) Crystal McMahon (RA)* Dan Harper (RA)* Pathways to Housing, Inc. New York, NY Sheryl Silver* Sam Tsemberis Georgia Boothe Joanna Price (NCC)* Ollie Pianteri (NCC) Ed Nolan (NCC) Allison Zalinger (RA) LifeWorks, NW Portland, OR Phyllis Maynard Maqsuda Kabir (NCC) Trinity Wilson (NCC) Anthony Kramer (RA)* Mari Meyer (RA) Mental Health Center of Denver Denver, CO Roy Starks Mary Khoury (NCC) Shawna McGuckin (RA)* The Mental Health Center of Greater Manchester Manchester, NH Margaret Almeida* Peter Janelle Harry Cunningham Amy Szczechowicz Susan Guarino (NCC) Mary Ellen Smith (RA)* John Moscarito (RA) Selma Deyak (RA) Neighboring Mentor, OH Kenneth G. Gill Jennifer Perkins (NCC) Karen Knavel (RA)* Robin Deemer (RA) RISE, Inc. Spring Lake Park, MN Don Lavin Joan Distler Sheri Walczak (NCC) Jennifer Gajewski (RA) Jeannie Rhinhart (NCC Assistant) South Middlesex Opportunity Council, Inc. Framingham, MA Jeffrey Handler Laurie Abruzzi (NCC) Meagan Ray-Windrush (RA)* Southeastern Mental Health Authority Norwich, CT William Newkirk Stephenie Guess Claire Burchfield* Kathryn Hatfield (NCC) Faraz Wasti (RA)* Kevin Larson (NCC Assistant)* *Former site director or former MHTS study site research staff. Acknowledgement-2 Southwest Connecticut Mental Health Systems Bridgeport, CT Rebecca Allen* James Pisciotta Celeste Cremin-Endes Daniel Wartenberg Kathleen Hammond (NCC) Howard Goldberg (RA)* Avis Harper-Brooks (RA)* St. Luke’s House Bethesda, MD Cindy Ostrowski* Larry Abramson Karen Mann (NCC)* Kasaan Holmes (RA)* Lauren Matthews (RA)* Mandi Browning (NCC Assistant) Joan Clement (NCC Assistant) The Thresholds (North and South) Chicago, IL Virginia Fraser Cynthia Kelly (NCC)* Ann Naumes (NCC) Laurel Lipkin (NCC) Susan Kaiser (RA)* Chad Cancro (RA) Isabela Krzos (RA) Wyandot Center for Community Behavioral Health, Inc. Kansas City, KS Leslie Young Dena White (NCC)* Mica Willis (NCC) Christine Swenson (RA) *Former site director or former MHTS study site research staff. Acknowledgement-3
© Copyright 2024