INSTITUTIONAL EFFECTIVENESS MANUAL A GUIDE FOR ASSESSMENT AND CONTINUOUS IMPROVEMENT PUBLISHED BY THE DIVISION OF INSTITUTIONAL RESEARCH, PLANNING AND EFFECTIVENESS January 2013 BROWARDCOLLEGEISACCREDITEDBYTHECOMMISSIONONCOLLEGESOFTHESOUTHERNASSOCIATION OFCOLLEGESANDSCHOOLSTOAWARDASSOCIATEANDBACCALAUREATEDEGREES.V ISITTHE COMMISSIONONCOLLEGESATHTTP://SACSCOC.ORGFORQUESTIONSABOUTTHEACCREDITATIONOF BROWARDCOLLEGE. Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness TABLEOFCONTENTS 1 Preface ...................................................................................................................................pg 4 2 About this Manual..................................................................................................................pg 5 3 Terminology ...........................................................................................................................pg 6 4 Accreditation and Institutional Effectiveness ........................................................................pg 7 5 Institutional Effectiveness at Broward College 5.1 Description ....................................................................................................................pg 8 5.2 Institutional Effectiveness Planning vs. Strategic Planning .........................................pg 9 5.3 Institutional Effectiveness Model.. .............................................................................. pg 10 5.4 Important IE and SACS Deadlines for the 2012-13 IE Cycle .....................................pg 10 5.5 Institutional Effectiveness Staff ....................................................................................pg 11 6 TaskStream Institutional Effectiveness Tool 6.1 Defining Units: Who Needs to Complete an IE Plan? .................................................pg 12 6.2 Using TaskStream .........................................................................................................pg 13 6.3 Institutional Effectiveness and Outcomes Assessment Template .................................pg 14 6.4 Template for Assessing Educational Programs ............................................................pg 15 6.5 Template for Assessing Administrative, Student Support, Community Units .............pg 16 7 Quality Assurance Review Process 7.1 Reviewers: Who Needs to Review My IE Plan? ..........................................................pg 17 7.2 Review Rubric: Academic Program Assessment Plan and Report ...............................pg 18 7.3 Review Rubric: Administrative, Student Support, Community/Public Svc Units .......pg 19 8 Best Practices: Academic Areas 8.1 Developing a Mission Statement ..................................................................................pg 20 8.2 Identifying, Writing and Prioritizing Academic Outcomes ..........................................pg 21 8.3 Assessing Academic Outcomes ....................................................................................pg 22 9 Best Practices: Administrative, Student Support, Community Units 9.1 Developing a Mission Statement ..................................................................................pg 23 9.2 Identifying, Writing and Prioritizing Unit Outcomes ...................................................pg 24 9.3 Assessing Unit Outcomes .............................................................................................pg 25 Page2 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Table of Contents 10 Using Findings for Continuous Improvement ..............................................................pg 26 11 Training ..........................................................................................................................pg 27 12 FAQ................................................................................................................................pg 28 13 Appendices A: Example Educational Program IE Plan...................................................................pg 30 B: Example Administrative, Student Support, Community Unit IE Plan....................pg 36 Page3 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 1. Preface Dear Colleagues: Thank you for your time and effort in helping make our Institutional Effectiveness process at Broward College a success. The Office of Institutional Effectiveness prepared this manual specifically to assist college staff, administrators and faculty in developing, implementing and assessing strong, integrated, college-wide plans that add to the quality of Broward College’s teaching, learning and overall institutional effectiveness. Several people contributed to the development and completion of this manual: Drs. Gail Wisan, Amy Gregory, Mrs. Mildred Foster and Mr. Gregg Sekscienski. I also would like to extend my appreciation to Mrs. Laurie Heacock for her diligent and thoughtful work on the overall document. To them, I am very grateful for their contributions. If you need any assistance or have questions about institutional effectiveness or planning assessment processes, please contact me. Suggestions for improving this manual are always welcome. Sincerely, Rigoberto J. Rincones-Gómez, Ph.D. Associate Vice President Division of Institutional Research, Planning and Effectiveness Email: [email protected] | Office: 954- 201-7857 | Fax: 954-201-7322 Page4 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 2. About this Manual This manual provides a conceptual and hands-on guide for the Broward College community. This guide is intended to assist each and every academic program, educational and student support area, administrative unit, and community and public service area to identify its expected outcomes, measure the extent to which the program or unit achieves its outcomes, and provide data and evidence of improvement based on an analysis of its assessment results. The evaluation of institutional effectiveness (IE) is essential for continuous improvement in institutional quality and for demonstrating the college is effectively accomplishing its mission. Users of this manual can expect to understand: Commonly used institutional effectiveness terminology The college’s comprehensive, institutional effectiveness process How to develop institutional effectiveness plans for educational programs How to develop institutional effectiveness plans for administrative, student support and community/public service areas How to input institutional effectiveness plans into TaskStream (the college’s IE tool) How institutional effectiveness supports SACS-COC accreditation Important dates and deadlines for the IE cycle How to use institutional effectiveness results for continuous improvement Page5 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 3. Terminology Term Common Definition Accountability The obligation of a college to account for its activities, accept responsibility for them and disclose the results in a transparent manner to the public including the state, federal government and accrediting body. Action Specific change that will be implemented to improve programs or units based on results/findings from a previous assessment cycle. Action Status The status of a program or unit’s actions when measures have been updated or findings have been added. This would typically be entered when preliminary finding are reported. Goal A statement of a desired future state or condition, stated more generally than an outcome. Learning Assessment The process of determining the extent to which students have mastered some instructional objective or competency. Outcomes The short-, medium- and long-term desired effects or changes. Administrative Outcomes Benefits that departments or administrative units receive or the college receives as a whole. Learning Outcomes Changes in knowledge, attitudes, behaviors, values and opportunities that occur as a result of learning. Learning outcomes may be identified at the student, course and program level. Educational and student support areas should also identify learning outcomes. Program Outcomes Benefits that students receive as a result of the completion of an entire program of study. Mapping Aligning outcomes to a strategic goal, core value or to the college’s mission. Mission A clear description of the purpose of the program or unit and who it serves. Unit An academic degree-granting program, an academic or student support area, an administrative support area or a community/public service area. Workspace Designated area in TaskStream where IE plans and findings are entered. Page6 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 4. Accreditation & Institutional Effectiveness The Southern Association of Colleges and Schools Commission on Colleges (SACS-COC) is the regional body for the accreditation of degree-granting higher education institutions in the South. In their 2012 edition of The Principles of Accreditation: Foundations for Quality Enhancement, institutional effectiveness is addressed in Core Requirement 2.5 and is defined as follows: “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate the institution is effectively accomplishing its mission.” Whereas the Core Requirements are foundational requirements, The Principles of Accreditation: Foundations for Quality Enhancement also contains Comprehensive Standards, which are operational and represent best practices. The expectation for Institutional Effectiveness is addressed in Comprehensive Standard 3.3: “The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: 3.3.1.1 educational programs, to include student learning outcomes 3.3.1.2 administrative support services 3.3.1.3 academic and student support services 3.3.1.4 research within its mission, if appropriate 3.3.1.5 community/public service within its mission, if appropriate” Page7 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness 5. IE E at Brow ward Colleege 5.1 Description he Commissiion on Colleeges, Browarrd College m must complyy with To mainttain accreditation with th the core requirement r s and standaards containeed in the Prinnciples of Acccreditation: Foundations for Qualiity Enhancem ment and witth the policies and proceedures of thee Commissioon on Collegges. In the con ntext of institutional effeectiveness, Broward B Colllege strives to be the beest in class amongst all Florida peer p collegess. The Strateegic, Strateggic Implemenntation and D Detailed Implementation Plan ns provide th he roadmap to o reach this end vision. E Each membeer of the college community works as part of a teaam within a division, offfice or deparrtment to translate these o actions to drive Browaard College forward. f plans into To advan nce success through t orgaanizational structure, thee Division off Institutionaal Research, Planning g and Effectiv veness (IRPE E) is organizzed into fourr areas or off ffices: Instituutional Plannning, Institutio onal Effectiveness, Institu utional Reseearch and Buusiness Intellligence. Page8 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness 5.2 Instittutional Efffectiveness Planning P vs. Strategic P Planning It is usefu ful to distingu uish between n strategic planning and institutionall effectiveneess planning. Strategic planning is a systemic process p of en nvisioning a desired futuure, and trannslating that vision intto broadly defined d goalss or objectivees and a sequuence of stepps to achieve them. Thee Broward College Strategic Plan, for instancee, extends thee college’s vvision througgh 2017. In contraast, institutional effectiveeness plannin ng operates on an annuaal cycle wherre each program//unit examin nes its purpose, identifiess expected ooutcomes, sellects assessm ment tools annd strategiess to collect data, d and anaalyzes and reeports findinngs for continnuous qualitty improvem ment. Innstitutional E Effectivenesss Plannning Sttrategic Plan nning • • • • • Collegee level Vision focused Wide view v Flexiblle means Begins with desired d end • • • • • Unnit level Miission focuseed Naarrow view Sppecific meanss Beegins with reeview of pastt asssessment cyccle or review w of currrent status ((if new) 5.3 Instittutional Efffectiveness Model M Institutio onal effectiveeness at Brow ward Colleg ge is missionn-driven and includes a sset of ongoinng and systeematic, instittutional proccesses and prractices that encompass:: The T assessmeent of academ mic, adminisstrative, studdent support and commuunity/public seervice prograams and serv vices. The T developm ment and imp plementation n of program m/unit assesssment and evvaluation plaans. The T use of ev valuation resu ults to fosterr a culture off evidence aand continuous improvem ment. Page9 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness Overall Institutio onal Effecttiveness M Model 5.4 Impo ortant IE an nd SACS Deeadlines forr the 2012–113 Cycle 7/2/2012 2012-13 IEE Cycle Startts A Aug 12 7/2/2012 10/5/12 Dead dline for units to en nter IE plans Sep 12 2 12/19/12 2 Deadline for u units to enter IEE preliminary fin ndings Aug 12 - Dec 12 Fall TTerm Oct 12 N Nov 12 IEE Training S Sessions 9/6/1 12 - 9/28/12 Dec 12 Jan 13 IE Status Report 1/15/2013 Page10 1/22/2013 IE Status Presented to BC BOT F Feb 13 5/10//13 Deadline ffor units to enter finaal findings for the 20 012-13 IE Cycle Jan 13 - May 13 M Winter Term Mar 13 Apr 13 Maay 13 Dead dline for units to uplo oad IE evidence IEE Status R Report 5/24/13 6/1 11/2013 6/28/2013 20 012-13 IE Cyycle Ends Jun 13 6/28/2013 IE Staatus Presented to BC BOT 6/18/2 2013 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 5.5 Institutional Effectives Staff Dr. Rigoberto Rincones-Gómez, Associate Vice President of Institutional Research, Planning and Effectiveness Dr. Gail Wisan, District Director of Institutional Effectiveness Dr. Amy Hyman Gregory, Interim Director of Institutional Effectiveness Ms. Mildred Foster, Accreditation and Document Analyst of Institutional Effectiveness Page11 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 6. TaskStream: Institutional Effectiveness Tool The college is committed to using TaskStream as an IE tool through 2014. 6.1 Defining Units: Do I Need to Complete an IE Plan? As stated in the Procedure Manual shown below, “all College administrators are responsible for the implementation and execution of institutional effectiveness plans for the organizational departments and/or programs for which they are assigned administrative responsibility.” Page12 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 6.2 Using TaskStream Broward College has implemented TaskStream’s customizable Accountability Management System (TaskStream’s AMS) to support continuous improvement projects and manage college wide and program level accountability processes. To request access to TaskStream and the creation of a workspace, please contact Dr. Amy Hyman Gregory at 954-201-7543 or [email protected]. The TaskStream User Guide is accessible from the IRPE Division SharePoint web site in Workplace. To assist you in completing your online IE plans in TaskStream, paper templates are provided in section 6.3. Page13 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 6.3 Institutional Effectiveness & Outcome Assessment Template Note: If you have already entered your 2012-13 assessment plan into TaskStream, you do not need to fill out a paper template or follow the steps below. If your unit is an educational program, use Template 1. If your unit is an administrative, student support or community/public service unit, use Template 2. STEPS: 1. Please enter your unit’s mission statement in Item 1 of the Template. 2. Please define new individual outcomes or use/select outcomes from your 2011-12 assessment plan. 3. Please map each of your outcomes to the appropriate strategic goal(s), core value(s), and/or the college’s mission. 4. Please identify the action(s)/activities based on your 2011-12 assessment findings and link them to specific outcomes. Then complete the following for each action/activity: Action details Implementation plan (timeline) Responsible personnel Resources needed If budget approval is required Budget request amount Priority (low, medium or high) 5. Please select one of the four options (not started, in progress, completed or not implemented) at the time you report your findings. 6. Please follow the directions included in Item 6 of the template. 7. Assessment findings will be completed after data have been gathered and analyzed. 8. Please follow the directions included in Item 8 of the template. Page14 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Page15 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Page16 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 7. Quality Assurance Review Process 7.1 Reviewers: Who Needs to Review My IE Plan? To provide quality assurance, institutional effectiveness planning includes a review process. Review rubrics are designed to facilitate this process, promoting quality control, providing guidance to reviewers and ensuring consistency in standards. Once an assessment plan or report is completed, the TaskStream workspace manager must submit their plan/report for review. The assessment plans for academic programs are typically reviewed by Associate and Academic Deans followed by final reviews by the Provost and Campus President. The assessment plans for student support programs are reviewed by the Vice President of Student Affairs. The assessment plans for administrative and community/public service units are reviewed by the appropriate Vice President. Page17 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 7.2 Review Rubric: Academic Program Assessment Plan and Report Page18 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 7.3 Review Rubric: Administrative, Student Support, Community/Public Service Units Page19 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 8. Best Practices: Academic & Educational Programs 8.1 Developing a Purpose or Mission Statement A purpose or mission statement is a clear description of the purpose of a program and who it serves. A mission statement guides the practices of the program. Ideally, a mission statement should: Concisely state the purpose of the program Indicate the primary function or activities of the program Specify who the program serves Clearly support the college’s broader mission Showcase what makes the program unique or distinctive 8.2 Identifying, Writing and Prioritizing Academic Outcomes Academic outcomes are comprised of student learning outcomes and program outcomes. Student learning outcomes are typically included in a course syllabus and answer what students should know and be able to do at the time of course completion. Program learning outcomes answer what students should know and be able to do at the time of program completion. To identify academic outcomes, focus on the key skills a student gains from completing a course (student learning outcomes) or program (program outcomes) in terms of: Knowledge Attitudes Values Behaviors Opportunities Once you have identified a list of academic outcomes, evaluate them based on the following criteria to ensure they are: Learner centered versus teaching centered Central to the program’s mission Page20 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness Im mportant and d meaningfu ul for both faaculty and stuudents Connected C an nd mapped to o a goal in th he 2012-17 B Broward College Strateggic Plan, to a co ore value or to the colleg ge mission Actionable: A you y can take action as a result r of the outcome SMART: Sp pecific, Meassurable, Attaainable, Releevant, Timely Modelfo orWritingO Outcomes: A Asaresultoofstudents completin ng<insertprogram>,<iinsert%>w willbeable to<inserttactionverb b>. Example e:Asaresulltofstudentsscompletinggthe NursingR R.N.program m,95%willbeabletoppassthe NCLEXliccensingexam montheirfi firstattemptt. Next, priioritize the outcomes. o Do D not try to cover every domain. Foocus on a small number oof outcomess (top three) that are: Widely W relevaant Highly H valued d by faculty and students Reflective R of higher orderr critical skillls Most M useful for f informing g decisions that t can leadd to improvem ment Page21 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 8.3 Assessing Academic Outcomes Once you have defined your top three academic outcomes, the next step is to choose an assessment method. There are two types of assessment methods: direct and indirect. Direct assessment methods measure student performance by examining samples of actual student work at the course, program or institutional level. Indirect assessment methods measure outcomes related to student learning—such as student perceptions—versus evidence of direct student learning. Examples of Direct and Indirect Measures of Academic Learning Outcomes Direct Measures Course Indirect Measures Research project Course evaluation Tests and quizzes Student hours spent on service learning Arts performance Lab assignment Program/Unit Employer rating of student performance Focus groups with students, faculty and employers Licensure exam pass rate Program review data Capstone project Job placement Internship supervisor rating Employer or graduate surveys Portfolio Institutional Performance on achievement tests Rubric scores for general education courses or other courses required of all students Performance on tests of writing, critical thinking or general knowledge Locally developed, commercial or national satisfaction and engagement surveys, e.g., Community College Survey of Student Engagement (CCSSE), Noel-Levitz Graduation rates Retention rates Transcript analyses that examine patterns of course selection and grading Adapted from the Middle States Commission on Higher Education. Student Learning Assessment: Options and Resources. Chapter 3, Evaluating Student Learning: 27-53; 2007. Page22 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 9. Best Practices: Administrative, Student Support, Community Units 9.1 Developing a mission statement A mission statement is a clear description of the purpose of a unit and who it serves. A mission statement guides the practices of the unit. Ideally, a mission statement should: Concisely state the purpose of the unit Indicate the primary function or activities of the unit Specify who the unit serves Clearly support the college’s broader mission Showcase what makes the unit unique or distinctive 9.2 Identifying, Writing and Prioritizing Outcomes for Administrative, Student Support, Community/Public Service Units In addition to the academic outcomes described in Section 8, administrative, student support and community/public service units also include administrative outcomes. Administrative outcomes align with unit mission statements and answer how the college—including students, faculty and staff-—benefits as a result of utilizing the services or functions provided by these units. Ideally, these outcomes: Address performance: efficiency, effectiveness, quality, service, cost Inform decision-making and planning for innovation and change Focus attention on what matters the most: key functions, services, processes Key questions to ask when identifying administrative outcomes include: To achieve our mission how should we appear to students, faculty, staff, other units, the college as a whole? Page23 “Efficiencyisdoingthingsright; effectivenessisdoingtheright things.” PeterF.Drucker Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness What W businesss processes must we exccel at to bestt benefit thosse we serve?? How H is our un nit planning for innovatiion and in m meeting the fuuture needs oof those we seerve? How H do we develop d and manage m reso ources (humaan and fiscall)? What W is the most m highly valued v servicce we providde institutionnally? Once you u have identiified a list off administrattive outcomees, evaluate them based on the following criteria to o ensure they y are: Central C to you ur area’s misssion Under U your purview Distinctive D to o your unit Connected C an nd mapped to o a goal in th he 2012-17 B Broward College Strateggic Plan, to a co ore value or to the colleg ge mission Actionable: A you y can take action as a result r of the outcome SMART: Sp pecific, Meassurable, Attaainable, Releevant, Timely TipforWrritingOutco omes:Usea actionverbs suchasincrrease,decreease, improve,reeduce,minim mize,maximize,complette,promote,,provide,deesign, develop,offfer,conduct,,deploy,pro ocess,implem ment,expan nd…. Example:A Asaresulto ofmeetingw withanadvis isor,95%off allnewstud dents developeda aneducation nalplandurringtheirfirrstsemesterr. Next, priioritize the outcomes. o Do D not try to cover every domain. Foocus on a small number oof outcomess (top three) that are: Widely W relevaant Highly H valued d by the stafff in your uniit Page24 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Indicative of what your target audience knows, can do or values as a result of using your service Most useful for informing decisions that can lead to improvement. 9.3 Assessing Administrative, Student Support, Community/ Public Service Outcomes Once you have defined your top three outcomes, the next step is to choose an assessment method. See Section 7.3 on page 19 of this guide for a description and examples of direct and indirect assessment methods for academic areas. Typically the methods most useful for assessing administrative outcomes include: Faculty/staff and student satisfaction and engagement surveys (e.g., % students student satisfied with Advising and Enrollment Services, % of faculty and staff satisfied with service from the Help Desk) Focus group sessions targeted to the policies, processes and procedures of a unit (e.g., admission process, financial aid process, testing and placement policy) Evaluation forms from attendees of a program or training conducted by your unit (e.g., % of new students satisfied with orientation, etc.) Unit statistics (e.g., call abandonment rate, resolution rate, % of requests processed on time, training expenditures/employee, student/staff ratios, % transcripts evaluated within x days of receipt, etc.) Page25 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness 10.Using U Find dings for Continuou C s Improveement The goal of assessmeent is continu uous improv vement. Eachh outcome m measure asseessed should result nott only in traccking and monitoring, bu ut in actions designed too yield improovement. Thhe use of resultss is crucial in n closing thee feedback lo oop and for ssuccessful asssessment. with all stakkeholders, To facilittate use, the results shou uld be broadlly shared andd discussed w including g faculty, adm ministrators, staff, studeents, advisoryy boards andd the commuunity, as appropriaate. These diiscussions sh hould guide the developm ment of speccific strategiies that are sustainab ble, feasible and designed to improvee program, uunit or studennt learning ooutcomes. a may require r resouurces, it is crritical that im mprovementt Because the implemeentation of actions strategiess requiring funds fu be link ked to the bu udget processs by means oof a budget rrequest. For example,, an improveement strateg gy may require investmeents in technnology, profeessional developm ment or the hiring h of add ditional staff.. If funds aree not availabble, then the implementaation strategy may m require the reallocaation of resou urces. Somee improvemeent strategiess do not requuire fiscal ressources but in nvestments in i time and collaboration c n which requuire planningg such as changes to t policies, course c redesign, schedulling changess or program m modificatioons. Page26 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 11. Training An Institutional Effectiveness training program was launched at Broward College in September 2012. The training program has three components: 1. Assessment Planning 2. TaskStream 3. Reviewer Training Group and individual training sessions are held during the yearly assessment cycle. Each year, assessment cycle timelines are posted on the IRPE SharePoint web site in WorkPlace and widely shared with the Broward College community. 1. Assessment Workshop: How to Create Better Assessment Plans and Write Better Outcomes Description: This workshop will explain the components of an Institutional Effectiveness Plan (mission, program and learning outcomes, mapping, actions, strategies and measures) and using your findings/results for improvement. Participants will learn: how to write better outcomes, create good assessment plans, plan for assessment, how to make assessment useful for your department/unit. 2. TaskStream Training Workshop Description: This workshop will cover the basics of entering institutional effectiveness assessment information into TaskStream. Participants will develop the knowledge necessary to create/revise assessment plans, map outcomes to specific goals and initiatives, and enter assessment plans, findings, and supporting documentation into their area’s Workspace(s). A TaskStream user account is needed to enroll in this workshop. 3. Reviewer Training Workshop Description: Reviewer training includes suggested reviewer structure, the purpose and use of rubrics, example rubrics and information on how to review plans and reports in TaskStream. Page27 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 12. FAQS Q. Are all college administrators required to complete an institutional effectiveness plan? A. Yes, all College administrators are responsible for the implementation and execution of institutional effectiveness plans for the organizational departments and/or programs for which they are assigned administrative responsibility. Q. How do I input my IE plan? A. Via TaskStream. Paper templates are available to help you organize your plan before entering into TaskStream. Q. How do I acquire a TaskStream account and the creation of a workspace? A. Contact Dr. Amy Hyman Gregory in IRPE at 954-201-7543 or [email protected] Q. Who has to review my IE plan? A. The assessment plans for academic programs are typically reviewed by Associate and Academic Deans followed by final reviews by the Provost and Campus President. The assessment plans for student support programs are reviewed by the Vice President of Student Affairs. The assessment plans for administrative and community/public service units are reviewed by the appropriate Vice President. Q. How many outcomes should I include in my IE plan? A. Three is recommended Q. What are the criteria for determining if I developed a good outcome measure? A. Specific, Measurable, Attainable, Relevant, Timely (SMART) Q. What do outcomes have to map and connect to? A. A goal in the strategic plan, a core value or to the college’s mission Q. Who do I contact for questions regarding TaskStream use, training or troubleshooting? A. Contact Dr. Amy Hyman Gregory in IRPE at 954-201-7543 or [email protected] Q. Who do I contact regarding assessment workshops, assistance writing outcomes, developing rubrics or curriculum maps? A. Contact Dr. Gail Wisan in IRPE at 954-201-7482 or [email protected] Q. Who do I contact to schedule meetings, trainings or workshops? A. Contact Mildred Foster in IRPE at 954-201-7480 or [email protected] Page28 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness Ap ppendix A ACT TUAL TASK KSTREAM M EXAMPLE E Institutiona al Effectiven ness & Outccome Assesssment for E Educational Program Fiscal Yeear: _7/1/20 012-6/30/201 13________ ______________ Educatio onal Progra am: Radiatio on Therapy y A.S. degreee (Health Sciences) Specific Coursse (if applicab ble): Steps to o Creating an a IE Plan iin TaskStreeam 1. Missio on Statemen nt/Academicc Program Goals G (Whaat are studentts expected tto know andd be able to o do at the en nd of the pro ogram/course? What skiills should sttudents havee acquired baased on thee program off study and/o or specific co ourse?) The misssion of the Radiation R Theerapy Prograam is to preppare competeent, ethical aand culturallly sensitive entry-level therapists who w are capab ble of functiooning in an ever-changinng health caare delivery system as a result of beiing dedicated d to lifelong self-directed learning. T The program m is plishing its mission m throu ugh the use of effective and diverse instructionaal committeed to accomp methods that encomp pass both traaditional as well w as technnology-basedd strategies. 2. Learn ning/Program m Outcomees and Meassures (Studeent Learnin ng Outcomes are specific, measu urable, and reesults oriented changes within w studeents that can be evaluatedd based on classro oom learning g. Program m Outcomes are measuraable end resuults studentss are expecteed to achiev ve after comp pleting the entire e prograam.): Performaance The grad duate will bee able to effeectively applly the skillss and knowleedge related d to the radiation n therapy field Career Development D t The grad duate will bee able to partticipate in career deevelopment based upon selfassessmeent, perform mance appraissals, work setting and a special in nterests to fo oster a sense of life-long g learning The Presscription The grad duate will bee able to perfform safe radiation n therapy treatments com mpetently based on n the prescrip ption so as to o minimize risk to th he patient/cliient, self, and others. Page29 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness See measures below in Assessment Plan. 3. Mapping (Outcomes should be mapped to the appropriate strategic goal of the 2012-2017 BC strategic plan, core values, and/or to the specific aspect of the BC mission): Outcomes are mapped to Broward College Strategic Plan 2012-2017, student success Goal 1. 4. Actions (Actions are specific changes used to improve your program based on the previous cycle’s assessment findings): Increase the variety of teaching methods utilized To begin utilizing clicker technology, web enhancing of lectures and labs Action: Continue web enhancement of the Radiation Therapy Program This Action is associated with the following Findings Finding on student performance. 1.2 To Begin Web Enhancement of Lecture Courses To enhance the students access to material by providing different styles and methods of delivery. Also allowing the students access to the educational material 24/7 Action details: Develop one or more blended course for the program Implementation Plan (timeline): To have the first blended course up and running during fall 20141 and survey the results in 20142. Key/Responsible Personnel: faculty, program manager, associate dean Resources needed (IT,IR, etc.): student satisfaction and increasing academic and clinical performance Budget approval required? (Describe): course and training necessary are offered through the professional development. Budget request amount: $0.00 Priority: Medium Page30 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 5. Action Status (Action Status indicates the current status of each action taken, whether it is completed, is still in progress, etc. Action Status may be reported later in the year as you record progress or at the same time you report your current assessment cycle findings): Report status in preliminary report, December 2012 and final report, 2013 6. Assessment Plan (Input your current assessment plan carrying over outcomes and measures from the previous cycle and/or including new outcomes and measures implemented. Include a description of each measure, an acceptable and ideal target, an assessment timeline, and the responsible personnel for each): Measure: Performance Program level; Direct - Other Performance The graduate will be able to effectively apply the skills and knowledge related to the radiation therapy field Details/Description: At the end of the program, the graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others as observed in the classroom, lab and clinical arenas. Measurement will entail direct observation of student providing simulated or actual treatment. Acceptable Target: 85% of the graduates will demonstrate the above. Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually - at the end of Term III. Key/Responsible Personnel: Faculty Clinical Instructors The Prescription The graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others. Measure: The Prescription Program level; Direct - Other Details/Description: At the end of the program, the graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others as observed in the classroom, lab and clinical arenas. Acceptable Target: 85% of the graduates will demonstrate the above. Page31 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually at the end of term III. Key/Responsible Personnel: Faculty Clinical Instructors Measure: Data Collection Program level; Direct - Other Data Collection The graduate will be able to demonstrate competence in performing specific data-collection techniques related to the prescription. Health Care Delivery Systems The graduate will be able to use human and material resources and services to provide highquality, efficient, and cost effective radiation therapy services based on patient/client goals and outcomes Details/Description: At the end of the program, the graduate will be able to demonstrate competence in performing specific datacollection techniques, as delegated by the supervising radiation therapist as observed in the classroom, lab and clinical arenas. Acceptable Target: 85% of the graduates will demonstrate the above. Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually - at the end of Term III. Key/Responsible Personnel: Faculty Instructors Measure: Health Care Delivery System Program level; Direct - Other Details/Description: At the end of the program, the graduate will be able to use human and material resources and services to provide high-quality, efficient, and cost effective radiation therapy services based on patient/client goals and outcomes as observed in the classroom, lab and clinical arenas. Acceptable Target: 85% of the graduates will demonstrate the above. Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually - at the end of Term III. Key/Responsible Personnel: Faculty Instructors 7. Assessment Findings (Summarize the findings for each outcome and measure, including supporting evidence such as surveys, data, tables, and indicate specifically how these findings will be used for improvement.): Page32 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Evidence must be provided on each outcome. Measure: Performance Program level; Direct - Other Details/Description: At the end of the program, the graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others as observed in the classroom, lab and clinical arenas. Acceptable Target: 85% of the graduates will demonstrate the above. Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually - at the end of Term III. Key/Responsible Personnel: Faculty Clinical Instructors Findings for Performance Performance The graduate will be able to effectively apply the skills and knowledge related to the radiation therapy field Summary of Findings: At the end of Term III all of the graduating students were able to perform safe radiation therapy procedures. This was based on their ability to read the prescriptions and perform the treatments accurately and safely. This was documented on the Rotational evaluations, the therapist final evaluations and the photon and electron treatment evaluations. Results: Acceptable Target Achievement: Exceeded; Ideal Target Achievement : Exceeded How do you plan to use these findings for continuous improvement? We plan to continue to monitor and evaluate the graduating students in all aspects of a radiation therapy, including reading prescription. room preparations and delivery. Substantiating Evidence: Electron treatment eval prescription.pdf (Adobe Acrobat Document) Areas utilized for performance grades Summer 2011.pdf (Adobe Acrobat Document) Monthly-Rotational Eval performance.pdf (Adobe Acrobat Document) Areas utilized for performance Photon Evaluation performance docx.pdf (Adobe Acrobat Document) Areas utilized for performance Sample of electron Eval...PDF (Adobe Acrobat Document) Sample of Monthly rotational evals.PDF (Adobe Acrobat Page33 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Document) Sample of photon eval.PDF (Adobe Acrobat Document) Sample of Therapist eval.PDF (Adobe Acrobat Document) Therapists Final Evaluation Performance.pdf (Adobe Acrobat Document) Areas utilized for performance Measure: The Prescription Program level; Direct - Other The Prescription The graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others. Details/Description: At the end of the program, the graduate will be able to perform safe radiation therapy treatments competently based on the prescription so as to minimize risk to the patient/client, self, and others as observed in the classroom, lab and clinical arenas. Acceptable Target: 85% of the graduates will demonstrate the above. Ideal Target: 100% of the graduates will demonstrate the above. Assessment Timeline: Annually at the end of term III. Key/Responsible Personnel: Faculty Clinical Instructors Findings for The Prescription Summary of Findings: At the end of Term III all of the graduating students were able to perform safe radiation therapy procedures. This was based on their ability to read the prescriptions and perform the treatments accurately and safely. This was documented on the Rotational evaluations, the therapist final evaluations and the photon and electron treatment evaluations. The students were also able to safely and accurately run quality assurance on the machines to make sure that the correct dose per prescription was maintained, and this was demonstrated in the clinic during warm up procedures. Results: Acceptable Target Achievement: Exceeded; Ideal Target Achievement : Exceeded How do you plan to use these findings for continuous improvement? We plan to continue to monitor and evaluate the graduating students in all aspects of a radiation therapy, including reading prescription. room preparations and delivery and warm up procedures. Substantiating Evidence: CT Simulation Evaluation Form prescription docx.pdf (Adobe Acrobat Document) Page34 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness Areas utilized for prescription skills Electron treatment eval prescription.pdf (Adobe Acrobat Document) Areas utilized for prescription skills grades Summer 2011.pdf (Adobe Acrobat Document) Monthly-Rotational Eval prescription.pdf (Adobe Acrobat Document) Areas utilized for prescription skills Photon Evaluation prescription.pdf (Adobe Acrobat Document) Areas utilized for prescription skills Sample of electron eval..PDF (Adobe Acrobat Document) Sample of Monthly rotational evals.PDF (Adobe Acrobat Document) Sample of photon eval.PDF (Adobe Acrobat Document) Sample of Therapist eval.PDF (Adobe Acrobat Document) Sample Warm up Eval..PDF (Adobe Acrobat Document) Samples of CT Simulation evals..PDF (Adobe Acrobat Document) Therapists Final Evaluation Prescription.pdf (Adobe Acrobat Document) Areas utilized for prescription skills Warm-up Competency. eval prescription.pdf (Adobe Acrobat Document) Areas utilized for prescription skills 8. Budget Request (If funding is needed to support an outcome you must complete a Budget Request Evaluation Form indicating the status of your budget request, whether it has been 100% approved, partially approved, or not approved). Page35 Institutionaal Effectivenesss Manual Division of Innstitutional Reesearch, Planninng and Effectivveness Ap ppendix B MODE EL EXAMP PLE Institu utional Effecctiveness & Outcome Assessment A ffor Administrative, Stu udent Support and d Communiity/Public Seervice Unit Fiscal Yeear: 7/1/201 12-6/30/2013 3 Administraative Unit: ☐ Yes ☐ N No Student Support Seervices Unit:: X Yes ☐ No N Com mmunity/Pu ublic Servicee Unit: ☐ Y Yes ☐ No me: Disabiliity Services (Division of o Student A Affairs) Unit Nam Steps to o Creating an a IE Plan iin TaskStreeam 1. Missio on Statemen nt (What serrvices do you u provide? T To whom aree theses servvices provideed? For stu udent support, how do you y plan to im mpact studennt learning ooutcomes?) Disability y Services (D DS) helps en nsure equal educational e aaccess and oopportunity ffor all studennts with docu umented disabilities. In n addition, DS collaborattes with facuulty, adminisstrators, and staff to ensuree the facilitattion of reaso onable and ap ppropriate acccommodatiions for studdents with documen nted disabilitties in accord dance with Section S 504 oof the Federral Rehabilitaation Act off 1973 and d Title III of the Americaans with Dissabilities Actt. 2. Outco omes and Measures M (Ad dministrativee and Studen nt Learningg Outcomes are specificc, measu urable, and reesults oriented): Outcomee 1 example: Disability Services S (DS S) will providde accurate aand timely innformation tto studen nts in their fiirst semesterr at Broward d College so tthat they cann self-identiffy as needing DS if apprropriate. Examplles of direct measures: 1,)counts 1 of workshops w pprovided durring student orientation; 2.) Inforrmation on DS D website evaluated e witth rubric; 3.)) Hits on thee Broward D DS website; 44.) Countts of studentss who are prrovided one on o one counnseling regarrding DS serrvices duringg the first month m of scho ool; Outcomee 2 example: Disability y Services will w provide faculty f with timely inforrmation abouut the studennts in their classes who are seelf-identified d as needing accommodaations and sppecify the naature of the needed d appropriatte accommod dation(s). Examples of direct measures: m 1.)) time line do ocument creaated for students, facultty and staff regard ding accomm modations; 2.) Direct evidence of o distribution of time lin ne to studentts, faculty, annd staff (e.g., meeting minutees, email disstributions; student s newsspaper adverrtisements. Page36 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness 3. Mapping (Outcomes should be mapped to the appropriate strategic goal of the 2012-2017 BC strategic plan, core values, and/or to the specific aspect of the BC mission): Disability Services outcome about providing accurate and timely information to students maps to the Broward College Strategic Plan student success goal 1 and also Broward College's mission. 4. Actions (Actions are specific changes used to improve your unit based on the previous cycle’s assessment findings): 1.) Each semester, Disability Services will provide monitoring reports regarding the distribution of accommodation information to faculty and information about the accommodations their students will need. 2.) Disability Services will provide information to faculty one week earlier than the prior semester. 3.) Disability Services will provide training for volunteer note takers who will then provide service for students with documented disabilities who require assistance with note taking. 5. Action Status (Action Status indicates the current status of each action taken, whether it is completed, is still in progress, etc. Action Status may be reported later in the year as you record progress or at the same time you report your current assessment cycle findings): When the preliminary report is filed (December 2012), the Action Status needs to be completed; Is Action "in process," "completed," or "not started." 6. Assessment Plan (Input your current assessment plan carrying over outcomes and measures from the previous cycle and/or including new outcomes and measures implemented. Include a description of each measure, an acceptable and ideal target, an assessment timeline, and the responsible personnel for each): For example, carry over (i.e., continue from the prior year) Outcome 2 : Disability Services will provide faculty with timely information about the students in their classes who are self-identified as needing accommodations and specify the nature of the needed appropriate accommodation(s). Direct Measure: number of information sessions provided for faculty with a rubric evaluating information content; Indirect Measure: survey of faculty who report what they learned and their satisfaction. Acceptable target: provide information at faculty orientation; Ideal target: provide information at information session on each campus. Targets related to information learned at information sessions. Responsible person: Director of Disability Services; Budget: budget for coffee, copying, and printing of information material. Add new outcome: Disability Services will recruit and train note takers in classes with students with documented disabilities. Direct Measures: count of recruitment activities; number of note Page37 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness takers recruited; number of note takers trained; Add new outcome: Disability Services will compare the learning outcomes (including test performance) of students with disabilities with access to note takers as compared with classes without note takers for students with documented disabilities. 7. Assessment Findings (Summarize the findings for each outcome and measure, including supporting evidence such as surveys, data, tables, and indicate specifically how these findings will be used for improvement.): Findings completed in the preliminary report by December 2012 and again by June 2013. Summarize findings for each outcome and then provide evidence of findings. For Example, analyze success data for students with documented disabilities who have access to note takers in their class as compared with students with documented disabilities who do not have access to note takers. Compare both to success data on non-disabled students. If note-taking action/strategy is working, students using the service should exhibit the same level of success (e.g., grade performance) as students without disability in the same class. 8. Budget Request (If funding is needed to support an outcome you must complete a Budget Request Evaluation Form indicating the status of your budget request, whether it has been 100% approved, partially approved, or not approved). Budget: budget for coffee, copying, and printing of information material. Page38 Institutional Effectiveness Manual Division of Institutional Research, Planning and Effectiveness This page intentionally left blank Page39
© Copyright 2024