Terms of Reference for Quantitative performance evaluation population based sample survey
Terms of Reference for Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program A. Background: The Strengthening Household Ability to Respond to Development Opportunities II (SHOUHARDO II) Program is funded through United States Agency for International Development (USAID) Food for Peace (FFP) Title II, operating from June 2010 to May 2015. The five-year Multi-Year Assistance Program (MYAP) builds on the previous phase (SHOUHARDO) which established an effective, integrated model for reducing child malnutrition while contributing to greater livelihood security and women’s empowerment. Funded by the USAID, Government of Bangladesh (GoB) and CARE USA for approximately USD 130 million. SHOUHARDO II is one of the world’s largest nonemergency food security programs and plays an influential role in Bangladesh’s poverty alleviation efforts. The program is implemented in four regions (North Char, Mid-Char, Haor and Coastal), reaching 11 districts, 31 Upazilas, 172 Unions and 1,573 villages (see map in Annex 2). The overall goal of SHOUHARDO II is to “transform the lives of 370,000 Poor and Extreme Poor (PEP) households in 11 of the poorest and most marginalized districts in Bangladesh by reducing their vulnerability to food insecurity.” Sixteen Partner NGOs (PNGOs) are responsible for 90 percent of the program implementation, while CARE Bangladesh implements the 10 percent through direct delivery. While technical and operational capacity varies somewhat among PNGOs, each benefit from significant administrative and technical support from CARE. SHOUHARDO II maintains close working relationships with a number of technical partners, whose skill sets compliment and broaden CARE-B’s expertise. This approach allows SHOUHARDO II to leverage technical skills and experience that provides beneficiaries with access to relevant technical expertise, technology, etc. These include the International Rice Research Institute’s Cereal System Initiative for South Asia (IRRI CSISA), Chittagong Veterinary & Animal Science University (CVASU), International Union for Conservation of Nature (IUCN), Fruit Tree Improvement Program (FTIP) and Conservation of Black Bengal Goat (CBBG) Units of Department of Animal Breeding and Genetics Bangladesh Agricultural University Mymensingh, World Fish, and the Regional Integrated Multi-Hazard Early Warning System (RIMES). Each of the partners’ primary focus is on SO1, with the exception of RIMES, which supports early warning activities under SO5. The program partners with the GoB through Project Advisory and Coordinating Committees (PACC) at multiple levels as well as through government provision of technical training provided to field staff and beneficiaries on key topics related to agriculture, livestock, fisheries, health and disaster risk management. In order to achieve the SHOUHARDO II goal, USAID, CARE-Bangladesh, and PNGOs have established the following Strategic Objectives (SO) and Intermediate Results (IR): -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 1 of 17 SO1: "Availability of" and "access to" nutritious foods enhanced and protected for 370,000 PEP households. - IR1.1: Improved and diversified agriculture systems developed and linked with private and public services. - IR1.2: Increased household income among PEP in the target communities. SO2: Improved health, hygiene and nutrition status of 281,0001 children under 2 years of age. - IR2.1: "Access to" and "utilization of" health and nutrition services improved to care givers of children under 2 years of age. - IR2.2: Care givers of children under 2 adopt improved health, hygiene and nutrition behavior and caring practices. SO3: PEP women and adolescent girls empowered in their families, communities and Union Parishad. - IR3.1: Influence of PEP women and adolescent girls in decision making increased. - IR3.2: Local support systems strengthened to reduce Violence Against Women (VAW). SO4: Local elected bodies and government service providers responsiveness and accountability to the PEP increased. - IR4.1: Nation Building Departments (NBD) and Union Parishads proactively work to address the needs of the PEP, especially women. - IR 4.2: PEP access to entitlements and services increased, including safety nets and natural resources. SO5: Targeted community members and government institutions are better prepared for, mitigate, and respond to disasters and adapt to climate change. - IR5.1: Disaster contingency systems in place and functioning. - IR5.2: Influence local and national humanitarian assistance initiatives. B. Purpose: The purpose of the “population based” quantitative performance evaluation survey is to collect and compare data on the key indicators with the baseline study and mid-term survey data to measure performance of SHOUHARDO II strategic objectives, and intermediate results. The primary audience of the survey report will be CARE, as well as USAID (in particular FFP/Washington and USAID/Bangladesh) and CARE program partners. The Government of Bangladesh (GOB) is also a secondary user of the findings of the survey. Findings from the survey will be used to draw lessons learned for the selection, design, and implementation of future programs in Bangladesh. CARE will also make extensive use of findings from the survey to document and disseminate best practices and lessons learned. C. Scope of Work (SoW): The quantitative performance evaluation survey shall be undertaken as a “population based” household survey, using the sampling method proposed below and in Annex 1. The survey must follow the baseline and mid-term sampling methods and the sampling frames. 1 Revised to 176,706 (Ref. PREP 2102) -------------------------------------------------------------------------------------------------------------------------------------- ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 2 of 17 There will be two distinct components to the survey – the first taking a statistically representative sample of the populations where the program only targets the PEP households, including food transfers to PEP pregnant women and lactating mothers (following the MCHN approach); and the second population where the Program transfers food rations to all pregnant women and lactating mothers in the population, regardless of socio-economic status (following the PM2A approach). The survey will also be stratified by four regions - Coast, Haor, Mid Char and North Char. Annex 1 is referred for further details on sampling approach. 1. The contractor must follow the appropriate Performance Indicator Reference Sheets (PIRS) developed by Food for Peace (FFP) and Feed the Future (FtF), to finalize the survey instrument and methodology. In designing the instrument the contractor must consider including all disaggregate levels required by CARE and USAID/ FFP. They should also follow the USAID Evaluation Policy, M&E requirements and compliance, data quality, FtF Agricultural Indicators Guide, FANTA technical requirements, etc. 2. The survey instrument used in the baseline survey should be the starting point. If there is a need for rephrasing any question and changing style, the contractor must add a question with revised statement instead of changing an existing one. Changing a question will void the comparability. To capture performance results for new indicators (e.g., number of hectare) the contractor would require to add new questions. In addition, in consultation with SHOUHARDO II staff and the USAID/Bangladesh Mission, the contractor may add more questions to the survey instrument. 3. A Survey plan must be prepared and get approved by CARE and USAID/B before the survey implementation. The design document should include sampling strategy and sample size estimation, sampling frame and household listing, data treatment and analysis plan, training of enumerators, supervisors and others, field testing of the instruments, and oversight and quality control mechanisms. The contractor must specify the details of the sampling design in the survey design document in advance of field implementation. This document must include all of the following elements: i. The principal indicator and associated target group that will drive the sample size calculation for the entire survey. ii. The contractor should show the equation used for this calculation and the parameters used in the equation, including the design effect assumed for the principal indicator driving the sample size calculation. The calculation should take into account statistical power. iii. The number of households to be sampled in order to achieve the desired sample size for the target group (assuming that households may contain more than one or no eligible members from the target group). The contractor should give an indication of how the base sample size will be adjusted to account for the number of households that need to be visited and number of children 0 – 5 years of age (by age cohort) to be measured and data collated. Design effect should be used from the baseline and mid-term survey data. iv. The number of households to be sampled to account for anticipated household non-response. The contractor should indicate by how much the number of households to be sampled will be pre-inflated to account for household nonresponse. -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 3 of 17 v. Geographic or other criteria for stratification. The contractor should specify all stratification criteria and the total number of strata for all criteria. vi. The number of stages of sampling to be used. vii. Explanation of how the number of clusters and of households per cluster in the sample will be determined. viii. Explanation of how individual members in a sample household will be selected for interview (this is particularly important for anthropometric indicators, questions related to women at reproductive age, questions related to infant and young child feeding practices and farmers) ix. Definition of the clusters. The contractor should use tables to show the number of clusters that will be selected for each stratum. x. Explanation on the source of the information for the sampling frame, e.g. census lists or other national or internationally-sponsored surveys, such as the Demographic Health Surveys (DHS). The contractor should indicate how reliable and recent the frame information is. xi. A Probability Proportionate to Size (PPS) sampling mechanism should be used to randomly select the clusters. The contractor should use the number of households per cluster as the size measure and include a table of size measure and another showing the final list of selected clusters along with their probabilities of selection. Sampling: The contractor should finalize the number of household to be interviewed. A design effect of 2 may be used to estimate the sample size. The contractor should calculate the actual design effect from the baseline and mid term survey data and use those to estimate sample size more accurately. ‘Stunting’ is chosen as one of the prime indicators because it provides enough households to measure target change levels for most indicators. If the sample size derived using stunting is not found adequate to measure exclusive breastfeeding (EBF) indicator for children ages 0-5 months and the minimum acceptable diet (MAD) indicator for children ages 6-23 months then the contractor must indicate a reasonable plan to address this issue. Annex 1 – provides more detailed on sampling. The contractor must obtain approval from CARE and USAID/B on the sampling method and proposed sample sizes. xii. Indication that the contractor will use systematic sampling to select dwellings within clusters. This implies that for the sampled clusters, a list of all households, with household identification and location indicated, within these clusters must be obtained through either a preliminary pass on the cluster prior to interviewing or other existing sources. 4. Data Treatment and Analysis Plan: The contractor must prepare a data treatment and analysis plan to address the following elements: i. Indication of how and when data will be entered into the database, as well as the software, entry screen and intelligent controls, to be used for data entry and minimize entry error. Double-data entry is required; If smart phone, PDA, or tablet is used to capture data, name of the application and the strategy to double-check the data on a regular basis so that any inconsistencies can be identified immediately and corrective measures can be taken within a day; -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 4 of 17 5. 6. 7. 8. ii. Systematic Data quality checks, examine inconsistencies and edits (data cleaning, checking missing values and out liar and fixing issues) planned to ensure logical consistency and coherence, as well as an indication of the software and data entry screen to be used; Sampling weights to be included on the data file. The formulae used to calculate the sampling weights should be included as part of a data dictionary document. Different sampling weights will need to be calculated for separate analysis of each component (PM2A vs MCHN/ non PM2A) and of the program level aggregate. Note that a household non-response adjustment should be made to the sampling weights as part of the final weighting system and description on how the out-liars would be addressed; Indicator tabulation plan. Estimates should be produced for each stratum and for the overall level; Indication of which sub-groups, if any, for which the Contractor will produced estimators; i. To understand factors that explain the variation in change in stunting, household hunger scale, household dietary diversity score, and minimum acceptable diet, multivariate analysis model must be specified and presented in the tabulation plan. ii. The contractor should specify all intended bivariate and multivariate analysis in the tabulation plan; Thoughts on intended descriptive tables and cross-tabulation plan, correlation, regression analysis, weighting and various statistical tests like confidence interval, test of significance, p-value, standard deviation; dealing out-liars, etc. iii. Indication that confidence intervals associated with the indicators will be produced alongside the indicator estimates and that these will take into account the design effect associated with the complex sampling design. Additional statistical outputs are required for multivariate analysis, but should be provided in an appendix; and iv. Software to be used for data analysis and for conversion of anthropometric data into Z-scores. v. Description of methods for comparing the final survey data with the baseline and mid-term survey data, and tests to be used to detect a population level difference at 95 percent level of significance. The tools and methodologies should be finalized having concurrence from SHOUHARDO II management and USAID/ FFP, OFDHA/B. Field Procedure Manual: It is expected that the contractor will develop a field manual and precise definitions on technical terminologies to be used as part of the training materials for survey enumerators and supervisors and serve as reference material for staff in the field conducting the survey. The field manual should include instructions on how to sample dwellings within clusters, households within dwellings, and select individuals within households and importance to follow the list of sampled households. The manual should also give recommended best practices for conducting interviews and dealing with specific challenging situations, e.g. households that refuse to participate, and provide a household and individual respondent non-response follow-up strategy. The manual should describe the steps and process on use of weighing scales, height measurement boards, check-list and standardization. The manual should also describe the roles and responsibilities of the enumerators, supervisors, and other field staff and contain a detailed explanation of how to properly administer each question in the questionnaire. -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 5 of 17 9. The survey team is required to ensure statistically representative data collection within the household survey and anthropometric measurement with appropriate representation of regions, as well as PM2A and MCHN locations. 10. For the anthropometric data collection, the contractor must use international standard height boards and scales and ensure none produces error data or malfunction in any day during the survey. 11. To comply with USAID’s Open Data Policy, USAID/FFP will host the data to USAID’s Open Data portal. To comply, the contractor must submit the following: i. Raw data and the cleaned data files with all of the computed variables both in SPSS and CSV formats ii. SPSS (v20 or later) and STATA Outputs and Syntax files and weighting files in Microsoft Excel iii. Submit a data dictionary - essentially a definition and description of any of the fields provided in the dataset iv. The contractor must ask all the respondents of the survey for their consent to release their birth dates and any other identifying information and provide accurate data required for the survey. 12. Respondents. The contractor should ensure that individual enumerators interview and collect survey data only from the appropriate member(s) of the sampled households The contractor is also responsible for the following: - Hiring of all qualified and competent team members for the survey. This includes– team leader, technical experts, statistician, quality control officers, enumerators (including local), data entry operators and editors, and all others (for further details, please see an indicative team composition below) - Organizing training and field exercises for the team members - Designing data entry screen with intelligent control and testing - Finalizing Survey Questionnaires and manual - Printing of all survey materials such as Survey Questionnaires, Manual, List of sampled villages and households, check list, report, de-brief materials - Managing required number of Survey instruments (weighing scales and height/ length measurement boards maintaining international standard), standard weights, wooden boards, etc with back-up plan - All types of logistic arrangement including lodging, foods, transport The survey team must follow the FFP, FtF, FANTA and other associated guidance, process and requirements applicable for the quantitative performance evaluation population based sample survey. The data collection tools, sample selection, survey instruments, the work plan and all other critical aspects pertained to the survey should be finalized only after concurrence from SHOUHARDO II management and USAID/ FFP, and OFDHA/B. CARE would have full access to observe and review training sessions, survey preparation, instruments, field interviews and measurement, electronic data, data analysis and so on (if/ as needed by them at any point of the survey) and suggest for modification, which the contractor should follow. -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 6 of 17 D. Composition of Survey Team and Qualifications: Below is an indicative (not inclusive) team composition. The team should have a good mix of both women and men members including enumerators. The contractor must provide a detailed and more specific plan on team composition and numbers. The proposal should reflect the positions and roles of team members and describe the educational qualifications, years of experience, level of skills and competencies, background for each member during proposal submission: The contractor must ensure that only the qualified candidates are engaged for this survey. The composition and number of the team members has deliberately not been exactly defined, leaving it to discretion of the evaluation team, but suggest the following composition as a minimum: Evaluation Team Leader (1), Survey Expert (1), Technical/Sector Specialist (4 – 5), Statistical Expert/ Data Analyst (1), Field Survey Supervisor and Quality Control Officer (20 @ one per survey cluster), Data Editor (20, 1 per team), Data Entry Operators (data collection, entry, cleaning, etc should be done simultaneously and contractor should plan accordingly) and Enumerators (120 @ 6 per survey cluster). However, this is subject to CARE concurrence. The team members must have experience in managing large size of socio-economic and health and nutrition, anthropometric quantitative surveys for evaluating Title II development food assistance programs in Bangladesh or south Asia. The evaluation team must include a healthy mix of men and women; and expatriate individuals with substantial experience in Bangladesh and Bangladeshi experts. The survey required local enumerators having technical skills and fluent with local dialect and conversation. It is expected that the team will have expatriate and national members with experience and expertise in multiple technical and cross-cutting areas, such as: Sound expertise in designing and implementation of large scale of Socio economic and HHN survey especially on food security, anthropometric, women empowerment and local governance, CCA, is the prior focus, more specificallyMaternal and child health nutrition Food security and livelihood strategies Rights and governance Disaster risk reduction and preparedness Environmental considerations and climate change adaptation Gender equality and equity Community participation Sound in dealing with large volume of complex data sets and analysis for on above areas, draw sampling applying standard process and software i) Evaluation Team Leader: The team leader must be an expatriate with multi-dimensional proven skills and experience preferring food security and socio-economic and/or HHN anthropometrics, BCC. S/he should have minimum post-graduate degree in a relevant field and at least 15 years’ experience in international development and such surveys. S/he must have proven experience in carrying out USAID/ FFP funded program evaluations using quantitative methods. S/ he should be familiar with USAID compliance, data quality, USAID Evaluation Policy. M&E requirement FFP, FtF and FANTA technical requirements and guides. The team leader will be responsible for coordinating all evaluation activities, supervising the team, meeting all specified objectives, evaluating and monitoring systems, ensure data -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 7 of 17 quality and proper recording and analysis, collaborating with each partner, presenting the evaluation results, and submitting drafts and final reports according to the defined timeline. In addition to evaluation expertise, the team leader must have expertise in at least one of the sectoral or cross-cutting components and extensive experience of designing and leading large scale population based socio economic and anthropometric sample surveys. The evaluation team leader will require to take specific responsibility for assessing and analyzing the programs’ progress towards targets (in line with SHO II result framework and IPTT indicators), and potential benefits or likelihoods of achieving the outcomes. The evaluation team leader will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables in consultation with the team members. S/he will provide leadership to the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final report including recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to USAID/Bangladesh, GoB, MYAPs, CARE-B and key partners. The evaluation team leader will submit the draft report, present the report, and, after incorporating USAID & CARE-B staff comments, submit the final draft report to CARE and USAID/Bangladesh within the prescribed timeline. Evaluation team leader should have strong communication skills in English – written and oral. S/he should have ability to manage large team with multi disciplinary back ground and capacity to conclude the evaluation meaningfully. ii) Technical/Sector Specialists: The technical/sector specialists in the team must possess post-graduate degrees in the respective technical, sector or cross-cutting area that s/he will be assessing. Individual team members must have at least 15 years’ experience in international development, demonstrated technical skills and proven experience in the subject matter and conducting USAID/ FFP funded projects and program evaluations using statistically representative quantitative methods. The team must have one Socioeconomic/ Food security international expatriates, one HHN/ anthropometric international expatriates, and one international Statistician/ Data analyst. Other may be either national or international. It is expected that some of the members may cover more than one technical or cross-cutting area provided that they have demonstrated expertise and proven experience in the areas. iii) Survey Expert: S/he must have minimum post graduate degree (preferably in statistics, economics, social science) with minimum 10 year proven skills and well experienced to manage such a large and complex survey from designing to implementing in real field. The survey expert should have expertise in quantitative survey design and implementation, guide the team, control data quality and reporting. The survey expert should also have extensive skills in sampling, designing survey instruments, enumerator training, anthropometric survey, data analysis, interpretation and reporting. S/he should be well conversant on USAID/ FFP, FtF ( Feed The Future) Agricultural Indicators Guide and FANTA technical requirements and compliance, M&E requirements, USAID Evaluation Policy, prime indicators, data quality and such surveys. iv) Statistical Expert/ Data Analyst: The Statistical Expert/ Data Analyst must be an international expatriate having minimum post graduate degree (preferably in statistics, economics, social science) with minimum 10 year proven skills and experience in dealing with large volume of complex data sets in particular socio economic and -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 8 of 17 anthropometric dataset using latest SPSS, STATA, Anthro and MS Excel. S/he should be a real expert for data treatment, cleaning, processing, analyzing (descriptive, crosstabulation, correlation, bivariate, multivariate and regression analysis, weighting, etc) and conduct various statistical tests like confidence interval, test of significance, p-value, standard deviation and generate outputs properly. v) Field Survey Supervisor and Quality Control Officer: S/he must have minimum graduate degree with 5 years experience in the relevant field. S/he must have practical experience and conceptual clarity on team management, planning, team management and supervision, ability to resolve field problem, convincing ability and communicating skill. S/he should have good problem identification and writing skill (English) to update/ communicate the day-day field activities. S/he should have strong experience and skills on systematic data testing/ review both at field and desk in particular socio-economic and anthropometric survey and report back concisely and clearly. S/he should be a good team player with strong leadership ability to uphold team spirit and ability to work under pressure/ hardship, respectful to the people, good listener and strong questioning skill and ability to address and conclude manage field problems. The individuals should be familiar with regional/ local context, culture and conversion units. vi) Enumerators: The enumerators must have minimum graduate degree with 5 years experience in the relevant field, who carried quantitative socio-economic surveys and anthropometric measurement in rural areas hard to reach. S/he should be good team player and can work under pressure/ hardship, respectful to the people, good listener and possesses strong questioning skill, good hand writing and ability to manage field problems. The individuals should be familiar with regional/ local context, culture, conversion units and dialects (can speak fluently and understand well). The contractor should hire only those enumerators, who will qualify in post training test. Thus, they should include an additional number of enumerators in the training to get the required number after applying a standard screening process. E. Timeline: The quantitative end-line population based survey shall be undertaken between November and December 2014. Major Events CARE Bangladesh (CB) received details complete proposals including personnel to be engaged, how the Quantitative performance evaluation population based sample survey will be undertaken, and the budget CB provides feedback on selected proposal to Survey Team Contract agreed and signed by both parties Survey Team starts assignment Survey team holds meetings with SHOUHARDO II management to better understand the Program to help design the methodology and tools to be used in the Survey When / Deadline July 2014 August 2014 by no later than September 2014 October 2014 20-23 October 2014 -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 9 of 17 Major Events Survey Team submits detailed questionnaires, tools and manual/ guidelines to be applied in end-line survey to CB in English. These should be translated with back translation to Bengali for survey teams who will be working at ground CB approves submitted questionnaires, guides and methodology Training of Enumerators by Survey Team on tools and questionnaire and use of instruments (Weighing scale and Height/ Length measurement boards to be arranged by the firm) Revision of tools / questionnaires if required, and updating enumerators on changes “Quantitative” Data collection, anthropometric measurement and analysis (THIS PERIOD IS FIXED AND NONNEGOTIABLE) Presentation of survey results to CARE (date to be mutually agreed depending on progress on analysis) CB provides feedback to Survey Team on findings Survey Team to submit Final Report to CB (see H for deliverables) When / Deadline 28 October 2014 30 October 2014 03-06 November 2014 07-09 November 2014 15 November-31 December 2014 15 January 2015 31 January 2015 F. Proposal: Detailed proposals are expected from interested firms detailing the ‘methodology to be employed’; a ‘detailed work plan with timeline’; and a ‘detailed budget’, sample size with rationale and team composition with defined role, responsibility and functions , report structure, data quality assurance. A detailed profile of the firm and consultants and bench marks of the survey teams to be engaged must also be provided. SHOUHARDO II Program brief, Base line and Mid Term Evaluation Reports, SHOUHARDO Final Evaluation Report and selected publications are available on www.carebangladesh.org/shouhardoII G. Point of Contact: Once the contract is signed, the contact person at CARE is Marc Nosbach, Chief of Party, SHOUHARDO II, CARE Bangladesh. All queries should be directed to [email protected]. Technical guidance and queries will be provided by Wadud Abdul, M&E Coordinator, [email protected]. CARE’s responsibility: CARE will provide the following documents to the contractor: - Complete list of all working villages with the number of households per village to the contractor to construct sampling frame - List of households for the program working villages (updated in October 2014) - IPTT matrix - Survey reports and back-ground materials, which can be found from CARE/ SHOUHARDO web site -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 10 of 17 H. Deliverables: 1. The following deliverables are to be submitted to CARE within the timeline: The Contractor is responsible for: a) Pertinent permissions, insurance, and other required permits i). Obtaining all the necessary permissions for implementing the quantitative evaluation data collection. ii). Adhering to country and local formalities and obtaining any required permits related to data collection from human subjects and logistics of survey implementation, including any necessary Internal Review Board (IRB) approvals, as well as health and accident insurance, salary, and taxes for all enumerators and supervisors. Deliverable: Evidence of insurances and permits for implementing survey and other data collection activities in electronic form b) Survey plan including detailed survey implementation plan (DSIP) i). Specifying details for methodology, critical tasks, anticipated outputs, date-bound timelines, resource needs, and responsible person(s). Composition of a standard field survey team, including expected tasks and responsibilities of each team member, should also be described. ii). Detailing a sampling plan for the quantitative population-based household survey that responds to the elements specified in Section C3. Deliverable: Survey plan including sampling plan, and detailed implementation plan reviewed and approved by CARE, USAID/FFP, USAID/Bangladesh Mission. c) Quantitative survey instrument which must take into account the instrument used in the baseline survey. Additional questions can be added to the instrument if needed. Adapting the questionnaire to the local context if additional questions are to be added to the instrument. Translating the approved questionnaire instrument and manual from English into Bangla. Back translating the questionnaire from Bangla to English with a second translator to ensure it is accurately translated in Bangla. Making any necessary changes to Bangla questionnaire based on the back translation. The questionnaire used in the baseline survey is already translated into Bangla, thus the contractor does not have to translate the major part of the questionnaire. Deliverable: Final Bangla and corresponding English questionnaires and manual reviewed and approved by CARE, USAID/FFP and USAID/ Bangladesh Mission d) Data treatment and analysis plan a. Detailing a data treatment and analysis plan that responds to the elements specified in section C4. Deliverable: Data treatment and analysis plan reviewed and approved by CARE, USAID/FFP and USAID/Bangladesh Mission e) Raw and cleaned data set, data dictionary/codebook, edit rules, outputs and syntax for data analysis, including syntax for variable transformations. Deliverables: i. Raw data set in SPSS and CSV formats; ii). Edit rules for cleaning data; iii). Data dictionary/codebook; iv). Syntax for all data analysis and variable transformations; v). Final data set that includes cleaned data, sampling weights at each stage, final sampling weights, and all derived indicators; and -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 11 of 17 vi). Sampling weights used to tabulate the aggregate-level estimates for the USAID/FFP Standard Indicators vii) All Output files in SPSS (v2 0 or later) f) Briefings for the CARE and USAID Bangladesh. Presenting findings, conclusions, lessons learned, and recommendations based on the quantitative performance evaluation survey. Deliverables: i). Mid-term and final briefings to CARE and final debriefings to USAID Bangladesh g) Draft quantitative evaluation survey report i). Not exceeding 50 pages, excluding appendices and attachments. The draft report must be presented in English. ii). Presenting the estimates and confidence interval for all indicators (impact and outcome) at the SHOUHARDO II program level and by components (PM2A vs targeted food transfers to PEP pregnant and lactating mothers); iii). Using appropriate tests of differences, determine the change at the underlying population level with confidence intervals. Deliverable: Draft quantitative evaluation survey report reviewed and approved by CARE and USAID h) Final quantitative evaluation survey report: This report will be a revised version of the quantitative evaluation survey report that incorporates the comments of CARE, USAID/FFP and the USAID Bangladesh Mission. The final report must be presented in English. Deliverable: Final quantitative evaluation survey reviewed and approved by CARE and USAID (i. printed copies - 3 sets of full report with all Annexes - multi color and binding and ii. full version in electronic report including all annexes, data, syntax, outputs, etc) i) Reporting Format Final Report: The evaluation team will submit a final report that incorporates CARE and USAID comments and suggestions within 15 working days after receiving comments from CARE and USAID/B on the draft final evaluation report. The team will follow the format given in the reporting requirement section below. The format of the final evaluation report should strike a balance between depth and length. The report must include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, evaluation design and methodology, findings, conclusions, lessons learned, and recommendations. Any substantially dissenting views by any team member, CARE or USAID on any of the findings or recommendations, a copy of the Scope of Work, a list of persons and organizations interviewed, methodology, tools, sample strategy, list of villages sampled, and other necessary documents must be presented in the annex. The report must not exceed 50 pages, excluding annexes, and must be submitted electronically in English. The evaluation team will edit and format the final report as appropriate to ensure a highquality deliverable. The final report should meet the following criteria to ensure a highquality deliverable: The findings should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project and what did not. Findings should be specific, concise, and supported by data using appropriate statistical software. -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 12 of 17 The main body of the report should present an analysis of the findings for each objective area, interpretation of the findings, and promising practices. This should include elaboration on major indicators with comparison with Baseline, Mid Term and recent national data aggregated and disaggregated by key attributes such as in aggregated and disaggregated manner by key attributes such as PM2A vs MCHN area by region and by well-being category, by sex (women and men), by age cohorts (e.g., U2s, U5s, Young) and program overall at minimum. The survey results should include central tendencies (mean, median, mode), percentile, confidence interval, test of significance, p-value, standard deviation, etc as applicable. The discussion and survey results also include program contribution pertained to CARE Unifying Framework for Poverty and Social Justice, Bangladesh National Food Policy Goal; Bangladesh Climate Change Strategy and Action Plan and FFP and USAID Bangladesh Strategies and MDG (Report format is referred for further details); The format for the quantitative evaluation survey report is as follows: i. Cover page, Table of Contents, List of Acronyms, List of Figures, List of Annex, List of Abbreviation and List of Glossary (English and Bengali Terms) Unit of Measure (Conversion Chart Standard and Local) and Months (English and Bengali) ; ii. Executive Summary should be a clear and concise stand-alone document that states the most salient findings, critical observation/ lessons, conclusions, and recommendations of the evaluation survey and gives readers the essential contents of the quantitative evaluation survey report in two or three pages. The Executive Summary helps readers to build a mental framework for organizing and understanding the detailed information within the report; concisely state the project purpose and background, key evaluation questions, methods, most salient findings and recommendations and surprises; iii. Introduction should include purpose, audience, and synopsis of task; should provide a brief country context, including a summary of any relevant history, demography, socio-economic status, program purpose, relation to MDG and GoB Country etc; iv. Methodology should describe sampling design, study methods, data collection techniques, data quality assurance, data management and analysis process constraints and limitations of the study process and rigor, and issues in carrying out the study; v. Tabular summary of results should present quantitative evaluation survey results in table form for all the indicators by components; and also by region; vi. Findings should present findings on all of the key indicators. Quantitative evaluation survey values must be presented in quantitative format and complemented by descriptive analysis for each stratum and at the aggregate SHOUHARDO II program level; vii. Conclusions and Recommendations should provide additional analysis of the data and results, drawing out programmatic and organizational recommendations for future program design. Recommendations must be relevant to program and context; viii. Issues should provide a list of key technical and/or administrative, if any, for SHOUHARDO II for which the quantitative evaluation survey was conducted; and -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 13 of 17 ix. Annexes should document the study methods, scope of work, schedules, interview lists and tables and be succinct, pertinent, and readable. All the Annexes should be succinct, pertinent, and easily readable and understandable. Annexes should also include if necessary, a statement of significant unresolved difference of opinion by key CARE Bangladesh and FFP/Washington, USAID/Bangladesh, GOB (line ministry), or members of the evaluation team on any of the findings or recommendations. x. References, including bibliographical documentation, meetings, interviews, and focus group discussions; xi. List of stakeholder group with number, type, and date of interactions; xii. Data collection instruments in English and the local language; xiii. Data sets in electronic format and data weighting files; xiv. Data dictionary and program files used to process the data in electronic format; xv. Other special documentation identified as necessary or useful. xvi. Data Tables showing progress against the IPTT (Indicator Performance Tracking Table) and SAPQ (Standardized Annual Performance Questionnaire) indicators (by region, PM2A/ MCHN, WBA- PEP/ Non-PEP), MDG Indicators in MS Excel v2007 spread sheet using the IPTT template; xvii. IPTT matrix and SAPQ should be presented in separate data tables with updated evaluation results for selected indicators (CARE will provide the list of indicators). The data should be presented in aggregated and disaggregated manner by key attributes such as PM2A vs MCHN area by region and by well-being category, by sex (women and men), by age cohorts (e.g., U2s, U5s, 0-5 months - as applicable) and program overall at minimum. The IPTT matrix should include central tendencies (mean, median, mode), percentile, confidence interval, test of significance, p-value, standard deviation, etc as applicable and reflect comparison against Baseline, MTR and Quantitative performance evaluation population based sample survey results. j. Data entry template: The contractor should provide the data entry template used for entering/ computing the survey data k. Data Quality: The contractor must provide a detailed plan and systematic approach on data quality review at primary sources; approach and steps to address the issues; cleaning electronic data and outputs. They should also share the field findings and actions to be applied to ensure data quality across. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with one-inch page margins at top/bottom and left/right. -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 14 of 17 Annex - 1 Sampling approach for population based quantitative evaluation survey for SHOUHARDO II Program CARE Bangladesh This proposed sampling approach is formulated referring to the methodology followed during baseline in line with FANTA Sampling Guide, 19972. This must maintain USAID/ FFP requirements and allow to make a scientific comparison between the PM2A and MCHN approaches and other key attributes. The quantitative evaluation survey aims to conduct a population based household sample survey. The sampling method is multistage cluster sampling. This also aims to sample an equal number of villages (cluster) and households for both PM2A and MCHN approaches. All these efforts will allow to make a comparison between the PM2A and MCHN target areas within and across the four regions – comparison with SHOUHARDO results is also intended. The survey intends to distribute an equal number of sample size across the 4 regions (Coast, North Char, Mid Char and Haor) in a way that is proportional to the size of the population of the region rather than just taking the same number from each region. The same principle will be applied to distribute the sample size between the two treatment strata of PM2A areas (villages) versus MCHN/PEP areas (villages). The data needs to be weighted for each of the four regions and PM2A/MCHN sites. The weights are to be computed as the total number of households in each area/sampled number of households in each area. Sampling survey location The survey will apply probability proportionate to size (PPS) for selecting the sample, with villages as the sampling unit. A total of 50 villages from each of the four regions (25 villages for PM2A area and 25 for MCHN area) will be sampled. This will total to 200 sampled villages across the Program (of which, 100 villages will be PM2A areas and the remaining 100 for MCHN areas). Selection of sample households for both socio-economic and HHN survey (+ anthro measures) The survey will select a statistically representative sample to compare proportion variables between PM2A and MCHN sites and across the four regions. The indicator on 'Stunting' will be used to make the calculation. To measure a 10 percentage point change in a proportion with initial value of 0.5 with 95 significance level, 80% power and 2 design effect, the minimum sample size is 606. Adding 10% cushion for non-response resulting in a sample size of 6663 households needed for the socio-economic survey, per region. The sampled households will be selected from the sampled clusters equally applying simple random sampling methodology. The contractor is also suggested to use the actual nonresponse rates from the baseline and midterm surveys to calculate the sample size more accurately. 2 The contractor should follow later version or addendum (if available) and must comply with USAID/ FFP requirements 3 The minimum sample size needed to measure a 10% change in a proportion with 95% level of significance and 80% power is 606. A non-response adjustment factor of 10% gives a minimum total initial sample size of 666 households per region. [FANTA Sampling Guide by Robert Magnani for details.] -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 15 of 17 The following parameters will be used to determine the sample size. Za=.95 Zb=.80 P1=.5 P2=.4 D= 2 To keep the sample size reasonable, the survey will select one single sample which will be used to conduct the survey combining both the health questions, anthropometric measurement and the socio-economic questions. This will be calculated using the proportion of the population in Bangladesh that is aged 0-59 months and 6-59 months and the average household size. The final evaluation survey must use the statics from the SHOUHARDO II baseline survey. Therefore the sample size will equate to 606/(11.9%*5)=1,017 or may require to differ slightly due to valid logic. Hence, the survey needs to visit 1,017 households in order to find 606 children aged 6-59 months. Adding another 10% for non-response cushion, the sample size will finally be 1,119 households per region - see Table 1 below. The contractor should finalize the sample number with rationale. Table 1 Region 1 Region 2 Region 3 Region 4 MCHN areas Combined SES + Health survey 1,119 Combined SES + Health survey 1,119 Combined SES + Health survey 1,119 Combined SES 4,476 + Health survey 1,119 PM2A areas Combined SES + Health survey 1,119 Combined SES + Health survey 1,119 Combined SES + Health survey 1,119 Combined SES 4,476 + Health survey 1,119 Sub-Total HHs to visit 2,238 2,238 2,238 2,238 Grand total Totals 8,952 8,952 The households will randomly be sampled from the entire population using the above equation. 44-45 households will randomly be selected from each of the sampled villages, with equal distribution for both PM2A and MCHN areas. This will equate to a total of 8,952 4 households for the entire survey (4,476 households for PM2A and 4,476 for MCHN sites). 4 The number may be varied slightly based on the sampling method/ design -------------------------------------------------------------------------------------------------------------------------------------- ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 16 of 17 Annex 2: SHOUHARDO II Program Areas -------------------------------------------------------------------------------------------------------------------------------------ToR- Quantitative performance evaluation population based sample survey for USAID Title II CARE SHOUHARDO II Program Page 17 of 17
© Copyright 2024