4. HOW do we Monitor and Evaluate Nutrition Interventions?

Monitoring and Evaluating Emergency Nutrition programs

Monitoring is the ongoing collection, analysis, and interpretation of program data, this includes monitoring activities (inputs), services (outputs) and outcomes (Results). The primary purpose of monitoring data is to assess one’s performance and for program improvement. The aim of monitoring is that the data reported should be reviewed, acted upon on an ongoing basis to inform program implementation (to modify it as per the need). Without a monitoring system in place, you cannot track or demonstrate the results of the program. If monitoring is done properly, it will also allow for trend analysis and cross comparison between different times, seasons, and geographic space.

Evaluation is a process of data collection designed to assess the effectiveness of the project in attaining its originally stated objectives and the extent to which observed changes are attributable to the project. It’s often done at the end of the project but could be planned at strategic periods during the project life span. It can use rigorous study designs e.g., experimental design or quasi-experimental-involving control groups.

Both Monitoring and Evaluation need clear stated goals and objectives and usually have four components: Inputs, processes, outputs, and outcomes. Inputs are the set of resources dedicated to a Program including human and financial resources, physical facilities, equipment, and operational policies that enable services to be delivered (i.e., staff, facility, equipment’s, storage rooms, supplies etc.). Process refers to the set of activities which are carried out to achieve the objectives of the project such as procurement and distribution of ready to use food rations, community mobilization, screening conducting IYCF and CMAM trainings etc. Outputs are the results obtained because of the execution of activities such as number of children who received treatment for acute malnutrition, numbers of staff performing better as a result of training, number of people having access and using nutrition services, number of functional and operating nutrition centers after support etc. Outcome is the set of results expected to occur at the population level due to Program activities and generation of Program outputs. The intermediate outcomes often result directly from project outputs such as reduced number of children with SAM (due to increased access to treatment services). The long-term outcome often comes as result of the actions of the intermediate outcomes such as drop in prevalence of malnutrition in the area over time, reduced child mortality, morbidity, and better health.

The M&E plan

The purpose of the M&E Plan is to serve as a framework for activities to demonstrate accountability and improve the quality of activity implementation and outcomes for participants. The M&E Plan should serve as a roadmap for activity staff, documenting M&E approaches and processes in sufficient detail. It should demonstrate that you have a rigorous system for monitoring and evaluating activity performance in a way that produces accurate, reliable, and useful data in a timely manner for decision making’.

Components of a M&E Framework and Plan

To develop a Monitoring and Evaluation framework you need to determine your objectives and what nutrition intervention you want to implement. Based on this you decide what is to be monitored and evaluated (The activities); who will be responsible for monitoring and evaluation of the activities, the resources you will need, when monitoring and evaluation activities are planned; and how they will be carried out. You can follow these steps.

Determine indicators for monitoring and reporting of nutrition programs based on requirements outlined in the nutrition proposal (objectives) or response plan. The indicators must include those specified in the project proposal and you must be able to describe them and how they are computed in the M&E plan. It’s important to disaggregate indicators into sex and age groups especially as most donors have specific requirements for this type of disaggregation. The indicators selected must provide evidence that defines the extent to which project interventions are successful in achieving the set objectives. Make sure that the specific purpose of indicators to be collected is relevant for decision-making and avoid collecting data, indicators and information that will not be used or is already being collected by other sectors.
Determine the data collection methods – Map out if you will be using a digital platform or manual registration systems and consider existing capacities such as internet connectivity, training level of people collecting the data etc. Following that map out the resources you need to build the capacity of relevant personnel to collect this data using agreed methods and consolidated tools for cross comparison over space (geography) and time. Any disaggregation (e.g., by age, gender, economic status/income, or geographic location) should be factored into tools of data collection to enable the system to capture these.
Determine who will be responsible for the data collection – this is the person who will collect the primary data and who is responsible for checking its accurately recorded and captured in the data collection tool (registers, books etc.)
Determine the frequency of data collection – this can be based on the normal time of service delivery, for example the time of OTP service delivery (weekly) or based on when it’s feasible to collect it. For example, since recovery can take time, collecting this data from registration books on a monthly basis is recommended to get a meaningful analysis.
Determine the Means of verification of the data (the source of the data) – This is where the data is stored, the primary source of it. This can be things such as training reports, OTP registers or cards, IYCF KAP survey report or counselling reports etc.

You can also incorporate into the M&E framework a workplan of when activities are supposed to be conducted, a baseline, a target and milestone (i.e., mid and end projects) for each indicator, this is especially useful if some of the indicators are captured through assessments such as SMART or KAP surveys.

Source of Nutrition information

Nutrition programs draw data and information from direct sources as well as from other information systems. The main nutrition data and information sources include:

Routine data collected through the nutrition program or the health information systems. This includes data on how many children were screened, how many were admitted to programs, their outcomes, data on supplementation and training etc.,
Population based data includes large scale surveys such as National Demographic and Health Survey, Multiple Indicator Cluster Survey (MICS), nutrition SMART surveys, Maternal Infant and Young Child Nutrition (MIYCN) KAP assessments, CMAM program coverage surveys among others.
Early Warning System – These can include joint Food Security and Nutrition Seasonal Assessments including situation analysis with Integrated Phase Classification for acute malnutrition and food Insecurity (IPC AMN and AFI) protocols, Mass screening data, Special studies, and operational research.

Evaluation of Emergency Nutrition Programs

An evaluation process demands formulating the objectives against which the program is going to be evaluated. For instance, from an implementation agency perspective the question could be, is the program performing as expected? and from a donor’s perspective the question could be ‘is the program worth continuing or extending’? Therefore, the question “How do we tell if a program has an effect?” is incomplete without knowing why one needs to know. Common reasons for doing an evaluation are to decide whether to continue the existing program or not, to redesign the program if necessary, or to decide whether to do similar program elsewhere. Whether you do this evaluation internally or externally there are a few important evaluation criteria you must consider including relevance, effectiveness, efficiency, coverage, quality of services and impact. Since most evaluations tend to be done by external partners you must ensure you are meeting all the evaluation criteria and design your nutrition intervention in such a way that is aligned to these criteria.

RELEVANCE/APPROPRIATENESS – A national nutrition strategy and strong government commitment supporting the relevance and sustainability of CMAM programming is essential. You must show how the specific nutrition interventions are in line with the national nutrition strategy and how they have contributed to strengthening the strategic coordination of the government to reach the most vulnerable. This means the program should be implemented in areas where the need is greatest as identified by the Ministry of health or the nutrition cluster. It is important that you show how you contributed to national capacity-building of health professionals and community workers, to policy and system development and to the engagement of other key stakeholders.

EFFICIENCY – Many of the ways of maximizing the efficiency of a nutrition program include:

Prepositioning of stock is cost effective in the sense that if you buy RUTF in bulk this will serve as buffer stock. In case of an emergency or a scale up it will save both freight cost and time as it takes on average 2-3 months to procure RUTF internationally. If the supply chain is not reliable there will be interruptions in treatment protocol and affect the quality of program implementation. Hence for the programs to be efficient there is a need to preposition supplies, and ensure stock availability by keeping contingency stock
Use of Existing infrastructure and resources: As economy is related to how to minimize the costs of inputs to reach desirable outputs. One shining example of this is the community-based nutrition program that uses existing health service platforms to deliver nutrition specific interventions. The utilization of health care staff and community volunteers to carry out these services is much cheaper than employing staff or setting up nutrition centers. Hence you should ensure nutrition is well integrated with Health and CMAM is delivered through the health service delivery points. The health care staff normally require training only and the use of health community volunteers is free. Hence transforming CMAM into a permanent and regular program integrated with the national health system is the most sustainable option country-wise.
Cost per beneficiary is also important to look at. Treatment of severe acute malnutrition is generally an expensive program which on average globally costs $200 USD/child. This cost varies from country to country but can be reduced if Ready to use Therapeutic foods (RUTF) are sourced locally in countries producing a high quality RUTF. Projects with smaller beneficiary caseload and high activity cost are not good value for money. Ideally the implementing partner should have a high beneficiary number with lower costs; however, cost per beneficiary can rise in the case of refugee camps which have higher costs due to infrastructure costs (clinics). National NGOs may also have lower beneficiary costs due to the lower human resource costs than international NGOs. The cost per beneficiary can be calculated by dividing the total beneficiaries under a CMAM component with the cost of the activity. Child health days for the delivery of essential nutrition intervention such as Vitamin A supplementation, deworming can rapidly enhance coverage much more than standalone campaigns, hence its more efficient use of resources.
Efficient use of Ready to use therapeutic and Supplementary Food (RUTF, RUSF) is also an essential part of efficient use or resources. If RUTF/RUSF is misused, sold, or shared with other household members, then the child will not gain weight and the length of stay in the CMAM program will be longer. This will also increase re-admission rates; hence for the efficient use of RUTF/RUSF, monitoring and educating caregivers is important to improve program quality.
The short-term stop-start gap funding arrangement of nutrition programs is not likely to yield substantial reduction in GAM unless a multi-sectoral long-term strategy is developed and implemented. A more preventive approach (nutrition sensitive programs) and longer-term financing (multi-year) is likely to save emergency funds, strengthen the national health system, and improve nutritional status in the country. Country offices should be able to pass on the benefits of multi-year financing to their implementing partners. Drawing a line between emergency and development funding is not conducive to bridging the gaps and ensuring sustainability.

EFFECTIVENESS – Community based Management of Acute Malnutrition (CMAM) is one of the 13 highly cost-effective direct nutrition interventions identified in the scaling up nutrition framework for action. Supporting community-based interventions increases cost-effectiveness, coverage, program quality and equity. The desired outcome of a CMAM program is often to reduce GAM rate and this can be assessed through the cost of percentage reduction in SAM and MAM prevalence. The impact of the number of lives saved can be assessed through the cost of child malnutrition deaths averted or Disability-adjusted life years (DALYs) averted. Both are examples of cost-effectiveness analysis. However, to do this the benefits must be monetized, and this might be difficult to do for in an emergency setting due to lack of data, unclear causality, lack of access, etc. Hence a qualitative assessment needs to be carried out to provide a picture of effectiveness of projects and response. Obtaining beneficiary feedback and opinions of the response is one crucial component of value for money (VfM) measurement of effectiveness. Misperceptions by the community may pose sizeable constraints to the effectiveness of a CMAM program through shyness of community members to use CMAM services, therefore community assessments and sensitization prior to program implementation are critical.

The timing of the response or relief is an important measure of cost-effectiveness. Nutrition programs can be delayed by bureaucratic procedures which delay the contracting and release of funding to the implementing partner. Consider MOU drafting or agreements early on during the project planning phase when working with government or local partners. One of the other things to consider is whether the arrival of any nutrition supplies is timely. Cost savings aspects in a CMAM program includes procuring supplies when the international market price is low, the use of health care workers and community volunteers for implementation of activities, conducting assessment using local expertise and staff and minimizing the cost of international consultancies, prepositioning of stock in case of a scale up following onset of emergency.

COVERAGE – In nutrition program treatment coverage is an important aspect of effectiveness. It is important to measure if all children with acute malnutrition are accessing the services available. The sphere standard of coverage for CMAM program 50% for rural areas; 70% for host communities and 90% in refugee or camp settings. This can be measured by conducting a coverage survey. Such a survey will enable you to understand the barriers to service uptake and appropriate action can then be taken to improve the treatment coverage. An example of poor coverage could be misperceptions by the community, and this may pose sizeable constraints to the effectiveness of a CMAM program; therefore, community assessments and sensitization prior to program implementation and during are critical to maximize program uptake. If full coverage survey cannot be conducted, the % of the expected total reach of the program is a good indication of effectiveness. Low reach (below the milestone) is an indication that the program is not reaching all the children with acute malnutrition needing the service.

QUALITY – One way to check the quality of the nutrition services delivered is to monitor their performance outcome indicators. These are then compared to the Sphere standards which are:

Recovery: >75%
Death: 3-10%
Defaulter: <15%

These performance indicators must be consistently collected across sites and the reporting rate must be satisfactory (above 80%) otherwise it will be difficult to assess the effectiveness of the program. You should indicate how capacity needs assessments have been undertaken for efficient use of resources and well-targeted training; capacity development activities should be followed up to improve the quality of service and skills. You can mention how you have identified the need for the training delivered and the subsequent follow-up to ensure the skills have been transferred. Wider benefits of the program need to be described, for example how the treatment of malnutrition has long-term benefits for not only the household but also the community and society.

Good beneficiary feedback is also a sign of an effective response; nutrition program with strong community participation in its design, implementation and evaluation phase is more likely to be effective. You should therefore report on the beneficiary feedback and how the program capitalized on community participation.

IMPACT – To show the impact of a nutrition program or intervention in an area you must be able to demonstrate any positive changes in nutritional status, growth, or diet and this must be compared to some baseline line study. For example, the nutritional status of children before the program started versus nutritional status post intervention is one way to demonstrate this however you must consider seasonality into your assessment (both assessments must be done within the same seasons). Any changes during the project period are worth noting such as any improvement in dietary diversity, exclusive breastfeeding rate, improved uptake of health services and attendance because of nutrition education and counselling. Where possible, it is highly recommended that program anthropometric data collected on the beneficiaries are complemented by cross-sectional surveys in the target population (including participants and non-participants) and in similar populations not participating in the program (control or comparison group). Ideally, cross-sectional surveys such as a SMART and IYCF KAP, including at least one during the time before the initiation of the program (baseline), should be carried out on a regular basis. Because the logistic problems involved in measuring the same individual through time are not practical, therefore the longitudinal approach is not recommended.