EAC M&E at the Organization Level

Country and Partner Selection

EAC operates a robust M&E system at the organization level. When the programme first began in 2012, a country selection process was undertaken. Because the goal of reaching 10 million OOSC in a few years was so ambitious, it was determined that first priority countries would have at least 500,000 OOSC according to current UNESCO statistics. This cut line indicated 35 countries meeting this criterion. The next step was to determine where the OOSC were located in each country, and which organizations were already actively engaged. The Education Policy and Data Center (EPDC) in Washington, DC was contracted to provide datasets for each of the identified countries, illustrating UNESCO and Household Survey Data showing numbers of OOSC in each country and geographic location on individual country maps. EPDC also created a matrix for each country showing current active projects in education and a synopsis of their work. Following this initial country overview, EAC compiled extensive individual country opportunity assessment documents including research on the political and economic context, the education  system, the barriers children faced in accessing education, the national education strategy, and potential partner contacts. (The EPDC database and individual country opportunity assessments are available on the EAC shared drive.) The next step was to determine whether any current or new organizations in the country might be interested in focusing on, or expanding to include OOSC, as a partner with EAC. This involved Exploratory Visits to selected countries to meet with potential partners, senior government officials, and community stakeholders. Organizations that were aligned with EAC goals were invited to submit EOIs and the review process was undertaken.

Programme Data

EAC maintains a robust database of information on progress toward EAC goals to reach OOSC, as well as tracking progress of individual projects. Once projects are operational for two years, they begin to report retention of students in their programs and completion rates over time. The overall retention and completion rates are compiled by M&E staff and reported to management. Database design is continually refined to reflect current data needs as the programme matures.


EAC due diligence processes result in high selectivity of partners

 

Monitoring Tools

The regular six-monthly M&E, technical and financial reports serve as monitoring tools, giving a status report of progress towards targets, challenges encountered, and mitigation efforts to address them. Semi-annual reporting templates and guidelines are included in Annex D. Post-award reviews of projects serve as an additional check on project status and compliance and are selectively conducted by EAA’s internal audit personnel and contracted with PwC.

At the organizational level, additional checks on project progress are carried out through Site Monitoring Visits (SMV), prompted by an interest in observing the functioning of an innovative model, or providing assistance in resolving issues or challenges encountered by the project. Visits to project offices, field sites, and interviews with stakeholders are typically part of the SMV. Reports from the SMV team are compiled and submitted to the Executive Director for approval and stored with the M&E Department. A detailed Site Monitoring Guide providing suggested documents to review, questions to ask, and interviews to conduct for each project component and stakeholder is included in Annex E.

Project and stakeholder consultations through telecommunication or in person are another monitoring tool and problem-solving mechanism to help ensure projects stay on track. The M&E team is heavily involved in data reconciliation and cleaning, as well as proper data recording, with every report submission. The technical team also engages personally with each project management team to ensure data and activities reported are consistent with submissions to the M&E database. Further, the Technical Department consults regularly with project managers to ensure compliance with EAC procedures, accuracy in reporting, and resolving issues. Likewise the M&E and Operations departments interact with their counterparts in the projects. Members of the three departments engage in regular internal discussions regarding specific project status to ensure every department is aware of all issues that are pending resolution. This personalized approach to project quality assurance helps to prevent greater challenges and issues later, and also maintains good working relationships with partners.

OOSC data verification site visits are conducted on a sample of projects. The purpose is to ensure project integrity of procedures for collecting, handling, recording, and transmitting data on OOSC. On occasion, a more comprehensive Data Quality Assurance (DQA) site visit is conducted with more extensive review of data collection and transmittal procedures to verify the authenticity of the data and accuracy in reporting. DQAs may be contracted with international education M&E specialists in this field.

Evaluation Tools

As part of impact assessment, EAC engages external consultants to conduct case studies to delve more deeply into the factors and conditions that precipitated project achievements. TORs for case study investigations are issued and consultants contracted over several weeks, with a combination of document review and site visits to examine the factors contributing to the project outcomes. Case studies are recommended by M&E, technical and operations teams based on their knowledge of projects and their potential as innovative methodologies.

Formative Performance Monitoring is a structured internal process jointly conducted with M&E, technical, operations, and engagement teams as a check on consistency in application of procedures, standards, and messaging. Teams meet together to discuss assessments of individual projects, responses to partners, EAC internal mechanisms, coordination with EAA processes, archiving of data and information, and communication materials.

External project evaluations are occasionally contracted with outside consultants when recommended by EAC programme teams and management. These external evaluations are conducted when project evaluations indicate a need for additional information. Often these evaluations are critical in determining whether a project continues, is considered for a second phase, or could to be highlighted as a model of success.

External Evaluation of the EAC programme is scheduled after several years of operation, as a check on programme accomplishments and KPIs, validation of data reported, staff capacity, and compliance with EAA and EAC procedures.


Data Dissemination

Sharing data and information on the outcomes of the overall EAC programme are compiled from the M&E database against KPIs and reported regularly to EAA management. Additional outreach includes website postings, programme brochures, presentations at conferences, and high level briefings for upper management, all based on M&E data archived on the database. A sample database dashboard illustrates the kinds of information generated.


Research

Research is a key component of the EAC M&E function. One of EAC’s objectives is to influence the global agenda concerning OOSC, as well as to be a thought leader in international development education. Occasional research investigations, including literature review, global trend analysis, assessment of historical practices, and country specific or regional education indices, all contribute to the global knowledge base on OOSC. A Perspective on Practice and Policy series, a Technical report series, and Research Publications are produced, published, distributed and posted on EAC’s website.

Audit

EAC undergoes regular audits of the EAC programme, individual projects, and project achievements. After two years of EAC operations, an external audit was conducted by Her Highness’ Office (HHO). Now, EAA has an internal staff officer who regularly conducts programme and project audits at his discretion. In addition, on occasion, PwC is contracted to conduct a project audit. Additionally, EAC has a clause in the grant agreements that project audits conducted within the project budget can be requested by EAC for review. EAC staff respond to internal and external audit findings with clarifications on how issues identified are addressed or what mitigations strategies are in place to resolve challenges faced. This sometimes involves working with partner projects to provide answers and sometimes prompts recommendations for project site visits. All audit findings are responded to until a point of mutually accepted conclusions.

Decision Authority

To ensure efficient and timely programme management, EAC assigns certain low-risk decisions to be negotiated and approved at the EAC Executive Director level. Decisions involving significant changes in grant agreements are made at the CEO and Executive Committee levels.   Distinction is made between decisions made prior to a fully executed grant agreement and after the agreement is signed and the project is underway. The following lists the ceilings recommended to be associated with each decision authority level, including the EAA Executive Committee, the EAA CEO, and the EAC Executive Director.

EAA Executive Committee Authority (significant changes to agreements)

Before an agreement is signed:

  • Funding approval for any project with an EAC commitment of more than 10m QR .
  • Any increases in the first year disbursements that are made between the time of Executive Committee approval and finalization of the agreement.

After an agreement is signed:

  • Any changes to the disbursement schedule.
  • Any decreases over 10% in OOSC life of project (LOP) target.
  • Any increases to EAC agreed budget ceiling.
  • Any decreases in co-funding that result in less than 40% of project cost.
  • Any increases over 10% in investment per child.
  • Any contractual amendments resulting from the above.

EAA CEO Authority (approval authority with actions reported to Executive Committee)

  • Funding approval for any project with an EAC commitment of less than 10m QR.
  • Signature authority on all EAC project agreements and contracts.
  • Signature authority on all EAC project amendments.
  • Any early project closure.
  • No-cost extension requests, for reasons such as:
    • Finalizing project activities not completed because of extenuating circumstances or unforeseen challenges;
    • Delays in receiving EAC funding;
    • Changes in allocation of EAC funds which improve project performance as recommended after technical team review and EAC Executive Director approval.

EAC Executive Director Authority (low-risk, minimal change parameters, technical discretion)

Before an agreement is signed:

  • Any change in LOP OOSC target by 10% or less.
  • Any decrease in the budget or investment per child.

After an agreement is signed:

  • Any decrease of less than 10% in the LOP OOSC target depending on the partner’s performance/context such as country issues (sudden war, force majeure).
  • Any decrease in retention of 20% or less of LOP OOSC target, depending on the partner’s performance/context such as country issues (sudden war, force majeure).
  • Any change in the investment per child by 10% or less.
  • Any budget realignment when shifting budget between EAC approved cost categories when such shift exceeds 10% of an annual line item:  e.g. salaries, equipment, other direct cost, within the total budget ceiling.
  • Any decrease in co-funding that maintains at least  40% of project cost.
  • Any payment recommendation that allows for the partner to retain 50% of any remaining balance on hand after allowing for commitments and the next six months of planned expenditure.
  • Any payment recommendation for a project with more than 20% underperformance towards cumulative annual enrolment targets with a remediation strategy approved by EAC.

The delineation of decision authority is standard procedure defined according to best practice and in alignment with the organizations’ values, integrity, and goals. Levels of authority are articulated to reduce confusion and unnecessary duplication of effort, and to advance the interest of negotiating timely decisions.