The SAT of ProFuturo: Transforming Digital Education Through Assessment

Continuous improvement is essential to ensure that socio-educational intervention programs are effective and relevant. In this regard, evaluative processes are crucial for measuring the impact, effectiveness, and efficiency of the implemented actions. However, these processes pose challenges within social organizations, related, among others, to factors such as information collection, the difficulty of finding suitable indicators, or adapting evaluative methodologies to various sociocultural contexts.

The ability to measure impact, adapt to diverse contexts, and evolve to address challenges has led ProFuturo to develop the SAT (School Assessment and Transformation Tool), a tool that has become the cornerstone of assessment and planning for this key organization in driving digital education in vulnerable environments.

In this article, we will explain what the tool is and how it has been carefully designed to help measure the impact of implemented programs, thus improving this organization’s ability to transform education and the opportunities of the girls and boys it works with.

Assessment and Evidence Always in Focus

ProFuturo is a very large and complex global program that addresses educational issues, particularly digital education, comprehensively and in very diverse contexts. In figures, we would be talking about more than 5,000 schools in 45 countries. Since its inception, the program has always sought to have tools to know the progress of the educational centers in which it operates, as well as the impact generated by the project, so the organization already had a significant culture of measurement and monitoring, which has materialized in the preparation of more than 20 studies and research that have allowed improving processes and increasingly focusing its interventions.

In 2020, when the organization already operated in more than 20 countries, the development of a global monitoring system began to collect all the information generated from its different platforms and digital sources, including an annual survey for teachers, students, school directors, and program coordinators. From there, ProFuturo developed a panel within its analytical system and began processes of analysis and quarterly monitoring reports with the participation of all actors involved in the program. The SAT was the logical evolution of this system. One step further. Its strengths: the vast amount of data that ProFuturo is capable of capturing and processing (250 million records daily) and a studied network of indicators that offer a much more comprehensive view of each school center and its status within the project.

Next, we will explain in more detail what the SAT consists of and how it works.

SAT: A Sophisticated System to Learn from Data

The SAT is much more than an assessment tool. It is an integral system meticulously designed to evaluate multiple dimensions of socio-educational intervention and analyze the level of appropriation that schools have of the program. It also provides information on the performance of the centers and their potential for continued growth.

Quantitative and Qualitative Information

The tool consists of 12 criteria, each addressing specific aspects of the program’s implementation in schools. These criteria are integrated by 32 key indicators, providing a detailed and nuanced panorama of the educational reality in each institution.

These criteria are:

1. Achievement of beneficiary goals.
2. Frequency of pedagogical use.
3. Motivation of participating teachers.
4. Commitment of the management team.
5. Care and maintenance of technological equipment.
6. Competent project leader teachers.
7. Integration of the program’s technology and innovation in educational practices.
8. Motivation of non-participating teachers.
9. Security of technological resources.
10. Availability of technological equipment.
11. Teacher training and advising.
12. Availability of operational connectivity.

The first seven indicators are diagnostic criteria, used to measure the performance of schools and their level of program appropriation. The last five define a work plan for each school.

The collection of information for the SAT is carried out in two ways: automatically and through a rubric completed by the team on the ground. The automatic collection is done through the indicators of the ProFuturo Monitoring System (SM), which processes data from its platforms and systems in real-time. These are the quantitative data that allow us to segment the schools. On the other hand, the rubric, completed by each of the coaches responsible for accompanying the school, provides a qualitative and contextualized perspective on the educational institution’s situation and gives us the keys to develop a transformation plan adapted to its specific circumstances.

Thus, the SAT works on two levels. On one hand, it provides a global snapshot that allows us to know the general situation of all the program’s schools, and on the other, it offers a very detailed level of detail, school by school. In the first application of the SAT in 2023, more than 4,600 centers were diagnosed, allowing not only segmentation according to the level of appropriation but also a deeper analysis based on specific criteria. This detailed information enables ProFuturo to focus on aspects that require global improvements, adapting its strategies and providing a more personalized approach to each school through the development of work plans for each educational center.

Based on the collected data, the system classifies schools into five levels of appropriation:

1. Initial level, where schools are mainly located that do not have teacher motivation or management commitment.
2. Basic level, which includes a wide variety of schools, essentially those that, although they may have teacher motivation and management commitment, do not yet have pedagogical use frequency of the Program.
3. Intermediate level, where schools meet beneficiary goals and have usage frequency. However, in some cases, work is needed on motivation, commitment, or equipment care.
4. Advanced level, includes schools that meet goals, have usage frequency, are motivated and committed, and take care of the equipment. However, they still lack a competent project leader teacher or have not integrated technology pedagogically.
5. Transformation level, where schools meet all diagnostic criteria and, therefore, stand out above the rest in program appropriation.

The Path to Transformation

As mentioned, this diagnosis leads to the creation of personalized prescriptive plans, designed based on the particular situation of each school, defining specific objectives to work on throughout the school year. The aim of these plans is to achieve the digital transformation of the centers according to the following components:

• Training proposals for teachers.
• Educational proposals for students.
• Support to educational centers for the program implementation.
• Development of responsible teachers or project leaders. Identification and preparation of those profiles within the school that will ensure the program’s sustainability.
• Virtual communities.
• Awareness of the school community. Incorporating and involving the educational community in the program and building a social fabric in schools and local communities.
• Platform management. For schools to be autonomous, they must assume certain platform management actions.
• Support. Provide attentive, reactive, and proactive support for incident resolution, monitoring, and continuous training.
• Equipment. Maintenance of the necessary equipment to implement the program.
• Monitoring and evaluation. Essential for achieving the planned results.

This methodology allows not only addressing the immediate needs of the schools but also contributing to a long-term positive impact on education quality.

Learnings and Future Plans

Now that the tool is designed and launched, we can talk about lessons learned and next steps. Regarding learnings, we can summarize them as follows:

1. Learning to learn from data to become an organization that constantly improves based on experience. Having a robust data collection infrastructure and reporting results is not enough. In this case, it has been necessary to design a series of mechanisms for all organization members to participate in reflective processes where quantitative results are analyzed to understand the causes, the context in which they occur, and their consequences.
2. Drawing conclusions for each management level and agile decision-making mechanisms. The reflection processes implied by the tool are carried out at many levels (schools, public and private partner organizations, ProFuturo management teams…). Drawing useful conclusions for each management level and acting in a concerted manner is a complex process that requires effective communication channels and clear, agile decision-making that must be constantly reviewed and improved.
3. Communicating effectively based on the role of each participant in the process. Each actor involved in the program (teachers, directors, educational authorities…) has a different interest concerning the results. For this reason, reports are generated according to the needs of each actor, taking into account the impact that such communication generates.
4. Offering each school what it needs by being creative and innovative. With almost 5,000 schools in 45 countries, this is one of the main challenges. Offering each center a personalized plan based on its needs, thus modifying the intervention carried out by the accompanying teams, from the frequency of their visits, the trainings they offer, introducing new elements or platforms, that is, being creative and innovative to overcome the challenges presented in each center.

As next steps, ProFuturo plans to increase the frequency of tool use from once to twice a year and to include new criteria as new data becomes available. Additionally, it aims to incorporate new profiles in the data collection process (e.g., principals and teachers, in addition to coaches), which would allow for triangulation and even more rigorous results.

Evaluation and assessment processes of social intervention projects are fundamental for measuring the impact and effectiveness of initiatives designed to address social problems and promote positive change in communities. The generation, systematization, and use of evidence are also crucial. In this sense, the SAT goes a step further, being a system that facilitates data-driven decision-making and allows working based on evidence in a world where, on some occasions, research, policy, and educational practice seem to be disconnected.