Student sparring partners help civil society groups evaluate their impact

In the coming decade, the city of Bremen, Germany, wants to create a car-free city centre, and Filip Wätjen is working to inform its 570,000 citizens about climate-friendly transport alternatives. He is part of Autofreier StadTraum Bremen (Auto-free City Bremen), a non-profit citizen group (NPO) that presents sustainable mobility concepts in spaces across the city every September during European Mobility Week, an initiative of the European Commission.

But Wätjen, who is in charge of evaluating the group’s impact, would like to know how effective they are in reaching citizens. In spring 2021, as part of the project course “Monitoring and Evaluation of Social Programmes”, a team of Hertie School students helped him do just that. They set up a system to evaluate and prioritise target groups and analyse the outcomes of the NPO’s efforts.

“The students have a certain know-how they are acquiring in the course, and we have a particular need,” said Wätjen. “We are a relatively small operation, and I wanted to know what I could achieve with my limited resources. It was a mutual learning experience.”

The project course introduced students to methods for evaluating social programmes – with a view to creating practical solutions for their partner organisations. It was taught by Ruth Ditlmann, Professor of Social Psychology, who joined the Hertie School faculty at the beginning of 2021. Ditlmann has worked for many years with civil society organisations on projects that build democratic values and encourage different groups to engage with each other, often through sport. In her research, she tries to understand the conditions that can help transfer such values to individuals and ultimately to create positive change in society.

Two of the three partner organisations came to Ditlmann via Phineo, a non-profit group that offers analysis and consulting for effective social engagement. Ditlmann had previously worked with the group, and Phineo also recently supported Autofreier StadTraum with a grant to help expand their activities as part of the Initiative Mobilitätskultur. Phineo suggested organisations from their networks that they thought would most benefit from participating in the course. Professor Ditlmann also helped the students find a third partner through her international contacts.

Hands-on learning in project design and working with partners

One of the goals was for students to carefully consider their partners’ needs, says Ditlmann. Students were encouraged to be ambitious in their evaluation designs, but also to tailor them to different requirements. “An important part of the task is finding the right balance between methodological rigor and a good fit for partners’ needs and resources,” says Ditlmann. “This is a practical challenge that students will face all the time in the future, and I want them to have an opportunity to engage with this and learn how to ponder difficult tradeoffs.”

Project courses are a regular feature at the Hertie School. They offer students a chance to gain hands-on experience solving real-world problems, conducting research and analysis while working with outside partners to put their in-class learnings to practical use.

“My goal was to create this kind of double value – for the NPOs and for the students,” says Ditlmann. “So the students could deliver something the organisation really needed, but which they may not have time or resources to do themselves.”

The student teams evaluated the projects of three civil society organisations between February and May 2021. Due to the COVID-19 pandemic, the work took place online. Ditlmann guided the students in their objectives, design, instruments, measurement and reporting.  She also arranged to give both the students and their project partners access to a digital tool developed by TolaData, a Berlin-based company that provides monitoring and evaluation software for non-profit projects. TolaData enables organisations to track indicators, monitor progress and report on their projects using a single platform. 

“Students built this infrastructure for each project partner,” Ditlmann explains. “They developed questionnaires and set up the system so responses could be fed into it.” The software allows users to collect, import, organise and analyse datasets from multiple sources and use configurable dashboards to visualize the information. TolaData was available to assist with their software throughout the course and share insights into how practitioners use it. 

Designs to meet partners’ needs

Finn Krueger, a first-year Master of Public Policy student was on the team working with Willkommenslotsen, or “Welcome Guides”, at the IHK zu Kiel (Chamber of Industry and Commerce in Kiel). Willkommenslotsen, a Germany-wide programme funded by the Federal Ministry for Economic Affairs and Energy (BMWi), actively approaches businesses to raise awareness about how refugees can enrich their organisations. They help fill job and apprenticeship vacancies and provide general legal and practical information. To make it easier for the IHK zu Kiel in this task, amid rising numbers of refugees and a relatively small staff, the students set up a system to help them understand how effective their work is, says Krueger. 

“We basically set up a platform to monitor the entire process,” he says. “We had evaluating aspects, such as the number of calls made to companies after meeting with a job candidate, preparing the candidate for a job interview, arranging interviews or jobs.”  The students conducted interviews with companies using qualitative methods and set up key performance indicators such as how many calls were made to schedule a single interview, to give insights into, for example, whether the IHK zu Kiel was connecting with the right companies or whether their persuasive skills were effective. 

They also developed a feedback mechanism ­– a quantitative questionnaire the staff could distribute at the end of events or after interviewing companies, to find out what was important for them, why they want to hire a refugee – or not. “Based on this information, the IHK zu Kiel could decide to call different companies or find matches more easily,” says Krueger. “Anna Schetle, who was our partner there, found it very useful because the monitoring system allowed her to meet requirements set by her funders – like a certain number of job placements.” 

Lea Endres and Kara Ketchum, both first-year Master of Public Policy students, worked with Peace Players, an international civil society organisation that uses basketball programmes to bridge the divide between different groups of people in conflict areas. Endres and Ketchum worked with the group’s Middle East operations, which engages over 500 Palestinian and Israeli youth each year in basketball training, conflict resolution education and leadership development. 

They discovered that a large, international organisation required a somewhat different approach. “Obviously, we thought, what can we contribute to this amazing organisation that has already a lot of experience with evaluation? And we felt a like we might not have the level of expertise that they needed and also maybe not the perfect background,” says Endres. “We tried to evaluate what we could add. What could we revise in the strategy? So we started at a very basic level looking at how they collect data and where the data was stored.”

“Professor Ditlmann was very supportive,” says Ketchum. “In the beginning, when we were kind of playing around with ideas and tweaking them, she helped us to really drill deeper into what we wanted to do and made it clear that it's okay to say no, which was really helpful because of course we wanted to please the partner organisation and meet all their needs.” But to serve the partner as best possible it is also important to be realistic about what could be accomplished given available resources and time, she noted.

A mutual learning journey

Ditlmann also invited several guest speakers from social projects during the semester: Ute Volz from Eleven, a non-profit institution that funds impactful organisations with mentoring programs and similar support for children and youth, and Ulrike Garanin from JOBLINGE, a collaboration of the private, public, and volunteer sectors helping prepare disadvantaged young people for the job market. Volz and Garanin spoke to students about the state of impact assessment in their organisations and in German civil society.

At the end of the course, the student groups and partners came together for presentations about their mutual learning.

“We saw how the students developed systems for the organisations to check their progress regularly, using instruments like questionnaires,” says Ditlmann. “The students acted as sparring partners that helped the organisations focus on their work and prioritise activities. With the new systems, they can do an internal learning check every few months.” Ditlmann hopes to teach the class again in 2022.  

Wätjen of Autofreier StadTraum Bremen said he would be eager to participate again, noting that the project generated some very useful outcomes. “We provided a reflection for each other,” said Wätjen. “It was an exciting journey.”