Moving the needle on social outcomes

It's often said that what gets measured gets changed. But how should we measure the social outcomes of an infrastructure project? How do we know how a project really impacts the environment, the local economy, and the need to cut carbon emissions?

On November 13, 2023, Mott MacDonald hosted a panel at the annual conference of the Canadian Council for Public-Private Partnerships (CCPPP). The title was "Moving the Needle: How Tangible Measurement Can Ensure Infrastructure Delivers on Government Policy."

Moderated by Mott MacDonald, the panel brought together David Ho of Infrastructure Ontario, Steven Nackan of Aecon Concessions, and Steven Robins of the Canada Infrastructure Bank to share their views on the topic.

Progress toward measuring outcomes

How are we doing when it comes to measuring outcomes?

According to Nackan, “I think we’re doing pretty well in the sense that the projects we’re delivering are having tremendous impact. But I don’t think we do particularly well in the measurement of it.”

“When we start too quickly to measure, we start to focus too quickly on very simplistic reporting, things that feel gratifying — how many trees have we saved, how many cars have we taken off the road — but not necessarily measuring what’s valuable. But if you added up all of the cars that we claim we would take off the road in all these different projects, there wouldn’t be any cars on the road anymore.”

“For us, we think about putting impact measurement at the core of our returns,” said Robins. “When we invest, we’re not investing solely for financial return. We’re looking to get a greenhouse gas reduction outcome, homes connected to broadband, new transit ridership, outcomes we feel we do need to measure to justify putting our capital at risk.”

Ho said, “There’s been for many years an inappropriately narrow focus and obsession on quantitative value for money. As an agency we’ve historically wanted to anchor ourselves on that analysis and what it would mean to procure and ultimately deliver a project on time and on budget.”

“We feel so much more connected now to what the upstream policy choices were for the project to exist. One of the things that’s remarkable is how at different points in time in the lifecycle of a project there are different reasons for a project to be pursued.”

Using the numbers

Ho emphasized the importance of quantification and defining KPIs in measuring social outcomes.

“In the healthcare sector we would love to be conducting more deliberate post-occupancy evaluations for hospitals being built. How have the infrastructure and technology choices manifested in things like shorter patient stay and better clinical staff retention? How would you convert those benefits into a model that could quantify for government at the time of business case that a particular scope is worth pursuing? Right now we have none of that.”

Nackan, however, cautioned against moving too quickly, noting that “contractualizing impact metrics too soon could lead to a risk of dispute. Especially if and when outcomes aren’t realized. We need to take our time with this and start slow to ensure the right metrics are developed. We need to incentivize the private sector as a first step before embedding terms into contracts.”

Robins agreed that flexibility is needed before construction begins, as it allows the private sector to innovate during the procurement phase to define the desired outcomes and how to measure them.

Telling the story

“Numbers can never capture the whole picture,” Robins pointed out. “We’ve gone through a journey from writing down some version of what we thought the benefits of the project were, to recognizing the gaps when you try to quantify everything.”

“For example, we have been working with the Indigenous communities we’re partnering with to understand how they tell the stories of the impact of those projects. In turn, how do we capture this impact in a way that’s respectable, captures that story, and then track it over time?”

What are the next steps?

“There is always going to be a risk with change,” said Ho. “We need to get better at identifying the risks of not changing our approach. This will help us adopt new approaches to impact measurement. There needs to be an ongoing effort to think about how to value things differently, beyond the financial value.”

“Anybody who is in this industry should be very proud to be involved in it, because it is delivering tremendous benefit,” said Nackan. “We’ve evolved from being very focused on technical issues, big cost issues, disputes, and day-to-day firefighting to thinking about the outcomes and the big impact that these projects are having.”

Indeed, infrastructure delivers significant benefits and delivers positive outcomes to society. As our industry continues to evolve, increased focus needs to be placed on how we measure these outcomes ­— social, economic, environmental, climate, and governance — as more complex projects are being delivered. We have not yet cracked the nut on outcomes measurement on infrastructure projects, but as the panelists highlighted, we are well underway.
__________________________

To learn more about our Advisory service offerings such as Business Casing, Structured Decision Making, and Alternative Procurement methodologies in North America, or if you are interested in joining our growing team, please contact our Advisory Practice leader Joshua Ogier at joshua.ogier@mottmac.com.