Impact measurement

Introduction

What is it and how does it align with traditional venture building?

Before scaling up, it’s advisable to conduct rigorous evaluation of impact (e.g. Wagestream) as it’s harder to pivot models after scaling. These evaluations are often done with expert partners (e.g. academia).

Following confirmation that the desired impact is being achieved, it’s important to remember that as a company scales, new user groups are being acquired which can alter impact dynamics (e.g. Airbnb’s users moving from people with under-utilised assets to professionals buying property for use on Airbnb). Having a few proxy indicators for impact can help to monitor this (see Wagestream’s six-monthly update on impact).

Detailed understanding of the customer and the extent to which you’re solving a problem is fundamental to any successful venture. Impact measurement goes one step further by asking what effect using a product has on their quality of life.

At this stage, businesses usually have a clear understanding of financial KPIs to track, and the same would be expected on their specific impact KPIs, e.g. does the average impact per unit or customer change with the cohorts?

How does it drive impact?

Rigorous impact measurement provides a wealth of insights which can be used to enhance impact by identifying what’s working, what’s not and why.

How does it drive commercial value?

Rigorous evaluation of impact can help to generate a more detailed understanding of the customer, improve the product, motivate key stakeholders and drive customer acquisition through more powerful messaging.

In practice

What does good look like?

In an ideal world, impact KPIs track performance across the entire theory of change (ToC): inputs, outputs, outcomes, impact and unintended consequences. So a good place to start is to define a ToC and brainstorm KPIs for each section.

Impact KPIs could be used to answer the following questions:

1. What impact (i.e. change in outcomes) does our product create?

2. For which users (and how does it vary by user)?

3. For how many users (note, this is not the same as total users)

4. How does our product create this change?

5. What unintended impacts has this product created?

In reality, there are many constraints on what can be measured (costs, time, capacity, motivation, technical expertise, etc.). These questions can narrow down where the focus should be:

1. What is most material to the impact thesis?

2. What has already been validated externally (by e.g. academia)?

3. What is easiest/most efficient to measure?

4. What is the most valuable to measure (e.g. because it's useful for driving sales)?

5. How does this change over time?

Some examples:

1. Wagestream's pilot evaluation in partnership with 60 decibels and supported by Big Society Capital

2. Wagestream's six-monthly impact assessment

3. Second Nature's academic research and success stories

4. Togetherall's data on effectiveness (scroll to 'What our members say') and case studies