Ways to ‘see’ a system: tracking innovation trends

Find the right data

We analyse various large datasets to capture different trends in innovation in each of Nesta’s mission areas. To date we have investigated trends in low-carbon heating innovation and food technologies, and we’re now researching trends in technologies to support early child development.

The choice of datasets used across these projects is inspired by Everett Rogers’ framework on the diffusion of innovation. To capture research trends in the UK, we use the open-access Gateway to Research database. It contains information about projects supported by UK Research and Innovation, which is responsible for around half of total UK public spend on research and development.

We also examine global research trends by using OpenAlex, a large, open-source dataset of publication data, including research paper summaries and metadata such as the authors’ affiliations and number of citations.

Looking at patent data allows us to cover innovations that are closer to commercialisation. For this purpose we have explored platforms like PatSnap as well as open-source data from Google Patents database.

To track investments into businesses, we use the Crunchbase database, which contains information about the amount of money raised by companies across the world from investment deals.

We complement the insights about research and investment with ‘public discourse analysis’ to better understand how technologies are being talked about in the public arena.

We have used openly available records of parliamentary debates (Hansard) and the open access platform of The Guardian news website – and we will explore other sources for analysing media coverage in the future.

Use the data to identify innovations

One of the main tasks is to identify instances of technologies or innovations in the research, investment and public discourse datasets.

In our first iterations of this approach, we experimented with unsupervised machine learning methods, such as clustering and topic modelling, as well as simpler keyword searches.

Now, we are also using supervised machine learning for training classifiers (machine learning algorithms used to organise data) of different technologies as this gives us the greatest control to tailor the results specifically to our mission areas. Supervised learning, however, requires labelling a lot of data and so we are also trying out generative AI (large language models) for this purpose.

Characterise the trends

To interpret and communicate the trends around research, investment and public discourse, we have prototyped a data-driven typology that categorises these trends as dormant, emerging, hot or stabilising.

Our typology takes into account the magnitude (average level) and growth (eg, increase in the past five years) of time series related to a particular technology, such as the number of new research projects or the amount of investment across different years.

Growth is one of the main attributes when evaluating emerging technologies, whereas magnitude can help us see how established the innovations and technologies are and where they might be in their life cycle.

Strengthen the quantitative data with strategic foresight

Whilst data science can illuminate the patterns in a system, on its own it cannot answer the ‘why’ or the ‘so what’ behind the trends.

Once the trends are identified, we also draw on the wider toolbox of strategic foresight methods to explore how these trends might play out for society. This helps us to critically appraise the hype cycle. Simply because a technology is ‘hot’ does not mean that it will be impactful, or indeed that the net effect will be positive for society.

For example, when we explored innovation in food systems, we combined quantitative analysis with qualitative ‘sensemaking’. This included surveying a panel of investors, scientists, industry experts, and policymakers on the downstream social impact of the innovations surfaced through the analysis (for example, food reformulation), and unpacking the potential impact on obesity specifically.