360Giving & Open Contracting – What We Learned From Our DataDive

By Mor Rubinstein June 27, 2018

In April we ran a Data Dive with DataKind UK looking at funding to the criminal justice sector in the UK. This is our second blog about what we found and learned. You can read the first blog here.

We asked DataKind to run a Data Dive for us as we wanted to link 360Giving data to other datasets and see what we could find. With lots of contracts and grants data being shared by the Ministry of Justice (MoJ) we decided to look at spending on criminal justice activities and see if we could follow where the money goes.

Why contracting data?

We know from NCVO’s Almanac that government is the single biggest funder of voluntary sector activities in the UK. In FY 2015/16, it provided £15.3 billion to the third sector, £11 billion of which was contracts and fees. We also know that government has two kinds of funding:

Grants: The government’s definition of a grant is a sum of money awarded to an organisation in anticipation of it being applied for an agreed purpose. This purpose may be very specific, for example to fit a smoke alarm in an old person’s house; or less specific, for example to promote fire safety among older people. Check out more types of grants here.
Contracts: These are defined as an agreement between the government and a legal entity to deliver services. As with grants, they can range hugely in size and scope.

Since 2003, government has awarded fewer grants to charities but issued more contracts (see the Almanac data for more on this). Luckily for us, the UK has committed to publish all government grants and contracts data openly and MoJ started to do this in 2017. This means we were able to compare grants and contracts issued for criminal justice activities. The results were interesting!

Challenges of working with Open Contracting data

Challenge #1 – Getting the data

Since we already had data for MoJ grants, we needed to get the ministry’s contracts data. One of the easiest ways to access UK contracting data is via Contracts Finder, but there are limitations to what you can query on it. We needed to query all the data, and luckily for us the Open Contracting Partnership had already arranged for data from the Contracts Finder OCDS API to be cached in a Google BigQuery table, so we used that. But not all data analysts feel comfortable using Google cloud services, so we played with BigQuery and learned by doing.

Challenge #2 – JSON

The Open Contracting Data Standard (OCDS) is complex and includes many stages of the procurement pipeline from tender to implementation. It needs to represent one to many relationships, and for this is uses a JSON format, which fits for that purpose.

However, data analysts prefer to use tabular data, which shows one to one relationships and is easier to query. Querying from so many different tables can be tricky. You can’t just dive into the schema – you have to learn it, and that takes a while. Our suggested solution is to create a bank of SQL queries of the most common questions around contracting data. For example, awards/tenders per department or total amounts of spending. This will support more efficient data analysis.

Challenge #3 – There is no signposting for different types of contracts

The next challenge we faced was understanding which contracts and awards go to capital funding (buildings, constructions) compared to service delivery. Within service delivery, we looked at the type of activities that charities would provide, such as rehabilitation programmes.

As charities are not the main suppliers of government contracts, it would have been useful to know the type of organisations that applied for the contract, e.g. charity, company, CIC, etc. This could be worth adding to the schema, and if the information already exists then it would be helpful to signpost it better.

We found 358 contracts issued by MoJ between 2014-2017. After a manual cleaning and reading every contract description looking for social services contracts, we were left with 78 contracts. As expected, the majority of them were for construction or IT services.

Challenge #4 – When you don’t have identifiers…

We identified a key issue with the UK Open Contracting dataset. It only has organisation identifiers – in this case company numbers – for contracts issued from November 2017 onwards. As mentioned in previous posts, organisation identifiers are crucial for making sure we are looking at the right organisation. Names are not a good identifier since an organisation can have different names. To join the contracts dataset to the 360Giving dataset, we needed to use the same identifiers. After we cleaned the dataset of irrelevant contracts, we manually added company numbers using data from Open Corporates and Find That Charity.

What we found

One of the Data Dive participants was Rafael Garcia, who used to work with contracting data at Transparencia Mexico. He looked at the cleaned dataset and this is what he found:

48 organisations were awarded at least one MoJ contract in the last two years. Of these, 34% were charities (see graph A below) and were awarded 25% of the total contracts (see graph B). However, in terms of value, charities were awarded only 1% of the total amount awarded by MoJ for statutory services (see graph C)

Just over 90% of the total amount of contracts were awarded to Community Rehabilitation Companies (CRCs). For more on CRC’s, see this NAO report. There are 21 CRCs run by eight parent organisations.

The CRC Sodexo Justice Services was awarded both the highest number of contracts (10 contracts) and the highest sum amount (£1,172,972). See graphs D and E.

What did we learn from this Data Dive?

At our first attempt of looking at joining up charities and contracting data, we learned that it is challenging to use this data and manual cleaning is needed in order to make the data easier to work with.

While we discovered that the current contracting dataset shows that there are not a lot of contracts awarded to charities, we can now start to look for these contracts at the local level. That would be a challenge though, because as we mentioned above, the contracts type is not tagged. In addition, we need contracts for more local authorities.

We hope to work with contracts data in the future to see how it can help us understand financial flows. We hope that this first attempt and what we learned will support working with contracts and grants data in the future.