Data literate public administrations across Europe are increasingly using big data as part of their operations, however its full potential remains untapped, exclusive Polivisu research shows
Big data has seen no shortage of attention in recent years. Advocates from all walks of life have touted its potential to improve virtually every aspect of our society, from health to security to transport. Besides individuals, companies and governments are often presented as key beneficiaries of the innovative solutions that big data makes possible. So far so good, but do these statements match the facts on the ground? Is big data still a buzzword that has yet to hit the mainstream?
Keen to explore the situation in more detail, PoliVisu set out on a fact-finding mission and conducted its own survey targeting mostly public sector organisations based in Europe. We found that in general big data is perceived positively, and the majority of respondents declared using it at the level of individual units or departments. However, the wider application across different areas of policy making appears to be limited. The lack of long-term big data strategy; established frameworks for quality, privacy and confidentiality; dedicated training programs; and organisation-wide awareness of how exactly big data can help in different areas are among the main impeding factors highlighted by the participants.
The survey was carried out between February and April 2018, yielding a total of 131 responses, 122 of which came from within the EU. The majority of respondents represented local governments (58%) and regional authorities (19%), who occupied various roles within an organisation: technical officer (24%), project manager (17%), director (15%), data officer (12%), to name just the most frequent ones.
Anyone with a basic interest in the subject will be familiar with the three Vs of big data (volume, velocity, variety). We started our questionnaire by asking what makes data big? To the traditional definition we added two more elements: variability and process. Volume was defined as extremely large amounts of data; velocity - an astonishing speed at which data is collected; variety - the different types of data that are available; variability – inherent inconsistencies, anomalies and outliers; process – the way data is captured, managed, analysed and shared. Our participants rated volume as being the most important aspect (79%), followed by variety (63%) and process (53%).
Next, the survey explored the extent of big data use and it is at this point that a very interesting pictured started to emerge. Although the majority of respondents (72%) said they or their colleagues use big data in their day-to-day job, only 21% agreed their organisation takes full advantage of the available opportunity. Many more disagreed (33%). According to some, the problem lies in the lack of appropriate skills needed to work with big data. For others, the problem has more to do with the recent nature of big data. “The area is relatively new and the necessary arrangements are just being implemented,” said one respondent from local government.
Indeed, the necessary arrangements, such as data strategy and the various frameworks for managing quality, privacy and confidentiality are in the early stages of development or don’t exist at all.
It is also telling that a vast majority of respondents (74%) said their organisations don’t provide any special training on big data. It seems that this is largely due to the prevailing preference to hire rather than train. “Usually we hire instead of providing training,” said a respondent from Tartu, Estonia. Similar preference was echoed by a respondent from Drenthe, Netherlands. “At the moment we tend to focus on hiring new staff. Any training that happens is on-the-job, whereby people learn by doing.” On-the-job training was also mentioned by a respondent from Salo, Finland. “People upskill themselves in their spare time at work. We provide no special training yet.”
As mentioned before, 72% respondents said they or their colleagues use big data as part of their job. “Non-users” thus represent just less than a third of the sample. In our survey we also investigated why these organisations don’t use big data. Interestingly, almost all (97%) of those who fall into this category said “small data” is sufficient for their needs. For many, the lack of clear purpose (74%) and skills (60%) is an important barrier, as is the cost which was raised by 40% of non-users. 31% said the needed big data simply does not exist. Interestingly, 57% respondents from this group said big data would improve the performance of their organisation.
So it is mainly for organisations that see value in big data but for various reasons haven’t started using it that PoliVisu prepared the following recommendations based on survey results, and in particular the results of interviews with cities that have made progress in that direction:
Conduct an audit to find out exactly what data your organisation has. Interviews with several cities showed that municipalities may have big data hidden on some server within a specific department, they may simply be unaware of this fact
Study the good practices of cities leading the way, identify priority use cases where big data can be applied, then develop pilots and demonstrators to make a business case for the opportunity and why it is worth pursuing
Once a big data path has been chosen it is advisable not to “walk it alone,” as there are probably potential partners on the “doorstep” willing to work together, who could provide the missing expertise and/or resources
Hire good specialists to work with data, statistics and analytics, or ensure there is an appropriate training program in place so that in-house staff always have the right skills to meet big data needs of the organisation
Secure senior leadership buy-in to promote big data culture across the whole organisation, not just an individual department, which can only create more silos
Rather than seeing it as a one-off exercise, develop a systematic approach to big data, one that spans several stages, including collection, analysis and application
Be open to experimentation as it is hard to know in advance what the outcome may be once you put big data to good use, for example by making it available as open data and sharing it with developers at a hackathon