The Joy of Data (2016) - BBC - Documentary
It sometimes seems we're being deluged with data. Wave upon wave of news and messages. Submerged by step counts. Constantly bailing out to make room for more.
We buy it, surf it, occasionally drown in it and with modern technology quantify ourselves and everything else with it. Data is the new currency of our time. Data has become almost a magic word for...anything.
Crime and lunacy and literacy and religion and...drunkenness. You name it, somebody was gathering information about it. It offers the ability to be transformationally positive. It's, in one sense, just the reduction in uncertainty. So what exactly is data? How is it captured, stored, shared and made sense of? The engineers of the data age are people that most of us have never heard of, despite the fact that they brought about a technological and philosophical revolution, and created a digital world that the mind boggles to comprehend. This is the story of THE word of our times...
..how the constant flow of more and better data has transformed society... ..and is even changing our sense of ourselves. I can't believe this is my life now. So come on in, because the water's lovely. My name is Hannah Fry, I'm a mathematician, and I'd like to begin with a confession. I haven't always loved data.
The truth is mathematicians just don't really like data that much. And for most of my professional life I was quite happy sitting in a windowless room with my equations, describing the world around me. You can capture the arc of a perfect free kick or the beautiful aerodynamics of a race car. The mathematics of the real world is clean and ordered and elegant, everything that data absolutely isn't. There was one moment that helped to change my mind.
It was in 2011 when I came across a little game that a teenage Wikipedia user called Mark J had invented. Now, Mark noticed that if you hit the first link in the main text of any Wikipedia page and then do the same for the next page, a pattern emerges. So the page for data, for example, links from "set" to "maths" to "quantity" to "property" and then "philosophy", which after a few more links will loop back onto itself. Now, the page "egg" ends up in the same place, and even that famously philosophical boyband One Direction will take you all the way through to "philosophy", although you have to go through "science" to get there. The same goes for "fungi", or "hairspray", "marmalade", even "mice", "dust" and "socks". It was a very strange finding and it called for some statistics.
Another Wikipedia user, Il Mare, wrote a computer program to try and investigate this phenomenon. Now, he discovered, amazingly, that for almost 95% of Wikipedia pages, you will end up getting to "philosophy" eventually. Now, that's pretty cool, but how did it change my mind about data? Well, the pattern that Mark J discovered and the data that was captured and analysed, it revealed a hidden mathematical structure, because Wikipedia is just a network with loops and chains hidden all over the place and it's something that can be described beautifully using mathematics.
For me this was the perfect example of how there are two parallel universes. There's the tangible, noisy, messy one, the one that you can see and touch and experience. But there's also the mathematical one, where I think the key to our understanding lies.
And data is the bridge between those two universes. Our understanding of everything from cities to crime, global trade, migration and even disease... ..it's all underpinned by data. Take this for example. Rural Wiltshire and a dairy farm gathering data from its cows wearing pedometers. We can't be out here 24-7.
The pedometers help us to have our eyes and ears everywhere. It turns out when cows go into heat they move around a lot more than normal. Constant monitoring of their steps and some background mathematics reveal the prime time for insemination. We'll be able to look at the data and within 24 hours there'll be a greater chance of getting her in calf. Data-driven farming is now big business, turning a centuries-old way of life into precision science.
Pretty much every industry you can think of now relies on data. We all agree that we are undergoing a major revolution in human history. The digital world replacing the analogue world. A world based on data, they are made of codes rather than a world made of biological or physical data, that is extraordinary. Why philosophy at this stage? Because when you face extraordinary challenges, the worst thing you can do is to get close to it. You need to take a long run-up.
The bigger the gap, the longer the run-up. And the run-up is called philosophy. In the spirit of taking a long run-up, we'll start with the word itself. "Data" is originally from the Latin "datum", meaning "that which is given". Data can be descriptions... ..counts, or measures... ..of anything... ..in any format. It's anything that when analysed becomes information, which in turn is the raw material for knowledge, the only true path to wisdom.
Look at the data on data. And before the scientific and industrial revolution, the word barely gets a look in, in English. But then it starts to appear in print as scientists and the state gather, observe and create more and more of it. This arrival of the age of data would change everything. Industrial Revolution Britain.
For Victorians, booming industry and the growth of major cities were changing both the landscape and daily life beyond recognition. Into this scene stepped an unlikely man of numbers, William Farr, one of the first people to manage data on an industrial scale. William Farr had a quite unusual upbringing, in that he was actually the son of a farm labourer but who had managed to get a medical education, which was really very unusual for someone of his class. Farr very quickly became absorbed in the study of statistics. He was particularly interested, as you might expect for somebody with medical training, in public health, life expectancy and about causes of death.
For anyone interested in statistics, there was only one place to be. Somerset House in London was home to the General Register Office, where, in 1839, Farr found his dream job. From up there in the north wing, William Farr, the apothecary, medical journalist and top statistician, would really rule the roost. Now, this place was almost like a factory. Here, they would collect, process and analyse vast amounts of data. So in would come the census returns, the records of every single birth, death and marriage in the country, and out would come the big picture, the usable information that could help inform policy and reform society.
I think it's sometimes difficult for us to remember just how little people knew in the early 19th century about the changes that Britain was going through. So when Farr did an analysis of population density and death rate, he was able to show that life expectancy in Liverpool was absolutely atrocious. It was far, far worse than the surrounding areas.
This came as a surprise to a lot of people who believed that Liverpool, a coastal town, was actually quite a salubrious place to live. At Somerset House, Farr spearheaded a revolution in the systematic collection of data to uncover the real picture of this changing society. Its scale and ambition was described in a newspaper at the time.
"In arched chambers of immense strength and extent "are, in many volumes, the "genuine certificates of upwards of 28 million persons "born into life, married or passed into the grave." Here, every person was recorded equally. A revolutionary idea.
"Here are to be found the records of nonentities, "side-by-side with those once learned in the law "or distinguished in "literature, art or science." But what really motivated William Farr was not just data collection, it was the possibility that data gathered could be analysed to help overcome society's greatest ill. Cholera was probably the most feared of all of the Victorian diseases. The terrifying thing was that you could wake up in the morning and feel absolutely fine, and then be dead by the evening. Between the 1830s and the 1860s, tens of thousands died in London alone.
The control of infectious diseases like cholera, which no-one fully understood, became the greatest public-health issue of the time. However great London might have looked back then, it would have smelled absolutely terrible. At that point, the Victorians didn't have really a great way of disposing of human waste, so it would have flowed down the gutters into open sewers and out into the Thames. Now, the city smelt so bad that it was pretty plausible that the foul air was responsible for carrying the disease. Farr collected a huge range of data during each cholera outbreak to try to identify what put people most at risk from the bad air.
He used income-tax data to try and measure the affluence of the different boroughs that were affected by cholera. He asked his friends at the Royal Observatory to provide data on the temperature and climatic conditions. But the one that he thought was most convincing was about the topography. It was about the elevation above the Thames. Using the data, Farr suggested a mathematical law of elevation.
Its equations described how cholera mortality falls the higher you live above the Thames. Now, he published his report in 1852, which the Lancet described as one of the most remarkable productions of type and pen in any age and country. The only problem was that Farr's work, although elegant and meticulous, was fundamentally flawed. Farr stuck to the prevailing theory that cholera was spread by air. Such is the power of the status quo.
But in 1866, 5,500 people died in just one square mile of London's East End, and that data made Farr change his mind. When Farr came to write his next report, the data told a different story which proved the turning point in combating the disease. The common factor among those who died was not elevation or air but sewage-contaminated drinking water. With this new report, Farr may seem to have contradicted much of his own work, but I think that this is the perfect example of what data can do. It provides that bridge essential to scientific discovery, from theory to proof, problem to solution. Good data, even in huge volumes, does not guarantee that you will arrive at the truth.
But, eventually, when the weight of the data tips the balance, even the strongest-held beliefs can be overcome. Of course, it was the weight of the data itself which, with the dawn of the 20th century, was becoming increasingly hard to manage. Data stored long form in things like census ledgers could take the best part of a decade to process, meaning the stats were often out of date.
When you're dealing with figures like these, it's one thing. But when you're counting the population like this it's quite a different matter. A deceptively simple solution got what's now called the information revolution under way, encoding data as holes punched in cards. These cards are passed over sorting machines, each of which handles 22,000 cards a minute.
By the 1950s, data processing and simple calculations were routinely mechanised, laying the groundwork for the next generation of data-processing machines. They would be put to pioneering work in a rather unlikely place. In a grand London dining hall, a group of men and women, many in their 80s and 90s, have gathered for a special work reunion. At its peak, their employer, J Lyons, purveyor of fine British tea and cakes, had hundreds of tea shops nationwide. There are hundreds of items of food.
All these in a varying quantity each day are delivered to a precise timetable to the tea shops. These people aren't former J Lyons bakers or tea-shop managers. They were hired for their mathematical skills. Lyons had a huge amount of data which has to be processed, often very low-value data. So, for example, the transaction from a tea shop would be a cup of tea. But each one had a voucher and had to be recorded, and had to go to the accounts for business reasons and for management reasons.
Every calculation you did, not only you had to do it twice, but you had to get it checked by someone else as well. The handling of these millions and millions of pieces of data, the storage of that data, are the key of the business problem. The Lyons team took the world by surprise when, in 1951, they unveiled the Lyons Electronic Office, or Leo for short. At this point, only a handful of computers existed, and they were used solely for scientific and military research, so a business computer was a radical reimagining of what this brand-new technology could be for.
Each manageress has a standing order depending on the day of the week. She speaks by telephone to head office, where her variations are taken quickly onto cards. What the girl hears, she punches.
The programme is fed first, laying down the sequence for the multiplicity of calculations Leo will perform. It was the first opportunity to process large volumes of clerical work, take all the hard work out of it, and put it on an automatic system. Before Leo, working out an employee's pay took an experienced clerk eight minutes, but with Leo that dropped to an astonishing one and a half seconds. It was all so exciting because we were breaking new ground the whole time. Absolutely everything which we did has never been done before.
By anybody anywhere. I don't think we realised the kind of transformation we were part of. The post-war years saw a boom in the application of this new computing technology. Leo ran on paper, tape and cards, but soon machines with magnetic tape and disks were developed, allowing for greater data storage and faster calculations.
As more businesses and institutions adopted these new machines, application of mathematics to a whole host of new, real-world challenges took off. And the word "data" went from relatively obscure to ubiquitous. "Data" has become almost a magic word for anything. The truth is that it is a kind of interface today between us and the rest of the world.
In fact, between us and ourselves, we understand our bodies in terms of data, we understand society in terms of data, we understand the physics of the universe in terms of data. The economy, social science, we play with data, so essentially it is what we interact with most regularly every day. Data underpins all human communication, regardless of the format. And it was the desire to communicate effectively and efficiently that led to one of the most important academic papers of the 20th century. A mathematical theory of communication has justifiably been called the Magna Carta for the information age. It was written by a very young and bright employee of Bell Laboratories, the American centre for telecoms research that was founded by one of the inventors of the telephone, Alexander Graham Bell.
Now, this paper was written by Claude Shannon in 1948 and it would effectively lay out the theoretical framework for the data revolution that was just beginning. Those that knew him described Shannon as a lifelong puzzle solver and inventor. To define the correct path it registers the information in its memory.
Later, I can put him down in any part of the maze that he has already explored and he will be able to go directly to the goal without making a single false turn. During World War II he worked on data-encryption systems, including one used by Churchill and Roosevelt. But at Bell Labs, Claude Shannon was trying to solve the very civilian problem of noisy telephone lines. # There's a call, there's a call # There's a call for you # There's a call on the phone for you. # In that analogue world of 20th-century phones, your speech was converted into an electrical signal using a handset like this and then transmitted down a series of wires.
The voice signals would travel along the wire, be detected by the receiver at the other end and then be converted back into sound waves to reach the ear of whoever had picked up. The problem was, the further the electrical signal travelled down the line, the weaker it would get. PHONE LINE CRACKLES Eventually you couldn't even hear the conversation for the amount of noise on the line. And you could boost the signal but it would mean boosting the noise, too.
Shannon's genius idea was just as simple as it was beautiful. The breakthrough was converting speech into an incredibly simple code. ON PHONE: Hello? First the audio wave is detected, then sampled. Each point is assigned a code of ones and zeros and the resulting long string of digits can then be sent down the wire with the zeros as brief low-voltage signals and ones as brief bursts of high voltage.
From this code, the original audio can be cleanly reconstructed and regenerated at the other end. ON PHONE: Hello? Shannon was the first person to publish the name for these ones and zeros, the smallest possible pieces of information, and they are called bits or binary digits, and the real power of the bit and the mathematics behind it applies way beyond telephones. They offered a new way for everything, including text and pictures, to be encoded as ones and zeros. The possibility to store and share data digitally in the form of bits was clearly going to transform the world. If anyone has to be identified as the genius who developed the foundational science of mathematics for our age, that is certainly Claude Shannon.
Now, one thing has to be clarified, the theory developed by Shannon is about data transmission and it has nothing to do with meaning, truth, relevance, importance of the data transmitted. So it doesn't matter whether the zero and one represent an answer to, "Heads or tails?", or to the question, "Will you marry me?", for a theory of information is data anyway and if it is a 50-50 chance that you will or will not marry me or that it is heads or tails, the amount of information, the Shannon information, communicated is the same. Shannon information is not information like you or I might think about it. Encoding any and every signal using just ones and zeros is a pretty remarkable breakthrough. However, Shannon also came up with a revolutionary bit of mathematics.
That equation there is the reason you can fit an entire HD movie on a flimsy bit of plastic or the reason why you can stream films online. I'll admit, it might not look too nice, but... don't get put off yet, because I'm going to explain how this equation works using Scrabble. Imagine that I created a new alphabet containing only the letter A. This bag would only have A tiles inside it and my chances of pulling out an A tile would be one.
You'd be completely certain of what was going to happen. Using Shannon's maths, the letter A contains zero bits of what's called Shannon information. Let's say then I got a little bit more creative, but not much, and had an alphabet with two letters, A and B, and equal numbers of both in this bag. Now my chances of pulling out an A are going to be a half and each letter contains one bit of Shannon information. Of course, when transmitting real messages, you'll use the full alphabet. But English, as with every other language, has some letters that are used more frequently than others.
If you take a quite common letter like H, which appear about 5.9% of the time, this will have a Shannon information of 4.1 bits. And incidentally, a Scrabble score of four. Of course, there are some much more exotic and rare letters, like Z, for instance, which appears about 0.07% of the time. That gives it 10.5 bits and Scrabble score of ten.
Bits measure our uncertainty. If you're guessing a three-letter word and you know this letter is Z, it gives you a lot of information about what the word could be. But if you know it's H, because it is a more common letter with less information, you're more uncertain about the answer.
Now if you wrap up all that uncertainty together, you end up with this, the Shannon entropy. It's the sum of the probability of each symbol turning up times the number of bits in each symbol. And this very clever bit of insight and mathematics means that the code for any message can be quantified. Not every letter, or any other signal for that matter, needs to be encoded equally. The digital code behind a movie like this one of my dog, Molly, for example, can usually be compressed by up to 50% without losing any information. But there's a limit.
Compressing more might make it easier to share or download, but the quality can never be the same as the original. DOG BARKS You can't really overstate the impact that Shannon's work has had, because without it we wouldn't have JPEGs or Zip files or HD movies or digital communications. But it doesn't just stop there, because while the mathematics of information theory doesn't tell you anything about the meaning of data, it does begin to open up a possibility of how we can understand ourselves and our society, because pretty much anything and everything can be measured and encoded as data. We say that signals flow through human society, that people use signals to get things done, that our social life is, in many ways, the sending back and forth of signals. So what is a signal? It's, in one sense, just the reduction in uncertainty. What it means to receive a signal is to be less uncertain than you were before and so, another way to think of measuring or quantifying signal is in that change in uncertainty.
Using Shannon's mathematics to quantify signals is common in the world of complexity science. It's rather less familiar to historians. I love maths, I love its precision, I love its beauty. I absolutely love its certainty, and that, Simon can bring that mathematical worldview, that mathematical certainty to what I work with.
The reason behind this remarkable marriage between history and science is the analysis of the largest single body of digital text ever collated about ordinary people. It's the Proceedings of London's Old Bailey, the central criminal court of England and Wales, which hosted close to 200,000 trials between 1674 and 1913. There are 127 million words of everyday speech in the mouths of orphans and women and servants and ne'er-do-wells, of criminals, certainly, but also people from every rank and station in society. And that made them unique.
What's exciting about the Old Bailey and the size of the dataset, the length and magnitude of it, is that not only can we detect a signal, but we are able to look at that signal's emergence over time. Shannon's mathematics can be used to capture the amount of information in every single word, and like the alphabet, the less you expect a word, the more bits of information it carries. Imagine that you walk into a courtroom at the time and you hear a single word, the question we ask is how much information does that word carry about the nature of the crime being tried? You hear the word "the". It's common across all trials and so gives you no bits of information. Most words you hear are poor signals of what's going on. But then you hear "purse".
It conveys real information. Then comes "coin", "grab" and "struck". The more rare a word, the more bits of information it carries, the stronger the signal becomes. One of the clearest signals that we see in the Old Bailey, one of the clearest processes that comes out, is something that is known as the civilising process.
It's an increasing sensitivity to, and attention to, the distinction between violent and nonviolent crime. If, for example, somebody hit you and stole your handkerchief, in the 18th-century context, in 1780, you would concentrate on the handkerchief. More worried about a few pence worth of dirty linen than the fact that somebody just broke your nose or cracked a rib. The fact that 100 years later, by 1880, every concern, every focus, both in terms of the words used in court, but also in terms of what people were brought to court for, focus on that broken nose and that cracked rib, speaks to a fundamental change in how we think about the world and how we think about how social relations work.
Look at the strongest word signals for violent crime across the period. In the 18th century, the age of highwaymen, words relating to property theft dominate. But by the 20th century, it's physical violence itself and the impact on the victim that carry the most weight.
That notion that one can trace change over time by looking at language and how it's used, who deploys it in what context, that I think gives this kind of work its real power. There are billions of words, there's all of Google Books, there's every printed newspaper, there is every speech made in Parliament, every sermon given at most churches. All of it is suddenly data and capable of being analysed. The rapid development of computers in the mid 20th century transformed our ability to encode, store and analyse data. It took a little longer for us to work out how to share it. This place is home to one of the most important UK scientific institutions, although it's one you've probably never heard of before.
But since the 1900s, this place has advanced all areas of physics, radio communications, engineering, materials science, aeronautics, even ship design. NPL, the National Physical Laboratory, in south-west London is where the first atomic clock was built and where radar and the Automatic Computer Engine, or Ace, were invented. The Ace computer was the brainchild of Alan Turing, who came to work here right after the Second World War. Now, Turing's contributions to the story of data are undoubtedly vast, but more important for our story is another person who worked here with Turing, someone who arguably is even less well known than this place, Donald Davies.
Davies worked on secret British nuclear weapons research during the war... ..later joining Turing at NPL, climbing the ranks to be put in charge of computing in 1966. As well as the new digital computers, Davies had a lifelong fascination with telephones and communication. His mother had worked in the Post Office telephone exchange, so even when he was a kid, he had a real understanding of how these phone calls were routed and rerouted through this growing network, and that was the perfect training for what was to follow.
What was Donald Davies like, then? He was a super boss because he was very approachable. Everybody realised he'd got huge intellect but not difficult with it. Very nice guy.
Davies' innovation was to develop, with his team, a way of sharing data between computers, a prototype network. Donald had spotted that there was a need to connect computers together and to connect people to computers, not by punch cards or paper tape or on a motorcycle, but over the wires, where you can move files or programs, or run a program remotely on another computer, and the telephone network is not really suited for that. In the pre-digital era, sending an encoded file along a telephone line meant that the line was engaged for as long as the transmission took. So the opportunity here was because we owned the site, 78 acres with some 50 buildings, we could build a network. Davies' team sidestepped the telephone problem by laying high-bandwidth data cables before instituting a new way of moving data around the network.
The technique he came up with was packet switching, the idea being that you take whatever it is you're going to send, you chop it up into uniform pieces, like having a standard envelope, and you put the pieces into the envelope and you post them off and they go separately through the network and get reassembled at the far end. To demonstrate this idea, Roger and I are convening NPL's first-ever packet-switching data-dash... ..which is a bit more complicated than your average sports-day event. The course is a data network. There are two computers, represented here as the start and finish signs.
Those computers are connected by a series of network cables and nodes. In our case, cables are lines of cones and the connecting nodes are Hula Hoops. Having built it, all we need now are some willing volunteers. And here they are. NPL's very own apprentices.
So welcome to our packet-switching sports day. We've got two teams, red and blue. 'Both teams are pretending to be data 'and they're going to have to race.' You're going to start over there where it says "start", kind of obvious, and you're trying to get through to the end as quickly as you possibly can. You can't just go anywhere, you have to go through these hoops to get to the finish line, these little nodes in our network. You're only allowed to travel along the lines of the cones, but only if there's nobody else along that line.
All clear? OK, there is one catch. All of you who are in the red team, we are going to tie your feet together. So you've got to travel round our network as one big chunk of data. Those of you who are in blue, you are allowed to travel on your own, so it's slightly easier. 'The objective is for both teams to deposit their beanbags 'in the goal in the right order, one to five.'
EXCITED CHATTER Get in the hoop! Get in the hoop! Bring out your competitive spirit here. We've got packets versus big chunks of data. I'm going to time you. Everyone ready? OK, over to you, Roger. TOOT! Remember, you can't go down the route until it's clear.
'The red and blue teams are exactly the same size, 'let's say five megabytes each. 'But their progress through the network is clearly very different.' THEY LAUGH OK, blues, you took 13 seconds, pretty impressive. Reds, 20 seconds.
That's a victory for the packet switchers. Well done, you guys! Well done, you guys. The impact that packet switching has had on the world, I mean, it sort of came from here and then spread out elsewhere. It did indeed, we gave the world packet switching, and the world, of course, being America, they took it on and ran with it. This little race, Donald Davies' packet switching, was adopted by the people that would go on to build the internet, and today, the whole thing still runs on this idea. Let's say I want to e-mail you a picture of Molly.
First, it will be broken up into over 1,000 data packets. Each one is stamped with the address of where it's from and where it's going to, which routers check to keep the packets moving. Regardless of the order they arrive, the image is reassembled, and there she is. This is quite a cool thing, right, that you've got one of the original creators of packet switching right here and you can ask him... Every time you're like... Well, do anything, really. "Why is my internet running so slowly?" THEY LAUGH Don't ask me! We've come a very long way in just a few decades.
Around 3.4 billion people now have access to the internet at home and there are around four times the number of phones and other data-sharing devices online, the so-called Internet of Things. Just by being alive in the 21st century with our phones, our tablets, our smart devices, all of us are familiar with data. Really embrace your inner nerd here, because every time you wander around looking at your screen, you are gobbling up and churning out absolutely tons of the stuff. Our relationship with data has really changed - it's no longer just for specialists, it's for everyone.
There's one city in the UK that's putting the sharing and real-time analysis of data at the heart of everything it does - Bristol. Using digital technology, we take the city's pulse. This data is the route to an open, smart, liveable city, a city where optical, wireless and mesh networks combine to create an open, urban canopy of connectivity. Taking the pulse of the city under a canopy of connectivity might sound a bit sci-fi, or like something from a broadband advert.
But if you just hold on to your cynicism for a second, because Bristol are trying to build a new type of data-sharing network for its citizens. There's a city-centre area which now has next-generation or maybe the generation after next of superfast broadband and then that's coupled to a Wi-Fi network, as well. The question is, what can you do with it? We would have a wide area network of very simple Internet of Things sensing devices that just monitor a simple signal like air quality or traffic queued in a traffic jam. Once you've got all this network infrastructure, you can get an awful lot, a really huge amount of data arriving to you in real time.
What's happening here is a city-scale experiment to try and develop and test what's going to be called the programmable city of the future. It relies on Bristol's futuristic network, vast amounts of data from as many sensors as possible and a computer system that can simulate and effectively reprogram the city. The computer system can intervene. It could reroute traffic and we can actually radio out to individuals, so maybe they get a message on their smartphone or perhaps a wrist-mounted device, saying, "If you have asthma, perhaps you should get indoors."
Once you create that capacity for anything and everything in the city to be connected together, you can really start to re-imagine how a city might operate. We are starting to experiment with driverless cars and, in order for driverless cars to work, they have to be able to communicate with the city infrastructure. So, your car needs to speak to the traffic lights, the traffic lights need to speak to the car, the cars to speak to each other. All of that requires a completely different set of infrastructure.
Of course, as the amount of data a city can share grows, the computing power needed to do something useful with it must grow, too. And for that, we have the cloud. For example, imagine trying to analyse all of Bristol's traffic data, weather and pollution data on your home computer. It could take a year.
Well, you could reduce that to a day by getting 364 more computers, but that's expensive. A cheaper option is sharing the analysis with other computers over the internet, which Google worked out first, but they published the basics and now free software exists to help anyone do the same. Big online companies rent their spare computers for a few pence an hour. So, now anyone like me or you can do big data analytics quickly for a few quid. Such computing power is something we could never have dreamt of just a few years ago, but it will only fulfil its potential if we can share our own data in a safe and transparent way.
If Bristol Council wanted to know where your car was at all times but could use that information to sort of minimise traffic jams, how would you feel about something like that? Er, I'm not sure if I'd particularly like it. I think it is up to me where I leave my car. I understand the idea of justifying it with all these great other ideas, but I still probably wouldn't like it very much. If they are using it for a better purpose, then yeah, but one should know how they are using it and why they'll be using it, for what purpose.
I'd like to imagine a world in which all the data that was retained was used for the greater good of mankind, but I can't imagine a circumstance like that in the world that we have today. We live in a modern society, where if you don't let your data out there, not in the public domain, but in a secure business domain, then you can't take part in society, really. Unsurprisingly, people are pretty wary about what happens to their data. We need to be careful that civil liberties are not eroded, because otherwise the technology is likely to be rejected. I think it's an area where us as a society have yet to sort of fully understand what the correct way forward is and therefore it is very much a discussion.
It's not a lecture, it's not a code, it's one where we are co-producing and co-forming these sorts of rules with people in the city, in order to sort of help us work out what the right and wrong things to do are. It will be intriguing to watch Bristol grapple with the technological and ethical challenges of being our first data-centric city. In all these contexts, Internet of Things... ..new forms of health care, smart cities, what we're seeing is an increase in transparency. You can see through the body, you can see through the house, you can see through the city and the square, you can see through society.
Now, transparency may be good. It's something that we may need to handle carefully in order to extract the value from those data to improve your lifestyle, your social interactions, the way in which your city works and so on. But it also needs to be carefully handled, because it's touching the ultimate nerve of what it means to be human. So how much data should you give away? Traffic management is one thing but when it comes to health care, the stakes, the risks and benefits are even higher. And in Bristol, with a project called Sphere, they're pushing the boundaries here, too.
The population is getting older, and an ageing population needs more intense health care, but it's very difficult to pay for that health care in institutions, paying for nurses and doctors. So, the key insight of the Sphere team was that it's now possible to arrange, in a house, lots of small devices where each device is monitoring a simple set of signals about what's going on in that house. There might be monitors for your heart rate or your temperature, but there might also be monitors that notice, as you're going up and down stairs, whether you're limping or not. They've invited me to go and spend a night in this very experimental house, but unfortunately, I'm not allowed to tell you where it is. The project is a live-in experiment and will soon roll out to 100 homes across Bristol.
It's a gigantic data challenge, overseen by Professor Ian Craddock. So, that's one up there, then? Yes, that's one of the video sensors and we have more sensors in the kitchen. We have another video camera in the hall and some environmental sensors, and a few more in here. The house can generate 3-D video, body position, location and movement data from a special wearable.
How much data are you collecting, then? So, when we scale from this house to 100 houses in Bristol, in total we'll be storing over two petabytes of data for the project. Lord. So, on my computer at home, I don't even have a terabyte hard drive and you're talking about 20,000 of those. Yes. I mean, you know, the interaction of people with their environment and with each other is a very complicated and very variable thing and that's why it is a very challenging area, especially for data analysts, machine learners, to make sense of this big mass of data. I'm happy to find out that the research doesn't call for cameras in the bedroom or bathroom, but I do have to be left entirely on my own for the night.
The very first thing I'm going to do is pour myself a nice bloody big glass of wine. There we go. So, that nice glass of wine that I'm enjoying isn't completely guilt-free, because I've got to admit to it to the University of Bristol. I have to keep a log of everything I do, so that the data from my stay can be labelled with what I actually got up to. In this way, I'll be helping the process of machine learning, teaching the team's computers how to automatically monitor things like cooking, washing and sleeping, signals in the data of normal behaviour.
In the interests of science. 'I was also asked to do some things that are less expected.' Oh! I spilled my drink. 'The team need to learn to detect out-of-the-ordinary behaviour, too, 'if they want to, one day, spot specific signs of ill health.'
Right, I'm going to run this back to the kitchen now. It's a fairly strange experience. I think the temperature sensors, the humidity sensors, the motion sensors, even the wearable I don't have a problem with at all.
For some reason the body position is the one that's getting me. On the flipside, though, I would go absolutely crazy to have this data. This is the most wonderful... My goodness me. Everything you could learn about humans. It would be so brilliant. One thing I wanted to do was to do something completely crazy just to see if they can spot it in the data. Just to kind of test them. OK, ready? I can't believe this is my life now.
'Anyone can get the data from my stay online if they fancy trying to find 'my below-the-radar escape. 'The man in charge of machine learning, Professor Peter Flach, 'has the first look.' Between nine and ten, you were cooking.
Correct. Then you went into the lounge. You had your meal in the lounge. You know what? I ate on the sofa.
And you were watching crap television. I was watching crap television? I've been found out. We didn't switch the crap-television sensor on. That's not on here, but OK.
So, you were in the lounge sort of until 11:30. Correct. Then you went upstairs, there's a very clear signal here. And then, from then on, there isn't a lot of movement. I was in bed. So, I guess you were in bed.
Sleeping. Normal activities, like cooking or being in bed, are relatively straightforward to spot. But what about the weird stuff? This is yesterday, again. I can see it. I can see the moment. You can see the moment? I can see it, yeah. There's something happening here which is sort of rather quick.
You've been in the lounge for quite a while and then, suddenly, there's a brief move to the kitchen here and then very quick cleaning up in the lounge. I wasted good wine on this experiment. Good wine? Humans are extraordinarily good at spotting most patterns. For machines, the task is much more challenging, but, once they've learned what to look for, they can do it tirelessly. I suppose, in the long run, if you are going to scale this up to more houses, you can't have people sifting through these graphs trying to find...
I mean, you have to train computers to do them. You have to train computers to do them. One challenge that we are facing is that our models, our machine learning classifiers and models, need to be robust against changes in layout, changes in personal behaviour, changes in the number of people that are in a house.
And maybe we are wildly optimistic about what it can do, but we are in the process of trying to find out what it can do, at what cost, at what... ..invasion into privacy, and then we can have a discussion about whether, as a society, we want this or not. If this type of technology rolls out, machines will be modelling us in mathematical terms and intervening to help keep us healthy in real time - and that's completely new.
It's true that our fascination with machine, or artificial, intelligence is as old as computers themselves. Claude Shannon and Alan Turing both explored the possibilities of machines that could learn. But it's only today, with torrents of data and pattern-finding algorithms, that intelligent machines will realise their potential. You'll hear a lot of heady stuff about what's going to happen when we mix big data with artificial intelligence. A lot of people, understandably, are very anxious about it.
But, for me, despite how much the world has changed, the core challenge is the same as it always was. It doesn't matter if you are William Farr in Victorian London trying to understand cholera or in one of Bristol's wired-up houses, all you're trying to do is to understand patterns in the data using the language of mathematics. And machines can certainly help us to find those patterns, but it takes us to find the meaning in them.
We should be worried about what we're going to do with these smart technologies, not about the smart technologies in themselves. They are in our hands to shape our future. They will not shape our futures for us. In the blink of an eye, we have gone from a world where data, information and knowledge belonged only to the privileged few, to what we have now, where it doesn't matter if you're trying to work out where to go on holiday next or researching the best cancer treatments. Data has really empowered all of us. Now, of course, there are some concerns about big corporations hoovering up the data traces that we all leave behind in our everyday lives, but I, for one, am an optimist as well as a rationalist and I think that if we can marshal together the power of data, then the future lies in the hands of the many and not just the few.
And that, for me, is the real joy of data. MUSIC: Good Vibrations by The Beach Boys
2021-02-28 16:24
