[Update: NB: this post only addresses one side of big data – the more commercial one – and doesn’t touch on the enormous wealth of other applications of huge data sets (environmental, medical, etc.). I’ll try to cover those in another post soon.]

There’s been an enormous amount of talk across the whole of the business world about Big Data. It’s one of the biggest themes out there at the moment, and yet it’s difficult to find anyone actioning it in a positive, sustainable way. This is leading us straight into a disaster, as the brilliant Jonathan MacDonald explains in his latest post. It needn’t be so. One of the cruelest ironies of the whole Big Data debacle so far is that the struggle for personal data has all but eclipsed the potential of all the other data that’s out there. Which is a shame, really.

Look at it this way: you know what you know, and I know what I know. You can tell me what you know and then I apply what I know, and this is how we arrive at a decision. Simple, right? This is the way humans work, at the most basic level. So what if you knew a lot more about my stuff and the same amount about your own? Or vice versa? Would that lead to better decisions? Maybe, maybe not. When we know each other better, we’re better able to persuade. But when we know ourselves, we make the best decisions.

So what if you could know a lot more about your stuff, and I could know a lot more about mine? Wouldn’t that be the ideal? Then we’d both make better decisions, and we’d both understand them better too.

The right application of big data and personal data could make this a reality. But at the rate we’re going, it may well not.

Data is, quite simply, stuff we could know. Businesses gather data about all kinds of things, but that doesn’t necessarily mean they know what the data contains. Take retailers. They have data on what’s purchased in their shops, at what time of day and in what quantity. They have security cameras that show the flow of foot traffic through the door (and past it) and through the merchandise. They have equivalent data for their digital presence. What are they doing with all this information? Sadly, most are using it to target marketing. But it could do so much more.

Data about what’s purchased at what time of day can help to optimise the supply chain – are the right items being stocked? Are there things that regularly sit on shelves until they need to be marked down? What are the patterns and how can they be improved?

Data about the flow of traffic into and past the door can be used to evaluate window displays and signage – are people interested in what they see? Is the right merchandise on show? Which are the items that have the most impact? Which are the ones that get people to turn around and go inside?

Data about the flow of traffic within the shop can be used to optimise merchandise displays. Many clothing retailers now rearrange their collections every few months, because someone told them that keeping people moving (even if it’s because they’re lost) takes them through more merchandise and increases the likelihood they’ll see something they want to buy. But is that really the best way to treat people? What if we looked at traffic data and tried to spot the patterns in how people shop? Maybe we could create paths through the shop that took people past things they might like, without being lost or annoyed.

These are just some obvious beginnings – things these businesses could know that they might not know today. None of them are to do with what they know about an individual customer, but all of them could help to improve the customer’s experience.

Speaking of customers, let’s focus there for a moment. The customer’s job is to know what they like. And we are perfectly capable of identifying the things we like (and don’t) when they’re put in front of us. It’s rather difficult to persuade a woman to like, for example, a pair of shoes she thinks are rather ugly. It’s even more difficult to persuade her she doesn’t like the ones she likes. True, getting to know her makes it possible to show her things she’s more likely to like – this is why local boutiques are still such a pleasure to visit, as I wrote in my piece for Fjord about The Shopping Revolution.

It is this idea of ‘the more you know, the better you suggest’ that has so many businesses salivating over personal data. As Jonathan quite rightly states, when it comes to that stuff, the most personal is the most valuable – the better you know someone, the more likely it is you can make them happy. But that’s the thing – you have to want to make them happy, not just make them spend more money. There is a human element to all this that’s being ignored, just as the potential power of the data is being ignored.

Having a lot of data about someone is not the same as knowing them, just as having a massive data set about traffic and purchases doesn’t mean you understand your shop. Getting to know someone is about more than facts. It’s about interpretation – the subtleties that come out through conversation. It’s about trust. It’s a two-way exchange.

Yes, spies have spent centuries perfecting the art of getting to know people clandestinely, but apart from Bond and Bourne, we don’t really like spies all that much. There’s a reason for that – we like to make our own decisions about who knows us, how and why.

Where things start to get really interesting again is in what would happen if we knew ourselves better – or if we could make better use of our own personal data. For example, I might be on the lookout for a pair of mid-heel boots in grey to fill a gap in my footwear wardrobe (yes, I went there. Clichés are clichés for a reason). I might let my favourite boutiques in Berlin know (giving them that bit of personal data), and get them to ring me if something interesting comes in. But what if I also could, when I head out for lunch in London and have a bit of time to browse, release that information, just for an hour or so, to see who’s got boots that might be a match*? That would be great – I’d find and buy them more quickly. And if I liked the shop, maybe I’d start a conversation with them, introduce myself, give them a bit of information about me. Maybe I wouldn’t. But if I did and then they misused it, I’d regret it and distrust them.

This is what we mean when we talk about having conversations. Is it magic? No. Is it labour intensive? Yes. Is it worth it? Absolutely. But this has always been the case. Big data hasn’t created this opportunity, or changed it in any substantive way. It has the potential to amplify it, certainly. It also has potential in a number of other ways. But people are people, and need to be treated as such. Anyone who tells you more data will make that any different is selling you something you don’t want.

*For more reading on this sort of thing, see Doc Searls‘ VRM project.