Why The World Is Going Crazy Over Big Data

From the 1980s into the 1990s, a pair of data scientists, Geoff Hinton and Yann LeCun laid the foundations of artificial intelligence. They came up with the idea of a neural net, a kind of statistical tool that could be trained, rather than programmed. Because of its similarities to the human mind, they borrowed the term “neural” and used it as a metaphor.

The problem was that back in the 1980s, the idea wasn’t actually very useful. Neural nets require thousands, sometimes millions, of pieces of data to train. But because the internet was basically non-existent and sensors were expensive, there was hardly any data to actually feed these systems. Neural net technology was seen as interesting from a theoretical perspective, but the idea never took off, simply because the world wasn’t generating enough data to make it operational.

Wikimedia Commons

For the next thirty years, right up until 2010, researchers and programmers focused on developing artificial intelligence through the so-called “expert systems” approach. This was an approach where a career programmer would literally sit down with a machine and write code for every eventuality. The machines were fine, so long as you asked them the right questions. But as soon as you or the environment strayed from what they had been programmed to do, they failed.

Cloud Field Day Roundtable – Big Data from Stephen Foskett on Vimeo.

Everything changed, however, after 2010, when data scientists realized that the internet and embedded sensors were generating enormous amounts of data that could be fed to Hinton’s and LeCun’s neural nets, (machine algorithms that learn more like people). Once they started feeding systems information from the web, they discovered that machines could do things which had previously been thought impossible, like drive cars or tell the difference between a cat and a dog.

Now the world is going totally crazy over big data because of it’s potential to help machines learn. Intelligent machines have the potential to revolutionize human existence and possibly free us from any material constraints.

The Big Data Job Market

As you can probably imagine, the job market for anybody who can manipulate big data or build AI systems is exploding. Institutions, like The Data Incubator, are creating bridge courses designed to help feed talented graduate students into the industry. And companies like Facebook and Google are buying up entire university computer science departments in the hope of capturing all available people with the skills they need to launch the next industrial revolution.

Demand for people with data skills is so high these days that some students, fresh from universities, are being offered six-figure salaries on arrival, with pay doubling within a couple of years. Data fluency is that valuable to companies right now. At least right now, it seems as if this is where the real potential in the job market lies: in the ability to develop and train intelligent systems.

Almost all of the fastest growing job segments in the economy have some relationship to computers and big data. And most top paying jobs in industries outside the traditional IT sector now demand that candidates have some sort of big data or artificial intelligence experience.

Looks like it’s time to get on the big data hype train.