Technology

Welcome to the dark side

Big data is creating dark data, a long tail of unused and unseen information. Dark data is risky, but also a chronically under-exploited resource.

14 December 2020

Several years ago, tech royalty and Pivotal chairperson Paul Maritz explained the cloud revolution as such: when PCs replaced mainframes, the cost of a CPU cycle collapsed, and one result was the graphic user interface, as it was computationally ‘cheap’ enough to now track a mouse cursor. Maritz said that the cloud is doing the same again, dramatically pushing down the price of a CPU cycle.

He was right – the much cheaper access to greater computing power, for example, is the direct reason why artificial intelligence is maturing and spreading so rapidly. Big data is another such consequence – before scalable virtualised infrastructure, it was too difficult to track and store large data sets. But today, companies such as YouTube can handle users uploading 60 minutes of video every second, averaging at 50 megabytes per minute of video, or 3 gigabytes a second.

ITWeb Premium

Get 3 months of unlimited access
No credit card. No obligation.

Already a subscriber Log in