Magnetic tape storage has found a new lease of life with a record 152.9 exabytes shipped worldwide in 2023 — and the rise of generative AI is a key factor
Magnetic tape storage might be one of the oldest storage technologies, but it appears to have found a new lease of life thanks to the generative AI boom.
A record 152.9 exabytes of total (compressed) tape capacity was shipped in 2023, according to the report from the companies behind the Linear Tape-Open Technology (LTO) program, namely Hewlett Packard Enterprise (HPE), IBM, and Quantum.
That’s growth of just over 3% compared to 2022, which the report puts down to rapid growth in data generation and the increased infrastructure needs of hyperscaler cloud providers and enterprise customers.
It seems that tape is turning out to be useful for storing the huge amounts of unstructured data that result from the adoption of new technologies such as AI.
“As AI and related technologies cause increases in storage requirements and costs, LTO tape technology makes the management of vast amounts of data more cost-effective and sustainable while delivering comprehensive protection against data breaches, which continue to grow in number and sophistication,” the report said.
The vast majority of enterprise data is unstructured and often ignored; however, the rise of generative AI has made this sort of data potentially extremely valuable, either for training AI models or to mine through with AI to find insights.
Some tech investment firms have even suggested that desire to get hold of this unstructured data could lead to a corporate acquisition boom. Either way, finding a way to host these gigantic datasets has become more important for enterprise customers.
Why magnetic tape storage is still in vogue
Magnetic tape has been used to store data since the 1950s, although the LTO format itself only dates back to the 1990s.
Even if it’s an old technology, it is widely used for good reason. Tape is considered to be low cost, highly portable, and can store very large amounts of data. It’s an extremely durable storage choice which can last for up to 30 years.
LTO also features error correction algorithms with an error rate of 1 in 10^19. This, the LTO companies said, would mean it would take a whole year, with 130 tape drives writing data continually, to encounter an error that could not be fixed by LTO’s error correction technology.
The current, ninth generation, technology can store up to 18 terabytes (TB) of data on a single cartridge (45TB compressed); the LTO roadmap plans to increase capacity up to 144TB by generation 12. Since the first LTO products arrived, over 5.6 million drives, 371 million cartridges and 517,958 billion GB of media capacity have been shipped, the companies said.
The companies argue that tape has an 86% lower total cost of ownership than disk and a 66% lower total cost of ownership with cloud storage over a ten-year basis.
The major disadvantage of tape is, of course, that it’s much slower at transferring data which is why it is often used for offline backup situations where reliable storage is more important than speed.
“The probability of losing data through degradation, however, is far less than it being maliciously encrypted in a ransomware attack, or destroyed by fire, flood or acts of war in the data center,” the companies note.
Indeed, one of the other reasons for the ongoing popularity of tape is as a backup technology, something that has become essential as a result of the rise of ransomware. Many companies will use the 3-2-1, or even the 3-2-1-1 backup rule, which means having at least three copies data stored on two different types of storage mediums, one of which is off-site and (for addition peace of mind) one that is off-line: tape is usually the ‘1’ in either case.
“LTO tape will endure as a critical component of storage architectures across the enterprise, especially as tape technology itself continues to improve to handle even larger volumes of data and more intensive workloads,” said Tom Coughlin, storage analyst and president of Coughlin Associates, quoted in the LTO companies’ press release
“The latest annual tape capacity shipment media report signifies the continued importance of LTO tape technology in the digital storage hierarchy not only today, but also far into the future as storage needs evolve in complexity and cost,” he said.
