Archival data may not require high performance computing, but it does need to stay accessible for productivity, legal, business value, analytics, and compliance. IT must secure that data against ransomware and other types of cyberattacks. And IT needs to do all this at reasonable costs, even given extraordinary projected data growth.
Active archiving technology is constantly evolving as marketplace demand increases. We recently released our 2022 Report: The Active Archiving Ecosystem: Building a Flexible Archival Repository Your Way, highlighting the increased demand for new data management strategies and the benefits and innovations of active archive solutions.
Top innovations within active archiving include Artificial Intelligence and Machine Learning (AI/ML), sustainability, analytics, and compliance.
By Guest Blogger, Dr. Shawn O. Brume Sc. D., IBM Tape Evangelist and Strategist
According to a study by McKinsey, the average lifespan of companies listed in Standard & Poor’s is less than 18 years! That means that tape technology is already in business almost 4 times longer than the average S&P will survive. Tape technology celebrated 70 years young on May 21st. Tape has been and continues to be the most transforming data storage technology in history.
In the 50’s it was the only viable technology for storing data generated by the few computers in existence. In the 60’s tape took the world to the moon and preserved the data for usage nearly 40 years later when it was retrieved to assist in modern space explorations. By the 70’s Tape was dominating storage, transforming the financial industry by providing the ability to access data on accounts with minimal human intervention. The 80’s and 90’s continued the transformation of data availability by performing transactional data storage for ATMs, but also was key in the investigation of the space shuttle Challenger disaster; an investigation enhanced as a result of the durability of tape even when submerged in saltwater.
Today tape lives in the data center, preserving Zettabytes of data. Data being preserved and utilized across nearly every industry, examples:
Healthcare – Data preserved on tape is being utilized to develop new predictive health services. Digital medical records can be retained for the life of patients and shared across organizations.
Financial – Online transaction retention ensures customers valuable financial data is protected in the eventuality of a cyber-attack. Mortgage loans are preserved without fear of tampering.
Cloud – Data stored in public clouds are growing at a 30% faster rate than traditional storage. Cloud providers rely on tape to provide data durability and low-cost storage subscriptions.
Tape’s popularity has often been driven by the low cost of storage, modern data storage requires so much more including cyber-resiliency, data durability and low carbon footprints that enable sustainable IT.
Cyber Resiliency – Tape is the only true airgap data storage solution available.
Data Durability – Tape has a native single copy durability of 11- Nines. This means the likelihood of a single bit failure is 1 in 100 Petabytes.
Sustainability – At scale tape technology is 96% lower carbon footprint than highly dense HDD storage (when comparing OCP Bryce canyon and IBM tape technology with 27PB of data).
If preserving data, in a cyber-resilient solution, at low cost, with relatively low carbon impact meets your business outcomes, then why wait? Clearly tape is here to stay and surging in usage across nearly every business use case.
Happy 70-years to an amazing technology!
For more information about technology since tape’s introduction, check out this post from my colleague Mike Doran.
For more information on current tape products see the IBM product page.
The 2021 Active Archive Alliance annual market report has just been released, entitled “Saved by the Data. Active Archive Leads the Way in a Mid-Pandemic World”.
Certainly, the COVID pandemic was a shock to many companies and put tremendous strain on operations, revenue, and profit. But those companies who had already implemented a sensible active archive strategy were at a competitive advantage thanks to their ability to intelligently manage access to their data.
I think active archiving, the practice of keeping data online all the time and easily accessible to users, is a hot concept in storage right now because it is really about optimization – getting the right data in the right place, at the right time, and at the right cost.
We know that IT budgets are not keeping up with the relentless growth of data. We also know that 60% to 80% of data quickly becomes archival. Typically after 30, 60, or 90 days, files become static and the frequency of access drops off. So why keep that kind of data on expensive primary storage?
Why not let intelligent data management software that is typical of an active archive solution move that data by user-defined policy from high performance, expensive tiers, to lower performance but more cost-effective tiers like economy disk or tape systems, or even cloud? All while maintaining transparent access for users.
We know that the value of data is increasing, retention periods are getting longer, and users want to maintain ready access to their data without IT staff intervention. But we also need to worry about the bottom line, about efficiency, compliance, sustainability, and cybersecurity! Active archiving provides the right solutions to these worries and that’s why it is such a hot concept in storage today.
But enough said, read the full report here and check out what Alliance members had to say in their related virtual conference.
Archival data is piling up faster than ever as organizations are quickly learning the value of analyzing vast amounts of previously untapped digital data. Industry studies consistently find that the vast majority of all digital data is rarely, if ever, accessed again after it is stored. However, this is changing now with the emergence of big data analytics made possible by Machine Learning (ML) and Artificial Intelligence (AI) tools that bring data back to life and tap its enormous value for improved efficiency and competitive advantage.
The need to securely store, search for, retrieve and analyze massive volumes of archival content is fueling new and more effective advancements in archive solutions. These trends are further compounded as an increasing number of businesses are approaching hyperscale levels with significant archival capacity requirements.
An active archive resolves complexity by leveraging the benefits of an intelligent data management layer, and an increasing number of effective software products that address this are now available. Access and management of data is getting more complex and is requiring modern strategies with intelligent data management techniques. These strategies and techniques are now being enhanced by AI to further improve and automate data management.
I just spent a full day at a meeting of the Active Archive Alliance and as I was flying home it occurred to me that it’s time for data storage managers to rise up from the sleepy status quo of buying more disk arrays to address runaway data growth problems. It’s time to wake up and smell the sweet aroma of freshly made modern data tape (sort of like that new car smell if you don’t know).
Why do that you ask? Because best practices and undeniable facts say so. Consider the following:
Data goes through a lifecycle from hot to cold, that is to say from a period of active use to a period of inactivity. This can happen in as little as 30 days or less.
Inactive data should not stay on primary storage devices. It takes up space on expensive storage media, consumes more energy and adds to the backup burden.
What to do? Delete it? You probably can’t get permission to delete it, all data is now potentially valuable with new artificial intelligence (AI) and analytic tools emerging to derive value from that data. But you can move it and stop copying it!
Where do you move it to? Put it in an active archive consisting of low cost disk cache and even lower cost long term storage like a high density automated tape library. To store one petabyte of data for 10 years in a tape library will cost around $220,000 depending on your TCO variables. Alternatively, you could spend $900,000 on HDD and around $1,300,000 for cloud. Need more capacity? Tape libraries easily scale by adding more slots and tapes. You can export full tapes and plug new ones in. Move the full tapes offsite and get the benefit of air gap since the data is physically isolated from other networks. At least you know that data can’t be accessed and held for ransom.
Getting end user access requests for that data all of a sudden? Move it back to disk cache and serve it from there. When done, move it back to the tape library. Tape is super-fast, 360 MB a second and file access is made easier and faster with LTFS.
How to orchestrate all this? Intelligent data management solutions help move data automatically. Leverage metadata and AI tools to analyze files and move them off primary storage if they don’t belong there.
Does this sound like a tiered storage strategy? It is and it’s also known as an active archive. This is a best practice used by the biggest and most advanced data generating companies in the industry. If it works for them, it will work for you too.
There’s a lot of hype in the storage industry with lots of folks looking for new, better ways to do things. But some things are tried and true, like tape, with the benefits of constantly evolving capacities, performance, reliability and long term archivability. So wake up and smell the tape…put your data where it belongs and get on with your day!
Kevin Benitez
Product Marketing Manager
FUJIFILM Recording Media U.S.A., Inc.
Over the past decade, the Media and Entertainment (M&E) industry has experienced a considerable increase in the amount of data generated due to the transition from traditional media workflows to fully digital environments. Today, the retention and accessibility of digital assets and video are incredibly vital to maintaining a competitive advantage. Fujifilm understands M&E companies’ digital storage challenges; that’s why companies like MLB, The LA Kings Ice Hockey Team, Chainsaw Edit, and others have turned to Fujifilm tape to ensure the integrity of their video assets while drastically reducing long-term storage costs.
Today, tape storage is used in modern infrastructure to deliver high storage capacities with low cost of ownership compared to other storage solutions.
Modern M&E companies continue incorporating data tape storage into their environments to combat costs. Nowadays storage requirements are on a different scale from where M&E companies first started, and data will continue to increase exponentially as the industry moves from HD to 4K, and soon to 8K recording. Today, a single digital 4K camera can record up to 1.5 TB for every hour of filming. Before this, companies like the LA Kings only recorded 200 GB in an entire year. Most IT budgets in the industry are not growing enough to support today’s data deluge—of which data storage can consume 70%.
“Tape storage in general is the lowest cost storage of any type of storage,” says Storage Economics expert Brad Johns
The cost of using LTO-7 tape is as low as $0.01/GB which can be 7 times less expensive than disk storage over a ten year period.* Additionally, tape doesn’t use any energy when it’s not being used, on the other hand, disk systems use 76 times more electrical power than a similarly configured tape system.* “Tape storage in general is the lowest cost storage of any type of storage,” says Brad Johns, founder of Brad Johns Consulting LLC, a storage consulting firm.
Why are leading M&E companies turning to tape?
Extremely cost-effective
Highly reliable
Portable to use at remote locations
Scalable to extremely large capacities
Open standards to allow the interchange of files
LTO tape is an ideal solution for M&E companies. LTO is an open format designed for interoperability and together with LTFS, provides easy data access and management—perfect for easy file share, high performance, and improved workflow.
How are M&E companies are using tape today?
NewBay Media has compiled how LTO technology and LTFS manage every stage of content creation/ management:
Production. LTO technology with LTFS protects original content with on-site backup copy, reduces camera media inventory costs, and enables the interchange of content between production sites and post-production.
Post-Production. LTO technology with LTFS offers a low-cost storage solution for work-in-progress, scales to meet large capacities, provides a standard means of interchange across the post-production ecosystem, and gives users the ability to offload less active content from expensive, high-performance flash or disks.
Distribution. LTO technology with LTFS supports the transfer of large amounts of digital content at low cost and serves as the de-facto standard for submission of content—to studios and between business partners.
Archiving. LTO technology with LTFS is ideal for long-term storage due to its durability, reliability, and low cost of operation. It scales to meet very large capacity requirements and supports rapid restoration for the repurposing of content.
Tape’s low-cost acquisition price per gigabyte and TCO advantage compared with other storage mediums make tape the most cost-effective technology for long-term data retention. Today, M& E companies can compare the total cost of ownership of data retention using TCO calculators created by Brad Johns Consulting, https://page.dternity.net/TCO.html, and the LTO consortium, https://www.lto.org/resources/.
More M&E companies are seeing the advantages of LTO tape, which can store massive amounts of data and combat ever-increasing storage costs across production, post-production, distribution, or archiving.
*Source: ESG report “Analyzing the Economic Value of LTO Tape for Long-Term Data Retention.”
*Source: The Clipper Group “Continuing the Search for the Right Mix of Long-Term Storage Infrastructure —A TCO Analysis of Disk and Tape Solutions.”
In this video, Fred Moore, president of Horison Information Strategies explains the benefits of an active archive including improving tape performance, providing random access to tape data, and eliminating mount time. Watch the video here for more information:
By Rich Gadomski
V.P. Marketing
FUJIFILM Recording Media U.S.A., Inc.
Have you ever noticed the grainy quality of surveillance footage shown on the local news broadcasts? Some unfortunate citizen is getting assaulted after making an ATM withdrawal and you feel like you’re watching a 1930s Charlie Chaplin rerun. The reason for this is that most organizations charged with surveilling public places can’t afford to do so in high resolution and certainly can’t afford to keep the surveillance content for any considerable period of time. Yet wouldn’t it be nice to easily ID the suspect and maybe go back to other video databases to see if there’s a pattern occurring over time?
As a society we want better security and swifter justice. We have the technology to achieve this but we don’t necessarily have the budget. The good news is that a record number of surveillance cameras are shipping year after year. They are also becoming more affordable and resolutions are steadily increasing to include 4K or better.
In fact, demand for video surveillance systems is continuing to explode driven by several factors including increased security threats, legislation, IoT applications, law enforcement applications and increasing affordability of surveillance cameras.
All of this is taking place at a time when society is demanding more transparency into what is happening in our public places and in law enforcement actions. This has resulted in a steady increase in evidentiary content from facility security cameras to body worn cameras, dashboard cameras, interview room audio and video recordings, cell phone data and others. This proliferation of surveillance technology has given rise to the “transparency paradox” where the public demands more recorded evidence which creates the need for more data storage. But we only have limited budgets that can’t handle the increasing data retention burdens.
This is where the concept of an active archive comes into focus providing a solution where organizations can afford to maintain online access to all of their surveillance content in a multi-tiered storage system that leverages the speed of flash and disk with the superior economics of tape. In a typical active archive environment, the file system extends across all storage tiers from primary flash/disk to long term, economy tape storage. Content moves by policy from high cost primary storage to low cost, long term storage. Typically this means LTFS LTO tape featuring the highest capacity, longest archival life, highest reliability and lowest total cost of ownership.
With an active archive in place, organizations can easily and cost effectively maintain more surveillance content at higher resolutions for longer retention periods and can maintain this on-premises to safely control chain of custody.
Expect demand for active archives to solve the “transparency paradox.” It should make the local news more engaging too!
Usage of Cookies
Cookies are important to the proper functioning of a site. To improve your experience, we use cookies to remember log-in details and provide secure log-in, collect statistics to optimize site functionality and deliver content tailored to your interest. By continuing to use this site you are giving us your consent to do this. For more information you can read our Privacy Policy.
Kevin Benitez
By Rich Gadomski