It’s that time of year again to look ahead at what the New Year might bring in terms of data storage. Looking back at our predictions for 2022 to see how well we did, I think we nailed 4 out 5.
The 27th Conference of the Parties to the United Nations Framework Convention on Climate Change (COP 27) concluded on November 20, 2022 with some poignant comments by António Guterres, the Secretary-General of the United Nations:
“It will take each and every one of us fighting in the trenches each and every day. Together, let’s not relent in the fight for climate justice and climate ambition. We can and must win this battle for our lives”.
Reading these words inspired me to look back on the year of 2022 and acknowledge that we in the IT industry are focused on sustainability as evidenced by the trade show agendas that commonly addressed this issue.
Sustainability has been a hot issue at all of the trade shows I have attended this year from Data Center World in Austin back in March, to the Sports Video Group Forum in July, to the Flash Memory Summit in August to the Open Compute Summit in October. So it was not surprising that SC22 held in Dallas earlier this month also had a heavy sustainability agenda for its HPC target market.
I had the privilege of attending the OCP 2022 Summit held in San Jose last week with the theme of “Empowering Open”. I say privilege because it was inspirational for me, and I suspect for many of the 4,000 or so attendees as well. The reason for that was the announcement by OCP Foundation CEO George Tchaparian (pictured at right) that the four tenets of OCP that have acted as guiding principles for 11 years now, namely Efficiency, Openness, Scalability, and Impact are to be enhanced with a fifth tenet, and that is Sustainability. It is the mission of OCP to foster innovation in the data center industry and now sustainability must be a part of all the ideas that OCP incubates and advances. George shared his vision that the modern data center will not only be scalable but also sustainable!
John Monroe, a long-time storage industry expert and Gartner analyst turned independent consultant, recently published a new report entitled “The Escalating Challenge of Preserving Enterprise Data”. The report, co-sponsored by Fujifilm and Twist Bioscience, looks at the supply and demand for SSD, HDD and tape technologies from 2022 to 2030. The findings and conclusions in John’s report are surprising to say the least and should serve as a wake-up call for executives in both the end user and vendor communities. Below are some summaries and excerpts taken from the report and a link is provided to view/download the full report.
By Guest Blogger Peter Faulhaber, former president and CEO, FUJIFILM Recording Media U.S.A., Inc.
The Hyperscale Data Center (HSDC) secondary storage market is quickly emerging, requiring advanced solutions for petascale and exascale storage systems, not currently available. According to HORISON Information Strategies, HSDCs currently use around 3% of the world’s electrical energy. Due to the massive energy footprint of HSDCs, climate protection measures have become increasingly important in recent years, with cloud computing offering the greatest advantages for sustainable operation by reducing the energy and carbon footprint over the entire data life cycle.
The slowing rate of HDD and tape technology development roadmaps in recent years, along with HDD and tape storage supplier consolidations are particularly concerning trends to HSDCs. Neither HDD nor tape technology is currently positioned by itself to effectively meet the enormous HSDC storage requirements that future performance and capacity demands. High technical asset specificity requires significant R&D investment, yet have limited ROI potential outside of hyperscalers.
HSDCs manage over 60% of the world’s data today with a CAGR of 35 – 40%, with a growing need for cost-effective secondary storage that still meets certain performance thresholds.
The vendors and manufacturers are dis-incentivized to invest in novel technology; the risk reward is not high enough, while HSDCs are leveraging their buying and bargaining power. Manufacturers need to invest hundreds of millions to bring innovative solutions to market in a long development cycle, without a commitment from the HSDC market.
As a result, the secondary storage market is left with incremental investments in existing technologies and moves slowly.
The conditions are set for a widening gap between customer demands and product solutions in the secondary storage market.
The current “vendor-driven” strategy will not keep pace with HSDC requirements for secondary storage as such offerings fall far behind HSDC curves. Photonics, DNA, glass, and holographic experiments are attempting to address the market, and have been in labs for decades, but most have drawbacks, and none are on the near-term horizon for customer deployment. These initiatives show that a change is needed to get ahead of the demand curve.
However, the opportunity also exists to mitigate this risk by bringing the interested parties together to share the risk reward paradigm. HSDCs need a quantum leap, which only comes with significant investment, best shared by the interested parties.
The Semiconductor Research Corporation (SRC) addressed the concept of vertical market failure in September 2021 in its published article “New Trajectories for Memory and Storage,” stating, “The prospect of vertical market failure can be mitigated by private sector market participants through risk-share agreements between customers and suppliers, as well as increased vertical integration.”
Without change, current technologies will fall far behind HSDC demand curves, and the current vendor-driven trajectory increases the likelihood of un-met demand and stagnation of growth for all involved.
The LTO Technology Provider Companies (IBM, HPE, and Quantum) issued a press release earlier this week announcing record capacity shipments for LTO in 2021 of 148 Exabytes (148,000 Petabytes) compressed (up from 105 EB compressed in 2020, +40%). More and more of the world’s data is being stored on LTO data tape. That’s good news for the IT industry! Is it not? After all, end users and service providers need:
Industry Pundits React
Some industry pundits, biased toward the HDD industry, took the opportunity to downplay the news. They said the data is inaccurate or insignificant compared to the capacity shipments for HDD last year. Really? Does tape technology threaten the market for HDD? Is it still about tape vs. disk in their minds? Have trains, trucks, and ships put air freight out of business? Or does a more strategic thought process say: “These technologies complement each other. We need both to meet the needs of end-users, storage service providers, and society itself…”
Analysts Predict Huge Zettabyte Demand
Indeed, if the big industry analysts firms are right, we will need to be storing more than 11.0 zettabytes of data in 2025. Just one zettabyte would require 55 million 18.0 TB HDDs or 55 million LTO-9 tape cartridges. Should we store all of that data on HDD, whether it is hot, warm, cool, or cold? Of course, we can’t just delete excess data. Now that we can analyze the data and derive a competitive advantage from it, the value of data has increased and we need to store more and more data for longer periods of time. As a result, the projections for the amount of persistent data to be stored are growing exponentially. We will need huge amounts of flash, HDD, tape, and even future storage solutions like DNA to address the data storage challenge.
A Strategic Approach to Data Storage
Change of Attitude Needed
But this is the wrong attitude and exactly what has to change to make a difference. Collectively, if we all switch off a light and all turn the car’s engine off, we will make a difference. We might even get motivated for more change! How about installing LED light bulbs or investing in an electric vehicle? Or maybe make the commitment and take the leadership on a renewable energy installation. Attitudes have to change, believing we can make a difference collectively. If data is inactive, why keep it on energy-intensive, constantly spinning disk? Are we all doing whatever it takes to make a difference?
New Flagship UN Report Is a Wake-up Call
If we believe the latest studies on climate change coming out of the United Nations, we need to start quickly taking any action we can. A new UN report on climate change from earlier this month indicated that harmful carbon emissions in the last decade have never been higher in earth’s history. It’s proof that the world is on a “fast track” to climate disaster. UN Secretary General Antonio Guterres has warned that it’s ‘now or never’ to limit global warming to 1.5 degrees C. Climate change is the result of more than a century of unsustainable energy and land use, lifestyles, and patterns of consumption and production. Guterres adds, “This is not fiction or exaggeration. It is what science tells us will result from our current energy policies. We are on a pathway to global warming of more than double the 1.5-degrees C limit” that was agreed in Paris in 2015. To limit global warming to around 1.5 C (2.7 F), the IPCC report insists that global greenhouse gas emissions will have to peak “before 2025 at the latest, and be reduced by 43% by 2030.”
Reducing Energy Consumption and CO2 Emissions with Tape
To help increase awareness and understanding of energy consumption in data storage, a number of whitepapers have been published highlighting alternative options for storage including LTO data
Other research shows that tape consumes 87% less energy than equivalent amounts of HDD storage. When CO2 emissions are analyzed over the entire product lifecycle (from raw materials to production to distribution, usage, and disposal) of HDD and tape, studies show a 95% reduction in CO2 in favor of tape compared to HDD. The same study shows Total Cost of Ownership for long-term data storage can be reduced by more than 70% by using tape instead of HDD. At the same time, tape can provide an effective defense against cybercrime via a physical air gap. All of this is possible by taking a strategic storage approach, where cool or cold data that has aged and is infrequently accessed gets moved from expensive primary storage to economical and environmentally friendly tape systems, online or offline.
Data Center World Attendees Get It
In my last blog on my visit and presentation at Data Center World in Austin last month, I mentioned that I was encouraged by the DCW attendees that I met and listened to in my session and other sessions. They are genuinely concerned about the environment and worried about what kind of planet we will be leaving behind for our kids and grandchildren. They recognize the opportunity to improve sustainability in data center operations and are committed to it. But since then it has occurred to me that maybe sustainability is more of a focus for facility teams. Perhaps the top-down pressure from the C-suite has yet to be widely applied to the data storage management teams. However, in the quest to achieve the needed sustainability goals, no stone can remain unturned.
Observing Earth Day for Future Generations
With Earth Day being observed today, let’s commit to strategically taking action in response to global warming and climate change. Let’s start changing attitudes from “It won’t make a difference” to “Collectively, we can make a difference.” Let’s look at the bright side of increasing LTO capacity shipments instead of the dark, self-serving side. Let’s think about the planet that’s home for us and the future generations of our families to come.
I had the opportunity to present at AFCOM’s Data Center World (DCW) exhibit and conference in Austin, Texas yesterday. The first thing I have to share about this experience is how surreal it was to get back on an airplane! It was my first trip since COVID started two years ago with many zoom presentations and virtual conferences since then. But not much has changed about air travel. The seating is still cramped, the flight was packed full, and my dog gets more snacks in a four hour period than I did on my four hour flight!
Committed to Sustainability
Sustainability is a hot topic these days and was one of the main themes of this year’s DCW. It was also the topic I presented on, specifically “How to Avoid CO2 Emissions in Long Term Storage with Modern Data Tape Technology.” The good news is that the DCW attendees that I met and listened to in other sessions are genuinely concerned about the environment and worried about what kind of planet we will be leaving behind for our kids and grandchildren. They recognize the opportunity to improve sustainability in data center operations and are committed to it.
Key Questions about Storage
At the outset of my presentation, I asked for a show of hands for those directly involved in data storage. I was not surprised to confirm my suspicion that there would be few if any attendees to raise a hand, since AFCOM’s DCW is more about facilities management than storage management. But I was also glad to see this because we need everyone to be advocates for any possible sustainability improvements in IT operations. So I asked my audience to lean on their colleagues in storage and pose two simple questions to them: “If data has gone cold and is infrequently accessed, why are we keeping it on energy intensive tiers of storage like constantly spinning and heat producing HDD arrays? Why not move it to eco-friendly tape?” The attendees in my session admitted they can feel the power drain and heat being produced by endless disk arrays in their data centers.
Climate Change and Global Warming
I began my presentation by setting the stage on global warming from the forest fires in 2020, to the Texas deep freeze in early 2021, to the fact that July of 2021 was the hottest month ever on earth. Add to this the dire reports from the U.N. in late 2021 and early 2022. All this has led to changing consumer sentiment demanding that governments do more. Thankfully they are. Corporate attitudes are also changing from resistance to action on climate and we will be seeing more CSOs (Chief Sustainability Officers) being appointed and implementing change top down. Even Wall Street and the SEC are getting in on the act, demanding reporting and disclosures on corporate sustainability initiatives.
Energy Intensive IT Industry
Next, I confirmed what we all know, that the IT industry is energy intensive and its demand for energy is rapidly increasing. The demand curve for energy looks similar to the demand curve for data storage. Driven by digital transformation, IDC expects persistent data that needs to be stored to grow from 2.0 ZB in 2016 to more than 11.0 ZB in 2025, a CAGR of 27%. Suffice it to say no one in the audience really understood what a zettabyte was or that just one zettabyte was equal to the capacity of 55 million LTO-9 data cartridges or 55 million 18.0 TB HDDs. That’s a lot of storage requirement for one zettabyte, let alone 11.0 zettabytes in 2025. We are going to need a lot of flash, disk and tape to handle that kind of volume!
Renewable Energy plus Conservation
Next came the conversation about renewables and how Greenpeace has done a great job advocating for more use of renewables in data centers, especially the cloud hyperscalers. But from the looks of progress being made on this front, renewable sources of energy likely can’t come on line fast enough or cheaply enough, or in sufficient volume to satisfy the energy needs of the massive data center industry. While Fujifilm is a big fan of renewables (we use it ourselves for our LTO plant in Boston) what’s really needed is a combination of renewables and energy conservation. How about turning off those lights and HDDs before leaving the office each night!
The Data Life Cycle
When it comes to conserving energy in data storage, one needs to understand a few simple principles related to the “data lifecycle.” Data quickly goes cold and access frequency drops off dramatically after 30, 60 or 90 days. At the same time, data retention periods are getting longer, sometimes reaching indefinite time periods. This is where data tiering saves the day as cold data can move from expensive, energy intensive tiers of storage to economy, eco-friendly tiers like modern data tape.
Advantages of Eco-Friendly Tape
I then shared the research findings from Brad Johns Consulting in his two white papers where tape consumes 87% less energy and produces 87% less CO2 than equivalent amounts of HDD storage. When analyzed over the total product lifecycle from procurement of raw materials to production to distribution to usage and finally disposal, tape produces 95% less CO2 than HDD and produces 80% less e-waste. I also shared the results of an IDC study that shows migrating more cold data from tape to HDD could result in an avoidance of 664 million metric tons of CO2 on a global basis by 2030. That’s the CO2 equivalent of 144 million automobiles being taken off the road for a full year! I also referenced research by IBM showing a side by side compare of TS4500 tape library and Bryce Canyon HDD where the IBM gear produced 80% less CO2 over a ten year period than the Bryce Canyon system. To round things out, I shared the end user perspective from an executive roundtable where Microsoft Azure stated:
“When you take the material savings and power savings, tape actually does offer quite a bit of advantage compared to other technologies that are on the market today.”
Since my audience wanted to know more, I briefly covered tape’s other benefits including:
I concluded by saying that data growth is here to stay and the volumes of valuable data are getting enormous. What the industry needs to do in support of strategic data storage management and sustainability objectives is this:
“Get the right data, in the right place, at the right time, at the right cost, and…at the right energy consumption level.”
I think the attendees got the message and now see modern tape storage as part of the carbon reduction answer for the data centers of today and tomorrow. It was well worth the snack deprived four hour flight!
I think it’s safe to say people like surveys, probably not everyone, but most people do. Why? Experts in the field suggest that people are willing to take surveys because respondents feel their opinions are valued and that their answers will be used and may even result in a benefit to society. They feel their participation will impact something they care about, and they want to share their opinion with those who will listen and act on the information.
Surveying the C-Suite on Sustainability
So it’s not surprising that Fujifilm got a great response rate to a recently launched survey entitled Awareness Survey on Environmental Issues in the Digital Domain. As many as 1,200 C-suite professionals responded including CEOs, CFOs, CSOs, CTOs, and CIOs from companies of 100 or more employees in the United States, Germany, Japan, and China.
The survey revealed that there is a growing awareness around broader environmental issues among corporate leaders, and that’s great news as the negative impacts of global warming and climate change keep piling up, flood after flood, wildfire after wildfire, and storm after storm.
When it comes to IT infrastructure specifically, the majority of U.S. respondents believe sustainability improvements in IT services and equipment can positively impact climate change, but 40% indicated that they did not know or were unsure if data storage can have a negative environmental impact and increase the cost of doing business.
Increasing Data Storage Requirements
Data storage can certainly be energy-intensive. This is a problem that is only getting worse as the value of data rises with the ability to analyze and derive competitive advantage from it. As a result, demand for long-term data retention is increasing. In fact, data to be stored is expected to grow from just 2.0 zettabytes in 2016 to 4.1 ZB in 2020 and is expected to reach 11.1 ZB in 2025 according to a recent whitepaper from IDC. Just one ZB is a vast amount of data equal to one million petabytes that would need 55 million 18 TB hard disk drives (HDDs) or 55 million 18 TB LTO-9 tapes to store. The environmental impact of the energy required to support this volume of storage is greatly underestimated, as are the associated carbon emissions. When asked in the survey what barriers exist for those who have not considered more eco-friendly data storage options, 31% in the U.S. cited a lack of awareness or understanding of the issue.
Hot vs. Cold Data
There was also a lack of awareness pertaining to frequently accessed “hot” data and less frequently accessed “cold” data, with 36% of respondents saying they either don’t or are unsure if they differentiate between the two. And 35% don’t realize that differentiating between hot and cold data can impact sustainability, affordability, and security. An interesting fact about data is that it quickly goes cold and access frequency drops off significantly after just 30, 60, or even 90 days. In fact, industry analysts estimate that 60% to 80% of all data stored is cold and qualifies as “archival”. Yet through inertia, that data often remains on energy intensive, constantly spinning and heat-producing tiers of storage like hard disk drives.
Reducing Energy Consumption and CO2 Emissions with Tape
To help increase awareness and understanding of this issue, a number of whitepapers have been published highlighting alternative options for storage including LTO data tape. A recent IDC whitepaper shows how migrating cold data from HDDs to LTO tape can reduce data centers’ CO2 emissions by 43.7% by 2030, avoiding 664 M metric tons of CO2 cumulatively. Other research shows that tape consumes 87% less energy than equivalent amounts of HDD storage. When CO2 emissions are analyzed over the entire product lifecycle (from raw materials to production to distribution, usage, and disposal) of HDD and tape, studies show a 95% reduction in CO2 in favor of tape compared to HDD. The same study shows Total Cost of Ownership for long-term data storage can be reduced by more than 70% using tape instead of HDD. All of this is possible by taking a storage optimization approach, where data that has aged and is infrequently accessed, otherwise known as cold data, gets moved from expensive primary storage like solid-state flash drives and HDDs to economical and environmentally friendly tape systems.
As far as security is concerned, tape is also playing a role in cybercrime prevention with air gap capabilities, WORM, and encryption. Intelligent data management software, typical in today’s active archive environments, can automatically move data from expensive, energy-intensive tiers of storage to more economical and energy-efficient tiers based on user-defined policies. By moving inactive data out of primary storage, the ransomware attack surface can also be reduced.
Renewable Energy Plus Conservation
Another interesting point from the survey reveals that 51% of participants said that their companies are using renewable energy to reduce carbon emissions, while 22% said they are doing so via climate protection projects and 13% through carbon offsets. Renewable energy is a key factor in reducing CO2 emissions and Fujifilm is a fan (see photo at right of our LTO plant in Bedford, MA). But alone renewables likely can’t come online fast enough or cheaply enough to keep up with data growth rates of between 30% – 60% annually in major data centers today. That’s why conservation has to be part of the equation. The very first metric to be analyzed in data center energy efficiency is simply the amount of energy that’s being consumed.
Alternative Data Storage Options
Finally, 81% of respondents noted that they would consider an alternative data storage option that is more sustainable and affordable. That option exists in the form of today’s modern and highly advanced data tape systems that offer the lowest energy consumption and cost profile. Add to that its best-in-class reliability rating of any storage media and longest archival life. So for the benefit of society, let’s act on the information that the survey reveals. It’s really just a question of getting the right data, in the right place, at the right time.
The Arrival of the Zettabyte Era
The data storage market has clearly entered the “zettabyte era” where new capacity shipments have exceeded a massive one zettabyte for a couple of years now. The data storage requirements are being driven by the phenomenon of “digital transformation” and the rising value of data that needs to be stored for longer periods of time, and in some cases, indefinitely. Further accelerating the zettabyte era is the other era we are all in, that being the “pandemic era”. With this era comes the unanticipated need for an unexpected remote workforce and the ever-expanding internet with its proliferation of online apps.
Pandemic Related Supply Shortages
The pandemic has brought with it related disruptions to the global supply chain including shortages of semiconductor chips. It’s been tough to get modern goods from toys to notebooks to refrigerators to automobiles. The combination of zettabyte and pandemic era has even put a strain on supply chains and the availability of SSDs and HDDs needed to support the digital transformation. This has been the cause of fluctuating prices based on quarterly supply and demand swings.
Supply Chain Challenges Persist
While pandemic-related labor shortages have delayed the production and distribution of goods, other factors are making matters worse. How about global warming, climate change, and the ensuing natural disasters that have had negative impacts on the supply chain? How about international rivalries and tensions impacting the availability of key components? Or cybercriminals shutting down vital infrastructure? Bottom line: industry pundits say we can expect supply chain hassles to continue throughout 2022.
Supply Chain Contingency Planning in Data Storage
Faced with supply chain risks in any industry, it’s always good to have contingency plans to mitigate risk and ensure ongoing operations. The IT industry is no exception where the availability of commodities that we may take for granted can be interrupted by any of the factors listed above from unforeseen demand to pandemic-related shortages to global warming, trade wars, and cybercrime.
A great way to avoid supply chain disruptions in the availability of primary storage devices like SSDs and HDDs is to employ intelligent data management software, typical of active archive solutions, that will automate the migration of data from these potentially supply chain affected devices to a modern, automated tape library. Since 60 to 80 percent of data quickly goes cold after a short period of time, why keep it stored on higher performing, expensive, and energy-intensive devices? Given the global supply chain uncertainty, 3 good reasons to migrate data from primary storage devices to tape storage are:
The above actually makes sense even in the absence of supply chain concerns. Since data to be stored is growing at a CAGR of around 30% versus IT budget growth somewhere in the low single digits, the IT industry needs to find a more cost-effective storage solution. With the increasing value of data and indefinite retention periods, the long-term archival profile of tape coupled with best-in-class reliability actually makes sense.
Finally, we all have to engage in the battle against global warming and climate change if we are to preserve the planet that we inhabit. Studies show that tape systems consume 87% less energy than equivalent amounts of disk storage and produce 95% less CO2 emissions than disk over the product lifecycle. Other studies show that collectively, the global IT industry could avoid as much as 664 million metric tons of CO2 emissions by strategically moving more data to tape systems. As data cools off or goes cold, it should migrate to less expensive, less energy-intensive, and more secure tiers of storage.
Once the pandemic era finally subsides, it will be environmental calamities brought on by climate change and the relentless threat of cybercriminals that will have long-term impacts on supply chains.
We can help you reduce cost, decrease vendor lock-in, and increase productivity of storage staff while ensuring accessibility and longevity of data.
Contact Us >