When Crawford Media Services realized that the data it was archiving for clients was quickly outgrowing the scale and resilience limits of RAID technology, the company looked for a better alternative—one that could create an archive of unlimited size and preserve digital assets in perpetuity with 100% reliability. The answer combined Quantum’s Lattus object storage with a Quantum Artico archive powered by StorNext data management.
One of the country’s preeminent media service companies, Crawford Media Services provides clients with everything from post-production audio services to descriptive metadata writing to asset management and archiving. The company’s list of projects covers the gamut: from episodic television stalwarts like The Walking Dead to major motion pictures like Lincoln and Hidden Figures to industry leaders, such as Coca-Cola, PBS, and Viacom.
“One of the ways that we have stayed successful and kept growing over the last 30 years,” explains Steve Davis, Crawford Executive Vice President and Chief Technology Officer, “is by constantly being aware of where the industry is moving and making sure that we offer our clients the level and kind of services that will keep them competitive over the long term.”
A perfect example of Crawford’s intelligent growth is the company’s innovative approach to archiving. The company watched the seismic changes as media production shifted from linear recording to a completely file-based approach, and it recognized that major changes were going to be needed on the storage side.
“Initially, people used a file-based workflow for new work and for some selected existing content when it needed to be reused, but we saw that was going to change,” says Davis. “People stopped keeping their old viewing and editing platforms as they aged, and much of the old, linear media started reaching the end of its useful life. That meant that thousands and thousands of hours of material was going to need to be digitized and stored somewhere. We realized that would be a major challenge for our customers —and an opportunity for us to create a next-generation archive.”
CAPACITY REQUIREMENT TESTS THE LIMITS OF RAID FOR ARCHIVING
Crawford already offered a sophisticated archive service for its clients based on tried and true combinations of existing technologies—RAID 6 arrays and digital tape—adding their own digital asset manager, Engage, to let clients view proxies and request full-resolution copies of material.
“Like everyone else, we replicated files for protection on disk and put a tape copy in deep archive—way down in the salt mines—for the ultimate DR protection,” explains Davis. “But, we also recognized that we were fast approaching the limits of that technology, and that we needed to make fundamental changes if we were going to continue to serve our customers.”
The big issue was the long-term storage of preservation-grade media files, ones that had the same resolution as the full-production originals. “The files we are talking about are huge, with 220Mbps bit rates and volumes of 100GB or more for every hour of content,” says Davis, “and we are talking about thousands and thousands of them moving to file-based storage that needs to last in perpetuity.”
This massive amount of data was quickly outpacing RAID-based disk systems. “RAID gives you redundancy at the level of the disk. As you get more disks and they get larger—now 4TB disks are common—the odds of a failure go up. The consequences are massive, and disk rebuilds can last for days. During that time data is vulnerable, and the system’s performance is degraded.”
Crawford decided that it needed to move to a different system before its archive grew very far beyond a petabyte—a watermark that the company reached last year. Before that time, however, the IT team had spent almost two years investigating and planning the technology changes it would need to support its second-generation archive, called AMBER.
INVESTIGATING ERASURE CODING AND OBJECT STORAGE
Very early on, the Crawford Media team had a strong interest in storage based on erasure coding. The team was convinced it could solve the problem of managing a huge and growing archive over an extended—essentially unlimited—period of time.
Erasure coding is a forward error correction methodology developed more than 50 years ago. It has been used extensively by NASA and telecommunications firms to ensure the highest levels of data integrity in situations where errors were very likely to occur. The basic technique breaks data into many discrete units, includes parity information with each, and then spreads the data, with redundancy, over many different storage nodes. If errors occur, or if the disk that data is stored on fails, the data is automatically recreated from the other locations. In erasure code-based systems, disks can be replaced with a broader range of choices, there is virtually no down time when disks fail, and the integrity of the data is much higher than with conventional RAID.
“Erasure coding is much more resilient than RAID and provides better integrity, but it also requires more compute power, so it has a somewhat higher latency,” Davis explains. “So, today it isn’t the best choice for high-performance media operations, but it’s perfect for a large-scale archive like AMBER.”
LEVERAGING LATTUS FOR A SCALABLE, RESILIENT ARCHIVE
“Our challenge was to develop a digital archive that would become more robust as it grew in size, rather than more fragile. We explored solutions that would allow us to keep large data sets cost-effectively in perpetuity without degradation, and to survive the costs of media and technology refreshes over time. After extensive research and testing, it became clear that object storage with erasure code—and specifically, Lattus—was the ideal choice for our AMBER archive.” Lattus provides several distinct advantages.
“We knew the technology behind Lattus and had been very impressed with its maturity, resilience, and scalability,” says Davis. Quantum also offers an appliance-based implementation, Artico, that integrates Lattus with Quantum’s StorNext data management platform. Quantum’s StorNext File System makes it easy to use the archive, and its API enables integration with Engage. It also supports tape for clients who rely on removable media for content transfer. “Experience and support were also very important to us. Quantum is a premier storage company with as much experience in media and archiving as any company on the planet.”
SCALABILITY TO PRESERVE DATA AND REDUCE COSTS
“Lattus lets us use either SSDs or conventional hard disks as our cache,” says Davis. “And Lattus lets us scale the system easily and non-disruptively in small increments, allowing us to grow at the rate that our data increases while minimizing up-front capital investment. It even allows us to incorporate new technologies easily. We simply add newer drives to the existing pool, and they coexist with the older storage. With RAID, we would have needed a forklift upgrade to do that. All of this helps us offer our clients substantially lower rates than conventional cloud services.” Unlike other object storage approaches, Lattus also creates storage that is immediately consistent, even when data is spread across multiple geographies, so that read operations always produce correct results.
The new Crawford AMBER archive was initially sized to accommodate the company’s current 1.2PB client archive pool and provide room for additional growth. Clients view their content in proxy view through a web interface, using Engage, which sends full resolution copies to an FTP server, tape, or other delivery system. “One of the strengths of the system is that our clients are not limited to recovering whole files,” Davis explains. “StorNext supports partial file recovery, so users can minimize data movement by bringing back specific timecode ranges.”
The AMBER archive built on Lattus has just started operating, but it is already making a difference. “Once the system was up and tuned, it became transparent and requires very little management. Files go in, where they are safer than on any RAID,” says Davis. “Clients see content through Engage just like before, and have it delivered quickly at preservation-level quality. “But, Lattus gives us the ability to provide a level of archiving service that we have not seen from anyone else in the industry—100% data integrity, migration to newer technology in the future through non-disruptive scalability, and the option of storing duplicate copies of data in different locations for an additional level of DR protection.”