Erasure Code: Next-Level Digital Archive Technology
26 July 2016 - 7:12, by , in AMBER, Comments off

Recently, a Quantum Lattus storage system was installed to upgrade our digital archive. Using a cutting edge technology known as erasure code, Lattus provides a level of immunity to system failure and data corruption not previously achievable.


Chief Technical Officer Steve Davis discussed erasure code and its significance to our digital archive service:


Creating a permanent archive of digital data is a real challenge. Why? Because the most practical digital storage technologies, like spinning disks and data tape use magnetic media. Guess what… magnetic media is not archival. All those little bits of magnetic information stored on a hard drive slowly lose their charge, and the mechanism wears out, and becomes obsolete, and the information is even bombarded and corrupted by cosmic rays. The combination of these assaults on your information is called entropy.  Even if no one accidentally erases your data or lightning strikes your storage, if you leave your data in one place long enough entropy will catch up with you. So… every so often you have to copy your data to fresh media before there is too much entropy. In addition to refreshing the media, every scheme for archiving data has relied upon having duplicate copies of the data. The idea is that if one of your copies is lost or corrupted you can fall back on the others. The problem with all of these strategies is that they are expensive, time consuming, and difficult to manage.


A few years back we became aware of a new approach to data storage that promised to be a breakthrough for digital archives. The technology is known as erasure code. When data is written to disk it is first encoded with an algorithm that provides forward error correction. Forward error correction is a technique that assumes some of the data will be lost or corrupted and has the ability to mathematically reconstruct the original data from only a partial return of the encoded data. Once the data has been encoded, it is also broken up into smaller units called “slices” and spread across many storage nodes. When the data is read back the slices are re-assembled and decoded. If any (or even several) of the slices are missing, the forward error correction makes it possible to rebuild the data perfectly from the remaining slices. By building the storage out of many small nodes and utilizing extremely clever policies to spread the slices the probability of data loss from entropy or system failure becomes fantastically small. Many “nines of data durability” as we say in the storage biz.


Erasure code does bloat the data footprint a bit, and there is a read/write performance penalty, but the advantages of erasure code for an archive are overwhelming. Perfect data integrity, immunity to entropy, a much smaller data footprint than replication, high immunity to system failure, automatic re-balancing of data and media refresh as the archive grows. After several years of investigating and testing erasure code systems, Crawford Media Services is excited to announce that we have just upgraded our archive storage to an erasure code system known as Lattus, built and supported by Quantum. With Lattus we believe that Crawford Media Services now offers the highest level of data durability and data integrity of any digital archive service.

About author:
EVP, CTO. Steve most recently served as the Executive Vice President and CTO of Crawford Media Services. His career in media began in the late seventies as an audio engineer during the heyday of analog recording and film production. In 1985 he joined Crawford Communications as it was known then, to direct their audio department. During his stint managing audio Steve served for many years on the board of the Society of Professional Audio Recording Services, where he wrote “The SPARS Time Code Primer,” that was used as a college text for over a decade. He later became a vice president of Crawford’s post production division. In 2007 Steve founded a new division of the company, Crawford Media Management, to provide services around the emerging challenges of managing file-based media. These services currently include digital migration, a managed digital archive, Crawford’s Engage asset management service, and Metaforce, a distributed workforce for creating descriptive metadata.

Comments are closed here.