Jack Schofield

Digital dark age looms

A huge quantity of data has rapidly fallen into a black hole because its format has become obsolete. Jack Schofield goes in search of lost words.

Jack Schofield

The Guardian, Thursday 9 January 2003

When monks were compiling William the Conqueror's Domesday Book in 1085, they probably didn't expect it to last 1,000 years. But they would surely have been shocked by the idea that it would be unreadable in 10 to 20 years, or even 50 years. That, sad to say, is the position most of our digital data is in today. When our descendents look back at the dawn of the "information age" in the UK, they may see a black hole.

A famous case in point: the BBC's contemporary Domesday project, produced with the help of hundreds of thousands of schoolchildren. The videodiscs were rendered almost unreadable in 15 years, when the hardware needed to play them became obsolete. But check your own records before you point an accusing finger. Can you still play your 45rpm singles, vinyl LPs, or reel-to-reel tapes? What about those old Standard 8 or Super 8 or Betamax movies? If you lived through the personal computer revolution, then you may well have dust-encrusted 5.25in or 8in floppy disks containing irreplaceable data. It may be on Sinclair Spectrum cassettes or QL Microdrives, 3in Amstrad disks, and other odd formats. Maybe you can still get that data back. However, the longer you leave it, the more difficult and more expensive it will be, until it eventually it becomes impossible.

Children who have grown up with the IBM PC, launched in 1981, have enjoyed an unusual degree of stability, thanks to the humble DOS floppy. Most rival systems eventually switched to a compatible format, including Amstrad (with the PCW word processor) and Apple (with the Macintosh). But ubiquity is no guarantee of long-term survival. Floppies will soon go the way of punched cards, which were a standard way of storing machine readable data from the US census in 1890 until the late 1970s.

"I still have data written on punched cards, and they are good for hundreds of years," quips Josh Krischer, vice president for systems and storage research at Gartner, "just find me a card reader!"

Krischer says companies should change their data storage technologies "every five years, because after that, the cost of maintenance is inhibitory." You have to keep migrating data to new media, or lose it. Unfortunately, only large companies take data preservation seriously, according to Claus Egge, who runs International Data Corporation's European storage research programme. "In small and medium-sized businesses, very few people realise they ought to be doing better," he says.

"And when it comes to consumers, they have no clue about backing their stuff up, really." There's not much chance of getting them to think about data migration and preservation strategies.

The good news is that all is not lost, and this month the Camileon project announced that it had "rescued" the BBC Domesday system by emulating the BBC Micro and videodisc player in software. Camileon is a joint project between universities in Michigan and Leeds, with the acronym standing for Creative Archiving at Michigan & Leeds: Emulating the Old on the New. This three year project was set up to investigate the use of computer emulation as a way of preserving data, and Paul Wheatley, the UK project manager, says his team devoted its final year to the Domesday discs as a "proof of concept".

"Traditionally, people have looked at using migration - taking the data and moving it to a new format - but the problem is you have to go through so many migrations, because the platforms that you migrate to will also become obsolete," says Wheatley.

The Camileon approach is to convert the data into a digital bytestream, thus removing most hardware compatibility problems. "It's easy to preserve a bytestream - that's basic computer science. Then it's just a question of how you interpret the bytestream, which is where emulation comes in. If you can't render the data, it's just a meaningless stream of bits and bytes."

The idea is to split data preservation in two, and create "a modular tool with input and output formats". You create input filters to convert whatever needs preserving into bytestreams. In the Domesday case, this involved reading all the data into a PC and digitising the analogue video content. Later, on demand, you create output tools to convert the bytestream to whatever format is required. On the Domesday project, they started with a BBC Micro emulator downloaded from the net. "Given that digital preservation work is woefully underfunded, we need to make use of whatever's out there," Wheatley says.

Apart from physical incompatibilities, there are software issues: the programs used to create data, and the file formats used to store it. In these cases, there are rarely any standards, except what the market dictates. The problem is that the market changes. Microsoft's Word document format is dominant today, but in the 1980s, the market leader was Word Perfect, and before that, almost everyone used WordStar. Microsoft's Excel now sets the standard for spreadsheet files, but Lotus 1-2-3 had dominated the market for a decade, after taking the crown from the original spreadsheet, VisiCalc.

Users are split between those who believe there's safety in numbers and those who think there's safety in standards. Of course, both sides believe in standards. "Safety in numbers" really means de facto or "market standards", which may sometimes be ratified and turned into de jure standards. The real questions are whether a file format has been published - in which case it can be implemented by someone else if the supplier goes bust - and whether only one program can read and write files in that format. If so, your files are at the mercy of that program's supplier.

But while open standards are an advantage, they are not a panacea, as the web's display format, HTML (Hypertext Markup Language), illustrates. In theory, the standard is set by the World Wide Web Consortium (W3C). Any company can produce software to generate HTML - examples include Macromedia's DreamWeaver and Microsoft FrontPage - and anybody can write a browser to read it. But as a matter of fact, different programs generate different HTML, while the same HTML looks different in different browsers, or even in the "same" browser on different platforms (such as Internet Explorer on Windows and Mac). Regrettably, web designers therefore tend to support the market standard (Internet Explorer for Windows) rather than the published standard.

Things should improve, thanks to the almost universal backing for XML (eXtensible Markup Language). In particular, there's a noble attempt to create open XML formats for office documents as part of the development of OpenOffice the open source version of Sun's Star Office. Microsoft is also supporting XML in the next version of Office, which is now undergoing beta testing. And on Sunday, Oasis, the Organization for the Advancement of Structured Information Standards, held the first teleconference for its new Open Office XML Format Technical Committee, which aims to ensure interoperability between different implementations of the file format.

So could XML solve that piece of the puzzle? Paul Wheatley is not convinced. He says: "There are quite a few people in the preservation world who think that XML is the solution. I'm a bit more sceptical. Standards don't last forever: they change, and new standards come along. The real test of whether it's succeeded will come in 50 years' time."

At the Public Record Office in Kew, David Ryan sees XML documents as just another kind of document to be preserved, but says they will be using it internally.

"With the storage system we're building at the moment, even if data is sent to us in a proprietary format, the metadata, which describes the data, will be in XML format," he says. "But at the end of the day, we're not storing information, we're storing records. If we convert it to another format, that's not the record. We can't go around telling people which formats to use because that's changing the record. We have to take what people create."

So, if a government department produces Word documents or PDF files, asking for XML versions for the nation's archive is not keeping a proper record. Ryan does think that using open file formats could make it "more economically feasible" to make records publicly available, but that's a separate issue.

Even if all these technical issues were solved, there are still huge problems with trying to preserve today's digital data for the future. These include copyright laws and digital rights management.

First, there's the problem of getting hold of digital records to preserve. Neil Beagrie, programme director for digital preservation at JISC (Joint Information Systems Committee of the Higher and Further Education Funding Councils), points out that while print publishers have to deposit copies with copyright libraries, "there's no legal deposit for electronic publications. We need the same sort of provision for electronic documents. We can't rely on accidental preservation, or we're going to lose things that we'll later regret."

Chris Mole, Labour MP for Ipswich, hopes to propose this in a private member's bill, but whether he gets the chance a matter of luck. "I find that rather sad myself," sighs Beagrie.

Second, having got the records, will archivists be able to do anything with them? Wheatley raises the question of whether anybody would be allowed to look at the "rescued" Domesday data even if they own the original discs. "Common sense says it's just a copy, but the law might say it's a new version. Really we need a change in the law: we need a special provision to allow libraries and archives to take action to preserve digital materials."

That may not go down well with Hollywood, the music industry and other copyright owners who are starting to use digital rights management (DRM) systems to prevent copying. "DRM clearly locks you in," says Beagrie, "so you'll have it, but you won't be able to access it, unless you own a technology museum."

Clearly something must be done, which is why 19 organisations - including the British Library, JISC, and the Public Record Office - formed the Digital Preservation Coalition to coordinate their efforts, according to JISC's Maggie Jones. The Domesday effort showed that records could be saved, she says, but it took a massive effort. "Doing that every time is not going to be a feasible proposition. And however passionately people feel about [digital preservation], the key issue is: Who's going to fund it?"

Two years ago, the Americans allocated almost $100m to their National Digital Information Infrastructure and Preservation Program.

"We're watching that with some envy," says Beagrie, "and we'd probably like to see something like that in the UK."

Дата добавления: 2015-11-14