Off Topic — HabitableZone

Off Topic » in reply to Information storage

re: Information storage

Interesting stats, all right.

I take issue with their use of a compressed scanned image as the benchmark--a scanned image is a representation of a representation, and assuming a 5:1 compression ratio is purely arbitrary. What counts is the plain text, which is the original information. It too can be compressed, BTW, but for some reason they didn't count that. The visual components of documents are another matter, and perhaps a good rule of thumb is to add another amount of bytes equal to the text.

To relate it to the question of archiving books for future generations, I think that talking about scanning books and then transcribing the scans down through the years is a little misplaced.

Aside from the unnecessary bulk of the data, consider the problem of language evolution. Representing text as text will make it much easier to perform machine translation to a then-modern language, such as Galactic Standard. For purists who'd like to see the page as it was originally published, the page layout and the font glyphs can be included in the archive file, i.e. much like today's Acrobat PDF format.

There's a sharp dividing line between pre-computer and computer-produced books and documents. Paper books can be scanned and OCR'd, and the visuals stored as images, and that's about the best we can do.

Documents produced on computer can be archived in a much richer form that preserves a lot more info for future generations. Archived documents should be stored by their creators using the original sources, because those usually contain far more data than ends up in the printed document. I archived the source files for my two published books, and a couple of years ago I was able to produce an "author's cut" for one of them as a PDF, complete with four chapters that the publisher cut. (That's a 4.3M PDF--beware.)

CAD and other vector-based drawing formats are an excellent example: They're pure information, and can be rendered later on better and better output devices. A 3D model that has to be rendered to 2D for printing today could, someday, be rendered as a 3D hologram.

A similar situation applies to source files such as imagery. Digital cameras can produce high-resolution images, but that resolution is severely dowgraded to produce the typical 300 dpi printed page. A future higher-res output device...well, you get the idea.

It's kind of like being able to produce a "director's cut" of a movie that used CGI. What would Disney's Tron look like if it were rendered with today's technology?

A few random thoughts based on things I read on the thread...

Last I heard, consumer-grade CDs and DVDs were estimated to last about 10 years (not 50), though that's just an estimate and your longevity will vary.

Whether people have the will to transcribe is a real problem, but I see it diminishing over time as storage capacity increases. The more information that can be stored on a single instance of a storage medium, the less labor is involved in making copies. CDROMs hold about 700 megabytes, DVDs 4.7 gigs (or 8+ gigs for dual-layer), so it takes roughly seven/fourteen times the amount of labor to transcribe CDs as DVDs. The day's not too far off when you can have a chip that holds a terabyte, and based on the linked article, it could hold all the books published in one year. Pop it into your computer and say "copy", and all the world's books are backed up.

There are concerns about forgetting data formats, and that's a reasonable concern. It's one argument for storing text as text, because 8-bit ASCII is so simple and universal it's not likely to be forgotten. The newer Unicode 16-bit format, that attempts to encode all the glyphs from all possible alphabets, is far more complex, but it's also well documented (in 8-bit ASCII--it bootstraps) and is unlikely to be forgotten.

The same isn't true of the complex graphic formats, like TIFF, used to hold scanned images. However, OTOH, archivists plan for that, and include format-decoding instructions with archived imagery. See NASA's Planetary Data System for a pretty good example. The format metadata is stored as ASCII, so leverages its universality.

Log in or register to post.

The whole thread (18 posts)