Email Archiving and the Dilemma it Poses for Records Managers

Records managers face a very difficult dilemma today and frankly, I do not think there is a solution yet, or a silver bullet. I believe that some company, or a brilliant records manager, will find a solution in the next few years and become the hero of the industry. Until then, we will continue to see articles written like this one that discuss the importance of the silver bullet solution to email archiving and records management.

In case you have not caught on to what I am talking about, here is the problem: After the Federal Rules of Civil Procedure was amended to include ESI, or electronically stored information, on December 1, 2006, the discovery of email messages became a reality, and a nightmare, for companies. And here is the rub: Companies were not prepared for this amendment and even 15 months later, many companies are scrambling to decide how to marry records retention schedules to email content.

Some of the solutions proposed to date include but are not limited to:

Save all email - courts love this solution because it can be easily shown that the company is violating its own records retention schedule;

Buy an Enterprise Content Management (ECM) solution and write complicated formulas for matching words and phrases with words and phrases found in the records retention schedule. Currently, this "Automated" solution only has a 78% success rate according to recent studies. Imagine telling your management that you can solve your email archiving problem with only a 22% error rate. I can hear the gasps now;

Set up an email folder structure system in the current email system (e.g., Outlook or GroupWise) and ask employees to drag each email to its appropriate folder based on the contents of the records retention schedule. Yikes. Studies have shown that employees may rebel and find alternate ways to write email if they have to drag every email, every day into individual folders based on a records retention schedule that covers all identified record classes at your company.

Use a Combination of items 2 and 3 above. At present, this is the solution most often advocated by ECM vendors.

Large companies send and receive millions of emails daily. Even small companies get an overwhelming number of emails daily. A typical email message can be a few sentences to several pages, and, optionally, have one or more attachments. In terms of records management, let us explain what constitutes an email message: Is it the body of the email? The subject line? The distribution? The attachments? The metadata? Well, the answer is, "YES," to all parts of an email message. Now the question becomes, "Now what?" I am with you. I frankly cannot see a good solution. While I believe it is possible to create complicated formulas to improve on the 22% error rate, there will be no good way to prove it one way or the other.

To set a records retention period for an email message, the Records Manager should take their authorized records retention schedule (and yikes, 59% of companies do not even have a retention schedule) and figure out a way to relate content to all parts of an email message. I can think of two solutions to this dilemma, both of which are ridiculous:

The company gives the Records Manager unlimited funds and personnel to review every email (you can understand why this will not happen) or
The company communicates a mandate that email messages must not be used as a storage device but rather as a communication delivery method. Naturally employees would rebel at this solution.

Conclusion:

The dilemma of setting records retention schedules for each email still exists today and there is no solution on the horizon that makes sense to employees and to the bottom line of companies. Do not get me wrong: There are companies that will attempt to use an automated solution and then there are others that will attempt to use the drag and drop methods but these methods are not likely succeed in the long run. Companies are likely to give up when employees rebel and the training costs go through the roof, to no avail.

So what should Records Managers do?

Records Managers should continue to gain as much knowledge as possible about email archiving and records retention systems. Records Managers should listen to webinars, attend workshops and conferences, and keep abreast of emerging technologies and companies in the area of ECM and Email Archiving. Records Managers should be ready when a solution presents itself.

Image Archiving - HDR and RAW Formats

HDR and RAW formats are effectively digital files that are used only for storage.  Neither format provides a useful image for purposes of display.  So which is better for archiving my high dynamic range files.  It pays to know how these storage or capture formats differ in order to make an informed decision.

In an earlier article I wrote that a single RAW file captures around 10EV in dynamic range while a 32-bit HDR file captures a virtually unlimited tonal range through the application of floating point luminance values.  This translates into around 76 orders of magnitude in dynamic range, far exceeding the luminance found on even the brightest sunny day.  HDR files capture a true high dynamic range.

Because the HDR file is lossless, one might think that it provides a better storage solution than a RAW file.  Perhaps. The problem lies, however, in the simple fact that the creation of an HDR file from a RAW file depends on the conversion software one uses.  Merging three to five RAW files into a single HDR merged file will give different results depending on the software one uses and even between different versions of the same software as later versions generally provide more sophisticated results.  Merging algorithms are proprietary each having different working thresholds for pixel conversion; each algorithm, therefore, produces a slightly different result.

In the final analysis, archiving RAW files as source images is superior to archiving HDR merged files, however, RAW files are subject to degradation over time so I recommend that one use a redundant backup system.  I use an off-site backup as well as backing up my RAW and HDR files on DVDs.  While RAW source file storage and backup is ideal, I also recommend that one store converted HDR merged files, especially if one works with a single vendor's software.  I work with Photomatix Pro exclusively so I find that storing my HDR merged files as well as the RAW source files provides me with an ideal solution.  I don't believe one can take too many precautions when storing important image files.

All that being said, remember that the converted HDR file is not useful as an image file for purposes of display.  Displaying a 32-bit HDR file on a low dynamic range computer monitor one finds that the displayed image has apparent areas where the highlights are completely blown and the shadows are completely clipped; details of the HDR image are not apparent in the display of the HDR merged file on a standard computer monitor.  Tone mapping of the HDR merged file is necessary, not only for viewing the details on your monitor but is also a requirement if one wishes to print the merged image.  Tone mapping, in its simplest terms, is a way to convert the 32-bit HDR merged file into a 16-bit TIFF file or an 8-bit JPEG file that is now useful as an image for viewing and printing.