General Letter 2001-1

Standards for the Use of Imaging Technology for Storage, Retrieval, and Disposition of Public Records

Policy Statement

Contents

• Part I. Introduction
  • Authority
  • Scope
  • Purpose
  • Legal Issues
  • Policy on Disposal of Original Public Records
• Part II. Definitions
• Part III. Planning the System
  • Needs Analysis
  • Retention Requirements
  • Cost Analysis
• Part IV. Selecting the System: Specifications
• Appendix 1 - Certificate of Compliance
• References
• Standards

---

Date: February 1, 2001
To: Administrative Heads Of State Agencies; State Agency Records Management Liaison Officers;  Administrative Heads Of Municipalities; Town Clerks; All Other State Agency and Municipal Records Custodians and Records Management Personnel
From: Eunice G. DiBella, Public Records Administrator

Standards for the Use of Imaging Technology for Storage, Retrieval, and Disposition of Public Records

(Replaces General Letter 94-1, Optical Imaging Technology and Public Records: Policy Statement)

Part I: Introduction

• Authority
  The Office of the Public Records Administrator and State Archives issues this Statement under authority granted it by Connecticut General Statutes (CGS) §11-8, §11-8a and §7-109.

• Scope
  The following provides a policy for using computer based digital imaging systems for storing and reproducing digital images of official public records of all state and local government entities in Connecticut onto optical or magnetic storage media. It addresses the retention and disposal of original public records reformatted on such media without hard copy or microfilm backup, recommends ways of integrating computer based digital imaging systems in a government records setting, and offers legal guidelines. See Appendix 1 for Certificate of Compliance, Digital Imaging Standards for Public Records.

• Purpose
  The purpose of this policy is to ensure the continued accessibility and retrievability of information stored in imaging systems.

• Legal Issues
  Public Act 97-89 "An Act Concerning the Recording, Copying and Maintenance of Certain Public Records" is an important piece of legislation because it allows electronic imaging for recording documents into the land records, provided that the Public Records Administrator approves the imaging system. This Public Act also amended several sections of the General Statutes. They are §1-7, §1-8, §1-14, §1-16, §1-17, §7-23, and §7-27a. Specifically, CGS §1-7 states that "When any officer, office, court, commission, board, institution, department, agent or employee of the state, or of any political subdivision thereof, is required or authorized by law or has the duty to record or copy any document, plat, paper, or instrument of writing, such recording or copying may be done by any photographic, micrographic, electronic imaging or other process, which clearly and accurately copies, photographs or otherwise reproduces the original document, plat, paper or instrument of writing. Each such photographic, micrographic, electronic imaging or other process shall be subject to the approval of the Public Records Administrator. Properly certified reproductions of any record made under the provisions of this section shall be admissible in evidence in the same manner and entitled to the same weight as copies made and certified from the original."

  Public Act 97-89 amended several sections of the general statutes to allow electronic imaging for the recording of documents into the land records, provided that the Public Records Administrator approves the imaging system. An electronic imaging system will be approved if it (a) meets these guidelines, and (b) provides for a human readable copy that is either microfilm or paper meeting the requirements for paper or microfilm established by the Public Records Administrator.

  Public Act 00-92 "An Act Concerning The Duties of Town Clerks and The Establishment of Ethics Agencies by Special Districts" in part deals with the issue of marginal notations. CGS §7-29 was changed to allow for making a marginal notation on a digitized image. "If the land records are not maintained in a paper form, the town clerk shall make the notation on the digitized image of the first page of such mortgage or lien in a form or manner approved by the Public Records Administrator."

  CGS §52-180 "Admissibility of Business Entries and Photographic Copies," allows for properly microfilmed or photographically reproduced records to be admitted as evidence into court as business records, just like traditional hard copies, provided a proper evidentiary foundation has been laid. The statute does not mention computer based digital imaging technology.

  Printed records reproduced by a computer based imaging system are not photographic copies. Some states have changed their evidence statutes to include imaging systems. Connecticut has not done so.

  It is clear that CGS §1-7 includes copies of land recordings that are made using computer based digital imaging technology. It is not clear if it extends to certified copies of other types of records.

  Any agency contemplating using a computer based digital imaging system should be aware of these legal issues. It is crucial that the digital imaging guidelines outlined here be adopted to ensure the authenticity of the documents created. Consultation with appropriate legal counsel regarding rules of evidence is advisable.

• Policy on Disposal of Original Public Records
  This Office will approve disposal authorizations for original public records reformatted on a computer based digital imaging system only if the original records have an approved retention period of ten (10) years or less.

  The Public Records Administrator may approve the disposal of original public records having a retention period of more than 10 years or as having permanent/archival value and reformatted on a digital imaging system if the agency also retains a security copy of the record in a human-readable storage medium approved by the Public Records Administrator, and the security copy is maintained in an organized record-keeping system.

Part II: Definitions

• "Agency" means a state agency or municipality or political subdivison as referred to in CGS §11-8 falling under the authority of the records management program administered by the Connecticut State Library.

• "Archival" means and pertains to records which have been appraised by the State Archivist to have permanent value to the State of Connecticut and which may be scheduled for transfer to the State Library or to an archival repository approved by the Public Records Administrator.

• "Backfile Conversion" means a process of scanning, indexing and inspecting a large existing collection of documents.

• "Human-readable storage medium" means paper, a photograph, a photocopy, or a microform, including, but not limited to, microfilm, microfiche, computer output microfilm, and aperture cards. 

• "Imaging system" means a computer based digital system used to store documents and records electronically by recording a digital reproduction of a scanned document or record and a suitable method of indexing and retrieving the stored images.

• "Metadata" is documentation of information and data about the imaging system hardware, software and storage file systems.

• "Off-site location" means a premises, building, or structure that is separate and apart from the premises, building, or structure that houses an agency's primary imaging system.

• "Public records or files" in accordance with CGS §1-200 means any recorded data or information relating to the conduct of the public's business prepared, owned, used, received or retained by a public agency, whether such data or information be handwritten, typed, tape-recorded, printed, photostated, photographed or recorded by any other method.

• "Records retention and disposition schedule" means a comprehensive list of record series which indicates for each series the length of time it is to be maintained until it is reviewed for destruction or archival retention. It also indicates retention in active and inactive storage.

• "State Archives" means the department within the State Library having custody of the records appraised for permanent retention by the State Archivist.

• Definitions of other technical terms used in this document are defined in the AIIM technical report AIIM TR2-1998, "Glossary of Document Technologies"

Part III: Planning The System

• Needs Analysis

  • Understand that purchasing any computer based digital imaging system constitutes a large ongoing financial commitment to hardware, software, training, media, and data conversion requirements. Remember that hardware and software change rapidly, that media used to record data on one system may be unreadable on another (or even on a later version of the recording program), that technology becomes obsolete, and that vendors go out of business.
  • Analyze what needs to be done and evaluate the benefits of using an imaging system to do it. In evaluating the need for a computer based digital imaging system, consider the ramifications of the disposal policy above in Part I. Item E (Policy on Disposal of Original Public Records) and the section in Part I. Item D (Legal Issues).
  • Identify the type and volume of public records that would be stored on optical or magnetic media. Reformatting information on computer based digital imaging technology is most cost effective for a large quantity of records that are frequently retrieved. Records that become inactive after a very short period of time may not be cost-effective candidates.

• Retention Requirements

  • Records stored or reformatted onto a computer based digital imaging system must conform with the retention requirements listed on a records retention schedule issued by the Public Records Administrator in accordance with CGS §11-8, §11-8a, and §7-109.
  • All disposal requests will be reviewed by the Public Records Administrator and the State Archivist. Records may not be destroyed or transferred until the Office of the Public Records Administrator has returned a signed disposal authorization form to the requester.

• Cost Analysis

  • The cost of implementing an imaging system depends on several key components of an information system: hardware, software, and support which includes reengineering, training, maintenance, and management associated with this technology. It also includes services for implementation, communications and integration.
  • Another major cost component is backfile conversion of existing documents. If access to archived documents is necessary, backfile conversion is essential. Backfile conversion costs can be very expensive and could easily exceed the cost of the imaging system.
  • Labor requirements are another significant cost factor. Although automated indexing has reduced some of the expense, personnel will still be needed for document preparation and scanning. An imaging system may also require that certain employees be reclassified because of new job responsibilities.
  • Imaging may also require more advanced and costly computer networks. Most local area networks can handle an imaging application up to a certain point. But as the number of users and requests increases, the network may experience slowdowns in response and data transfer times. Upgrading a network to a higher bandwidth to accommodate an increase in users or imaging traffic, can be a significant cost factor.
  • Remember that you may need to use parallel media (paper and image) until the new system is fully functional.
  • Budget on average 15-20% per year of the original system acquisition cost for upgrades, training, maintenance and supplies. Some costs may be shared among departments. Unless these costs are factored into the continuing support of system maintenance and improvement, the system is in danger of becoming obsolete and requiring a far greater cost outlay to restore its effectiveness. Also, records stored in an outdated system tend to be at greater risk than those in a well-supported system. Continued planning and budgeting for the migration of long-term and archival records is essential for the success of any digital imaging project. Refer to Part IV, F.
  • Outsourcing may be cheaper than an in-house imaging system. If the application is outsourced, however, it is vital that the agency not give up control and oversight of the project.

Part IV: Selecting The System: Specifications

• Open Architecture: Require an open systems architecture for digital imaging applications or require vendors to provide a bridge to systems with non-proprietary configuration.
  Although the term open systems architecture is defined in various ways, public officials should follow a system design approach that permits future component upgrades with minimal degradation of system functions. This open system architecture allows the system to be upgraded over time without a significant risk of records loss. It also supports the importing and exporting of digital images to and from other sources. The flexibility of an open system architecture enables long-term records to be accessed and transferred, as well as upgraded or modified, from one hardware or software platform to another. One key factor in achieving an open system architecture is the adoption of non-proprietary standards.

• File Format: Use a non-proprietary digital image file format. If using a proprietary format, provide a bridge to a non-proprietary digital image file format.
  A digital image file format is a structured container for information about each digital image and the image data. Information about the digital image file includes, but is not limited to, its name, width, length, resolution, and compression techniques. The computer requires this information to interpret the digital image. It is essential to use a non-proprietary image file format to ensure the ability to transfer successfully digital images between different systems or when a system is upgraded or modified.

  American National Standard Institute (ANSI)/Association for Information and Image Management (AIIM) MS53-1993, Standard Recommended Practice - File Format for Storage and Exchange of Images - Bi-Level Image File Format: Part I details a standard definition for file formats. Despite the existence of a standard, there is not an agreed-upon, industry-wide image format standard. Many digital imaging systems use the Tagged Image File Format, or TIFF. Because different versions of TIFF exist (TIFF-5, TIFF-4, etc.) there is still no absolute guarantee that images can be transported seamlessly from one system to another. Comprehensive documentation of the digital image file format, including TIFF, is recommended.

  A number of other file formats exist, such as Graphics Interchange Format (GIF), Joint Photographic Experts Group (JPEG), and Bitmap (BMP). These file formats are commonly used in conjunction with hypertext markup language (HTML) for internet and intranet applications. Many systems or third-party graphics packages will convert images from one to another, although often with unpredictable results.

• Compression: Use International Telecommunications Union (ITU) Group 3 and Group 4 compression techniques or have the vendor provide a bridge to these techniques.
  The large file sizes of typical scanned documents require digital image compression to support data transmission and to promote storage efficiency. Today most digital imaging systems use standard compression algorithms to "shrink" images. Standard compression techniques are instrumental in ensuring a migration strategy for records needed for long-term use. Two international standards are currently available. Using compression techniques conforming to either of these specifications will increase the likelihood that the images can be used with other technologies or migrated between systems.

• Encryption: An agency shall not use any software or hardware that utilizes an encrypted software source code.

• Storage Media: Where data longevity or records integrity is a primary concern use a recording media that is not rewritable.
  The storage capacity of optical disks versus paper is a primary advantage to the use of digital imaging systems. However, optical disks are not the only option. Other storage solutions that can be used with digital imaging systems include output to microfiche or microfilm, digital tape, and magnetic disks. The selection of a storage media may depend on budget considerations for the agency.

  The issues of data longevity and integrity must be considered when selecting optical storage media. There are a variety of optical disks on the market today such as Write Once Read Many (WORM), Rewritable, and Compact Disk-Read Only Memory (CD-ROM)/Compact Disk-Recordable (CD-R). Each has its own advantages and disadvantages. WORM, CD-R, and COLD (Computer Output to Laser Disk) are not rewritable. These media offer a high level of data security because data cannot be altered without destroying the medium itself. If a record is no longer needed, software may allow the pointer to the data to be disabled, preventing normal access. Because the data cannot truly be deleted, however, it may remain accessible by other means.

  An agency using optical storage media shall only use nonerasable write-once-read-many (WORM) disks. Disk drives and multifunction disk drives capable of recording on WORM disks or rewriteable disks may be used. However, only optical storage disks manufactured exclusively as WORM shall be used.

• Migration: Provide specific plans for an ongoing process of migrating long-term and archival records from older to newer hardware and software platforms.
  Municipalities and agencies must ensure that their long-term and archival records are continually accessible. While the physical system could be operational for ten years or more, its technology will often be superseded within two to three years. If the system stores records with retention periods exceeding the life span of the hardware and software, it is essential that the administrator plan for future data migration. A migration strategy documents how an organization will safely and completely transfer long-term and archival records from one generation of hardware and software to another generation. The strategy should be written and maintained with the system documentation. Current strategies for migrating digital imaging system records include: upgrading equipment and software as technology evolves and periodically recopying optical or magnetic storage media as required; recopying optical or magnetic storage media based upon projected longevity and/or periodic verification of the records; or, transferring the data from an obsolete generation of optical or magnetic storage media to a newly-emerging technology, in some cases bypassing the intermediate generation that is mature but at risk of becoming obsolete.

• Indexing and Image Retrieval
  • An agency using an imaging system shall index each image stored by the system with a specific or unique identifier. The index shall contain information sufficient to enable the agency to retrieve images and records in an efficient and effective manner based on the anticipated needs of its users. The index shall have the same retention period as the information and shall be migrated at the same time as the information.
  • An agency shall implement procedures and a regular timetable for verifying the accuracy of index entries. Verification may be accomplished visually, by duplicative typing of index data, or by any other method that accomplishes positive index data verification. An agency shall test its indexing verification method to ensure that the index permits the retrieval of images in an efficient and effective manner.
  • All index information relating to stored images shall be retained on magnetic or optical media that are designed to function with the imaging system. An agency may retain index data on the same medium where recorded images are stored, if the agency also retains a separate copy of the index.
  • If an agency replaces any storage media, then the agency shall maintain the integrity of the index as well as the images onto the new medium. The new medium shall be capable of reading and fully converting all information stored on the replaced disk.

• Error Detection and Correction
  • Error detection and correction is the ability to predict the point at which an optical storage disk is no longer readable. This ability is critical if the recopying of disks is to take place at the appropriate time. Refer to ANSI/AIIM MS 59-1996, Media Error Monitoring and Reporting Techniques for Verification of Stored Data on Optical Digital Data Disks.
  • An agency shall specify that the small computer standard interface (SCSI) command "Write and Verify" is used when writing data to optical or magnetic storage disks. The command is valuable for assessing how accurately the scanned information is transferred from the central processing unit of the computer to the digital imaging system. The command requires verification from the system that the digital image is correctly written to the disk and provides additional protection for continued access to long-term records.
  • An agency shall acquire a utility program that monitors the amount of disk space used to relocate data sectors when errors are detected.

• Labeling
  • Offline storage media shall be kept in protective cartridges. Such media shall have a human readable label and should be labeled as completely as possible to ensure future use and accessibility.
  • The following types of information may be included on the label: (1) name of agency; (2) the unit or division that is responsible for the records on the disk; (3) sequential number or other specific identifier that identifies the disk in the series of disks used by the system; (4) date of manufacture; (5) identification of the disk as the master or backup storage copy; (6) records series title and item number from the retention and disposition schedule; (7) creation dates of the records; (8) beginning and ending dates during which images were recorded onto the disk; (9) software needed to access the digital images or index stored on the disk.

• Environmental Conditions for Optical Storage Disks
  • In accordance with ANSI/AIIM TR25-1995, Use of Optical Disks for Public Records, users should obtain and follow the manufacturer's recommendations for the storage and use of optical disks. An agency should request that the vendor supply specifications for the storage of digital optical disks and ensure that office conditions meet these specifications.
  • An agency shall provide a stable storage and use environment for optical disks. Technical specialists recommend a stable environment with a temperature between 65 and 75 degrees Fahrenheit and a relative humidity between 30 and 50 per cent. Adverse storage conditions, especially high humidity, can cause rapid deterioration of the media.
  • All disk media shall be kept free of condensation.
  • An optical disk is affected by dust, debris, and fingerprints. An agency shall not remove the plastic cartridge protecting a disk. A cartridge shutter shall not be opened to expose the optical disk's recording surface. To protect disks from warping, they should not be subject to pressure and should be stored in an upright position when not in the disk drive.
  • An agency shall not store disk media under other objects, in a place that could readily permit water damage to occur, in a manner or place that is likely to subject the disks to physical shocks, shattering due to rapid temperature change, or damage due to exposure to direct sunlight, other light sources, or sources of heat, oil, or solvents.

• Backup Storage Media
  • An agency shall periodically prepare backup storage copies of recorded information and shall store the backup media at a secure, environmentally controlled, off-site location.
  • An agency shall store its backup image media and backup index media together.
  • An agency shall update its backup storage media on a regular basis.
  • An agency shall copy or migrate, or copy and migrate, its backup storage media and original image media at the same time.

• Scanning and Testing for Quality
  • An agency shall establish scanning quality control procedures and shall periodically document the use of procedures.
  • To help ensure the integrity of long-term and archival records stored on the system, staff members should perform a visual quality evaluation of each index entry and scanned image before writing the digital image to optical or magnetic media. Visual inspection must be a part of operational procedures in order to verify the completeness and accuracy of scanning after documents have been transferred to a disk.
  • Overall system quality control is best when the scanned image is temporarily stored on magnetic media, as this allows corrections to be made by re-scanning. Depending on the system configuration, corrections may be performed at the scanner capture station or at designated inspection/re-scan workstations.
  • Training and supervision of the operations staff is a key factor in maintaining acceptable image and index quality as well as user satisfaction with the system.
  • An agency shall test and certify scanner performance as a routine part of a regularly established schedule of maintenance for operating an imaging system.

• Image Resolution
  • When determining document scanning resolution, consider data storage requirements, document scanning throughput rates, and the accurate reproduction of the image. The resolution of scanned images shall be sufficient to permit the recording of an accurate image of the public record. An agency shall use the AIIM technical report AIIM TR26-1993, "Resolution as it Relates to Photographic and Electronic Imaging," as a guide for established resolution criteria.
  • A digitized image consists of black and white dots or picture elements (pixels) measured in dots per inch (dpi). The higher the number of dpi, the higher the legibility of the reproduced image. Images scanned at higher dpi rates, however, use more storage space on the disk and may require longer scanning times.
  • The selection of scanning density involves a trade-off between image clarity, storage capacity and speed. When selecting a scanner, ask the vendor to perform a quality test on a broad sampling of agency documents at various dpi settings so that an appropriate end-to-end throughput rate and resolution can be determined.
  • Standard letter quality records shall be scanned at not less than 200 pixels per inch. Photographs and other higher quality or more detailed records shall be scanned at not less than 400 pixels per inch. Engineering drawings shall be scanned at not less than 200 pixels per inch. A higher scanning density (600 dpi or higher) is appropriate for deteriorating documents, and documents with visual elements such as, engineering drawings, maps, or documents with background detail. A trial and error method should be used to find the minimal acceptable image.

• System Security
  An agency shall ensure that the system's software is designed to track use of and access to the system and will attempt to prevent or indicate the occurrence of unauthorized access to sensitive records.

• Document Authentication
  • The software will ensure that records were created, edited, and deleted according to documented procedures and will maintain this information in chronological order.
  • Legal retention periods shall be included in retention and disposition schedules in accordance with CGS §11-8.

• Metadata
  The metadata file is necessary to assure that future changes in personnel will have the information necessary for migration to newer systems. It will also facilitate any state audits when necessary for compliance to state laws.
  • An agency shall prepare a procedures manual and flowchart indicating the normal routine of the agency and the role that the imaging system assumes in normal activities. The documentation shall describe the procedures to be followed for all of the following: document preparation, document scanning, scanner testing, method of indexing, verification of index data, image editing, image enhancing, image and index updating, and image and index deleting.
  • An agency shall maintain a description of file layouts and content, including, but not limited to the following information: (1) identification of all imaging system hardware and software by manufacturer, model, release and installation dates. Provisions shall also be made to include updates, upgrades and their installation dates; (2) the location of files on the computer system; (3) how the files are laid out; (4) the meaning of the information in the file directory; (5) the formats used to encode the contents of files (as described in IV. B, C, D and E); (6) a plan to enable the storage media recorded by the system to be read by another brand or type of system if the system's hardware or software fails or otherwise becomes permanently inoperable (as described in IV.F); (7) a description of security procedures (as described in IV.N); (8) a plan to provide for major upgrades of hardware and software to avoid obsolescence or the loss of recorded information (as described in IV.F); (9) a contingency plan to allow the agency to continue functioning during temporary system's downtime, which shall also be included in the procedures manual (as described in IV.K); and (10) a description of the hardware and software used for the detection and correction of error codes, including the placement of error correcting codes and a description and orientation of preformatted blocks on a disk (as described in IV.H).

• System Administrator
  • An agency shall identify a system administrator to monitor the installation, operation, and security of the imaging system as well as the training of assigned personnel. It is recommended that there be a backup system administrator to be responsible for the system in the administrator's absence.
  • The system administrator shall maintain documentation pertaining to the imaging system (metadata file), including any offline storage media, and its procedures. In addition to any other record required by this policy, the system administrator shall maintain all of the following records:
    • A description of the imaging system and the storage media hardware, including equipment specifications, names and addresses of manufacturers, equipment model numbers, equipment maintenance requirements, and technical operating manuals.
    • Records reflecting the results of the performance appraisal and reliability testing performed at the time of system installation.
    • Before an imaging system is purchased, the system administrator shall arrange to have the agency's retention and disposition schedule amended to include the imaging of records, the retention of the human-readable copies, and the storage of backup copies and shall ensure that the new schedule is approved by the Public Records Administrator.
    • A written procedure governing the regular removal of images and indexes in accordance with established retention schedules and in accordance with security provisions established in this policy. The system administrator must have a plan for recreating the indexes and a record of this shall be maintained.
  • The system administrator shall be responsible for documenting and updating the metadata file, and for planning and administering data migration as required according to IV.P.2.
  • The system administrator shall be responsible for administering system security as described in IV.N.

• Safeguard Plan; Vendor Codes Escrow; Notice from Vendor
  • Before acquiring an imaging system, an agency shall prepare a written plan that includes safeguards to ensure the agency's continued ability to access and retrieve image data if the agency's vendor no longer supports the system's hardware or software or if the system becomes inoperable due to damage, malfunction, or obsolescence.
  • By contract, an agency shall require its imaging system vendor to:
    • Deposit a copy of the imaging system's application software codes and associated documentation in an escrow account with a bank, archive, or other institution acceptable to the agency for transmission to the agency if the vendor's business fails. The vendor must verify that the code is updated every year. If the vendor goes out of business, the agency must be given a copy of the source code. The contract must state that "upon termination of this contract, all data/tapes must be returned to this agency." The system administrator must have this documentation.
    • Notify the system administrator about the discontinuation of the imaging system or its product line, changes or upgrades to the imaging system, or the cessation of service support for the imaging system.
    • In order to maintain the integrity of the agency's records, the agency shall retain ownership and access to all information maintained by the vendor either at the location of the agency or at the vendor's location.

• Annual System Review
  • An agency must periodically review its metadata file to verify all of the following:
    • That the statistical error rate and any loss of recorded information are being tracked.
    • That satisfactory scanner performance and reliability are being maintained by comparing currently scanned and recreated hard copy images of test targets with original benchmark target images.
    • That the scanner testing logbook is being properly maintained and that all necessary procedures are being followed and documented.
    • That all required targets are being scanned on a routine basis.
    • That backup storage copies of images and indexes are routinely produced on separate media and stored at an off-site location.
    • That human-readable copies of records are created, managed, and stored as required by this policy.
    • That images stored on master use and backup storage disks have not deteriorated.
    • That security measures have been developed and are being utilized.
    • That comprehensive documentation about appropriate aspects of the imaging system has been created and retained.