***** File HALGUIDE.TXT Comet Halley Archive User's Guide to IHW CD-ROM Volumes 19-23 Contents 1. Background and Scope of Compact Disc 2. Accessing the Disc 3. Data Products on these Discs 4. Supplemental Directories 5. Filenaming Conventions 6. References 1. BACKGROUND AND SCOPE OF COMPACT DISC This compact disc (CD-ROM), and the series of discs of which it is a part, contains observations of comet P/Halley obtained during the 1985-1986 apparition. The resulting digital archive is the main product of the International Halley Watch (IHW), which was formed in 1981 to mount an international campaign to observe the comet over a long baseline in time. The collection of remote (mostly ground-based) data archived here spans a wide wavelength range and is augmented (on other discs) by measurements made in situ by the spacecraft which encountered the comet in 1986 March. The remote observations are associated with nine IHW Disciplines: Astrometry, Infrared Studies, Large-Scale Phenomena (L-SP), Meteor Studies, Near-Nucleus Studies, Photometry and Polarimetry, Radio Studies, Spectroscopy and Spectrophotometry, and Amateur Observations. The complete IHW digital archive is comprised of four subsets of CD-ROMs containing: o Compressed Halley images from the L-SP Discipline (Volumes 1-18), o Halley data from all IHW Disciplines (Vols. 19-23), o IHW "Trial Run" data of comets P/Crommelin & P/Giacobini-Zinner (Vol. 24), o Halley spacecraft data (Vols. 25-26) The total collection for Halley's Comet contains more than 37,000 independent files and 15 Gbytes of data. This document is intended as a guide to Volumes 19-23. [CDTREE.TXT and DATATREE.TXT are useful files within the DOCUMENT directory to which the archive user may wish to refer to visualize the structure and content of these discs as he or she reads this document.] 2. ACCESSING THE DISC These discs have been manufactured following the normal specifications for Compact Disc Read Only Memory (CD-ROM) as well as the logical structure outlined by the International Standards Organization (ISO) in the No. 9660 document. A basic system requirement demands that the CD-ROM reader and software driver support this standard. At this writing (July 1990), primary access to the disc will be using IBM-PC or compatible computers. However, software support from vendors for the Apple MacIntosh II and some workstations (SUN, MicroVAX) is now available. If you are using a PC style machine, the reader that you bought should include the appropriate hardware (a PC-bus or SCSI controller board and cable) and software to address your device. Software packages should include a device driver for your machine and extensions to DOS that allow the reader to "look" like another random access disk drive; the de facto standard here is the Microsoft Extensions (MSCDEX 2.0 or higher). Install the specific device driver software as directed by the documentation accompanying the hardware. Usually, this will involve specifying the device driver in a DEVICE= line in the CONFIG.SYS file and invoking the extensions package from your AUTOEXEC.BAT file. This will automatically configure the CD-ROM reader when the computer is booted. Remember that the CD-ROM reader is assigned a drive name which you have either chosen with the appropriate command line switch (/L:H assigns the CD- ROM reader to drive H:) or allowed to default to the next available device letter in your system. Typical PC configurations will have two floppies (A: and B:), and a hard disk (C:). Put your drive at some letter higher than that. Remember that many DOS commands work on the CD-ROM but that it is a read-only medium. Some of the useful DOS functions are: DIR - directory listing CHDIR - change directory (also CD) TYPE - list the contents of a file on the screen; useful for normal text with , delimiters COPY - copy file(s) to another device PRINT - print file on hard copy device Notice that commands such as DEL and MKDIR are not available since the CD-ROM is a read-only medium. Furthermore, some compact disc software packages will invoke screen plots that may depend on the DOS program GRAPHICS.COM. In this case, execute it in your AUTOEXEC.BAT file or before you run the software package. The IHW CD-ROMs have been designed to allow maximum access to the data using existing software developed within NASA, as well as user-supplied programs. In particular, each data file was originally supplied in the FITS format, and also has a detached PDS label to describe the data structure. Furthermore, the normal FITS files have had the headers placed in files (with extension .HDR) separate from the data (.DAT, .IBG, .IMG, .TAB). The headers have been separated to allow unrestricted access to the data by non-FITS programs. The sizes of all header and data files have been preserved as integer multiples of 2880 bytes (required by FITS) in order to facilitate the reconstruction of the original FITS bytestream by concatenating the data file with the appropriate header file. 3. DATA PRODUCTS ON THESE DISCS The majority of the ground-based data is located in directories whose organization is chronological. The directory names themselves are based on one or more parameters of time, the complete set being Year, Month, Day, and UT Hour; the time referred to reflects the approximate beginning of the directory's data contents (filenaming conventions are discussed in Section 5). We have restricted directories to a reasonable number of files while allowing enough information for useful browsing; 256 was adopted as the desired maximum number, which includes datafiles, headers, and PDS labels. Given the large variation of the temporal density of IHW observations throughout the apparition, the "reasonable N" < 256 criterion resulted in directories widely divergent in duration. For example, all observations for 1983 were deposited in one directory, whereas for 1986 April there were many days which required directories only 3 hours wide (sample directory name: Y1986\M04\D10\H18). The smallest hourly subdivision is, in fact, 3 hours (03,06,09,....hours UT). No subdirectory was created for days on which data were not submitted. Across the entire set of ground-based data discs (Volumes 19-23), the typical file count in a directory is 50, and the average byte count is 1.0 Mbyte. Not all the IHW remote data on these discs (Volumes 19-23) reside on one directory level, although the majority of them--those from six of the Disciplines--do. Those six Disciplines are: o Infrared Studies o Large-Scale Phenomena (browse images and dataless headers) o Near-Nucleus Studies o Photometry and Polarimetry o Radio Studies o Spectroscopy and Spectrophotometry The approach has been to include Halley datafiles (and headers and labels) from these Disciplines in the appropriate chronological directory (e.g., Y1986\M04\D10\H18). The full-resolution images from the Large-Scale Phenomena (L-SP) Discipline have been deposited on a different series of IHW discs--Volumes 1-18--in compressed form. Details of the compression algorithm and the accompanying FITS proposal are in the file FITSCOMP.TXT in the DOCUMENT directory of those discs. Descriptions of the discs' full contents and structure, as well as the rationale for performing data compression, are to be found in the documentation on Volumes 1-18 and in the LS_PHEN.APX file in the \DOCUMENT\APPENDIX directory of this disc. A useful file which shows the time range of each L-SP compressed image disc is SUMINFO.TXT, located in the DOCUMENT directory of Volume 18. Inclusion of subsampled, "browse" versions of the same L-SP images on these discs (Vols. 19-23) was thought to be important for inter-disciplinary comparisons between all types of IHW data. These images are restricted to a maximum of 256 one-byte pixels on a side, and preserve the original sampling geometry. They may be especially important for "quick look" assessments of the full-resolution images, which are relatively time-consuming to decompress. The data for three IHW Disciplines--Astrometry, Amateur Observations, and Meteor Studies--are stored in different locations on these discs (Volumes 19-23), as follows. Astrometry and Amateur Observations are located one directory level below the "main data level" discussed above (sample path for Astrometry is: Y1985\M12\D12\H18\ASTROM), whereas the data for the Meteor Studies Discipline are contained in 59 files in a dedicated directory on Volume 23. It is to be noted that on occasion, a high density of either Astrometry or Amateur files drove the temporal widths of the "main data level" since the N(max)=256 criterion applied to the lowel levels (Astrometry and Amateur) as well. Some Disciplines submitted supplemental data which include filter tables, non-comet images, flat fields, and laboratory spectra. These files are stored in the CALIB directory of Volume 23, and the numeric part of their filename begins at a higher number (4001) than does the actual Halley data (0001). The reader should consult Section 5 for more details on filenaming. A large number of scientists contributed to this archive project. They are listed in the \DOCUMENT\OBSERVER directory of the these discs, by the IHW Discipline to which they contributed data. The contributing observatories, with their IHW system codes, are listed in the file OBSCODES.TXT (directory \DOCUMENT\OBSVTRY of Volume 23). Individual observers and their various instruments are, of course, also identified in the FITS header that accompanies the data. 4. SUPPLEMENTAL DIRECTORIES The volume and directory structure of this disc conform to the Level-1 standard specified by the ISO. This format is widely accepted and used on a variety of machines. The AAREADME.TXT file in the root directory introduces the user to the full extent of the archive and the contents of the individual subdirectories. There are four directories (DOCUMENT, EPHEM, INDEX and SOFTWARE) on this disc that contain supplementary files. The DOCUMENT directory contains text files that give the background to this CD-ROM project, present a general guide to its use, and detail experience with previous CD-ROM products, including a test disc of comet Giacobini-Zinner data (also archived by the IHW) and two test discs of Halley data (refer to VOLINFO.TXT). A discussion of the FITS and PDS formats and the metadata used specifically for the Halley data is located in the files FITS_IHW.TXT and PDS_IHW.TXT. This file (HALGUIDE.TXT) and IMAGUIDE.TXT on Volumes 1-18 are meant to serve as general overviews of the discs and their contents. Documents in the APPENDIX subdirectory, written by the IHW Discipline Specialists, contain information on data collection, subsequent processing steps, and archiving techniques, at the Discipline level. In the INDEX directory, tables of useful information have been indexed in various forms in order to allow automated searching of the data. The QUIK.IDX index contains a selected set of mandatory FITS keywords from all Disciplines. On each of Volumes 19-23, QUIK.IDX includes only the observations on that disc. Volume 23 has an additional, "summary quick index", QUIK_SUM.IDX, which includes all observations contained in Volumes 19-23; the last field in QUIK_SUM.IDX includes the Volume number. A set of tables in the subdirectory NETABLES contains the metadata/data from the proposed printed archive, organized by network and subnetwork and chronologically ordered in each index. In this subdirectory, also, are more complete indices of FITS keywords for five of the IHW Disciplines. The filenames (Disciplines) are: NETAMATV.IDX (Amateur Observations), NETLARGE.IDX (Large-Scale Phenomena), NETMETR.IDX and NETMETV.IDX (Meteor Studies), NETRADIO.IDX (Radio Science), and NETSPECT.IDX (Spectroscopy and Spectrophotometry). We constructed a separate index called PATHTABL.IDX to specify the full path to each datafile; these are organized by disc, and a summary version is contained on Volume 23. We attempted to make all index tables transportable to relational DBMS by delimiting the tables and providing structure (.STR) and dBASE-compatible (.DBF) files. Further information about IHW indices is contained in the file INDXINFO.TXT. The SOFTWARE directory contains source code and executables for display of imaging and spectral data, interpolation of ephemeris tables, reading of FITS tables, and manipulation of metadata. To be specific, IMDISP.EXE contains various utilities for manipulating visual data on image display devices; IMDISP was originally developed by the Planetary Data System (PDS) at the Jet Propulsion Laboratory (JPL), and has been augmented and improved by them and by outside users. The interpolation software is meant to be used on the EPHEM.TAB file in the EPHEM directory; the algorithm uses values of ephemeris data for 7 consecutive integral days to perform the interpolation. The Fortran source code is called OBSNTERP.FOR, which we have compiled and linked on VAX and PC computers; the resulting executables for VAX/VMS and MS-DOS operating systems are VAXNTERP.EXE and PCNTERP.EXE, respectively. Also provided on these discs is a "FITS Table Browser" called FTB.EXE, which was developed by the Astronomical Data Center (ADC) of the National Space Science Data Center (NSSDC). Several other support programs for manipulating the metadata--FITSUTIL, FITSXTND, FITS2TXT, and TXT2FITS--are also provided. The archive user should take note of the fact that on the L-SP compressed image discs (Volumes 1-18), additional source code and executables exist for compression and decompression of the large image files contained on those discs. 5. FILENAMING CONVENTIONS The convention for naming files on the IHW CD-ROMs was proposed by the Lead Center and NASA/Goddard Space Flight Center (GSFC) personnel to include a unique data qualifier for the data. Specifically, a set of subnet codes was established to enable identification of the IHW Discipline/subdiscipline from the filename itself. A CD-ROM running number and file extension complete the filename (example: LSPN0059.IBG). A short list organized by Discipline and listing subnetwork names and letter codes is given below: Discipline Number Subnet (Exp) Subnet Code ______________________________________________________________________ Astrometry 1 none ASTR Infrared Studies 2 Photometry IRPH Polarimetry IRPOL Spectroscopy IRSP Image IRIM Filter Table IRFT Filter Curve IRFC Large-Scale Phenomena 3 none LSPN Near Nucleus Studies 4 none NNSN Photometry Polarimetry 5 Broad Band Mag PMAG Narrow Band Flux PFLX Polarimetry PPOL Stokes PSTOKE Radio Science 6 Continuum RSCN Occultation RSOC OH RSOH Radar RSRDR Spectral Line RSSL Spectroscopy Spectrophot 7 none SPEC Amateur Observations 8 Drawing AMDR Photography AMPG Spectroscopy AMSP Visual Magnitude AMV Meteor Studies 9 Radar MSNRDR Visual Estimates MSNVIS ______________________________________________________________________ Concerning the numeric portion of filenames, calibration files for IRIM, IRSP, LSPN, and SPEC begin at 4001, whereas the Halley data themselves for all disciplines and subdisciplines start at 0001. The International Halley Watch agreed early in the project that all data would be submitted by the individual Disciplines to the JPL Lead Center using the FITS format. When the decision was made to distribute this information on CD-ROM, it was determined that the data had to have even broader accessibility. For this reason, the original FITS files, with contiguous headers and data, were split into separate files. The original FITS byte stream could then be recovered by concatenating the appropriate header and data files. The tabular data, even though presented as valid FITS data records, are really just ASCII byte strings that can be exported to many machines. In addition, detached PDS labels were constructed to allow parallel definition of the data files for the Planetary Data System. (We thankfully acknowledge the help of M. Martin, PDS, for assisting the IHW through version 2.0 of the Object Description Language to describe the SPECTRUM object.) The file extensions follow suggestions by the Planetary Data System (SPIDS v1.1) for tabular and image data, and are listed below: .DAT - other non-image and non-tabular data .FIT - original FITS file .HDR - FITS header records .IBG - data records for subsampled browse image .IMG - image data records .LBL - detached PDS stream format .TAB - table data records as ASCII The five PDS objects in this archive are FITS_LABEL (header), IMAGE, TABLE, TEXT, and SPECTRUM; a label occurs for each datafile. Our aim was to construct a basic PDS label for each data file on the CD-ROM. Files that remain in the original FITS form do not have a PDS label; these can occur as DOCUMENT files and datafiles for Comet Crommelin and Comet Giacobini-Zinner. Inclusion of the PDS labels allows use of CD-ROM imaging software already available from JPL. The PDS labels are text files, with lines terminated by the characters. The total size of the files is maintained in the required SFDU_LABEL. 6. REFERENCES Martin, T., Martin, M., Braun, M., Johnson, T., Davis, R., and Mehlman, R., SPIDS: Standards for the Preparation and Interchange of Data Sets, JPL D-4683: October 3, 1988. "Information Processing -- Volume and File Structure of CD-ROM for Information Interchange", Reference Number ISO 9660: 1988(E), Developed in coordination with the National Information Standards Organization, National Bureau of Standards, Administration 101, Library E-106, Gaithersburg, MD.