Menu: Skip within this page

 

Data Archives

MPC @ SBN

MPC Home

MPC Annex

MPC FEEDBACK

The MPC Users' Group (MUG) invites community feedback on any issues, concerns, or questions about the MPC.

SBN FEEDBACK

Please provide feedback on the SBN website.

PDS Nodes

PDS Support

How to Prepare Data

PDS3 Format

This page is currently being kept/maintained while some datasets are grandfathered in. But new datasets are expected to be in PDS4 format. So please read the PDS4 instructions.

Preparing data for submission to the PDS includes not only labelling of data files and collection of documentation, but also content verification and usually at least some measure of reformatting to get the data into one of the standard deep archive formats required by the PDS. The formatting and documentation requirements of the PDS were developed in conjunction with astronomers to ensure that the data archived will be accessible long past the lifetime of the mission, its collaborators, or its hardware/software installation. The PDS format is specifically designed to be an archive format, not a working data format. Listed on this page are pointers to information and software tools to assist you in preparing and verifying data for submission to the PDS.

Introductions

The Planetary Data System (PDS) is an archive of data products from NASA planetary missions and supporting ground observations sponsored by the NASA Office of Space Science. A useful mission archive, as required by most NRAs and AOs, includes formatted raw data from each instrument, data calibrated to physical units, and derived products. Derived data products are those based on further processing of the calibrated data, or on combinations of data from different sources, and include things like maps, overlays, and comparative tables. The archive should contain sufficient documentation of the mission, the instruments, and the calibrations so that future generations of scientists can intelligently use and, if required, even recalibrate the data on contemporary computing platforms, without recourse to the original mission personnel and facilities.

The PDS has an Archive Preparation Guide available to guide new data preparers through the archiving process. Preparing data for submission to the PDS includes not only the labeling of data files and collection of documentation, but also content verification and usually at least some measure of reformatting to get the data into one of the standard deep archive formats required by the PDS. The formatting and documentation requirements of the PDS were developed in conjunction with astronomers to ensure that the data archived will be accessible long past the lifetime of the mission, its collaborators, or its hardware/software installation. The PDS format is specifically designed to be an archive format, not a working data format.

The On-Line Archiving Facility (OLAF) (for PDS3) is a tool that data preparers can use to prepare and submit data and supporting documentation to the PDS for review and archiving. OLAF can accommodate tabular ASCII data and simple, single-image FITS files, whether from ground-based or spacecraft observations. OLAF simplifies the generation of the label, index and catalog files required for archiving these simple data formats.

If you are submitting observations of a solar system small body, please follow these Small Body Target Name Conventions to ensure that users can find your targets in the PDS catalog database.

If you are using a coordinate system of a small body and this system has not been approved by IAU or you want to update the IAU approved coordinate system for some object, then SBN should review your coordinate system before you deliver data using the coordinate system. The procedure and requirements are described in the document Coordinate Systems at PDS-SBN.

A Word About Data Formats

It has been our experience that simple data formats have several major advantages over more complex or proprietary formats:

  1. They are immediately accessible to a much wider section of the science community.
  2. They are stable over decades, maintaining accessibility for the projected life of the archive.
  3. They are easier to document and verify, facilitating efficient review and archiving.

Therefore, the Small Bodies Node is a strong advocate of simple data structures, especially in high-level mission data products. The simple data structures are homogeneous arrays and flat (ASCII or binary) tables. Not surprisingly, these are also the formats well supported by the Flexible Image Transport System (FITS) standard, which provides a very general and widely accessible format for exchanging files between various platforms.
Note that, from an archiving point of view, headers (and trailers) that are appended to the data structures present no major problems, as long as they do not interrupt the byte stream which constitutes the observational data. Thus any compliant FITS file can be easily prepared for PDS archiving without modifying the original FITS file because the headers occur before the individual data structures.

PDS Standards Documents

In addition to the printed documents available from the PDS, there are on-line versions accessible by Web browser:

Archive Preparation Guide
This workbook provides a cook-book approach to preparing data for submission to the PDS.
(no longer exists for PDS3)
Information for Data Providers (PDS4)
Information for Proposers
The PDS Information for Proposers page includes links to various guides, tools and examples for use in preparing proposal containing archiving elements.
PDS Standards Reference
This reference contains the detailed PDS standards for things like archive layout, data label requirements, units of measure, time values and other aspects of the PDS archives.
various formats
Planetary Science Data Dictionary (PSDD)
The PSDD contains the definitions for all keywords and data structures used in and available for archive datasets. A flat ASCII version of the PSDD is required to run some of the software tools; there is a Web interface into the dictionary which allows you to retrieve specific keyword definitions.
Data Dictionary Search

PDS Tools

A number of tools have been developed for generating and verifying PDS labels. The complete inventory of general PDS utilities is available from the Engineering Node. The tools our data preparers have found most useful are:

SBN Tools

The SBN has also developed suites of small utilities for working with the types of data we most frequently encounter. The majority of these routines are written in Perl; some are written in ANSI-standard C. All are maintained in a linux environment. The source code for these routines is available from the SBN, as is, from our software archives. Utilities available include:

Verifying Data Submissions

Most of the format and content validators used by the PDS archivists are available to download and run locally. It is highly recommended that data preparers run at least the PDS Validation Tool (VTool), available from the PDS Software Download page, on all labels and catalog files and to correct all syntactic and semantic errors reported. (Warnings about standard values and keywords which are not yet in the data dictionary but which are scheduled to be added may be ignored.)

The same PDS Software Download page has other types of validators available, as does the Tool Registry page of the main PDS site, for use on particular types of data or for checking entire data sets. The Volume Validation Tool, for example, can perform referential integrity and indexing checks on an entire data set.

If you are submitting data via the On-Line Archiving Facility, OLAF (for PDS3) will run the appropriate validation routines on the data as it creates the labels and other PDS-required files.