DICAL README ------------------------------ PROJECT COMPONENTS ------------------- DICAL - The main calibration routine. This guy is the workhorse and does the actual calibration DICAL_GUI - Graphical front-end to DICAL DICAL_CONFIG - Graphical utility to specify where DICAL will get calibration data FILES ------- DICAL* - Mostly modules in the calibration pipeline and definition files for the pipeline. The two exceptions are DICAL_GUI and DICAL_CONFIG which are separate parts of the project. GUI* - Files for the DICAL_GUI front end Everything else - Tools used in numerous modules STEP APPLIED IN PROCESSING -------------------------- Calibrate Temperatures and Voltages in the Header - dical_tempvolt.pro Decompression - dical_decompress.pro Flag Saturated Pixels - dical_flagsat.pro ADC Bit Weighting (invalid) - dical_bitweight.pro Linearize (IR only) - dical_lineardn.pro XTalk Removal - dical_xtalk.pro Dark Subtraction - dical_darkframe.pro Normalize Quadrant Gains (invalid) - dical_gain.pro Flat Field - dical_flatfield.pro Flag Bad Pixels - dical_fixbadpixs.pro Transfer Smear Removal (VIS only) - dical_desmear.pro Absolute Calibration - dicalir_dntorad.pro & dical_radtoif.pro (IR) dicalvis_dntoradif.pro (VIS) Reclaim Bad Pixels and Gaps - dical_fillgaps.pro Cosmic Ray Removal - dical_despike.pro Denoising - dical_denoise.pro Deconvolution - dical_mtf.pro Geometric Calibration (invalid)- dical_geom.pro RUNNING THE PIPELINE -------------------- For command line running, see the documentation in DICAL For graphically running, use DICAL_GUI. A single image can be calibrated by opening the raw FITS file. For batch processing, open a text file with a different file name on each line. ACESSING CALIBRATION FILES -------------------------- All of the calibration files are stored on the SDC in: .../Epoxi/Data/Cal/ Each sub directory specifies a different type of calibration file. All of the files are named using the following format: -YYMMDD--.fit For example, a spectral map valid for images after May 6, 2004 for IR mode 3 at 135.23K would be labeled: HRIIR-040506-3-135_23.fit Pointers to all of these files are stored in an SQL database along with some extra calibration data. To access any calibration data, use the DI_SQLQuery routine. If the query returns a remote filename , then use the getLocFn routine to convert the remote filename into a local one. If needed, the file is downloaded using wget. To update the database, use resetdb_dical.sh followed by filldb_dical.csh. Both of these files are found in the Db/ directory. WHEN NEW CALIBRATION FILES ARRIVE --------------------------------- When calibration files arrive, they are usually not in a compatible format. To fix this, they must be converted to FITS files with the filename format mentioned above. Since each source is different, I have created scripts/programs as I need them to convert the files. All of these are scattered throughout the Data/ directory. Once the files are converted, they need to be moved to the proper calibr ation files directory and the database needs to be updated. All calibration files must have the following keywords in the header: MSNCONFG (= 'FLIGHT' if the adjustment was derived from flight data) FNORIG DATEDLV AUTHOR For data that is only stored in the SQL database, like the wavelength for each VIS filter, there is a script/program in the Db/ directory that is called by filldb_dical.csh RERUNNING DATA ON THE SDC ------------------------ If you have a fully functioning SDC, the following steps must be performed on the SDC order to rerun data with new calibration files or algorithms. 1. Place new calibration files in their appropriate directory 2. Run the DiCal/Db/resetdb_dical.sh script 3. Run the DiCal/Db/filldb_dical.csh script 4. Turn off the cron jobs and make sure jobs finish 5. Touch any files in the Data/FLIGHT/RAWMRGFITS directory that you want recalculated so that those files have a current time stamp 6. Turn the cron jobs back on FILE HISTORY ------------------------------ Created - Mark Desnoyer(md246@cornell.edu) Feb 11,2005 Added pipeline specifics and how to rerun data - Mark Densoyer May 22, 2006 Changed location to Epoxi - BTCarcich 02.October, 2007