OSIRIS Optical, Spectroscopic, and Infrared Remote Imaging System OSIRIS Calibration Pipeline RO-RIS-MPAE-RP-XXX / D-3 Issue: Draft Revision: 3 19/08/2009 Prepared by: Cecilia Tubiana Approval Sheet prepared by: Cecilia Tubiana (signature/19.08.2009) approved by: put name here (signature/date) Document Change Record Iss./Rev. Date Pages affected Description Draft 23 April 2009 all first draft 19 August 2009 first revision sec 1.1 pipeline version sec 2.2 general description sec 2.2.4 partially updated sec 2.2.9 updated sec 2.2.11 modified sec 2.2.12 added Table of contents 1 General aspects 1 1.1 Scope 1 1.2 Introduction 1 1.3 Applicable Documents 2 1.4 Reference Documents 2 2 THE PIPELINE 3 2.1 Processing levels of the images 3 2.2 Structure of the pipeline 3 2.2.1 TANDEM ADC OFFSET 4 2.2.2 COHERENT NOISE 5 2.2.3 SEPARATION OF THE ELEMENTS IN THE FRAME 6 2.2.4 BIAS SUBTRACTION 7 2.2.5 EXPOSURE TIME NORMALIZATION 7 2.2.6 DARK CURRENT SUBTRACTION --> to be removed from the pipeline 8 2.2.7 BAD PIXELS AND BAD COLUMNS REMOVAL 8 2.2.8 FLAT FIELD CORRECTION 9 2.2.9 ABSOLUTE CALIBRATION 9 2.2.10 DISTORTION CORRECTION 10 2.2.11 SIGMA (or ERROR ESTIMATE) MAP 11 2.2.12 PIXEL QUALITY MAP 12 2.3 Folders 12 List of Figures Figure 1: Structure of the OSIRIS CCDs 2 List of Tables Table 1 Summary of the routines belonging to the OSIRIS calibration pipeline. 13 1 General aspects 1.1 Scope This document describes the pipeline for calibration of the images acquired by both OSIRIS cameras: the Wide Angle Camera (WAC) and the Narrow Angle Camera (NAC). The document presents the pipeline in its current state and it will be updated when necessary. The current pipeline version is SNAPSHOT_20090629. Additionally, this document contains auxiliary information (e.g. some of the technical details) about each calibration step and calibration frames. 1.2 Introduction The pipeline has been developed to calibrate OSIRIS science frames. Both NAC and WAC use 2k x 2k pixel backside illuminated CCDs (RD1). The OSIRIS CCD has 2048 columns and 2052 lines (2 + 2048 + 2). The serial register contains 2148 pixels (2 + 48 + 2048 + 48 + 2). A sketch of the CCD is shown in Figure 1. The two extra lines at the bottom of the CCD (colored in red in Figure 1) are generally discarded. The two extra lines at the top of the CCD (colored in green in Figure 1), which are read out after the image data, are called overclocking lines. The first two extra pixels in the serial register (colored in fuchsia in Figure 1) are ignored, thus 48 extra pixels (colored in light green in Figure 1) appear at both ends of the serial register. These pixels, which are read out before or after the image data are called pre-clocked pixels and post-clocked pixels, respectively. Pre-clocked and post-clocked pixels do not represent physical pixels of the CCD. Overclocking lines are used to determine the charge transfer efficiency, while pre-clocked and post-clocked pixels are used to estimate the bias and the electronic noise level of the image. Figure 1: Structure of the OSIRIS CCDs The cameras use a dual 14 bit Analogue to Digital Converter (ADC) configuration giving an effective 16 bit ADC system (ADC-TANDEM). NAC and WAC CCDs are equipped with two amplifiers (A and B) which can be used individually or in parallel, to speed up the readout process. The CCDs can be readout using hardware windowing and hardware binning (1x1, 2x2, 4x4 and 8x8). 1.3 Applicable Documents no. document name document number, Iss./Rev. AD1 OSIRIS user manual RO-RIS-MPAE-SP-025 AD2 OSIRIS Experiment Data Record and Software Interface Specification (EDR/SIS) RO-RIS-MPAE-ID-018 1.4 Reference Documents no. document name document number, Iss./Rev. RD1 OSIRIS user manual RO-RIS-MPAE-SP-025 RD2 Rosetta-OSIRIS to Planetary Science Archive Interface Control Document RO-RIS-MPAE-ID-015 RD3 CCD Readout Board – (CRB) RO-RIS-MPAE-SP-020 RD4 OSIRIS Calibration Report RO-RIS-MPAE-RP-147 2 THE PIPELINE 2.1 Processing levels of the images The OSIRIS data levels are (RD2): Level 0: raw header, raw image. Level 1: calibrated header, raw image. Level 2: calibrated header, radiometric calibrated image (in W m-2 sr-1 nm-1). Level 3: calibrated header, radiometric calibrated and optical distortion corrected image. The Ground Support Operating System (GSEOS) saves the image data as ORISIS level 0 image files. OSIRIS level 0 images are processed by a software application (tmi2pds.exe) which calibrates the header information and adds information like the spacecraft position and orientation. The output is OSIRIS level 1 images, which are used as input for the calibration pipeline. OSIRIS level 2 and level 3 images are generated by the pipeline. The OSIRIS data levels correspond to CODMAC (Committee On Data Management, Archiving, and Computing) levels: OSIRIS level n = CODMAC level n-1 Thus, the listed OSIRIS data levels correspond to CODMAC levels 1-4, respectively. 2.2 Structure of the pipeline The routine to calibrate the OSIRIS WAC and NAC images is osiris_calibrate_dir.pro. The routine generates level 2 and level 3 images of all the level 1 images stored in the input directory. Moreover, this routine converts the calibrated images (level 2 and level 3) to .jpg and stored them in a separate folder. To calibrate one single image, the routine osiris_calibrate_file.pro can be called directly. The disadvantage is that the level 2 and level 3 .jpg images are not created, since .jpg images are generated by osiris_calibrate_dir.pro. The following steps are applied by the pipeline to calibrate a raw image: 1. Removal of tandem ADC offset 2. Removal of coherent noise 3. Separation of the various frame elements: pre-clocked and post-clocked pixels, overclocking lines and image (or active area) 4. Subtraction of bias 5. Normalization to exposure time 6. Removal of bad pixels and bad columns 7. Division by flat field 8. Conversion to radiometric units (absolute calibration) 9. Correction of geometric distortion Steps 1 to 9 create level 2 images, while step 10 creates level 3 images. Each of these steps is described in the following paragraphs. A summary of the routines belonging to the OSIRIS calibration pipeline is given in Table 1. Radiometric calibrated images (level 2) are created by the procedure osiris_level1_to_level2.pro, while the geometric distortion corrected images (level 3) are generated by osiris_level2_to_level3.pro. In addition to the calibrated images, the procedures generated quality maps and sigma maps for level 2 and level 3 images (described in Sec. 2.2.11). Most of the calibration steps require a calibration frame or file in order to be performed. These files are stored in the folder: *\osiris_calibration_pipeline\database\..., where * = OSIRIS calibration pipeline directory (e.g.: bias frames: *\osiris_calibration_pipeline\database\bias). If more than one version of the calibration files exists, the pipeline looks automatically for the latest version, which corresponds to the one with the highest version number. 2.2.1 TANDEM ADC OFFSET Both NAC and WAC are equipped with two 14 bits ADCs (ADC-LOW and ADC-HIGH) to digitalize the CCD readout. The readout electronics can use the ADCs separately (either ADC-LOW or ADC-HIGH) or together in a dual 14 bits ADC configuration (ADC-TANDEM). The latter option gives an effective dynamical range of 16 bits, which is resolved using a sub-ranging technique. The signal is split between the two ADCs as follows (RD3): when 0 ? DN ? 214 - 1 ==> ADC-LOW is used when 214 ? DN ? 216 - 1 ==> ADC-HIGH is used. When using the dual configuration, an ADC offset is added to the ADC-HIGH in order to well distinguish from which ADC the signal comes. Thus, the digitalized signal is: DN = ADC-LOW output when 0 ? DN ? 214 - 1 DN = ADC-HIGH output + ADC offset when 214 ? DN ? 216 - 1, where ADC-LOW output and ADC-HIGH output represent the digital signal corresponding to the analogue signal entering in the ADC. The ADC offset is applied only to ADC-HIGH, when the ADCs work in dual configuration (ADC = TANDEM). Thus, no correction is needed to ADC-HIGH when the two ADCs are used separately and no correction needs to be applied to ADC-LOW whether only one ADC is used or both ADCs are used in the dual configuration. The ADC offset is removed by the procedure: osiris_calibrate_adc_offset.pro. The procedure looks for the pixels where the value ? 214 and to these pixels it subtract the ADC offset. The CCD has two readout channels (A and B) which can also be used in parallel for faster readout. The ADC offset correction has to be applied whether one of the two readout channel is used or they are used in parallel (amplifier = BOTH). The procedure handles separately the cases amplifier = A, amplifier = B and amplifier = BOTH. In fact the ADC offset is different for amplifier A and amplifier B and moreover when reading out through one of the amplifiers or both of them simultaneously (offset A ? offset B ? offset A(BOTH) ? offset B(BOTH)). 2.2.2 COHERENT NOISE The NAC and WAC signal chains are exposed to noise generated in the CCD readout board (CRB) power converter modules and in the data processing unit (DPU) power converter module. The CRB power converter modules are the dominant noise source. The presence of coherent noise in the images is due to the use of twisted pair cables instead of coaxial type interconnection cables. Both, NAC and WAC CRB power converter modules are synchronised with the corresponding pixel readouts. The converters contain two primary switches in push-pull, where one switch is active for the even pixels while the other switch is active for the odd pixels. Both switches produce different noise effects. Thus, we have different coherent noise on odd and on even pixels, which produces a black-and-white pattern upon the image data with amplitudes of up to 25 DU. This pattern is vertically structured if extra pixels are not included, while it is diagonally structured (with 45° inclination) if the extra pixels are included. The coherent noise can be effectively removed. In case for readout the two amplifiers are used in parallel, the two parts of the CCD are processed separately (RD4). The removal of coherent noise is done by the procedure: osiris_calibrate_noise.pro. The routine osiris_rmv_chn_noi.pro makes the forward Fast Fourier Transform (FFT) of the input image. If the coherent noise is vertically structured, the maximum amplitude of the FFT is localized in the central pixel of the bottom line of the CCD. If, instead, the coherent noise is diagonally structured, the maximum amplitude of the FFT is localized in the central pixel of the CCD. After low frequencies are discarded, the inverse FFT of the FFT image is performed. The frame, which has the same dimensions as the input one, contains only the coherent noise and it is subtracted from the input frame. 2.2.3 SEPARATION OF THE ELEMENTS IN THE FRAME Each frame is composed of three elements: pre-clocked and post-clocked pixels, overclocking lines and image. 1. Pre-clocked and post-clocked pixels: 48 + 48 pixels in the serial register (colored in light green in Figure 1) which are read out before and after, respectively, the image data. They do not represent physical pixels of the CCD and are filled by the output amplifier. They can be used to estimate the actual bias and electronic noise level of the image. 2. Overcloking lines: two extra lines at the top of the CCD (colored in green in Figure 1) which are readout after the image data. These lines are used to determine the charge transfer efficiency. 3. Image: 2048 x 2048 pixels which correspond to the active area of the CCD. The three elements are separated by the procedure: osiris_seperate_frames.pro. The procedure handles overclocking lines first, then the pre-clocked and post-clocked pixels, separating them from the image. In case the readout is done using the two amplifiers in parallel, the keyword ‘prepix’ contains pre-clocked pixels of the amplifier A and ‘postpix’ contains pre-clocked pixels of the amplifier B. Nowadays, OSIRIS processes and downlinks the pre-clocked and post-clocked pixel and the overclocking lines sections seperately. The TMI2.10 telemetry decoder saves the sections as separate level 0 files: - NAC_2007-02-24T18.28.36.011Z_ID10_0053001004_F16.IMG: image file (ID) - NAC_2007-02-24T18.28.36.011Z_PA10_0053001004_F16.IMG: pre-clocked pixels amplifier A (PA) - NAC_2007-02-24T18.28.36.011Z_PB10_0053001004_F16.IMG: pre-clocked pixels amplifier B (PB) The files containing pre-clocked and post-clocked pixel and overclocking lines exist only for un-calibrated images (level 0). For level 1 images, pre-clocked and post-clocked pixel and overclocking lines images are stored as image objects in the image file itself. However, since they are optional data their existence depends on the image acquisition mode. In level 2 and level 3 images they will not be present anymore. The routine osiris_seperate_frames.pro is still part of the pipeline in order to treat also old data, in which the three elements (pre-/post-clocked pixels, overclocking lines and image) are stored in the same frame and need to be separated on ground. 2.2.4 BIAS SUBTRACTION The bias level is an electrical offset which is applied to the CCD before readout, first, to make the CCD working and, second, to shift the zero-level of the CCD to a positive value in order to properly sample the readout noise. The bias frame is obtained with a very short exposure (ideally a zero-seconds exposure) and the shutter closed. The bias level is the mean value of the counts for such a bias frame. The bias correction is done by subtracting the median of the bias frames from the science image. As bias we can use: 1. pre-clocked and post-clocked pixels. 2. Bias frame acquired close in time to the images and stored in the same directory as the image itself. 3. Bias frame stored in the database. The procedure to bias-subtract the images is: osiris_subtract_bias.pro. The routine uses only the bias frame stored in the database to bias subtract the images. Based on bias frames obtained in calibration campaigns before launch, the bias is flat across the CCD. A unique bias value is subtracted for all CCD pixels. The bias level depends on the camera used (WAC or NAC), the operational mode (readout done using a single amplifier or both amplifiers in parallel), the amplifier used, the hardware binning, the hardware windowing and the internal CRB configuration. Due to the aforementioned dependences, the routine checks the image settings reading the header of the image and extracting the needed information. Amplifier A and amplifier B are handled separately, since they have different bias levels. 2.2.5 EXPOSURE TIME NORMALIZATION -- ?t is only an approximation – need to take into account the actual blade motion Each image is normalized to 1 s of exposure time. This is done by the procedure: osiris_exposuretime.pro. The commanded exposure time (in s) is read from the header of the image. This time is then corrected for the difference between the commanded and the actual exposure time (?t = 0.0027 s): exp_timeactual ? exp_time = exp_timecommanded – ?t. ?t = 0.0027 s is the time that the shutter takes to move the blades through the blade overlap. In general ?t = 0.0027 is only an approximation, it would be needed to swap to a calibration which takes the actual motion of the blades into account. The image is divided by the exposure time (exp_time). NB: The start time of the observation contained in the filename is shifted by a few seconds with respect to the actual start time, which is stored in the image header. 2.2.6 DARK CURRENT SUBTRACTION --> to be removed from the pipeline The pixels which are now corrected by this procedure will be then marked in the quality map. Thermal agitation in the silicon layers of the CCD produces electrons which are indistinguishable from photoelectrons. Thermal agitation gives rise to a dark current which, in absence of light, follows the law: where A and B are constants, k is the Boltzmann constant and T is the CCD operational temperature. Based on calibration campaigns performed in 2001 and 2003, the dark charge generation is less than 0.1 e-/s for temperature T < 210 K, thus negligible. The WAC and NAC operational temperature is 180 K, thus the dark current is negligible during nominal OSIRIS operations and no dark current correction should be needed. Nowadays, the dark current is smaller than the readout noise. An electrostatic discharge damaged a 9x9 pixels area close to the physical center of the NAC CCD. For these pixels the dark current is not negligible and it has to be subtracted. These hot pixels do not show any temperature dependence behavior while their intensity is exposure time dependent (most likely with a linear dependence). The procedure for dark current subtraction is: osiris_darkcurrent_removal.pro. The dark current frame is normalized to 1 s before to be subtracted from the image. No temperature dependence is implemented. Any dark current correction is needed and applied to the WAC CCD. 2.2.7 BAD PIXELS AND BAD COLUMNS REMOVAL -- Better algorithm to correction NAC column 995 is needed, not only using the median value of the neighboring columns (the correction is needed for compression). Correction: Stubbe, Laurent, Nick. -- Remove the bottom pixel of column 995. -- No bad pixel removal is done. NAC images display a hot column: column number 995. The pixels of this column have an intensity of few DN higher than the neighboring pixels (RD4). The procedure osiris_badpixel_removal.pro corrects the NAC image subtracting the median of the difference between the bad column and the neighboring columns (columns number 994 and 996). Column 995 is however normally replaced by the Instrument Flight Software (IFSW) before compressing the image. Thus, the pipeline does not do any correction since the image has already been corrected. The major problem regarding this calibration step is that the two surrounding columns (994 and 996), which are used to correct column 995, unfortunately also show some problems. Therefore the correction does not work properly. 2.2.8 FLAT FIELD CORRECTION The flat field corrects for the non-uniform sensitivity of the CCD. Two kinds of non-uniformities can be distinguished: a pixel-to-pixel variation and a gradient across the chip. Thus, when the CCD is exposed to a uniform radiation, the resulting image is not uniform. These non-uniformities are corrected flat fielding the image: dividing the image by the flat field frame. Since there is no possibility to get flat field exposures in flight, we use flat field frames obtained during a calibration campaign. The OSIRIS flat fields were created using an integrating sphere with calibrated halogen lamps (for NIR and visible) and xenon lamps (for UV) (RD4). The flat field correction is done by the procedure: osiris_flatfield.pro. The procedure checks the filter combination and the hardware binning of the images to be flat fielded. Since only full frame flat fields are available, if the science images are binned, they are flat fielded by binning the corresponding flat field. 2.2.9 ABSOLUTE CALIBRATION Images are intensity calibrated: they are converted from DN/s to W m-2 sr-1 nm-1. The calibration constants for the absolute calibrations are obtained from the observation of standard stars (e.g. Vega, 16 Cyg (A+B), 58 Aql and Eps Aqr). Standard star images are bias subtracted, exposure time normalized, flat field corrected and the cosmic rays are removed. After the center of the star is identified, its flux is measured with the technique of aperture photometry (in DN/s). The tabulated flux of the star at the central wavelength of the filter used (in W m-2 nm-1) is determined from its tabulated magnitude (magstar): where fluxsun is the flux of the sun at the central wavelength of the filter (in W m-2 nm-1) and mstar and msun are the magnitude of the star and of the sun, respectively. The calibration constant is: where scale is the pixel scale of the camera (in sr). The calibration constant is in DN s-1/ W m-2 nm-1 sr-1. If the standard star is not a solar type star (e.g. Vega), the calibration constant must be scaled to solar colors: where and are the sun flux and star flux integrated on the filter passband (in W m-2), respectively, and Fsun(?0) and Fstar(?0) the flux of the sun and of the star at the central wavelength of the filter??0 (in W m-2 nm-1), respectively. The procedure for the absolute calibration is: osiris_abscal.pro. For the determination of the NAC calibration coefficients Vega and 16CygA+B observations performed in February 2007 and November 2007 have been used. Due to the low S/N ratio of 16CygA+B measurements in UV, for this wavelength range the calibration coefficients have been determined using only Vega measurements. The calibration coefficients are obtained using a weighted average of the available measurements: where is the calibration coefficient at wavelength ?, is the weight and is its standard deviation of the calibration coefficient. 2.2.10 DISTORTION CORRECTION As last step, the geometric distortion is removed from the images. This step is done by the procedure: osiris_level2_to_level3.pro. Two different sub-routines are used to correct WAC and NAC images, since two different methods are used. The routine osiris_level2_to_level3_wac.pro removes the geometric distortion from the WAC images, giving as output a re-sampled image having the same scale all over the image (i.e. as the image would have been acquired from an ideal distortion free optical system). The reference focal length (i.e. focal length for undistorted images) is 135.68 mm, corresponding to 20.52''/pix which is the mean scale on the optical axis of the camera (center of the field of view). The reference pixel for distortion removal is (1043, 937), i.e. the optical center (optical axis direction) for the camera. The coefficients kx and ky are determined, starting from tabulated values and calibration frames, via a third order polynomial fit of the surface that describes the value of the focal length for each pixel of the CCD: where are the undistorted coordinates expressed in function of the image coordinated . The coefficients for distortion removal (kx and ky) are the coefficients of the third-order polynomial fit. To obtain the corrected level 3 images, the level 2 images are wrapped according to the tabulated kx and ky distortion removal coefficients. Whether the level 3 images are photometrically corrected or not is still on debate. The routine osiris_level2_to_level3_nac.pro removes the geometric distortion from the NAC images. The de-distorted coordinates corresponding to each pixel are pre-calculated in FITS images using the distortion coefficients of the camera, calibrated on star fields. Each image is then re-sampled according to these coordinates. The FITS images are loaded at the beginning of each calibration session. Nowadays, level 3 images are not photometrically corrected. 2.2.11 SIGMA (or ERROR ESTIMATE) MAP Together with the calibrated images, the pipeline produces error estimate maps and pixel quality maps. The ERROR ESTIMATE MAP is a float image with the same dimension as the image itself, which contains the relative error for each pixel of the image (in %) determined using Poisson statistics: where is the intensity of the pixel with coordinates (i, j) in number of electrons. Since the Poisson statistics are done using the intensity in number of electrons, the image intensity has to be converted from DN to number of electrons and this is done using: where and are the intensity in number of electrons and DN, respectively, and gain is the number of electrons per DN (for OSIRIS WAC and NAC gain = 3.1 e-/DN in HIGH gain mode and gain = 15.5 e-/DN in LOW gain mode). NB: The routine osiris_calc_poisson.pro produces the error estimate map for level 2 and level 3 images. 2.2.12 PIXEL QUALITY MAP The PIXEL QUALITY MAP is an 8-bit image with the same dimension as the image itself and contains a quality estimate of each pixel. The quality map exists for data level 2 and higher. The pixel quality map for level 2 and level 3 images is produced by the routine: osiris_mk_quality_map.pro. The quality estimate values stored in the quality map are generated by setting a given bit to value 1 for specific effects. If more effect is present in the data several different bits can be set. The following value a possible: Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 0 Value 128 64 32 16 8 4 2 1 Effect BAD SAT DIM WARM LOSSY NLIN CONV SQRT BAD: Pixel is marked as bad (garbage data!) SAT: Pixel was saturated during the exposure (garbage data!) DIM: Pixel is marked as dim (low sensitivity – probably garbage data!) WARM: Pixel is marked as warm (increased and slightly varying ensitivity) – use with caution LOSSY: Pixel has seen lossy image compression NLIN: Pixel was expososed into the non linear DN range of the CCD CONV: Pixel has seen gauss convolution filtering as part of the image compression SQRT: Pixel has seen sqrt filtering as part of the image compression The general rule for the quality map is that low absolute values means good data and high absolute values mean suspect data. 2.3 Folders procedure folder main routine for calibration osiris_calibrate_dir.pro * \pipeline main routine for calibration osiris_calibrate_file.pro * \pipeline ADC offset osiris_calibrate_adc_offset.pro * \pipeline coherent noise osiris_calibrate_noise.pro * \pipeline osiris_rmv_chn_noi.pro * \pipeline\noise separate frame osiris_seperate_frames.pro * \pipeline bias subtraction osiris_subtract_bias.pro * \pipeline exposure time normalization osiris_exposuretime.pro * \pipeline dark current removal osiris_darkcurrent_removal.pro * \pipeline bad column removal osiris_badpixel_removal.pro * \pipeline flat fielding osiris_flatfield.pro * \pipeline absolute calibration osiris_abscal.pro * \pipeline distortion correction osiris_level2_to_level3.pro *\pipeline\level2to3 osiris_level2_to_level3_nac.pro *\pipeline\level2to3 osiris_level2_to_level3_wac.pro *\pipeline\level2to3 error estimate map osiris_calc_poisson.pro * \pipeline quality map osiris_mk_quality_map.pro * \pipeline Table 1 Summary of the routines belonging to the OSIRIS calibration pipeline. OSIRIS Reference: RO-RIS- Issue: Draft Rev.: - Date: Page: 11