From bb93e01db17697dde3b4e64132824cd265849a84 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Fri, 11 Aug 2023 13:28:17 -0400 Subject: [PATCH 01/10] Added description for the application of the saturation image and when the former algorithm of using just a single scalar value is still applied. Updated the history and some reformat of heading levels. Updated the order of major processing steps for UVIS. Noted unit conversion happens in the wf32d portion of the processing during FLATCORR. Updated the discussion of some of the pipeline components (e.g., wf3cte). Please note there are STILL updates which need to be done to this documentation, particular for additional components and the IR pipeline. --- docs/wfc3tools/calwf3.rst | 1 + docs/wfc3tools/history.rst | 11 ++-- docs/wfc3tools/ir_pipeline.inc | 27 +++++----- docs/wfc3tools/uvis_pipeline.inc | 88 ++++++++++++++++++-------------- docs/wfc3tools/wf32d.rst | 25 ++++++--- docs/wfc3tools/wf3ccd.rst | 53 ++++++++++++++----- docs/wfc3tools/wf3cte.rst | 20 ++++++-- docs/wfc3tools/wf3ir.rst | 26 +++++++--- docs/wfc3tools/wf3rej.rst | 19 +++++-- 9 files changed, 180 insertions(+), 90 deletions(-) diff --git a/docs/wfc3tools/calwf3.rst b/docs/wfc3tools/calwf3.rst index a8db7bc..3212bd5 100644 --- a/docs/wfc3tools/calwf3.rst +++ b/docs/wfc3tools/calwf3.rst @@ -262,6 +262,7 @@ but the program's exit code will still be checked for successful completion. +========================================= Types of Files Used as Input to `calwf3` ======================================== diff --git a/docs/wfc3tools/history.rst b/docs/wfc3tools/history.rst index 2797499..5811df4 100644 --- a/docs/wfc3tools/history.rst +++ b/docs/wfc3tools/history.rst @@ -6,12 +6,15 @@ Software Update History for HSTCAL.CALWF3 .. warning:: IRAF version of WFC3 no longer maintained or delivered, use WFC3TOOLS in HSTCAL or call the executable from your operating system command line. With version 3.3 the pipeline now produces two versions of each calibrated file, one set with the CTE correction applied and one set without the CTE correction applied -**Updates for Version 3.7.0 16-Feb-2022 - MDD** - - Implemented algorithm change to use an image to detect and flag full-well saturation versus a simple scalar. +**Updates for Version 3.7.0 08-Jun-2023 - MDD** + - Implemented a change to use an image to detect and flag full-well saturation versus a simple scalar. The new routine, dofwsat.c, is similar to what has been done for calacs. The WFC3 implementation is more complicated in that there are serial virtual overscan columns, as well as binned images, - to accommodate. The detection/flagging occurs after blev and bias correction while the output is - still in counts. + to accommodate. The detection/flagging occurs after blev and bias corrections while the output is + still in counts. If the SATUFILE keyword is missing from the FITS header, or the keyword does + not have a valid filename as a value, the code will revert to using the original method of flagging + full-well saturation. The flagging will be done in doDQI and use a single value as the saturation + threshold. **Updates for Version 3.6.2 27-May-2021 - MDD** - Fixed bug to address calwf3.e crashing (Abort trap: 6) when taking an existing _ima.fits (IR) file diff --git a/docs/wfc3tools/ir_pipeline.inc b/docs/wfc3tools/ir_pipeline.inc index 60ca483..6ca5e2f 100644 --- a/docs/wfc3tools/ir_pipeline.inc +++ b/docs/wfc3tools/ir_pipeline.inc @@ -1,6 +1,7 @@ .. _ir_pipeline: +=========== IR Pipeline =========== @@ -22,7 +23,7 @@ IR pipeline output files using the RAW file as input: Data Quality Initialization (DQICORR) -------------------------------------- +===================================== Initialize the data quality array for the image using the reference file specified in its header with BPIXTAB. The DQ array is no longer updated to reflect any TDF transition during the sample. If you want to update DQ pixel values yourself before running further processing, do it after this first step has been completed, remembering that the data in this extension is always in units of UNSIGNED INTEGER. The following table lists the DQ flag values and their meanings: @@ -52,7 +53,7 @@ RESERVED2 32768 can't use Estimate the signal in the zero read (ZSIGCORR) ------------------------------------------------ +=============================================== This step measures the signal between the super zero read in the linearity reference file (NLINFILE) and the science zero read exposure, the steps are roughly as follows: @@ -67,19 +68,19 @@ This step measures the signal between the super zero read in the linearity refer * This step acutally subtractes the super zero read from the science zero read instead of calculating an estimated signal based on the first read and zero read + estimated exposure time between them so that the difference in readout time for subarrays is not an issue. Bias Correction (BLEVCORR) --------------------------- +========================== This step subtracts the bias level using the reference pixels around the perimeter of the detector, the boundries fo the reference pixels are defined in the OSCNTAB reference file. There are 5 reference pixels on each end of each row, but 1 is ignored on each side, for a total of 8 being used per row. The resistent mean of the standard deviation of all the reference pixels in the image is subtracted from the entire image and the value is stored in the MEANBLEV keyword in the output image header. The reference pixels are left in place in the IMA output image through processeing, but the final FLT image has been trimmed to just the science pixels. Zero read subtraction (ZOFFCORR) --------------------------------- +================================ The original zero read is subtracted from all groups in the science image, including the zeroth read itself, combining the DQ arrays with a logical OR. The ERR and SAMP arrays are unchanged and the TIME arrays are subtracted from each other. The exposure time for the group being corrected is reduced by an amount equal to the exposure time of the zero-read. At this point we've subtracted the mean bias using the reference pixels (BLEVCORR) and added back in the signal from the super zero read (done at the end of ZSIGCORR). What's left in the zero read of the science image is the superbias subtracted signal. The TIME and SAMP arrays are saved to the FLT image only AFTER the CRCORR step has been completed. Error array initialization --------------------------- +========================== The errors associated with the raw data are estimated according to the noise model for the detector which currently includes a simple combination of detector readnoise and poisson noise from the pixel. Readnoise and gain are read from the CCDTAB reference file. The ERR array continues to be summed in quadrature as the SCI array is processed. Inside the final FLT image, the ERR array is calculated by CRCORR as the calculated uncertainty of the count-rate fit to the multiaccum samples. @@ -89,7 +90,7 @@ The errors associated with the raw data are estimated according to the noise mo Detector Non-linearity Correction (NLINCORR) --------------------------------------------- +============================================ The integrated counts in the science images are corrected for the non-linear response of the detectors, flagging pixels which extend into saturation (as defined in the saturation extension of the NLINFILE reference image. The observed response of the detector can be represented by two regimes: @@ -135,14 +136,14 @@ The format of the linearity reference file: Dark Current Subtraction (DARKCORR) ------------------------------------ +=================================== The reference file listed under the DARKFILE header keyword is used to subtract the dark current from each sample. Due to potential non-linearities in some of the signal components, such as reet-related effecets in the first one or two reads of an exposure, the dark current subtraction is not aplied by simply scaling a generic reference dark image to the exposure time and then subtracting it. Instead, a library of dark current images is maintained that includes darks taken in each of the available predefined multiaccum sample sequences, as well as the available sub-array readout modes. The multiaccum dark reference file is subtracted read-by-read from the stack of science image readouts so that there is an exact match in the timings and other characteristics of the dark image and the science image. The subtraction does not include the reference pixel. The ERR and DQ arrays from the reference dark file are combined with the SCI and DQ arrays from the science image, but the SAMP and TIME arrays are unchanged. The mean of the dark image is saved to the MEANDARK keyword in the output science image header. Photometry Keywords (PHOTCORR) ------------------------------- +============================== The PHOTCORR step is performed using tables of precomputed values instead of calls to SYNPHOT. The correct table for a given image must be specified in the IMPHTTAB header keyword in order for calwf3 to perform the PHOTCORR step. The format of the file for the IR detectors is: @@ -163,14 +164,14 @@ where each extension contains the photometry keyword information for that specif * PHOTBW: the bandpass RMS width Conversion to Signal Rate (UNITCORR) ------------------------------------- +==================================== This step converts the science data from a time-integrated signal to a signal rate by dividing the SCI and ERR arrays for reach readout by the TIME array. No reference file is needed. The BUNIT keyword in the output data header reflects the appropriate data units. The FLATCORR keyword is checked to decide on proper units for BUNIT and skip this step if "countrate" is found. If FLATCORR is set to "complete", then the units should be electrons, otherwise they are counts (the digitized signal from the FPA). Fit accumulating signal and identify cosmic ray hits (CRCORR) -------------------------------------------------------------- +============================================================= This step fits the accumulating signal up the image ramp and identifies cosmic-ray hits for each sample using the `Fixsen et al (2000) `_ methods. @@ -206,13 +207,13 @@ The result of this step is stored as a single imset in the output FLT file. In Flatfield Correction (FLATCORR) -------------------------------- +=============================== This step corrects for sensativity variations across the detector by dividing the images by one or more reference flatfields (taken from the PFLTFILE, DFLTFILE or LFLTFILE header keywords). The mean gain from all the amps is used to convert to the image to units of electrons. Errors and DQ flags from the flatfields are combined with the science data errors and flag, the TIME and SAMP arrays are unchanged. Calculation of image statistics -------------------------------- +=============================== The min, mean, maxmin and max SNR (for the SCI and ERR) for data values flagged as "good" in the DQ array (i.e. zero) are calculated and stored in the output SCI image header, the reference pixels are not used. This is performed for all samples in the IMA file as well as the FLT image but the input data is not modified in any way. Updated keywords in the science header include: @@ -226,7 +227,7 @@ The min, mean, maxmin and max SNR (for the SCI and ERR) for data values flagged Reject cosmic rays from multiple images (RPTCORR) -------------------------------------------------- +================================================= Reject cosmic rays from multiple images. REPEAT-OBS exposures get combined with :ref:`wf3rej`. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIC (ocrrj) and WFPC2(crrej), providing a well-tested and robust procedure. CR-SPLIT is not used for the IR channel. All dithered observation get combined with Astrodrizzle (see `Astrodrizzle `_ ), which will also correct for geometric distortion. .. warning:: diff --git a/docs/wfc3tools/uvis_pipeline.inc b/docs/wfc3tools/uvis_pipeline.inc index 5cc0670..150a4eb 100644 --- a/docs/wfc3tools/uvis_pipeline.inc +++ b/docs/wfc3tools/uvis_pipeline.inc @@ -1,6 +1,7 @@ .. _uvis_pipeline: +============= UVIS Pipeline ============= @@ -16,15 +17,16 @@ As of `calwf3 v3.3`, the calwf3 pipeline processes all UVIS data twice, once wit * Calculate and remove and CTE found in the image (PCTECORR) * Calculate a noise model for each pixel and record in the error array (ERR) of the image (NOISCORR) -* Initialize the data quality (DQ) array of the image (DQICORR) +* Initialize the data quality (DQ) array of the image based on BPIXTAB , flag A-to-D saturation, and + potentially flag full-well saturation (DQICORR) * Correct for A-to-D conversion errors where necessary, currently skipped (ATODCORR) * Subtract bias level determined from overscan regions (BLEVCORR) * Subtract the bias image (BIASCORR) +* If applicable, employ a full-well saturation image, SATUFILE, to flag affected pixels. * Detect and record SINK pixels in the DQ mask (performed if DQICORR is set to PERFORM) * Subtract the post-flash image, if applicable (FLSHCORR) * Scale and subtract the dark image (DARKCORR) -* Perform flatfielding (FLATCORR) -* Perform unit conversion (FLATCORR) +* Perform flatfielding and unit conversion (FLATCORR) * Perform shutter-shading correction where necessary, currently skipped (SHADCORR) * Populate photometric header keywords (PHOTCORR) * Calculate image statistics for the header @@ -36,8 +38,13 @@ of certain DN values, so this correction is not needed. The SHADCOR step which for differential exposure time across the detector caused by the amount of time it takes for the shutter to open and close completely is an insignificant effect. This step is always set to OMIT. +If BLEVCORR and BIASCORR are performed, a full-well saturation image (SATUFILE) is applied to +flag affected pixels. This is the updated and preferred method of flagging such pixels. If the +SATUFILE keyword is missing from the FITS header, then the flagging of full-well saturated pixels +is done during the DQICORR step using a single constant value as the threshold. + Correction For Charge Transfer Efficiency (PCTECORR) ----------------------------------------------------- +==================================================== The charge transfer (CTE) of the UVIS detector has been declining over time as on-orbit radiation damage creates charge traps in the CCDs. Faint sources, in particular, can suffer large flux losses or even be lost entirely if observations are not planned and analyzed carefully. The CTE depends on the morphology of the source, the distribution of electrons in the field of view, and the population of charge traps in the detector column between the source and the transfer register. Further details regarding the understanding of the WFC3/UVIS charge transfer efficiency (CTE) are presented in several documents. Please see `WFC3 ISR 2021-13 `_, as well as other ISRs found in the WFC3 documentation `_. The PCTECORR step aims to mitigate the flux loss incurred from CTE. @@ -45,7 +52,7 @@ More information on this part of the pipeline can be found in the :ref:`wf3cte` Error Array Initialization (NOISCORR) -------------------------------------- +===================================== The image error array is initialized. The function examines the ERR extension of the input data to determine the state of the array. The input _raw image contains an empty ERR array. If the ERR array has already been expanded and contains values other than zero, then this function does nothing. Otherwise it will initialize the ERR array by assigning pixel values based on a simple noise model. The noise model uses the science (SCI) array and for each pixel calculates the error value :math:`\sigma` in units of DN: The NOISCORR calibration step keyword is not explicitly listed in the image header (i.e., it is not @@ -59,9 +66,9 @@ The CCDTAB reference file contains the bias, gain and readnoise values used for Data Quality Array Initialization (DQICORR) -------------------------------------------- +=========================================== -This step initializes the data quality array by reading a table of known bad pixels for the detector, as stored in the Bad Pixel reference table BPIXTAB. The types of bad pixels that can be flagged are: +This step initializes the data quality array by reading a table of known bad pixels for the detector, as stored in the Bad Pixel reference table BPIXTAB. In addition to the bad pixel types in the table, the types of bad pixels that can be flagged are: =============== ====== ======================================================= NAME VALUE DESCRIPTION @@ -86,29 +93,10 @@ CROSSTALK 16384 ghost or crosstalk RESERVED2 32768 cannot use =============== ====== ======================================================= -Sink Pixel Detection and Marking --------------------------------- - -Sink pixels are a type of image defect. These pixels contain a number of charge traps and under-report the number of electrons that were generated in them during an exposure. These pixels can have an impact on nearby upstream or downstream pixels. Though they often only impact one or two pixels when the background is high, they can impact up to 10 pixels if the background is low. - -Flagging of SINK pixels in the DQ extension of calibrated images is controlled with the DQICORR header keyword, happens after the bias correction has been performed, and is done in the amp-rotated C-D-A-B full image format used and described in the CTE correction. When set to perform, the sink pixels are located and flagged with help from the SNKCFILE reference image. Given the reference image, the procedure for flagging the sink pixel in science data involves: - -* Extract the MJD of the science exposure -* Go through the reference image pixel by pixel looking for those pixels with values greater than 999, which indicates that the current pixel is a sink pixel. The value of this pixel in the reference file corresponds to the date at which this pixel exhibited the sink behavior. -* If the turn on date of the sink pixel is after the exposure date of the science image, then we ignore the sink pixel in this exposure and move on to the next pixel -* If the turn on date of the sink pixel is before the exposure date of the science image, then this science pixel was compromised at the time of the exposure. The corresponding DQ extension pixel for this science pixel is flagged with the "charge trap" flag of 1024. -* If the pixel "below" the sink pixel in the long format image has a value of -1 in the reference image, then it is also flagged with the "charge trap" value in the DQ extension. We then proceed vertically "up" from the sink pixel and compare each pixel in the reference file to the value of the sink pixel in the science exposure at hand. If the value of the sink pixel in the exposure is below the value of the upstream pixel in the reference image, we flag that pixel with the "charge trap" value in the DQ extension. We continue to flag pixels until the value of the pixel in the reference image is zero or until the value of the sink pixel in the exposure is greater than the value of the upstream pixel in the reference image. - - -`WFC3 ISR 2014-19 `_ -has a detailed analysis on detection of the sink pixels, while the strategy for flagging them is discussed in `WFC3 ISR 2014-22 `_. - -Sink pixels were originally only flagged in full frame science images, but since `calwf3 v3.4` sink pixel flagging has also been done -in subarray images. **The pipeline currently does no further analysis or correction on pixels which have been flagged as affected by sink pixels** - +If the newer (mid-2023) SATUFILE FITS keyword is missing or invalid in the input image header, the full-well saturated pixels are flagged during the DQICORR step using a single value as the threshold. However, the newer technique is to flag the full-well saturated pixels in a sub-step after BLEVCORR/BIASCORR using a full two-dimensional image as the threshold. Overscan Bias Correction (BLEVCORR) ------------------------------------ +=================================== The location of the overscan regions in a raw image varies, depending upon the type of readout that is performed. The overscan regions are used to monitor the instrument as well as provide a measure of the bias level at the time the detector was exposed. The bias level which is calculated for subtraction is done on a line-by-line basis in the image. If the image has no overscan region the BIAS level to be subtracted is obtained from the CCDTAB reference file. Otherwise, the columns to use for the calculation are referenced in the OSCNTAB reference file. The OSCNTAB refers to all regions in pixel coordinates, even when the image is binned. A bias drift calculation is made if there are virtual overscan pixels which exist, if neither of the virtual overscan regions are specified, then the physical overscan region is used. @@ -156,15 +144,36 @@ The mean value of all the bias levels which were subtracted is recorded in the S TRIMY1 - Number of lines to trim off beginning of each column TRIMY2 - Number of line to trim off end of each column - Bias Correction (BIASCORR) --------------------------- +========================== This step subtracts the two dimensional bias structure from the image using the superbias reference image listed in the header keyword BIASFILE. The dimensions of the image are used to distinguish between full-frame and subarray images. Because the bias image is already overscan-subtracted, it will have a mean pixel value of less than one. The BIASFILE has the same dimensions as a full-size science image, complete with overscan regions. Only after completion of :ref:`wf3ccd` are the science images trimmed to their final calibrated size. Since the same reference image is used for full-frame and subarray images, `calwf3` will extract the matching region from the full-size bias file and apply it to the subarray image. +If both the BLEVCORR and BIASCORR steps are performed, and the input image contains a valid FITS SATUFILE keyword in the primary header, then the full-well saturation image identified by the SATUFILE keyword will be usedto define the saturation threshold for flagging at this stage. + +Sink Pixel Detection and Marking +================================ + +Sink pixels are a type of image defect. These pixels contain a number of charge traps and under-report the number of electrons that were generated in them during an exposure. These pixels can have an impact on nearby upstream or downstream pixels. Though they often only impact one or two pixels when the background is high, they can impact up to 10 pixels if the background is low. + +Flagging of SINK pixels in the DQ extension of calibrated images is controlled with the DQICORR header keyword, happens after the bias correction has been performed, and is done in the amp-rotated C-D-A-B full image format used and described in the CTE correction. When set to perform, the sink pixels are located and flagged with help from the SNKCFILE reference image. Given the reference image, the procedure for flagging the sink pixel in science data involves: + +* Extract the MJD of the science exposure +* Go through the reference image pixel by pixel looking for those pixels with values greater than 999, which indicates that the current pixel is a sink pixel. The value of this pixel in the reference file corresponds to the date at which this pixel exhibited the sink behavior. +* If the turn on date of the sink pixel is after the exposure date of the science image, then we ignore the sink pixel in this exposure and move on to the next pixel +* If the turn on date of the sink pixel is before the exposure date of the science image, then this science pixel was compromised at the time of the exposure. The corresponding DQ extension pixel for this science pixel is flagged with the "charge trap" flag of 1024. +* If the pixel "below" the sink pixel in the long format image has a value of -1 in the reference image, then it is also flagged with the "charge trap" value in the DQ extension. We then proceed vertically "up" from the sink pixel and compare each pixel in the reference file to the value of the sink pixel in the science exposure at hand. If the value of the sink pixel in the exposure is below the value of the upstream pixel in the reference image, we flag that pixel with the "charge trap" value in the DQ extension. We continue to flag pixels until the value of the pixel in the reference image is zero or until the value of the sink pixel in the exposure is greater than the value of the upstream pixel in the reference image. + + +`WFC3 ISR 2014-19 `_ +has a detailed analysis on detection of the sink pixels, while the strategy for flagging them is discussed in `WFC3 ISR 2014-22 `_. + +Sink pixels were originally only flagged in full frame science images, but since `calwf3 v3.4` sink pixel flagging has also been done +in subarray images. **The pipeline currently does no further analysis or correction on pixels which have been flagged as affected by sink pixels.** + Post-Flash Correction (UVIS ONLY) (FLSHCORR) --------------------------------------------- +============================================ WFC3 has post-flash capability to provide a means of mitigating the effects of Charge Transfer Efficiency (CTE) degradation. When FLSHCORR=PERFORM, this routine subtracts the post-flash reference image, FLSHFILE, from the science image step. The success of the post-flash operation during the exposure is first verified by checking the keyword FLASHSTA. The FLSHFILE is renormalized to the appropriate post-flash current level (LOW, MED, HIGH) recorded in the FLASHCUR keyword, and the flash duration (FLASHDUR) is then subtracted from the science image. The mean value of the scaled post-flash image is written to MEANFLSH in the output SCI extension header. Different members of an association can have different values of SHUTRPOS because it varies by exposure, and this is fine for calibration because the references files are populated separately for each exposure. @@ -187,13 +196,13 @@ Further reading: Dark Current Subtraction (DARKCORR) ------------------------------------ +=================================== The reference file listed under the DARKFILE header keyword is used as the reference dark image. In the UVIS, the dark image is scaled by EXPTIME and FLASHDUR. The reference file pointed to with DARKFILE is used for the non-CTE corrected data, while the reference file pointed to with DRKCFILE is used for the CTE corrected data. FLATCORR --------- +======== Correct the image for pixel quantum efficiency using the reference image specified by the FLATFILE keyword in the header. This actually consists of correction using up to 3 reference flat images: @@ -206,15 +215,16 @@ The pipeline is currently only using the P-flats. If two or more reference files Subarray science images use the same reference file as the full-frame images; `calwf3` will extract the appropriate region from the reference file and apply it to the subarray input image. + Unit Conversion to Electrons ----------------------------- -Conversion from DN to ELECTRONS no longer depends on FLATCORR being set to PERFORM as all images are converted appropriately. The actual unit conversion is formally among one of the last steps performed in FLATCORR. +============================ +The calibration reference flat image is divided by the calibrated gain value, and then the science image is divided by the flat. Any calibration reference file data which is in units of electrons and is used in `calwf3` prior to the unit conversion step, has the gain applied before use to ensure the calibration data and the input data are in consistent units. Shutter Shading Correction (SHADCORR) -------------------------------------- +===================================== This step corrects the science image for differential exposure time across the detector cased by the amount of time it takes for the shutter to completely open and close, which is a potentially significant effect only for images with very short exposure times (less than ~5 seconds). Pixels are corrected based on the exposure time using the relation: @@ -227,7 +237,7 @@ WFC3 tests have shown that the shutter shading effect is insignificant (< 1%), e Photometry Keywords (PHOTCORR) ------------------------------- +============================== The PHOTCORR step is performed using tables of precomputed values. Instead of calls to SYNPHOT, it uses the reference table specified in the IMPHTTAB header keyword. Each DETECTOR uses a different table. @@ -270,7 +280,7 @@ where each extension contains the photometry keyword information for that specif Flux normalization for UVIS1 and UVIS2 (FLUXCORR) -------------------------------------------------- +================================================= The FLUXCORR step was added in `calwf3 v3.2` as a way to scale the UVIS chips so that the flux correction over both chips is uniform. This required new keywords which specify new PHOTFLAM values to use for each chip as well as a keyword to specify the scaling factor @@ -294,5 +304,5 @@ This step is performed by default in the pipeline and the PHOTFLAM keyword will Cosmic-ray rejection --------------------- +==================== Associations with more than one member, which have been associated using either CR-SPLIT or REPEAT-OBS, will be combined using :ref:`wf3rej`. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrj) and WFPC2(crrej), providing a well-tested and robust procedure. The DRZ and DRC products will be created from the association, with `AstroDrizzle `_, which will also correct for geometric distortion. diff --git a/docs/wfc3tools/wf32d.rst b/docs/wfc3tools/wf32d.rst index 4e84026..87351eb 100644 --- a/docs/wfc3tools/wf32d.rst +++ b/docs/wfc3tools/wf32d.rst @@ -11,7 +11,8 @@ This routine performs the remaining series of tasks in the UVIS pipeline. The wf32d primary functions include: * DARKCORR: dark current subtraction - * FLATCORR: flat-fielding + * FLATCORR: flat-fielding and conversion to electrons + * SHADCORR: apply shutter shading correction (currently skipped) * PHOTCORR: photometric keyword calculations * FLUXCORR: photometric normalization of the UVIS1 and UVIS2 chips @@ -20,8 +21,20 @@ executed, after which the switch will be set to COMPLETE in the corresponding output files. See `Section 3.4.3 of the WFC3 Data Handbook `_ for more information. +Running `wf32d` from a Python Terminal +======================================= + +.. code-block:: shell + + from wfc3tools import wf32d + wf32d(filename) + + +Input Parameters for the Python Interface +----------------------------------------- + Parameters -========== +~~~~~~~~~~ input : str or list Name of input files, such as @@ -69,13 +82,13 @@ Parameters Returns -======= +~~~~~~~ None Usage -===== +~~~~~ .. code-block:: python @@ -83,8 +96,8 @@ Usage wf32d(filename) -Command Line Options for the wf32d executable -============================================= +Command Line Options for the `wf32d` C Executable +================================================= .. code-block:: shell diff --git a/docs/wfc3tools/wf3ccd.rst b/docs/wfc3tools/wf3ccd.rst index 3a51a54..710ad47 100644 --- a/docs/wfc3tools/wf3ccd.rst +++ b/docs/wfc3tools/wf3ccd.rst @@ -6,12 +6,17 @@ wf3ccd This routine contains the initial processing steps for all the WFC3 UVIS channel data. These steps are: - * DQICORR - initializing the data quality array - * ATODCORR - perform the a to d conversion correction + * DQICORR - initialize the data quality array with values from BPIXTAB, flag for A-to-D + saturation, and potentially flag for full-well saturation using scalar value as + the threshold (fall-back algorithm) + * ATODCORR - perform the a-to-d conversion correction * BLEVCORR - subtract the bias level from the overscan region * BIASCORR - subtract the bias image + * Flag for full-well saturation using a two-dimensional image (new algorithm) + * Detect and record SINK pixels in the DQ mask (performed if DQICORR is set to PERFORM) * FLSHCORR - subtract the post-flash image +The `wf3ccd` processes everything in counts. If a calibration reference file is in units of electrons when used during the wf3ccd processing, the calibration data are divided by the gain before use. The conversion to electrons happens in the wf32d component. `wf3ccd` first subtracts the bias and trims the overscan regions from the image. If an associated set of UVIS CR-SPLIT or REPEAT-OBS images is being processed, all of the overscan-trimmed images are sent through `wf3rej` to be combined and receive cosmic-ray rejection. The resulting combined image then receives final calibration with `wf32d`, @@ -26,16 +31,28 @@ executed, after which the switch will be set to COMPLETE in the corresponding output files. See `Section 3.4.2 of the WFC3 Data Handbook `_ for more information. -Displaying output from wf3ccd in a Jupyter Notebook -=================================================== +Running `wf3ccd` from a Python Terminal +======================================= + +.. code-block:: shell + + from wfc3tools import wf3ccd + wf3ccd(filename) + + +Displaying output from `wf3ccd` in a Jupyter Notebook +----------------------------------------------------- When calling `wf3ccd` from a Jupyter notebook, informational text output from the underlying `wf3ccd.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ccd`. As output is read from the underlying program, the `wf3ccd` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3ccd` to the Python logging system by passing the `logging.debug` function or similar.) If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. +Input Parameters for the Python Interface +----------------------------------------- + Parameters -========== +~~~~~~~~~~ input : str or list Name of input files, such as @@ -50,8 +67,10 @@ Parameters Name of the output FITS file. dqicorr : str, optional, default="PERFORM" - Update the dq array from bad pixel table. Allowed values are "PERFORM" - and "OMIT". + Update the dq array from bad pixel table, as well as flag the A-to-D saturation. + If the comparatively new FITS keyword (mid-2023) SATUFILE is missing or not + populated in the input file, the full-well saturation will also be flagged using + a single value as the threshold. Allowed values are "PERFORM" and "OMIT". atodcorr : str, optional, default="PERFORM" Analog to digital correction. Allowed values are "PERFORM" and "OMIT". @@ -63,6 +82,14 @@ Parameters biascorr : str, optional, default="PERFORM" Subtract bias image. Allowed values are "PERFORM" and "OMIT". + NOTE: Strictly speaking, the application of the full-well saturation *image* is + not a calibration step (i.e., there is no SATCORR), but the application + of a 2D image to flag pixels versus using a single scalar value to flag + saturated pixels as previously done in DQICORR will be done in doFullWellSat() + after BLEVCORR and BIASCORR. This correction should only be done if both + BLEVCORR and BIASCORR have been performed. This flagging is only applicable + for the UVIS. + flashcorr : str, optional, default="PERFORM" Subtract post-flash image. Allowed values are "PERFORM" and "OMIT". @@ -78,13 +105,13 @@ Parameters Returns -======= +~~~~~~~ None Usage -===== +~~~~~ .. code-block:: python @@ -92,8 +119,8 @@ Usage wf3ccd(filename) -Command Line Options for the wf3ccd executable -============================================== +Command Line Options for the `wf3ccd` executable +================================================ .. code-block:: shell @@ -102,9 +129,9 @@ Command Line Options for the wf3ccd executable Input may be a single filename, and the options include: * -v: verbose -* t: print time stamps +* -t: print time stamps * -dqi: udpate the DQ array -* -atod: perform gain correction +* -atod: perform a-to=d gain correction * -blev: subtract bias from overscan * -bias: perform bias correction * -flash: remove post-flash image diff --git a/docs/wfc3tools/wf3cte.rst b/docs/wfc3tools/wf3cte.rst index 7be7c57..40c0a53 100644 --- a/docs/wfc3tools/wf3cte.rst +++ b/docs/wfc3tools/wf3cte.rst @@ -84,16 +84,28 @@ The standalone call of `wf3cte` will produce a RAC fits (\*_rac_tmp.fits) file b For more information please see `the WFC3 CTE webpage `_ and `Section 3.4.1 of the WFC3 Data Handbook `_. +Running `wf3cte` from a Python Terminal +======================================= + +.. code-block:: shell + + from wfc3tools import wf3cte + wf3cte(filename) + + Displaying output from `wf3cte` in a Jupyter Notebook -===================================================== +----------------------------------------------------- When calling `wf3cte` from a Jupyter notebook, informational text output from the underlying `wf3cte.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3cte`. As output is read from the underlying program, the `wf3cte` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3cte` to the Python logging system by passing the `logging.debug` function or similar.) If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. +Input Parameters for the Python Interface +----------------------------------------- + Parameters -========== +~~~~~~~~~~ input : str or list Name of input files, such as @@ -116,13 +128,13 @@ Parameters Returns -======= +~~~~~~~ None Usage -===== +~~~~~ .. code-block:: python diff --git a/docs/wfc3tools/wf3ir.rst b/docs/wfc3tools/wf3ir.rst index 2c6455c..af9ce53 100644 --- a/docs/wfc3tools/wf3ir.rst +++ b/docs/wfc3tools/wf3ir.rst @@ -30,16 +30,28 @@ will be set to COMPLETE in the corresponding output files. See `Section 3.4.4 of the WFC3 Data Handbook `_ for more information. -Displaying output from wf3ir in a Jupyter Notebook -================================================== +Running `wf3ir` from a Python Terminal +======================================= + +.. code-block:: shell + + from wfc3tools import wf3ir + wf3ir(filename) + + +Displaying output from `wf3ir` in a Jupyter Notebook +---------------------------------------------------- When calling `wf3ir` from a Jupyter notebook, informational text output from the underlying `wf3ir.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ir`. As output is read from the underlying program, the `wf3ir` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3ir` to the Python logging system by passing the `logging.debug` function or similar.) If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. +Input Parameters for the Python Interface +----------------------------------------- + Parameters -========== +~~~~~~~~~~ input : str Name of input files, such as @@ -65,13 +77,13 @@ Parameters Returns -======= +~~~~~~~ None Usage -===== +~~~~~ .. code-block:: python @@ -79,8 +91,8 @@ Usage wf3ir(filename) -Command Line Options for the wf3ir executable -============================================= +Command Line Options for the `wf3ir` C Executable +================================================= .. code-block:: shell diff --git a/docs/wfc3tools/wf3rej.rst b/docs/wfc3tools/wf3rej.rst index 00932d8..54f0590 100644 --- a/docs/wfc3tools/wf3rej.rst +++ b/docs/wfc3tools/wf3rej.rst @@ -40,16 +40,27 @@ as specified by CRSIGMAS. See `Section 3.4.5 of the WFC3 Data Handbook for more information. +Running `wf3rej` from a Python Terminal +======================================= + +.. code-block:: shell + + from wfc3tools import wf3rej + wf3rej([filename1, filename2]) + + Displaying Output from `wf3rej` in a Jupyter Notebook -===================================================== +----------------------------------------------------- When calling `wf3rej` from a Jupyter notebook, informational text output from the underlying `wf3rej.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3rej`. As output is read from the underlying program, the `wf3rej` Python wrapper will call `log_func` with the contents of each line. Note that `print` is an obvious choice for a log function, but this also provides a way to connect `wf3rej` to the Python logging system by passing the logging.debug function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. +Input Parameters for the Python Interface +----------------------------------------- Parameters -========== +~~~~~~~~~~ input : str or list Name of input files, such as @@ -99,13 +110,13 @@ Parameters Returns -======= +~~~~~~~ None Usage -===== +~~~~~ .. code-block:: python From 3716901cf993a82c05e6f609f363f408a2cac807 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 14:35:14 -0400 Subject: [PATCH 02/10] Updated warning message at the beginning of the history document. --- docs/wfc3tools/history.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wfc3tools/history.rst b/docs/wfc3tools/history.rst index 5811df4..223b94a 100644 --- a/docs/wfc3tools/history.rst +++ b/docs/wfc3tools/history.rst @@ -4,7 +4,7 @@ Software Update History for HSTCAL.CALWF3 ***************************************** -.. warning:: IRAF version of WFC3 no longer maintained or delivered, use WFC3TOOLS in HSTCAL or call the executable from your operating system command line. With version 3.3 the pipeline now produces two versions of each calibrated file, one set with the CTE correction applied and one set without the CTE correction applied +.. warning:: The IRAF version of WFC3 is no longer maintained or delivered; use WFC3TOOLS in HSTCAL or call the executable, calwf3.e, from your operating system command line. With version 3.3, the pipeline now produces two versions of each calibrated file: one set with the CTE correction applied and one set without the CTE correction applied. **Updates for Version 3.7.0 08-Jun-2023 - MDD** - Implemented a change to use an image to detect and flag full-well saturation versus a simple scalar. From 5c38b4676063cb3a56834e8a1682ccd331522848 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 14:48:52 -0400 Subject: [PATCH 03/10] Typos addressed in ir_pipeline.inc. --- docs/wfc3tools/ir_pipeline.inc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/wfc3tools/ir_pipeline.inc b/docs/wfc3tools/ir_pipeline.inc index 6ca5e2f..3323b40 100644 --- a/docs/wfc3tools/ir_pipeline.inc +++ b/docs/wfc3tools/ir_pipeline.inc @@ -65,12 +65,12 @@ This step measures the signal between the super zero read in the linearity refer * low signal pixels are masked out by setting them to zero * the NLINCOR file has an extension with saturation values for each pixel which is referenced here. Pixels which are saturated in the zeroth or first reads are flagged in the DQ and the number of found saturated pixels are reported. * This step works poorly for bright targets which are already begining to saturate in the zeroth and first reads -* This step acutally subtractes the super zero read from the science zero read instead of calculating an estimated signal based on the first read and zero read + estimated exposure time between them so that the difference in readout time for subarrays is not an issue. +* This step acutally subtracts the super zero read from the science zero read instead of calculating an estimated signal based on the first read and zero read + estimated exposure time between them so that the difference in readout time for subarrays is not an issue. Bias Correction (BLEVCORR) ========================== -This step subtracts the bias level using the reference pixels around the perimeter of the detector, the boundries fo the reference pixels are defined in the OSCNTAB reference file. There are 5 reference pixels on each end of each row, but 1 is ignored on each side, for a total of 8 being used per row. The resistent mean of the standard deviation of all the reference pixels in the image is subtracted from the entire image and the value is stored in the MEANBLEV keyword in the output image header. The reference pixels are left in place in the IMA output image through processeing, but the final FLT image has been trimmed to just the science pixels. +This step subtracts the bias level using the reference pixels around the perimeter of the detector, the boundries of the reference pixels are defined in the OSCNTAB reference file. There are 5 reference pixels on each end of each row, but 1 is ignored on each side, for a total of 8 being used per row. The resistent mean of the standard deviation of all the reference pixels in the image is subtracted from the entire image and the value is stored in the MEANBLEV keyword in the output image header. The reference pixels are left in place in the IMA output image through processing, but the final FLT image has been trimmed to just the science pixels. Zero read subtraction (ZOFFCORR) @@ -82,7 +82,7 @@ The original zero read is subtracted from all groups in the science image, inclu Error array initialization ========================== -The errors associated with the raw data are estimated according to the noise model for the detector which currently includes a simple combination of detector readnoise and poisson noise from the pixel. Readnoise and gain are read from the CCDTAB reference file. The ERR array continues to be summed in quadrature as the SCI array is processed. Inside the final FLT image, the ERR array is calculated by CRCORR as the calculated uncertainty of the count-rate fit to the multiaccum samples. +The errors associated with the raw data are estimated according to the noise model for the detector which currently includes a simple combination of detector readnoise and Poisson noise from the pixel. Readnoise and gain are read from the CCDTAB reference file. The ERR array continues to be summed in quadrature as the SCI array is processed. Inside the final FLT image, the ERR array is calculated by CRCORR as the calculated uncertainty of the count-rate fit to the multiaccum samples. .. math:: @@ -92,7 +92,7 @@ The errors associated with the raw data are estimated according to the noise mo Detector Non-linearity Correction (NLINCORR) ============================================ -The integrated counts in the science images are corrected for the non-linear response of the detectors, flagging pixels which extend into saturation (as defined in the saturation extension of the NLINFILE reference image. The observed response of the detector can be represented by two regimes: +The integrated counts in the science images are corrected for the non-linear response of the detectors, flagging pixels which extend into saturation (as defined in the saturation extension of the NLINFILE reference image). The observed response of the detector can be represented by two regimes: * At low and intermediate signal levels the detector response deviates from the incident flux in a way that is correctable using the following expression @@ -138,14 +138,14 @@ The format of the linearity reference file: Dark Current Subtraction (DARKCORR) =================================== -The reference file listed under the DARKFILE header keyword is used to subtract the dark current from each sample. Due to potential non-linearities in some of the signal components, such as reet-related effecets in the first one or two reads of an exposure, the dark current subtraction is not aplied by simply scaling a generic reference dark image to the exposure time and then subtracting it. Instead, a library of dark current images is maintained that includes darks taken in each of the available predefined multiaccum sample sequences, as well as the available sub-array readout modes. The multiaccum dark reference file is subtracted read-by-read from the stack of science image readouts so that there is an exact match in the timings and other characteristics of the dark image and the science image. The subtraction does not include the reference pixel. The ERR and DQ arrays from the reference dark file are combined with the SCI and DQ arrays from the science image, but the SAMP and TIME arrays are unchanged. The mean of the dark image is saved to the MEANDARK keyword in the output science image header. +The reference file listed under the DARKFILE header keyword is used to subtract the dark current from each sample. Due to potential non-linearities in some of the signal components, such as reet-related effects in the first one or two reads of an exposure, the dark current subtraction is not applied by simply scaling a generic reference dark image to the exposure time and then subtracting it. Instead, a library of dark current images is maintained that includes darks taken in each of the available predefined multiaccum sample sequences, as well as the available sub-array readout modes. The multiaccum dark reference file is subtracted read-by-read from the stack of science image readouts so that there is an exact match in the timings and other characteristics of the dark image and the science image. The subtraction does not include the reference pixel. The ERR and DQ arrays from the reference dark file are combined with the SCI and DQ arrays from the science image, but the SAMP and TIME arrays are unchanged. The mean of the dark image is saved to the MEANDARK keyword in the output science image header. Photometry Keywords (PHOTCORR) ============================== -The PHOTCORR step is performed using tables of precomputed values instead of calls to SYNPHOT. The correct table for a given image must be specified in the IMPHTTAB header keyword in order for calwf3 to perform the PHOTCORR step. The format of the file for the IR detectors is: +The PHOTCORR step is performed using tables of precomputed values instead of calls to SYNPHOT. The correct table for a given image must be specified in the IMPHTTAB header keyword in order for ``calwf3`` to perform the PHOTCORR step. The format of the file for the IR detectors is: :: @@ -209,7 +209,7 @@ The result of this step is stored as a single imset in the output FLT file. In Flatfield Correction (FLATCORR) =============================== -This step corrects for sensativity variations across the detector by dividing the images by one or more reference flatfields (taken from the PFLTFILE, DFLTFILE or LFLTFILE header keywords). The mean gain from all the amps is used to convert to the image to units of electrons. Errors and DQ flags from the flatfields are combined with the science data errors and flag, the TIME and SAMP arrays are unchanged. +This step corrects for sensitivity variations across the detector by dividing the images by one or more reference flatfields (taken from the PFLTFILE, DFLTFILE or LFLTFILE header keywords). The mean gain from all the amps are used to convert to the image to units of electrons. Errors and DQ flags from the flatfields are combined with the science data errors and flag, the TIME and SAMP arrays are unchanged. Calculation of image statistics From 8d22d9f0b7121f331aa57ff6327ab332a5f2dc2e Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 14:52:48 -0400 Subject: [PATCH 04/10] Fixed quoting of calwf3. --- docs/wfc3tools/ir_pipeline.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wfc3tools/ir_pipeline.inc b/docs/wfc3tools/ir_pipeline.inc index 3323b40..de8e438 100644 --- a/docs/wfc3tools/ir_pipeline.inc +++ b/docs/wfc3tools/ir_pipeline.inc @@ -145,7 +145,7 @@ The reference file listed under the DARKFILE header keyword is used to subtract Photometry Keywords (PHOTCORR) ============================== -The PHOTCORR step is performed using tables of precomputed values instead of calls to SYNPHOT. The correct table for a given image must be specified in the IMPHTTAB header keyword in order for ``calwf3`` to perform the PHOTCORR step. The format of the file for the IR detectors is: +The PHOTCORR step is performed using tables of precomputed values instead of calls to SYNPHOT. The correct table for a given image must be specified in the IMPHTTAB header keyword in order for `calwf3` to perform the PHOTCORR step. The format of the file for the IR detectors is: :: From 70c621bb6b967f17303077d37b02cfdfdc16f715 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 15:04:28 -0400 Subject: [PATCH 05/10] Removed use of parentheses. --- docs/wfc3tools/wf3cte.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wfc3tools/wf3cte.rst b/docs/wfc3tools/wf3cte.rst index 40c0a53..db0eebd 100644 --- a/docs/wfc3tools/wf3cte.rst +++ b/docs/wfc3tools/wf3cte.rst @@ -96,7 +96,7 @@ Running `wf3cte` from a Python Terminal Displaying output from `wf3cte` in a Jupyter Notebook ----------------------------------------------------- -When calling `wf3cte` from a Jupyter notebook, informational text output from the underlying `wf3cte.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3cte`. As output is read from the underlying program, the `wf3cte` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3cte` to the Python logging system by passing the `logging.debug` function or similar.) +When calling `wf3cte` from a Jupyter notebook, informational text output from the underlying `wf3cte.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3cte`. As output is read from the underlying program, the `wf3cte` Python wrapper will call `log_func` with the contents of each line. `print` is an obvious choice for a log function, but this also provides a way to connect `wf3cte` to the Python logging system by passing the `logging.debug` function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. From ff8afdcfc87a887dbc36ebf094e3696ce1bc20a2 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 15:06:57 -0400 Subject: [PATCH 06/10] Fixed typos found during review. --- docs/wfc3tools/wf3ccd.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/wfc3tools/wf3ccd.rst b/docs/wfc3tools/wf3ccd.rst index 710ad47..9512ffb 100644 --- a/docs/wfc3tools/wf3ccd.rst +++ b/docs/wfc3tools/wf3ccd.rst @@ -16,7 +16,7 @@ This routine contains the initial processing steps for all the WFC3 UVIS channel * Detect and record SINK pixels in the DQ mask (performed if DQICORR is set to PERFORM) * FLSHCORR - subtract the post-flash image -The `wf3ccd` processes everything in counts. If a calibration reference file is in units of electrons when used during the wf3ccd processing, the calibration data are divided by the gain before use. The conversion to electrons happens in the wf32d component. +The `wf3ccd` processes everything in counts. If a calibration reference file is in units of electrons when used during the `wf3ccd` processing, the calibration data are divided by the gain before use. The conversion to electrons happens in the `wf32d` component. `wf3ccd` first subtracts the bias and trims the overscan regions from the image. If an associated set of UVIS CR-SPLIT or REPEAT-OBS images is being processed, all of the overscan-trimmed images are sent through `wf3rej` to be combined and receive cosmic-ray rejection. The resulting combined image then receives final calibration with `wf32d`, @@ -43,7 +43,7 @@ Running `wf3ccd` from a Python Terminal Displaying output from `wf3ccd` in a Jupyter Notebook ----------------------------------------------------- -When calling `wf3ccd` from a Jupyter notebook, informational text output from the underlying `wf3ccd.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ccd`. As output is read from the underlying program, the `wf3ccd` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3ccd` to the Python logging system by passing the `logging.debug` function or similar.) +When calling `wf3ccd` from a Jupyter notebook, informational text output from the underlying `wf3ccd.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ccd`. As output is read from the underlying program, the `wf3ccd` Python wrapper will call `log_func` with the contents of each line. `print` is an obvious choice for a log function, but this also provides a way to connect `wf3ccd` to the Python logging system by passing the `logging.debug` function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. @@ -131,7 +131,7 @@ Input may be a single filename, and the options include: * -v: verbose * -t: print time stamps * -dqi: udpate the DQ array -* -atod: perform a-to=d gain correction +* -atod: perform a-to-d gain correction * -blev: subtract bias from overscan * -bias: perform bias correction * -flash: remove post-flash image From fe40c3aa42722476f5b492fe020b9a7b5b66c5ff Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 15:32:19 -0400 Subject: [PATCH 07/10] Added a section for displaying output in a Jupyter notebook. Same text as seems to be duplicated among the multiple modules. --- docs/wfc3tools/wf32d.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/wfc3tools/wf32d.rst b/docs/wfc3tools/wf32d.rst index 87351eb..8526b62 100644 --- a/docs/wfc3tools/wf32d.rst +++ b/docs/wfc3tools/wf32d.rst @@ -29,6 +29,13 @@ Running `wf32d` from a Python Terminal from wfc3tools import wf32d wf32d(filename) +Displaying output from `wf32d` in a Jupyter Notebook +----------------------------------------------------- + +When calling `wf32d` from a Jupyter notebook, informational text output from the underlying `wf32d.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf32d`. As output is read from the underlying program, the `wf32d` Python wrapper will call `log_func` with the contents of each line. `print` is an obvious choice for a log function, but this also provides a way to connect `wf32d` to the Python logging system by passing the `logging.debug` function or similar. + +If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. + Input Parameters for the Python Interface ----------------------------------------- From 87f46c412f891dbd6e4601dc5d8d9671a11f35c6 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 15:34:43 -0400 Subject: [PATCH 08/10] Removed parentheses. --- docs/wfc3tools/wf3ir.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wfc3tools/wf3ir.rst b/docs/wfc3tools/wf3ir.rst index af9ce53..885ba53 100644 --- a/docs/wfc3tools/wf3ir.rst +++ b/docs/wfc3tools/wf3ir.rst @@ -42,7 +42,7 @@ Running `wf3ir` from a Python Terminal Displaying output from `wf3ir` in a Jupyter Notebook ---------------------------------------------------- -When calling `wf3ir` from a Jupyter notebook, informational text output from the underlying `wf3ir.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ir`. As output is read from the underlying program, the `wf3ir` Python wrapper will call `log_func` with the contents of each line. (`print` is an obvious choice for a log function, but this also provides a way to connect `wf3ir` to the Python logging system by passing the `logging.debug` function or similar.) +When calling `wf3ir` from a Jupyter notebook, informational text output from the underlying `wf3ir.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3ir`. As output is read from the underlying program, the `wf3ir` Python wrapper will call `log_func` with the contents of each line. `print` is an obvious choice for a log function, but this also provides a way to connect `wf3ir` to the Python logging system by passing the `logging.debug` function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. From cbb1476ac74227e14b8e581994812a65daf97ee9 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 17:01:02 -0400 Subject: [PATCH 09/10] Improved description when input is a string. --- docs/wfc3tools/wf3rej.rst | 5 +++-- wfc3tools/wf3rej.py | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/wfc3tools/wf3rej.rst b/docs/wfc3tools/wf3rej.rst index 54f0590..a825a16 100644 --- a/docs/wfc3tools/wf3rej.rst +++ b/docs/wfc3tools/wf3rej.rst @@ -52,7 +52,7 @@ Running `wf3rej` from a Python Terminal Displaying Output from `wf3rej` in a Jupyter Notebook ----------------------------------------------------- -When calling `wf3rej` from a Jupyter notebook, informational text output from the underlying `wf3rej.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3rej`. As output is read from the underlying program, the `wf3rej` Python wrapper will call `log_func` with the contents of each line. Note that `print` is an obvious choice for a log function, but this also provides a way to connect `wf3rej` to the Python logging system by passing the logging.debug function or similar. +When calling `wf3rej` from a Jupyter notebook, informational text output from the underlying `wf3rej.e` program will be passed through `print` as the calibration runs by default, and show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `wf3rej`. As output is read from the underlying program, the `wf3rej` Python wrapper will call `log_func` with the contents of each line. Note that `print` is an obvious choice for a log function, but this also provides a way to connect `wf3rej` to the Python logging system by passing the `logging.debug` function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. @@ -64,6 +64,7 @@ Parameters input : str or list Name of input files, such as + * common separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) * a Python list of filenames * a partial filename with wildcards (``*flt.fits``) * an at-file (``@input``) @@ -142,7 +143,7 @@ Command Line Options for the `wf3rej` C Executable Example - Process data with timestamps and a custom cosmic ray rejection table: - wf3rej.e ibfma4jqq_flt.fits,ibfma4jtq_flt.fits output.fits -t -table mycrejtab.fits + wf3rej.e iaao01k8q_flc.fits,iaao01k9q_flc.fits output.fits -t Example - Print the code version and exit: wf3rej.e -r diff --git a/wfc3tools/wf3rej.py b/wfc3tools/wf3rej.py index 203bf51..9d7380b 100644 --- a/wfc3tools/wf3rej.py +++ b/wfc3tools/wf3rej.py @@ -13,7 +13,7 @@ $ wf3rej.e input output [-options] Input can be a comma-delimited list of files, and an output file must be specified. - $ wf3rej.e ibfma4jqq_flt.fits,ibfma4jtq_flt.fits output.fits -t -table mycrejtab.fits + $ wf3rej.e iaao01k8q_flc.fits,iaao01k9q_flc.fits output.fits -t Where the C executable options include: @@ -62,6 +62,7 @@ def wf3rej(input, output, crrejtab="", scalense=0., initgues="", ---------- input : str or list Name of input files, such as + - common separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) - a Python list of filenames - a partial filename with wildcards (``*flt.fits``) - an at-file (``@input``) From 333a79e16ed7c20ff1605bc5dc5799b8cfd36102 Mon Sep 17 00:00:00 2001 From: Michele De La Pena Date: Wed, 16 Aug 2023 17:48:39 -0400 Subject: [PATCH 10/10] Fixed typo. --- docs/wfc3tools/wf3rej.rst | 2 +- wfc3tools/wf3rej.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/wfc3tools/wf3rej.rst b/docs/wfc3tools/wf3rej.rst index a825a16..d804c93 100644 --- a/docs/wfc3tools/wf3rej.rst +++ b/docs/wfc3tools/wf3rej.rst @@ -64,7 +64,7 @@ Parameters input : str or list Name of input files, such as - * common separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) + * comma-separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) * a Python list of filenames * a partial filename with wildcards (``*flt.fits``) * an at-file (``@input``) diff --git a/wfc3tools/wf3rej.py b/wfc3tools/wf3rej.py index 9d7380b..54393de 100644 --- a/wfc3tools/wf3rej.py +++ b/wfc3tools/wf3rej.py @@ -62,7 +62,7 @@ def wf3rej(input, output, crrejtab="", scalense=0., initgues="", ---------- input : str or list Name of input files, such as - - common separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) + - comma-separated (no spaces) filenames (``iaao01k8q_flc.fits,iaao01k9q_flc.fits``) - a Python list of filenames - a partial filename with wildcards (``*flt.fits``) - an at-file (``@input``)