In this example, I will refer to a fictitious night of data which, in order of acquisition, looks like this.
1) from PC, ftp all frames to your account on Sun, and convert to imh format.
2) put Exptime, Darktime, ST, Airmass, HJD into headers.
3) prepare Zero frame.
4) copy correct temperature Dark to current directory.
5) prepare Flat for each filter used.
6) for each observation set, determine overscan from local bias and subtract off star images.
7) for all star observations, apply Zero, Dark, and Flats.
8) notify team members where images can be found.
9) ACLa will backup all processed and raw images to tape.
Updated 1999 July 08
a FYI, ACL = Dr. Andy C. Layden (a.k.a. YFL, Your Fearless Leader)
OVERVIEW: In this example, we will process the data in a directory called /usr/Kracken2/layden/BGTEL/99JUN03. I recommend you store the unprocessed *.fit images in a subdirectory called RAW - that way if something goes wrong with processing the images, you can easily recover the originals.
1) on the Sun, create the directories you will need, e.g.:
> cd /usr/Kracken2/layden/BGTEL
> mkdir 99JUN03
> cd 99JUN03
> mkdir RAW
2) from PC, enter FTP... (full instructions to go here)
3) copy the *.fit images, changing the format from fits to imh. First, create a list of all the raw images, called 'f2i.cl'. Then edit this file into an IRAF script (a list of IRAF commands which can be run in a batch) which reads a *.fits file, adds 0 to it, and writes it out as an *.imh file in integer format in the current directory (99JUN03). Here are some handy emacs keystrokes that will simplify the editing. Click on the link to see the finished script for our example. Finally, run the IRAF script.
> files RAW/*.fit >> f2i.cl
> emacs f2i.cl
> cl < f2i.cl
OVERVIEW: The image header contains many "keywords" and their values that are required for image processing and analysis (try "imhead yourfile.imh l+"). In particular, ccdproc needs the keywords "EXPTIME" and "DARKTIME" to do the dark current corrections, but CCDSoft writes the exposure time into the keyword "EXPOSURE" - we will have to copy the value from EXPOSURE into the EXPTIME and DARKTIME. Also, we will express the time of our variable star observations in "Heliocentric Julian Date" (HJD), the number of days since some semi-arbitrary date about 2.5 million days ago. IRAF needs to know the Universal Time (UT), date (DATE-OBS), exposure time (EXPTIME), and Sidereal Time (ST) of the observation in order to compute HJD. We accomplish all this using a task written by Nick Pearson:
FILL IN!
OVERVIEW: The 8 or-so bias frames taken at the beginning a/o end of the night will be overscan subtracted (based on median counts), and then combined into a master frame, Zero.imh, which contains a 2-D map of the intensity-structure in the bias. The median value of Zero.imh should be 0. For an individual star frame, the median bias level will be determined from the single bias frame taken soon afterward, and the 2D bias structure will be removed using Zero.
1) make a list of the bias frames taken at the beginning a/o end of the night. Edit out any of the 'local biases' taken at the end of each observing set (if you called them, e.g., SW_And1z1.imh, they should not appear).
> files *bias*.imh >> bias1.cl
> emacs bias1.cl
2) determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the output file in our example.
> imstat @bias1.cl fields="image,midpt" lower=930 upper=1060 >> bias1.median
3) type out the file bias1.median and compute the mean of the individual median values. Write this number down as it will resurface in Preparing the Flat Field Images.
4) create an IRAF script to subtract these values from the images from which they were derived. Thus, the median of each image will go from ~975 to 0. Again, these emacs keystrokes will be helpful in the editing. The link shows what the final script looks like.
> emacs bias1.cl
5) open the following packages in iraf. The link shows the suggested parameters for the ccdred task.
> noao.imred.ccdred
> epar ccdred
6) combine these overscan-subtracted biases into a high signal-to-noise map of the 2-D structure produced when the chip is read out. The final zero frame is called Zero.imh.
> epar zerocombine
> zerocombine
7) display Zero.imh to be sure it looks reasonable. Our zeros are usually pretty uniform in the upper-right, while a horizontal "ridge" exists at Y< 70 (5-10 counts higher than the upper-right) and a vertical "trough" exists at X<120 (5-10 counts lower than the upper-right). Discuss your Zero.imh with ACL if there are any questions.
> display Zero.imh 1
OVERVIEW: Every time we take an image, whether there is light falling on the chip or not, a small number of counts appear in each pixel due to "dark current" in the chip. The amount of dark current is proportional to the exposure time (i.e., it appears at a constant rate, counts/sec). The amount of dark current varies from pixel to pixel, and tends to be larger when the chip is operated at a higher temperature. To correct for the dark current, we have created a number of "dark frames" &endash; high signal-to-noise images recording the counts in each pixel &endash; each at a different temperature. In this step, we select the dark frame with the temperature closest to the temperature setpoint in our observations. Later, when we run CCDPROC on the star and skyflat images, IRAF will scale this dark frame (darktime=1000sec) to the exposure time (=darktime) of each image, and subtract off the scaled dark image.
1) first, see which temperatures have dark frames available:
> imhead /usr/Kracken2/layden/KR2/BGTEL/DARKS/DarkT*.imh
2) copy to your directory the image with the temperature closest to your setpoint. In our example, the setpoint was -14C, so we have chosen DarkTm15.imh:
> imcopy /usr/Kracken2/layden/KR2/BGTEL/DARKS/DarkTm15.imh DarkTm15.imh
OVERVIEW: The purpose of the flat field images is to remove the variations in sensitivity from one pixel to the next. To do this, we obtain several 'skyflat' images of the sky (presumably a source of uniform illumination over the chip) in each filter. We combine the images to remove any cosmic rays or stars, which would produce defects on the final flat field (if, on a frame of AN_Ser, the variable or comparison star fell on one of these defects, the photometry for that star would be incorrect!).
To ensure that the stars change position from one skyflat to the next, we move the telescope a bit between exposures. However, its easy to forget that in the excitement of taking skyflats, so we will double-check here that there are no 'overlap images'. We need a bare minimum of 3 skyflats in each filter for the star-rejection techniques in IRAF to work; 5 is better, and >10 is best, especially if there are some clouds or other non-uniformities in the sky while you were taking flats.
The ideal skyflat has between 7000 and 11,000 counts &endash; this ensures plenty of signal (60,000 to 94,000 electrons per pixel) while allowing good dynamic range above (up to 16,300 counts) and below (down to 0 counts) to record the sensitivities of especially "hot" and "cold" pixels. However, it is sometimes not possible to get such well-exposed counts. Skyflats with count levels between 4000-7000, and between 11,000-13,000 are ok; skyflats with 2000-4000 counts will even do in a pinch. There is some "art" involved in knowing whether it is best to add in or leave out extra skyflats of less-than-ideal quality, so if you have any questions or doubts, please discuss them with ACL. This is a critical part of the processing procedure!
1) make a list the V skyflats and a list of the I skyflats. Use the 'files' command, and edit it with emacs.
> files *sflat*.imh >> sflats
> cp sflats sflatv
> emacs sflatv (remove all I skyflats)
> repeat for the I skyflats, call the file 'sflati'
2) determine the median counts in each skyflat. This will help you decide which flats to keep and which to reject (see OVERVIEW). Also review the exposure times (don't use any shorter than 10sec!).
> imstat @sflats fields="image,midpt" lower=930 upper=INDEF >> sflat.median
> hselect @sflats $I,exptime yes >> sflat.exptime
3) check to be sure there are no 'overlap images'. Display the V skyflats in the order they were obtained and blink them. If the stars move so that they don't overlap on sequential images, all should be well. If some of the images do overlap (e.g., you forgot to move the telescope between them), you can only use one (1) of the overlapped images. Pick the one with the counts closest to 9000, and/or with the exposure time closest to 30 sec (shorter exposures may have a little 'shutter effect', longer ones tend to have more faint stars). In our example, I did the following:
> set stdimage=imt1024
> display sflatV03 1
> display sflatV04 2
> display sflatV05 3
> display sflatV06 4
> blink the images, see ACL if you don't know how.
> display sflatV07 1
>display sflatV13 2
>display sflatV14 3
>display sflatV15 4
> blink the images
I found that sflatV05 and sflatV06 were overlapped. V05 had median=7245 at EXPTIME=26sec while V06 had median=6734 at EXPTIME=32sec. Tough call, I used V05. The rest were ok.
4) Delete the names of the images you do not want to use from sflatv. Reasons for rejection could include (a) star overlap, (b) low/high counts, (c) exposure time to short. See ACL if you have any doubts!
> emacs sflatv
5) repeat Steps 3 and 4 for the I skyflats.
6) process the skyflats. Here we will overscan-subtract them, trim them (the rows Y>1020 and the columns X<3 are bad), zero-correct and dark-correct them. As part of this process, Zero.imh will also get trimmed.
> epar ccdproc (this version is specifically for the flats, don't use it for the stars!)
> ccdproc @sflatv
> ccdproc @sflati
7) combine the V skyflats into a single, high signal-to-noise, starless final flat, "FlatV.imh". Ditto for the I skyflats, producing "FlatI.imh". If you want to learn more about the way the stars and cosmic rays get rejected, type "help flatcombine" and focus on the rejection algorithms.
> epar flatcombine
> flatcombine @sflatv out=FlatV.imh
> flatcombine @sflati out=FlatI.imh
8) display your final flats to be sure they look reasonable. There should be no obvious stars, and the intensities should drop as you move from the center toward the corners. There may be some "bad pixels" scattered here and there that just won't flatten out. Discuss the flats with ACL if there are any questions.
> display FlatV.imh 1
> display FlatI.imh 2
OVERVIEW: While observing, you should have taken a bias image after each observation set of a star (VI*IV*VIZ), and the name should look like "AN_Ser1z1.imh" for the bias after the first observation set on the star AN Ser. I will refer to these as "local biases" to distinguish them from the sets of biases you took at the beginning/end of the night. We will compute the overscan value for the local bias associated with each observing set (again, we will use the median of the intensities over the entire frame), and then subtract this value off each of the star images in that set. We have noticed that the bias (overscan) level varies by 0-5 counts over the course of several hours, and this is our attempt to correct for this variation. In the following section, the 2-D bias structure will be removed using Zero.imh.
1) make a list of the local bias frames. Edit out any of the biases taken at the beginning/end of night, if they appear.
> files *z1.imh >> bias2
> emacs bias2.cl
2) determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the file bias2.median in our example.
> imstat @bias2 fields="image,midpt" lower=930 upper=1060 >> bias2.median
3) using 'files', create a list of all the images you took. Then remove the names of all the non-star images (biases, flats, Zero, Flat*, Dark*). Imhead the list to ensure you got it right.
> files *.imh >> stars.all
> emacs stars.all
> imhead @stars.all
4) create an IRAF script to subtract these values from the star images with which they are associated. Use these emacs keystrokes to turn the list into a script. The link shows what the final script looks like. Run the script. The resulting images have been overscan subtracted &endash; we will complete their processing in the next section.
> cp stars.all bias2.cl
> emacs bias2.cl
> cl < bias2.cl
OVERVIEW: in the previous section, we removed the mean bias level (overscan) from each of the star images. Here, we will run CCDPROC in IRAF to (a) trim the images, (b) subtract Zero.imh to remove the 2-D structure produced when the chip is read out, (c) subtract off the appropriate-temperature dark image (ccdproc scales the dark image intensities by the ratio DARKTIME(star_image)/DARKTIME(dark_image)), and (d) divide by the appropriate flat field image (FlatV.imh for the V star images and FlatI.imh for the I star images). The resulting images should be completely processed and ready for photometry!
1) copy and edit 'stars.all' to make a list of all the V-filter star images and a list of all the I-filter star images. It should be easy with emacs to run down each list and delete the lines (using "^k") with the wrong filter.
> cp stars.all stars.v
> emacs stars.v (keep only names like "AN_Ser2v3.imh)
> cp stars.all stars.i
> emacs stars.i (keep only names like "AN_Ser4i3.imh)
2) set the parameters for CCDPROC (note that these are different from the parameters used on the skyflats!), and run CCDPROC separately for the V and I images.
> epar ccdproc
> ccdproc @stars.v flat=FlatV.imh
> ccdproc @stars.i flat=FlatI.imh
7) display several of the V and several of the I images to be sure the processing looks reasonable. The sky should appear pretty uniformly gray (i.e., constant brightness). Discuss the images with ACL if there are any questions.
> display AN_Ser1v2 1
Once you are satisfied with your processing, email ACL with the directory path of your final images, and a list of all the stars which you observed. I will inspect the images, and if all looks good, I will notify the people who are responsible for doing the photometry on the various stars. These people will copy the images into their own accounts for the photometry. ACL will make tape copies of the raw and processed images for the archive. If we need to reprocess or re-photometer the data at some time in the future, we can recover the images from the tapes. ACL will tell you when the images can be deleted from your account.