In this example, we will refer to a fictitious night of data which, in order of acquisition, looks like this . Here, you have taken bias images after every observing set (the *1z1.fit images). These will help you account for any drifts that may occur in the bias level over time, which may be important if your images have a sky level very close to zero (e.g., very short exposure times or very narrow filters). This procedure involves some additional, tedious steps. If you are doing differential variable star photometry (e.g., for the RR Lyrae program), you can save yourself time and trouble by using the basic processing procedure.
1) From PC, ftp all frames to your account on baade, and convert to imh format.
2) Put Exptime, Darktime, ST, Airmass, HJD into headers.
3) Prepare the Zero frame.
4) Copy correct temperature Dark frame to current directory.
5) Prepare the Flat Field frames for each filter used.
6) For each observation set, determine overscan from local bias and subtract off star images.
7) For all star observations, apply Zero, Dark, and Flats.
8) Notify team members where images can be found.
9) ACLa will backup all processed and raw images to tape.
At any point, if you have trouble, you may get a jumpstart from the troubleshooting page.
Updated 2000 Feb 23 - ACL
a FYI, ACL = Dr. Andy C. Layden (a.k.a. YFL, Your Fearless Leader)
Overview: In this example, we will process the data in a directory called /b1/layden/BGTEL/99JUN03. I recommend you store the unprocessed *.fit images in a subdirectory called RAW - that way if something goes wrong with processing the images, you can easily recover the originals.
1) On baade, enter IRAF and create the directories you will need. In our example, this is:
> cl cl> cd /b1/layden/BGTEL cl> mkdir 99JUN03 cl> cd 99JUN03 cl> mkdir RAW cl> pwd /b1/layden/BGTEL/99JUN03
2) From PC, enter FTP [we usually leave it "minimized" along the bottom bar; if its not there, navigate to C:\Program Files\WS_FTP and click on WS_FTP]
3) There are a bunch of files and programs that you will need for processing. Here, we will copy them all to the current directory on baade. [Hint: Cut/paste with the mouse to ensure accuracy.]
cl> cp /b1/layden/BGTEL/PROCESS/* .
4) The following IRAF script will convert all of the raw, FITS-format images (RAW/*.fit) into IMH-format images in the current directory (*.imh). [To see the things it has done, type "page readwrite.cl"].
cl> cl < fits2imh.cl cl> pwd /b1/layden/BGTEL/99JUN03 cl> imhead *.imh(look at all those files!!)
Updated 2000 Feb 23 - ACL
Overview: The image header contains many "keywords" and their values that are required for image processing and analysis (try "imhead yourfile.imh l+"). In particular, ccdproc needs the keywords "EXPTIME" and "DARKTIME" to do the dark current corrections, but CCDSoft writes the exposure time into the keyword "EXPOSURE" - we will have to copy the value from EXPOSURE into the EXPTIME and DARKTIME. Also, we will express the time of our variable star observations in "Heliocentric Julian Date" (HJD), the number of days since some semi-arbitrary date about 2.5 million days ago. IRAF needs to know the Universal Time (UT), date (DATE-OBS), exposure time (EXPTIME), and Sidereal Time (ST) of the observation in order to compute HJD.
We accomplish all this using a task written by Nick Pearson. It will ask you a bunch of questions, and you should respond as follows [pwd first to check you are in the right directory, IRAF should be running].
cl> pwd /b1/layden/BGTEL/99JUN03 cl> cl < headadd.clName of file containing stars RA and Dec: stars.allist Epoch of coordinates: 2000 UT at beginning of night: 12 03 14.3 (for example) LST at beginning of night: 07 14 00.0 (for example)
The UT and LST entries are the times you wrote on the top of the logsheet -- they enable the program to compute the ST at any time during the night. Errors here secretly propagate throughout the analysis procedure, so take care! If you don't have the UT/ST zero-point from the logsheet, look here.
This corrects all the header information. A log of all the operations performed is written to the file "headadd.log", and the file "headadd_go.cl" is the script that was run.
Just to be sure all went well, compare the old values (from CCDSoft) with the new values (that IRAF wants). They should be the same:
cl> hselect *.imh $I,date-obs,date-ut,time-obs,ut yes cl> hselect *.imh $I,exposure,exptime,darktime yes
Also, look to see that all the star images got reasonable values of ST, airmass, and HJD (compare with the log sheets you wrote while observing).
cl> hselect *.imh $I,st,airmass,hjd yes
Note: if there are any errors, contact ACL. The program is pretty tricky, and it is not 100% "bulletproof".
Updated 2000 Feb 23 - ACL
Overview: The 8 or so bias frames taken at the beginning and/or end of the night will be overscan subtracted (based on median counts), and then combined into a master frame, Zero.imh, which contains a 2-D map of the intensity-structure in the bias. The median value of Zero.imh should be about 0.
For an individual object frame (e.g., SW_And1v2.imh), the median bias level will be determined from the single bias frame taken soon afterward (e.g., AN_Ser1z1.imh), and the 2D bias structure will be removed using Zero. We do it this way because there is some evidence that the median bias level changes slowly by 3-10 adu over the course of several hours. However, the 2D structure in the bias is stable over time (but fairly subtle, so we need to average many bias frames to measure it accurately).
If you did not obtain single bias frames after each observation set (e.g., SW_And1z1.imh), go here.
... Updated 2000 Feb 14 - ACL
1) Make a list of the bias frames taken at the beginning and/or end of the night. Edit out any of the 'local biases' taken at the end of each observing set (if you called them, e.g., SW_And1z1.imh, they should not be included in the list).
cl> files *bias*.imh >> bias1.list cl> emacs bias1.list
2) Determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the output file in our example.
cl> imstat @bias1.list fields="image,midpt" >> bias1.cl
3) Type out the file "bias1.cl" and compute the mean of the individual median values. Write this number down as it will resurface in Preparing the Flat Field Images.
4) Create an IRAF script to subtract these values from the images from which they were derived. Thus, the median of each image will go from ~975 to 0. Again, these emacs keystrokes will be helpful in the editing. The link shows what the final script looks like. Run the script.
cl> emacs bias1.cl cl> cl < bias1.cl
5) Open the following packages in IRAF. The link shows the suggested parameters for the CCDRED task.
cl> noao no> imred im> ccdred cc> epar ccdred
6) Combine these overscan-subtracted biases into a high signal-to-noise map of the 2-D structure produced when the chip is read out. The final zero frame is called Zero.imh.
cl> epar zerocombine cl> zerocombine
7) Display "Zero.imh" to be sure it looks reasonable. Our zeros are usually pretty uniform in the upper-right, while a horizontal "ridge" exists at Y< 70 (5-10 counts higher than the upper-right) and a vertical "trough" exists at X<120 (5-10 counts lower than the upper-right). Discuss your Zero.imh with ACL if there are any questions.
cl> display Zero.imh 1
Updated 2000 Feb 23 - ACL
Overview: Every time we take an image, whether there is light falling on the chip or not, a small number of counts appears in each pixel due to "dark current" in the chip. The amount of dark current is proportional to the exposure time (i.e., it appears at a constant rate, counts/sec). The amount of dark current varies from pixel to pixel, and tends to be larger when the chip is operated at a higher temperature. Here is an example.
To correct for the dark current, we have created a number of "dark frames" -- high signal-to-noise images recording the counts in each pixel -- each at a different temperature. In this step, we select the dark frame with the temperature closest to the temperature setpoint in our observations. Later, when we run CCDPROC on the star and skyflat images, IRAF will scale this dark frame (darktime=1000sec) to the exposure time (=darktime) of each image, and subtract off the scaled dark image.
1) First, see which temperatures have dark frames available:
cl> imhead /b1/layden/BGTEL/DARKS/DarkT*.imh
2) Copy to your directory the image with the temperature closest to your setpoint. In our example, the setpoint was -14C, so we have chosen DarkTm15.imh:
cl> imcopy /b1/layden/BGTEL/DARKS/DarkTm15.imh DarkTm15.imh
Updated 2000 Feb 23 - ACL
Overview: The purpose of the flat field images is to remove the variations in sensitivity from one pixel to the next. To do this, we obtain several "skyflat" images of the sky (presumably a source of uniform illumination over the chip) in each filter. We combine the images to remove any cosmic rays or stars, which would produce defects on the final flat field (if, on a frame of AN_Ser, the variable or comparison star fell on one of these defects, the photometry for that star would be incorrect!).
To ensure that the stars change position from one skyflat to the next, we move the telescope a bit between exposures. However, it is easy to forget that in the excitement of taking skyflats, so we will double-check here that there are no "overlap images" (consecutive images where the stars lie at the same XY positions). We need a bare minimum of 3 skyflats in each filter for the star-rejection techniques in IRAF to work; 5 is better, and >10 is best, especially if there are some clouds or other non-uniformities in the sky while you were taking flats.
The ideal skyflat has between 7000 and 11,000 counts -- this ensures plenty of signal (60,000 to 94,000 electrons per pixel) while allowing good dynamic range above (up to 16,300 counts) and below (down to 0 counts) to record the sensitivities of especially "hot" and "cold" pixels. However, it is sometimes not possible to get such well-exposed frames. Skyflats with count levels between 4000-7000, and between 11,000-13,000 are ok; skyflats with 2000-4000 counts will even do in a pinch. There is some "art" involved in knowing whether it is best to add in or leave out extra skyflats of less-than-ideal quality, so if you have any questions or doubts, please discuss them with ACL. This is a critical part of the processing procedure!
1) Make a list the V skyflats and a list of the I skyflats. Use the FILES command, and edit it with emacs.
cl> files *sflat*.imh >> sflats cl> cp sflats sflatv cl> emacs sflatv (remove all I skyflats) cl> cp sflats sflati cl> emacs sflati (remove all V skyflats)
2) Determine the median counts in each skyflat. This will help you decide which flats to keep and which to reject (see Overview). Also review the exposure times (don't use any shorter than 10sec!).
cl> imstat @sflats fields="image,midpt" >> sflat.median cl> hselect @sflats $I,exptime yes >> sflat.exptime
3) Check to be sure there are no "overlap images". Display the V skyflats in the order they were obtained and blink them. If the stars move so that they don't overlap on sequential images, all should be well. If some of the images do overlap (e.g., you forgot to move the telescope between them), you can only use one (1) of the overlapped images. Pick the one with the counts closest to 9000, and/or with the exposure time closest to 30 sec (shorter exposures may have a little "shutter effect", longer ones tend to have more faint stars). In our example, I did the following:
* If you don't have one already, open an ximtool (image display window) by typing in your IRAF window:cc> !ximtool & cc> set stdimage=imt1024 cc> display sflatV03 1 cc> display sflatV04 2 cc> display sflatV05 3 cc> display sflatV06 4* blink the images (see ACL if you don't know how).
cc> display sflatV07 1 cc> display sflatV13 2 cc> display sflatV14 3 cc> display sflatV15 4* blink the images
* I found that sflatV05 and sflatV06 were overlapped. V05 had median=7245 at EXPTIME=26sec while V06 had median=6734 at EXPTIME=32sec. Tough call -- I used V05. The rest were OK.
4) Delete the names of the images you do not want to use from sflatv. Reasons for rejection could include (a) star overlap, (b) low/high counts, (c) exposure time too short. See ACL if you have any doubts!
cc> emacs sflatv
5) Repeat Steps 3 and 4 for the I skyflats.
6) Now is the time to remove the median bias level (about 975, we jotted this down in Step 3 of "Preparing a Zero Frame". We will write a short script to do this using the task IMARITH. The final script, called "sflat_corr.cl", should look analagous to this. Run the script.
cc> cp sflatv sflat_corr.cl cc> cat sflati >> sflat_corr.cl cc> emacs sflat_corr.cl cc> cl < sflat_corr.cl
7) Process the skyflats using CCDPROC. Here we will trim them (the rows Y>1020 and the columns X<3 are bad), zero-correct them, and dark-correct them. As part of this process, "Zero.imh" will also get trimmed. Note: be careful to epar ccdproc to match the example in the link below!!
cc> epar ccdproc cc> ccdproc @sflatv zero=Zero.imh cc> ccdproc @sflati zero=Zero.imh
8) Combine the V skyflats into a single, high signal-to-noise, starless final flat, "FlatV.imh". Ditto for the I skyflats, producing "FlatI.imh". If you want to learn more about the way the stars and cosmic rays get rejected, type "help flatcombine" and focus on the rejection algorithms.
cc> epar flatcombine cc> flatcombine @sflatv out=FlatV.imh cc> flatcombine @sflati out=FlatI.imh
9) Display your final flats to be sure they look reasonable. There should be no obvious stars, and the intensities should drop as you move from the center toward the corners. There may be some "bad pixels" scattered here and there that just won't flatten out. Discuss the flats with ACL if there are any questions.
cc> display FlatV.imh 1 cc> display FlatI.imh 2
Updated 2000 Feb 23 - ACL
Overview: While observing, you should have taken a bias image after each observation set of a star (VI*IV*VIZ), and the name should look like "AN_Ser1z1.imh" for the bias after the first observation set on the star AN Ser. I will refer to these as "local biases" to distinguish them from the sets of biases you took at the beginning/end of the night.
We will compute the overscan value for the local bias associated with each observing set (again, we will use the median of the intensities over the entire frame), and then subtract this value off each of the star images in that set.
The reason for this is because we have noticed that the bias (overscan) level varies by 0-5 counts over the course of several hours, and this is our attempt to correct for this variation. In the following section, the 2-D bias structure will be removed using "Zero.imh".
1) Make a list of the local bias frames. Edit out any of the biases taken at the beginning/end of night, if they appear.
cc> files *z1.imh >> bias2.list cc> emacs bias2.list
2) Determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the file "bias2.median" in our example.
cc> imstat @bias2 fields="image,midpt" upper=1060 >> bias2.median
3) Using FILES, create a list of all the star (object) images you took. Hopefully all the star/object images included the underscore character ("_") to distinguish them from the calibration images. Check for and remove the names of any non-star images (biases, flats, Zero, Flat*, Dark*) using emacs. CCDLIST the list to ensure you got it right.
cc> files *_*.imh >> stars.all cc> emacs stars.all cc> ccdlist @stars.all
4) Create an IRAF script to subtract these values from the star images with which they are associated. Use these emacs keystrokes to turn the list into a script. The link shows what the final script looks like. Run the script. The resulting images have been overscan subtracted -- we will complete their processing in the next section.
cc> cp stars.all bias2.cl cc> emacs bias2.cl cc> cl < bias2.cl
Updated 2000 Feb 23 - ACL
Overview: In the previous section, we removed the mean bias level (overscan) from each of the star images. Here, we will run CCDPROC in IRAF to (a) trim the images, (b) subtract Zero.imh to remove the 2-D structure produced when the chip is read out, (c) subtract off the appropriate-temperature dark image, and (d) divide by the appropriate flat field image ("FlatV.imh" for the V star images and "FlatI.imh" for the I star images). The resulting images should be completely processed and ready for photometry!
1) Copy and edit "stars.all" to make a list of all the V-filter star images and a list of all the I-filter star images. It should be easy with emacs to run down each list and delete the lines (using "^k") with the wrong filter.
cc> cp stars.all stars.v cc> emacs stars.v (keep only names like "AN_Ser2v3.imh) cc> cp stars.all stars.i cc> emacs stars.i (keep only names like "AN_Ser4i3.imh)
2) Set the parameters for CCDPROC (note that these are different from the parameters used on the skyflats!), and run CCDPROC separately for the V and I images. CCDLIST the results to ensure the processing went well -- all the star images should be [TZDF] now.
cc> epar ccdproc cc> ccdproc @stars.v zero=Zero.imh flat=FlatV.imh cc> ccdproc @stars.i zero=Zero.imh flat=FlatI.imh
7) Display several of the V and several of the I images to be sure the processing looks reasonable. The sky should appear pretty uniformly gray (i.e., constant brightness). Discuss the images with ACL if there are any questions.
cc> display AN_Ser1v2 1
Updated 2000 Feb 23 - ACL
Once you are satisfied with your processing, email ACL with the directory path of your final images, and a list of all the stars which you observed. I will inspect the images, and if all looks good, I will notify the people who are responsible for doing the photometry on the various stars. These people will copy the images into their own accounts to do the photometry.
ACL will make tape copies of the raw and processed images for the archive. If we need to reprocess or re-photometer the data at some time in the future, we can recover the images from the tapes.
ACL will tell you when the images can be deleted from your account.
Updated 2000 Feb 23 - ACL