In this example, I will refer to a fictitious night of data which, in order of acquisition, looks like this (ie, you have taken bias images after every observing set). If you did not do this, go here.
1) From PC, ftp all frames to your account on baade, and convert to imh format.
2) Put Exptime, Darktime, ST, Airmass, HJD into headers.
3) Prepare the Zero frame.
4) Copy correct temperature Dark frame to current directory.
5) Prepare the Flat Field frames for each filter used.
6) For each observation set, determine overscan from local bias and subtract off star images.
7) For all star observations, apply Zero, Dark, and Flats.
8) Notify team members where images can be found.
9) ACLa will backup all processed and raw images to tape.
At any point, if you have trouble, you may get a jumpstart from the troubleshooting page.
Updated 2000 Feb 21 - ACL
a FYI, ACL = Dr. Andy C. Layden (a.k.a. YFL, Your Fearless Leader)
Overview: In this example, we will process the data in a directory called /b1/layden/BGTEL/99JUN03. I recommend you store the unprocessed *.fit images in a subdirectory called RAW - that way if something goes wrong with processing the images, you can easily recover the originals.
1) On baade, enter IRAF and create the directories you will need. In our example, this is:
> clcl> cd /b1/layden/BGTEL
cl> mkdir 99JUN03
cl> cd 99JUN03
cl> mkdir RAW
cl> pwd
/b1/layden/BGTEL/99JUN03
2) From PC, enter FTP [we usually leave it "minimized" along the bottom bar; if its not there, navigate to C:\Program Files\WS_FTP and click on WS_FTP]
3) There are a bunch of files and programs that you will need for processing. Here, we will copy them all to the current directory on baade. [Hint: Cut/paste with the mouse to ensure accuracy.]
cl> cp /b1/layden/BGTEL/PROCESS/* .
4) The following IRAF script will convert all of the raw, FITS-format images (RAW/*.fit) into IMH-format images in the current directory (*.imh). [To see the things it has done, type "page readwrite.cl"].
cl> cl < fits2imh.clcl> pwd
/b1/layden/BGTEL/99JUN03
cl> imhead *.imh
(look at all those files!!)
Updated 2000 Feb 14 - ACL
Overview: The image header contains many "keywords" and their values that are required for image processing and analysis (try "imhead yourfile.imh l+"). In particular, ccdproc needs the keywords "EXPTIME" and "DARKTIME" to do the dark current corrections, but CCDSoft writes the exposure time into the keyword "EXPOSURE" - we will have to copy the value from EXPOSURE into the EXPTIME and DARKTIME. Also, we will express the time of our variable star observations in "Heliocentric Julian Date" (HJD), the number of days since some semi-arbitrary date about 2.5 million days ago. IRAF needs to know the Universal Time (UT), date (DATE-OBS), exposure time (EXPTIME), and Sidereal Time (ST) of the observation in order to compute HJD.
We accomplish all this using a task written by Nick Pearson. It will ask you a bunch of questions, and you should respond as follows [pwd first to check you are in the right directory, IRAF should be running].
cl> pwd/b1/layden/BGTEL/99JUN03
cl> cl < headadd.cl
Name of file containing stars RA and Dec: stars.allistEpoch of coordinates: 2000
UT at beginning of night: 12 03 14.3 (for example)
LST at beginning of night: 07 14 00.0 (for example)
The UT and LST entries are the times you wrote on the top of the logsheet -- they enable the program to compute the ST at any time during the night. Errors here secretly propagate throughout the analysis procedure, so take care! If you don't have the UT/ST zero-point from the logsheet, look here.
This corrects all the header information. A log of all the operations performed is written to the file headadd.log, and the file headadd_go.cl is the script that was run.
Just to be sure all went well, compare the old values (from CCDSoft) with the new values (that IRAF wants). They should be the same:
cl> hselect *.imh $I,date-obs,date-ut,time-obs,ut,exposure,exptime,darktime yes
Also, look to see that all the star images got reasonable values of ST, airmass, and HJD (compare with the log sheets you wrote while observing).
cl> hselect *.imh $I,st,airmass,hjd yes
Note: if there are any errors, contact ACL. The program is pretty tricky, and it is not 100% "bulletproof".
Updated 2000 Feb 14 - ACL
Overview: The 8 or so bias frames taken at the beginning and/or end of the night will be overscan subtracted (based on median counts), and then combined into a master frame, Zero.imh, which contains a 2-D map of the intensity-structure in the bias. The median value of Zero.imh should be about 0.
For an individual object frame (e.g., SW_And1v2.imh), the median bias level will be determined from the single bias frame taken soon afterward (e.g., AN_Ser1z1.imh), and the 2D bias structure will be removed using Zero. We do it this way because there is some evidence that the median bias level changes slowly by 3-10 adu over the course of several hours. However, the 2D structure in the bias is stable over time (but fairly subtle, so we need to average many bias frames to measure it accurately).
If you did not obtain single bias frames after each observation set (e.g., SW_And1z1.imh), go here.
... Updated 2000 Feb 14 - ACL
1) Make a list of the bias frames taken at the beginning and/or end of the night. Edit out any of the 'local biases' taken at the end of each observing set (if you called them, e.g., SW_And1z1.imh, they should not be included in the list).
> files *bias*.imh >> bias1.cl
> emacs bias1.cl
2) determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the output file in our example.
> imstat @bias1.cl fields="image,midpt" lower=930 upper=1060 >> bias1.median
3) type out the file bias1.median and compute the mean of the individual median values. Write this number down as it will resurface in Preparing the Flat Field Images.
4) create an IRAF script to subtract these values from the images from which they were derived. Thus, the median of each image will go from ~975 to 0. Again, these emacs keystrokes will be helpful in the editing. The link shows what the final script looks like.
> emacs bias1.cl
5) open the following packages in iraf. The link shows the suggested parameters for the ccdred task.
> noao.imred.ccdred
> epar ccdred
6) combine these overscan-subtracted biases into a high signal-to-noise map of the 2-D structure produced when the chip is read out. The final zero frame is called Zero.imh.
> epar zerocombine
> zerocombine
7) display Zero.imh to be sure it looks reasonable. Our zeros are usually pretty uniform in the upper-right, while a horizontal "ridge" exists at Y< 70 (5-10 counts higher than the upper-right) and a vertical "trough" exists at X<120 (5-10 counts lower than the upper-right). Discuss your Zero.imh with ACL if there are any questions.
> display Zero.imh 1
OVERVIEW: Every time we take an image, whether there is light falling on the chip or not, a small number of counts appear in each pixel due to "dark current" in the chip. The amount of dark current is proportional to the exposure time (i.e., it appears at a constant rate, counts/sec). The amount of dark current varies from pixel to pixel, and tends to be larger when the chip is operated at a higher temperature. To correct for the dark current, we have created a number of "dark frames" &endash; high signal-to-noise images recording the counts in each pixel &endash; each at a different temperature. In this step, we select the dark frame with the temperature closest to the temperature setpoint in our observations. Later, when we run CCDPROC on the star and skyflat images, IRAF will scale this dark frame (darktime=1000sec) to the exposure time (=darktime) of each image, and subtract off the scaled dark image.
1) first, see which temperatures have dark frames available:
> imhead /usr/Kracken2/layden/KR2/BGTEL/DARKS/DarkT*.imh
2) copy to your directory the image with the temperature closest to your setpoint. In our example, the setpoint was -14C, so we have chosen DarkTm15.imh:
> imcopy /usr/Kracken2/layden/KR2/BGTEL/DARKS/DarkTm15.imh DarkTm15.imh
OVERVIEW: The purpose of the flat field images is to remove the variations in sensitivity from one pixel to the next. To do this, we obtain several 'skyflat' images of the sky (presumably a source of uniform illumination over the chip) in each filter. We combine the images to remove any cosmic rays or stars, which would produce defects on the final flat field (if, on a frame of AN_Ser, the variable or comparison star fell on one of these defects, the photometry for that star would be incorrect!).
To ensure that the stars change position from one skyflat to the next, we move the telescope a bit between exposures. However, its easy to forget that in the excitement of taking skyflats, so we will double-check here that there are no 'overlap images'. We need a bare minimum of 3 skyflats in each filter for the star-rejection techniques in IRAF to work; 5 is better, and >10 is best, especially if there are some clouds or other non-uniformities in the sky while you were taking flats.
The ideal skyflat has between 7000 and 11,000 counts &endash; this ensures plenty of signal (60,000 to 94,000 electrons per pixel) while allowing good dynamic range above (up to 16,300 counts) and below (down to 0 counts) to record the sensitivities of especially "hot" and "cold" pixels. However, it is sometimes not possible to get such well-exposed counts. Skyflats with count levels between 4000-7000, and between 11,000-13,000 are ok; skyflats with 2000-4000 counts will even do in a pinch. There is some "art" involved in knowing whether it is best to add in or leave out extra skyflats of less-than-ideal quality, so if you have any questions or doubts, please discuss them with ACL. This is a critical part of the processing procedure!
1) make a list the V skyflats and a list of the I skyflats. Use the 'files' command, and edit it with emacs.
> files *sflat*.imh >> sflats
> cp sflats sflatv
> emacs sflatv (remove all I skyflats)
> repeat for the I skyflats, call the file 'sflati'
2) determine the median counts in each skyflat. This will help you decide which flats to keep and which to reject (see OVERVIEW). Also review the exposure times (don't use any shorter than 10sec!).
> imstat @sflats fields="image,midpt" lower=930 upper=INDEF >> sflat.median
> hselect @sflats $I,exptime yes >> sflat.exptime
3) check to be sure there are no 'overlap images'. Display the V skyflats in the order they were obtained and blink them. If the stars move so that they don't overlap on sequential images, all should be well. If some of the images do overlap (e.g., you forgot to move the telescope between them), you can only use one (1) of the overlapped images. Pick the one with the counts closest to 9000, and/or with the exposure time closest to 30 sec (shorter exposures may have a little 'shutter effect', longer ones tend to have more faint stars). In our example, I did the following:
> set stdimage=imt1024
> display sflatV03 1
> display sflatV04 2
> display sflatV05 3
> display sflatV06 4
> blink the images, see ACL if you don't know how.
> display sflatV07 1
>display sflatV13 2
>display sflatV14 3
>display sflatV15 4
> blink the images
I found that sflatV05 and sflatV06 were overlapped. V05 had median=7245 at EXPTIME=26sec while V06 had median=6734 at EXPTIME=32sec. Tough call, I used V05. The rest were ok.
4) Delete the names of the images you do not want to use from sflatv. Reasons for rejection could include (a) star overlap, (b) low/high counts, (c) exposure time to short. See ACL if you have any doubts!
> emacs sflatv
5) repeat Steps 3 and 4 for the I skyflats.
6) process the skyflats. Here we will overscan-subtract them, trim them (the rows Y>1020 and the columns X<3 are bad), zero-correct and dark-correct them. As part of this process, Zero.imh will also get trimmed.
> epar ccdproc (this version is specifically for the flats, don't use it for the stars!)
> ccdproc @sflatv
> ccdproc @sflati
7) combine the V skyflats into a single, high signal-to-noise, starless final flat, "FlatV.imh". Ditto for the I skyflats, producing "FlatI.imh". If you want to learn more about the way the stars and cosmic rays get rejected, type "help flatcombine" and focus on the rejection algorithms.
> epar flatcombine
> flatcombine @sflatv out=FlatV.imh
> flatcombine @sflati out=FlatI.imh
8) display your final flats to be sure they look reasonable. There should be no obvious stars, and the intensities should drop as you move from the center toward the corners. There may be some "bad pixels" scattered here and there that just won't flatten out. Discuss the flats with ACL if there are any questions.
> display FlatV.imh 1
> display FlatI.imh 2
OVERVIEW: While observing, you should have taken a bias image after each observation set of a star (VI*IV*VIZ), and the name should look like "AN_Ser1z1.imh" for the bias after the first observation set on the star AN Ser. I will refer to these as "local biases" to distinguish them from the sets of biases you took at the beginning/end of the night. We will compute the overscan value for the local bias associated with each observing set (again, we will use the median of the intensities over the entire frame), and then subtract this value off each of the star images in that set. We have noticed that the bias (overscan) level varies by 0-5 counts over the course of several hours, and this is our attempt to correct for this variation. In the following section, the 2-D bias structure will be removed using Zero.imh.
1) make a list of the local bias frames. Edit out any of the biases taken at the beginning/end of night, if they appear.
> files *z1.imh >> bias2
> emacs bias2.cl
2) determine the overscan value for each bias. We will use the median intensity value of the entire image. The link shows the file bias2.median in our example.
> imstat @bias2 fields="image,midpt" lower=930 upper=1060 >> bias2.median
3) using 'files', create a list of all the images you took. Then remove the names of all the non-star images (biases, flats, Zero, Flat*, Dark*). Imhead the list to ensure you got it right.
> files *.imh >> stars.all
> emacs stars.all
> imhead @stars.all
4) create an IRAF script to subtract these values from the star images with which they are associated. Use these emacs keystrokes to turn the list into a script. The link shows what the final script looks like. Run the script. The resulting images have been overscan subtracted &endash; we will complete their processing in the next section.
> cp stars.all bias2.cl
> emacs bias2.cl
> cl < bias2.cl
OVERVIEW: in the previous section, we removed the mean bias level (overscan) from each of the star images. Here, we will run CCDPROC in IRAF to (a) trim the images, (b) subtract Zero.imh to remove the 2-D structure produced when the chip is read out, (c) subtract off the appropriate-temperature dark image (ccdproc scales the dark image intensities by the ratio DARKTIME(star_image)/DARKTIME(dark_image)), and (d) divide by the appropriate flat field image (FlatV.imh for the V star images and FlatI.imh for the I star images). The resulting images should be completely processed and ready for photometry!
1) copy and edit 'stars.all' to make a list of all the V-filter star images and a list of all the I-filter star images. It should be easy with emacs to run down each list and delete the lines (using "^k") with the wrong filter.
> cp stars.all stars.v
> emacs stars.v (keep only names like "AN_Ser2v3.imh)
> cp stars.all stars.i
> emacs stars.i (keep only names like "AN_Ser4i3.imh)
2) set the parameters for CCDPROC (note that these are different from the parameters used on the skyflats!), and run CCDPROC separately for the V and I images.
> epar ccdproc
> ccdproc @stars.v flat=FlatV.imh
> ccdproc @stars.i flat=FlatI.imh
7) display several of the V and several of the I images to be sure the processing looks reasonable. The sky should appear pretty uniformly gray (i.e., constant brightness). Discuss the images with ACL if there are any questions.
> display AN_Ser1v2 1
Once you are satisfied with your processing, email ACL with the directory path of your final images, and a list of all the stars which you observed. I will inspect the images, and if all looks good, I will notify the people who are responsible for doing the photometry on the various stars. These people will copy the images into their own accounts for the photometry. ACL will make tape copies of the raw and processed images for the archive. If we need to reprocess or re-photometer the data at some time in the future, we can recover the images from the tapes. ACL will tell you when the images can be deleted from your account.