In this example, we will refer to a fictitious night of data which, in order of acquisition, looks like this. In this example, we will not use any bias images taken after an observing set (the *1z1.fit images). If you have very short exposure times or very narrow filters (so the sky level is near zero), you may want to follow a more tedius but exact processing procedure.
1) From PC, ftp all frames to your account on baade, and convert to imh format.
2) Put Exptime, Darktime, ST, Airmass, HJD into headers.
3) Prepare the Zero frame.
4) Copy correct temperature Dark frame to current directory.
5) Prepare the Flat Field frames for each filter used.
6) Process all star observations (do Trim, Zero, Dark, and Flat corrections).
7) Notify team members where images can be found.
8) ACLa will backup all processed and raw images to tape, and inform you when images can be deleted.
At any point, if you have trouble, you may get a jumpstart from the troubleshooting page.
Updated 2000 Feb 23 - ACL
a FYI, ACL = Dr. Andy C. Layden (a.k.a. YFL, Your Fearless Leader)
Overview: In this example, we will process the data in a directory called /b1/layden/BGTEL/99JUN03. I recommend you store the unprocessed *.fit images in a subdirectory called RAW - that way if something goes wrong with processing the images, you can easily recover the originals.
1) On baade, enter IRAF and create the directories you will need. In our example, this is:
> cl cl> cd /b1/layden/BGTEL cl> mkdir 99JUN03 cl> cd 99JUN03 cl> mkdir RAW cl> pwd /b1/layden/BGTEL/99JUN03
2) From PC, enter FTP [we usually leave it "minimized" along the bottom bar; if its not there, navigate to C:\Program Files\WS_FTP and click on WS_FTP95LE]
3) There are a bunch of files and programs that you will need for processing. Here, we will copy them all to the current directory on baade. [Hint: Cut/paste with the mouse to ensure accuracy.]
cl> cp /b1/layden/BGTEL/PROCESS/* .
4) The following IRAF script will convert all of the raw, FITS-format images (RAW/*.fit) into IMH-format images in the current directory (*.imh). [To see the things it has done, type "page readwrite.cl"].
cl> cl < fits2imh.cl cl> pwd /b1/layden/BGTEL/99JUN03 cl> imhead *.imh(look at all those files!!)
5) Keep the FITS-format images in /RAW ! They are a convenient backup in case you make an error in later processing steps. Plus, we will make a tape copy of the raw and processed images for our archive. ACL will inform you when it is OK to delete the images in /RAW .
Updated 2000 Feb 23 - ACL
Overview: The image header contains many "keywords" and their values that are required for image processing and analysis (try "imhead yourfile.imh l+"). In particular, CCDPROC needs the keywords "EXPTIME" and "DARKTIME" to do the dark current corrections, but CCDSoft writes the exposure time into the keyword "EXPOSURE" - we will have to copy the value from EXPOSURE into the EXPTIME and DARKTIME.
Also, we will express the time of our variable star observations in "Heliocentric Julian Date" (HJD), the number of days since some semi-arbitrary date about 2.5 million days ago. IRAF needs to know the Universal Time (UT), UT-date (DATE-OBS), exposure time (EXPTIME), and Sidereal Time (ST) of the observation in order to compute HJD.
We accomplish all this using a task written by Nick Pearson. It will ask you a bunch of questions, and you should respond as follows [pwd first to check you are in the right directory, IRAF should be running].
cl> pwd /b1/layden/BGTEL/99JUN03 cl> cl < headadd.clName of file containing stars RA and Dec: stars.allist Epoch of coordinates: 2000 UT at beginning of night: 12 03 14.3 (for example) LST at beginning of night: 07 14 00.0 (for example)
The UT and LST entries are the times you wrote on the top of the logsheet -- they enable the program to compute the ST at any time during the night. Errors here secretly propagate throughout the analysis procedure, so take care! If you don't have the UT/ST zero-point from the logsheet, look here.
This corrects all the header information. A log of all the operations performed is written to the file "headadd.log", and the file "headadd_go.cl" is the script that was run.
Just to be sure all went well, compare the old values (from CCDSoft) with the new values (that IRAF wants). They should be the same:
cl> hselect *.imh $I,date-obs,date-ut,time-obs,ut yes cl> hselect *.imh $I,exposure,exptime,darktime yes
Also, look to see that all the star images got reasonable values of ST, airmass, and HJD (compare with the log sheets you wrote while observing).
cl> hselect *.imh $I,st,airmass,hjd yes
Note: if there are any errors, contact ACL. The program is pretty tricky, and it is not 100% "bulletproof".
Updated 2000 Feb 23 - ACL
Overview: The 8 or so bias frames taken at the beginning and/or end of the night will be trimmed (the rows Y>1020 and the columns X<3 are bad), and then combined into a master frame, "Zero.imh", which contains a 2-D map of the intensity-structure in the bias. The median value of Zero.imh should be about 975.
1) Make a list of the bias frames taken at the beginning and/or end of the night (when the dome was shut and the lights were off). These are probably the only biases you have. Check to be sure using IMHEAD. If there are other biases in your list, delete the filenames using emacs [Hint: these emacs keystrokes may be helpful in your editing here and elsewhere].
cl> files *bias*.imh >> bias.list cl> imhead @bias.list cl> emacs bias.list (if needed)
2) Open the following packages in IRAF. The link shows the suggested parameters for the CCDRED task.
cl> noao no> imred im> ccdred cc> epar ccdred
3) Use CCDPROC to trim the bad columns off the bias images, and convert them to real format. Usually you would use EPAR to edit the parameters for CCDPROC, but I have a shortcut that avoids typos. The file dpar.ccdproc_bias is a file which you have already copied to your account (give it a look-see with PAGE).
cc> cl < dpar.ccdproc_bias cc> lpar ccdproc cc> ccdrproc @bias.list
4) Check out the results using CCDLIST. The images are now smaller in size, [1020,1020] rather than [1024,1024]. They are also real format rather than short (integer), so they take up more space on the disk. The [T] means that only the processing step Trim was performed on these images.
cc> ccdlist @bias.list
5) Combine these trimmed biases into a high signal-to-noise map of the 2-D structure produced when the chip is read out. The final zero frame is called "Zero.imh".
cc> cl < dpar.zerocomb cc> zerocombine
6) Display "Zero.imh" to be sure it looks reasonable. Here is an example. Our zeros are usually pretty uniform in the upper-right, while a horizontal "ridge" exists at Y< 70 (5-10 counts higher than the upper-right) and a vertical "trough" exists at X<120 (5-10 counts lower than the upper-right). The median value of Zero.imh should be around 975 counts. Discuss your Zero.imh with ACL if there are any questions.
cc> display Zero.imh 1 cc> imstatistics Zero.imh fields="image,midpt"
Updated 2000 Feb 23 - ACL
Overview: Every time we take an image, whether there is light falling on the chip or not, a small number of counts appears in each pixel due to "dark current" in the chip. The amount of dark current is proportional to the exposure time (i.e., it appears at a constant rate, counts/sec). The amount of dark current varies from pixel to pixel, and tends to be larger when the chip is operated at a higher temperature. Here is an example.
To correct for the dark current, we have created a number of "dark frames" -- high signal-to-noise images recording the counts in each pixel -- each at a different temperature. In this step, we select the dark frame with the temperature closest to the temperature setpoint in our observations. Later, when we run CCDPROC on the star and skyflat images, IRAF will scale this dark frame (darktime=1000sec) to the exposure time (=darktime) of each image, and subtract off the scaled dark image.
1) First, see which temperatures have dark frames available:
cc> imhead /b1/layden/BGTEL/DARK/DarkT*.imh
2) Copy to your directory the image with the temperature closest to your setpoint. In our example, the setpoint was -14C, so we have chosen "DarkTm15.imh":
cc> imcopy /b1/layden/BGTEL/DARK/DarkTm15.imh LDark.imh
Updated 2000 Feb 23 - ACL
Overview: The purpose of the flat field images is to remove the variations in sensitivity from one pixel to the next. To do this, we obtain several "skyflat" images of the sky (presumably a source of uniform illumination over the chip) in each filter. We combine the images to remove any cosmic rays or stars, which would produce defects on the final flat field (if, on a frame of AN_Ser, the variable or comparison star fell on one of these defects, the photometry for that star would be incorrect!).
To ensure that the stars change position from one skyflat to the next, we move the telescope a bit between exposures. However, it is easy to forget that in the excitement of taking skyflats, so we will double-check here that there are no "overlap images" (consecutive images where the stars lie on the same XY positions). We need a bare minimum of 3 skyflats in each filter for the star-rejection techniques in IRAF to work; 5 is better, and >10 is best, especially if there are some clouds or other non-uniformities in the sky while you were taking flats.
The ideal skyflat has between 7000 and 11,000 counts -- this ensures plenty of signal (60,000 to 94,000 electrons per pixel) while allowing good dynamic range above (up to 16,300 counts) and below (down to 0 counts) to record the sensitivities of especially "hot" and "cold" pixels. However, it is sometimes not possible to get such well-exposed frames. Skyflats with count levels between 4000-7000, and between 11,000-13,000 are OK; skyflats with 2000-4000 counts will even do in a pinch. There is some "art" involved in knowing whether it is best to add in or leave out extra skyflats of less-than-ideal quality, so if you have any questions or doubts, please discuss them with ACL. This is a critical part of the processing procedure!
1) Make a list the V skyflats and a list of the I skyflats. Use the FILES command, and edit it with emacs.
cc> files *sflat*.imh >> sflats cc> cp sflats sflatv cc> emacs sflatv (remove all I skyflats) cc> cp sflats sflati cc> emacs sflati (remove all V skyflats)
2) Determine the median counts in each skyflat. This will help you decide which flats to keep and which to reject (see Overview). Also review the exposure times (don't use any shorter than 10sec!).
cc> imstat @sflats fields="image,midpt" >> sflat.median cc> hselect @sflats $I,exptime yes >> sflat.exptime
3) Check to be sure there are no "overlap images". Display the V skyflats in the order they were obtained and blink them. If the stars move so that they don't overlap on sequential images, all should be well. If some of the images do overlap (e.g., you forgot to move the telescope between them), you can only use one (1) of the overlapped images. Pick the one with the counts closest to 9000, and/or with the exposure time closest to 30 sec (shorter exposures may have a little "shutter effect", longer ones tend to have more faint stars). In our example, I did the following:
* If you don't have one already, open an ximtool (image display window) by typing in your IRAF window:cc> !ximtool & cc> set stdimage=imt1024 cc> display sflatV03 1 cc> display sflatV04 2 cc> display sflatV05 3 cc> display sflatV06 4* blink the images (see ACL if you don't know how).
cc> display sflatV07 1 cc> display sflatV13 2 cc> display sflatV14 3 cc> display sflatV15 4* blink the images
* I found that sflatV05 and sflatV06 were overlapped. V05 had median=7245 at EXPTIME=26sec while V06 had median=6734 at EXPTIME=32sec. Tough call -- I used V05. The rest were OK.
4) Delete the names of the images you do not want to use from sflatv. Reasons for rejection could include (a) star overlap, (b) low/high counts, (c) exposure time too short. See ACL if you have any doubts!
cc> emacs sflatv
5) Repeat Steps 3 and 4 for the I skyflats.
6) Process the skyflats. Here we will trim them, zero-correct, and dark-correct them using the task CCDPROC. Ensure all is well with CCDLIST (the processed files should have [TZD]).
cc> cl < dpar.ccdproc_flat cc> ccdproc @sflatv zero=Zero.imh cc> ccdproc @sflati zero=Zero.imh cc> ccdlist @sflatv,@sflati
7) Combine the V skyflats into a single, high signal-to-noise, starless final flat, "FlatV.imh". Ditto for the I skyflats, producing "FlatI.imh". If you want to learn more about the way the stars and cosmic rays get rejected, type "help flatcombine" and focus on the rejection algorithms.
cc> cl < dpar.flatcomb cc> flatcombine @sflatv out=FlatV.imh cc> flatcombine @sflati out=FlatI.imh cc> ccdlist Flat*.imh
8) Display your final flats to be sure they look reasonable. There should be no obvious stars, and the intensities should drop as you move from the center toward the corners. There may be some "bad pixels" scattered here and there that just won't flatten out. Discuss the flats with ACL if there are any questions.
cc> display FlatV.imh 1 cc> display FlatI.imh 2
Updated 2000 Feb 23 - ACL
Overview: Here, we will run CCDPROC on the star (object) images to (a) trim them, (b) subtract "Zero.imh" to remove the bias level (about 975 counts) and the 2-D structure produced when the chip is read out, (c) subtract off the appropriate-temperature dark image, and (d) divide by the appropriate flat field image ("FlatV.imh" for the V star images and "FlatI.imh" for the I star images). The resulting images should be completely processed and ready for photometry!
1) Using FILES, create a list of all the star images you took. For the RR Lyrae program, they should all have an underscore ("_") in them. Check to be sure no spurious images (biases, flats, Zero, Flat*, Dark*) have crept in. If they have, remove their names from the list. IMHEAD the list to ensure you got it right.
cc> files *_.imh >> stars.all cc> emacs stars.all cc> imhead @stars.all
2) Copy and edit "stars.all" to make a list of all the V-filter star images and a list of all the I-filter star images. It should be easy with emacs to run down each list and delete the lines (using "^k") with the wrong filter.
cc> cp stars.all stars.v cc> emacs stars.v (keep only names like "AN_Ser2v3.imh") cc> cp stars.all stars.i cc> emacs stars.i (keep only names like "AN_Ser4i3.imh")
3) Set the parameters for CCDPROC and run CCDPROC separately for the V and I images. Use CCDLIST to ensure that the processing has gone well ([TZDF] for all star images).
cc> cl < dpar.ccdproc_star cc> ccdproc @stars.v zero=Zero.imh flat=FlatV.imh cc> ccdproc @stars.i zero=Zero.imh flat=FlatI.imh cc> ccdlist @stars.all
4) Display several of the V and several of the I images to be sure the processing looks reasonable. The sky should appear pretty uniformly gray (i.e., constant brightness). On some images, there are long, curving features -- we think these are caused by reflections inside the telescope, the light of bright stars glinting off a corner or shiny surface. For now, we gotta live with them. Discuss the images with ACL if there are any questions.
cc> display AN_Ser1v2 1
Updated 2000 Feb 23 - ACL
Once you are satisfied with your processing, email ACL with the directory path of your final images, and a list of all the stars which you observed. I will inspect the images, and if all looks good, I will notify the people who are responsible for doing the photometry on the various stars. These people will copy the images into their own accounts for the photometry.
ACL will make tape copies of the raw and processed images for the archive. If we need to reprocess or re-photometer the data at some time in the future, we can recover the images from the tapes.
ACL will tell you when the images can be deleted from your account.
Updated 2000 Feb 23 - ACL
