Several tools developed at STScI, to support planning and/or analysis of HST or JWST observations, use a wide variety of reference data that is carefully verified and validated previously to archiving them and ingesting them into all the systems. The INS Reference Data for Calibration and Tools (ReDCaT) Management Team works closely with instrument teams and developers to define, manage, and distribute all these files as well as to certify their correctness and suitability to be used with other related systems or by other tools or instruments that might benefit from them.
The process of creating quality instrument specific reference files starts with the Instrument Teams, as they are responsible for the various forms of calibration data pertinent to their instrument. Following a delivery protocol, the teams create, validate, test, and submit the reference files to the ReDCaT Team, which in the process of preparation for delivery to the Archive, Operations, and other data analysis tools, performs additional validation. The result is the ingest of compliant reference files into the Calibration Reference Data System (CRDS) and the Tools Reference Data System (TRDS).
CRDS is a metadata-based system that maps datasets to data using mapping rules. The calibration pipelines, pre-archive processing, and reprocessing of HST and JWST observations, use CRDS to identify the best reference files for the data. They can be retrieved directly using CRDS or retrieved from the STScI Archive for recalibration at a user's home site.
TRDS, on the other hand, provides configuration management for the data used by other analysis tools. It currently supports reference files used by HST ETC, JWST ETC, pysynphot and the user community. The files within TRDS can include instrument specific files, cross instrument files, a large set of stellar and galaxy atlases or models, as well as a suite of calibration standards. The ReDCaT Team carefully evaluates them for their applicability to all the tools and instruments that use them and documents their provenance and metadata.
This page provides information for the reference files delivered to CRDS or TRDS.
Calibration Reference Files
Each instrument has reference files with a specific format. The ICD-47 document provides with the description of all the HST reference files. In the case of JWST, the JWST Calibration Pipeline documentation offers this information. Users can retrieve the reference files via the online CRDS pages (HST, JWST)] or the CRDS command line program, bestrefs. CRDS is installed implicitly with the conda installation of HST or JWST calibration software. For HST the CRDS bestref command updates the dataset headers with the best references for a given context (pmap file), the default is the current context. In the case of JWST, the calibration step that uses a reference file calls the bestref function to find the best match.
Retrieving Reference Files via the HST Archive Interface
If you need to request individual or multiple files from the MAST archive website, use the first nine characters of the file name followed by an asterisk, or the entire unique name given on the SI web pages linked to below. Retrieving the reference files from MAST requires overriding the retrieval options. That is, once in the "Retrieval Options" window, you must override the defaults by entering the extension(s) of the file(s) requested. For example, if you are retrieving a bias reference file (with extension bia), enter "bia" in the box "or enter a specific extension:" and then submit your request.
Retrieving Reference Files via CRDS
If you would like to use the CRDS Tools to get individual or multiple files, you can follow these steps:
- For HST reference files, go to hst-crds.stsci.edu
- For JWST reference files, go to jwst-crds.stsci.edu
- Click on an instrument
- Click on a reference file type
- Use the Search box on the right to filter the results
- Click on a filename (Do not right click on the filename) you will be taken to another page
- On this page, click on 'download.'
Alternatively, you can download one or all cached reference files for one or all the instruments via the CRDS command line. For this follow the instructions at HST CRDS Command Line for HST instruments or JWST CRDS Command Line for JWST instruments.
Receiving Notification about New Reference Files
If you wish to find out about new reference files for a specific HST instrument, you may subscribe to a mailing list designed specifically for that purpose. The xxx_reffiles_upd mailing lists, (where xxx is the instrument name; acs, cos, stis, or wfc3) are used to send out an e-mail whenever an instrument team delivers a new reference file and the database is updated. To subscribe to any of these lists, send a blank e-mail to listname-subscribe-request @ maillist.stsci.edu (leave the subject and body blank). Example: acs_reffiles_upd-subscribe-request @ maillist.stsci.edu
Tools Reference Files
The ReDCaT Team manages all files used by Pysynphot and HST ETC. These can be classified in four categories:
- Pysynphot Throughput Tables
- Thermal Emissivity Tables
- Pysynphot Master Files
- Astronomical Catalogs
These files are available to the astronomical community for use with these or other tools.