|
The Data Record Management System (DRMS) provides software utilities, plus a C Application Programming Interface (C-API), for accessing GDC-SDO data. To obtain a DRMS account please contact burston [at] mps.mpg.de
There are 2 important concepts regarding the DRMS that users should know.
i) It stores the meta-data (such as keywords in FITS headers) in a PostgreSQL database to provide rapid and complex searching through the large volume of data
ii) However, it stores corresponding segments (such as a FITS image) elsewhere, in a Storage Unit Management System (SUMS) which is typically just some large online storage area. For HMI/AIA, SUMS will store the segments in a compressed FITS format with most of its keywords stripped. It only retains the keywords which are absolutely necessary to read the segment (such as SIMPLE, BITPIX, NAXES, NAXIS1, NAXIS2, BSCALE, BZERO plus compression information).
Summarily, GDC-SDO users cannot expect to have acess to self-contained FITS files complete with all header information. Instead the DRMS provides utilities for accessing this information transparently. Here, we mention some of the most useful DRMS sofware tools. The reader is once again encouraged to visit the JSOC web site for additional tools and futher documention.
show_series: a utility to determine which series are avaliable at the GDC-SDO, e.g.
- display help
> show_series --help
- list all series
> show_series
- list series with "hmi" as part of the name
> show_series hmi
- list series with "schunker.MT" as part of the name
> show_series "schunker.MT"
show_info: a utility for accessing meta-data (keywords) and segment (image) information
- display help will produce a long list of useful options
> show_series --help
- count how many records available for the HMI (preliminary or test) dopplergrams
> show_info "hmi_test.V_45s" -c
- display T_REC keyword for the very first available frame
> show_info "hmi_test.V_45s" key=T_REC n=1
- check the online status for the segment (i.e. image) for the first frame: Y => online or NA=> offline
> show_info "hmi_test.V_45s" key=T_REC n=1 -o
- get T_REC keyword for the very last available frame
> show_info "hmi_test.V_45s" key=T_REC n=-1
- get keywords T_OBS, WAVELNTH, check online status for the latest 3 records at the wavelength 1600 A
> show_info "aia_test.lev1[?WAVELNTH=1600?]" n=-3 key=WAVELNTH,T_OBS -o
- get AIA (test) level 1 filename with full path to SUMS disk at time=2010-06-09T13:58:18.57Z
> show_info "aia_test.lev1[2010-06-09T13:58:18.57Z]" -P seg=image_lev1
- get the online location for the 12 minute cadence series for wavelength 171 A. Starting at time=2010-06-16T13:40:00.57Z and displaying for the following 1 hour (1h).
> show_info "aia_test.lev1[2010-06-16T13:40:00.57Z/1h][?WAVELNTH=171?][?mod(T_OBS_INDEX-34,720)<= 12?]" -P seg=image_lev1
The key piece of information here is [?mod(T_OBS_INDEX-34,720)<= 12?] 12 corresponds to the typical cadence of the wavelength you are looking at i.e. 12s in this case, but you will need to change this to e.g. 24 for other wavelengths that have a 24s cadence. The value 720 => 720s = 12 minutes and 34 are numbers you don't need to change. Also note the time in the square brakets "[]" does not have questions marks inside "??" whereas WAVELNTH and mod(blah) are flanked by question marks. The one without question marks implies the prime key which you can see by inspecting the output of "show_info ... -j". For AIA the prime key is T_OBS, but for HMI the prime key is T_REC.
- get the retention time for a particular segment. This is important, as once the retention expires, SUMS will automatically delete it
> show_info mps_schunker.Ic_test n=1 -R
Mapping Tracking Series: some info on contents of some series
Some series do not indicate whether they are MDI or HMI.
HMI Intensity cube: mps_schunker.Ic_512x512x33
HMI Doppler cube: mps_schunker.MT_1024x1024x1441 (big fat sunspot, yes I know it is not 24 hours)
> show_info mps_schunker.MT_100x100x101 -o
Tells you how many data arrays are online (there may be records which do not necessarily have corresponding data online).
>show_info mps_schunker.MT_100x100x101 n=2
n is the number of records to show info for (can be negative to count from the end)
>show_info mps_schunker.Ic_512x512x33 n=10 -A
shows the segname (or name of the fits file you will want to get)
>show_info mps_schunker.Ic_512x512x33 n=10 -P
shows the path name to all of the data in the series
>show_info mps_schunker.Ic_512x512x33 n=1 -a
lists all the keywords
>show_info mps_schunker.Ic_512x512x33 n=1 key=T_OBS_FI
lists the value of the specified keyword (in this case, T_OBS
>show_info mps_schunker.Ic_512x512x33[2010.04.24_23:33:00_TAI] -S
Tells you the storage unit number - this is unique for each record and is quite useful.
Reading from IDL
Download fitsio_read_image.pro and fitsio.so . Edit the ".pro" by changing the "LIB = " directory to point to wherever you have saved the "fitsio.so" file.
You can run this in IDL like this:
img = fitsio_read_image(filename [, hd] [, /single | /double] )
e.g.
IDL> vel = fitsio_read_image("/SUM4/D281474977119287/S00000/Dopplergram.fits",hd)
IDL> aia1600 = fitsio_read_image("/SUM2/D50000662/S00000/image_lev1.fits",hd)
Here are some usage comments from Keh-Cheng (modified by me):
The "fitsio_read_image" IDL routine will read uncompressed, gzip-compressed, and tile-compressed FITS images
1. Second parameter hdr is optional
2. Keywords /single and /double force the output to be single-precision and double-precision float, respectively
3. FITS checksums in the input are verified
4. If you have a recent version of the ASTRON IDL library installed and the fpack/funpack utility (part of the CFITSIO library distribution) is on your PATH, you can also use mrdfits() or readfits() to read tile-compressed FITS files produced at JSOC. The calling sequence is
image = mrdfits(filename, 1, ...)
image = readfits(filename, exten_no=1, /fpack, ...)
where ... is whatever positional or keyword parameters you normally use with those routines. However, I do not recommend them for reading DRMS/SUMS FITS files for two reasons: they are slow, and they convert integer BLANK values to normal floating-point values (instead of NaN) in the presence of BSCALE/BZERO. However, you might find them perfectly acceptable
------------------------------------------------------
Here are some other functions for use in IDL to read the array and it's header from the DRMS provided you know the series (and storage unit number) you need (examples included at the top of the codes).
read_array_from_drms.pro
read_head_from_drms.pro
|
