Accessing Data from the Data Record Management System
You require a Data Record Management System (DRMS) account to access data from the GDC-SDO. To request a DRMS account please contact zhichao [at] mps.mpg.de. The Data Record management System (DRMS) was developed by Stanford University. It provides utilities, libraries, and a Application Programming Interface written in C (C-API) for accessing GDC-SDO data.
Things you need to know
-
log onto one of the seismo interactive machines (e.g., seismo16) to use the Data Record
Management System
-
the meta-data (i.e. keywords commonly found in FITS headers) are
separated from the FITS images. Thus, you will not have acess to
self-contained FITS files complete with all header information. You
will use the DRMS for accessing this information.
-
HMI and AIA FITS images are compressed. Popular coding languages
can easily read compressed images directly.
- Please have a look at SUMS Activity Report for precautions.
Examples below demonstrate how to access data from the DRMS directly from
- command-line utilities; all users must know how to use these
- IDL (see bottom of this page)
- MATLAB
If you want to access DRMS data using languages not listed, please contact zhichao [at] mps.mpg.de
DRMS command line utilities
show_series: a DRMS utility to determine which data series are available
- display help
> show_series --help
- list all series
> show_series
- list all series containing "hmi" as part of the name
> show_series "hmi"
- list all series containing "aia" as part of the name
> show_series "aia"
show_info: a DRMS utility for accessing the meta-data (i.e. keywords) and segment (i.e. FITS image) information
- display help will produce a long list of useful options
> show_info --help
- print "JSOC Series Definition" for e.g. aia.lev1 series. This is a very useful command to learn about a particular DRMS series. It will list all the all the keywords available along with a short description of each. It also lists the segment(s) (i.e. image) information for the series such as the segment name (required for searching) as well as the size of the size and compression information.
> show_info "aia.lev1" -j
- count how many records are available for the HMI LOS Magnetograms
> show_info "hmi.M_45s" -c
- display the T_REC keyword for the very first available frame for the Dopplergrams
> show_info "hmi.V_45s" key=T_REC n=1
- check the online status for the segment (i.e. FITS image) for the first frame: Y => online or NA=> offline
> show_info "hmi.V_45s" key=T_REC n=1 -o
- T_REC keyword for the very last available frame
> show_info "hmi.V_45s" key=T_REC n=-1
- keywords T_REC, WAVELNTH, check online status for the latest 3 records at the wavelength 1600 A
> show_info "aia.lev1[?WAVELNTH=1600?]" n=-3 key=WAVELNTH,T_REC -o
- online location of HMI Dopplergram FITS image at specified date/time
> show_info "hmi.V_45s[2010.12.13_23:53:15_TAI]" -P seg=Dopplergram
- retention time for a particular segment (i.e. FITS image); once the retention expires, SUMS will delete image.
> show_info "hmi.V_45s[2010.12.13_23:53:15_TAI]" -R
DRMS command line utilities to access 12-minute cadence Level-1 FITS images
The DRMS utility show_info is used to get the locations of the 12-min cadence FITS images. It is suggested you bookmark this page and cut-and-paste everytime you need this command. Try the following example
> show_info "aia.lev1[2010-12-01T00:00:01Z/2h][?WAVELNTH=131?][?mod(cast(T_REC as bigint)-34,720)<=12?]" -P seg=image_lev1 key=T_REC,WAVELNTH
which gives you a list containing
- full paths to the level-1 FITS images for which the above command requires the "-P" flag and the "seg=image_lev1" option (seg is short for segment or FITS image). To determine the segment name (image_lev1 in this case) of various series, print the jsoc series definition with "show_info aia.lev1 -j" and look at the available segments.
- T_REC time-stamps for each segment (image) are listed by specifing key=T_REC. The description of T_REC is also in the jsoc series definition (see it is listed as a primary key)
- a time range that is determined by the first "square-bracket" search criteria [2010-12-01T00:00:01Z/2h]. It starts from a date/time of 2010-12-01T00:00:01Z and ends after 2 hours, hence the /2h. Other options include m for minute and h for hour. You can also specify start and end times such as [2010-12-01T00:00:01Z-2010-12-01T01:48:11Z]. Note the time criteria in the square brakets is not flanked by question marks "[?..?]" whereas the others (to be described shortly do). This is because it is a primary key which can be seen in the jsoc series definition.
- the wavelengths are listed by specifying key=WAVELNTH. In addition, we have chosen to select only the 131 A wavelength by including a search criteria of [?WAVELNTH=131?]. If you are interested in a different wavelength, then choose a different number, or if you are interested in several wavelengths, then you can do something like [?WAVELNTH=171 and WAVELNTH=193?] for 171 A and 193 A.
- the criteria for selecting the 720 s (i.e 12-min) cadence is [?mod(cast(T_REC as bigint)-34,720)<=12?]]; 34 is a number to get images starting near the beginnning of each hour (for the 1600A and 1700A data change this to 44); 720 corresponds to the 720s (12min) cadence; 12 corresponds to the full cadence in seconds of the wavelength you are looking at i.e. 12s for 131 A
- For the 1600A, 1700A, or 4500A data, the command has to be modified to reflect that these series have a full cadence of 24s, 24s, and 3600s respectively. Also, the value 34 changes to 44 for the 1600 A and 1700 A wavelengths.
Once you are comfortable with all the various options, the most useful command may be to produce a list containing just the paths to the segemets (FITS images), which can then be easily read into any scientific code you many have. To do this, you would omit "key=T_REC,WAVELNTH" from the above command and include the "-q" flag (quiet) so list titles are printed to screen. The screen output can then easily be redirected to some text file.
> show_info "aia.lev1[2010-12-01T00:00:01Z/2h][?WAVELNTH=131?][?mod(cast(T_REC as bigint)-34,720)<=12?]" -qP seg=image_lev1 > aia.lev1.FITS.files.list.txt
Access DRMS from within your IDL code
You can access the DRMS directly from within an IDL code by spawning show_info and show_series.
If you know the full paths to the compressed DRMS data that you are interested in (e.g. via show_info ... -PA command), then you can use the following IDL function to read the compressed FITS file into an IDL array
array = fitsio_read_image(filename [, hdr] [, /single | /double] [, /chksum])
The following sequence of commands are examples for the compressed AIA and HMI data
- log onto one of the seismo interactive machines e.g. seismo1.mps.mpg.de, seismo16.mps.mpg.de
- Download the following files: fitsio_read_image.pro and fitsio.so and make sure they are in the same directory.
- AIA example: open a command consol and type the following
> setup idl
> idl
IDL> .r fitsio_read_image.pro
IDL> array=fitsio_read_image("/SUM41/D114565302/S00000/image_lev1.fits",hdr); read AIA Level-1 image
IDL> array[(array ne array)] = -1000.; For plotting you may or may not need to set any any NaN values "where or if they exist" to some value of your choosing. This is typically not required for AIA level-1 data since they have not been cropped, however, HMI has been cropped and replaces missing values with NaNs values.
IDL> tvscl, rebin(array,512,512); rebin the image so it fits in a display window on your screen and plot
IDL> print, hdr; display header information
Please contact zhichao [at] mps.mpg.de if you have any problems.
An alternative method to read compressed images. If you have a recent version of the ASTRON IDL library installed and the fpack/funpack utility (part of the CFITSIO library distribution) is on your PATH, you can also use mrdfits() or readfits() to read tile-compressed FITS files produced at JSOC. The calling sequence is
image = mrdfits(filename, 1, ...)
image = readfits(filename, exten_no=1, /fpack, ...)
where ... is whatever positional or keyword parameters you normally use with those routines. However, I do not recommend them for reading DRMS/SUMS FITS files for two reasons: they are slow, and they convert integer BLANK values to normal floating-point values (instead of NaN) in the presence of BSCALE/BZERO. However, you might find them perfectly acceptable.
Here are some other functions for use in IDL to read the array and it's header from the DRMS provided you know the series (and storage unit number) you need (examples included at the top of the codes).
An alternative method to read compressed images. If you have a recent version of the ASTRON IDL library installed and the fpack/funpack utility (part of the CFITSIO library distribution) is on your PATH, you can also use mrdfits() or readfits() to read tile-compressed FITS files produced at JSOC. The calling sequence is
image = mrdfits(filename, 1, ...)
image = readfits(filename, exten_no=1, /fpack, ...)
where ... is whatever positional or keyword parameters you normally use with those routines. However, I do not recommend them for reading DRMS/SUMS FITS files for two reasons: they are slow, and they convert integer BLANK values to normal floating-point values (instead of NaN) in the presence of BSCALE/BZERO. However, you might find them perfectly acceptable.