THEMIS
Data Analysis Software User’s Guide
THM-SOC-120
Timothy Quinn, THEMIS Science Operations
Ken Bromund, THEMIS Science SFW
Jim McTiernan, THEMIS Science SFW
Dr. John Bonnell, THEMIS EFI
Dr. Alain Roux, THEMIS SCM
Dr. Uli Auster, THEMIS FGM
Dr. Davin Larson, THEMIS SST
Dr. Jim McFadden, THEMIS ESA
Dr. Harald Frey, THEMIS ASIs
Dr. Hannes Schwarzl, THEMIS Science ACS
Manfred Bester, THEMIS Mission Operations Manager
Vassilis Angelopoulos, THEMIS Principal Investigator
Document Revision Record
|
Rev. |
Date |
Description of Change |
Approved By |
|
1 |
2006-11-13 |
Draft |
- |
|
2 |
2006-12-11 |
Refine naming conventions. Reformat, reorganize. |
|
|
3 |
2006-12-20 |
Document thmsw_0_3b_20061220, incorporate comments received from post-AGU meeting. |
|
|
|
|
|
|
|
|
|
|
|
Distribution List
|
Name |
Email |
|
Jim Lewis, U.C. Berkeley |
jwl@ssl.berkeley.edu |
|
Dr. Tai Phan, U.C. Berkeley |
phan@ssl.berkeley.edu |
|
Dr. Forrest Mozer, UCB |
fmozer@ssl.berkeley.edu |
|
Dr. Robert Ergun, LASP |
ree@fast.colorado.edu |
|
Dr. Robert Lin, UCB |
rlin@ssl.berkeley.edu |
|
Dr. Chris Cully, LASP |
cully@colorado.edu |
|
Dr. Michael Ludlam |
mludlam@ssl.berkeley.edu |
|
Dr. Ellen Taylor, U.C. Berkeley |
ertaylor@ssl.berkeley.edu |
|
Dr. Krishan Khurana, UCLA |
kkhurana@igpp.ucla.edu |
|
Dr. David Sibeck, NASA GSFC |
david.g.sibeck@nasa.gov |
TBD List
|
Identifier |
Description |
Table
of Contents
Document
Revision Record...................................................................................... 2
Distribution List.......................................................................................................... 2
TBD List............................................................................................................................. 2
1. Introduction.......................................................................................................... 4
1.1 Purpose
and Scope..................................................................................................... 4
1.2 Applicable
Documents............................................................................................... 4
2. Structure
of THEMIS data analysis programs......................................... 5
2.1 General
rules.............................................................................................................. 5
2.2 Software
Organization................................................................................................ 5
2.3 List
of key Routines and functions............................................................................. 6
2.4 Availability
of Code and Documentation.................................................................... 7
2.5 Graphical
User Interface............................................................................................. 8
3. User’s
Guide............................................................................................................. 8
3.1 System
Requirements................................................................................................. 8
3.2 How
to get started...................................................................................................... 9
3.2.1 Code
Download......................................................................................................... 9
3.2.2 Installation
and configuration........................................................................................ 9
3.2.3 Data
Access............................................................................................................. 10
3.2.4 Plot
your first plots of ground or space data................................................................... 12
3.3 How
to modify code for your own use..................................................................... 12
3.4 How
to contribute code and modifications............................................................... 13
1. Introduction
1.1 Purpose and Scope.
The THEMIS analysis software is derived from an IDL set of routines that operate on CDFs from existing missions such as Cluster, Wind, Polar and FAST. This software is being re-organized by the THEMIS software developers and scientists, and checked into a software management database—using the freeware Subversion (SVN)—for maintenance, tracking updates and contributions by a distributed team, and ensuring self-compatibility. In addition to the command-line invoked IDL routines, a graphical user interface that prompts the user for opening, analyzing and plotting the THEMIS data is being developed, and is intended to facilitate use of the most useful of the THEMIS routines.
1.2 Applicable Documents.
- THM_SYS_012_PDMP THEMIS Project Data Management Plan
- THM_SOC_101_TIME THEMIS TIME Definition
- THM_SOC_108_GMAG_L1_VARNAMES THEMIS GMAG Variable Name Def’s
- THM_SOC_111_SUNSENSPROC THEMIS SUN SENSOR Science Processing
- THM_SOC_112_ATTPAIPROC THEMIS Science ATT & Inertia Determ.
- THM_SOC_113_FGM_CALPROC THEMIS FGM CAL File and Processing
- THM_SOC_114_SCM_CALPROC THEMIS SCM CAL File and Processing
- THM_SOC_115_EFI_CALPROC THEMIS EFI CAL File and Processing
- THM_SOC_116_ESA_CALPROC THEMIS ESA CAL File and Processing
- THM_SOC_117_SST_CALPROC THEMIS SST CAL File and Processing
- SAI-SPEC-1079A (Oct 26, 2005) THEMIS Coordinate systems
- SAI-RPT-0722a (September, 2006) Probe Alignment Report (MSSS data, p18)
- pturin e-mail on Faro alignment results (9/28/06) FGM, SCM mag alignments
- THM-MB-MEC-005-Magnetometer clocking r7.pdf MAG clocking angles
2. Structure of THEMIS data analysis programs
The programs are IDL-based. They are used to open, analyze and plot L1 or L2 data, process L1 data into L2 files, and also translate data into other products (e.g., ascii) as necessary.
2.1 General rules
Routines operate on (as input and/or output) both arrays and tplot variables (i.e. handles to structures which contain data as well as information about how to display that data in time plots), with an option to do either one (or both).
The naming convention is that all THEMIS-specific routines begin with thm_ followed by the function (action) followed by the type of data operated upon. E.g., thm_load_xxx reads xxx data.
Routines are as general and encompassing as possible; they rely on keywords (or data file name) for branching out to specific functionality.
Some examples of routines which follow the THEMIS naming convention:
|
thm_load_esa |
loads esa data |
|
thm_cal_mom |
calibrates all (ESA and SST) moment data from L1 data to
physical quantities. |
|
thm_pa_sst |
Combines SST psd, and FGM and provides pitch angles dist. |
|
thm_cotrans |
Coordinate transformation from any to any Earth centered,
in particular from GSE to other.
It uses the THEMIS-specific STATE file as an input, but is not
specifically for transformation of any on type of THEMIS data. |
|
thm_crib_ask |
Crib sheet for loading and displaying All Sky Keogram
data. |
2.2 Software Organization
The themis directory is for THEMIS-specific routines. The general directory (currently named ssl_general) is for routines which are useful for more than one space-science mission for which THEMIS team members are developing code. The external directory is for packages developed and maintained by other groups, but which are required for writing (and running) the themis and/or general routines. The distinction between general and external can be blurry…
The THEMIS SOFTWARE distribution is a snapshot of /disks/socware/idl (or perhaps a subset of this, if other instruments are added in the future). The idl directory of the distribution includes the following files and sub-directories.
|
thmsw_doc.html |
Documentation in
HTML format, including alphabetical list of all routines. |
||
|
themis/ |
setup_themis |
Sample setup script for csh (UNIX) |
|
|
ground/ |
routines for loading, processing and plotting ground-based
data. e.g. thm_load_gmag,
thm_gmag_stackplot, imageplot, mosaicplot |
||
|
spacecraft/ |
fields/ |
routines for loading, processing and plotting particles
data, e.g. thm_proc_fgm |
|
|
sst/ |
(Should be particles/) routines for loading, processing and plotting particles
data, e.g. thm_load_sst, thm_cal_sst, thm_pa_sst |
||
|
orbit/ |
Routines for reading state and performing coordinate
transformations, e.g. thm_load_state, thm_despin, thm_ssl2spg, thm_dsl2gse,
thm_cotrans |
||
|
examples/ |
Crib sheets, like thm_crib_ask |
||
|
common/ |
Themis-specific tools useful to multiple data types and
instruments, eg themis_w (The THEMIS GUI) |
||
|
ssl_general/ [should be general/ ?] |
CDF/ |
CDF utilities, including cdf2tplot, and perhaps other utilites like cdf2flat, cdf2ascii, cdf_load (into IDL arrays) |
|
|
analysis/ |
|
||
|
cotrans/ |
Incorporate ROCOTLIB here, or keep it as external? |
||
|
tplot/ |
General time-plotting utilites |
||
|
misc/ |
Miscellaneous routines used by tplot and other ssl_general
routines. |
||
|
orbitplot/ |
General orbit-plotting utility. |
||
|
external/ |
SSW_gen_idl |
direct import of a portion of SSW general IDL routines |
|
|
ROCOTLIB |
direct import of rocotlib – thm_cotrans, etc are
wrappers around this. |
||
|
CDAWeb |
|
||
|
Radars |
|
||
|
facility 1 |
|
||
|
QSAS |
|
||
2.3 List of key Routines and functions
To give a better idea of how the THEMIS IDL programming interface looks, here is the documentation for a few key routines and functions.
See section 2.4 for information on how to get a complete, up-to-date listing.
Loading data:
|
thm_load_esa |
loads esa data |
|
thm_load_efi |
loads efi waveforms |
|
thm_load_fgm |
loads all fgm waveforms (choice of fgl,
fgh…) |
|
thm_load_spec |
loads EFI and SCM spectra |
|
thm_load_gmag |
loads ground mag data for any ground
station (by keyword) or all available.
Keywords to choose high lat and low lat. |
|
thm_load_hsk |
loads housekeeping data – all or one
by keyword. Important for FGM (temperatures) but also other instruments. |
Plotting:
|
tplot |
General purpose time plotting utility for
creating stack plots of waveforms and sonograms. |
|
tlimit |
Zoom into or out of a tplot |
Calibration: from L1 data to physical quantities
|
thm_cal_mom |
calibrates all (ESA and SST) moment data |
|
thm_cal_sst |
calibrates SST data provides counts,
eflux, flux and psd (phase space density or distribution function) data |
|
thm_cal_esa |
same as thm_cal_sst, above, but for ESA
data. |
Beyond calibration: combining physical quantities to create higher level data products:
|
thm_pa_sst |
Combines SST psd, and FGM and provides
pitch angles dist. |
|
thm_pa_esa |
same as above, but combines ESA and FGM. |
Coordinate transformations:
|
thm_despin |
takes us from SSL to DSL, can be also:
thm_ssl2dsl |
|
thm_ssl2spg |
if needed (EFI) to go back to geom. (SPG)
from Spinning Sunsensor L-vec. (SSL) |
|
thm_dsl2gse |
from DSL to one Earth centered (GSE is
easiest) |
|
thm_cotrans |
from any to any Earth centered, in
particular from GSE to other. |
Crib sheets:
|
thm_crib_ask |
Crib sheet for loading and displaying All
Sky Keogram data. |
|
thm_crib_tplot |
Crib sheet for using tplot, using GMAG
data as an example. |
For more information, use:
· Crib sheets included in the idl/themis/examples directory of the software distribution.
· HTML help included with the software distribution
·
IDL XDOC widget.
At the IDL prompt, type:
xdoc
·
IDL doc_library procedure. At the IDL prompt, type:
doc_library, ‘command_name’
Sample THEMIS routine documentation:
THM_LOAD_GMAG
Procedure: THM_LOAD_GMAG,station
keywords:
TRANGE= (Optional)
Time range of interest (2 element
array).
/VERBOSE : set to
output some useful info
/SUBTRACT_AVERAGE ;
Example:
thg_load_mag,'bmls'
Notes:
This routine is (should
be) platform independent.
$LastChangedBy: davin-win $
$LastChangedDate:
2006-12-15 12:29:37 -0800 (Fri, 15 Dec 2006) $
$LastChangedRevision: 107 $
$URL:
svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/thmsoc/trunk/idl/themis/ground/thm_load_gmag.pro
$
2.4 Availability of Code and Documentation.
The latest THEMIS code (with documentation, but not yet automatically updated) is visible at
http://themis.ssl.berkeley.edu/themisdata/socware/bleeding_edge. This is a link to /disks/socware/idl, which is the integration/testing area for the latest data analysis software at UCB SSL. There is no zip file of the latest code. We may implement daily builds of zip files and documentation in the future.
Zipped and tested [ideally!] releases are in separate directories under
http://themis.ssl.berkeley.edu/themisdata/socware
For example, the latest release as of the writing of this document (December 20, 2006) is in:
http://themis.ssl.berkeley.edu/themisdata/socware/thmsw_0_3b_20061220/
Each release folder contains a zip file for
download. The zip file includes
IDL files and HTML documentation.
HTML-browsable documentation is available within each release folder at
idl/thmsw_doc.html
Even more up-to-date and detailed access to
the THEMIS code will be available through a web interface to the SVN repository
(read-only) http:// or svn:// URL to be determined. This is for access to the very latest version of the code,
plus possibly access to the SVN change log.
2.5 Graphical User Interface
The philosophy of the graphical user interface GUI is to provide a convenient interface to the most generally useful capabilities of the command-line THEMIS IDL routines. General users can use the GUI without being required to know the individual IDL routines. However, users who want a jump-start to using the more powerful and customizable IDL programming interface may begin with the GUI and then use the GUI to generate a script of the underlying IDL commands which perform the same function. This script can then be used as a starting point for creating an IDL program to process THEMIS data, or can be used as a crib sheet for using the THEMIS IDL programming interface.
Practice [???]
The GUI code is bundled together with the THEMIS IDL code distribution, they are downloaded and installed as a single package.
3. User’s Guide
3.1 System Requirements
The THEMIS Data Analysis Software requires IDL on Windows or a UNIX-like operating system like Solaris, Linux, or Mac OS X. You must have IDL 6.3 to read the THEMIS CDF data files. Alternatively, you can use an earlier version of IDL, and apply the patch available at http://cdf.gsfc.nasa.gov/html/idl62_or_earlier_and_cdf3_problems.html
Even if you have IDL 6.3, you will want to install the patch if you use IDL to write CDF files. This is because of a bug in IDL 6.3 which will be fixed in 6.4.
For Mac OS X, you must
install the X11 package and start it before you can run IDL. X11 comes with your Mac, but is not
installed by default.
3.2 How to get started
Getting started is simply a matter of downloading the code, setting up your IDL path, and telling the THEMIS SW where put local data (if the default won’t work for you).
3.2.1 Code Download.
You can download the code as described above in Availabililty of Code and Documentation.
3.2.2
Installation and configuration.
Choose the location where you want to unzip the installation.
Unzip the .zip file. It will create a folder with the same name as the zip file. The folder will contain an idl directory.
Installation and configuration consists of setting up your IDL path, creating a local data directory, and telling the THEMIS SW where put local data (if the default won’t work for you).
3.2.2.1 IDL
PATH setup
3.2.2.1.1 IDL
PATH Setup on Windows (and IDLDE on UNIX, Linux and Mac)
For Windows or IDLDE on UNIX, you can use the File->Preferences widget to set up the path so IDL can find the THEMIS IDL files.
UNIX Note: If you use IDLDE on UNIX-like systems, these instructions only work if you do not set the IDL_PATH environment variable before you call IDLDE. If IDLDE does not allow you to set the path by following these instructions, then follow the instructions for UNIX installation, below.
Start IDL (or IDLDE).
Go to File->Preferences
Select the ‘Path’ tab.
If <IDL_DEFAULT> is not present, press ‘Insert Standard Libraries’
Press Insert
Browse to find your installation and select the ‘idl/external’ directory
Check the box to indicate “search subdirectories”
Press Insert
Browse to find your installation and select the ‘idl/ssl_general’ directory
Check the box to indicate “search subdirectories”
Verify that ‘idl_external’ is last in the list.
Press Insert
Browse to find your installation and select the ‘idl/themis’ directory
Check the box to indicate “search subdirectories”
3.2.2.1.2 IDL PATH Setup for IDL Command Line (UNIX, Linux or Mac OS X)
For the command line version of IDL, installation consists of setting up the IDL_PATH environment variable, and a variable which points to your local data directory. The THEMIS Data Analysis Software distribution includes setup scripts which help set the IDL_PATH variable correctly.
If you use csh or tcsh, the recommendation is to put the following in your .cshrc
setenv IDL_BASE_DIR /path/to/thmsw_v_sb_yyyymmdd/idl
source
$IDL_BASE_DIR/themis/setup_themis
If you use bash shell, the recommendation is to put the following in your .bashrc (Linux) or .bash_profile (Mac)
IDL_BASE_DIR=/path/to/thmsw_v_sb_yyyymmdd/idl
.
$IDL_BASE_DIR/themis/setup_themis_bash
In the above examples, IDL_BASE_DIR should be set to the full path name to the idl directory inside the THEMIS Data Analysis Software release.
3.2.2.2 Data Directory Setup
The THEMIS Data Analysis Software requires a local data directory in which THEMIS data files can be cached. The THEMIS will attempt to create the local data directory for you at the following default location, depending on your operating system.
|
OS |
LOCAL_DATA_DIR |
|
Windows |
C:\data\themis |
|
Solaris, Linux, Mac
OS X |
/disks/data/themis |
If you don’t have administrative privileges to create the data directory in the above locations, you can have your system administrator create it for you, or you can configure the THEMIS Data Analysis Software to use an alternate location. See Remote Data Access and Local Data Cache for information about configuring an alternate location.
3.2.3 Data Access
The THEMIS software
will automatically create a local data cache which mirrors the structure of the
THEMIS data archive. The software
is written such that the default settings will work for the majority of users. The location of the THEMIS data archive
is also found automatically by the software.
3.2.3.1 Loading
THEMIS Data
To download some data and load it into IDL, you simply specify a timespan and type of data you want to load. If the data does not exist locally on your local_data_dir, it will be downloaded automatically before it is loaded into IDL.
3.2.3.1.1 Loading Data With the GUI
At the IDL prompt, type
themis_w
Select a timespan, data type, and satellite or ground station. Use the control key to select more than one item the menus. Press the Load button to load the data.
3.2.3.1.2 Loading Data from the IDL Command Line
At the IDL prompt, type:
timespan, ‘2006-11-11’, 1, /day
Then, use one of the thm_load commands, e.g.
thm_load_gmag, ‘ccnv’
3.2.3.2 Managing Your Data Cache
There currently exist
some experimental routines for managing your THEMIS data file cache
- thm_file_download – can be used to
download all types of THEMIS data for a given time span.
- thm_file_cleanup – clean up data
files that have not been accessed in a long time.
3.2.3.3 Configuring
Local Data Cache and Remote Access Locations
If necessary, data locations
may be configured at IDL startup (for example, if you want the data to reside
in a non-standard location). Data
locations may also be changed dynamically during an IDL session.
The following table summarizes the controls which can be used, in order of precedence:
· !THEMIS system variable
· environment variable settings
· themis_environment.pro
The following table summarizes the settings available:
|
!themis structure
element |
Environment Variable
|
Description |
|
local_data_dir |
THEMIS_DATA_DIR |
a writable, local
directory in which to cache data files |
|
remote_data_dir |
THEMIS_REMOTE_DATA_DIR |
URL to a data
archive where THEMIS data can be found. By default, http://sprg.ssl.berkeley.edu/data/themis |
3.2.3.3.1 Configuration at startup
Since environment configuration varies from system, the THEMIS Data Analysis Software includes a file which you can use to define a procedure which will set up your environment from with IDL.
1. Make a copy of the idl/themis/common/themis_environment.pro in a directory outside the THEMIS Data Analysis Software distribution (or edit it in place, if you don’t mind losing your changes when you update your software)
2. Add the directory which contains your copy of themis_environment.pro to your IDL PATH, making sure that it is before the idl/themis directory. (See IDL PATH setup, above).
3. Edit your themis_environment file to set the environment variables to your liking. See the above table for the names of relevant environment variables.
3.2.3.3.2 On-the-Fly
configuration
The local data directory location can be changed on-the-fly by setting the !themis.local_data_dir system variable. For example, when you are connected to the network, you may choose to use a networked data dir:
!themis.local_data_dir = ‘\\justice\data\themis\’
The remote_data_dir can also be updated. For example, if you want the software to use only locally available data, and don’t want the software to try to download data, you can set:
!themis.remote_data_dir = ‘’
Note that you can only set the !themis variable in an IDL session after a thm_ command has already been run. Otherwise you will get an error, because the !themis system variable does not exist yet.
3.2.4 Plot your first plots of ground or space data
In the GUI, press ‘Call Tplot’
You can select the data quantities you want to plot from the ‘Loaded Data’ list.
In the command line, type
tplot_names
to see available data quantities. Type, e.g.,
tplot, 1
to plot the first quantity.
For more details on getting started, see the crib sheet at
themis/examples/thm_crib_tplot.pro
or other crib sheets in the examples directory.
3.3 How to modify code for your own use
This section gives ideas about you can move beyond the generic capability provided by the GUI and crib sheets.
GUI-based analysis
1. Where to go for help
a. xdoc (start it from the IDL> prompt)
b. HTML documentation.
2. How to string multiple operations on a pipeline
3. How to create a new operation and corresponding GUI in your local toolkit
4. How to save pipeline into a crib sheet (transfer to command line based analysis)
a. Use the History and ‘Clear History’ buttons
Command line-based analysis (crib-sheets)
- Where to go for help
- doc_library
- xdoc
- HTML documentation included with the distribution.
- The code itself.
- Modify GUI-generated code
- Modify existing analysis crib sheets
- Write your own analysis module – an IDL routine - that operates on data
- Transfering between data arrays and tplot variables
- Plotting data
- IDL help with routines
3.4 How to contribute code and modifications
If you want to contribute code to the THEMIS IDL SW
distribution, you can email us your routines, or you can become part of the
THEMIS development team and work directly with the SVN repository. We can give you access if you provide
us with your SSH public key. See
the THEMIS SOC SW document for details on working with the THEMIS SVN
repository.