This page was created by the IDL library routine `mk_html_help2`.

Last modified: Tue Aug 5 18:17:40 2025.

# Directory Listing of Routines

# general/spedas_tools/spd_download

- SPD_CDF_CHECK_DELETE
  Renames or deletes cdf or netcdf files, if they can't be opened.
- SPD_COPY_FILE
- SPD_DOWNLOAD
- SPD_DOWNLOAD_CALLBACK
- SPD_DOWNLOAD_EXPAND
- SPD_DOWNLOAD_EXTRACT
- SPD_DOWNLOAD_FILE
- SPD_DOWNLOAD_HANDLER
- SPD_DOWNLOAD_MKDIR
- WRAPPER FOR SPD_DOWNLOAD THAT RETURNS LOCAL VERSIONS OF FILES, IF
- SPD_DOWNLOAD_TEMP
- SPD_GET_PROXY
- SPD_NETURL_ERROR2MSG
  returns a list, where the index is the cURL error code and the item is the status message that the code represents;

# Routine Descriptions

## SPD_CDF_CHECK_DELETE

[Next Routine] [List of Routines]

    NAME:
       spd_cdf_check_delete

    PURPOSE:
       Renames or deletes cdf or netcdf files, if they can't be opened.
       This fuction can be used to cleanup downloaded files with problems.
       By default, the files are renamed to: filename + '.todelete'

    INPUT:
       filenames: Array of cdf or netcdf filenames (full path).
       (filename extension is expected to be '.nc' for netcdf and '.cdf' for cdf files)

    KEYWORDS:
       iscdf: Force cdf check.
       isnetcdf: Force netcdf check.
       (if both keywords are used, netcdf will be prefered)
       delete_file: The file will be deleted.

    OUTPUT:
       Array of renamed or deleted files.
       Renamed files have the file extension '.todelete'.

    EXAMPLES:
       deleted_files = spd_cdf_check_delete(["file1.cdf", "file2.nc"], /delete)

    HISTORY:
    $LastChangedBy: nikos $
    $LastChangedDate: 2021-06-10 14:39:13 -0700 (Thu, 10 Jun 2021) $
    $LastChangedRevision: 30039 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_cdf_check_delete.pro $

(See general/spedas_tools/spd_download/spd_cdf_check_delete.pro)

## SPD_COPY_FILE

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_copy_file

    Purpose:
      Primarily a helper function for spd_download.  This function will
      copy a single file and return the path to the copy.  Existing files
      will only be overwritten if the source is newer.

    Calling Sequence:
      path = spd_copy_file( source=source, destination=destination
                           [,no_update=no_update] [,force_copy=force_copy] )

    Input:
      source:  Path to source file
      destination:  Path to destination file
      no_update:  Flag to not overwrite existing file
      force_copy:  Flag to always overwrite existing file
      file_mode:  Bit mask specifying permissions for new files (see file_chmod)
      dir_mode:  Bit mask specifying permissions for new directories (see file_chmod)

    Output:
      return value:  Full path to destination file if successful, empty string otherwise

    Notes:
      -precedence of boolean keywords:
         force_download > no_update > default behavior

    $LastChangedBy: egrimes $
    $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
    $LastChangedRevision: 22594 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_copy_file.pro $

(See general/spedas_tools/spd_download/spd_copy_file.pro)

## SPD_DOWNLOAD

[Previous Routine] [Next Routine] [List of Routines]

    Function:
      spd_download

    Purpose:
      Download one or more remote files and return their local paths.
      This function can be used in place of file_retrieve.

    Calling Sequence:

        paths = spd_download( [remote_file=remote_file] | [,remote_path=remote_path]
                              [,local_file=local_file]  [,local_path=local_path] ... )

    Example Usage:

      Download file to current directory
      ------------------------------------
        ;"file.dat" will be placed in current directory
        paths = spd_download( remote_file='http://www.example.com/file.dat' )

      Download file to specific location
      ------------------------------------
        ;remote file downloaded to "c:\data\file.dat"
        paths = spd_download( remote_file='http://www.example.com/file.dat', $
                              local_path='c:\data\')

      Download multiple files to specified location and preserve remote file structure
      -------------------------------------
        ;remote files downloaded to "c:\data\folder1\file1.dat"
        ;                           "c"\data\folder2\file2.dat"
        paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $
                              remote_path = 'http://www.example.com/', $
                              local_path = 'c:\data\')

    Input:

      Specifying File names
      ---------------------
        remote_file:  String or string array of URLs to remote files
        remote_path:  String consisting of a common URL base for all remote files

          -The full remote URL(s) will be:  remote_path + remote_file
          -If no url scheme is recognized (e.g. 'http') then it is
           assumed that the url is a path to a local resource (e.g. /mount/data/)

        local_file:  String or string array of local destination file names
        local_path:  String consisting of a common local path for all local files

          -The final local path(s) will be:  local_path + local_file
          -The final path(s) can be full or relative
          -If local_file is not set then remote_file is used
          -If remote_file is a full URL then only the file name is used

          -Remote file names may contain wildcards:  *  ?  [  ]

      Download Options
      ---------------------
        last_version:  Flag to only download the last in file in a lexically sorted
                       list when multiple matches are found using wildcards
        no_update:  Flag to not overwrite existing file
        force_download:  Flag to always overwrite existing file
        no_download:  Flag to not download remote files

        user_agent:  Specifies user agent reported to remote server. (string)

        file_mode:  Bit mask specifying permissions for new files (see file_chmod)
        dir_mode:  Bit mask specifying permissions for new directories (see file_chmod)

        no_wildcards: assume no wild cards in the requested url/filename

        ssl_verify_peer: set this to verify the authenticity of the peer's SSL certificate (HTTPS)
        ssl_verify_host: set this to verify the authenticity of the server certificate (HTTPS)

        disable_cdfcheck: Useful for large files

      IDLnetURL Properties
      ---------------------
        All IDLnetURL properties may be specified as keywords to this routine.
        See "IDLnetURL Properties" in the IDL documentation for more information.
        Some examples are:
        ------------------
          url_query
          url_username
          url_password
          url_port
          proxy_hostname
          proxy_username
          proxy_password

      Other
      ---------------------
        progress_object:  Object reference for status updates

      Backward Support
      ---------------------
        local_data_dir:  mapped to local_path
        remote_data_dir:  mapped to remote_path
        no_server:  mapped to no_download
        no_clobber:  mapped to no_update
        valid_only:  only return paths to files that exist (ideally this would be default)

    Output:
      return value:  String array specifying the full local path to all requested files

    Notes:
      -unsupported file_retrieve keywords:
         progress, preserve_mtime, progobj, ignore_filesize,
         ignore_filedate, archive_ext, archive_dir, min_age_limit

    $LastChangedBy: nikos $
    $LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $
    $LastChangedRevision: 27093 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download.pro $

(See general/spedas_tools/spd_download/spd_download.pro)

## SPD_DOWNLOAD_CALLBACK

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_ui_download_callback

    Purpose:
      A callback function for the idlneturl object as used by spd_download_file

    Calling Sequence:
      N/A - function name set as idlneturl object's callback_function property

    Input/Output:
      status:  See "Using Callbacks with the IDLnetURL Object" in IDL documentation
      progress:  See "Using Callbacks with the IDLnetURL Object" in IDL documentation
      data:  Custom data structure for passing variables from spd_download_file
             to this function.
              {
                net_object:   reference to current idlneturl object
                msg_time:  pointer to time of the last status message
                msg_data:  bytes transferred as of msg_time
                progress_object:  reference to applicable status output object
                error: pointer to flag denoting whether an error occurred in this code
                       (to be used by handler later)
               }

    Output:
      return_value:  1 if everything is OK, 0 if operation should be canceled

    Notes:

    $LastChangedBy: egrimes $
    $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
    $LastChangedRevision: 22594 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_callback.pro $

(See general/spedas_tools/spd_download/spd_download_callback.pro)

## SPD_DOWNLOAD_EXPAND

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_download_expand

    Purpose:
      Check remote host for requested files and apply wildcards
      by downloading and parsing remote index file.

    Calling Sequence:
      spd_download_extract, url, last_version=last_version

    Input:
      url:  String array of URLs to remote files.
      last_version:  Flag to only use the last file in a lexically sorted list
                     when wildcards result in multiple matches.

    Output:
      url:  String array of matching remote files or empty string if no matches are found.

    Notes:

    $LastChangedBy: nikos $
    $LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $
    $LastChangedRevision: 27093 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_expand.pro $

(See general/spedas_tools/spd_download/spd_download_expand.pro)

## SPD_DOWNLOAD_EXTRACT

[Previous Routine] [Next Routine] [List of Routines]

    Function:
      spd_download_extract

    Purpose:
      Helper function to parse <a>(link) tags from html index files.

    Calling Sequence:
      return_value =  spd_download_extract(string [,/relative] [,/normal]
                                                  [,no_parent_links=no_parent_links])

    Input:
      string_array:  String array containing the html index file to be parsed
      relative:  Set to strip out everything but the filename from a link
      normal:  Set to links that don't have '*' or '?'
               (don't think this should every actually happen, but option retained just in case)
      no_parent_links:  Set to the parent domain to automatically exclude backlinks to the parent directory

    Output:
      return_value:  An empty string or array of strings with link destinations

    Notes:
      Copied from file_http_copy subroutine extract_html_links_regex, original notes below:

         "The _regex version of this routine is replacing the original version because
          the old version made assumptions about the formatting of the .remote-index.html file
          that were dependent upon the type of web server that was producing the file.  We think that
          these bugs took so long to show up because Apache servers are extremely common.
          Modification prompted so that file_http_copy can work more reliably rbspice & rb-emfisis
          New version:
          #1 Handles html that doesn't place the href attribute exactly one space after the link tag
          #2 Handles cases where the server doesn't include newlines, or where multiple links are
             included per line of returned html by the server"

    $LastChangedBy: aaflores $
    $LastChangedDate: 2015-02-18 16:27:58 -0800 (Wed, 18 Feb 2015) $
    $LastChangedRevision: 17004 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_extract.pro $

(See general/spedas_tools/spd_download/spd_download_extract.pro)

## SPD_DOWNLOAD_FILE

[Previous Routine] [Next Routine] [List of Routines]

    Function:
      spd_download_file

    Purpose:
      Primarily a helper function for spd_download.  This function will download
      a single file and return the path to that file.  If a local file exists it
      will only download if the remote file is newer.

    Calling Sequence:
      path = spd_download_file(url=url, [,filename=filename] ... )

    Input:
      url:  URL to remote file. (string)
      filename:  Specifies local file path and name.  If a full path is not
                 provided it will assumes a relative path. (string)

      user_agent:  Specifies user agent reported to remote server. (string)
      headers:  Array of HTML headers to be sent to remote server.
                Other IDLneturl properties are passed through _extra but
                this one is extracted so that defaults can be appended:
                  -"User-Agent" is added by default
                  -"If-Modified-Since" is added if file age is checked

      no_update:  Flag to not overwrite existing file
      force_download:  Flag to always overwrite existing file
      string_array:  Flag to download remote file and load into memory as an
                     array of strings.

      min_age_limit:  Files younger than this (in seconds) are assumed current

      file_mode:  Bit mask specifying permissions for new files (see file_chmod)
      dir_mode:  Bit mask specifying permissions for new directories (see file_chmod)

      progress_object:  Status update object
      disable_cdfcheck: Useful for large files
      _extra:  Any idlneturl property (except callback_*) can be passed via _extra

    Output:
      return value:
        local file path (string) - if a file is downloaded or a local file is found
        empty string (string) - if no file is found
        file contents (string array)  - if /string_array is set

    Notes:
      -precedence of boolean keywords:
         string_array > force_download > no_update > default behavior
      -checks contents of "http_proxy" environment variable for proxy server

    $LastChangedBy: nikos $
    $LastChangedDate: 2024-11-20 07:53:04 -0800 (Wed, 20 Nov 2024) $
    $LastChangedRevision: 32966 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_file.pro $

(See general/spedas_tools/spd_download/spd_download_file.pro)

## SPD_DOWNLOAD_HANDLER

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_download_handler

    Purpose:
      Handle errors thrown by the IDLnetURL object used in spd_download_file.
      HTTP responses will be caught and handled.  All other exceptions will
      be reissued for a higher level error handler to catch.

    Calling Sequence:
      spd_download_handler, net_object=net_object, url=url, filename=filename

    Input:
      net_object:  Reference to IDLnetURL object
      url:  String specifying URL of remote file
      filename:  String specifying path (full or partial) to (requested) local file
      callback_error:  Flag denoting that an exception occurred in the callback function

    Output:
      None

    Notes:

       egrimes, 3/20/17 - no longer reissuing last error, now meaningful printing error
                          msg from spd_neturl_error2msg map

    $LastChangedBy: egrimes $
    $LastChangedDate: 2017-03-20 10:49:45 -0700 (Mon, 20 Mar 2017) $
    $LastChangedRevision: 22996 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_handler.pro $

(See general/spedas_tools/spd_download/spd_download_handler.pro)

## SPD_DOWNLOAD_MKDIR

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_download_mkdir

    Purpose:
      Manually create any non-existent directories in a local path
      so that file permissions can be set.

    Calling Sequence:
      spd_download_mkdir, path, mode

    Input:
      path:  Local path (full or relative) to the requested file's destination.
             The filename is assumed to be included.
      mode:  Bit mask specifying permissions for any created directories.
             See file_chmod documentation for more information.

    Output:
      error:  Flag output from recursive calls denoting that no
              directories should be created.

    Notes:

    $LastChangedBy: egrimes $
    $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
    $LastChangedRevision: 22594 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_mkdir.pro $

(See general/spedas_tools/spd_download/spd_download_mkdir.pro)

## SPD_DOWNLOAD_TEMP

[Previous Routine] [Next Routine] [List of Routines]

    Function:
      spd_download_temp

    Purpose:
      Create a random numeric filename suffix for temporary files.

    Calling Sequence:
      suffix = spd_download_temp()

    Output:
      Returns 12 digit numeric string preceded by a period

        e.g. ".555350461348"

    Notes:

    $LastChangedBy: jimm $
    $LastChangedDate: 2017-03-08 16:06:47 -0800 (Wed, 08 Mar 2017) $
    $LastChangedRevision: 22927 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_temp.pro $

(See general/spedas_tools/spd_download/spd_download_temp.pro)

## SPD_GET_PROXY

[Previous Routine] [Next Routine] [List of Routines]

    Procedure:
      spd_get_proxy

    Purpose:
      Gets "http_proxy" environment variable and adds parsed proxy
      properties to an input structure for use with idlneturl.

    Calling Sequence:
      spd_get_proxy, structure

    Arguments:
      structure: Structure to which the idlneturl proxy properties will be appended.
                 Existing properties will not be overwritten.  If undefined or not
                 valid then a structure with said properties will be returned.

    Notes:

    $LastChangedBy: aaflores $
    $LastChangedDate: 2015-08-28 14:36:40 -0700 (Fri, 28 Aug 2015) $
    $LastChangedRevision: 18666 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_get_proxy.pro $

(See general/spedas_tools/spd_download/spd_get_proxy.pro)

## SPD_NETURL_ERROR2MSG

[Previous Routine] [Next Routine] [List of Routines]

     FUNCTION:
         spd_neturl_error2msg

     PURPOSE:
         returns a list, where the index is the cURL error code and the item is the status message that the code represents;
         useful for finding meaningful download error messages

     $LastChangedBy: egrimes $
     $LastChangedDate: 2017-03-20 08:55:34 -0700 (Mon, 20 Mar 2017) $
     $LastChangedRevision: 22990 $
     $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_neturl_error2msg.pro $

(See general/spedas_tools/spd_download/spd_neturl_error2msg.pro)

## WRAPPER FOR SPD_DOWNLOAD THAT RETURNS LOCAL VERSIONS OF FILES, IF

[Previous Routine] [List of Routines]

    NAME:
     wrapper for spd_download that returns local versions of files, if
     they are available. SPd_download only will return a local version if
     the file is specified without wildcards.
    CALLING SEQUENCE:
     paths = spd_download_plus( [remote_file=remote_file] | [,remote_path=remote_path]
                              [,local_file=local_file] [,local_path=local_path] ... )
    Example Usage:

      Download file to current directory
      ------------------------------------
        ;"file.dat" will be placed in current directory
        paths = spd_download( remote_file='http://www.example.com/file.dat' )

      Download file to specific location
      ------------------------------------
        ;remote file downloaded to "c:\data\file.dat"
        paths = spd_download( remote_file='http://www.example.com/file.dat', $
                              local_path='c:\data\')

      Download multiple files to specified location and preserve remote file structure
      -------------------------------------
        ;remote files downloaded to "c:\data\folder1\file1.dat"
        ;                           "c"\data\folder2\file2.dat"
        paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $
                              remote_path = 'http://www.example.com/', $
                              local_path = 'c:\data\')

    Input:

      Specifying File names
      ---------------------
        remote_file:  String or string array of URLs to remote files
        remote_path:  String consisting of a common URL base for all remote files

          -The full remote URL(s) will be:  remote_path + remote_file
          -If no url scheme is recognized (e.g. 'http') then it is
           assumed that the url is a path to a local resource (e.g. /mount/data/)

        local_file:  String or string array of local destination file names
        local_path:  String consisting of a common local path for all local files

          -The final local path(s) will be:  local_path + local_file
          -The final path(s) can be full or relative
          -If local_file is not set then remote_file is used
          -If remote_file is a full URL then only the file name is used

          -Remote and local file names may contain wildcards:  *  ?  [  ]

      Download Options
      ---------------------
        last_version:  Flag to only download the last in file in a lexically sorted
                       list when multiple matches are found using wildcards
        no_update:  Flag to not overwrite existing file
        force_download:  Flag to always overwrite existing file
        no_download:  Flag to not download remote files

        user_agent:  Specifies user agent reported to remote server. (string)

        file_mode:  Bit mask specifying permissions for new files (see file_chmod)
        dir_mode:  Bit mask specifying permissions for new directories (see file_chmod)

        no_wildcards: assume no wild cards in the requested url/filename

      IDLnetURL Properties
      ---------------------
        All IDLnetURL properties may be specified as keywords to this routine.
        See "IDLnetURL Properties" in the IDL documentation for more information.
        Some examples are:
        ------------------
          url_query
          url_username
          url_password
          url_port
          proxy_hostname
          proxy_username
          proxy_password

      Other
      ---------------------
        progress_object:  Object reference for status updates

      Backward Support
      ---------------------
        local_data_dir:  mapped to local_path
        remote_data_dir:  mapped to remote_path
        no_server:  mapped to no_download
        no_clobber:  mapped to no_update
        valid_only:  only return paths to files that exist (ideally this would be default)

    Output:
      return value:  String array specifying the full local path to all requested files

    Notes:
      -unsupported file_retrieve keywords:
         progress, preserve_mtime, progobj, ignore_filesize,
         ignore_filedate, archive_ext, archive_dir, min_age_limit
      -Note that the program still only returns the local file if a)
       the local_file keyword is set, or the local_path and remote_file
       keywords are set.
    $LastChangedBy: jimm $
    $LastChangedDate: 2017-02-06 15:51:25 -0800 (Mon, 06 Feb 2017) $
    $LastChangedRevision: 22741 $
    $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/general/spedas_tools/spd_download/spd_download_plus.pro $

(See general/spedas_tools/spd_download/spd_download_plus.pro)