This page was created by the IDL library routine
mk_html_help2.
Last modified: Thu Aug 6 12:53:14 2020.
NAME: spd_cdf_check_delete PURPOSE: Renames or deletes cdf or netcdf files, if they can't be opened. This fuction can be used to cleanup downloaded files with problems. By default, the files are renamed to: filename + '.todelete' INPUT: filenames: Array of cdf or netcdf filenames (full path). (filename extension is expected to be '.nc' for netcdf and '.cdf' for cdf files) KEYWORDS: iscdf: Force cdf check. isnetcdf: Force netcdf check. (if both keywords are used, netcdf will be prefered) delete_file: The file will be deleted. OUTPUT: Array of renamed or deleted files. Renamed files have the file extension '.todelete'. EXAMPLES: deleted_files = spd_cdf_check_delete(["file1.cdf", "file2.nc"], /delete) HISTORY: $LastChangedBy: nikos $ $LastChangedDate: 2019-04-29 12:00:37 -0700 (Mon, 29 Apr 2019) $ $LastChangedRevision: 27131 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_cdf_check_delete.pro $
(See general/spedas_tools/spd_download/spd_cdf_check_delete.pro)
Procedure:
spd_copy_file
Purpose:
Primarily a helper function for spd_download. This function will
copy a single file and return the path to the copy. Existing files
will only be overwritten if the source is newer.
Calling Sequence:
path = spd_copy_file( source=source, destination=destination
[,no_update=no_update] [,force_copy=force_copy] )
Input:
source: Path to source file
destination: Path to destination file
no_update: Flag to not overwrite existing file
force_copy: Flag to always overwrite existing file
file_mode: Bit mask specifying permissions for new files (see file_chmod)
dir_mode: Bit mask specifying permissions for new directories (see file_chmod)
Output:
return value: Full path to destination file if successful, empty string otherwise
Notes:
-precedence of boolean keywords:
force_download > no_update > default behavior
$LastChangedBy: egrimes $
$LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
$LastChangedRevision: 22594 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_copy_file.pro $
(See general/spedas_tools/spd_download/spd_copy_file.pro)
Function:
spd_download
Purpose:
Download one or more remote files and return their local paths.
This function can be used in place of file_retrieve.
Calling Sequence:
paths = spd_download( [remote_file=remote_file] | [,remote_path=remote_path]
[,local_file=local_file] [,local_path=local_path] ... )
Example Usage:
Download file to current directory
------------------------------------
;"file.dat" will be placed in current directory
paths = spd_download( remote_file='http://www.example.com/file.dat' )
Download file to specific location
------------------------------------
;remote file downloaded to "c:\data\file.dat"
paths = spd_download( remote_file='http://www.example.com/file.dat', $
local_path='c:\data\')
Download multiple files to specified location and preserve remote file structure
-------------------------------------
;remote files downloaded to "c:\data\folder1\file1.dat"
; "c"\data\folder2\file2.dat"
paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $
remote_path = 'http://www.example.com/', $
local_path = 'c:\data\')
Input:
Specifying File names
---------------------
remote_file: String or string array of URLs to remote files
remote_path: String consisting of a common URL base for all remote files
-The full remote URL(s) will be: remote_path + remote_file
-If no url scheme is recognized (e.g. 'http') then it is
assumed that the url is a path to a local resource (e.g. /mount/data/)
local_file: String or string array of local destination file names
local_path: String consisting of a common local path for all local files
-The final local path(s) will be: local_path + local_file
-The final path(s) can be full or relative
-If local_file is not set then remote_file is used
-If remote_file is a full URL then only the file name is used
-Remote file names may contain wildcards: * ? [ ]
Download Options
---------------------
last_version: Flag to only download the last in file in a lexically sorted
list when multiple matches are found using wildcards
no_update: Flag to not overwrite existing file
force_download: Flag to always overwrite existing file
no_download: Flag to not download remote files
user_agent: Specifies user agent reported to remote server. (string)
file_mode: Bit mask specifying permissions for new files (see file_chmod)
dir_mode: Bit mask specifying permissions for new directories (see file_chmod)
no_wildcards: assume no wild cards in the requested url/filename
ssl_verify_peer: set this to verify the authenticity of the peer's SSL certificate (HTTPS)
ssl_verify_host: set this to verify the authenticity of the server certificate (HTTPS)
disable_cdfcheck: Useful for large files
IDLnetURL Properties
---------------------
All IDLnetURL properties may be specified as keywords to this routine.
See "IDLnetURL Properties" in the IDL documentation for more information.
Some examples are:
------------------
url_query
url_username
url_password
url_port
proxy_hostname
proxy_username
proxy_password
Other
---------------------
progress_object: Object reference for status updates
Backward Support
---------------------
local_data_dir: mapped to local_path
remote_data_dir: mapped to remote_path
no_server: mapped to no_download
no_clobber: mapped to no_update
valid_only: only return paths to files that exist (ideally this would be default)
Output:
return value: String array specifying the full local path to all requested files
Notes:
-unsupported file_retrieve keywords:
progress, preserve_mtime, progobj, ignore_filesize,
ignore_filedate, archive_ext, archive_dir, min_age_limit
$LastChangedBy: nikos $
$LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $
$LastChangedRevision: 27093 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download.pro $
(See general/spedas_tools/spd_download/spd_download.pro)
Procedure:
spd_ui_download_callback
Purpose:
A callback function for the idlneturl object as used by spd_download_file
Calling Sequence:
N/A - function name set as idlneturl object's callback_function property
Input/Output:
status: See "Using Callbacks with the IDLnetURL Object" in IDL documentation
progress: See "Using Callbacks with the IDLnetURL Object" in IDL documentation
data: Custom data structure for passing variables from spd_download_file
to this function.
{
net_object: reference to current idlneturl object
msg_time: pointer to time of the last status message
msg_data: bytes transferred as of msg_time
progress_object: reference to applicable status output object
error: pointer to flag denoting whether an error occurred in this code
(to be used by handler later)
}
Output:
return_value: 1 if everything is OK, 0 if operation should be canceled
Notes:
$LastChangedBy: egrimes $
$LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
$LastChangedRevision: 22594 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_callback.pro $
(See general/spedas_tools/spd_download/spd_download_callback.pro)
Procedure:
spd_download_expand
Purpose:
Check remote host for requested files and apply wildcards
by downloading and parsing remote index file.
Calling Sequence:
spd_download_extract, url, last_version=last_version
Input:
url: String array of URLs to remote files.
last_version: Flag to only use the last file in a lexically sorted list
when wildcards result in multiple matches.
Output:
url: String array of matching remote files or empty string if no matches are found.
Notes:
$LastChangedBy: nikos $
$LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $
$LastChangedRevision: 27093 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_expand.pro $
(See general/spedas_tools/spd_download/spd_download_expand.pro)
Function:
spd_download_extract
Purpose:
Helper function to parse <a>(link) tags from html index files.
Calling Sequence:
return_value = spd_download_extract(string [,/relative] [,/normal]
[,no_parent_links=no_parent_links])
Input:
string_array: String array containing the html index file to be parsed
relative: Set to strip out everything but the filename from a link
normal: Set to links that don't have '*' or '?'
(don't think this should every actually happen, but option retained just in case)
no_parent_links: Set to the parent domain to automatically exclude backlinks to the parent directory
Output:
return_value: An empty string or array of strings with link destinations
Notes:
Copied from file_http_copy subroutine extract_html_links_regex, original notes below:
"The _regex version of this routine is replacing the original version because
the old version made assumptions about the formatting of the .remote-index.html file
that were dependent upon the type of web server that was producing the file. We think that
these bugs took so long to show up because Apache servers are extremely common.
Modification prompted so that file_http_copy can work more reliably rbspice & rb-emfisis
New version:
#1 Handles html that doesn't place the href attribute exactly one space after the link tag
#2 Handles cases where the server doesn't include newlines, or where multiple links are
included per line of returned html by the server"
$LastChangedBy: aaflores $
$LastChangedDate: 2015-02-18 16:27:58 -0800 (Wed, 18 Feb 2015) $
$LastChangedRevision: 17004 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_extract.pro $
(See general/spedas_tools/spd_download/spd_download_extract.pro)
Function:
spd_download_file
Purpose:
Primarily a helper function for spd_download. This function will download
a single file and return the path to that file. If a local file exists it
will only download if the remote file is newer.
Calling Sequence:
path = spd_download_file(url=url, [,filename=filename] ... )
Input:
url: URL to remote file. (string)
filename: Specifies local file path and name. If a full path is not
provided it will assumes a relative path. (string)
user_agent: Specifies user agent reported to remote server. (string)
headers: Array of HTML headers to be sent to remote server.
Other IDLneturl properties are passed through _extra but
this one is extracted so that defaults can be appended:
-"User-Agent" is added by default
-"If-Modified-Since" is added if file age is checked
no_update: Flag to not overwrite existing file
force_download: Flag to always overwrite existing file
string_array: Flag to download remote file and load into memory as an
array of strings.
min_age_limit: Files younger than this (in seconds) are assumed current
file_mode: Bit mask specifying permissions for new files (see file_chmod)
dir_mode: Bit mask specifying permissions for new directories (see file_chmod)
progress_object: Status update object
disable_cdfcheck: Useful for large files
_extra: Any idlneturl property (except callback_*) can be passed via _extra
Output:
return value:
local file path (string) - if a file is downloaded or a local file is found
empty string (string) - if no file is found
file contents (string array) - if /string_array is set
Notes:
-precedence of boolean keywords:
string_array > force_download > no_update > default behavior
-checks contents of "http_proxy" environment variable for proxy server
$LastChangedBy: nikos $
$LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $
$LastChangedRevision: 27093 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_file.pro $
(See general/spedas_tools/spd_download/spd_download_file.pro)
Procedure:
spd_download_handler
Purpose:
Handle errors thrown by the IDLnetURL object used in spd_download_file.
HTTP responses will be caught and handled. All other exceptions will
be reissued for a higher level error handler to catch.
Calling Sequence:
spd_download_handler, net_object=net_object, url=url, filename=filename
Input:
net_object: Reference to IDLnetURL object
url: String specifying URL of remote file
filename: String specifying path (full or partial) to (requested) local file
callback_error: Flag denoting that an exception occurred in the callback function
Output:
None
Notes:
egrimes, 3/20/17 - no longer reissuing last error, now meaningful printing error
msg from spd_neturl_error2msg map
$LastChangedBy: egrimes $
$LastChangedDate: 2017-03-20 10:49:45 -0700 (Mon, 20 Mar 2017) $
$LastChangedRevision: 22996 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_handler.pro $
(See general/spedas_tools/spd_download/spd_download_handler.pro)
Procedure:
spd_download_mkdir
Purpose:
Manually create any non-existent directories in a local path
so that file permissions can be set.
Calling Sequence:
spd_download_mkdir, path, mode
Input:
path: Local path (full or relative) to the requested file's destination.
The filename is assumed to be included.
mode: Bit mask specifying permissions for any created directories.
See file_chmod documentation for more information.
Output:
error: Flag output from recursive calls denoting that no
directories should be created.
Notes:
$LastChangedBy: egrimes $
$LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $
$LastChangedRevision: 22594 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_mkdir.pro $
(See general/spedas_tools/spd_download/spd_download_mkdir.pro)
Function:
spd_download_temp
Purpose:
Create a random numeric filename suffix for temporary files.
Calling Sequence:
suffix = spd_download_temp()
Output:
Returns 12 digit numeric string preceded by a period
e.g. ".555350461348"
Notes:
$LastChangedBy: jimm $
$LastChangedDate: 2017-03-08 16:06:47 -0800 (Wed, 08 Mar 2017) $
$LastChangedRevision: 22927 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_temp.pro $
(See general/spedas_tools/spd_download/spd_download_temp.pro)
Procedure:
spd_get_proxy
Purpose:
Gets "http_proxy" environment variable and adds parsed proxy
properties to an input structure for use with idlneturl.
Calling Sequence:
spd_get_proxy, structure
Arguments:
structure: Structure to which the idlneturl proxy properties will be appended.
Existing properties will not be overwritten. If undefined or not
valid then a structure with said properties will be returned.
Notes:
$LastChangedBy: aaflores $
$LastChangedDate: 2015-08-28 14:36:40 -0700 (Fri, 28 Aug 2015) $
$LastChangedRevision: 18666 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_get_proxy.pro $
(See general/spedas_tools/spd_download/spd_get_proxy.pro)
FUNCTION:
spd_neturl_error2msg
PURPOSE:
returns a list, where the index is the cURL error code and the item is the status message that the code represents;
useful for finding meaningful download error messages
$LastChangedBy: egrimes $
$LastChangedDate: 2017-03-20 08:55:34 -0700 (Mon, 20 Mar 2017) $
$LastChangedRevision: 22990 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_neturl_error2msg.pro $
(See general/spedas_tools/spd_download/spd_neturl_error2msg.pro)
NAME:
wrapper for spd_download that returns local versions of files, if
they are available. SPd_download only will return a local version if
the file is specified without wildcards.
CALLING SEQUENCE:
paths = spd_download_plus( [remote_file=remote_file] | [,remote_path=remote_path]
[,local_file=local_file] [,local_path=local_path] ... )
Example Usage:
Download file to current directory
------------------------------------
;"file.dat" will be placed in current directory
paths = spd_download( remote_file='http://www.example.com/file.dat' )
Download file to specific location
------------------------------------
;remote file downloaded to "c:\data\file.dat"
paths = spd_download( remote_file='http://www.example.com/file.dat', $
local_path='c:\data\')
Download multiple files to specified location and preserve remote file structure
-------------------------------------
;remote files downloaded to "c:\data\folder1\file1.dat"
; "c"\data\folder2\file2.dat"
paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $
remote_path = 'http://www.example.com/', $
local_path = 'c:\data\')
Input:
Specifying File names
---------------------
remote_file: String or string array of URLs to remote files
remote_path: String consisting of a common URL base for all remote files
-The full remote URL(s) will be: remote_path + remote_file
-If no url scheme is recognized (e.g. 'http') then it is
assumed that the url is a path to a local resource (e.g. /mount/data/)
local_file: String or string array of local destination file names
local_path: String consisting of a common local path for all local files
-The final local path(s) will be: local_path + local_file
-The final path(s) can be full or relative
-If local_file is not set then remote_file is used
-If remote_file is a full URL then only the file name is used
-Remote and local file names may contain wildcards: * ? [ ]
Download Options
---------------------
last_version: Flag to only download the last in file in a lexically sorted
list when multiple matches are found using wildcards
no_update: Flag to not overwrite existing file
force_download: Flag to always overwrite existing file
no_download: Flag to not download remote files
user_agent: Specifies user agent reported to remote server. (string)
file_mode: Bit mask specifying permissions for new files (see file_chmod)
dir_mode: Bit mask specifying permissions for new directories (see file_chmod)
no_wildcards: assume no wild cards in the requested url/filename
IDLnetURL Properties
---------------------
All IDLnetURL properties may be specified as keywords to this routine.
See "IDLnetURL Properties" in the IDL documentation for more information.
Some examples are:
------------------
url_query
url_username
url_password
url_port
proxy_hostname
proxy_username
proxy_password
Other
---------------------
progress_object: Object reference for status updates
Backward Support
---------------------
local_data_dir: mapped to local_path
remote_data_dir: mapped to remote_path
no_server: mapped to no_download
no_clobber: mapped to no_update
valid_only: only return paths to files that exist (ideally this would be default)
Output:
return value: String array specifying the full local path to all requested files
Notes:
-unsupported file_retrieve keywords:
progress, preserve_mtime, progobj, ignore_filesize,
ignore_filedate, archive_ext, archive_dir, min_age_limit
-Note that the program still only returns the local file if a)
the local_file keyword is set, or the local_path and remote_file
keywords are set.
$LastChangedBy: jimm $
$LastChangedDate: 2017-02-06 15:51:25 -0800 (Mon, 06 Feb 2017) $
$LastChangedRevision: 22741 $
$URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_4_0/general/spedas_tools/spd_download/spd_download_plus.pro $
(See general/spedas_tools/spd_download/spd_download_plus.pro)