This page was created by the IDL library routine
mk_html_help2
.
Last modified: Wed Dec 20 10:37:44 2023.
NAME: spd_cdf_check_delete PURPOSE: Renames or deletes cdf or netcdf files, if they can't be opened. This fuction can be used to cleanup downloaded files with problems. By default, the files are renamed to: filename + '.todelete' INPUT: filenames: Array of cdf or netcdf filenames (full path). (filename extension is expected to be '.nc' for netcdf and '.cdf' for cdf files) KEYWORDS: iscdf: Force cdf check. isnetcdf: Force netcdf check. (if both keywords are used, netcdf will be prefered) delete_file: The file will be deleted. OUTPUT: Array of renamed or deleted files. Renamed files have the file extension '.todelete'. EXAMPLES: deleted_files = spd_cdf_check_delete(["file1.cdf", "file2.nc"], /delete) HISTORY: $LastChangedBy: nikos $ $LastChangedDate: 2021-06-10 14:39:13 -0700 (Thu, 10 Jun 2021) $ $LastChangedRevision: 30039 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_cdf_check_delete.pro $
(See general/spedas_tools/spd_download/spd_cdf_check_delete.pro)
Procedure: spd_copy_file Purpose: Primarily a helper function for spd_download. This function will copy a single file and return the path to the copy. Existing files will only be overwritten if the source is newer. Calling Sequence: path = spd_copy_file( source=source, destination=destination [,no_update=no_update] [,force_copy=force_copy] ) Input: source: Path to source file destination: Path to destination file no_update: Flag to not overwrite existing file force_copy: Flag to always overwrite existing file file_mode: Bit mask specifying permissions for new files (see file_chmod) dir_mode: Bit mask specifying permissions for new directories (see file_chmod) Output: return value: Full path to destination file if successful, empty string otherwise Notes: -precedence of boolean keywords: force_download > no_update > default behavior $LastChangedBy: egrimes $ $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $ $LastChangedRevision: 22594 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_copy_file.pro $
(See general/spedas_tools/spd_download/spd_copy_file.pro)
Function: spd_download Purpose: Download one or more remote files and return their local paths. This function can be used in place of file_retrieve. Calling Sequence: paths = spd_download( [remote_file=remote_file] | [,remote_path=remote_path] [,local_file=local_file] [,local_path=local_path] ... ) Example Usage: Download file to current directory ------------------------------------ ;"file.dat" will be placed in current directory paths = spd_download( remote_file='http://www.example.com/file.dat' ) Download file to specific location ------------------------------------ ;remote file downloaded to "c:\data\file.dat" paths = spd_download( remote_file='http://www.example.com/file.dat', $ local_path='c:\data\') Download multiple files to specified location and preserve remote file structure ------------------------------------- ;remote files downloaded to "c:\data\folder1\file1.dat" ; "c"\data\folder2\file2.dat" paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $ remote_path = 'http://www.example.com/', $ local_path = 'c:\data\') Input: Specifying File names --------------------- remote_file: String or string array of URLs to remote files remote_path: String consisting of a common URL base for all remote files -The full remote URL(s) will be: remote_path + remote_file -If no url scheme is recognized (e.g. 'http') then it is assumed that the url is a path to a local resource (e.g. /mount/data/) local_file: String or string array of local destination file names local_path: String consisting of a common local path for all local files -The final local path(s) will be: local_path + local_file -The final path(s) can be full or relative -If local_file is not set then remote_file is used -If remote_file is a full URL then only the file name is used -Remote file names may contain wildcards: * ? [ ] Download Options --------------------- last_version: Flag to only download the last in file in a lexically sorted list when multiple matches are found using wildcards no_update: Flag to not overwrite existing file force_download: Flag to always overwrite existing file no_download: Flag to not download remote files user_agent: Specifies user agent reported to remote server. (string) file_mode: Bit mask specifying permissions for new files (see file_chmod) dir_mode: Bit mask specifying permissions for new directories (see file_chmod) no_wildcards: assume no wild cards in the requested url/filename ssl_verify_peer: set this to verify the authenticity of the peer's SSL certificate (HTTPS) ssl_verify_host: set this to verify the authenticity of the server certificate (HTTPS) disable_cdfcheck: Useful for large files IDLnetURL Properties --------------------- All IDLnetURL properties may be specified as keywords to this routine. See "IDLnetURL Properties" in the IDL documentation for more information. Some examples are: ------------------ url_query url_username url_password url_port proxy_hostname proxy_username proxy_password Other --------------------- progress_object: Object reference for status updates Backward Support --------------------- local_data_dir: mapped to local_path remote_data_dir: mapped to remote_path no_server: mapped to no_download no_clobber: mapped to no_update valid_only: only return paths to files that exist (ideally this would be default) Output: return value: String array specifying the full local path to all requested files Notes: -unsupported file_retrieve keywords: progress, preserve_mtime, progobj, ignore_filesize, ignore_filedate, archive_ext, archive_dir, min_age_limit $LastChangedBy: nikos $ $LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $ $LastChangedRevision: 27093 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download.pro $
(See general/spedas_tools/spd_download/spd_download.pro)
Procedure: spd_ui_download_callback Purpose: A callback function for the idlneturl object as used by spd_download_file Calling Sequence: N/A - function name set as idlneturl object's callback_function property Input/Output: status: See "Using Callbacks with the IDLnetURL Object" in IDL documentation progress: See "Using Callbacks with the IDLnetURL Object" in IDL documentation data: Custom data structure for passing variables from spd_download_file to this function. { net_object: reference to current idlneturl object msg_time: pointer to time of the last status message msg_data: bytes transferred as of msg_time progress_object: reference to applicable status output object error: pointer to flag denoting whether an error occurred in this code (to be used by handler later) } Output: return_value: 1 if everything is OK, 0 if operation should be canceled Notes: $LastChangedBy: egrimes $ $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $ $LastChangedRevision: 22594 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_callback.pro $
(See general/spedas_tools/spd_download/spd_download_callback.pro)
Procedure: spd_download_expand Purpose: Check remote host for requested files and apply wildcards by downloading and parsing remote index file. Calling Sequence: spd_download_extract, url, last_version=last_version Input: url: String array of URLs to remote files. last_version: Flag to only use the last file in a lexically sorted list when wildcards result in multiple matches. Output: url: String array of matching remote files or empty string if no matches are found. Notes: $LastChangedBy: nikos $ $LastChangedDate: 2019-04-25 15:32:56 -0700 (Thu, 25 Apr 2019) $ $LastChangedRevision: 27093 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_expand.pro $
(See general/spedas_tools/spd_download/spd_download_expand.pro)
Function: spd_download_extract Purpose: Helper function to parse <a>(link) tags from html index files. Calling Sequence: return_value = spd_download_extract(string [,/relative] [,/normal] [,no_parent_links=no_parent_links]) Input: string_array: String array containing the html index file to be parsed relative: Set to strip out everything but the filename from a link normal: Set to links that don't have '*' or '?' (don't think this should every actually happen, but option retained just in case) no_parent_links: Set to the parent domain to automatically exclude backlinks to the parent directory Output: return_value: An empty string or array of strings with link destinations Notes: Copied from file_http_copy subroutine extract_html_links_regex, original notes below: "The _regex version of this routine is replacing the original version because the old version made assumptions about the formatting of the .remote-index.html file that were dependent upon the type of web server that was producing the file. We think that these bugs took so long to show up because Apache servers are extremely common. Modification prompted so that file_http_copy can work more reliably rbspice & rb-emfisis New version: #1 Handles html that doesn't place the href attribute exactly one space after the link tag #2 Handles cases where the server doesn't include newlines, or where multiple links are included per line of returned html by the server" $LastChangedBy: aaflores $ $LastChangedDate: 2015-02-18 16:27:58 -0800 (Wed, 18 Feb 2015) $ $LastChangedRevision: 17004 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_extract.pro $
(See general/spedas_tools/spd_download/spd_download_extract.pro)
Function: spd_download_file Purpose: Primarily a helper function for spd_download. This function will download a single file and return the path to that file. If a local file exists it will only download if the remote file is newer. Calling Sequence: path = spd_download_file(url=url, [,filename=filename] ... ) Input: url: URL to remote file. (string) filename: Specifies local file path and name. If a full path is not provided it will assumes a relative path. (string) user_agent: Specifies user agent reported to remote server. (string) headers: Array of HTML headers to be sent to remote server. Other IDLneturl properties are passed through _extra but this one is extracted so that defaults can be appended: -"User-Agent" is added by default -"If-Modified-Since" is added if file age is checked no_update: Flag to not overwrite existing file force_download: Flag to always overwrite existing file string_array: Flag to download remote file and load into memory as an array of strings. min_age_limit: Files younger than this (in seconds) are assumed current file_mode: Bit mask specifying permissions for new files (see file_chmod) dir_mode: Bit mask specifying permissions for new directories (see file_chmod) progress_object: Status update object disable_cdfcheck: Useful for large files _extra: Any idlneturl property (except callback_*) can be passed via _extra Output: return value: local file path (string) - if a file is downloaded or a local file is found empty string (string) - if no file is found file contents (string array) - if /string_array is set Notes: -precedence of boolean keywords: string_array > force_download > no_update > default behavior -checks contents of "http_proxy" environment variable for proxy server $LastChangedBy: nikos $ $LastChangedDate: 2022-10-03 10:00:54 -0700 (Mon, 03 Oct 2022) $ $LastChangedRevision: 31147 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_file.pro $
(See general/spedas_tools/spd_download/spd_download_file.pro)
Procedure: spd_download_handler Purpose: Handle errors thrown by the IDLnetURL object used in spd_download_file. HTTP responses will be caught and handled. All other exceptions will be reissued for a higher level error handler to catch. Calling Sequence: spd_download_handler, net_object=net_object, url=url, filename=filename Input: net_object: Reference to IDLnetURL object url: String specifying URL of remote file filename: String specifying path (full or partial) to (requested) local file callback_error: Flag denoting that an exception occurred in the callback function Output: None Notes: egrimes, 3/20/17 - no longer reissuing last error, now meaningful printing error msg from spd_neturl_error2msg map $LastChangedBy: egrimes $ $LastChangedDate: 2017-03-20 10:49:45 -0700 (Mon, 20 Mar 2017) $ $LastChangedRevision: 22996 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_handler.pro $
(See general/spedas_tools/spd_download/spd_download_handler.pro)
Procedure: spd_download_mkdir Purpose: Manually create any non-existent directories in a local path so that file permissions can be set. Calling Sequence: spd_download_mkdir, path, mode Input: path: Local path (full or relative) to the requested file's destination. The filename is assumed to be included. mode: Bit mask specifying permissions for any created directories. See file_chmod documentation for more information. Output: error: Flag output from recursive calls denoting that no directories should be created. Notes: $LastChangedBy: egrimes $ $LastChangedDate: 2017-01-13 11:24:43 -0800 (Fri, 13 Jan 2017) $ $LastChangedRevision: 22594 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_mkdir.pro $
(See general/spedas_tools/spd_download/spd_download_mkdir.pro)
Function: spd_download_temp Purpose: Create a random numeric filename suffix for temporary files. Calling Sequence: suffix = spd_download_temp() Output: Returns 12 digit numeric string preceded by a period e.g. ".555350461348" Notes: $LastChangedBy: jimm $ $LastChangedDate: 2017-03-08 16:06:47 -0800 (Wed, 08 Mar 2017) $ $LastChangedRevision: 22927 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_temp.pro $
(See general/spedas_tools/spd_download/spd_download_temp.pro)
Procedure: spd_get_proxy Purpose: Gets "http_proxy" environment variable and adds parsed proxy properties to an input structure for use with idlneturl. Calling Sequence: spd_get_proxy, structure Arguments: structure: Structure to which the idlneturl proxy properties will be appended. Existing properties will not be overwritten. If undefined or not valid then a structure with said properties will be returned. Notes: $LastChangedBy: aaflores $ $LastChangedDate: 2015-08-28 14:36:40 -0700 (Fri, 28 Aug 2015) $ $LastChangedRevision: 18666 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_get_proxy.pro $
(See general/spedas_tools/spd_download/spd_get_proxy.pro)
FUNCTION: spd_neturl_error2msg PURPOSE: returns a list, where the index is the cURL error code and the item is the status message that the code represents; useful for finding meaningful download error messages $LastChangedBy: egrimes $ $LastChangedDate: 2017-03-20 08:55:34 -0700 (Mon, 20 Mar 2017) $ $LastChangedRevision: 22990 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_neturl_error2msg.pro $
(See general/spedas_tools/spd_download/spd_neturl_error2msg.pro)
NAME: wrapper for spd_download that returns local versions of files, if they are available. SPd_download only will return a local version if the file is specified without wildcards. CALLING SEQUENCE: paths = spd_download_plus( [remote_file=remote_file] | [,remote_path=remote_path] [,local_file=local_file] [,local_path=local_path] ... ) Example Usage: Download file to current directory ------------------------------------ ;"file.dat" will be placed in current directory paths = spd_download( remote_file='http://www.example.com/file.dat' ) Download file to specific location ------------------------------------ ;remote file downloaded to "c:\data\file.dat" paths = spd_download( remote_file='http://www.example.com/file.dat', $ local_path='c:\data\') Download multiple files to specified location and preserve remote file structure ------------------------------------- ;remote files downloaded to "c:\data\folder1\file1.dat" ; "c"\data\folder2\file2.dat" paths = spd_download( remote_file = ['folder1/file1.dat','folder2/folder2.dat'], $ remote_path = 'http://www.example.com/', $ local_path = 'c:\data\') Input: Specifying File names --------------------- remote_file: String or string array of URLs to remote files remote_path: String consisting of a common URL base for all remote files -The full remote URL(s) will be: remote_path + remote_file -If no url scheme is recognized (e.g. 'http') then it is assumed that the url is a path to a local resource (e.g. /mount/data/) local_file: String or string array of local destination file names local_path: String consisting of a common local path for all local files -The final local path(s) will be: local_path + local_file -The final path(s) can be full or relative -If local_file is not set then remote_file is used -If remote_file is a full URL then only the file name is used -Remote and local file names may contain wildcards: * ? [ ] Download Options --------------------- last_version: Flag to only download the last in file in a lexically sorted list when multiple matches are found using wildcards no_update: Flag to not overwrite existing file force_download: Flag to always overwrite existing file no_download: Flag to not download remote files user_agent: Specifies user agent reported to remote server. (string) file_mode: Bit mask specifying permissions for new files (see file_chmod) dir_mode: Bit mask specifying permissions for new directories (see file_chmod) no_wildcards: assume no wild cards in the requested url/filename IDLnetURL Properties --------------------- All IDLnetURL properties may be specified as keywords to this routine. See "IDLnetURL Properties" in the IDL documentation for more information. Some examples are: ------------------ url_query url_username url_password url_port proxy_hostname proxy_username proxy_password Other --------------------- progress_object: Object reference for status updates Backward Support --------------------- local_data_dir: mapped to local_path remote_data_dir: mapped to remote_path no_server: mapped to no_download no_clobber: mapped to no_update valid_only: only return paths to files that exist (ideally this would be default) Output: return value: String array specifying the full local path to all requested files Notes: -unsupported file_retrieve keywords: progress, preserve_mtime, progobj, ignore_filesize, ignore_filedate, archive_ext, archive_dir, min_age_limit -Note that the program still only returns the local file if a) the local_file keyword is set, or the local_path and remote_file keywords are set. $LastChangedBy: jimm $ $LastChangedDate: 2017-02-06 15:51:25 -0800 (Mon, 06 Feb 2017) $ $LastChangedRevision: 22741 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/tags/spedas_6_0/general/spedas_tools/spd_download/spd_download_plus.pro $
(See general/spedas_tools/spd_download/spd_download_plus.pro)