`squirrel.base`¶

class Selection(database, persistent=None)[source]¶

Bases: object

Database backed file selection (base class for Squirrel).

Parameters:	database (`Database` or `str`) – Database instance or file path to database. persistent (`str`) – If given a name, create a persistent selection.

In the Squirrel framework, a selection is conceptually a list of files to be made available in the application. Instead of using Selection directly, user applications should usually use its subclass Squirrel which adds content indices to the selection and provides high level data querying.

By default, a temporary table in the database is created to hold the names of the files in the selection. This table is only visible inside the application which created it. If a name is given to persistent, a named selection is created, which is visible also in other applications using the same database.

Besides the filename references, desired content kind masks and file format indications are stored in the selection’s database table to make the user choice regarding these options persistent on a per-file basis. Book-keeping on whether files are unknown, known or if modification checks are forced is also handled in the selection’s file-state table.

Paths of files can be added to the selection using the add() method and removed with remove(). undig_grouped() can be used to iterate over all content known to the selection.

get_database()[source]¶

Get the database to which this selection belongs.

Returns:	`Database` object

add(paths, kind_mask=255, format='detect')[source]¶

Add files to the selection.

Parameters:	paths (iterator yielding `str` objects) – Paths to files to be added to the selection.

remove(paths)[source]¶

Remove files from the selection.

Parameters:	paths (`list` of `str`) – Paths to files to be removed from the selection.

iter_paths()[source]¶

Iterate over all file paths currently belonging to the selection.

Returns:	Iterator yielding file paths.

get_paths()[source]¶

Get all file paths currently belonging to the selection.

Returns:	List of file paths.

undig_grouped(skip_unchanged=False)[source]¶

Get inventory of cached content for all files in the selection.

Param:	skip_unchanged: if `True` only inventory of modified files is yielded (`flag_modified()` must be called beforehand).

This generator yields tuples ((format, path), nuts) where path is the path to the file, format is the format assignation or 'detect' and nuts is a list of Nut objects representing the contents of the file.

flag_modified(check=True)[source]¶

Mark files which have been modified.

Parameters:	check – If `True` query modification times of known files on disk. If `False`, only flag unknown files.

Assumes file state is 0 for newly added files, 1 for files added again to the selection (forces check), or 2 for all others (no checking is done for those).

Sets file state to 0 for unknown or modified files, 2 for known and not modified files.

class SquirrelStats(**kwargs)[source]¶

Bases: pyrocko.guts.Object

Container to hold statistics about contents available from a Squirrel.

See also Squirrel.get_stats().

♦ nfiles¶

int

Number of files in selection.

♦ nnuts¶

int

Number of index nuts in selection.

♦ codes¶

list of tuple of str objects objects, default: []

Available code sequences in selection, e.g. (agency, network, station, location) for stations nuts.

♦ kinds¶

list of str objects, default: []

Available content types in selection.

♦ total_size¶

int

Aggregated file size of files is selection.

♦ counts¶

dict of dict of int objects objects, default: {}

Breakdown of how many nuts of any content type and code sequence are available in selection, counts[kind][codes].

♦ tmin¶

builtins.float (pyrocko.guts.Timestamp), optional

Earliest start time of all nuts in selection.

♦ tmax¶

builtins.float (pyrocko.guts.Timestamp), optional

Latest end time of all nuts in selection.

class Squirrel(env=None, database=None, cache_path=None, persistent=None)[source]¶

Bases: pyrocko.squirrel.base.Selection

Prompt, lazy, indexing, caching, dynamic seismological dataset access.

Parameters:

Parameters:	env (`SquirrelEnvironment` or `str`) – Squirrel environment instance or directory path to use as starting point for its detection. By default, the current directory is used as starting point. When searching for a usable environment the directory `'.squirrel'` or `'squirrel'` in the current (or starting point) directory is used if it exists, otherwise the parent directories are search upwards for the existence of such a directory. If no such directory is found, the user’s global Squirrel environment `'$HOME/.pyrocko/squirrel'` is used. database (`Database` or `str`) – Database instance or path to database. By default the database found in the detected Squirrel environment is used. cache_path (`str`) – Directory path to use for data caching. By default, the `'cache'` directory in the detected Squirrel environment is used. persistent (`str`) – If given a name, create a persistent selection.

env (SquirrelEnvironment or str) – Squirrel environment instance or directory path to use as starting point for its detection. By default, the current directory is used as starting point. When searching for a usable environment the directory '.squirrel' or 'squirrel' in the current (or starting point) directory is used if it exists, otherwise the parent directories are search upwards for the existence of such a directory. If no such directory is found, the user’s global Squirrel environment '$HOME/.pyrocko/squirrel' is used.
database (Database or str) – Database instance or path to database. By default the database found in the detected Squirrel environment is used.
cache_path (str) – Directory path to use for data caching. By default, the 'cache' directory in the detected Squirrel environment is used.
persistent (str) – If given a name, create a persistent selection.

Provides a unified interface to query seismic waveforms, station and sensor metadata, and event information from local file collections and remote data sources. Query results are promptly returned, even for very large collections, thanks to a highly optimized database setup working behind the scenes. Assemblage of a data selection is very fast for known files as all content indices are cached in a database. Unknown files are automatically indexed when added to the selection.

Features

Efficient[1] lookup of data relevant for a selected time window.
Metadata caching and indexing.
Modified files are re-indexed as needed.
SQL database (sqlite) is used behind the scenes.
Can handle selections with millions of files.
Data can be added and removed at run-time, efficiently[1].
Just-in-time download of missing data.
Disk-cache of meta-data query results with expiration time.
Efficient event catalog synchronization.
Always-up-to-date data coverage indices.
Always-up-to-date indices of available station/channel codes.

[1] O log N performance, where N is the number of data entities (nuts).

Queries are restricted to the contents offered by the files which have been added to the Squirrel (which usually is a subset of the information collected in the attached global file meta-information database).

By default, temporary tables are created in the attached database to hold the names of the files in the selection as well as various indices and counters. These tables are only visible inside the application which created it. If a name is given to persistent, a named selection is created, which is visible also in other applications using the same database.

Paths of files can be added to the selection using the add() method.

add(paths, kinds=None, format='detect', check=True, progress_viewer='terminal')[source]¶

Add files to the selection.

Parameters:

Parameters:	paths (`list` of `str`) – Iterator yielding paths to files or directories to be added to the selection. Recurses into directories. If given a `str`, it is treated as a single path to be added. kinds (`list` of `str`) – Content types to be made available through the Squirrel selection. By default, all known content types are accepted. format (`str`) – File format identifier or `'detect'` to enable auto-detection.

paths (list of str) – Iterator yielding paths to files or directories to be added to the selection. Recurses into directories. If given a str, it is treated as a single path to be added.
kinds (list of str) – Content types to be made available through the Squirrel selection. By default, all known content types are accepted.
format (str) – File format identifier or 'detect' to enable auto-detection.

Complexity: O(log N)

reload()[source]¶

Check for modifications and reindex modified files.

Based on file modification times.

add_virtual(nuts, virtual_paths=None)[source]¶

Add content which is not backed by files.

Parameters:	nuts (iterator yielding `Nut` objects) – Content pieces to be added. virtual_paths (`list` of `str`) – List of virtual paths to prevent creating a temporary list of the nuts while aggregating the file paths for the selection.

Stores to the main database and the selection.

add_source(source)[source]¶

Add remote resource.

Parameters:	source (subclass of `Source`) – Remote data access client instance.

add_fdsn(*args, **kwargs)[source]¶

Add FDSN site for transparent remote data access.

Arguments are passed to FDSNSource.

add_catalog(*args, **kwargs)[source]¶

Add online catalog for transparent event data access.

Arguments are passed to CatalogSource.

iter_nuts(kind=None, tmin=None, tmax=None, codes=None, naiv=False, kind_codes_ids=None)[source]¶

Iterate content entities matching given constraints.

Parameters:

Parameters:	kind (`str`, `list` of `str`) – Content kind (or kinds) to extract. tmin (timestamp) – Start time of query interval. tmax (timestamp) – End time of query interval. codes (`tuple` of `str`) – Pattern of content codes to be matched. naiv (`bool`) – Bypass time span lookup through indices (slow, for testing). kind_codes_ids (`list` of `str`) – Kind-codes IDs of contents to be retrieved (internal use).

kind (str, list of str) – Content kind (or kinds) to extract.
tmin (timestamp) – Start time of query interval.
tmax (timestamp) – End time of query interval.
codes (tuple of str) – Pattern of content codes to be matched.
naiv (bool) – Bypass time span lookup through indices (slow, for testing).
kind_codes_ids (list of str) – Kind-codes IDs of contents to be retrieved (internal use).

Complexity: O(log N) for the time selection part due to heavy use of database indices.

Yields Nut objects representing the intersecting content.

Query time span is treated as a half-open interval [tmin, tmax). However, if tmin equals tmax, the edge logics are modified to closed-interval so that content intersecting with the time instant t = tmin = tmax is returned (otherwise nothing would be returned as [t, t) never matches anything).

Time spans of content entities to be matched are also treated as half open intervals, e.g. content span [0, 1) is matched by query span [0, 1) but not by [-1, 0) or [1, 2). Also here, logics are modified to closed-interval when the content time span is an empty interval, i.e. to indicate a time instant. E.g. time instant 0 is matched by [0, 1) but not by [-1, 0) or [1, 2).

get_nuts(*args, **kwargs)[source]¶

Get content entities matching given constraints.

Like iter_nuts() but returns results as a list.

get_time_span(kinds=None)[source]¶

Get time interval over all content in selection.

Complexity O(1), independent of the number of nuts.

Returns:	(tmin, tmax)

get_deltat_span(kind)[source]¶

Get min and max sampling interval of all content of given kind.

Parameters:	kind (`str`) – Content kind
Returns:	(deltat_min, deltat_max)

iter_kinds(codes=None)[source]¶

Iterate over content types available in selection.

Parameters:	codes – if given, get kinds only for selected codes identifier