Remote Access (pyavs.remote)

Note

Loads data directly from the public AWS S3 release bucket, on demand, without downloading the whole dataset first. See Data Access for details and Quick Start Guide for a walkthrough.

AVSRemote mirrors the local loaders (pyavs.load_meg_raw(), pyavs.load_experiment_log(), etc.): each method fetches the file(s) it needs from S3 into a local cache, then delegates to the existing loader function unchanged – “the same loaders, a different byte source.”

Client

Remote AVS client — load subject/session/trial data directly from the public S3 release bucket, without downloading the whole dataset first.

AVSRemote mirrors the local loaders (pyavs.load_meg_raw(), pyavs.load_experiment_log(), etc.): each method resolves the release-relative path(s) it needs via the same Layout the local API uses, fetches them into a local cache via S3Store, then delegates to the existing loader function against that cache. Nothing about the loaders themselves changes — this is “the same loaders, a different byte source.”

v1 scope: whole-file fetches only, at the same subject/session/(whole epoch file) granularity the local API already supports. Not built: querying epochs by content (e.g. “every fixation on a dog, across subjects”) without downloading each session’s full epoch file first — that needs a catalog and chunk-level range reads, both designed but deferred; see release/remote_dataloader_design.md.

class pyavs.remote.client.AVSRemote(cache_root: str | Path | None = None, bucket: str = 'kietzmannlab-avs', region: str = 'us-west-2', verbose: bool = True)[source]

Bases: object

Load AVS data on demand from the public S3 release bucket.

Parameters:

  • cache_root (str or Path, optional) – Local cache directory. Defaults to ~/.cache/pyavs/<bucket>. Once populated, this directory is itself a valid (partial) avs-public tree — pointing pyavs.set_data_path() at it works too.

  • bucket (str, optional) – S3 bucket name (default: the public AVS release bucket).

  • region (str, optional) – Bucket region (default: 'us-west-2').

  • verbose (bool, optional) – Log size/time/cache-location feedback for each fetch (default: True). Set False for silent fetching.

Examples

>>> avs = AVSRemote()
>>> explog = avs.load_experiment_log(1, 1)
>>> epochs = avs.load_epochs(1, 1, event_type='fixation_scene')  # one whole session
>>> dogs = avs.epochs(event_type='fixation_scene').where("object_label == 'dog'")
>>> dog_epochs = dogs.load()  # range-read only the matching epochs, across subjects
__init__(cache_root: str | Path | None = None, bucket: str = 'kietzmannlab-avs', region: str = 'us-west-2', verbose: bool = True)[source]
property data_path: str

Local cache root, usable directly as a data_path= for the local API.

load_meg_raw(subject_id: int, session: int, run: int, preload: bool = False, verbose: bool = True) → mne.io.Raw[source]

Fetch and load one raw MEG run. See pyavs.load_meg_raw().

load_meg_preprocessed(subject_id: int, session: int, run: int, preload: bool = False, verbose: bool = True) → mne.io.Raw[source]

Fetch and load one Maxwell-filtered MEG run. See pyavs.load_meg_preprocessed().

load_experiment_log(subject_id: int, session: int, output_prefix: str = 'as') → DataFrame[source]

Fetch and load the experiment log. See pyavs.load_experiment_log().

load_eye_events(subject_id: int, session: int, preprocessed: bool = True, output_prefix: str = 'as') → Tuple[DataFrame, DataFrame][source]

Fetch and load eye-tracking events + messages. See pyavs.load_eye_events().

load_epochs_h5(subject_id: int, session: int, event_type: str = 'epochs') → Tuple[Dict[str, ndarray], DataFrame, Dict[str, Any]][source]

Fetch and load one session’s raw epoch arrays. See pyavs.io.read.load_epochs_h5().

load_epochs(subject_id: int, session: int, event_type: str = 'fixation_scene') → mne.Epochs[source]

Fetch and load one session’s epochs as an mne.Epochs with metadata attached (including object_label/object_id).

This is whole-session granularity — the same as the local pyavs.io.read.load_epochs() — not a filtered/indexed query. Filter the returned epochs.metadata locally after loading (e.g. epochs[epochs.metadata.object_label == 'dog']).

epochs(event_type: str | None = None, subject_id: int | None = None, session: int | None = None) → EpochQuery[source]

Open a content-indexed query over every epoch in the dataset.

Downloads (and caches) the small epoch catalog on first call, then filters entirely locally – no bulk data is fetched until EpochQuery.load() is called. This is what makes “every fixation on a dog, across subjects” answerable without downloading each session’s full epoch file.

Parameters:

  • event_type (str, optional) – Restrict to 'fixation_scene' or 'saccade_scene' (default: both).

  • subject_id (int, optional) – Restrict to one subject (default: all).

  • session (int, optional) – Restrict to one session (default: all).

Return type:

EpochQuery

Notes

Only epochs whose underlying h5 has actually been uploaded to the bucket can be .load()-ed; the catalog itself covers the whole released dataset regardless of upload progress. A query spanning un-uploaded sessions raises RemoteFileNotFoundError on .load().

load_anatomical(subject_id: int) → str[source]

Fetch and return the path to the defaced T1 volume. See pyavs.load_anatomical().

pyavs.remote.client.open_remote(cache_root: str | Path | None = None, bucket: str = 'kietzmannlab-avs', region: str = 'us-west-2', verbose: bool = True) → AVSRemote[source]

Open a remote AVS client backed by the public S3 release bucket.

Parameters:

  • cache_root (str or Path, optional) – Local cache directory (default: ~/.cache/pyavs/<bucket>).

  • bucket (str, optional) – S3 bucket name (default: the public AVS release bucket).

  • region (str, optional) – Bucket region (default: 'us-west-2').

  • verbose (bool, optional) – Log size/time/cache-location feedback for each fetch (default: True). Set False for silent fetching.

Return type:

AVSRemote

Content-Indexed Epoch Queries

Content-indexed epoch queries over the AVS release.

This is the versatile piece of the remote dataloader: filter fixations or saccades by metadata (fixated object, scene, kinematics, subject, session, …) across the whole dataset, then fetch only the matching epochs’ HDF5 chunks over HTTP range reads – never a whole session’s epoch file just to pull out a handful of rows.

Mechanism, measured end-to-end against the live bucket (matches the ~290x-less-data finding in release/remote_dataloader_design.md ss3): each epoch h5 is chunked one HDF5 chunk per epoch (pyavs.io.write.save_population_codes_h5’s chunk_epochs=1), so opening the remote file over fsspec’s HTTPFileSystem with cache_type='none' and reading specific epoch indices issues one HTTP range request per chunk rather than downloading the file. cache_type='none' matters: fsspec’s default block-caching would pull in ~8x more bytes than needed for this scattered access pattern (measured in the design doc).

Moving fewer bytes doesn’t by itself make this fast: each chunk is its own HTTP round trip, so a serial loop over hundreds of scattered epochs is latency-bound, not bandwidth-bound – measured on a real 220-epoch/4-file query: 116.9 MB range-read (68x less than the ~8 GB those 4 files total), but 100.5s wall clock, actually slower than the ~94s a whole-file download of the same 8 GB would take at observed sync throughput.

Concurrency here has to be process-based, not thread-based – measured, not assumed. A first attempt used a thread pool; it made things even slower (150.2s on the same query), because h5py wraps every HDF5 C-library call in a process-global lock (HDF5 is not built thread-safe by default), so concurrent threads calling into h5py – even each with its own file handle, even with fsspec’s instance cache disabled – still serialize. A ProcessPoolExecutor sidesteps this entirely: each worker process gets its own independent copy of the HDF5 library and its own lock, so the actual network waits genuinely overlap. Same 220-epoch/4-file query with an 8-process pool: 24.4s – 4.1x faster than the serial attempt, 6.2x faster than the failed thread-pool attempt, and now genuinely faster than whole-file download too, on top of the 68x bandwidth saving. This keeps HDF5/h5py entirely as-is – no format change, no manual chunk parsing – _read_task below still just calls ordinary h5py indexing, only spread across processes instead of one loop.

Read tasks are split both across files (the common “one query, many sessions/subjects” case) and, within one file, across sub-batches when a single file has enough matching epochs to be worth it – so a query concentrated in one session benefits too, not just cross-subject queries.

class pyavs.remote.query.EpochQuery(metadata: DataFrame, store: S3Store)[source]

Bases: object

A lazy, filterable view over the epoch catalog.

Built via pyavs.remote.AVSRemote.epochs(), not directly. .where() only filters a local metadata table – no bulk data moves until .load() is called.

Parameters:

  • metadata (pd.DataFrame) – The (possibly already filtered) catalog rows this query covers.

  • store (S3Store) – Used by load() to range-read the matching epochs.

Examples

>>> q = avs.epochs(event_type='fixation_scene').where("object_label == 'dog'")
>>> len(q)
62
>>> epochs = q.load(picks=['grad'])
__init__(metadata: DataFrame, store: S3Store)[source]
where(expr: str) → EpochQuery[source]

Filter by a pandas.DataFrame.query expression over the catalog columns (object_label, sceneID, duration, subject, session, fix_sequence, …). Returns a new, narrower EpochQuery – no data is fetched.

load(picks: Sequence[str] = ('grad', 'mag'), max_workers: int = 8) → mne.Epochs[source]

Range-read only the matching epochs and assemble them into one mne.Epochs, row-aligned with .metadata.

Chunk reads run concurrently across a process pool (see module docstring for why threads don’t work for this) – each is an independent HTTP request, so overlapping them cuts wall-clock time roughly in proportion to max_workers for queries spread across enough files/epochs to fill the pool.

Parameters:

  • picks (sequence of str, optional) – Which ROI arrays to read (default: both 'grad' and 'mag', matching the local API’s default combination).

  • max_workers (int, optional) – Concurrent read processes (default: 8). Higher isn’t free – each worker is a full process (measured startup cost is already included in the ~6.4x speedup this default achieves), and bucket-side throttling under heavy concurrency is unmeasured; 8 is an untuned starting point, not a validated ceiling.

Return type:

mne.Epochs

S3 Store

On-demand fetching from the public AVS S3 bucket.

Mirrors pyavs.scenes.fetch.fetch_scene_image()’s download pattern: a plain HTTPS GET (the bucket is public-read, so no credentials or AWS SDK are needed), written atomically (temp file + rename) so an interrupted download never leaves a corrupt file behind, and cached locally so repeat calls skip the network.

v1 only fetches whole objects. Chunk-level HTTP range reads for content-indexed epoch queries (e.g. “every fixation on a dog, across subjects”) are a separate, larger piece of work — see release/remote_dataloader_design.md, not implemented here.

exception pyavs.remote.store.RemoteFileNotFoundError[source]

Bases: FileNotFoundError

Raised when the bucket has no object at the requested key.

class pyavs.remote.store.S3Store(cache_root: str | Path | None = None, bucket: str = 'kietzmannlab-avs', region: str = 'us-west-2', timeout: int = 30, verbose: bool = True)[source]

Bases: object

Fetches objects from the public AVS S3 bucket, caching them locally.

Parameters:

  • cache_root (str or Path, optional) – Local directory to cache fetched objects under, mirroring the release tree’s relative layout exactly (so a Layout pointed at cache_root resolves to the same paths). Defaults to ~/.cache/pyavs/<bucket>.

  • bucket (str, optional) – S3 bucket name (default: the public AVS release bucket).

  • region (str, optional) – Bucket region (default: 'us-west-2').

  • timeout (int, optional) – HTTP request timeout in seconds (default: 30).

  • verbose (bool, optional) – Log size/time/cache-location feedback for each fetch (default: True). Set False to fetch silently.

__init__(cache_root: str | Path | None = None, bucket: str = 'kietzmannlab-avs', region: str = 'us-west-2', timeout: int = 30, verbose: bool = True)[source]
url_for(dst: str) → str[source]

Public HTTPS URL for a release-relative key, e.g. 'sub-01/ses-01/meg/as01a01.fif'.

fetch(dst: str, force: bool = False) → Path[source]

Fetch one object into the local cache, returning its path.

Parameters:

  • dst (str) – Release-relative key, identical to manifest.tsv’s dst column (e.g. 'derivatives/pyavs/sub-01/ses-01/epochs/sub-01_ses-01_task-avs_fixation_scene_epochs.h5').

  • force (bool, optional) – Re-download even if a cached copy of the expected size already exists (default: False).

Returns:

Local cached path.

Return type:

Path

Raises:

RemoteFileNotFoundError – If the bucket has no object at dst.