Scene Analysis (pyavs.scenes)¶

The scenes module provides functions for analyzing visual scenes and mapping eye movements to objects.

Object Detection and Mapping¶

Object detection and mapping for pyAVS package.

This module provides memory-efficient functions for mapping eye tracking fixations to MSCOCO objects in scene images used in the Active Visual Semantics experiment.

Key features: - Compressed mask storage using RLE encoding (90-95% space reduction) - Spatial indexing for faster coordinate lookups - On-demand mask computation and loading - Memory usage scales with active objects, not total objects

class pyavs.scenes.objects.ObjectMaskMetadata(scene_id: int, category_id: int, category_name: str, bbox: Tuple[int, int, int, int], area: int, compressed_mask_key: str)[source]¶

Bases: object

Metadata for object masks to enable efficient storage and retrieval.

scene_id: int¶

category_id: int¶

category_name: str¶

bbox: Tuple[int, int, int, int]¶

area: int¶

compressed_mask_key: str¶

__init__(scene_id: int, category_id: int, category_name: str, bbox: Tuple[int, int, int, int], area: int, compressed_mask_key: str) → None¶

class pyavs.scenes.objects.CocoObjectMasker(annotation_dir: str, output_dir: str, mscoco_image_dir: str)[source]¶

Bases: object

Memory-efficient MSCOCO object masker using compressed storage.

Instead of storing full boolean masks, this class: 1. Stores RLE-compressed masks from COCO annotations directly 2. Creates spatial indices for fast coordinate lookups 3. Only decompresses masks when needed

This provides 90-95% reduction in storage space compared to full mask storage.

__init__(annotation_dir: str, output_dir: str, mscoco_image_dir: str)[source]¶

Initialize the object masker.

Parameters:

annotation_dir (str) – Path to MSCOCO annotation directory
output_dir (str) – Path to output directory for compressed masks
mscoco_image_dir (str) – Path to MSCOCO images directory

compute_masks_for_image(coco_id: int)[source]¶

Compute and store compressed object masks for a single image.

Parameters:: coco_id (int) – COCO image ID

compute_masks(coco_ids: int | List[int]) → str[source]¶

Compute compressed object masks for multiple images.

Parameters:: coco_ids (int or list of int) – COCO image ID(s)
Returns:: Path to metadata file
Return type:: str

load_mask_for_category(coco_id: int, category_id: int) → ndarray | None[source]¶

Load and decompress mask for a specific category.

Parameters:

coco_id (int) – COCO image ID
category_id (int) – Object category ID

Returns:

Decompressed boolean mask or None if not found

Return type:

np.ndarray or None

get_scene_metadata(coco_id: int) → List[ObjectMaskMetadata][source]¶: Get metadata for all objects in a scene.

close()[source]¶: Close any open resources.

class pyavs.scenes.objects.FixationObjectChecker(transformed_annotations_dir: str, use_cocostuff: bool = False)[source]¶

Bases: object

Fixation object checker using transformed AVS scene annotations.

This class works with pre-transformed annotations that match the processed scene format used in the AVS experiment. Annotations are loaded from JSON files created by the AVS scene annotation transformer.

__init__(transformed_annotations_dir: str, use_cocostuff: bool = False)[source]¶

Initialize the fixation object checker.

Parameters:

transformed_annotations_dir (str) – Path to directory containing transformed annotation JSON files
use_cocostuff (bool, default=False) – If True, use COCO-Stuff annotations (183 classes: 1 unlabeled + 80 things + 91 stuff + 11 missing). If False, use standard COCO annotations (80 thing classes only).

get_fixated_objects(coco_id: int, x_pos: float | ndarray, y_pos: float | ndarray, error_margin_pixels: int = 10) → Tuple[List[int], List[str]][source]¶

Check which objects are fixated at given coordinates with error margin tolerance.

This method first checks for direct hits at the exact fixation coordinates. If no object is found, it searches within an error margin around the fixation to account for eye tracker noise and calibration drift.

Parameters:

coco_id (int) – COCO image ID
x_pos (float or array) – Screen-centered x coordinates (pixels)
y_pos (float or array) – Screen-centered y coordinates (pixels)
error_margin_pixels (int, optional) – Search radius in pixels around fixation for nearest object (default: 10) This accounts for eye tracker noise and calibration drift.

Returns:

(object_category_ids, object_category_names)

Return type:

tuple

clear_cache()[source]¶: Clear the annotation cache.

pyavs.scenes.objects.get_fixated_objects(events_df: DataFrame, transformed_annotations_dir: str, verbose: bool = False, error_margin_pixels: int = 10, use_cocostuff: bool = True) → DataFrame[source]¶

Add object labels to fixation events using transformed AVS scene annotations.

This function uses pre-transformed annotations that match the processed scene format used in the AVS experiment, providing more accurate object detection. Includes error margin tolerance to account for eye tracker noise and calibration drift.

Parameters:

events_df (pd.DataFrame) – Eye tracking events dataframe
transformed_annotations_dir (str) – Path to directory containing transformed annotation JSON files
verbose (bool, optional) – Whether to print progress information (default: False)
error_margin_pixels (int, optional) – Search radius in pixels around fixation for nearest object (default: 10) This accounts for eye tracker noise and calibration drift.
use_cocostuff (bool, optional) – If True, use COCO-Stuff annotations (172 classes: 80 things + 91 stuff + 1 unlabeled). If False, use standard COCO annotations (80 thing classes only). Default is True (COCO-Stuff mode).

Returns:

Events dataframe with object_label and object_id columns added

Return type:

pd.DataFrame

Notes

COCO-Stuff mode (use_cocostuff=True) provides better coverage by including amorphous background regions like sky, grass, walls, water, etc. This typically increases fixation labeling coverage by 20-40% compared to COCO-only mode.

pyavs.scenes.objects.load_object_masks(scene_ids: int | List[int], masks_dir: str | None = None) → Dict[int, Dict[str, ndarray]][source]¶

Load precomputed RLE object masks for specified scene IDs.

Parameters:

scene_ids (int or list of int) – Scene ID(s) to load masks for
masks_dir (str) – Directory holding object_masks_metadata.json and compressed_masks/, as produced by CocoObjectMasker.

Returns:

Dictionary mapping scene IDs to object masks

Return type:

dict

Raises:

FileNotFoundError – If masks_dir is not given, or does not contain the metadata file. These masks are not part of the public release — see get_fixated_objects() for the annotation-based equivalent.

pyavs.scenes.objects.map_fixations_to_objects(fixations_df: DataFrame, scene_id: int, x_col: str = 'mean_gx', y_col: str = 'mean_gy', data_path: str | None = None, use_cocostuff: bool = False, transformed_annotations_dir: str | None = None) → DataFrame[source]¶

Map fixations to objects for a single scene.

Parameters:

fixations_df (pd.DataFrame) – Dataframe containing fixation data
scene_id (int) – COCO scene ID
x_col (str, optional) – Column name for x coordinates (default: ‘mean_gx’)
y_col (str, optional) – Column name for y coordinates (default: ‘mean_gy’)
data_path (str, optional) – avs-public root. If None, uses the configured data path.
use_cocostuff (bool, optional) – Use the COCO-Stuff annotations (183 classes) rather than the 80 COCO thing classes (default: False).
transformed_annotations_dir (str, optional) – Explicit annotation directory, overriding data_path.

Returns:

Fixations dataframe with object_id and object_label columns added.

Return type:

pd.DataFrame

pyavs.scenes.objects.categorize_objects(object_names: List[str], level: str = 'subcategory') → List[str][source]¶

Categorize object names into broader categories for RSA analysis.

Parameters:

object_names (list of str) – List of object names to categorize
level (str, optional) – Categorization level: ‘main_category’, ‘subcategory’, or ‘hierarchical’ Default: ‘subcategory’

Returns:

List of category names for each object

Return type:

list of str

Examples

>>> categorize_objects(['person', 'car', 'dog'], level='main_category')
['animate', 'inanimate', 'animate']

>>> categorize_objects(['person', 'car', 'dog'], level='subcategory')
['human', 'vehicle_ground', 'mammal_small']

pyavs.scenes.objects.sort_objects_by_category(object_names: List[str], level: str = 'subcategory') → Tuple[List[str], List[int]][source]¶

Sort objects by their categories and return sorted objects with indices.

Parameters:

object_names (list of str) – List of object names to sort
level (str, optional) – Categorization level for sorting: ‘main_category’, ‘subcategory’, or ‘hierarchical’ Default: ‘subcategory’

Returns:

(sorted_objects, sort_indices) where sort_indices maps new positions to original positions

Return type:

tuple

Examples

>>> objects = ['car', 'person', 'dog']
>>> sorted_objs, indices = sort_objects_by_category(objects)
>>> print(sorted_objs)  # ['person', 'dog', 'car'] (animate first, then inanimate)
>>> print(indices)      # [1, 2, 0] (person was at index 1, dog at 2, car at 0)

pyavs.scenes.objects.get_supercategory_palette()[source]¶: Return {supercategory: (r,g,b)} using husl palette, cached.

Image Cropping¶

Scene cropping utilities for pyAVS package.

This module provides functions for creating fixation-based crops from scene images and extracting regions of interest based on eye tracking data.

pyavs.scenes.crops.create_fixation_crops(eye_events_df: DataFrame, scene_images: Dict[int, str], config: PyAVSConfig, crop_size: Tuple[int, int] = (100, 100), output_dir: str | None = None, save_crops: bool = False, center_on: str = 'mean') → Dict[str, ndarray][source]¶

Create fixation-based crops from scene images.

Parameters:

eye_events_df (pd.DataFrame) – Eye tracking events dataframe with fixation locations
scene_images (dict) – Dictionary mapping scene IDs to image file paths
config (PyAVSConfig) – Configuration object with visual system parameters (required)
crop_size (tuple of int, optional) – Size of crops in pixels (width, height) (default: (100, 100))
output_dir (str, optional) – Directory to save crops. If None, crops are not saved
save_crops (bool, optional) – Whether to save crops to disk (default: False)
center_on (str, optional) – Coordinate type to center on (‘mean’, ‘start’, ‘end’) (default: ‘mean’)

Returns:

Dictionary mapping crop IDs to crop arrays

Return type:

dict

pyavs.scenes.crops.extract_scene_regions(scene_id: int, regions: List[Tuple[int, int, int, int]], scene_images: Dict[int, str] | None = None, data_path: str | None = None) → List[ndarray][source]¶

Extract rectangular regions from a scene image.

Parameters:

scene_id (int) – COCO scene ID
regions (list of tuple) – List of regions as (left, top, width, height) tuples
scene_images (dict, optional) – Dictionary mapping scene IDs to image paths
data_path (str, optional) – avs-public root. If None, uses the configured data path.

Returns:

List of extracted region arrays

Return type:

list of np.ndarray

pyavs.scenes.crops.create_object_based_crops(scene_id: int, object_ids: List[int], config: PyAVSConfig, crop_size: Tuple[int, int] = (100, 100), scene_images: Dict[int, str] | None = None, data_path: str | None = None, masks_dir: str | None = None) → Dict[int, ndarray][source]¶

Create crops centered on object centers of mass.

Parameters:

scene_id (int) – COCO scene ID
object_ids (list of int) – List of object category IDs to crop
config (PyAVSConfig) – Configuration object with visual system parameters (required)
crop_size (tuple of int, optional) – Size of crops in pixels (width, height) (default: (100, 100))
scene_images (dict, optional) – Dictionary mapping scene IDs to image paths
data_path (str, optional) – avs-public root, used to locate the scene image. If None, uses the configured data path.
masks_dir (str, optional) – Directory of precomputed RLE object masks. Not part of the public release — without it this function raises; see pyavs.scenes.objects.load_object_masks().

Returns:

Dictionary mapping object IDs to crop arrays

Return type:

dict

pyavs.scenes.crops.visualize_fixations_on_scene(scene_id: int, fixations_df: DataFrame, config: PyAVSConfig, scene_images: Dict[int, str] | None = None, data_path: str | None = None, figsize: Tuple[int, int] = (12, 8), save_path: str | None = None) → Figure[source]¶

Visualize fixations overlaid on scene image.

Parameters:

scene_id (int) – COCO scene ID
fixations_df (pd.DataFrame) – Dataframe containing fixation data for this scene
config (PyAVSConfig) – Configuration object with visual system parameters (required)
scene_images (dict, optional) – Dictionary mapping scene IDs to image paths
data_path (str, optional) – avs-public root. If None, uses the configured data path.
figsize (tuple of int, optional) – Figure size (width, height) (default: (12, 8))
save_path (str, optional) – Path to save the visualization

Returns:

Matplotlib figure object

Return type:

plt.Figure

COCO-Stuff Category Definitions¶

COCO-Stuff class definitions and utilities.

This module provides constants and utilities for working with COCO-Stuff annotations, which include 80 thing classes, 91 stuff classes, and 1 unlabeled class.

COCO-Stuff extends the COCO dataset with dense pixel-level annotations for amorphous regions (stuff) like sky, grass, walls, water, etc. This provides comprehensive scene segmentation for fixation object detection.

Class Structure: - Index 0: unlabeled (background) - Indices 1-91: Thing classes (80 actual classes with segmentation, 11 missing) - Indices 92-182: Stuff classes (91 amorphous regions) - Total: 183 classes (0-182)

References: - COCO-Stuff paper: https://arxiv.org/abs/1612.03716 - GitHub: https://github.com/nightrome/cocostuff - Labels: https://github.com/nightrome/cocostuff/blob/master/labels.md

Author: pyAVS development team

pyavs.scenes.cocostuff_classes.get_class_name(class_id: int) → str[source]¶

Get class name from COCO-Stuff class ID.

Parameters:: class_id (int) – COCO-Stuff class ID (0-182)
Returns:: Class name, or ‘unknown’ if ID is out of range
Return type:: str

Examples

>>> get_class_name(0)
'unlabeled'
>>> get_class_name(1)
'person'
>>> get_class_name(92)
'banner'
>>> get_class_name(182)
'wood'

pyavs.scenes.cocostuff_classes.get_class_id(class_name: str) → int | None[source]¶

Get COCO-Stuff class ID from class name.

Parameters:: class_name (str) – Class name (e.g., ‘person’, ‘sky-other’)
Returns:: Class ID (0-182), or None if name not found
Return type:: int or None

Examples

>>> get_class_id('person')
1
>>> get_class_id('banner')
92
>>> get_class_id('nonexistent')
None

pyavs.scenes.cocostuff_classes.is_thing_class(class_id: int) → bool[source]¶

Check if class ID represents a thing class.

Thing classes are countable objects with defined boundaries (1-91, excluding missing).

Parameters:: class_id (int) – COCO-Stuff class ID
Returns:: True if class is a thing, False otherwise
Return type:: bool

Examples

>>> is_thing_class(1)  # person
True
>>> is_thing_class(92)  # banner (stuff)
False
>>> is_thing_class(0)  # unlabeled
False

pyavs.scenes.cocostuff_classes.is_stuff_class(class_id: int) → bool[source]¶

Check if class ID represents a stuff class.

Stuff classes are amorphous regions without defined boundaries (92-182).

Parameters:: class_id (int) – COCO-Stuff class ID
Returns:: True if class is stuff, False otherwise
Return type:: bool

Examples

>>> is_stuff_class(92)  # banner
True
>>> is_stuff_class(182)  # wood
True
>>> is_stuff_class(1)  # person (thing)
False

pyavs.scenes.cocostuff_classes.get_annotation_type(class_id: int) → str[source]¶

Get annotation type: ‘thing’, ‘stuff’, ‘unlabeled’, or ‘unknown’.

Parameters:: class_id (int) – COCO-Stuff class ID
Returns:: Annotation type
Return type:: str

Examples

>>> get_annotation_type(0)
'unlabeled'
>>> get_annotation_type(1)
'thing'
>>> get_annotation_type(92)
'stuff'
>>> get_annotation_type(999)
'unknown'

pyavs.scenes.cocostuff_classes.get_summary() → dict[source]¶

Get summary statistics about COCO-Stuff classes.

Returns:: Dictionary with class counts and index ranges
Return type:: dict

Examples

>>> summary = get_summary()
>>> summary['total_classes']
183
>>> summary['num_things']
80
>>> summary['num_stuff']
91

License-Filtered Image Subsets¶

Extract MSCOCO images with permissive licenses for use in academic papers.

This module parses COCO annotation files to identify images with licenses that allow usage in academic publications (with proper attribution).

Usage:

python -m pyavs.scenes.coco_licenses –coco-dir /share/klab/datasets/avs/input/annotations/: –output permissive_images.csv

# With Flickr metadata enrichment python /home/student/p/psulewski/pyAVS/pyavs/scenes/coco_licenses.py –coco-dir /share/klab/datasets/avs/input/annotations/ –output permissive_images.csv –flickr-api-key YOUR_API_KEY

Author: psulewski

pyavs.scenes.coco_licenses.get_avs_scene_ids(avs_scenes_dir: str) → set[int][source]¶

Get set of COCO IDs for AVS scenes from the scenes directory.

Parameters:: avs_scenes_dir (str) – Path to AVS scenes directory containing scene images
Returns:: Set of COCO image IDs used as AVS scenes
Return type:: set[int]

pyavs.scenes.coco_licenses.extract_flickr_photo_id(flickr_url: str) → str | None[source]¶

Extract photo ID from Flickr static URL.

Parameters:: flickr_url (str) – Flickr static URL in format: http://farm{N}.staticflickr.com/{server}/{photo_id}_{secret}_{size}.jpg
Returns:: Photo ID if successfully extracted, None otherwise
Return type:: str | None

pyavs.scenes.coco_licenses.fetch_flickr_metadata(photo_id: str, api_key: str) → dict | None[source]¶

Fetch photo and owner metadata from Flickr API.

Parameters:

photo_id (str) – Flickr photo ID
api_key (str) – Flickr API key

Returns:

Dictionary with photo and owner metadata, or None if photo not found or error occurred. Keys include:

Owner info: flickr_username, flickr_realname, flickr_nsid, flickr_owner_location, flickr_path_alias
Photo info: flickr_title, flickr_description, flickr_date_taken, flickr_date_uploaded, flickr_last_update, flickr_page_url, flickr_license_id, flickr_views, flickr_tags
Location: flickr_latitude, flickr_longitude, flickr_geo_accuracy, flickr_locality, flickr_county, flickr_region, flickr_country

Return type:

dict | None

pyavs.scenes.coco_licenses.enrich_with_flickr_metadata(df: DataFrame, api_key: str) → DataFrame[source]¶

Add Flickr photo and owner metadata columns to DataFrame.

Parameters:

df (pd.DataFrame) – DataFrame with ‘flickr_url’ column
api_key (str) – Flickr API key

Returns:

DataFrame with added columns:

flickr_photo_id: Extracted photo ID from URL
Owner info: flickr_username, flickr_realname, flickr_nsid, flickr_owner_location, flickr_path_alias
Photo info: flickr_title, flickr_description, flickr_date_taken, flickr_date_uploaded, flickr_last_update, flickr_page_url, flickr_license_id, flickr_views, flickr_tags
Location: flickr_latitude, flickr_longitude, flickr_geo_accuracy, flickr_locality, flickr_county, flickr_region, flickr_country

Return type:

pd.DataFrame

pyavs.scenes.coco_licenses.extract_licensed_images(annotation_file: str, split: str | None = None, filter_permissive: bool = True) → DataFrame[source]¶

Extract image license metadata from COCO annotations.

Parameters:

annotation_file (str) – Path to COCO annotation JSON file (e.g., instances_val2017.json)
split (str, optional) – Split name to add as column (e.g., ‘train’, ‘val’)
filter_permissive (bool, default True) – If True, keep only images whose license is in PERMISSIVE_LICENSE_IDS. If False, return license metadata for all images regardless of license (e.g. to document per-image licenses for a fixed image set without excluding any of them).

Returns:

DataFrame containing image license metadata (all images, or only the permissively licensed ones if filter_permissive is True). Columns: coco_id, file_name, license_id, license_name, license_url, flickr_url, coco_url, width, height, split (if provided)

Return type:

pd.DataFrame

pyavs.scenes.coco_licenses.extract_from_coco_dir(coco_dir: str, filter_permissive: bool = True) → DataFrame[source]¶

Extract image license metadata from both train and val splits.

Parameters:

coco_dir (str) – Path to COCO annotations directory containing instances_train2017.json and instances_val2017.json
filter_permissive (bool, default True) – If True, keep only permissively licensed images. If False, return license metadata for all images regardless of license.

Returns:

Combined DataFrame with images from both splits

Return type:

pd.DataFrame

pyavs.scenes.coco_licenses.main()[source]¶: Main function for command line execution.

Scene Annotation Transformation¶

Scene Annotation Transformer for pyAVS

This script transforms MSCOCO object annotations to match the processed scene format used in the AVS experiment. It applies the same center-crop and resize transformations that were applied to scene images by scene_resizer.py.

The transformed annotations are stored in DATA_DIR/stimuli/annotations/coco_objects for use by the FixationObjectChecker.

Usage:: python -m pyavs.scenes.transform_scene_annotations [–avs-scenes-dir DIR] [–output-dir DIR] [–verbose]

Author: pyAVS development team

pyavs.scenes.transform_scene_annotations.crop_resize(image: Image, size: Tuple[int, int], ratio: Fraction, resample: int = 1) → Image[source]¶

Apply center crop and resize - matching scene_resizer.py logic.

Parameters:

image (PIL.Image) – Input image
size (tuple) – Target size (width, height)
ratio (Fraction) – Target aspect ratio
resample (int, optional) – Resampling filter. Use Image.LANCZOS for photos, Image.NEAREST for masks.

Returns:

Transformed image

Return type:

PIL.Image

pyavs.scenes.transform_scene_annotations.default_target_size_and_ratio(config: PyAVSConfig | None = None) → Tuple[Tuple[int, int], Fraction][source]¶

Target (width, height) and aspect ratio for AVS MEG-size scene images.

Parameters:: config (PyAVSConfig, optional) – Source of screen_size_pixels/screen_usage. Defaults to PyAVSConfig().
Returns:: (target_size, target_ratio), e.g. ((947, 710), Fraction(947, 710)).
Return type:: tuple

class pyavs.scenes.transform_scene_annotations.AVSSceneAnnotationTransformer(avs_scenes_dir: str, output_dir: str, mscoco_annotations_dir: str, mscoco_images_dir: str, use_cocostuff: bool = False, verbose: bool = False)[source]¶

Bases: object

Transforms MSCOCO annotations to match AVS processed scene format.

This class applies the same transformations (center-crop + resize) that scene_resizer.py applied to the original scene images.

__init__(avs_scenes_dir: str, output_dir: str, mscoco_annotations_dir: str, mscoco_images_dir: str, use_cocostuff: bool = False, verbose: bool = False)[source]¶

Initialize the transformer.

Parameters:

avs_scenes_dir (str) – Directory containing processed AVS scene images
output_dir (str) – Output directory for transformed annotations
mscoco_annotations_dir (str) – Directory containing MSCOCO annotation files
mscoco_images_dir (str) – Directory containing original MSCOCO images
use_cocostuff (bool, optional) – If True, load and process COCO-Stuff annotations (172 classes). If False, use only COCO instances (80 classes). Default: False for backward compatibility.
verbose (bool) – Enable verbose logging

crop_resize(image: Image, size: Tuple[int, int], ratio: Fraction, resample: int = 1) → Image[source]¶: Apply center crop and resize. See module-level crop_resize().

transform_scene_annotations(scene_filename: str) → bool[source]¶

Transform annotations for a single scene.

Parameters:: scene_filename (str) – Filename of the scene image
Returns:: True if successful, False otherwise
Return type:: bool

transform_all_scenes() → Dict[str, int][source]¶

Transform annotations for all scenes in the AVS scenes directory.

Returns:: Statistics about the transformation process
Return type:: dict

pyavs.scenes.transform_scene_annotations.main()[source]¶: Main entry point.

ANN Embeddings for Fixation Crops¶

Note

Unlike the rest of pyavs.scenes, this submodule and pyavs.captions are intentionally not re-exported from top-level pyavs – they’re submodule-only utilities used directly by the scripts that need them (e.g. compute_fixation_embeddings.py), not part of the core top-level API surface.

Neural network embeddings for fixation crops in pyAVS.

This module provides functions to extract ANN embeddings from stored fixation crop images using pre-trained models like ResNet50-EcoSet via thingsvision.

pyavs.scenes.embeddings.extract_embeddings_from_crops(crops_dir: str, output_dir: str, model_name: str = 'resnet50_ecoset_crop', layers: List[str] = ['avgpool'], batch_size: int = 64, device: str | None = None, weights_path: str | None = None, overwrite: bool = False, verbose: bool = False) → Dict[str, str][source]¶

Extract neural network embeddings from stored crop images using thingsvision.

This function follows the pattern from the old codebase, using thingsvision’s ImageDataset and DataLoader for efficient batch processing.

Parameters:

crops_dir (str) – Directory containing crop PNG files
output_dir (str) – Directory to save embeddings
model_name (str, default 'resnet50_ecoset_crop') – Model name for feature extraction
layers (list of str, default ['avgpool']) – Model layers to extract features from
batch_size (int, default 64) – Batch size for processing
device (str, optional) – Device to use (‘cuda’, ‘cpu’, ‘mps’). Auto-detected if None
weights_path (str, optional) – Path to custom model weights (e.g., EcoSet weights)
overwrite (bool, default False) – Whether to overwrite existing embeddings
verbose (bool, default False) – Print verbose output

Returns:

Dictionary mapping layer names to output file paths

Return type:

dict

pyavs.scenes.embeddings.get_default_ecoset_path() → str | None[source]¶

Get the path to EcoSet ResNet50 model weights, from PYAVS_ECOSET_WEIGHTS.

The EcoSet-trained ResNet50 checkpoint is a third-party model, not part of the AVS release. Set the PYAVS_ECOSET_WEIGHTS environment variable to a checkpoint of your own, or pass weights_path= explicitly, to use the ecoset_resnet50 model.

Returns:: Path to EcoSet weights if available, None otherwise
Return type:: str or None

pyavs.scenes.embeddings.get_available_models() → Dict[str, Any][source]¶

Get list of available models for crop embedding extraction.

Returns:: Dictionary of model categories and available models
Return type:: dict

pyavs.scenes.embeddings.create_bids_embeddings_path(subject_id: int, session: int, data_path: str, model_name: str) → str[source]¶

Create BIDS-compatible path for embeddings storage.

Parameters:

subject_id (int) – Subject ID
session (int) – Session number
data_path (str) – Base data path
model_name (str) – Model name for subdirectory

Returns:

BIDS-compatible path for embeddings

Return type:

str