39.1. MemoryStore

39.1.1. Core Memory Management

Mock memory module for documentation build.

This module provides mock objects for the memories.core.memory module to allow documentation to be built without requiring all dependencies.

class memories.core.memory.MemoryManager

Bases: object

Mock memory manager for documentation.

__init__(*args, **kwargs)

Initialize the mock memory manager.

initialize(*args, **kwargs)

Initialize memory systems.

store(*args, **kwargs)

Store data in memory.

retrieve(*args, **kwargs)

Retrieve data from memory.

39.1.2. MemoryStore Class

39.1.3. Memory Tiers

39.1.3.1. Hot Memory

class memories.core.hot.HotMemory

Bases: object

Hot memory layer using DuckDB for fast in-memory storage.

__init__()

Initialize hot memory with in-memory DuckDB.

async store(data, metadata=None, tags=None)

Store data in hot memory with metadata and tags.

Parameters:

• data (Any) – Data to store

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

Returns:

True if storage was successful, False otherwise

Return type:

bool

async retrieve(query, tags=None)

Retrieve data from hot memory.

Parameters:

• query (Dict[str, Any]) – Query parameters (can contain ‘id’ to retrieve specific data)

• tags (List[str] | None) – Optional tags to filter by

Returns:

Retrieved data or None if not found

Return type:

Dict[str, Any] | None

clear()

Clear hot memory.

Return type:

None

cleanup()

Clean up resources.

Return type:

None

__del__()

Destructor to ensure cleanup is performed.

async get_schema(data_id)

Get schema information for stored data.

Parameters:

data_id (str) – ID of the data to get schema for

Returns:

Dictionary containing schema information or None if not found

Return type:

Dict[str, Any] | None

39.1.3.2. Warm Memory

class memories.core.warm.WarmMemory

Bases: object

Warm memory layer using DuckDB for storage.

__init__(storage_path=None)

Initialize warm memory.

Parameters:

storage_path (str) – Optional path to store DuckDB files

async store(data, metadata=None, tags=None, db_name=None, table_name=None)

Store data in warm memory with metadata and tags.

Parameters:

• data (Any) – Data to store

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

• db_name (str | None) – Optional name of the database file to store in (without .duckdb extension)

• table_name (str | None) – Optional name for the table to create. If None, a name will be generated.

Returns:

• success: True if storage was successful, False otherwise

• data_id: The unique ID of the stored data

• table_name: The name of the table where data is stored

Return type:

Dict containing success status and table information

async retrieve(query=None, tags=None, db_name=None, table_name=None)

Retrieve data from warm memory.

Parameters:

• query (Dict[str, Any] | None) – Optional query parameters

• tags (List[str] | None) – Optional tags to filter by

• db_name (str | None) – Optional name of the database file to retrieve from (without .duckdb extension)

• table_name (str | None) – Optional name of the specific table to query

Returns:

Retrieved data or None if not found

Return type:

Dict[str, Any] | List[Dict[str, Any]] | None

clear()

Clear all data from warm memory.

Return type:

None

cleanup()

Clean up resources.

Return type:

None

__del__()

Destructor to ensure cleanup is performed.

async get_schema(data_id)

Get schema information for stored data.

Parameters:

data_id (str) – ID of the data to get schema for

Returns:

Dictionary containing schema information or None if not found

Return type:

Dict[str, Any] | None

get_connection(db_name=None)

Get a connection to a specific database file.

Parameters:

db_name (str | None) – Name of the database file (without .duckdb extension). If None, returns the default connection.

Returns:

DuckDB connection

Return type:

DuckDBPyConnection

async import_from_parquet(parquet_file, metadata=None, tags=None, db_name=None, table_name=None)

Import data from a parquet file into warm memory.

Parameters:

• parquet_file (str) – Path to the parquet file

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

• db_name (str | None) – Optional name of the database file to store in (without .duckdb extension)

• table_name (str | None) – Optional name for the table to create. If None, a name will be generated.

Returns:

• success: True if import was successful, False otherwise

• data_id: The unique ID of the stored data

• table_name: The name of the table where data is stored

Return type:

Dict containing success status and table information

async import_from_duckdb(source_db_file, tables=None, metadata=None, tags=None, db_name=None)

Import tables from another DuckDB database using direct table copying.

Parameters:

• source_db_file (str) – Path to the source DuckDB database file

• tables (List[str] | None) – Optional list of table names to import. If None, imports all tables.

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

• db_name (str | None) – Optional name of the database file to store in (without .duckdb extension)

Returns:

• success: True if import was successful, False otherwise

• imported_tables: List of imported table names

• data_ids: Dict mapping table names to data IDs

Return type:

Dict containing success status and table information

async import_from_csv(csv_file, metadata=None, tags=None, db_name=None, table_name=None)

Import data from a CSV file into warm memory.

Parameters:

• csv_file (str) – Path to the CSV file

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

• db_name (str | None) – Optional name of the database file to store in (without .duckdb extension)

• table_name (str | None) – Optional name for the table to create. If None, a name will be generated.

Returns:

• success: True if import was successful, False otherwise

• data_id: The unique ID of the stored data

• table_name: The name of the table where data is stored

Return type:

Dict containing success status and table information

39.1.3.3. Cold Memory

class memories.core.cold.ColdMemory

Bases: object

Cold memory layer using DuckDB for persistent storage.

__init__()

Initialize cold memory.

async register_external_file(file_path)

Register an external file in the cold storage metadata.

Parameters:

file_path (str) –

Return type:

None

async store(data, metadata=None, tags=None)

Store data in cold storage.

Parameters:

• data (Any) – Data to store (DataFrame or dictionary)

• metadata (Dict[str, Any] | None) – Optional metadata about the data

• tags (List[str] | None) – Optional tags for categorizing the data

Returns:

True if storage was successful, False otherwise

Return type:

bool

async retrieve(query)

Retrieve data from cold storage.

Parameters:

query (Dict[str, Any]) –

Return type:

Dict[str, Any] | None

async clear()

Clear all data from cold storage.

Return type:

None

async unregister_file(file_id)

Unregister a specific file from cold storage.

Parameters:

file_id (str) – ID of the file to unregister

Returns:

True if successful, False otherwise

Return type:

bool

async list_registered_files()

List all registered files and their metadata.

Return type:

List[Dict]

cleanup()

Cleanup resources.

Return type:

None

__del__()

Ensure cleanup is called when object is destroyed.

async get_all_schemas()

Get all file paths from cold storage metadata and extract their schemas.

async get_schema(data_id)

Get schema information for stored data.

Parameters:

data_id (str) – ID of the data to get schema for

Returns:

• columns: List of column names

  • dtypes: Dictionary mapping column names to their data types

  • type: Type of schema (e.g., ‘table’, ‘file’, ‘dataframe’)

  • source: Source of the schema (e.g., ‘duckdb’, ‘parquet’, ‘json’)

Returns None if data not found or schema cannot be determined

Return type:

Dictionary containing

39.1.3.4. Glacier Memory

class memories.core.glacier.GlacierMemory

Bases: object

Base class for Glacier Memory.

__init__()

Initialize Glacier Memory.

async retrieve(query)

Retrieve data from a glacier source.

Parameters:

query (Dict[str, Any]) – Query dictionary containing: - source: Name of the source (‘osm’, ‘overture’, etc.) - Other source-specific parameters

Returns:

Retrieved data or None if not found

Return type:

Optional[Dict[str, Any]]

Raises:

ValueError – If source is not supported or query is invalid

get_available_sources()

Get list of available data sources.

Returns:

List of supported source names

Return type:

List[str]

register_connector(name, connector)

Register a connector.

Parameters:

• name (str) –

• connector (Any) –

Return type:

None

39.1.4. Example Usage

from memories import MemoryStore, Config

# Initialize configuration
config = Config(
    storage_path="./data",
    hot_memory_size=50,  # MB
    warm_memory_size=200,  # MB
    cold_memory_size=1000  # MB
)

# Create memory store
memory_store = MemoryStore(config)

# Store data
data = {
    "timestamp": "2025-02-17T12:00:00",
    "location": {"lat": 40.7128, "lon": -74.0060},
    "measurements": {"temperature": 25.5, "humidity": 60}
}
memory_store.store(data)

# Retrieve data
result = memory_store.retrieve(
    query={"location.lat": {"$gt": 40}},
    time_range=("2025-02-17T00:00:00", "2025-02-17T23:59:59")
)

39.1.5. Configuration Options

The MemoryStore can be configured with various options:

• storage_path: Base path for storing memory data

• hot_memory_size: Size limit for hot memory in MB

• warm_memory_size: Size limit for warm memory in MB

• cold_memory_size: Size limit for cold memory in MB

• enable_compression: Enable data compression (default: True)

• compression_level: Compression level (1-9, default: 6)

• enable_encryption: Enable data encryption (default: False)

• encryption_key: Encryption key for secure storage