MADSci Resource Manager - MADSci Documentation

Tracks and manages the full lifecycle of laboratory resources - assets, consumables, samples, containers, and labware.

Table of Contents¶

Features
Installation
Usage
- Quick Start
- Manager Setup
Configuration
Resource Client
Resource Types
- Core Resource Hierarchy
- Usage Examples
Integration with MADSci Ecosystem
Advanced Operations
Resource Templates
Resource Locking and Concurrency Control
Resource Hierarchy Queries
Performance Optimization
Error Handling and Troubleshooting
Quick Reference

Features¶

Comprehensive resource types: Assets, consumables, containers with specialized behaviors
Complete history tracking: Full audit trail with restore capabilities
Container hierarchy: Supports racks, plates, stacks, queues, grids, and custom containers
Quantity management: Track consumable quantities with capacity limits
Query system: Find resources by type, name, properties, or relationships
Constraint validation: Prevents logical errors like negative quantities or overflow

Installation¶

See the main README for installation options. This package is available as:

PyPI: pip install madsci.resource_manager
Docker: Included in ghcr.io/ad-sdl/madsci
Example configuration: See example_lab/managers/example_resource.manager.yaml

Dependencies: PostgreSQL database (see example_lab)

Usage¶

Quick Start¶

Use the example_lab as a starting point:

# Start with working example
docker compose up  # From repo root
# Resource Manager available at http://localhost:8003/docs

# Or run standalone
python -m madsci.resource_manager.resource_server

Manager Setup¶

For custom deployments, see example_resource.manager.yaml for configuration options.

Configuration¶

The Resource Manager uses environment variables for configuration with a hierarchical precedence system. All settings have defaults suitable for development.

Environment Variables¶

Core Settings:

# Service Configuration
RESOURCE_HOST=localhost                    # Server hostname
RESOURCE_PORT=8003                        # Server port
RESOURCE_LOG_LEVEL=INFO                   # Logging level

# Database Configuration
RESOURCE_POSTGRES_HOST=localhost          # PostgreSQL hostname
RESOURCE_POSTGRES_PORT=5432              # PostgreSQL port
RESOURCE_POSTGRES_USER=madsci            # Database username
RESOURCE_POSTGRES_PASSWORD=madsci        # Database password
RESOURCE_POSTGRES_DATABASE=madsci        # Database name

# Manager Integration
RESOURCE_EVENT_MANAGER_URL=http://localhost:8001    # Event logging

Advanced Settings:

# Development/Testing
RESOURCE_LOCAL_MODE=false                 # Run without external dependencies
RESOURCE_ENABLE_CORS=true                # Enable CORS for web clients

# Performance Tuning
RESOURCE_MAX_CONNECTIONS=20              # Database connection pool size
RESOURCE_QUERY_TIMEOUT=30                # Query timeout in seconds
RESOURCE_LOCK_DEFAULT_DURATION=300       # Default lock duration (seconds)

# Security
RESOURCE_REQUIRE_AUTHENTICATION=false    # Enable authentication
RESOURCE_API_KEY_HEADER=X-API-Key        # API key header name

Local Mode Configuration¶

For development or testing without external dependencies:

export RESOURCE_LOCAL_MODE=true
export RESOURCE_EVENT_MANAGER_URL=""     # Disable event logging
python -m madsci.resource_manager.resource_server

Local Mode Limitations:

No event logging integration
No distributed locking coordination
Single-process operation only
In-memory or file-based storage options

When to Use Local Mode:

Unit testing and development
Offline development environments
Single-process applications
Quick prototyping and experimentation

When to Use Server Mode:

Production deployments
Multi-process/multi-node environments
Integration with other MADSci managers
Distributed laboratory setups
When you need event logging and audit trails

Production Configuration¶

Docker Compose (Recommended):

version: '3.8'
services:
  resource_manager:
    image: ghcr.io/ad-sdl/madsci:latest
    environment:
      RESOURCE_POSTGRES_HOST: postgres
      RESOURCE_POSTGRES_PASSWORD: ${DB_PASSWORD}
      RESOURCE_LOG_LEVEL: WARNING
      RESOURCE_MAX_CONNECTIONS: 50
    depends_on:
      - postgres
    ports:
      - "8003:8003"

Environment File (.env):

# Database credentials
DB_PASSWORD=secure_password_here
POSTGRES_PASSWORD=secure_password_here

# Resource Manager settings
RESOURCE_LOG_LEVEL=INFO
RESOURCE_REQUIRE_AUTHENTICATION=true
RESOURCE_API_KEY_HEADER=Authorization

Database Setup¶

Initial Setup:

# Using Docker Compose
docker compose up -d postgres
docker compose exec postgres psql -U madsci -d madsci -c "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"

# Manual PostgreSQL setup
createdb -U postgres madsci
psql -U postgres -d madsci -c "CREATE USER madsci WITH PASSWORD 'madsci';"
psql -U postgres -d madsci -c "GRANT ALL PRIVILEGES ON DATABASE madsci TO madsci;"

Schema Migration:

# The Resource Manager automatically creates tables on startup
from madsci.resource_manager.resource_server import ResourceManagerServer
from madsci.resource_manager.resource_server import ResourceManagerSettings

settings = ResourceManagerSettings()
server = ResourceManagerServer(settings)
# Tables created automatically when server starts

Configuration Validation¶

# Validate configuration before starting
from madsci.resource_manager.resource_server import ResourceManagerSettings

try:
    settings = ResourceManagerSettings()
    print(f"✓ Configuration valid")
    print(f"  Database: {settings.postgres_host}:{settings.postgres_port}")
    print(f"  Server: {settings.host}:{settings.port}")
except Exception as e:
    print(f"✗ Configuration error: {e}")

Resource Client¶

Use ResourceClient to manage laboratory resources:

from madsci.client.resource_client import ResourceClient
from madsci.common.types.resource_types import Asset, Consumable, Grid
from madsci.common.types.resource_types.definitions import ResourceDefinition

client = ResourceClient("http://localhost:8003")

# Add a new asset (samples, labware, equipment)
sample = Asset(
    resource_name="Sample A1",
    resource_class="sample",
    attributes={"compound": "aspirin", "concentration": "10mM"}
)
added_sample = client.add_resource(sample)

# Add consumables with quantities
reagent = Consumable(
    resource_name="PBS Buffer",
    resource_class="reagent",
    quantity=500.0,
    attributes={"units": "mL"},
)
added_reagent = client.add_resource(reagent)

# Create containers (plates, racks, etc.)
plate = Grid(
    resource_name="96-well Plate #1",
    resource_class="plate",
    rows=8,
    columns=12
)
added_plate = client.add_resource(plate)

# Place samples in containers
client.set_child(resource=added_plate, key=(0, 0), child=added_sample)

# Query resources
samples = client.query_resource(resource_class="sample", multiple=True)
consumables = client.query_resource(resource_class="reagent", multiple=True)

# Manage consumable quantities
client.decrease_quantity(resource=added_reagent, amount=50.0)  # Use 50mL
client.increase_quantity(resource=added_reagent, amount=100.0) # Add 100mL

# Resource history and restoration
history = client.query_history(resource_id=added_sample.resource_id)
client.remove_resource(resource_id=added_sample.resource_id)  # Soft delete
client.restore_deleted_resource(resource_id=added_sample.resource_id)

# Query resource hierarchy
hierarchy = client.query_resource_hierarchy(resource_id=added_plate.resource_id)
print(f"Ancestors: {hierarchy.ancestor_ids}")
print(f"Descendants: {hierarchy.descendant_ids}")

Resource Types¶

Core Resource Hierarchy¶

Base Types:

Resource: Base class for all resources
Asset: Non-consumable resources (samples, labware, equipment)
Consumable: Resources with quantities that can be consumed

Container Types:

Container: Base for resources that hold other resources
Collection: Supports random access by key
Row: Single-dimensional containers
Grid: Two-dimensional containers (plates, racks)
VoxelGrid: Three-dimensional containers
Slot: Holds exactly zero or one child (plate nests)
Stack: LIFO access (stacked plates)
Queue: FIFO access (sample queues)
Pool: Mixed/collocated consumables

Usage Examples¶

# Different container types
tip_box = Grid(resource_name="Tip Box", rows=8, columns=12, resource_class="tips")
plate_stack = Stack(resource_name="Plate Stack", resource_class="plate_storage")
sample_rack = Row(resource_name="Sample Rack", length=24, resource_class="rack")

# Container operations
client.set_child(resource=tip_box, key=(0, 0), child=tip_sample)    # Grid access
client.push(resource=plate_stack, child=new_plate)                  # Stack push
client.pop(resource=plate_stack)                                    # Stack pop
client.set_child(resource=sample_rack, key=5, child=sample)         # Row access

Integration with MADSci Ecosystem¶

Resources integrate seamlessly with other MADSci components:

Workflows: Reference resources in step locations and arguments
Nodes: Access resource information during actions
Data Manager: Link data to specific resources and samples
Event Manager: Track resource lifecycle events

# Example: Node action using resources
@action
def process_sample(self, sample_resource_id: str) -> ActionResult:
    # Get sample attributes from Resource Manager
    sample = self.resource_client.get_resource(sample_resource_id)

    # Process based on sample properties
    result = self.device.analyze(sample.attributes["compound"])

    # Update sample with results
    sample.attributes["analysis_result"] = result
    self.resource_client.update_resource(sample)

    return ActionSucceeded(data=result)

Advanced Operations¶

Resource Definitions¶

Use ResourceDefinition for idempotent resource creation:

from madsci.common.types.resource_types.definitions import ResourceDefinition

# Creates new resource or attaches to existing one
resource_def = ResourceDefinition(
    resource_name="Standard Buffer",
    resource_class="reagent"
)
resource = client.init_resource(resource_def)  # Idempotent

Bulk Operations¶

# Query multiple resources
all_samples = client.query_resource(resource_class="sample", multiple=True)
empty_containers = client.query_resource(is_empty=True, multiple=True)

# Batch operations for consumables
for reagent in reagents:
    client.decrease_quantity(resource=reagent, amount=usage_amounts[reagent.resource_id])

History and Auditing¶

# Full resource history
history = client.query_history(resource_id=sample.resource_id)

# Query by time range and change type
import datetime
recent_updates = client.query_history(
    start_date=datetime.datetime.now() - datetime.timedelta(days=7),
    change_type="Updated"
)

Resource Templates¶

ResourceTemplates provide reusable blueprints for creating standardized laboratory resources. Templates help ensure consistency across resource creation and reduce configuration errors.

Creating Templates¶

Templates are created from existing resources and can be customized with metadata:

from madsci.client.resource_client import ResourceClient
from madsci.common.types.resource_types import Grid, Consumable

client = ResourceClient("http://localhost:8003")

# Create a standard 96-well plate resource
standard_plate = Grid(
    resource_name="Standard 96-Well Plate",
    resource_class="plate",
    rows=8,
    columns=12,
    attributes={
        "well_volume": 200,  # µL
        "material": "polystyrene",
        "sterilized": True
    }
)

# Create template from the resource
plate_template = client.create_template(
    resource=standard_plate,
    template_name="standard_96_well_plate",
    description="Standard 96-well polystyrene plate for assays",
    required_overrides=["resource_name"],  # Must be customized when using
    tags=["plate", "96-well", "assay", "standard"],
    created_by="lab_manager",
    version="1.0.0"
)

Using Templates to Create Resources¶

Templates streamline resource creation with consistent defaults:

# Create new resources from template
assay_plate_1 = client.create_resource_from_template(
    template_name="standard_96_well_plate",
    resource_name="Assay Plate #001",
    overrides={
        "attributes": {"experiment_id": "EXP001", "assay_type": "ELISA"}
    }
)

assay_plate_2 = client.create_resource_from_template(
    template_name="standard_96_well_plate",
    resource_name="Assay Plate #002",
    overrides={
        "attributes": {"experiment_id": "EXP002", "assay_type": "cell_culture"}
    }
)

# Both plates inherit standard configuration but with custom attributes

Template Management Operations¶

Listing and Discovery:

# List all available templates
all_templates = client.list_templates()

# Filter templates by category
plate_templates = client.list_templates(base_type="container", tags=["plate"])
reagent_templates = client.list_templates(base_type="consumable", tags=["reagent"])

# Get templates organized by category
templates_by_category = client.get_templates_by_category()
# Returns: {"container": ["plate_template", "rack_template"], "consumable": ["buffer_template"]}

# Filter by creator
lab_templates = client.list_templates(created_by="lab_manager")

Template Metadata:

# Get detailed template information
template_info = client.get_template_info("standard_96_well_plate")

# Returns metadata dictionary:
# {
#   "description": "Standard 96-well polystyrene plate for assays",
#   "required_overrides": ["resource_name"],
#   "tags": ["plate", "96-well", "assay", "standard"],
#   "created_by": "lab_manager",
#   "version": "1.0.0",
#   "created_at": "2024-01-15T10:30:00Z",
#   "resource": <template_resource_object>
# }

Template Updates:

# Update template metadata
updated_template = client.update_template(
    template_name="standard_96_well_plate",
    updates={
        "description": "Updated standard 96-well plate with new specifications",
        "tags": ["plate", "96-well", "assay", "standard", "v2"],
        "version": "1.1.0",
        "attributes": {"well_volume": 250}  # Updated well volume
    }
)

Template Deletion:

# Remove template (permanent)
success = client.delete_template("obsolete_template")
if success:
    print("Template successfully deleted")

Template Use Cases¶

1. Standardized Labware:

# Create templates for common labware
tip_box_template = client.create_template(
    resource=Grid(resource_name="Standard Tip Box", rows=8, columns=12, resource_class="tips"),
    template_name="standard_tip_box",
    description="200µL tip box template",
    required_overrides=["resource_name"],
    tags=["tips", "consumable", "standard"]
)

# Create multiple tip boxes from template
for i in range(5):
    client.create_resource_from_template(
        template_name="standard_tip_box",
        resource_name=f"Tip Box #{i+1:03d}",
        overrides={"attributes": {"batch_number": f"TB{i+1:03d}"}}
    )

2. Reagent Standards:

# Create reagent template
buffer_template = client.create_template(
    resource=Consumable(
        resource_name="PBS Buffer",
        resource_class="buffer",
        quantity=1000.0,
        attributes={"pH": 7.4, "concentration": "1X", "units": "mL"}
    ),
    template_name="pbs_buffer_1x",
    description="1X PBS buffer, pH 7.4",
    required_overrides=["resource_name", "quantity"],
    tags=["buffer", "pbs", "cell_culture"]
)

# Create buffer instances
buffer_stock = client.create_resource_from_template(
    template_name="pbs_buffer_1x",
    resource_name="PBS Stock #001",
    overrides={"quantity": 5000.0, "attributes": {"lot_number": "PBS2024001"}}
)

3. Container Hierarchies:

# Template for plate storage systems
storage_template = client.create_template(
    resource=Stack(resource_name="Plate Storage", resource_class="storage", capacity=20),
    template_name="plate_storage_stack",
    description="Standard plate storage stack (20 plates)",
    required_overrides=["resource_name"],
    tags=["storage", "plate", "stack"]
)

# Create storage locations
incubator_storage = client.create_resource_from_template(
    template_name="plate_storage_stack",
    resource_name="Incubator Plate Stack",
    overrides={"attributes": {"temperature": 37, "humidity": 95}}
)

Template Best Practices¶

1. Use Meaningful Names and Tags:

# ✅ Good - Descriptive and searchable
client.create_template(
    resource=plate,
    template_name="corning_96_well_flat_bottom",
    tags=["plate", "96-well", "flat-bottom", "corning", "cell-culture"]
)

# ❌ Avoid - Generic and hard to find
client.create_template(resource=plate, template_name="plate1", tags=["lab"])

2. Define Required Overrides:

# ✅ Good - Enforce customization of unique fields
client.create_template(
    resource=sample,
    template_name="dna_sample_template",
    required_overrides=["resource_name", "attributes.sample_id", "attributes.source"]
)

# ❌ Avoid - No required overrides may lead to duplicate names
client.create_template(resource=sample, template_name="sample_template")

3. Version Your Templates:

# Version templates for tracking changes
client.create_template(
    resource=updated_plate,
    template_name="assay_plate_v2",
    description="Updated assay plate with improved specifications",
    version="2.0.0",
    tags=["plate", "assay", "v2"]
)

4. Organize with Categories:

# Use consistent tag hierarchies
consumable_tags = ["consumable", "reagent", "buffer"]
labware_tags = ["labware", "plate", "96-well"]
equipment_tags = ["equipment", "analyzer", "hplc"]

Resource Locking and Concurrency Control¶

MADSci provides comprehensive resource locking to prevent conflicts when multiple processes or nodes access the same resources concurrently.

Basic Resource Locking¶

from madsci.client.resource_client import ResourceClient

client = ResourceClient("http://localhost:8003")

# Acquire lock on a single resource
success = client.acquire_lock(
    resource=sample_plate,
    lock_duration=300.0,  # 5 minutes
)

if success:
    try:
        # Perform operations on the locked resource
        client.set_child(resource=sample_plate, key=(0, 0), child=new_sample)
        client.update_resource(sample_plate)
    finally:
        # Always release the lock
        client.release_lock(resource=sample_plate)

Context Manager for Automatic Lock Management¶

The recommended approach uses context managers for automatic lock acquisition and release:

# Single resource locking
with client.lock(sample_plate) as locked_plate:
    # Resource is automatically locked
    locked_plate.set_child(key=(0, 0), child=new_sample)
    locked_plate.update_resource()
    # Lock automatically released when exiting context

# Multiple resource locking (atomic)
with client.lock(reagent_bottle, sample_rack, plate_stack) as (reagent, rack, stack):
    # All resources locked atomically or operation fails
    reagent.decrease_quantity(amount=50.0)
    new_sample = rack.get_child(key=5)
    stack.push(child=finished_plate)
    # All locks released automatically

Advanced Locking Patterns¶

Lock Duration and Auto-Refresh:

# Custom lock duration with auto-refresh
with client.lock(
    resource=long_running_plate,
    lock_duration=60.0,     # 1 minute initial lock
    auto_refresh=True,      # Automatically extend lock if needed
) as locked_plate:
    # Perform long-running operations
    # Lock automatically refreshed every 30 seconds
    for i in range(96):  # Process each well
        process_well(locked_plate, well_position=i)

Lock Status Checking:

# Check if resource is currently locked
is_locked = client.is_locked(resource=sample_plate)

if not is_locked:
    with client.lock(sample_plate) as locked_plate:
        perform_analysis(locked_plate)
else:
    print("Resource currently in use by another process")

Error Handling and Lock Recovery¶

# Manual lock management with error handling
try:
    if client.acquire_lock(resource=critical_resource, lock_duration=120.0):
        try:
            # Critical operations
            perform_critical_work(critical_resource)
        finally:
            client.release_lock(resource=critical_resource)
    else:
        raise Exception("Failed to acquire lock on critical resource")
except Exception as e:
    print(f"Operation failed: {e}")

Best Practices for Resource Locking¶

1. Always Use Context Managers:

# ✅ Good - Automatic cleanup
with client.lock(resource) as locked_resource:
    work_with_resource(locked_resource)

# ❌ Avoid - Manual management prone to errors
client.acquire_lock(resource)
work_with_resource(resource)
client.release_lock(resource)  # May not execute if exception occurs

2. Lock Multiple Resources Atomically:

# ✅ Good - All locks acquired or none
with client.lock(plate, reagent, tip_rack) as (p, r, t):
    transfer_samples(from_plate=p, reagent=r, tips=t)

# ❌ Avoid - Deadlock potential
with client.lock(plate) as p:
    with client.lock(reagent) as r:  # Could deadlock if another process locks in reverse order
        transfer_samples(p, r)

3. Use Appropriate Lock Durations:

# Short operations - brief locks
with client.lock(sample, lock_duration=30.0) as s:
    result = quick_measurement(s)

# Long operations - longer locks with auto-refresh
with client.lock(plate_stack, lock_duration=300.0, auto_refresh=True) as stack:
    process_entire_stack(stack)  # May take several minutes

Integration with Node Actions¶

Resource locking integrates seamlessly with MADSci node actions:

from madsci.node_module.node_module import RestNode
from madsci.common.types.action_types import ActionResult, ActionSucceeded

class AnalyzerNode(RestNode):

    @action
    def analyze_sample(
        self,
        sample_plate_id: str,
        sample_position: tuple[int, int]
    ) -> ActionResult:
        # Acquire lock before manipulating resources
        with self.resource_client.lock(sample_plate_id) as plate:
            # Get sample from locked plate
            sample = plate.get_child(key=sample_position)

            # Perform analysis
            result = self.instrument.analyze(sample)

            # Update sample with results
            sample.attributes["analysis_result"] = result
            plate.set_child(key=sample_position, child=sample)

            return ActionSucceeded(data=result)

Resource Hierarchy Queries¶

The Resource Manager provides functionality to query the hierarchical relationships between resources, making it easy to understand parent-child relationships and navigate resource trees.

Understanding Resource Hierarchy¶

Resources can form hierarchical structures where:

Parent resources contain child resources (e.g., a plate contains samples)
Child resources belong to parent resources and have a specific key/position
Ancestor resources are all parents up the hierarchy chain
Descendant resources are all children down the hierarchy chain

Querying Resource Hierarchy¶

from madsci.client.resource_client import ResourceClient

client = ResourceClient("http://localhost:8003")

# Create a hierarchy: Rack -> Plate -> Sample
rack = Grid(resource_name="Sample Rack", rows=2, columns=3, resource_class="rack")
rack = client.add_resource(rack)

plate = Grid(resource_name="96-well Plate", rows=8, columns=12, resource_class="plate")
plate = client.add_resource(plate)
client.set_child(resource=rack, key=(0, 0), child=plate)

sample = Asset(resource_name="Sample A1", resource_class="sample")
sample = client.add_resource(sample)
client.set_child(resource=plate, key=(0, 0), child=sample)

# Query hierarchy for the plate (middle of the hierarchy)
hierarchy = client.query_resource_hierarchy(plate.resource_id)

print(f"Resource ID: {hierarchy.resource_id}")
print(f"Ancestors (closest to furthest): {hierarchy.ancestor_ids}")
print(f"Descendants by parent: {hierarchy.descendant_ids}")

# Example output:
# Resource ID: 01HQ2K3M4N5P6Q7R8S9T0V1W2X
# Ancestors: ['01HQ2K3M4N5P6Q7R8S9T0V1W2Y']  # [rack_id]
# Descendants: {
#     '01HQ2K3M4N5P6Q7R8S9T0V1W2X': ['01HQ2K3M4N5P6Q7R8S9T0V1W2Z']  # plate -> [sample_id]
# }

Hierarchy Query Results¶

The query_resource_hierarchy method returns a ResourceHierarchy object with:

ancestor_ids: List of parent resource IDs, ordered from closest to furthest
- [parent_id, grandparent_id, great_grandparent_id, ...]
- Empty list if the resource has no parents
resource_id: The ID of the queried resource
descendant_ids: Dictionary mapping parent IDs to their direct child IDs
- Recursively includes all descendant generations (children, grandchildren, great-grandchildren, etc.)
- Only includes direct parent-child relationships (no “uncle” or “cousin” resources)
- Key: parent resource ID, Value: list of direct child resource IDs
- Empty dictionary if no descendants exist

Use Cases¶

1. Navigate Up the Hierarchy:

# Find all containers holding a specific sample
sample_hierarchy = client.query_resource_hierarchy(sample_id)
for ancestor_id in sample_hierarchy.ancestor_ids:
    ancestor = client.get_resource(ancestor_id)
    print(f"Sample is contained in: {ancestor.resource_name}")

2. Navigate Down the Hierarchy:

# Find all contents of a container and their sub-contents
container_hierarchy = client.query_resource_hierarchy(container_id)
for parent_id, child_ids in container_hierarchy.descendant_ids.items():
    parent = client.get_resource(parent_id)
    print(f"{parent.resource_name} contains:")
    for child_id in child_ids:
        child = client.get_resource(child_id)
        print(f"  - {child.resource_name}")

3. Verify Containment Relationships:

# Check if one resource is an ancestor of another
def is_ancestor(potential_ancestor_id, resource_id, client):
    hierarchy = client.query_resource_hierarchy(resource_id)
    return potential_ancestor_id in hierarchy.ancestor_ids

# Check if one resource is a descendant of another
def is_descendant(potential_descendant_id, resource_id, client):
    hierarchy = client.query_resource_hierarchy(resource_id)
    for child_ids in hierarchy.descendant_ids.values():
        if potential_descendant_id in child_ids:
            return True
    return False

4. Build Resource Trees:

# Recursively build a complete resource tree
def build_resource_tree(resource_id, client, depth=0):
    resource = client.get_resource(resource_id)
    hierarchy = client.query_resource_hierarchy(resource_id)

    indent = "  " * depth
    print(f"{indent}{resource.resource_name} ({resource.resource_id})")

    # Process direct children
    if resource_id in hierarchy.descendant_ids:
        for child_id in hierarchy.descendant_ids[resource_id]:
            build_resource_tree(child_id, client, depth + 1)

# Start from a root resource
build_resource_tree(root_container_id, client)

Performance Considerations¶

Hierarchy Query Optimization¶

Hierarchy queries are optimized to fetch only direct parent-child relationships
For deep hierarchies, consider caching results if querying frequently
The query returns all direct ancestors and recursively traverses all descendants
Use sparingly for very large resource trees with many nested levels

Performance Optimization¶

Database Performance¶

Connection Pooling:

# Increase connection pool size for high-throughput environments
export RESOURCE_MAX_CONNECTIONS=50
export RESOURCE_CONNECTION_TIMEOUT=30

Query Optimization:

# Use specific filters to reduce query scope
samples = client.query_resource(
    resource_class="sample",
    attributes={"experiment_id": "EXP001"},  # Filter early
    multiple=True
)

# Avoid retrieving large result sets at once
batch_size = 100
offset = 0
while True:
    batch = client.query_resource(
        resource_class="sample",
        limit=batch_size,
        offset=offset,
        multiple=True
    )
    if not batch:
        break
    process_batch(batch)
    offset += batch_size

Resource Management Performance¶

Bulk Operations:

# Batch similar operations together
resource_updates = []
for sample_id, new_attributes in sample_updates.items():
    resource = client.get_resource(sample_id)
    resource.attributes.update(new_attributes)
    resource_updates.append(resource)

# Process batch
for resource in resource_updates:
    client.update_resource(resource)

Container Hierarchy Optimization:

# Cache hierarchy results for repeated access
hierarchy_cache = {}

def get_cached_hierarchy(resource_id):
    if resource_id not in hierarchy_cache:
        hierarchy_cache[resource_id] = client.query_resource_hierarchy(resource_id)
    return hierarchy_cache[resource_id]

# Use for repeated hierarchy traversals
for sample_id in sample_list:
    hierarchy = get_cached_hierarchy(sample_id)
    process_ancestors(hierarchy.ancestor_ids)

Lock Management Performance¶

Minimize Lock Duration:

# ✅ Good - Short lock scope
hierarchy = client.query_resource_hierarchy(plate_id)  # Outside lock
with client.lock(plate) as locked_plate:
    # Only critical operations inside lock
    locked_plate.set_child(key=(row, col), child=sample)

# ❌ Avoid - Long lock duration
with client.lock(plate) as locked_plate:
    hierarchy = client.query_resource_hierarchy(plate_id)  # Unnecessary lock usage
    locked_plate.set_child(key=(row, col), child=sample)

Concurrent Operations:

# Use separate threads for independent resource operations
import threading

def process_resource(resource_id):
    with client.lock(resource_id) as locked_resource:
        perform_analysis(locked_resource)

# Process multiple resources concurrently
threads = []
for resource_id in resource_list:
    thread = threading.Thread(target=process_resource, args=(resource_id,))
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join()

Memory Management¶

Large Container Handling:

# For containers with many children, avoid loading all at once
def process_large_container(container_id):
    container = client.get_resource(container_id)

    # Process children by key ranges instead of loading all
    if isinstance(container, Grid):
        for row in range(container.rows):
            for col in range(container.columns):
                child = container.get_child((row, col))
                if child:
                    process_child(child)
                    # Release reference to help GC
                    child = None

Resource Cleanup:

# Clean up large query results
large_result_set = client.query_resource(resource_class="sample", multiple=True)
try:
    for resource in large_result_set:
        process_resource(resource)
finally:
    # Explicit cleanup for large datasets
    large_result_set.clear()
    del large_result_set

Monitoring Performance¶

Query Timing:

import time

start_time = time.time()
resources = client.query_resource(resource_class="sample", multiple=True)
query_time = time.time() - start_time

if query_time > 5.0:  # Log slow queries
    print(f"Slow query detected: {query_time:.2f}s for {len(resources)} resources")

Lock Contention Monitoring:

lock_wait_start = time.time()
try:
    with client.lock(resource, lock_duration=30.0) as locked_resource:
        lock_wait_time = time.time() - lock_wait_start
        if lock_wait_time > 1.0:
            print(f"Lock contention: waited {lock_wait_time:.2f}s")

        perform_operation(locked_resource)
except TimeoutError:
    print("Failed to acquire lock - high contention detected")

Performance Best Practices¶

Use Specific Queries: Always include filters to reduce result set size
Batch Operations: Group similar operations together
Cache Hierarchy Results: For repeated hierarchy traversals
Minimize Lock Scope: Keep locked sections as short as possible
Monitor Query Performance: Log slow operations for optimization
Use Connection Pooling: Configure appropriate pool sizes for your workload
Clean Up Resources: Explicitly clean up large datasets when done

Error Handling and Troubleshooting¶

Common Error Scenarios¶

1. Resource Not Found:

from madsci.common.exceptions import ResourceNotFoundError

try:
    resource = client.get_resource("invalid_id")
except ResourceNotFoundError:
    print("Resource does not exist or has been removed")

2. Lock Acquisition Failures:

# Handle lock timeouts gracefully
try:
    with client.lock(resource, lock_duration=30.0) as locked_resource:
        perform_operation(locked_resource)
except TimeoutError:
    print("Resource is currently locked by another process")
    # Implement retry logic or queue the operation

3. Container Capacity Violations:

try:
    client.set_child(resource=full_container, key="A1", child=new_sample)
except ValueError as e:
    if "capacity" in str(e):
        print(f"Container is full: {e}")
        # Find alternative container or wait for space

4. Quantity Management Errors:

try:
    client.decrease_quantity(resource=reagent, amount=1000.0)
except ValueError as e:
    if "insufficient quantity" in str(e):
        print(f"Not enough reagent available: {e}")
        # Check current quantity and reorder if needed
        current = client.get_resource(reagent.resource_id)
        print(f"Current quantity: {current.quantity}")

Database Connection Issues¶

Connection Failures:

from madsci.client.resource_client import ResourceClient

try:
    client = ResourceClient("http://localhost:8003")
    # Test connection
    client.get_definition()
except Exception as e:
    print(f"Failed to connect to Resource Manager: {e}")
    # Check if service is running: docker compose ps
    # Check logs: docker compose logs resource_manager

Network Timeouts:

import httpx

client = ResourceClient(
    base_url="http://localhost:8003",
    timeout=30.0  # Increase timeout for slow operations
)

Performance Troubleshooting¶

Slow Hierarchy Queries:

# For deep hierarchies, query specific levels instead of full tree
hierarchy = client.query_resource_hierarchy(resource_id)
if len(hierarchy.ancestor_ids) > 10:
    # Consider caching results or limiting traversal depth
    pass

Large Batch Operations:

# Process resources in smaller batches to avoid timeouts
resources = client.query_resource(resource_class="sample", multiple=True)
batch_size = 50

for i in range(0, len(resources), batch_size):
    batch = resources[i:i + batch_size]
    for resource in batch:
        # Process batch
        pass

Resource State Issues¶

Debugging Resource State:

# Check resource history for unexpected changes
history = client.query_history(resource_id=problem_resource.resource_id)
for entry in history[-5:]:  # Last 5 changes
    print(f"{entry.timestamp}: {entry.change_type} - {entry.details}")

Recovering from Soft Deletes:

# Find and restore accidentally deleted resources
deleted_resources = client.query_resource(removed=True, multiple=True)
for resource in deleted_resources:
    if resource.resource_name == "important_sample":
        client.restore_deleted_resource(resource.resource_id)
        print(f"Restored: {resource.resource_name}")

Docker and Service Issues¶

Service Health Check:

# Check if Resource Manager is running
curl http://localhost:8003/health

# Check service logs
docker compose logs resource_manager

# Restart if needed
docker compose restart resource_manager

Database Connection:

# Check PostgreSQL connection
docker compose exec postgres psql -U madsci -d madsci -c "\dt"

# Check Resource Manager tables
docker compose exec postgres psql -U madsci -d madsci -c "\d resources"

Common Solutions¶

1. Service Won’t Start:

Verify PostgreSQL is running: docker compose ps postgres
Check port availability: lsof -i :8003
Review environment variables in docker-compose.yml

2. Resource Operations Fail:

Verify resource exists: client.get_resource(resource_id)
Check resource type compatibility
Ensure proper permissions/ownership

3. Locking Issues:

Check for orphaned locks: restart service to clear
Reduce lock duration for short operations
Implement lock retry logic with backoff

Quick Reference¶

Essential Operations¶

from madsci.client.resource_client import ResourceClient
from madsci.common.types.resource_types import Asset, Consumable, Grid

client = ResourceClient("http://localhost:8003")

# Basic CRUD
resource = client.add_resource(Asset(resource_name="Sample", resource_class="sample"))
resource = client.get_resource(resource.resource_id)
resource.attributes["status"] = "processed"
client.update_resource(resource)
client.remove_resource(resource.resource_id)

# Queries
samples = client.query_resource(resource_class="sample", multiple=True)
recent = client.query_resource(created_after="2024-01-01", multiple=True)

# Containers
plate = client.add_resource(Grid(resource_name="Plate", rows=8, columns=12))
client.set_child(resource=plate, key=(0, 0), child=sample)
child = plate.get_child((0, 0))

# Consumables
reagent = client.add_resource(Consumable(resource_name="Buffer", quantity=1000.0))
client.decrease_quantity(resource=reagent, amount=50.0)
client.increase_quantity(resource=reagent, amount=100.0)

# Locking
with client.lock(resource) as locked:
    locked.update_resource()

# Templates
template = client.create_template(resource=plate, template_name="standard_plate")
new_plate = client.create_resource_from_template("standard_plate", resource_name="Plate #2")

Common Resource Types¶

Type	Use Case	Key Features
`Asset`	Samples, labware	Non-consumable, trackable
`Consumable`	Reagents, tips	Quantity tracking, depletion
`Grid`	Plates, arrays	2D positioning (row, col)
`Stack`	Plate magazines	LIFO access (push/pop)
`Queue`	Conveyor systems	FIFO access (enqueue/dequeue)
`Row`	Tube racks	1D positioning
`Pool`	Mixed containers	Multiple consumables in one space

Key Environment Variables¶

RESOURCE_PORT=8003                    # Service port
RESOURCE_POSTGRES_HOST=localhost      # Database host
RESOURCE_LOCAL_MODE=false             # Standalone mode
RESOURCE_LOG_LEVEL=INFO              # Logging verbosity

Common Commands¶

# Start service
python -m madsci.resource_manager.resource_server

# With Docker
docker compose up resource_manager

# Health check
curl http://localhost:8003/health

# API documentation
open http://localhost:8003/docs

Examples: See example_lab/ for complete resource management workflows integrated with laboratory operations.

Database Migration Tools¶

MADSci includes automated database migration tools that handle schema changes and version tracking for the resource management system.

Features¶

Version Compatibility Checking: Automatically detects mismatches between MADSci package version and database schema version
Automated Backup: Creates PostgreSQL dumps before applying migrations to enable rollback on failure
Schema Migration: Uses Alembic to generate and apply database schema changes
Type Conversion Safety: Automatically handles PostgreSQL type conversions (e.g., VARCHAR to FLOAT) with safe SQL transformations
Location Independence: Can be run from any directory while finding its own configuration files

Usage¶

Standard Usage¶

# Run migration to current MADSci version
python -m madsci.resource_manager.migration_tool --db-url 'postgresql://user:pass@localhost:5432/resources'

# Migrate to specific version
python -m madsci.resource_manager.migration_tool --target-version 1.0.0

# Create backup only
python -m madsci.resource_manager.migration_tool --backup-only

# Restore from backup
python -m madsci.resource_manager.migration_tool --restore-from /path/to/backup.sql

# Generate new migration file
python -m madsci.resource_manager.migration_tool --generate-migration "Add new feature"

Docker Usage¶

When running in Docker containers, use docker-compose to execute migration commands:

# Run migration to current MADSci version in Docker
docker-compose run --rm -v $(pwd)/src:/home/madsci/MADSci/src resource-manager python -m madsci.resource_manager.migration_tool --db-url 'postgresql://user:pass@postgres:5432/resources'

# Migrate to specific version in Docker
docker-compose run --rm -v $(pwd)/src:/home/madsci/MADSci/src resource-manager python -m madsci.resource_manager.migration_tool --db-url 'postgresql://user:pass@postgres:5432/resources' --target-version 1.0.0

# Create backup only in Docker
docker-compose run --rm -v $(pwd)/src:/home/madsci/MADSci/src resource-manager python -m madsci.resource_manager.migration_tool --db-url 'postgresql://user:pass@postgres:5432/resources' --backup-only

# Generate new migration file in Docker
docker-compose run --rm -v $(pwd)/src:/home/madsci/MADSci/src resource-manager python -m madsci.resource_manager.migration_tool --db-url 'postgresql://user:pass@postgres:5432/resources' --generate-migration "Add new feature"

Server Integration¶

The Resource Manager server automatically checks for version compatibility on startup. If a mismatch is detected, the server will refuse to start and display migration instructions:

DATABASE MIGRATION REQUIRED! SERVER STARTUP ABORTED!
The database schema version does not match the MADSci package version.
To resolve this issue, run the migration tool and restart the server.

Backup Location¶

Backups are stored in .madsci/postgresql/backups/ with timestamped filenames:

Format: resources_backup_YYYYMMDD_HHMMSS.sql
Can be restored using the --restore-from option

Requirements¶

PostgreSQL server running and accessible
PostgreSQL tools (pg_dump, pg_restore, psql) installed
Appropriate database permissions for the specified user
Alembic for schema migration management