MADSci Event Manager - MADSci Documentation

Handles distributed logging and events throughout a MADSci-powered Lab.

MADSci Event Manager Architecture Diagram

Features¶

Centralized logging from distributed lab components
Event querying with structured filtering
Arbitrary event data support with standard schema
Python logging-style log levels
Alert notifications (email, etc.)

Installation¶

See the main README for installation options. This package is available as:

PyPI: pip install madsci.event_manager
Docker: Included in ghcr.io/ad-sdl/madsci
Example configuration: See example_lab/managers/example_event.manager.yaml

Dependencies: MongoDB database (see the example_lab)

Usage¶

Quick Start¶

Use the example_lab as a starting point:

# Start with working example
docker compose up  # From repo root
# Event Manager available at http://localhost:8001/docs

# Or run standalone
python -m madsci.event_manager.event_server

Manager Setup¶

For custom deployments, create an Event Manager definition:

madsci manager add -t event_manager

See example_event.manager.yaml for configuration options.

Client¶

You can use MADSci’s EventClient (madsci.client.event_client.EventClient) in your python code to log new events to the event manager, or fetch/query existing events.

from madsci.client.event_client import EventClient
from madsci.common.types.event_types import Event, EventLogLevel, EventType

event_client = EventClient(
    name="my_component",
    event_server="http://localhost:8001", # Update with the host/port you configured for your EventManager server
)

# Modern structlog-style API (recommended)
event_client.info("This logs a simple string at the INFO level")
event_client.debug("Debug message with context", key="value", count=42)
event_client.warning("Warning with structured data", resource_id="res-123")
event_client.error("Error occurred", exc_info=True)  # Includes traceback

# With event types for structured querying
event_client.info(
    "Workflow completed",
    event_type=EventType.WORKFLOW_COMPLETE,
    workflow_id="wf-123",
    duration_ms=1500,
)

# Context binding for persistent metadata
event_client = event_client.bind(experiment_id="exp-456")
event_client.info("All subsequent logs include experiment_id")

# Legacy API (still supported)
event = Event(
    event_type=EventType.NODE_CREATE,
    log_level=EventLogLevel.DEBUG,
    event_data="This logs a NODE_CREATE event at the DEBUG level."
)
event_client.log(event)

# Get the 50 most recent events
event_client.get_events(number=50)
# Get all events from a specific node
event_client.query_events({"source": {"node_id": "01JJ4S0WNGEF5FQAZG5KDGJRBV"}})

event_client.alert(event) # Will force firing any configured alert notifiers on this event

Logging with Context¶

Use the EventClient context system for hierarchical logging across components:

from madsci.common.context import event_client_context, get_event_client

# Establish context at entry points
with event_client_context(name="my_experiment", experiment_id="exp-123") as logger:
    logger.info("Starting experiment")

    # All nested operations share context
    with event_client_context(name="workflow", workflow_id="wf-456") as wf_logger:
        wf_logger.info("Running workflow")  # Includes both experiment_id and workflow_id

# In library code, inherit context automatically
def utility_function():
    logger = get_event_client()  # Uses context if available
    logger.info("Utility running")

See the Logging Guide for comprehensive documentation on structured logging and context management.

OpenTelemetry Integration¶

The EventClient integrates with OpenTelemetry for distributed tracing:

from madsci.client.event_client import EventClient, EventClientConfig

config = EventClientConfig(
    otel_enabled=True,
    otel_service_name="my_service",
    otel_exporter="otlp",
    otel_endpoint="http://localhost:4317",
)
client = EventClient(name="traced_component", config=config)

# Events automatically include trace_id and span_id
client.info("Traced event")  # Correlated with distributed traces

See the Observability Guide for the full observability stack setup.

Alerts¶

The Event Manager provides some native alerting functionality. A default alert level can be set in the event manager definition’s alert_level, which will determine the minimum log level at which to send an alert. Calls directly to the EventClient.alert method will send alerts regardless of the alert_level.

You can configure Email Alerts by setting up an EmailAlertsConfig (madsci.common.types.event_types.EmailAlertsConfig) in the email_alerts field of your EventManagerSettings.

Database Migration Tools¶

MADSci Event Manager includes automated MongoDB migration tools that handle schema changes and version tracking for the event management system.

Features¶

Version Compatibility Checking: Automatically detects mismatches between MADSci package version and MongoDB schema version
Automated Backup: Creates MongoDB dumps using mongodump before applying migrations to enable rollback on failure
Schema Management: Creates collections and indexes based on schema definitions
Index Management: Ensures required indexes exist for optimal query performance
Location Independence: Auto-detects schema files or accepts explicit paths
Safe Migration: All changes are applied transactionally with automatic rollback on failure

Usage¶

Standard Usage¶

# Run migration for events database (auto-detects schema file)
python -m madsci.common.mongodb_migration_tool --database madsci_events

# Migrate with explicit database URL
python -m madsci.common.mongodb_migration_tool --db-url mongodb://localhost:27017 --database madsci_events

# Use custom schema file
python -m madsci.common.mongodb_migration_tool --database madsci_events --schema-file /path/to/schema.json

# Create backup only
python -m madsci.common.mongodb_migration_tool --database madsci_events --backup-only

# Restore from backup
python -m madsci.common.mongodb_migration_tool --database madsci_events --restore-from /path/to/backup

# Check version compatibility without migrating
python -m madsci.common.mongodb_migration_tool --database madsci_events --check-version

Docker Usage¶

When running in Docker containers, use docker-compose to execute migration commands:

# Run migration for events database in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json'

# Create backup only in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json' --backup-only

# Check version compatibility in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json' --check-version

Server Integration¶

The Event Manager server automatically checks for version compatibility on startup. If a mismatch is detected, the server will refuse to start and display migration instructions:

DATABASE INITIALIZATION REQUIRED! SERVER STARTUP ABORTED!
The database exists but needs version tracking setup.
To resolve this issue, run the migration tool and restart the server.

Schema File Location¶

The migration tool automatically searches for schema files in:

madsci/event_manager/schema.json

Backup Location¶

Backups are stored in .madsci/mongodb/backups/ with timestamped filenames:

Format: madsci_events_backup_YYYYMMDD_HHMMSS
Can be restored using the --restore-from option

Requirements¶

MongoDB server running and accessible
MongoDB tools (mongodump, mongorestore) installed
Appropriate database permissions for the specified user

API Reference¶

The Event Manager provides a REST API for logging and querying events. The API is available at http://localhost:8001 by default.

Event Operations¶

POST /event¶

Log a new event to the system.

Request Body: Event object

{
  "event_type": "LOG_INFO",
  "log_level": 20,
  "event_data": "Event message or data",
  "alert": false
}

Response: The logged Event object with assigned event_id and event_timestamp.

GET /event/{event_id}¶

Retrieve a specific event by its ID.

Parameters:

event_id (path): The unique event identifier

Response: Event object or 404 if not found.

GET /events¶

Get the latest events from the system.

Query Parameters:

number (int, default: 100): Maximum number of events to return
level (int, default: 0): Minimum log level to include

Response: Dictionary mapping event IDs to Event objects.

POST /events/query¶

Query events using MongoDB selector syntax.

Request Body: MongoDB query selector object

{
  "source.node_id": "01JJ4S0WNGEF5FQAZG5KDGJRBV",
  "log_level": {"$gte": 20}
}

Response: Dictionary mapping event IDs to matching Event objects.

Utilization Analysis¶

GET /utilization/sessions¶

Generate session-based utilization reports.

Query Parameters:

start_time (string, optional): ISO format start time
end_time (string, optional): ISO format end time
csv_format (bool, default: false): Return data in CSV format
save_to_file (bool, default: false): Save CSV to server filesystem
output_path (string, optional): Server path to save CSV files

Response: JSON utilization report or CSV data.

Standard Manager Endpoints¶

The Event Manager also provides standard manager endpoints:

GET /definition: Get manager definition and metadata
GET /health: Get health status including database connectivity and event counts

Interactive API Documentation¶

Visit http://localhost:8001/docs when the Event Manager is running for interactive Swagger UI documentation.

MongoDB Setup¶

The Event Manager requires MongoDB for event storage. For local development:

Using Docker (recommended):

docker run -d -p 27017:27017 --name mongodb mongo:latest

Using the example lab: The example_lab includes a pre-configured MongoDB instance.

Configuration: Set the database URL in your environment or manager configuration:

# In manager YAML config
db_url: "mongodb://localhost:27017"
collection_name: "madsci_events"

Or as environment variables:

export EVENT_DB_URL="mongodb://localhost:27017"
export EVENT_COLLECTION_NAME="madsci_events"