Conducting Automated/Autonomous Experiments with MADSci

This notebook teaches you how to run interactive experiments using the MADSci ExperimentNotebook class, which is designed for cell-by-cell execution in Jupyter notebooks.

Goals¶

After completing this notebook, you should understand how to:

Use ExperimentNotebook for interactive, cell-by-cell experiment execution
Interact with MADSci manager services (workcell, resource, location, data) through the built-in clients
Track experiment lifecycle with start/end semantics

For a simpler, script-based approach see examples/example_experiment.py, which demonstrates ExperimentScript for run-once experiments.

What is an Experiment Application?¶

The MADSci software architecture uses a number of modular components, called managers, to organize the workflows, resources and datapoints that we will use to actually do science. In order to interface with the manager, we provide clients with predefined functions that make the manager functionality easy to understand and use.

MADSci provides several experiment modalities depending on your use case:

ExperimentNotebook: For interactive Jupyter notebook experiments with cell-by-cell execution (used in this tutorial)
ExperimentScript: For simple run-once experiments (see examples/example_experiment.py)
ExperimentTUI: For interactive terminal UI experiments
ExperimentNode: For experiments that operate as REST server nodes

All modalities inherit from ExperimentBase, which uses MadsciClientMixin for automatic client management. This means:

Lightweight composition-based design (no REST server overhead)
Automatic client initialization from lab context
All manager clients available as properties

Built-in Client Access¶

Through the MadsciClientMixin, all experiment classes provide automatic access to:

experiment_client: Experiment management and lifecycle
workcell_client: Workflow coordination and scheduling
resource_client: Resource and inventory tracking
data_client: Data capture, storage, and querying
location_client: Location management and positioning
event_client: Distributed event logging (also available as logger)
lab_client: Lab configuration and context

# Install dependencies (skip if already installed or pip not available)
import subprocess
import sys

try:
    import madsci.experiment_application  # noqa: F401
    import madsci.client  # noqa: F401
except ImportError:
    subprocess.run(
        [sys.executable, "-m", "pip", "install", "-q", "madsci.experiment_application", "madsci.client"],
        check=False,
    )

This demo will assume that you have the Example Lab found in the MADSci repo installed and running on your local machine. If this is not the case, follow the instructions found in the top level MADSci readme, including running just up

Setup¶

Below is an example class inheriting from the ExperimentNotebook class. This modality is designed for interactive, cell-by-cell execution in Jupyter notebooks with rich display output.

Unlike ExperimentScript (which uses a single run() call), ExperimentNotebook uses a start()/end() pattern that maps naturally to notebook cells.

from madsci.common.types.experiment_types import ExperimentDesign
from madsci.experiment_application.experiment_notebook import ExperimentNotebook


class ExampleExperimentNotebook(ExperimentNotebook):
    """An example notebook experiment."""

    experiment_design = ExperimentDesign(
        experiment_name="Example Notebook Experiment",
        experiment_description="An interactive notebook experiment demonstrating MADSci clients.",
    )


experiment = ExampleExperimentNotebook(
    lab_server_url="http://localhost:8000",
)
print(experiment.experiment_design)

An experiment class contains an ExperimentDesign object. This can be as simple as the name and description of the experiment, but can also contain checks to ensure that the resources of the lab are in the correct state to begin a run of the experiment, though this functionality is still being tested and refined.

Starting the Experiment¶

With ExperimentNotebook, you explicitly call start() to begin the experiment. This registers the experiment with the Experiment Manager and displays a status panel. You can then run experiment steps in subsequent cells, and call end() when finished.

# Start the experiment - this registers it with the Experiment Manager
experiment.start(run_name="Notebook Demo Run")

What can we get from an Experiment application?¶

Clients! The experiment class automatically inherits client management from MadsciClientMixin. When you create an ExperimentNotebook instance:

It gets the lab configuration from the Lab server
It automatically initializes clients for all MADSci managers
All clients are configured with the proper connection details
Clients are available as properties (e.g., self.workcell_client)

This eliminates the need to manually create and configure each client - they’re all ready to use!

ExperimentNotebook also provides notebook-friendly features like:

display(data, title): Rich-formatted output in notebook cells
start()/end(): Cell-by-cell lifecycle management
Context manager: with experiment: for single-cell experiments

Workcell Client¶

The Workcell Client interacts with the workcell manager, and can be used to send and monitor workflows, add nodes to the workcell and request state information about the workcell and its nodes. Below are some examples of this functionality.

from pathlib import Path

from madsci.common.types.parameter_types import ParameterInputFile
from madsci.common.types.step_types import StepDefinition
from madsci.common.types.workflow_types import WorkflowDefinition

##An Example Workflow, defined in code. This workflow can also be read from a yaml file
workflow_definition = WorkflowDefinition(
    name="Example Workflow",
    steps=[
        StepDefinition(
            # Human readable name of the step, not necessarily unique
            name="Run Liquidhandler Protocol",
            # Human readable description of the step
            description="Run the Liquidhandler",
            # Node on which this step will be executed, not necessary for workcell actions
            node="liquidhandler_1",
            # Action to be executed
            action="run_protocol",
            # Files required for this step
            files={
                "protocol": ParameterInputFile(
                    key="protocol", description="the liquid handler protocol file"
                )
            },
        ),
        StepDefinition(
            name="Transfer from liquid handler to plate reader",
            description="Transfer an asset from the liquid handler to the plate reader",
            node="robotarm_1",
            action="transfer",
            locations={
                "source": "liquidhandler_1.deck_1",
                "target": "platereader_1.plate_carriage",
            },
        ),
        StepDefinition(
            name="run platereader measurement",
            key="measurement",  # The key is a unique identifier for the step, unlike the name which is human readable and not unique
            description="measure a well on the plate reader",
            node="platereader_1",
            action="read_well",
        ),
        StepDefinition(
            name="Transfer from  plate reader to liquid handler",
            description="Transfer an asset from the liquid handler to the plate reader",
            node="robotarm_1",
            action="transfer",
            locations={
                "source": "platereader_1.plate_carriage",
                "target": "liquidhandler_1.deck_1",
            },
        ),
    ],
)

# Protocol file path (relative to this notebook's location)
protocol_path = Path("../example_lab/protocols/protocol.py")

try:
    workflow = experiment.workcell_client.start_workflow(
        workflow_definition,
        file_inputs={"protocol": protocol_path},
        prompt_on_error=False,
    )
except Exception:
    print("Workflow failed because there was no plate in source location!")

Resource Client¶

The workflow above failed because there was no plate in the location the robot arm was trying to pick up from. In order to fix this, we make use of the Resource Client. This client is automatically available through the MadsciClientMixin and allows us to:

Create plate assets imported from the provided MADSci resource types
Place resources into location slots where devices can access them
Track resource inventory and usage throughout experiments

Location Client¶

The Location Client also comes pre-configured and ready to use. It handles:

Saving robot positioning information for the workcell
Retrieving location definitions and their associated resources
Managing the spatial organization of your lab setup

Both clients work together seamlessly since they’re part of the integrated MADSci ecosystem.

from madsci.common.types.resource_types import Asset

# Define a plate and add it to the database using the resource manager
asset = experiment.resource_client.add_resource(
    Asset(resource_name="well_plate")
)

# Get the liquid_handler_deck_1 location from the location manager
liquid_handler_deck_1 = experiment.location_client.get_location_by_name(
    "liquidhandler_1.deck_1"
)

# Clear the liquid handler deck slot, just in case:
try:
    experiment.resource_client.pop(liquid_handler_deck_1.resource_id)
except Exception:
    print(f"Liquid handler deck was already empty")  # noqa


# Insert the plate into the liquid handler deck slot
experiment.resource_client.push(
    liquid_handler_deck_1.resource_id, asset.resource_id
)

# Get the plate reader location from the location manager

plate_reader_plate_carraige = (
    experiment.location_client.get_location_by_name(
        "platereader_1.plate_carriage"
    )
)

# Clear the plate reader slot just in case:
try:
    experiment.resource_client.pop(plate_reader_plate_carraige.resource_id)
except Exception:
    print(f"Plate reader carriage was already empty")  # noqa


# Try running the workflow again, this time it should succeed
workflow = experiment.workcell_client.start_workflow(
    workflow_definition,
    file_inputs={"protocol": protocol_path},
    prompt_on_error=False,
)

Data Client¶

The Data Client is automatically configured and used to send and retrieve data using different databases to store datapoints. Datapoints can be either files or JSON data. For file datapoints, they must be saved locally and reopened to be used, while JSON datapoints can be worked with directly.

The workcell manager has an internal data client that it uses to save the results of workflows, which can be retrieved using the experiment’s data_client property - no manual configuration needed!

We can also use experiment.display() to render results with Rich formatting in the notebook.

# Get the datapoint object returned by the measurement step
datapoint = workflow.get_datapoint(step_key="measurement")

# Save the datapoint value to a local file and print the result
experiment.data_client.save_datapoint_value(
    datapoint.datapoint_id, "test.txt"
)
with Path("test.txt").open() as f:
    result = f.read()
    experiment.display(
        {"measurement": result},
        title="Platereader Result",
    )

Complete Experiment Loop¶

Now we can put it all together, and combine the tools for a complete experiment loop. Since we already called experiment.start() earlier, the experiment is already being tracked by the Experiment Manager. We can use the built-in event_client (also available as logger) to log progress.

Note: For a standalone script version of this same pattern, see examples/example_experiment.py which wraps this logic in an ExperimentScript.run_experiment() method with automatic lifecycle management.

# Complete experiment loop:
desired_limit = 12
measurement = 9
while measurement > desired_limit:
    experiment.logger.info(
        f"Current measurement: {measurement}, target: {desired_limit}"
    )

    workflow = experiment.workcell_client.start_workflow(
        workflow_definition,
        file_inputs={"protocol": protocol_path},
        prompt_on_error=False,
    )
    datapoint = workflow.get_datapoint(step_key="measurement")

    experiment.data_client.save_datapoint_value(
        datapoint.datapoint_id, "test.txt"
    )
    with Path("test.txt").open() as f:
        measurement = float(f.read())
        experiment.logger.info(
            f"The result of the platereader was {measurement}"
        )

experiment.display(
    {"final_measurement": measurement, "desired_limit": desired_limit},
    title="Experiment Complete",
)

# End the experiment - this marks it as complete in the Experiment Manager
# and displays a summary panel
experiment.end()

Summary¶

ExperimentNotebook vs ExperimentScript¶

Feature	ExperimentNotebook	ExperimentScript
Use Case	Interactive Jupyter notebooks	Run-once scripts
Execution	Cell-by-cell with `start()`/`end()`	Single `run()` call
Display	Rich formatting with `display()`	Standard print output
Lifecycle	Manual `start()`/`end()` or context manager	Automatic via `manage_experiment()`
Override	Use clients directly between `start()`/`end()`	Override `run_experiment()`

Benefits of Built-in Client Management¶

The MadsciClientMixin integration (shared by all experiment modalities) provides:

Automatic Configuration: All clients are pre-configured with proper connection details
No Manual Setup: No need to manually create and configure individual clients
Consistent Interface: All manager services are accessed through a unified client pattern
Error Handling: Built-in connection management and error handling
Context Aware: Clients automatically use the current MADSci lab context
Lazy Initialization: Clients are only created when first accessed, improving startup performance

Next Steps¶

See examples/example_experiment.py for a standalone script-based experiment
See the MADSci CLI reference (docs/guides/cli_reference.md) for managing experiments from the command line
Explore the template catalog (docs/guides/template_catalog.md) for scaffolding new experiment projects