Conducting Automated/Autonomous Experiments with MADSci

This aims to teach you how to define and run automated/autonomous experiment applications using a MADSci Workcell.

Goals¶

After completing this notebook, you should understand how to define and use a MADSci ExperimentApplication to manage data, resources, logging and workflows for your scientific applications. You’ll also learn how the built-in client management through MadsciClientMixin simplifies interaction with all MADSci manager services.

What is an Experiment Application?¶

The MADSci software architecture uses a number of modular components, called managers, to organize the workflows, resources and datapoints that we will use to actually do science. In order to interface with the manager, we provide clients with predefined functions that make the manager functionality easy to understand and use.

The ExperimentApplication class provided by MADSci:

Inherits from RestNode (and thus MadsciClientMixin) providing automatic client management
Leverages the Lab manager to retrieve configuration information
Creates clients for all managers used in a MADSci experiment
Provides utility functions that make it easier to run experiments and handle data locally
Can operate as Nodes, allowing you to incorporate custom functionality into workflows, including data processing and AI control

Built-in Client Access¶

Through the MadsciClientMixin, ExperimentApplication provides automatic access to:

experiment_client: Experiment management and lifecycle
workcell_client: Workflow coordination and scheduling
resource_client: Resource and inventory tracking
data_client: Data capture, storage, and querying
location_client: Location management and positioning
event_client: Distributed event logging (also available as logger)
lab_client: Lab configuration and context

# Install dependencies
%pip install madsci.experiment_application madsci.client

This demo will assume that you have the Example Lab found in the MADSci repo installed and running on your local machine. If this is not the case, follow the instructions found in the top level MADSci readme, including running just up

Setup¶

Below is an example class inheriting from the Experiment Application class.

from madsci.common.types.experiment_types import ExperimentDesign
from madsci.experiment_application.experiment_application import ExperimentApplication


class ExampleExperimentApplication(ExperimentApplication):
    """An example experiment application."""

    experiment_design = ExperimentDesign(
        experiment_name="Example Experiment",
        experiment_description="An Example Experiment",
    )

    def startup_handler(self) -> None:
        """Demonstrate client access during startup."""
        # The event_client (logger) is automatically available
        self.event_client.info("Example experiment application starting up")

        # All other clients are also automatically available:
        # self.experiment_client - for experiment management
        # self.workcell_client - for workflow coordination
        # self.resource_client - for resource tracking
        # self.data_client - for data operations
        # self.location_client - for location management
        # self.lab_client - for lab configuration


experiment_application = ExampleExperimentApplication(
    lab_server_url="http://localhost:8000"
)
print(experiment_application.experiment_design)

An Experiment Application contains an ExperimentDesign object. This can be as simple as the name and description of the experiment, but can also contain checks to ensure that the resources of the lab are in the correct state to begin a run of the experiment, though this functionality is still being tested and refined.

What can we get from an Experiment application?¶

Clients! The experiment application automatically inherits client management from MadsciClientMixin. When you create an ExperimentApplication instance:

It gets the lab configuration from the Lab server
It automatically initializes clients for all MADSci managers
All clients are configured with the proper connection details
Clients are available as properties (e.g., self.workcell_client)

This eliminates the need to manually create and configure each client - they’re all ready to use!

Workcell Client¶

The Workcell Client interacts with the workcell manager, and can be used to send and monitor workflows, add nodes to the workcell and request state information about the workcell and its nodes. Below are some examples of this functionality.

from pathlib import Path

from madsci.common.types.parameter_types import ParameterInputFile
from madsci.common.types.step_types import StepDefinition
from madsci.common.types.workflow_types import WorkflowDefinition

##An Example Workflow, defined in code. This workflow can also be read from a yaml file
workflow_definition = WorkflowDefinition(
    name="Example Workflow",
    steps=[
        StepDefinition(
            # Human readable name of the step, not necessarily unique
            name="Run Liquidhandler Protocol",
            # Human readable description of the step
            description="Run the Liquidhandler",
            # Node on which this step will be executed, not necessary for workcell actions
            node="liquidhandler_1",
            # Action to be executed
            action="run_protocol",
            # Files required for this step
            files={
                "protocol": ParameterInputFile(
                    key="protocol", description="the liquid handler protocol file"
                )
            },
        ),
        StepDefinition(
            name="Transfer from liquid handler to plate reader",
            description="Transfer an asset from the liquid handler to the plate reader",
            node="robotarm_1",
            action="transfer",
            locations={
                "source": "liquidhandler_1.deck_1",
                "target": "platereader_1.plate_carriage",
            },
        ),
        StepDefinition(
            name="run platereader measurement",
            key="measurement",  # The key is a unique identifier for the step, unlike the name which is human readable and not unique
            description="measure a well on the plate reader",
            node="platereader_1",
            action="read_well",
        ),
        StepDefinition(
            name="Transfer from  plate reader to liquid handler",
            description="Transfer an asset from the liquid handler to the plate reader",
            node="robotarm_1",
            action="transfer",
            locations={
                "source": "platereader_1.plate_carriage",
                "target": "liquidhandler_1.deck_1",
            },
        ),
    ],
)

try:
    workflow = experiment_application.workcell_client.start_workflow(
        workflow_definition,
        file_inputs={"protocol": Path("../protocols/protocol.py")},
        prompt_on_error=False,
    )
except Exception:
    print("Workflow failed because there was no plate in source location!")

Resource Client¶

The workflow above failed because there was no plate in the location the robot arm was trying to pick up from. In order to fix this, we make use of the Resource Client. This client is automatically available through the MadsciClientMixin and allows us to:

Create plate assets imported from the provided MADSci resource types
Place resources into location slots where devices can access them
Track resource inventory and usage throughout experiments

Location Client¶

The Location Client also comes pre-configured and ready to use. It handles:

Saving robot positioning information for the workcell
Retrieving location definitions and their associated resources
Managing the spatial organization of your lab setup

Both clients work together seamlessly since they’re part of the integrated MADSci ecosystem.

from madsci.common.types.resource_types import Asset

# Define a plate and add it to the database using the resource manager
asset = experiment_application.resource_client.add_resource(
    Asset(resource_name="well_plate")
)

# Get the liquid_handler_deck_1 location from the location manager
liquid_handler_deck_1 = experiment_application.location_client.get_location_by_name(
    "liquidhandler_1.deck_1"
)

# Clear the liquid handler deck slot, just in case:
try:
    experiment_application.resource_client.pop(liquid_handler_deck_1.resource_id)
except Exception:
    print(f"Liquid handler deck was already empty")  # noqa


# Insert the plate into the liquid handler deck slot
experiment_application.resource_client.push(
    liquid_handler_deck_1.resource_id, asset.resource_id
)

# Get the plate reader location from the location manager

plate_reader_plate_carraige = (
    experiment_application.location_client.get_location_by_name(
        "platereader_1.plate_carriage"
    )
)

# Clear the plate reader slot just in case:
try:
    experiment_application.resource_client.pop(plate_reader_plate_carraige.resource_id)
except Exception:
    print(f"Plate reader carriage was already empty")  # noqa


# Try running the workflow again, this time it should succeed
workflow = experiment_application.workcell_client.start_workflow(
    workflow_definition,
    file_inputs={"protocol": Path("../protocols/protocol.py")},
    prompt_on_error=False,
)

Data Client¶

The Data Client is automatically configured and used to send and retrieve data using different databases to store datapoints. Datapoints can be either files or JSON data. For file datapoints, they must be saved locally and reopened to be used, while JSON datapoints can be worked with directly.

The workcell manager has an internal data client that it uses to save the results of workflows, which can be retrieved using the experiment application’s data_client property - no manual configuration needed!

# Get the datapoint object returned by the measurement step
datapoint = workflow.get_datapoint(step_key="measurement")

# Save the datapoint value to a local file and print the result
experiment_application.data_client.save_datapoint_value(
    datapoint.datapoint_id, "test.txt"
)
with Path.open("test.txt") as f:
    print("The result of the platereader was " + f.read())

Complete App¶

Now we can put it all together, and combine the tools for a complete experiment run. While doing this, we can use the Experiment Client to track the experiment progress.

Experiment Client¶

The Experiment Client communicates with the Experiment manager, and records the start and end of experiments, while making sure all workflows run by the experiment are properly attributed in the ownership info.

The built-in experiment_app.manage_experiment() function uses this client to create a context that will:

Start an experiment on context entry
End the experiment if an error is thrown or the script completes
Properly handle experiment lifecycle and state transitions

All of this happens automatically thanks to the integrated client management system!

# Complete App Main Function:
desired_limit = 12
with experiment_application.manage_experiment():
    measurement = 9
    while measurement > desired_limit:
        # Log experiment progress using the built-in event_client
        experiment_application.event_client.info(
            f"Current measurement: {measurement}, target: {desired_limit}"
        )

        workflow = experiment_application.workcell_client.start_workflow(
            workflow_definition,
            file_inputs={"protocol": Path("../protocols/protocol.py")},
            prompt_on_error=False,
        )
        datapoint = workflow.get_datapoint(step_key="measurement")

        # Use the data_client to save results locally
        experiment_application.data_client.save_datapoint_value(
            datapoint.datapoint_id, "test.txt"
        )
        with Path.open("test.txt") as f:
            measurement = float(f.read())
            experiment_application.event_client.info(
                f"The result of the platereader was {measurement}"
            )

    experiment_application.event_client.info(
        f"Measurement {measurement} is below desired limit {desired_limit}, ending experiment"
    )

Summary: Benefits of Built-in Client Management¶

The MadsciClientMixin integration in ExperimentApplication provides several key benefits:

Automatic Configuration: All clients are pre-configured with proper connection details
No Manual Setup: No need to manually create and configure individual clients
Consistent Interface: All manager services are accessed through a unified client pattern
Error Handling: Built-in connection management and error handling
Context Aware: Clients automatically use the current MADSci lab context
Lazy Initialization: Clients are only created when first accessed, improving startup performance

This design allows you to focus on your experiment logic rather than infrastructure management!