How does real-time campaign monitoring work?

MatCraft uses WebSocket connections to push real-time updates from running campaigns to the dashboard and connected clients. This eliminates the need for polling and provides an instant, reactive experience.

How It Works

1. When you open a campaign page in the dashboard, the frontend establishes a WebSocket connection to the backend at /ws/campaigns/{campaign_id}.
2. The backend subscribes to a Redis Pub/Sub channel for that campaign.
3. As the Celery worker progresses through optimization iterations, it publishes events to the Redis channel.
4. The backend forwards these events to all connected WebSocket clients.

Event Types

type CampaignEvent =
  | { type: "iteration_started"; iteration: number; timestamp: string }
  | { type: "surrogate_trained"; metrics: { train_loss: number; val_loss: number } }
  | { type: "candidates_proposed"; candidates: Candidate[]; iteration: number }
  | { type: "candidates_evaluated"; results: EvaluationResult[]; iteration: number }
  | { type: "iteration_completed"; iteration: number; best_value: number; pareto_size: number }
  | { type: "campaign_completed"; total_iterations: number; total_evaluations: number }
  | { type: "campaign_error"; error: string; iteration: number }
  | { type: "convergence_update"; metric: number; converged: boolean };

Frontend Integration

The dashboard uses these events to update visualizations in real-time:

• Convergence plot: New data points appear as each iteration completes.
• Pareto plot: The Pareto front is redrawn as new candidates are evaluated.
• Candidate table: New candidates appear with their predicted and (once evaluated) actual objective values.
• Progress indicator: Shows the current iteration, elapsed time, and estimated time remaining.

Python SDK Streaming

You can also stream events in the Python SDK:

from materia import Client

client = Client(base_url="https://api.matcraft.ai/api/v1", token="mc_live_...")

for event in client.campaigns.stream("camp-123"):
    if event.type == "iteration_completed":
        print(f"Iteration {event.iteration}: best = {event.best_value:.4f}")
    elif event.type == "campaign_completed":
        print(f"Done! {event.total_iterations} iterations, {event.total_evaluations} evaluations")
        break

CLI Live Output

The CLI shows real-time progress when running a campaign:

$ materia campaign run --config my_material.yaml
[Iter  1/20] Training surrogate... done (0.8s, val_loss=0.0342)
[Iter  1/20] Optimizing acquisition... done (1.2s, 5 candidates)
[Iter  1/20] Evaluating candidates... done (0.3s)
[Iter  1/20] Best: water_flux=42.3, salt_rejection=93.1%
[Iter  2/20] Training surrogate... done (0.9s, val_loss=0.0215)
...

Scaling WebSockets

For high-concurrency deployments with many simultaneous dashboard users:

• The FastAPI backend uses async WebSocket handlers, supporting thousands of concurrent connections per process.
• Redis Pub/Sub handles cross-process message distribution when running multiple API server instances behind a load balancer.
• WebSocket connections are authenticated with the same JWT token used for REST API calls.