What is MatCraft's system architecture?

MatCraft follows a modern, layered architecture with clear separation between the frontend, backend API, task processing, and core optimization engine.

Architecture Overview

┌──────────────────┐     ┌──────────────────┐
│   Next.js        │     │   Python SDK /   │
│   Frontend       │     │   CLI            │
│   (Port 3000)    │     │                  │
└────────┬─────────┘     └────────┬─────────┘
         │                        │
         │  REST / WebSocket      │  REST
         │                        │
         ▼                        ▼
┌──────────────────────────────────────────┐
│          FastAPI Backend (Port 8000)      │
│  ┌────────┐ ┌────────┐ ┌──────────────┐ │
│  │ Routes │ │ Auth   │ │ WebSocket    │ │
│  │        │ │ (JWT)  │ │ Manager      │ │
│  └────────┘ └────────┘ └──────────────┘ │
└────────────────┬─────────────────────────┘
                 │
        ┌────────┴────────┐
        ▼                 ▼
┌──────────────┐  ┌──────────────┐
│  PostgreSQL  │  │    Redis     │
│  (Data)      │  │  (Queue +    │
│              │  │   Pub/Sub)   │
└──────────────┘  └──────┬───────┘
                         │
                         ▼
                 ┌──────────────┐
                 │   Celery     │
                 │   Workers    │
                 │  ┌────────┐  │
                 │  │materia │  │
                 │  │ core   │  │
                 │  └────────┘  │
                 └──────────────┘

Component Details

Frontend (Next.js 14)

• Framework: Next.js with App Router and React Server Components.
• Styling: Tailwind CSS with shadcn/ui component library.
• State management: Zustand for client state, React Query for server state.
• Real-time: WebSocket connection for live campaign progress updates.
• Visualization: Recharts for convergence plots, Plotly.js for interactive 3D Pareto surfaces.

Backend (FastAPI)

• Framework: FastAPI with async request handling.
• Authentication: JWT tokens with refresh rotation. OAuth2 support for GitHub/Google.
• Database ORM: SQLAlchemy 2.0 with async session management.
• API documentation: Auto-generated OpenAPI/Swagger at /docs.
• WebSocket: Real-time campaign updates pushed to connected clients.

Task Queue (Celery + Redis)

• Broker: Redis serves as both the message broker and result backend.
• Workers: Celery workers execute long-running tasks: surrogate training, CMA-ES optimization, and active learning iterations.
• Concurrency: Each worker runs multiple threads/processes depending on whether the task is CPU-bound (optimization) or I/O-bound (data import).

Core Library (materia)

• Location: src/materia/
• Pure Python: No web framework dependencies. Can be used standalone.
• Key modules: surrogate/ (MLP, ONNX), optimize/ (CMA-ES), active_learning/ (acquisition functions, convergence), analysis/ (Pareto computation), plugins/ (domain definitions).

Database (PostgreSQL)

• Schema: Normalized schema with tables for organizations, users, materials, campaigns, iterations, candidates, and measurements.
• Migrations: Managed with Alembic. Run materia db upgrade to apply.

This architecture ensures that the compute-intensive optimization runs asynchronously without blocking the API, while the frontend receives real-time updates via WebSocket.