Phoenix ML Platform Documentation

Technical documentation for the Phoenix ML Platform — a self-healing, model-agnostic real-time ML inference system built for modularity and production reliability.

What is Phoenix ML?

Phoenix ML is an end-to-end MLOps platform that provides:

• Real-time Inference: Model serving via REST API and gRPC with latency < 10 ms
• Model-Agnostic: Supports any model exported to ONNX (classification, regression, image, NLP)
• Self-Healing: Automatic data drift detection → alert → rollback → export fresh data → retrain
• Multi-Model Management: Champion/Challenger, A/B Testing, Canary Deployment, Shadow Mode
• Full Observability: Prometheus metrics, Grafana dashboards, Jaeger distributed tracing
• Event-Driven: Kafka for async event streaming and real-time prediction logging
• Automated Retraining: Export labeled prediction logs → merge with baseline → retrain on fresh distribution

Tech Stack

Architecture at a Glance

graph TD
    Client[Client] -->|REST / gRPC| API[FastAPI API Gateway]

    subgraph "Application"
        API --> PH[PredictHandler]
        PH --> IS[InferenceService]
        IS --> Router[Routing Strategy]
        Router --> CB[Circuit Breaker]
        CB --> Engine[ONNX / TensorRT / Triton]
    end

    subgraph "Data"
        IS -->|Features| Redis[(Redis)]
        API -->|CRUD| PG[(PostgreSQL)]
    end

    subgraph "Events"
        PH -.->|Async| Kafka{Kafka}
    end

    subgraph "Monitoring"
        Monitor[MonitoringService] --> Drift[DriftCalculator]
        Drift -.-> Alert[AlertManager]
        Alert -.-> Rollback[RollbackManager]
    end

    subgraph "Observability"
        API --> Prometheus
        Prometheus --> Grafana
        API --> Jaeger
    end

    Dashboard[React Dashboard] --> API

Documentation Sections

🏗 Architecture

• System Design — Full architecture: layer structure, data flow, infrastructure services, design patterns, self-healing flow
• Frontend Architecture — React dashboard: components, state management, API integration, testing strategy

📡 API & Deployment

• API Reference — All REST endpoints: inference, batch prediction, model management, drift monitoring, feature store
• Deployment Guide — Docker Compose stack (14+ services), Kubernetes Helm chart, environment config

📚 Guides

• Library API Guide — Use Phoenix ML as a Python library: programmatic inference, model loading, feature retrieval
• Customization Guide — Add models, implement custom engines/plugins, configure alerts, extend data loaders
• Monitoring & Alerting — Drift detection algorithms, anomaly detection, Prometheus metrics, Grafana dashboards, webhook alerts
• Troubleshooting — Common issues, error reference, debug commands, port mapping
• Contributing — Development setup, branching strategy, code standards, PR checklist

📋 Architecture Decision Records

📝 Blog Posts

• System Design Overview — Architecture decisions deep dive
• Self-Healing ML — Automated drift detection and model recovery
• Performance Optimization — Latency, throughput, and batching engineering

Quick Start

# Clone and setup
git clone https://github.com/vtnguyen04/phoenix_ML.git
cd phoenix_ML && cp .env.example .env

# Start all services
docker compose up -d
docker compose -f docker-compose.airflow.yaml up -d

# Train all models
uv run python scripts/generate_datasets.py
uv run python examples/credit_risk/train.py
uv run python examples/house_price/train.py
uv run python examples/fraud_detection/train.py
uv run python examples/image_classification/train.py
uv run python examples/sentiment/train.py

# Run full simulation
uv run python scripts/simulate_pipeline.py --fast

# Access
# API:       http://localhost:8001/docs
# Frontend:  http://localhost:5174
# MLflow:    http://localhost:5001
# Grafana:   http://localhost:3001
# Airflow:   http://localhost:8080

Project Stats

Repository: phoenix_ML