Monitoring MVP Systems

1 Learning Objectives

Key terms

Out of scope

• Enterprise observability platforms (Datadog, New Relic full stack)
• Full production monitoring infrastructure
• Advanced distributed tracing systems
• Complex real-time analytics pipelines

2 The Three Monitoring Layers

Every AI system you'll ship has three observability surfaces. Miss one and you'll be debugging in the dark.

Why silent failures are dangerous

A traditional web service tells you when it breaks — 500s in the logs, error rates spike, pagers go off. ML systems can keep returning HTTP 200 forever while quietly destroying business value:

• Recommendation engine still returns items, but they're stale and CTR has tanked.
• Fraud model still scores transactions, but new fraud patterns slip through.
• RAG assistant still answers questions, but cites a doc that was deprecated last quarter.
• Document classifier still labels — at 60% accuracy instead of 90%.

3 Four Systems, Four Monitoring Profiles

Each breakout team owns one of these in tonight's first lab. Notice how the critical metrics differ even though the "three layers" frame is the same.

4 Logs vs Metrics vs Alerts

The pipeline

Application logs
        ↓
Metrics aggregation  (counts, percentiles, distributions)
        ↓
Alert rules          (thresholds + dwell time)
        ↓
Dashboards / pages   (humans get the signal)
            

5 3-Hour Session Schedule

6 Hands-On Labs

Part A — Monitoring Framework

Create week31_ai_system_playbook/monitoring_framework.md. Include for your assigned system:

• Key metrics across the three layers
• Failure indicators (what does broken look like?)
• Alert thresholds with dwell times

Part B — Logging Script

Create monitoring/logging_demo.py. Use Python's stdlib logging — no external deps needed.

import logging, random, time

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def predict(req_id):
    latency = random.uniform(50, 600)
    logging.info(f"predict req_id={req_id} latency_ms={latency:.0f}")
    if latency > 500:
        logging.warning(f"slow predict req_id={req_id} latency_ms={latency:.0f}")
    if random.random() < 0.05:
        logging.error(f"model failure req_id={req_id}")

for i in range(20):
    predict(f"r{i:03d}")
    time.sleep(0.1)

Part C — Dashboard Plan

Create week31_ai_system_playbook/monitoring_dashboard.md. Sketch a one-screen layout with three panels:

• System health — request rate, error rate, latency p50/p95/p99
• Model metrics — accuracy / drift / confidence distribution over the last 24h
• User activity — DAU on the feature, engagement, override / feedback rate

For each panel: which metric, what time window, what threshold turns the panel red.

7 Anti-GenAI Requirement

8 Pre-class Reading

• Honeycomb — Observability vs Monitoring. Monitoring tracks predefined metrics; observability lets you ask new questions of system behavior. honeycomb.io/blog/observability-vs-monitoring
• Neptune — Monitoring ML Systems. Tracking accuracy, drift, and input distribution shifts. neptune.ai/blog/ml-model-monitoring
• Atlassian — Logging Best Practices. What to log, what not to log, why structured logs matter. atlassian.com/incident-management/logging

9 Closing Checklist

Before you leave tonight, every team should have:

• week31_ai_system_playbook/monitoring_framework.md
• week31_ai_system_playbook/monitoring_dashboard.md
• monitoring/logging_demo.py running locally
• The "Why these metrics matter for this AI system" section filled in

These artifacts roll forward into your capstone operations plan once topics drop.