Chapter 13 — Capstone: SmartPlant Decision Intelligence, End to End

Chapters 9 to 12 each built and measured one layer of SmartPlant.

In this chapter. Now we assemble them into the shape you’d deploy: edge topology, security zones, monitoring, governance and scaling to many assets. We also close the learning loop, where your own audit trail is used to fine-tune Kev on your plant.

The result is a path from sensor data to trusted industrial decisions. Code computes, Kev decides, people authorise, and fresh state proves the action happened.

13.1 What we’re shipping

Here’s the whole system on one page. Each layer was built in an earlier chapter, and the last column shows which Part I pattern it reuses, in case you want to look it up.

Only training_export is new here. Part I and Part II share one uv workspace, so the smartplant package depends directly on the Part I triage engine.

mastering-jev-guide/
├── pyproject.toml                    # uv workspace: triage-engine + smartplant (both editable)
├── capstone-project/src/triage/      # Part I (jev_client now also has KevDecisionClient)
├── smartplant-capstone/
│   ├── pyproject.toml                # package "smartplant", depends on triage-engine
│   ├── src/smartplant/
│   │   ├── sensor_simulator.py  fault_injector.py  edge_state_builder.py      # Ch9
│   │   ├── contract.py  context_enricher.py  models.py  decision_service.py   # Ch10
│   │   ├── fallback_rules.py  cmms_mock_service.py  audit_log.py              # Ch11
│   │   ├── action_orchestrator.py                                             # Ch11
│   │   ├── scenario_runner.py  benchmark_metrics.py  eval_dashboard.py        # Ch12
│   │   ├── training_export.py                                                 # Ch13
│   │   └── data/test_scenarios.json
│   └── tests/                        # 26 tests, no GPU or network needed
└── mini-projects/
    ├── 08-iot-sensor-simulator/run_simulator.py  kev_console.py
    ├── 09-smartplant-decision-service/run_decision_service.py
    ├── 10-closed-loop-controller/run_controller.py
    ├── 11-smartplant-eval-suite/run_eval.py
    └── 12-learning-loop/record_outcomes.py

The 26 tests need no GPU or network, so you can run them anywhere before going near a plant.

13.2 Where everything runs

On a plant, where a component runs matters as much as what it does. Here’s SmartPlant mapped onto the plant network.

A few design decisions shape that picture.

• Kev runs at Level 3, on the same box as the decision service, over localhost. Level 3 is site operations in the Purdue model. Telemetry never leaves the site, and a WAN outage doesn’t stop the plant from getting decisions.
• The edge box has no write path down to Levels 0 and 1, where the sensors, actuators and PLCs live. All it can do is create CMMS work orders and raise requests to the control room. Control actions stay with the operators and the PLC.
• Traffic follows zones and conduits, in the spirit of IEC 62443. The edge box talks to the CMMS and MES only through defined, firewalled conduits. Nothing in the IT zone can use the edge box as a route into the PLC network.
• Size the model to the box. Kev-0.8B runs on any 4 GB GPU or Apple Silicon for single-line pilots. Kev-4B needs an L40S-class GPU (or a 32 GB Mac) when you need its accuracy. Kev-9B gives extra headroom.

13.3 Bring it all up

You need two terminals. The first runs the Kev server from its own folder and virtual environment (Python 3.12 or 3.13). The second runs the book workspace: the tests, then the decision service, closed-loop controller and evaluation mini-projects. In step 2, DECISION_BACKEND=kev points SmartPlant at your local Kev, and KEV_API_KEY must match the key from step 1.

# 1. Kev (its own folder & venv, Python 3.12/3.13)
cd ~/src/kev && export KEV_API_KEY=$(openssl rand -hex 16)
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b@v1.0 --port 8009

# 2. The book workspace (another terminal)
cd ~/src/mastering-jev-guide && uv sync --group iot
export DECISION_BACKEND=kev KEV_BASE_URL=http://127.0.0.1:8009 KEV_API_KEY=...   # same key as above
uv run pytest -q                                                               # Part I + Part II
uv run python mini-projects/09-smartplant-decision-service/run_decision_service.py S03 S06 S09
uv run python mini-projects/10-closed-loop-controller/run_controller.py --kev-down-at 120
uv run python mini-projects/11-smartplant-eval-suite/run_eval.py --kev --kev-rps 20

Pin the checkpoint, either a tag like @v1.0 or a local runs/... directory, the same way Part I pinned JEV_MODEL. Your thresholds were calibrated (tuned to the model’s confidence scores) against one checkpoint, so if the checkpoint changes, they stop meaning what you think.

13.4 The learning loop: fine-tune Kev on your plant

Kev’s biggest advantage over a hosted API is that you can train it. Fine-tuning means continuing a trained model’s training on your own examples. The Kev README reports that fine-tuning lifted a support workload from 67.7 % to 73.6 % accuracy, and raised the share of cases it could automate at a 5 % error budget from 34 % to 48 %. On a consumer-finance set it went from 0.804 to 0.904. Each Kev-4B run cost about $1 of H100 time.

SmartPlant already records what you need. The audit log holds the snapshot Kev saw. When a technician closes the work order, the outcome carries expert labels, much like CMMS close-out codes: what the right call actually was. Pair the two and you have training data.

The training export does that pairing. Watch three things: examples_from_audit keeps only correlation IDs that have both a snapshot and an outcome, _labeled_example drops missing or out-of-range labels, and severity shifts from the plant’s 1–5 scale to a zero-based level.

# smartplant-capstone/src/smartplant/training_export.py
"""Turn the audit trail into Kev fine-tuning data: snapshot + contract + expert-confirmed labels.

Output is one JSON object per line in kev.train's format: each question carries a `label`
(choice -> option name, noul -> true/false, score -> zero-based level).
"""
from __future__ import annotations

import argparse
import json
import random
from collections import defaultdict
from pathlib import Path
from typing import Any, Iterable, Mapping

from smartplant.contract import ACTIONS, URGENCIES, build_questions

LABEL_KEYS = ("intervention_urgency", "operational_action", "standard_workflow", "operational_severity")


def _labeled_example(state: Mapping[str, Any], labels: Mapping[str, Any]) -> dict[str, Any] | None:
    if not all(k in labels for k in LABEL_KEYS):
        return None                                    # partial labels are not training data
    qs = build_questions(state, speculative=False)
    if labels["operational_action"] not in qs["operational_action"]["criteria"] or \
            labels["intervention_urgency"] not in URGENCIES or labels["operational_action"] not in ACTIONS:
        return None
    sev = int(labels["operational_severity"])
    if not 1 <= sev <= 5:
        return None
    qs["intervention_urgency"]["label"] = labels["intervention_urgency"]
    qs["operational_action"]["label"] = labels["operational_action"]
    qs["standard_workflow"]["label"] = bool(labels["standard_workflow"])
    qs["operational_severity"]["label"] = sev - 1          # plant scale 1..5 -> zero-based level
    return {"state": dict(state), "questions": qs}


def examples_from_audit(lines: Iterable[str]) -> list[dict[str, Any]]:
    by_corr: dict[str, dict[str, Any]] = defaultdict(dict)
    for line in lines:
        if not line.strip():
            continue
        e = json.loads(line)
        if e["event"] in ("snapshot", "outcome"):
            by_corr[e["correlation_id"]][e["event"]] = e["payload"]
    out = []
    for parts in by_corr.values():
        if "snapshot" in parts and "outcome" in parts:
            ex = _labeled_example(parts["snapshot"], parts["outcome"].get("expert_labels", {}))
            if ex:
                out.append(ex)
    return out


def write_split(examples: list[dict[str, Any]], out_dir: Path, heldout: float = 0.15, seed: int = 7) -> tuple[Path, Path]:
    rnd = random.Random(seed)
    rows = examples[:]
    rnd.shuffle(rows)
    k = max(1, int(len(rows) * heldout)) if len(rows) > 1 else 0
    out_dir.mkdir(parents=True, exist_ok=True)
    train, held = out_dir / "train.jsonl", out_dir / "heldout.jsonl"
    train.write_text("".join(json.dumps(r) + "\n" for r in rows[k:]), encoding="utf-8")
    held.write_text("".join(json.dumps(r) + "\n" for r in rows[:k]), encoding="utf-8")
    return train, held


def main() -> int:
    ap = argparse.ArgumentParser(description="Export expert-labeled decisions for kev.train")
    ap.add_argument("audit", type=Path, nargs="+")
    ap.add_argument("--out", type=Path, default=Path("kev-finetune"))
    args = ap.parse_args()
    ex = [x for p in args.audit for x in examples_from_audit(p.read_text(encoding="utf-8").splitlines())]
    train, held = write_split(ex, args.out)
    print(f"{len(ex)} labeled examples -> {train}, {held}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

write_split holds back 15 % of examples by default, which training never sees. The full cycle is export, fine-tune from kev-4b, benchmark on the held-out set, then serve the new checkpoint.

uv run smartplant-export audit/*.jsonl --out kev-finetune
cd ~/src/kev
uv run python -m kev.train --data ../mastering-jev-guide/kev-finetune/train.jsonl \
    --base Qwen/Qwen3.5-4B-Base --init_from jaredpalmer/kev-4b \
    --epochs 2 --lr 2e-5 --batch 1 --accum 8 --dtype bf16 --checkpointing 1 --device cuda --out runs/smartplant
uv run python -m kev.benchmark --run runs/smartplant --data ../mastering-jev-guide/kev-finetune/heldout.jsonl \
    --out runs/smartplant-eval
uv run --extra serve python -m kev.serve --run runs/smartplant --port 8009      # then re-run Ch12

Rules for a training set you can trust

• Labels must come from people, after the fact. Never use Kev’s own answers as labels. If you do, you’re just training it to agree with itself.
• Keep the contract fixed between export and serving. training_export rebuilds the questions with build_questions(), so the pruned option lists in training match what Kev will see in production.
• Hold out data by time and by asset. Test on last month’s motors, and on at least one asset the model never saw in training. That gives you a fair read on whether it generalises.
• Critical cases are rare, so oversample them in review rather than in the labels. Make sure every critical miss in the field gets reviewed and labelled.

Hands-on: close the learning loop in 10 simulated minutes

This walk-through follows one bearing from telemetry to training data. It uses the clean audit/smartplant.jsonl from Chapter 11’s hands-on: a 10-minute run with two work orders. If you don’t have it, recreate it:

rm -rf audit
uv run python mini-projects/10-closed-loop-controller/run_controller.py --offline --auto-approve \
  --steps 61 --every 6 --fault-start 6 --magnitude 2

Step 1: Close the work orders as the technician. In a real plant this happens when the technician completes the job in the CMMS. Here, record_outcomes.py asks you what you found and what the right call was, then appends an outcome event to the same hash-chained audit log.

uv run python mini-projects/12-learning-loop/record_outcomes.py audit/smartplant.jsonl

WO-1001  M-104: vib C_alert/rising_fast, temp normal  | system said URGENT/INSPECT sev 4 [fake]
  what did you find? outer race noise, bearing hot to touch
  (press Enter on any question to leave this work order unlabelled)
  urgency ['ROUTINE', 'PRIORITY', 'URGENT', 'CRITICAL']: URGENT
  action ['MONITOR', 'INSPECT', 'SCHEDULE_MAINTENANCE', 'TRANSFER_LOAD', 'REQUEST_SHUTDOWN']: SCHEDULE_MAINTENANCE
  standard workflow was enough? [y/n]: y
  severity 1-5: 4
  -> outcome recorded: {'intervention_urgency': 'URGENT', 'operational_action': 'SCHEDULE_MAINTENANCE', 'standard_workflow': True, 'operational_severity': 4}

Notice that the system’s call (INSPECT) and yours (SCHEDULE_MAINTENANCE) differ. That disagreement is exactly what fine-tuning learns from. If you press Enter on any question, the work order is closed without labels and is left out of training, because partial labels aren’t training data.

No time to type? --demo uses a scripted technician that labels from the bearing condition. It’s for trying the plumbing only: real labels must come from people who saw the machine.

uv run python mini-projects/12-learning-loop/record_outcomes.py audit/smartplant.jsonl --demo

Step 2: Export the training set.

uv run smartplant-export audit/smartplant.jsonl --out kev-finetune
jq -c '{zone: .state.telemetry.vibration_zone, labels: (.questions | with_entries(.value = .value.label))}' kev-finetune/*.jsonl

2 labeled examples -> kev-finetune/train.jsonl, kev-finetune/heldout.jsonl
{"zone":"C_alert","labels":{"intervention_urgency":"URGENT","operational_action":"SCHEDULE_MAINTENANCE","standard_workflow":true,"operational_severity":3}}
{"zone":"D_danger","labels":{"intervention_urgency":"CRITICAL","operational_action":"TRANSFER_LOAD","standard_workflow":false,"operational_severity":4}}

Each row pairs the exact state Kev saw with the technician’s answer. Severity 4 on the plant’s 1–5 scale is stored as level 3, because Kev’s score levels start at zero.

Step 3: Collect a few more 10-minute runs. Two examples only prove the mechanics. Give each run its own audit file, with slower and faster wear:

for m in 1 3; do
  uv run python mini-projects/10-closed-loop-controller/run_controller.py --offline --auto-approve \
    --steps 61 --every 6 --fault-start 6 --magnitude $m --audit audit/wear-$m.jsonl
  uv run python mini-projects/12-learning-loop/record_outcomes.py audit/wear-$m.jsonl --demo
done
uv run smartplant-export audit/*.jsonl --out kev-finetune

PLC protective trip latched: motor stopped by the PLC; loop ends until operator reset.
6 labeled examples -> kev-finetune/train.jsonl, kev-finetune/heldout.jsonl

At magnitude 3, vibration hits the protective limit before minute 10. The controller records the PLC trip and stops, and that work order is labelled like any other. Six examples is still far too few to train on; a real set needs hundreds of reviewed outcomes. The workflow, though, is the one you’ll run in production.

Step 4: Fine-tune and benchmark (GPU box). Run the commands from 13.4 against kev-finetune/: kev.train from kev-4b, then kev.benchmark on heldout.jsonl. Training needs a CUDA GPU; with a handful of examples, expect no real change in accuracy.

Step 5: Serve the new checkpoint and re-run the acceptance test.

# terminal 1
cd ~/src/kev && uv run --extra serve python -m kev.serve --run runs/smartplant --port 8009
# terminal 2
curl -s $KEV_BASE_URL/v1/models | jq '.'                                   # confirm the new checkpoint
uv run python mini-projects/11-smartplant-eval-suite/run_eval.py --kev --kev-rps 20
uv run python mini-projects/08-iot-sensor-simulator/kev_console.py snapshots.jsonl

Apply the Chapter 12 ship gate to the new kev line. Then replay the 10-minute timeline and compare it with the run you saved before fine-tuning. The new checkpoint ships only if critical misses stay at zero and agreement doesn’t drop. If it ships, pin it (a tag or a runs/... path) and re-tune the thresholds, because a new checkpoint changes what its confidence scores mean.

13.5 Production readiness

Working on the bench isn’t the same as running for months in a plant. Here’s what to watch, how to control change, and how to grow.

Monitoring

These signals cover Kev, the sensors and the human approval process.

Governance

• Version four things together, alongside the Kev checkpoint. These are the snapshot schema (smartplant-snapshot/1.0.0), the decision state (smartplant-state/1.0.0), the contract (smartplant-contract/1.0.0, plus its hash) and the policy (smartplant-policy/1.0.0). Any change to any of them triggers the Chapter 12 replay plus a run on the held-out set.
• Put the contract and thresholds under change control. They live in two reviewable files. Handle edits to them like PLC logic changes, with one reviewer from reliability engineering and one from operations.
• Retain the audit chain for at least as long as you keep your maintenance records.

Scaling to many assets and sites

• Run one builder per asset and one decision service per site. Kev batches requests well. The README reports about 101 requests per second for Kev-4B on an H100, so a single box can cover hundreds of assets at a one-minute decision cadence.
• Decide when something changes, not on every sample. The debounced trigger from Chapter 11 keeps the decision rate proportional to events, not to the number of sensors.
• Set limits per asset and thresholds per site. Calibrate separately for each asset class (motors and pumps behave differently), and recalibrate whenever a site changes its maintenance practice.
• Learn across the fleet. Export labels from all sites into one fine-tuning set. Then evaluate on a held-out site before you roll a new checkpoint out everywhere.

13.6 Rollout plan

Like any new control strategy, SmartPlant goes in stages, not everywhere at once.

Notice that there is deliberately no “fully autonomous” stage at the end. Load-transfer and shutdown requests always need a human to approve them, and protective functions always stay in the PLC.

13.7 Capstone deliverables checklist

Here’s where each item from the SmartPlant brief lives in the code.

The ship criterion is the same as Part I. SmartPlant ships when it beats the current process (threshold alarms plus manual triage) on three measures: critical-case misses, unnecessary interventions and time-to-decision. And it has to do that while every consequential action still needs a human approval and every protective function stays in the PLC.

Code computes. Kev decides. People authorise. Fresh state proves.