Sentinel

Problem

An AI agent writing code against a third-party API has no way to know the API moved. It reads the docs, writes a client, and ships. Three weeks later a vendor turns price from a number into a string, and the failure surfaces as a runtime crash in someone’s pipeline — far from the change that caused it.

Existing tooling doesn’t help here. OpenAPI diffing assumes the vendor publishes an accurate spec, which is often the fiction the bug was hiding behind. Contract testing assumes you control both sides. What an agent actually needs, mid-task, is a cheap answer to one question: is the live response still shaped the way I coded against?

What I built

Sentinel is an MCP server exposing four tools an agent can call: infer a JSON Schema from a live endpoint, save it as a named baseline, diff two schemas, or do all of it in one check_endpoint call. The output isn’t a raw diff — it’s a change list where every entry carries a severity, so an agent can branch on breaking without parsing prose.

The severity model is the product:

Severity	Meaning	Examples
breaking	existing consumers can crash	field removed, type replaced (number → string)
risky	consumers may crash on some payloads	newly optional, newly nullable, type broadened, new value in a declared enum
additive	safe for existing consumers	new field, type narrowed, a value dropping out of an enum

That last row is the one that took the most thought, and it’s deliberately asymmetric. A new value appearing in an enum is risky — your switch has no branch for it. A value disappearing is additive, because a consumer that already handled it will simply never see it again. Most diff tools flag both as changes of equal weight and bury the signal.

How it works

• Inference learns optionality instead of guessing it. A field is marked required only if it appeared in every sampled object in an array, so pointing the tool at a list endpoint yields a real required-vs-optional split rather than a snapshot of one lucky record.
• Enum candidates never gate a verdict. Small string sets are surfaced as x-enum-candidates, but because they’re inferred rather than declared, any change to them is only ever additive. Values shaped like identifiers — URLs, timestamps, UUIDs, digests, emails, numeric strings — are excluded from candidacy entirely, which is what stops every ID field from looking like a two-value enum.
• Baselines are the only state. They sit behind a small store interface: JSON files locally (commit them, and schema history rides along with code review), an Apify key-value store when hosted.
• The first check is free and self-seeding. check_endpoint against an unknown baseline saves the current schema and says so, rather than erroring. First call establishes truth; every later call detects drift.

Architecture & decisions

• One core, three transports. The tool layer is transport-agnostic: stdio for local agents, a stateless Streamable HTTP server, and an Apify standby deployment — the same handler in all three. The HTTP server builds a fresh MCP server per request and keeps no sessions, so horizontal scaling needs no shared state.
• The core is pure. Inference and diffing are plain functions with no MCP or I/O dependency, which is why they carry the bulk of the 80 unit tests and why the whole thing has three runtime dependencies.
• SSRF was a design constraint, not a hardening pass. The tools fetch caller-supplied URLs, and hosted callers are untrusted. Fetches are restricted to http/https, hostnames are DNS-resolved and checked against loopback, RFC 1918, link-local and cloud-metadata, CGNAT, and unique-local IPv6 ranges, and redirects are followed manually so every hop is re-validated rather than only the first. Local stdio deliberately opts out — checking http://localhost:8080/api is the primary local use case — while the hosted actor never can. I verified this in production against 169.254.169.254; it fails closed.
• Metering shapes the free tier. infer_schema and save_baseline are free and diff_schemas / check_endpoint are charged, which makes exploration free and answers metered. The baseline-creating first call is also free, since nothing was diffed yet.

Impact / results

Sentinel is live on the Apify Store as a monetized, standby MCP server with pay-per-event pricing, verified end to end against the deployed instance: tool discovery, baseline persistence across separate runs, drift detection on a real public API, and the SSRF guard refusing cloud-metadata access. It runs under Apify’s restricted permission model, which the platform’s own documentation suggested wouldn’t work with a runtime-opened named store — establishing that it does took a production test rather than a doc read.

The part I’d point to is the severity model. Detecting that two JSON documents differ is trivial; deciding which differences a consumer will survive is the actual problem, and it’s what makes the output something an agent can act on unattended.

What I’d do next

• Pin fetches to the validated IP via a custom dispatcher, closing the DNS-rebinding window between validation and connection.
• Support anyOf/allOf so hand-written vendor specs diff as cleanly as inferred ones.
• Per-caller baseline namespacing for genuinely multi-tenant hosted use.
• A Cloudflare Worker deployment for a free hosted tier, which needs the Workers MCP adapter rather than the Node-only HTTP transport.