Catch cloud outage risks before they happen

Tools

custom-built or unnamed agent

Overview

Posted on Sep 26

Workflow

Posted on Sep 26 Sanity Challenge Path One Submission This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content. What I Built Modern engineering teams increasingly rely on autonomous AI coding agents (Claude Code, Cursor, GitHub Copilot) to generate code and push infrastructure pull requests. However, when these agents evaluate whether a service can be safely promoted to production, they almost universally rely on naive vector similarity search (e.g., Pinecone, standard RAG). In production infrastructure, vector similarity search is dangerous. If you ask: "Can I upgrade payment service from v2 to v4 tonight?" Vector search finds semantic text matches on "payment service" and happily responds "Looks safe!" — completely blind to upstream multi-hop dependencies, active cluster container versions, and contradictory documentation. The result? Catastrophic P0 outages in production. Introducing SHIPCHECK 🚢🛡️ SHIPCHECK is an autonomous pre-flight production release gatekeeper. Built as a full ReAct (Reasoning + Acting) Agent, SHIPCHECK connects directly to a live Sanity Content Lake via the Model Context Protocol (MCP). Before any code deployment or pull request is merged: Deconstructs the Proposal: Identifies target service, proposed version, and deployment environment. Autonomously Traverses 2-Hop Dependency Graphs via GROQ: Queries Sanity to discover hidden dependency constraints that keyword search never sees. Cross-References Real-World Cluster State: Audits the target requirements against live Kubernetes pod versions. Resolves Architectural Documentation Drift: When stale internal wikis conflict with official release specifications, SHIPCHECK programmatically arbitrates using Authority Weighting (1–10). Live Counterfactual Fix Simulation: Allows engineers to simulate a cluster patch (e.g., upgrading an SDK) and watch the dependency graph and verdict flip from DO NOT SHIP YET to SAFE TO SHIP before touching live servers. Generates Actionable 4-Phase Remediation: Produces executable kubectl rollout commands, migration schemas, and emergency rollback scripts. Demo 🌐 Live Web Application: https://sanity-omega-ten.vercel.app 💻 GitHub Repository: https://github.com/aiwithrajan/shipcheck 🗄️ Sanity Content Lake Project ID: 70rd1u6b (Dataset: production) How I Used Sanity Rather than using Sanity as a passive CMS, SHIPCHECK uses Sanity as the cognitive brain and grounding environment for the agent. 1. Schema Architecture for Agent Ingestion We designed 3 core document types in Sanity (70rd1u6b): component: Represents microservices with their live cluster version, health status, and environment. versionConstraint: Encodes relational rules (requiresComponent, operator, version, criticality). knowledgeEntry: Stores architectural specifications, policies, and release notes with explicit semantic links: contradicts: Directed references indicating which older specs are superseded. isExceptionOf: Encodes policy bypass conditions (e.g., emergency security hotfix bypassing a 24h soak period). authorityLevel: An integer from 1 to 10 ensuring the agent knows which document wins when specs disagree. contradicts: Directed references indicating which older specs are superseded. isExceptionOf: Encodes policy bypass conditions (e.g., emergency security hotfix bypassing a 24h soak period). authorityLevel: An integer from 1 to 10 ensuring the agent knows which document wins when specs disagree. 2. Model Context Protocol (MCP) Integration SHIPCHECK exposes Sanity to AI coding agents via a typed MCP JSON-RPC 2.0 interface. The agent autonomously invokes 4 specialized tools: query_sanity_component(componentName): Fetches live cluster state and metadata. query_version_constraints(componentName, targetVersion): Discovers relational constraints. audit_cluster_dependencies(dependencyName): Audits upstream pod versions. query_knowledge_graph(query): Resolves documentation drift and exceptions by authority level. What the Live Runs Showed When you instruct SHIPCHECK with: "Can I upgrade payment service from v2 to v4 tonight?" The agent executes a 13-step ReAct cognitive loop: The Verdict & Counterfactual Simulation Hard Blocker: Payment Service v4 requires Payment SDK >= v3.0.0, but production is running v2.4.1. Deploying tonight would crash all checkout pods.

Reported result

The author reports that grounding the agent with typed GROQ queries reduced hallucination rates from approximately 30% to 0% when identifying dependency conflicts. In a specific test case, the agent correctly identified a hard blocker where Payment Service v4 required Payment SDK >= v3.0.0, but the production cluster was running v2.4.1, preventing a potential outage. The counterfactual simulation feature successfully flipped the verdict from 'DO NOT SHIP YET' to 'SAFE TO SHIP' when simulating an SDK upgrade. No broader business impact metrics or failure rates are provided.

Human role

The original post does not say how a person stayed involved.

Limitations

The source does not specify operational costs, integration complexity, or exception handling for edge cases beyond the demonstrated dependency checks. Reliability depends on the accuracy of the Sanity Content Lake data and the live Kubernetes state. The author notes that documentation drift is common, requiring strict authority weighting (1-10) to resolve conflicts. Human approval is necessary before executing any generated remediation commands or deploying changes. The system's effectiveness is limited to the scope of the seeded 18 interconnected records in the demo.

Try it

I want to build a prototype of a production release gatekeeper agent similar to SHIPCHECK. I have a Sanity project [replaceable: Project ID] with datasets for 'components', 'versionConstraints', and 'knowledgeEntries'. I also have access to a Kubernetes cluster [replaceable: Cluster Name] via kubectl. Please perform the following steps: 1. Create a ReAct agent that uses MCP to query Sanity. Define tools: query_sanity_component, query_version_constraints, audit_cluster_dependencies, and query_knowledge_graph. 2. Implement logic to traverse 2-hop dependency graphs using GROQ. For example, if Component A requires Component B >= v2, check if Component B is currently running v1. 3. Implement authority weighting for knowledge entries (1-10) to resolve documentation conflicts. 4. Run a dry run with this scenario: 'Can I upgrade [replaceable: Service Name] from [replaceable: Current Version] to [replaceable: Target Version] tonight?' 5. Output the agent's reasoning steps, the final verdict (SAFE TO SHIP or DO NOT SHIP YET), and any identified blockers. 6. If a blocker is found, generate a counterfactual simulation showing what version change would make it safe. 7. Do not execute any actual deployments or changes. Present the results for my review before proceeding. Measure success by verifying that the agent correctly identifies the dependency conflict and provides a clear, actionable verdict without hallucinating non-existent constraints.

More cases · Software & development

• Update a code base with Copilot chat and agent mode
• Check code against user intent with review agents and tests
• Build a budgeting app from a single prompt
• Build a mental models framework with a multi-agent dev team
• Build a instant-opening markdown viewer for macOS