Diagnose and roll back bad config during outages
Developer Advocate. Task: Developer Advocate Software Engineer One of our favorite mottos in Google Site Reliability Engineering (SRE) is: "Eliminate Toil." You hear it in the microkitchens in Zurich and the hallways in Mountain View. This motto refers to the SRE mission of replacing repetitive, manual work with engineered systems. But as a senior SRE once explained, this doesn't just mean writing a script to solve a problem once. It means building the automation that triggers that script at the exact right moment—often the hardest part. AI has already revolutionized how we write code, but what about how we operate it? Can AI safely solve operational problems? Can it assist operators during a high-pressure outage without taking away control? In this article, we’ll delve into real scenarios that Google SREs are solving today using Gemini 3 (our latest foundation model) and Gemini CLI—the go-to tool for bringing agentic capabilities to the terminal. The Scenario: Fighting "Bad Customer Minutes" Meet Ramón. Ramón is a Core SRE, meaning he works in the engineering group that develops the foundational infrastructure for all of Google's products: safety, security, account management, and data backends for multiple services. When infrastructure at this layer has a hiccup, it’s visible across a massive range of products immediately. Speed is vital, and we measure it in Bad Customer Minutes. Every minute the service is degraded burns through our Error Budget. To combat this, we obsess over MTTM (Mean Time to Mitigation). Unlike Mean Time to Repair (MTTR), which focuses on the full fix, MTTM is about speed: how fast can we stop the pain? In this space, SREs typically have a 5-minute Service Level Objective (SLO) just to acknowledge a page, and extreme pressure to mitigate shortly after. Our incident usually follows four distinct stages: Paging: The SRE gets alerted. Paging: The SRE gets alerted. Mitigation: We "stop the bleeding" to reduce Bad Customer Minutes, often before we even know why it broke. Mitigation: We "stop the bleeding" to reduce Bad Customer Minutes, often before we even know why it broke. Root Cause: Once users are happy, we investigate the underlying bug and fix it for good. Root Cause: Once users are happy, we investigate the underlying bug and fix it for good. Postmortem: We document the incident and add extensive action items on engineering teams, that are prioritised to ensure it never happens again. Postmortem: We document the incident and add extensive action items on engineering teams, that are prioritised to ensure it never happens again. Let's walk through a real (simulated) outage to see how Gemini CLI accelerates every step of this journey to keep MTTM low. Step 1: Paging and Initial Investigation Context: It's 11:00 AM. Ramón’s pager goes off for incident s_e1vnco7W2. When a page comes in, the clock starts ticking. Our first priority isn't to fix the code—it's to mitigate the impact on users. Thanks to the extensive work on Generic Mitigations by Google SREs, we have a defined, closed, set of standard classes of mitigations (e.g., drain traffic, rollback, restart, add capacity). This is a perfect task for an LLM: classify the symptoms and select a mitigation playbook. A mitigation playbook is an instruction created dynamically for an agent to be able to execute a production mutation safely. These playbooks can include the command to run, but also instructions to verify that the change is effectively addressing the problem, or to rollback the change. Ramón opens his terminal and uses Gemini CLI. Gemini immediately calls the fetch_playbook function from ProdAgent (our internal agentic framework). It chains together several tools to build context: get_incident_details: Fetches the alert data from our Incident Response Management system (description, metadata, prior instances, etc). get_incident_details: Fetches the alert data from our Incident Response Management system (description, metadata, prior instances, etc). causal_analysis: Finds causal relations between different time series behaviour and generic mitigation labels. causal_analysis: Finds causal relations between different time series behaviour and generic mitigation labels. timeseries_correlation: Finds pairs of time series that are correlated, which may help the agent find root causes and mitigation timeseries_correlation: Finds pairs of time series that are correlated, which may help the agent find root causes and mitigation; tools: Gemini. Human role: The original post does not say how a person stayed involved.; limitations: Requires integration with internal agentic frameworks (ProdAgent) and Model Context Protocol (MCP) servers for tools like incident management and code repositories. Human approval is mandatory for all production mutations to ensure safety and compliance. Audit trails are automatically generated, but policy enforcement layers must be configured to restrict high-risk actions..
Tools
Published on Agent Work Atlas:
Overview
Developer Advocate
Workflow
Developer Advocate Software Engineer One of our favorite mottos in Google Site Reliability Engineering (SRE) is: "Eliminate Toil." You hear it in the microkitchens in Zurich and the hallways in Mountain View. This motto refers to the SRE mission of replacing repetitive, manual work with engineered systems. But as a senior SRE once explained, this doesn't just mean writing a script to solve a problem once. It means building the automation that triggers that script at the exact right moment—often the hardest part. AI has already revolutionized how we write code, but what about how we operate it? Can AI safely solve operational problems? Can it assist operators during a high-pressure outage without taking away control? In this article, we’ll delve into real scenarios that Google SREs are solving today using Gemini 3 (our latest foundation model) and Gemini CLI—the go-to tool for bringing agentic capabilities to the terminal. The Scenario: Fighting "Bad Customer Minutes" Meet Ramón. Ramón is a Core SRE, meaning he works in the engineering group that develops the foundational infrastructure for all of Google's products: safety, security, account management, and data backends for multiple services. When infrastructure at this layer has a hiccup, it’s visible across a massive range of products immediately. Speed is vital, and we measure it in Bad Customer Minutes. Every minute the service is degraded burns through our Error Budget. To combat this, we obsess over MTTM (Mean Time to Mitigation). Unlike Mean Time to Repair (MTTR), which focuses on the full fix, MTTM is about speed: how fast can we stop the pain? In this space, SREs typically have a 5-minute Service Level Objective (SLO) just to acknowledge a page, and extreme pressure to mitigate shortly after. Our incident usually follows four distinct stages: Paging: The SRE gets alerted. Paging: The SRE gets alerted. Mitigation: We "stop the bleeding" to reduce Bad Customer Minutes, often before we even know why it broke. Mitigation: We "stop the bleeding" to reduce Bad Customer Minutes, often before we even know why it broke. Root Cause: Once users are happy, we investigate the underlying bug and fix it for good. Root Cause: Once users are happy, we investigate the underlying bug and fix it for good. Postmortem: We document the incident and add extensive action items on engineering teams, that are prioritised to ensure it never happens again. Postmortem: We document the incident and add extensive action items on engineering teams, that are prioritised to ensure it never happens again. Let's walk through a real (simulated) outage to see how Gemini CLI accelerates every step of this journey to keep MTTM low. Step 1: Paging and Initial Investigation Context: It's 11:00 AM. Ramón’s pager goes off for incident s_e1vnco7W2. When a page comes in, the clock starts ticking. Our first priority isn't to fix the code—it's to mitigate the impact on users. Thanks to the extensive work on Generic Mitigations by Google SREs, we have a defined, closed, set of standard classes of mitigations (e.g., drain traffic, rollback, restart, add capacity). This is a perfect task for an LLM: classify the symptoms and select a mitigation playbook. A mitigation playbook is an instruction created dynamically for an agent to be able to execute a production mutation safely. These playbooks can include the command to run, but also instructions to verify that the change is effectively addressing the problem, or to rollback the change. Ramón opens his terminal and uses Gemini CLI. Gemini immediately calls the fetch_playbook function from ProdAgent (our internal agentic framework). It chains together several tools to build context: get_incident_details: Fetches the alert data from our Incident Response Management system (description, metadata, prior instances, etc). get_incident_details: Fetches the alert data from our Incident Response Management system (description, metadata, prior instances, etc). causal_analysis: Finds causal relations between different time series behaviour and generic mitigation labels. causal_analysis: Finds causal relations between different time series behaviour and generic mitigation labels. timeseries_correlation: Finds pairs of time series that are correlated, which may help the agent find root causes and mitigation timeseries_correlation: Finds pairs of time series that are correlated, which may help the agent find root causes and mitigation
Reported result
The source describes a simulated scenario where Gemini CLI accelerates incident response by reducing context switching and automating postmortem generation. No measured business outcomes, MTTM reductions, or production metrics are reported. A proposed acceptance check is to verify that the agent correctly identifies the failing service component and generates a valid rollback command in a sandbox environment.
Human role
The original post does not say how a person stayed involved.
Limitations
Requires integration with internal agentic frameworks (ProdAgent) and Model Context Protocol (MCP) servers for tools like incident management and code repositories. Human approval is mandatory for all production mutations to ensure safety and compliance. Audit trails are automatically generated, but policy enforcement layers must be configured to restrict high-risk actions.
Try it
Act as an SRE using Gemini CLI to handle a simulated outage. Input: [Incident ID: s_e1vnco7W2], [Service: Core Infrastructure], [Alert: High latency]. Steps: 1. Use Gemini CLI to fetch incident details and analyze time-series correlations. 2. Propose a mitigation playbook (e.g., restart or rollback) based on symptoms. 3. Present the proposed command to me for approval. Do not execute until I say 'SGTM'. 4. If I approve, simulate the execution and report the result. 5. If the mitigation fails, analyze logs to find the root cause and suggest a code fix. Output: A step-by-step log of the agent's reasoning, the proposed commands, and my approval status. Ensure all actions are logged for audit purposes. Ask clarifying questions if the incident context is ambiguous.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.