AI Agents Cloud Engineering For DevOps Complete Guide

The development and operation of cloud environments has far exceeded the capacity of many organizations to manage them. With multiple Kubernetes clusters, multi-account cloud configurations, and an endless flow of alerts, DevOps teams are spending significantly more time responding to incidents than developing new features. AI agents for cloud engineering offer a completely different model - software that is capable of investigating problems, creating plans to fix those problems, and taking actions against your infrastructure, while operating within the confines of established safety nets. In this guide, you will learn what this looks like in the real world; where agentic DevOps is currently working and where it is not; and how to implement this model in your organization so that you do not put your production environment at risk.

AI agents for cloud engineering are software programs using large language models to create a plan for and complete cloud operations tasks using various tools and APIs. Instead of executing a set of instructions from a script, an agent will receive a goal (e.g., "determine why checkout latency has increased") and then determine what logs, metrics, and commands are needed to accomplish that goal, execute them, and adjust accordingly based on what is discovered.

What makes a DevOps Agent different than the current level of automation that many companies are utilizing? It is the loop. The agent will plan, act, observe the outcome of those actions, and then plan based upon that observation. This process continues until the task has been completed or the agent requires assistance from a person. That loop is what separates the DevOps Agents from most of today's automation efforts.

Scripts are the preferred method of completing predictable, repetitive tasks. Agents find their niche when there is no predetermined way of finding an answer.

Three forces are coming together: Infrastructure is becoming increasingly complex; hiring experienced Cloud/SRE professionals is challenging; and on-call burnout is a major retention issue. Simultaneously, the maturity of Tool-Calling within Large Language Models, along with standardizations such as the Model Context Protocol (MCP) that allow for easier connection between AI agents and existing systems used by teams, have transformed AI Infrastructure Automation from being just a prototype or demonstration into something worthy of serious evaluation.

The AI SRE Agent gets the most focus here. When there's a problem, an AI SRE Agent can pull up similar logs, look back to the last few times something was deployed, see what those measurements look like compared to a baseline, and then write up a summary of the issue along with what it thinks caused it in the incident channel. While the diagnosis may not always be 100% correct, it does save the engineer a great deal of time trying to gather information about what went wrong at 3:00 in the morning.

Agents have the capability to prioritize failed builds, determine which tests are non-deterministic (flaky), and provide recommendations to resolve issues through pull requests. Since the output generated by the agent is a proposed change that may be reviewed by others, it provides a relatively safe initial step into using agents in your DevOps processes.

The agent can write Terraform or another type of Infrastructure as Code (IaC) based upon a natural language request, create a Terraform plan, and identify if there is any difference between what has been declared in the code and what actually exists in the environment. The user will review the plan before deploying it.

The unseen costs of unused resources, excessively large databases, and forgotten stored files increase the overall cost of utilizing cloud-based services. Agents have the ability to review usage patterns, suggest right-sizing options, and create requests for changes to be made through opening tickets or creating pull requests containing suggested modifications. To better understand how to reduce your cloud-based service costs while avoiding downtime, please refer to our guide entitled "Reducing Cloud Infrastructure Costs Without Downtime."

Agents may evaluate configurations based on their own company's internal policies, identify excessive permissions in an IAM role, and gather supporting data for audit purposes. These actions are most effective when completed in a read-only manner.

The most common type of agent system that is used in cloud engineering has four levels.

• Reasoning layer: An LLM determines what needs to be done next based upon its understanding of the objective.
• Tool layer: Control over access to Kubernetes, Terraform, cloud provider API's, and ticketing systems; typically through MCP.
• Context layer: The agent has access to runbooks, architectural documentation, service owners' contact information, and past incident reports.
• Orchestration layer: a process that includes planning, acting, observing, and setting boundaries for actions such as limiting the number of actions to take, the amount of time available to complete those actions, and what an individual or group can do (permissions)

Context layer quality is generally more important than how smart the model is. A model that has access to your runbooks and can identify owners for each service will perform better than a model that is smarter, but "blind.

Teams can get in trouble by trying to jump directly to complete autonomy. A safer way is to move up the "ladder" only after you have established trust at each lower level.

• Assist: The agent assists by answering questions and summarizing information; however, it does not perform any tasks.
• Recommend: The agent provides recommendations on how to resolve an issue (e.g., provide a Pull Request, Remediate Plan) for consideration by a human to make the final decision.
• Act with approval: The agent can proceed with the execution of the changes only upon the completion of a formal approval process by humans.
• Act autonomously: The agent is able to perform a small number of well-defined tasks independently, with full activity logging and automatic recovery in case of error.

Teams will find themselves spending most of their time working on levels 1 and 2. It would be appropriate to use level 4 for those tasks that have a small blast radius (e.g., restarting a known flaky non-critical service) or when removing expired testing environments.

The decision to provide access to your organization's infrastructure via a probabilistic system warrants careful consideration. Both the potential threats associated with such systems as well as the means to protect against them are well known.

• Hallucinated or destructive commands: An agent can have "confidently wrong" or destructive/hallucinatory commands. Use least-privilege credentials, default to read-only, and make sure that Terraform plan reviews occur prior to applies.
• Prompt injection: Any type of log entry, ticket, or error message is a form of unverified input. An attacker may use a harmful string within a log entry to guide the actions of an agent. Therefore, treat all information that you retrieve as data and not as instructions, and limit what agents can do with the tools available to them.
• Runaway loops and cost: Cap the amount of time that can be used on each task, as well as how much money can be spent on that task.
• Missing context: An agent can't understand if a service is critical to the organization without being fed its ownership and criticality metadata; otherwise, an agent could treat a business-critical service as non-critical.
• Accountability gaps: To allow for both audit and post-incident review, each event should have an explanation of why the event occurred.

In each of the above cases, the general idea is that the amount of autonomy given to agents should be based on how easily an agent's actions can be undone. The easier it would be for an agent to reverse his/her actions, the more leeway the agent will have.

• Pick one narrow workflow: The best option would be to select a single, specific workflow such as failed build analysis or alert triage. Failed build analysis and alert triage both provide useful information as their final product.
• Start in a non-production environment: Let the software demonstrate its ability to function correctly without causing expensive errors.
• Connect read-only tools first: Only add the ability to make changes when the agent consistently makes correct recommendations.
• Document your runbooks: The quality of an agent's performance is limited by the amount of information he/she/it has access to. Documenting your run books is beneficial for agents and humans.
• Define approval gates: The definition of "approval gate." Beforehand, determine the actions that require the presence of a human being.
• Review and expand. Every week review your agents' choices; fill in any holes in their knowledge base; and begin to broaden the range of activities they perform.

In deciding if you should create a custom application or purchase an off-the-shelf product, consider whether the system will integrate with your current systems (stack), allow for approval workflow, produce audit trails, and enable you to specify exactly what parts of your environment it can access.

Measure what matters; do not measure based on how new it is. Useful metrics include MTTR, change failure rates, hours of toil that were eliminated, the number of alerts received by individuals, and money saved in cloud spending. A baseline must be established prior to the pilot to ensure that improvements can be proven as opposed to being described as an anecdote.

AI agents for cloud engineering do not replace the need for strong DevOps practices; rather, they enhance those practices. When teams have clearly defined runbooks, robust observability systems and strict processes for managing changes, they can begin to realize benefits from agents immediately. On the other hand, if teams lack these foundational elements, they will find that the agent highlights each of their weaknesses.

The simple way to go is to start small; allow humans to make decisions; track performance/results, and increase autonomy based upon how much faith you have in the technology.

If you are considering incorporating AI Agents into your Cloud Workflows, Hadrus Digital can assist you with assessing whether or not you are ready for this transition, as well as developing a plan for the safe implementation of these technologies. Contact us today to discuss how we may be able to support you in this journey.