5 Signs Your Infrastructure Needs a DevOps Overhaul Before You Scale

It's the difficulty part that they don't all magically appear; sometimes the problem is infrastructure. They grow stealthily until one crappy week, when all at once, everyone notices them.

These are some ways to tell that your infrastructure has urgent issues that require attention before the situation becomes a major problem.

Don't let a nervousness about sending a new feature scare you off. This is a process issue.

Healthy teams view deploys as a regular occurrence. Sick ones regard them as surgery! Your release process is not scaling with your product if your engineers wait until Tuesday morning to deploy, as no one wants to "risk the weekend," or if a release needs someone to sit and watch dashboards for an hour after the release.

A deploy should possibly be boring. If there's that, that is the first sign.

All systems experience incidents. That is normal. If you are on call all the time, and you get paged every night, that's not normal. If nobody wants to do on-call shifts because they know they have to forgo their sleep, that's not normal.

Typically, if a system makes a lot of noise whenever there's a problem, there are two issues at hand: it doesn't have the observability needed to detect problems early, or it doesn't have the resilience to recover. Both issues are exacerbated at scale, as more users are hitting the same tender bulbs.

Along with the team, pose a straightforward but important question, How long will it take to recover our production servers if we lose our servers right now?

Anywhere you can put the real answer someone's memory, an old folder of scripts, or "we'd figure it out"- is not documentation as code. Is recorded as tribal knowledge; that is, it doesn't represent much more than what those 1 or 2 individuals know.

Infra-as-Code tools such as Terraform/Pulumi are created to overcome this exact issue. Each server, each environment, and each configuration should be something that your whole team can view, version, and be able to regenerate it, and not something that just exists in one engineer's mind.

If you get angry e-mail or a support ticket as an initial alert of an outage, then your monitoring is too poor.

Growing systems require real observability: metrics, logs, and traces that are directly connected to alerts that are attended by your team. The goal is simple. You don't solve a problem for your customer if there is one that you know exists, but your user doesn't.

If the infrastructure cannot be documented or there's inconsistent documentation, or many manual stages involved to keep it running, new hires are project onboarding engineers, and that's its own project. When you hire, while ramp-up time continues to increase each and every time, it's a scaling issue right there.

Clean infrastructure, clean documentation, and repeatable processes enable new engineers to contribute in days, not weeks.

There's no need to start from scratch if you are making a DevOps overhaul. In most circumstances, it implies:

• Infrastructure-as-code management of manual infrastructure became a reality between 2015 and 2017. Manual infrastructure became Infrastructure-as-code from 2015 to 2017.
• Incorporating gated CI / CD – all deploys on the same tested path.
• Creating a trustworthy observability stack, and actioning on alerts
• Writing incident runbooks makes on-call less of a guessing game.
• Administering a one-click rollback process, rather than a war room.

This doesn't even necessitate a complete overhaul. It demands essentially sealing a gap at a time, beginning with the one that is easiest to recognize amongst the signs above.