Chaos Engineering

Google Cloud Platform

Azure

AWS

Kubernetes

Exploring Chaos Engineering With different Cloud Platforms

Saurabh Mishra

6 min readNov 9, 2022

Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

in another words,Chaos engineering is the practice of subjecting applications and services to real-world stresses and failures to build and validate resilience to unreliable conditions and missing dependencies.

A Brief History and Background

The concept of Chaos Engineering has been around since 2010, The Netflix Eng Tools team created Chaos Monkey. Chaos Monkey was created in response to Netflix’s move from physical infrastructure to cloud infrastructure provided by Amazon Web Services, and the need to be sure that a loss of an Amazon instance wouldn’t affect the Netflix streaming experience.

The Simian Army was born in 2011. The Simian Army added additional failure injection modes on top of Chaos Monkey that would allow testing of a more complete suite of failure states, and thus build resilience to those as well. “The cloud is all about redundancy and fault-tolerance. Since no single component can guarantee 100% uptime (and even the most expensive hardware eventually fails), we have to design a cloud architecture where individual components can fail without affecting the availability of the entire system.

They each have their own responsibilities, behaviors, and criteria for analysis. This approach makes it easy to introduce refined Chaos — Chaos that isn’t completely random and is focused on a specific type of learning In recent times, a lot of the larger cloud platforms such as AWS, Azure, etc., have started to provide their own built-in Chaos/fault injection tools, allowing you to run and report on experiments easily.

Netflix decided they would create a new role: the Chaos Engineer. Bruce Wong coined the term, and Dan Woods shared it with the greater engineering community via Twitter. Dan Woods explained, “I learned more about Chaos Engineering from Kolton Andrus than anyone else, he called it failure injection testing”.

In October of 2014, while Gremlin co-founder Kolton Andrus was at Netflix, his team announced Failure Injection Testing (FIT), a new tool that built on the concepts of the Simian Army, but gave developers more granular control over the “blast radius” of their failure injection. The Simian Army tools had been so effective that in some instances they created painful outages, causing many Netflix developers to grow wary of them. FIT gave developers control over the scope of their failure so they could realize the insights of Chaos Engineering, but mitigate potential downside.

This is great for the broadening of test coverage and knowledge of a testing type that in the past has been quite difficult to access (unless you’re building your own in-house one).

Gremlin publishes the first ever State of Chaos Engineering report in 2021. The report shows how the practice of Chaos Engineering has grown among organizations, key benefits of Chaos Engineering, how often top performing teams run chaos experiments, and more.

Principles of Chaos Engineering

. The network is reliable.

There is zero latency.
Bandwidth is infinite.
The network is secure.
Topology never changes.
There is one admin.
Transport cost is zero.
The network is homogeneous.

How is chaos engineering different from testing?

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” This definition established that it is a form of experimentation, which sits apart from testing.

Chaos Engineering Agenda

Be proactive, not reactive to failures.

Experiment often.

Robustness and resilience of your system should be SLA’s that you monitor.

Chaos Engineering in AWS

AWS Fault Injection Simulator (FIS)

AWS Fault Injection Simulator (FIS) is a fully managed service for running fault injection experiments to improve an application’s performance, observability, and resiliency. FIS simplifies the process of setting up and running controlled fault injection experiments across a range of AWS services, so teams can build confidence in their application behavior.

Chaos Engineering on Google Cloud

Google Cloud Chaos Monkey is an open-source tool written in F# that performs a few basic Chaos Experiments on Google Cloud instances. Instance target selection is pseudo-randomized and based on probabilities.

Reset: Resets the targeted instance.
Stop and Start: Stops the instance and then restarts it after 3 seconds.

Engineering Chaos Experiments on TensorFlow with Gremlin

It’s safe and easy to perform Chaos Experiments on Google Cloud Platform services like TensorFlow with Gremlin. As a clustered configuration TensorFlow is ideal for running Chaos Experiments to test the fault tolerance and stability of each clustered node. Gremlin can execute attacks that will show you the resilience of your TensorFlow cluster, including overloading CPU, and terminating instances.

Swapping Kubernetes Nodes in GKE with Chaos Toolkit

To induce Chaos on Google Cloud Kubernetes Engine clusters we can use the Chaos Toolkit Google Cloud driver. A relatively simple failure injection involves attacking a node pool with an HTTP load balancing library like Vegeta and then calling the swap_nodepool function to create a new node pool and drain the old pool. This forces pods to be moved to the new pool.

Chaos Engineering on Azure

Azure Chaos Studio

Azure Chaos Studio Preview is a fully managed chaos engineering experimentation platform for accelerating discovery of hard-to-find problems, from late-stage development through production. Disrupt your apps intentionally to identify gaps and plan mitigations before your customers are impacted by a problem.

Explore the benefits of Azure Chaos Studio

Subject your Azure applications to real or simulated faults

Observe how your applications respond to real-world disruptions

Integrate chaos experiments into any phase of quality validation

Use the same tools as Microsoft engineers to build resilience of cloud services

Conclusion

Chaos Engineering is a tool that we can use to improve the resilience of systems. As practitioners in this industry, our success depends not on removing the complexity from our systems, but on learning to live with it, navigate it, and optimize for other business-critical properties despite the underlying complexity.