3 years ago

Category: Database, Kubernetes, automation

by Besides fault injection, a full chaos engineering application consists of hypothesizing around defined steady states, running experiments in production, validating the system via test cases, and automating the testing.

To validate how TiDB withstands chaos, we implemented dozens of test cases in TiPocket, combined with a variety of inspection tools.

The random SQL generated by SQLsmith is executed on TiDB and MySQL, and various faults are injected into the TiDB cluster to test.

Here is how Argo fits in TiPocket: The sample workflow for our predefined bank test is shown below: In this example, we use the workflow template and nemesis parameters to define the specific failure to inject.