Body
MySQL HA Lab MySQL HA Lab · MySQL · High Availability

MySQL HA Lab: Learn MySQL InnoDB Cluster High Availability Hands-On with Docker

A hands-on lab series for MySQL 8.4 InnoDB Cluster on Docker: build a self-bootstrapping cluster, break it on purpose, measure failover, and write apps that survive it.

MySQL HA Lab Series: Learn MySQL InnoDB Cluster High Availability Hands-On
📘
This post is part of the MySQL HA Lab series: a hands-on MySQL 8.4 InnoDB Cluster on Docker.

Reading about high availability is easy. Watching your writes stop for 25 seconds while a primary dies, finding out where those seconds went, and getting them down to 6 is how it actually sticks.

This series is a hands-on lab for MySQL 8.4 InnoDB Cluster that runs entirely on your laptop with Docker Compose. Every lesson changes a few files, explains them setting by setting, and then breaks something on purpose: kill the primary, cut a node off the network, stall a secondary, crash a router. You measure what clients see, and then fix it.

⚠️
This is a learning lab. All nodes run on one machine, so it isn't real high availability. In production, run the nodes on separate hosts or availability zones.

What you will build

flowchart TB
    clients(["Clients"])
    subgraph LB["HAProxy (container: mysql-router)"]
        haproxy["HAProxy<br/>:6446 R/W · :6447 R/O · :6450 R/W split<br/>stats :8404"]
    end
    subgraph Routers["MySQL Router 8.4"]
        r1["mysql-router1"]
        r2["mysql-router2"]
    end
    subgraph GR["InnoDB Cluster: Group Replication"]
        direction LR
        m1["mysql1<br/>PRIMARY"]
        m2["mysql2<br/>SECONDARY"]
        m3["mysql3<br/>SECONDARY"]
        m1 <--> m2
        m2 <--> m3
    end
    subgraph Apps["Lab applications"]
        sb["labapp-springboot :8080"]
        go["labapp-go :8090"]
    end
    subgraph Mon["Monitoring"]
        exp["mysqld-exporter ×3"]
        prom["Prometheus :9090"]
        graf["Grafana :3000"]
    end
    clients --> haproxy
    Apps --> haproxy
    haproxy -- "health checks" --> r1
    haproxy -- "health checks" --> r2
    r1 & r2 -- "writes → primary" --> m1
    r1 & r2 -. "reads → secondaries" .-> m2
    r1 & r2 -. "reads → secondaries" .-> m3
    exp -- "one per node, direct" --> GR
    prom -- "scrape" --> exp
    prom -- "scrape :8404/metrics" --> haproxy
    prom -- "scrape /metrics" --> Apps
    graf --> prom

Lab architecture: clients reach the cluster through HAProxy and two MySQL Routers; Prometheus and Grafana watch every layer.

  • A 3-node InnoDB Cluster that builds itself: one docker compose up -d starts the nodes, creates the cluster, adds the members and starts the routers. Running it again is safe: it recovers from a full outage and re-adds missing nodes.
  • Two MySQL Routers behind HAProxy, so losing a router doesn't cut clients off.
  • Prometheus and Grafana with six provisioned dashboards, from "is everything OK?" down to a second-by-second failover timeline.
  • Two applications, in Spring Boot and in Go, that survive a failover without a single failed request.

One write, end to end

sequenceDiagram
    autonumber
    participant C as Client
    participant H as HAProxy :6446
    participant R as MySQL Router
    participant P as mysql1 (PRIMARY)
    participant S as mysql2 + mysql3
    C->>H: connect, INSERT
    H->>R: forward TCP to a healthy router
    R->>P: route to the current primary
    P->>S: send transaction for certification
    S-->>P: majority agrees
    P-->>R: COMMIT OK
    R-->>H: OK
    H-->>C: OK
    S->>S: apply in the background (may lag)

What happens to a single INSERT: the commit waits for a majority to agree, but the secondaries apply it in the background.

The lessons

#LessonYou learn
1Environment and SQL helpers.env, shell helpers, timeouts, database users, a transaction loop
2A self-bootstrapping InnoDB ClusterGroup Replication, node config, an idempotent AdminAPI bootstrap
3Failover, quorum and recoveryPrimary election, write downtime, quorum, how a node catches up
4Monitoring with Prometheus and Grafanamysqld-exporter, Prometheus, Grafana provisioning
5Highly available routing with HAProxyRouter bootstrap, HAProxy health checks, router failure tests
6Stale reads and read/write splittinggroup_replication_consistency, router port 6450
7Failover tuning, switchover and partitionssetPrimaryInstance, memberWeight, expel timeout, split brain
8A Spring Boot app that survives failoverRead/write pools, retry, idempotent writes, outage tracking
9The same app in Godatabase/sql pools, error classification, a shared metrics contract
BonusOperations and troubleshootingEvery command and every known problem with its fix

More lessons are planned: alerting, load testing with sysbench, backups and point-in-time recovery, then rolling upgrades, multi-primary mode and ClusterSet.

How to follow along

All code is on GitHub: mytechbytes/mysql-ha-lab. Each lesson has its own branch holding the repository as it was at the end of that lesson; main has everything.

  1. Start on main and follow the lessons in order. Each one ends with tests you run yourself.
  2. Use two terminals: one runs a transaction loop or an app's load generator, the other breaks something.
  3. Run source scripts/helpers.sh in every new terminal. It loads .env, writes the MySQL login file and defines the SQL helpers (mhelp lists them). Credentials never go on the command line.
  4. Before each lesson, make sure all three members are ONLINE (mstatus).

Requirements: Docker Desktop or Docker Engine with Compose v2, about 3 GB of free RAM (4 GB with both apps), and jq. The apps also need Java 25 and Maven, or Go, if you run them outside Docker.

Quick start

git clone https://github.com/mytechbytes/mysql-ha-lab.git && cd mysql-ha-lab
cp .env.example .env                      # then set every password and user
chmod 644 conf/*.cnf scripts/* .env
chmod -R a+rX monitoring haproxy

docker compose --profile monitoring up -d
docker logs --tail 100 -f cluster-init    # wait for the cluster status JSON

source scripts/helpers.sh
mstatus                                   # "status": "OK", 3 members ONLINE
URLWhat
localhost:6446 / 6447 / 6450MySQL read/write / read-only / read-write splitting, through HAProxy
http://localhost:8404/statsHAProxy: router health and sessions
http://localhost:9090Prometheus
http://localhost:3000Grafana → MySQL HA dashboards

Start with Lesson 1: Environment and SQL helpers.


The MySQL HA Lab series

  1. Series introduction (you are here)
  2. Lesson 1: Environment and SQL helpers
  3. Lesson 2: A self-bootstrapping InnoDB Cluster
  4. Lesson 3: Failover, quorum and recovery
  5. Lesson 4: Monitoring with Prometheus and Grafana
  6. Lesson 5: Highly available routing with HAProxy
  7. Lesson 6: Stale reads and read/write splitting
  8. Lesson 7: Failover tuning, switchover and partitions
  9. Lesson 8: A Spring Boot app that survives failover
  10. Lesson 9: The same app in Go
  11. Bonus: Operations and troubleshooting

All code is on GitHub: mytechbytes/mysql-ha-lab. Questions or a different result in your lab? Leave a comment below.

← Previous Extract Structured Data from Images Using Claude & Supabase Edge Functions Next → MySQL HA Lab, Lesson 1: Environment, Credentials and SQL Helpers
Part 1 of 11 · Collection MySQL HA Lab Back to collection →
Membership

Go deeper.

Members get the full technical guides, production checklists and the source repos behind every article.

You may also like