Replication cluster upgrade

Upgrading routers

Upgrade router instances one by one:

1. Stop one router instance.
2. Start this instance on the target Tarantool version.
3. Repeat the previous steps for each router instance.

After completing the router instances upgrade, perform the vshard.router check on each of them.

Upgrading storages

Before upgrading storage instances:

• Disable Cartridge failover: run

  tt cartridge failover disable
  

  or use the Cartridge web interface (Cluster tab, Failover: <Mode> button).

• Disable the rebalancer: run

  tarantool> vshard.storage.rebalancer_disable()
  

• Make sure that the Cartridge upgrade_schema option is false.

Upgrade storage instances by performing the following steps for each replica set:

1. Pick a replica (a read-only instance) from the replica set. Stop this replica and start it again on the target Tarantool version. Wait until it reaches the running status (box.info.status == running).
2. Restart all other read-only instances of the replica set on the target version one by one.
3. Make one of the updated replicas the new master using the applicable instruction from Switching the master.
4. Restart the last instance of the replica set (the former master, now a replica) on the target version.

5. Run box.schema.upgrade() on the new master. This will update the Tarantool system spaces to match the currently installed version of Tarantool. The changes will be propagated to other nodes via the replication mechanism later.

6. Run box.snapshot() on every node in the replica set to make sure that the replicas immediately see the upgraded database state in case of restart.

Once you complete the steps, enable failover or rebalancer back:

• Enable Cartridge failover: run

  tt cartridge failover set [mode]
  

  or use the Cartridge web interface (Cluster tab, Failover: Disabled button).

• Enable the rebalancer: run

  tarantool> vshard.storage.rebalancer_enable()
  

Rollback before the point of no return

If you decide to roll back before reaching the point of no return, your data is fully compatible with the version you had before the upgrade. In this case, you can roll back the same way: restart the nodes you’ve already upgraded on the original version.

Rollback after the point of no return

If you’ve passed the point of no return (that is, executed box.schema.upgrade()) during the upgrade, then a rollback requires downgrading the schema to the original version.

To check if an automatic downgrade is available for your original version, use box.schema.downgrade_versions(). If the version you need is on the list, execute the following steps on each upgraded replica set to roll back:

1. Run box.schema.downgrade(<version>) on master specifying the original version.
2. Run box.snapshot() on every instance in the replica set to make sure that the replicas immediately see the downgraded database state after restart.
3. Restart all read-only instances of the replica set on the initial version one by one.
4. Make one of the updated replicas the new master using the applicable instruction from Switching the master.
5. Restart the last instance of the replica set (the former master, now a replica) on the original version.

Then enable failover or rebalancer back as described in the Upgrading storages.

Replication check

Run box.info:

tarantool> box.info

Check that the following conditions are satisfied:

• box.info.status is running
• box.info.replication[*].upstream.status and box.info.replication[*].downstream.status are follow
• box.info.replication[*].upstream.lag is less or equal than box.cfg.replication_timeout, but it can also be moderately larger under a write load.
• box.info.ro is false at least on one instance in each replica set. If all instances have box.info.ro = true, this means there are no writable nodes. On Tarantool v. 2.10.0 or later, you can find out why this happened by running box.info.ro_reason. If box.info.ro_reason or box.info.status has the value orphan, the instance doesn’t see the rest of the replica set.

Then run box.info once more and check that box.info.replication[*].upstream.lag values are updated.

vshard.storage check

Run vshard.storage.info():

tarantool> vshard.storage.info()

Check that the following conditions are satisfied:

• there are no issues or alerts
• replication.status is follow

vshard.router check

Run vshard.router.info():

tarantool> vshard.router.info()

Check that the following conditions are satisfied:

• there are no issues or alerts
• all buckets are available (the sum of bucket.available_rw on all replica sets equals the total number of buckets)

Switching the master

• Cartridge. If your cluster runs on Cartridge, you can switch the master in the web interface. To do this, go to the Cluster tab, click Edit replica set, and drag an instance to the top of Failover priority list to make it the master.

• Raft. If your cluster uses automated leader election, switch the master by following these steps:

  1. Pick a candidate – a read-only instance to become the new master.
  2. Run box.ctl.promote() on the candidate. The operation will start and wait for the election to happen.
  3. Run box.cfg{ election_mode = "voter" } on the current master.
  4. Check that the candidate became the new master: its box.info.ro must be false.

• Legacy. If your cluster neither works on Cartridge nor has automated leader election, switch the master by following these steps:

  1. Pick a candidate – a read-only instance to become the new master.

  2. Run box.cfg{ read_only = true } on the current master.

  3. Check that the candidate’s vclock value matches the master’s: The value of box.info.vclock[<master_id>] on the candidate must be equal to box.info.lsn on the master. <master_id> here is the value of box.info.id on the master.

     If the vclock values don’t match, stop the switch procedure and restore the replica set state by calling box.cfg{ read_only == false } on the master. Then pick another candidate and restart the procedure.

After switching the master, perform the replication check on each instance of the replica set.