Master-replica: manual failover | Tarantool
How-to guides Replication tutorials Master-replica: manual failover

Master-replica: manual failover

Example on GitHub: manual_leader

This tutorial shows how to configure and work with a replica set with manual failover.

Before starting this tutorial:

  1. Install the tt utility.

  2. Create a tt environment in the current directory by executing the tt init command.

  3. Inside the instances.enabled directory of the created tt environment, create the manual_leader directory.

  4. Inside instances.enabled/manual_leader, create the instances.yml and config.yaml files:

    • instances.yml specifies instances to run in the current environment and should look like this:

      instance001:
      instance002:
      
    • The config.yaml file is intended to store a replica set configuration.

This section describes how to configure a replica set in config.yaml.

First, set the replication.failover option to manual:

replication:
  failover: manual

Define a replica set topology inside the groups section:

  • The leader option sets instance001 as a replica set leader.
  • The iproto.listen option specifies an address used to listen for incoming requests and allows replicas to communicate with each other.
groups:
  group001:
    replicasets:
      replicaset001:
        leader: instance001
        instances:
          instance001:
            iproto:
              listen:
              - uri: '127.0.0.1:3301'
          instance002:
            iproto:
              listen:
              - uri: '127.0.0.1:3302'

In the credentials section, create the replicator user with the replication role:

credentials:
  users:
    replicator:
      password: 'topsecret'
      roles: [replication]

Set iproto.advertise.peer to advertise the current instance to other replica set members:

iproto:
  advertise:
    peer:
      login: replicator

The resulting replica set configuration should look as follows:

credentials:
  users:
    replicator:
      password: 'topsecret'
      roles: [replication]

iproto:
  advertise:
    peer:
      login: replicator

replication:
  failover: manual

groups:
  group001:
    replicasets:
      replicaset001:
        leader: instance001
        instances:
          instance001:
            iproto:
              listen:
              - uri: '127.0.0.1:3301'
          instance002:
            iproto:
              listen:
              - uri: '127.0.0.1:3302'

  1. After configuring a replica set, execute the tt start command from the tt environment directory:

    $ tt start manual_leader
       • Starting an instance [manual_leader:instance001]...
       • Starting an instance [manual_leader:instance002]...
    
  2. Check that instances are in the RUNNING status using the tt status command:

    $ tt status manual_leader
    INSTANCE                      STATUS      PID
    manual_leader:instance001     RUNNING     15272
    manual_leader:instance002     RUNNING     15273
    

  1. Connect to instance001 using tt connect:

    $ tt connect manual_leader:instance001
       • Connecting to the instance...
       • Connected to manual_leader:instance001
    
  2. Make sure that the instance is in the running state by executing box.info.status:

    manual_leader:instance001> box.info.status
    ---
    - running
    ...
    
  3. Check that the instance is writable using box.info.ro:

    manual_leader:instance001> box.info.ro
    ---
    - false
    ...
    
  4. Execute box.info.replication to check a replica set status. For instance002, upstream.status and downstream.status should be follow.

    manual_leader:instance001> box.info.replication
    ---
    - 1:
        id: 1
        uuid: 9bb111c2-3ff5-36a7-00f4-2b9a573ea660
        lsn: 7
        name: instance001
      2:
        id: 2
        uuid: 4cfa6e3c-625e-b027-00a7-29b2f2182f23
        lsn: 0
        upstream:
          status: follow
          idle: 0.3893879999996
          peer: replicator@127.0.0.1:3302
          lag: 0.00028800964355469
        name: instance002
        downstream:
          status: follow
          idle: 0.37777199999982
          vclock: {1: 7}
          lag: 0
    ...
    

    To see the diagrams that illustrate how the upstream and downstream connections look, refer to Monitoring a replica set.

To check that a replica (instance002) gets all updates from the master, follow the steps below:

  1. On instance001, create a space and add data as described in CRUD operation examples.

  2. Open the second terminal, connect to instance002 using tt connect, and use the select operation to make sure data is replicated.

  3. Check that box.info.vclock values are the same on both instances:

    • instance001:

      manual_leader:instance001> box.info.vclock
      ---
      - {1: 21}
      ...
      
    • instance002:

      manual_leader:instance002> box.info.vclock
      ---
      - {1: 21}
      ...
      

    Note

    Note that a vclock value might include the 0 component that is related to local space operations and might differ for different instances in a replica set.

This section describes how to add a new replica to a replica set.

  1. Add instance003 to the instances.yml file:

    instance001:
    instance002:
    instance003:
    
  2. Add instance003 with the specified iproto.listen option to the config.yaml file:

    groups:
      group001:
        replicasets:
          replicaset001:
            leader: instance001
            instances:
              instance001:
                iproto:
                  listen:
                  - uri: '127.0.0.1:3301'
              instance002:
                iproto:
                  listen:
                  - uri: '127.0.0.1:3302'
              instance003:
                iproto:
                  listen:
                  - uri: '127.0.0.1:3303'
    

  1. Open the third terminal to work with a new instance. Start instance003 using tt start:

    $ tt start manual_leader:instance003
       • Starting an instance [manual_leader:instance003]...
    
  2. Check a replica set status using tt status:

    $ tt status manual_leader
    INSTANCE                      STATUS      PID
    manual_leader:instance001     RUNNING     15272
    manual_leader:instance002     RUNNING     15273
    manual_leader:instance003     RUNNING     15551
    

After you added instance003 to the configuration and started it, you need to reload configurations on all instances. This is required to allow instance001 and instance002 to get data from the new instance in case it becomes a master.

  1. Connect to instance003 using tt connect:

    $ tt connect manual_leader:instance003
       • Connecting to the instance...
       • Connected to manual_leader:instance001
    
  2. Reload configurations on all three instances using the reload() function provided by the config module:

    • instance001:

      manual_leader:instance001> require('config'):reload()
      ---
      ...
      
    • instance002:

      manual_leader:instance002> require('config'):reload()
      ---
      ...
      
    • instance003:

      manual_leader:instance003> require('config'):reload()
      ---
      ...
      
  3. Execute box.info.replication to check a replica set status. Make sure that upstream.status and downstream.status are follow for instance003.

    manual_leader:instance001> box.info.replication
    ---
    - 1:
        id: 1
        uuid: 9bb111c2-3ff5-36a7-00f4-2b9a573ea660
        lsn: 21
        name: instance001
      2:
        id: 2
        uuid: 4cfa6e3c-625e-b027-00a7-29b2f2182f23
        lsn: 0
        upstream:
          status: follow
          idle: 0.052655000000414
          peer: replicator@127.0.0.1:3302
          lag: 0.00010204315185547
        name: instance002
        downstream:
          status: follow
          idle: 0.09503500000028
          vclock: {1: 21}
          lag: 0.00026917457580566
      3:
        id: 3
        uuid: 9a3a1b9b-8a18-baf6-00b3-a6e5e11fd8b6
        lsn: 0
        upstream:
          status: follow
          idle: 0.77522099999987
          peer: replicator@127.0.0.1:3303
          lag: 0.0001838207244873
        name: instance003
        downstream:
          status: follow
          idle: 0.33186100000012
          vclock: {1: 21}
          lag: 0
            ...
    

This section shows how to perform manual failover and change a replica set leader.

  1. In the config.yaml file, change the replica set leader from instance001 to null:

    replicaset001:
      leader: null
    
  2. Reload configurations on all three instances using config:reload() and check that instances are in read-only mode. The example below shows how to do this for instance001:

    manual_leader:instance001> require('config'):reload()
    ---
    ...
    manual_leader:instance001> box.info.ro
    ---
    - true
    ...
    manual_leader:instance001> box.info.ro_reason
    ---
    - config
    ...
    
  3. Make sure that box.info.vclock values are the same on all instances:

    • instance001:

      manual_leader:instance001> box.info.vclock
      ---
      - {1: 21}
      ...
      
    • instance002:

      manual_leader:instance002> box.info.vclock
      ---
      - {1: 21}
      ...
      
    • instance003:

      manual_leader:instance003> box.info.vclock
      ---
      - {1: 21}
      ...
      

  1. Change a replica set leader in config.yaml to instance002:

    replicaset001:
      leader: instance002
    
  2. Reload configuration on all instances using config:reload().

  3. Make sure that instance002 is a new master:

    manual_leader:instance002> box.info.ro
    ---
    - false
    ...
    
  4. Check replication status using box.info.replication.

This section describes the process of removing an instance from a replica set.

Before removing an instance, make sure it is in read-only mode. If the instance is a master, perform manual failover.

  1. Clear the iproto option for instance003 by setting its value to {}:

    instance003:
      iproto: {}
    
  2. Reload configurations on instance001 and instance002:

    • instance001:

      manual_leader:instance001> require('config'):reload()
      ---
      ...
      
    • instance002:

      manual_leader:instance002> require('config'):reload()
      ---
      ...
      
  3. Check that the upstream section is missing for instance003 by executing box.info.replication[3]:

    manual_leader:instance001> box.info.replication[3]
    ---
    - id: 3
      uuid: 9a3a1b9b-8a18-baf6-00b3-a6e5e11fd8b6
      lsn: 0
      downstream:
        status: follow
        idle: 0.4588760000006
        vclock: {1: 21}
        lag: 0
      name: instance003
    ...
    

  1. Stop instance003 using the tt stop command:

    $ tt stop manual_leader:instance003
       • The Instance manual_leader:instance003 (PID = 15551) has been terminated.
    
  2. Check that downstream.status is stopped for instance003:

    manual_leader:instance001> box.info.replication[3]
    ---
    - id: 3
      uuid: 9a3a1b9b-8a18-baf6-00b3-a6e5e11fd8b6
      lsn: 0
      downstream:
        status: stopped
        message: 'unexpected EOF when reading from socket, called on fd 27, aka 127.0.0.1:3301,
          peer of 127.0.0.1:54185: Broken pipe'
        system_message: Broken pipe
      name: instance003
    ...
    

  1. Remove instance003 from the instances.yml file:

    instance001:
    instance002:
    
  2. Remove instance003 from config.yaml:

    instances:
      instance001:
        iproto:
          listen:
          - uri: '127.0.0.1:3301'
      instance002:
        iproto:
          listen:
          - uri: '127.0.0.1:3302'
    
  3. Reload configurations on instance001 and instance002:

    • instance001:

      manual_leader:instance001> require('config'):reload()
      ---
      ...
      
    • instance002:

      manual_leader:instance002> require('config'):reload()
      ---
      ...
      

To remove an instance from the replica set permanently, it should be removed from the box.space._cluster system space:

  1. Select all the tuples in the box.space._cluster system space:

    manual_leader:instance002> box.space._cluster:select{}
    ---
    - - [1, '9bb111c2-3ff5-36a7-00f4-2b9a573ea660', 'instance001']
      - [2, '4cfa6e3c-625e-b027-00a7-29b2f2182f23', 'instance002']
      - [3, '9a3a1b9b-8a18-baf6-00b3-a6e5e11fd8b6', 'instance003']
    ...
    
  2. Delete a tuple corresponding to instance003:

    manual_leader:instance002> box.space._cluster:delete(3)
    ---
    - [3, '9a3a1b9b-8a18-baf6-00b3-a6e5e11fd8b6', 'instance003']
    ...
    
  3. Execute box.info.replication to check the health status:

    manual_leader:instance002> box.info.replication
    ---
    - 1:
        id: 1
        uuid: 9bb111c2-3ff5-36a7-00f4-2b9a573ea660
        lsn: 21
        upstream:
          status: follow
          idle: 0.73316000000159
          peer: replicator@127.0.0.1:3301
          lag: 0.00016212463378906
        name: instance001
        downstream:
          status: follow
          idle: 0.7269320000014
          vclock: {2: 1, 1: 21}
          lag: 0.00083398818969727
      2:
        id: 2
        uuid: 4cfa6e3c-625e-b027-00a7-29b2f2182f23
        lsn: 1
        name: instance002
    ...
    
Found what you were looking for?
Feedback