Версия:

Руководство пользователя / Replication / Настройка набора реплик
Руководство пользователя / Replication / Настройка набора реплик

Настройка набора реплик

Настройка набора реплик

Настройка репликации мастер-реплика

Let us first bootstrap a simple master-replica set containing two instances, each located on its own machine. For easier administration, we make the instance files almost identical.

../../../_images/mr-1m-1r-twoway.png

Ниже пример файла экземпляра для мастера:

-- файл экземпляра для мастера
           box.cfg{
             listen = 3301,
             replication = {'replicator:password@192.168.0.101:3301',  -- URI мастера
                            'replicator:password@192.168.0.102:3301'}, -- URI реплики
             read_only = false
           }
           box.once("schema", function()
              box.schema.user.create('replicator', {password = 'password'})
              box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
              box.schema.space.create("test")
              box.space.test:create_index("primary")
              print('box.once executed on master')
           end)

где:

  • the box.cfg() listen parameter defines a URI (port 3301 in our example), on which the master can accept connections from replicas.

  • the box.cfg() replication parameter defines the URIs at which all instances in the replica set can accept connections. It includes the replica’s URI as well, although the replica is not a replication source right now.

    Примечание

    For security reasons, we recommend that administrators prevent unauthorized replication sources by associating a password with every user that has a replication role. That way, the URI for replication parameter must have the long form username:password@host:port.

  • the read_only = false parameter setting enables data-change operations on the instance and makes the instance act as a master, not as a replica. That is the only parameter setting in our instance files that will differ.

  • the box.once() function contains database initialization logic that should be executed only once during the replica set lifetime.

In this example, we create a space with a primary index, and a user for replication purposes. We also say print('box.once executed on master') so that it will later be visible on a console whether box.once() was executed.

Примечание

Репликация требует настройки прав. Права на доступ к спейсам можно задать напрямую для пользователя, под чьим именем запущен экземпляр. Но обычно права на доступ к спейсам задаются с помощью роли, которая затем присваивается пользователю, под чьим именем запущена реплика.

Here we use Tarantool’s predefined role named «replication» which by default grants «read» privileges for all database objects («universe»), and we can change privileges for this role as required.

In the replica’s instance file, we set the read_only parameter to «true», and say print('box.once executed on replica') so that later it will be visible that box.once() was not executed more than once. Otherwise the replica’s instance file is identical to the master’s instance file.

-- файл экземпляра для реплики
          box.cfg{
            listen = 3301,
            replication = {'replicator:password@192.168.0.101:3301',  -- URI мастера
                           'replicator:password@192.168.0.102:3301'}, -- URI реплики
            read_only = true
          }
          box.once("schema", function()
             box.schema.user.create('replicator', {password = 'password'})
             box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
             box.schema.space.create("test")
             box.space.test:create_index("primary")
             print('box.once executed on replica')
          end)

Примечание

The replica does not inherit the master’s configuration parameters, such as those making the checkpoint daemon run on the master. To get the same behavior, set the relevant parameters explicitly so that they are the same on both master and replica.

Теперь можно запустить два экземпляра. Мастер…

$ # запуск мастера
          $ tarantool master.lua
          2017-06-14 14:12:03.847 [18933] main/101/master.lua C> version 1.7.4-52-g980d30092
          2017-06-14 14:12:03.848 [18933] main/101/master.lua C> log level 5
          2017-06-14 14:12:03.849 [18933] main/101/master.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 14:12:03.859 [18933] iproto/101/main I> binary: bound to [::]:3301
          2017-06-14 14:12:03.861 [18933] main/105/applier/replicator@192.168.0. I> can't connect to master
          2017-06-14 14:12:03.861 [18933] main/105/applier/replicator@192.168.0. coio.cc:105 !> SystemError connect, called on fd 14, aka 192.168.0.102:56736: Connection refused
          2017-06-14 14:12:03.861 [18933] main/105/applier/replicator@192.168.0. I> will retry every 1 second
          2017-06-14 14:12:03.861 [18933] main/104/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 14:12:19.878 [18933] main/105/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.102:3301
          2017-06-14 14:12:19.879 [18933] main/101/master.lua I> initializing an empty data directory
          2017-06-14 14:12:19.908 [18933] snapshot/101/main I> saving snapshot `/var/lib/tarantool/master/00000000000000000000.snap.inprogress'
          2017-06-14 14:12:19.914 [18933] snapshot/101/main I> done
          2017-06-14 14:12:19.914 [18933] main/101/master.lua I> vinyl checkpoint done
          2017-06-14 14:12:19.917 [18933] main/101/master.lua I> ready to accept requests
          2017-06-14 14:12:19.918 [18933] main/105/applier/replicator@192.168.0. I> failed to authenticate
          2017-06-14 14:12:19.918 [18933] main/105/applier/replicator@192.168.0. xrow.cc:431 E> ER_LOADING: Instance bootstrap hasn't finished yet
          box.once executed on master
          2017-06-14 14:12:19.920 [18933] main C> entering the event loop

… (the display confirms that box.once() was executed on the master) – and the replica:

$ # запуск реплики
          $ tarantool replica.lua
          2017-06-14 14:12:19.486 [18934] main/101/replica.lua C> version 1.7.4-52-g980d30092
          2017-06-14 14:12:19.486 [18934] main/101/replica.lua C> log level 5
          2017-06-14 14:12:19.487 [18934] main/101/replica.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 14:12:19.494 [18934] iproto/101/main I> binary: bound to [::]:3311
          2017-06-14 14:12:19.495 [18934] main/104/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 14:12:19.495 [18934] main/105/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.102:3302
          2017-06-14 14:12:19.496 [18934] main/104/applier/replicator@192.168.0. I> failed to authenticate
          2017-06-14 14:12:19.496 [18934] main/104/applier/replicator@192.168.0. xrow.cc:431 E> ER_LOADING: Instance bootstrap hasn't finished yet

In both logs, there are messages saying that the replica was bootstrapped from the master:

$ # настройка реплики (из журнала мастера)
          <...>
          2017-06-14 14:12:20.503 [18933] main/106/main I> initial data sent.
          2017-06-14 14:12:20.505 [18933] relay/[::ffff:192.168.0.101]:/101/main I> recover from `/var/lib/tarantool/master/00000000000000000000.xlog'
          2017-06-14 14:12:20.505 [18933] main/106/main I> final data sent.
          2017-06-14 14:12:20.522 [18933] relay/[::ffff:192.168.0.101]:/101/main I> recover from `/Users/e.shebunyaeva/work/tarantool-test-repl/master_dir/00000000000000000000.xlog'
          2017-06-14 14:12:20.922 [18933] main/105/applier/replicator@192.168.0. I> authenticated
$ # настройка реплики (из журнала реплики)
          <...>
          2017-06-14 14:12:20.498 [18934] main/104/applier/replicator@192.168.0. I> authenticated
          2017-06-14 14:12:20.498 [18934] main/101/replica.lua I> bootstrapping replica from 192.168.0.101:3301
          2017-06-14 14:12:20.512 [18934] main/104/applier/replicator@192.168.0. I> initial data received
          2017-06-14 14:12:20.512 [18934] main/104/applier/replicator@192.168.0. I> final data received
          2017-06-14 14:12:20.517 [18934] snapshot/101/main I> saving snapshot `/var/lib/tarantool/replica/00000000000000000005.snap.inprogress'
          2017-06-14 14:12:20.518 [18934] snapshot/101/main I> done
          2017-06-14 14:12:20.519 [18934] main/101/replica.lua I> vinyl checkpoint done
          2017-06-14 14:12:20.520 [18934] main/101/replica.lua I> ready to accept requests
          2017-06-14 14:12:20.520 [18934] main/101/replica.lua I> set 'read_only' configuration option to true
          2017-06-14 14:12:20.520 [18934] main C> entering the event loop

Обратите внимание, что функция box.once() была выполнена только на мастере, хотя мы добавили box.once() в оба файла экземпляра.

Также можно было сначала запустить реплику.

$ # запуск реплики
          $ tarantool replica.lua
          2017-06-14 14:35:36.763 [18952] main/101/replica.lua C> version 1.7.4-52-g980d30092
          2017-06-14 14:35:36.765 [18952] main/101/replica.lua C> log level 5
          2017-06-14 14:35:36.765 [18952] main/101/replica.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 14:35:36.772 [18952] iproto/101/main I> binary: bound to [::]:3301
          2017-06-14 14:35:36.772 [18952] main/104/applier/replicator@192.168.0. I> can't connect to master
          2017-06-14 14:35:36.772 [18952] main/104/applier/replicator@192.168.0. coio.cc:105 !> SystemError connect, called on fd 13, aka 192.168.0.101:56820: Connection refused
          2017-06-14 14:35:36.772 [18952] main/104/applier/replicator@192.168.0. I> will retry every 1 second
          2017-06-14 14:35:36.772 [18952] main/105/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.102:3301

… а затем уже мастера:

$ # запуск мастера
          $ tarantool master.lua
          2017-06-14 14:35:43.701 [18953] main/101/master.lua C> version 1.7.4-52-g980d30092
          2017-06-14 14:35:43.702 [18953] main/101/master.lua C> log level 5
          2017-06-14 14:35:43.702 [18953] main/101/master.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 14:35:43.709 [18953] iproto/101/main I> binary: bound to [::]:3301
          2017-06-14 14:35:43.709 [18953] main/105/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.102:3301
          2017-06-14 14:35:43.709 [18953] main/104/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 14:35:43.709 [18953] main/101/master.lua I> initializing an empty data directory
          2017-06-14 14:35:43.721 [18953] snapshot/101/main I> saving snapshot `/var/lib/tarantool/master/00000000000000000000.snap.inprogress'
          2017-06-14 14:35:43.722 [18953] snapshot/101/main I> done
          2017-06-14 14:35:43.723 [18953] main/101/master.lua I> vinyl checkpoint done
          2017-06-14 14:35:43.723 [18953] main/101/master.lua I> ready to accept requests
          2017-06-14 14:35:43.724 [18953] main/105/applier/replicator@192.168.0. I> failed to authenticate
          2017-06-14 14:35:43.724 [18953] main/105/applier/replicator@192.168.0. xrow.cc:431 E> ER_LOADING: Instance bootstrap hasn't finished yet
          box.once executed on master
          2017-06-14 14:35:43.726 [18953] main C> entering the event loop
          2017-06-14 14:35:43.779 [18953] main/103/main I> initial data sent.
          2017-06-14 14:35:43.780 [18953] relay/[::ffff:192.168.0.101]:/101/main I> recover from `/var/lib/tarantool/master/00000000000000000000.xlog'
          2017-06-14 14:35:43.780 [18953] main/103/main I> final data sent.
          2017-06-14 14:35:43.796 [18953] relay/[::ffff:192.168.0.102]:/101/main I> recover from `/var/lib/tarantool/master/00000000000000000000.xlog'
          2017-06-14 14:35:44.726 [18953] main/105/applier/replicator@192.168.0. I> authenticated

В данном случае реплика ожидает доступности мастера, поэтому порядок запуска не имеет значения. Наша функция box.once() также будет выполняться однократно, только на мастере.

$ # реплика в итоге подключена к мастеру
          $ # и получила настройки (из журнала реплики)
          2017-06-14 14:35:43.777 [18952] main/104/applier/replicator@192.168.0. I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 14:35:43.777 [18952] main/104/applier/replicator@192.168.0. I> authenticated
          2017-06-14 14:35:43.777 [18952] main/101/replica.lua I> bootstrapping replica from 192.168.0.199:3310
          2017-06-14 14:35:43.788 [18952] main/104/applier/replicator@192.168.0. I> initial data received
          2017-06-14 14:35:43.789 [18952] main/104/applier/replicator@192.168.0. I> final data received
          2017-06-14 14:35:43.793 [18952] snapshot/101/main I> saving snapshot `/var/lib/tarantool/replica/00000000000000000005.snap.inprogress'
          2017-06-14 14:35:43.793 [18952] snapshot/101/main I> done
          2017-06-14 14:35:43.795 [18952] main/101/replica.lua I> vinyl checkpoint done
          2017-06-14 14:35:43.795 [18952] main/101/replica.lua I> ready to accept requests
          2017-06-14 14:35:43.795 [18952] main/101/replica.lua I> set 'read_only' configuration option to true
          2017-06-14 14:35:43.795 [18952] main C> entering the event loop

Контролируемое восстановление после сбоя

To perform a controlled failover, that is, swap the roles of the master and replica, all we need to do is to set read_only=true at the master, and read_only=false at the replica. The order of actions is important here. If a system is running in production, we do not want concurrent writes happening both at the replica and the master. Nor do we want the new replica to accept any writes until it has finished fetching all replication data from the old master. To compare replica and master state, we can use box.info.signature.

  1. Настройте read_only=true на мастере.

    # на мастере
              tarantool> box.cfg{read_only=true}
    
  2. Зарегистрируйте текущее состояние мастера с помощью box.info.signature, которое содержит общее количество всех LSN в векторных часах мастера.

    # на мастере
                tarantool> box.info.signature
    
  3. Подождите, пока сигнатура реплики не совпадет с сигнатурой мастера.

    # на реплике
                tarantool> box.info.signature
    
  4. Настройте read_only=false` на реплике, чтобы запустить операции записи данных.

    # на реплике
              tarantool> box.cfg{read_only=false}
    

These four steps ensure that the replica doesn’t accept new writes until it’s done fetching writes from the master.

Настройка репликации мастер-мастер

Now let us bootstrap a two-instance master-master set. For easier administration, we make master#1 and master#2 instance files fully identical.

../../../_images/mm-2m-mesh.png

Переиспользуем файл экземпляра для мастера из вышеописанного примера мастер-реплика.

-- файл экземпляра для любого из двух мастеров
          box.cfg{
            listen      = 3301,
            replication = {'replicator:password@192.168.0.101:3301',  -- URI мастера 1
                           'replicator:password@192.168.0.102:3301'}, -- URI мастера 2
            read_only   = false
          }
          box.once("schema", function()
             box.schema.user.create('replicator', {password = 'password'})
             box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
             box.schema.space.create("test")
             box.space.test:create_index("primary")
             print('box.once executed on master #1')
          end)

In the replication parameter, we define the URIs of both masters in the replica set and say print('box.once executed on master #1') so it will be clear when and where the box.once() logic is executed.

Now we can launch the two masters. Again, the launch order doesn’t matter. The box.once() logic will also be executed only once, at the master which is elected as the replica set leader at bootstrap.

$ # запуск мастера №1
          $ tarantool master1.lua
          2017-06-14 15:39:03.062 [47021] main/101/master1.lua C> version 1.7.4-52-g980d30092
          2017-06-14 15:39:03.062 [47021] main/101/master1.lua C> log level 5
          2017-06-14 15:39:03.063 [47021] main/101/master1.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 15:39:03.065 [47021] iproto/101/main I> binary: bound to [::]:3301
          2017-06-14 15:39:03.065 [47021] main/105/applier/replicator@192.168.0.10 I> can't connect to master
          2017-06-14 15:39:03.065 [47021] main/105/applier/replicator@192.168.0.10 coio.cc:107 !> SystemError connect, called on fd 14, aka 192.168.0.102:57110: Connection refused
          2017-06-14 15:39:03.065 [47021] main/105/applier/replicator@192.168.0.10 I> will retry every 1 second
          2017-06-14 15:39:03.065 [47021] main/104/applier/replicator@192.168.0.10 I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 15:39:08.070 [47021] main/105/applier/replicator@192.168.0.10 I> remote master is 1.7.4 at 192.168.0.102:3301
          2017-06-14 15:39:08.071 [47021] main/105/applier/replicator@192.168.0.10 I> authenticated
          2017-06-14 15:39:08.071 [47021] main/101/master1.lua I> bootstrapping replica from 192.168.0.102:3301
          2017-06-14 15:39:08.073 [47021] main/105/applier/replicator@192.168.0.10 I> initial data received
          2017-06-14 15:39:08.074 [47021] main/105/applier/replicator@192.168.0.10 I> final data received
          2017-06-14 15:39:08.074 [47021] snapshot/101/main I> saving snapshot `/Users/e.shebunyaeva/work/tarantool-test-repl/master1_dir/00000000000000000008.snap.inprogress'
          2017-06-14 15:39:08.074 [47021] snapshot/101/main I> done
          2017-06-14 15:39:08.076 [47021] main/101/master1.lua I> vinyl checkpoint done
          2017-06-14 15:39:08.076 [47021] main/101/master1.lua I> ready to accept requests
          box.once executed on master #1
          2017-06-14 15:39:08.077 [47021] main C> entering the event loop
$ # запуск мастера №2
          $ tarantool master2.lua
          2017-06-14 15:39:07.452 [47022] main/101/master2.lua C> version 1.7.4-52-g980d30092
          2017-06-14 15:39:07.453 [47022] main/101/master2.lua C> log level 5
          2017-06-14 15:39:07.453 [47022] main/101/master2.lua I> mapping 268435456 bytes for tuple arena...
          2017-06-14 15:39:07.455 [47022] iproto/101/main I> binary: bound to [::]:3301
          2017-06-14 15:39:07.455 [47022] main/104/applier/replicator@192.168.0.19 I> remote master is 1.7.4 at 192.168.0.101:3301
          2017-06-14 15:39:07.455 [47022] main/105/applier/replicator@192.168.0.10 I> remote master is 1.7.4 at 192.168.0.102:3301
          2017-06-14 15:39:07.455 [47022] main/101/master2.lua I> initializing an empty data directory
          2017-06-14 15:39:07.457 [47022] snapshot/101/main I> saving snapshot `/Users/e.shebunyaeva/work/tarantool-test-repl/master2_dir/00000000000000000000.snap.inprogress'
          2017-06-14 15:39:07.457 [47022] snapshot/101/main I> done
          2017-06-14 15:39:07.458 [47022] main/101/master2.lua I> vinyl checkpoint done
          2017-06-14 15:39:07.459 [47022] main/101/master2.lua I> ready to accept requests
          2017-06-14 15:39:07.460 [47022] main C> entering the event loop
          2017-06-14 15:39:08.072 [47022] main/103/main I> initial data sent.
          2017-06-14 15:39:08.073 [47022] relay/[::ffff:192.168.0.102]:/101/main I> recover from `/Users/e.shebunyaeva/work/tarantool-test-repl/master2_dir/00000000000000000000.xlog'
          2017-06-14 15:39:08.073 [47022] main/103/main I> final data sent.
          2017-06-14 15:39:08.077 [47022] relay/[::ffff:192.168.0.102]:/101/main I> recover from `/Users/e.shebunyaeva/work/tarantool-test-repl/master2_dir/00000000000000000000.xlog'
          2017-06-14 15:39:08.461 [47022] main/104/applier/replicator@192.168.0.10 I> authenticated