Chapter 20 InnoDB Cluster

InnoDB cluster uses Group Replication and therefore your server instances must meet the same requirements. See Section 17.7.1, “Group Replication Requirements”.

In addition, the provisioning scripts that MySQL Shell uses to configure servers for use in InnoDB cluster require access to Python (2.7 and above). On Windows MySQL Shell includes Python and no user configuration is required. On Unix Python must be found as part of the enviroment. To check that your system has Python configured correctly issue:

$ /usr/bin/env python

If a Python interpreter starts, no further action is required. If the previous command fails, create a soft link between /usr/bin/python and your chosen Python binary.

The following methods of installing InnoDB cluster are available:

MySQL Shell includes the AdminAPI, which is accessed through the dba global variable and its associated methods. The dba variable's methods enable you to administer your cluster, for example you use the dba.deploySandboxInstance() method to add a sandbox MySQL instance.

To list all available dba commands, use the dba.help() method. You can obtain detailed information for a specific method using the general format object.help('methodname'). For example:

mysql-js> dba.help('getCluster')

Retrieves a cluster from the Metadata Store.

SYNTAX
  <Dba>.getCluster([name])

WHERE
  name: Parameter to specify the name of the cluster to be returned.

DESCRIPTION

If name is not specified, the default cluster will be returned.

If name is specified, and no cluster with the indicated name is found, an error
will be raised.

In addition to this documentation, there is developer documentation for all dba methods in the JavaScript and Python developer documentation.

Initially deploying and using local sandbox instances of MySQL is a good way to start your exploration of InnoDB cluster. You can fully test out InnoDB cluster locally, prior to deployment on your production servers. MySQL Shell has built in functionality for creating sandbox instances. MySQL Shell creates the sandbox instances correctly configured to work with Group Replication in a locally deployed clustered scenario.

The dba module provides several functions for administration of sandbox instances. For this example setup, you create three sandbox instances. The AdminAPI provides a function for that purpose: dba.deploySandboxInstance().

Start MySQL Shell from a command prompt by issuing the command:

shell> mysqlsh

MySQL Shell provides two scripting languages: JavaScript and Python. Throughout this guide MySQL Shell is used primarily in JavaScript mode . When MySQL Shell starts it is in JavaScript mode by default. You switch into JavaScript mode, Python mode and SQL mode using the commands \js, \py, and \sql. Ensure you are in JavaScript mode by issuing the \js command, then execute:

mysql-js> dba.deploySandboxInstance(3310)

The argument passed to deploySandboxInstance() is the TCP port number where the MySQL Server instance listens for connections. By default the sandbox is created in a directory named $HOME/mysql-sandboxes/port on Unix systems. For Microsoft Windows systems the directory is %userprofile%\MySQL\mysql-sandboxes\port.

The root password for the instance is prompted for.

To add further server instances, use deploySandboxInstance(). For this example sandbox cluster add two more instances using different port numbers by issuing:

mysql-js> dba.deploySandboxInstance(3320)
mysql-js> dba.deploySandboxInstance(3330)

You now have three MySQL server sandbox instances running on ports 3310, 3320 and 3330.

The next step is to create the InnoDB cluster while connected to the seed MySQL Server instance. The seed instance is the instance that you are connected to via MySQL Shell and that you want to replicate to the other instances. In this example, the sandbox instances are blank instances, therefore we can choose any instance. In a production set up the seed instance would be the one which contains your existing data set and would be replicated to the other instances in the cluster.

Connect MySQL Shell to the seed instance, in this case the one at port 3310:

mysql-js> \connect root@localhost:3310

The syntax \connect is a shortcut for the MySQL Shell connect method shell.connect(). Alternatively use the following command:

mysql-js> shell.connect('root@localhost:3310')

Use the createCluster() method to create the InnoDB cluster with the currently connected instance as the seed:

mysql-js> var cluster = dba.createCluster('testCluster')

The createCluster() method returns the created cluster, the above statement assigns this to the cluster variable. The parameter passed to the createCluster() method is a symbolic name given to this InnoDB cluster, in this case testCluster. The resulting InnoDB cluster is assigned to the cluster variable. This function deploys the metadata to the selected instance, configures it for Group Replication and adds the instance as the seed of the new InnoDB cluster.

After validating that the instance is properly configured, it is added to the InnoDB cluster as the seed instance and the replication subsystem is started.

The provided sandbox instances are pre-configured to work with Group Replication, but if you use a pre-existing instance, it is possible that some configuration options might not be set in a compatible way. The createCluster() command ensures that the settings are correct and if not, it changes their values. If a change requires MySQL Server to be restarted, you are prompted to restart it manually whenever convenient.

In summary, when dba.createCluster() is executed, the following steps are carried out:

Once you have created a cluster, obtain the cluster instance variable using a command such as:

mysql-js> var cluster = dba.getCluster("testCluster")

You specify the name of the cluster you wish to obtain the instance variable for. If you do not specify the name of the cluster the default cluster is returned.

The next step is to add secondary instances to the InnoDB cluster. Any transactions that were executed by the seed instance are re-executed by each secondary instance as it is added. We use the sandbox instances that were created earlier.

The seed instance in this example was recently created, so it is nearly empty. Therefore, there is little data that needs to be replicated from the seed instance to the secondary instances. In a production environment, where you have an existing database on the seed instance, you could use a tool such as MySQL Enterprise Backup to ensure that the secondaries have matching data before replication starts. This avoids the possibility of lengthy delays while data replicates from the primary to the secondaries. Once the cluster is formed, writes to the primary result in data being replicated to the secondaries.

Add the second instance to the InnoDB cluster:

mysql-js> cluster.addInstance('root@localhost:3320')

The root user's password is prompted for.

Add the third instance:

mysql-js> cluster.addInstance('root@localhost:3330')

The root user's password is prompted for.

At this point you have created a cluster with three instances: a primary, and two secondaries.

mysql-js> cluster.addInstance('root@localhost:3330', {verbose: true})

You can only specify localhost in addInstance() if the instance is a sandbox instance. This also applies to the implicit addInstance() after issuing createCluster().

In order for client applications to handle failover, they need to be aware of the InnoDB cluster topology. They also need to know whether an instance is the PRIMARY in single-primary mode, or is "R/W" in multi-primary mode. While it is possible for applications to implement that logic, MySQL Router provides this functionality for you and is designed for InnoDB cluster.

The recommended deployment of MySQL Router is on the same host as the application. In this tutorial, everything is running on a single host, so you deploy MySQL Router to the same host.

Assuming MySQL Router is already installed (see Installation), the only required step is to bootstrap it with the location of the InnoDB cluster metadata server. The following does this using all default settings:

shell> mysqlrouter --bootstrap root@localhost:3310 --user=mysqlrouter 

You are prompted for the instance password and encryption key for MySQL Router to use. This encryption key is used to encrypt the instance password used by MySQL Router to connect to the cluster. The ports you can use to connect to the InnoDB cluster are also displayed.

MySQL Router connects to the InnoDB cluster, fetches its metadata and configures itself for use. The generated configuration creates two TCP ports: one for read-write sessions (which redirects connections to "R/W" instances) and one for read-only sessions (which redirects connections to one of the SECONDARY instances).

Once bootstrapped and configured, start MySQL Router (or set up a service for it to start automatically when the system boots):

shell> mysqlrouter &

You can now connect a MySQL client, such as MySQL Shell to one of the incoming MySQL Router ports and see how the client gets transparently connected to one of the InnoDB cluster instances. To see which instance you are actually connected to, simply query the port status variable.

shell> mysqlsh --uri root@localhost:6442
mysql-js> \sql
Switching to SQL mode... Commands end with ;
mysql-sql> select @@port;
+--------+
| @@port |
+--------+
|   3310 |
+--------+
1 row in set (0.00 sec)

To test if failover works, simulate an unexpected halt by killing the PRIMARY instance using the dba.killSandboxInstance() function and check that one of the other instances takes over automatically.

mysql-js> dba.killSandboxInstance(3310)

Then you can again check which instance you are connected to. The first SELECT statement fails as the connection to the original PRIMARY was lost. MySQL Shell automatically reconnects for you and when you issue the command again the new port is confirmed.

mysql-js> \sql
Switching to SQL mode... Commands end with ;
mysql-sql> SELECT @@port;
ERROR: 2013 (HY000): Lost connection to MySQL server during query
The global session got disconnected.
Attempting to reconnect to 'root@localhost:6446'...
The global session was successfully reconnected.
mysql-sql> select @@port;
+--------+
| @@port |
+--------+
|   3330 |
+--------+
1 row in set (0.00 sec)

This shows that the InnoDB cluster provided us with automatic failover, that MySQL Router has automatically reconnected us to the new PRIMARY instance, and that we have high availability.

You can bring the instance that you killed back online.

mysql-js> dba.startSandboxInstance(3310)
mysql-js> cluster.rejoinInstance('root@localhost:3310')
mysql-js> cluster.status()

Use the Cluster object's status() method to check a cluster's status:

mysql-js> cluster.status()

This retrieves the current InnoDB cluster status which the server instance you are connected to is aware of and outputs a status report. It is important to understand that the instance's state in the cluster influences the information in the status report. A member which has left the cluster would provide a different view of the cluster compared to a instance which belongs to the cluster.

The instance status is one of the following:

The mode indicates either R/W (read and writable) or R/O (read only). In single-primary mode, only the instance marked "R/W" can execute transactions that update the database, so it is the PRIMARY. If that instance becomes unreachable for any reason (like an unexpected halt), one of the remaining "R/O" instances automatically takes over its place and becomes the new "R/W" PRIMARY. In multi-primary mode, multiple instances are marked as "R/W" and there is no elected PRIMARY.

To check the status of the InnoDB cluster at a later time, you can get a reference to the InnoDB cluster object by connecting to any of its instances. However, if you want to make changes to the InnoDB cluster, you must connect to a "R/W" instance. For information about how the InnoDB cluster is running, use the status() method:

mysql-js> var cluster = dba.getCluster()
mysql-js> cluster.status()
{
    "clusterName": "test",
    "defaultReplicaSet": {
        "status": "Cluster tolerant to up to ONE failure.",
        "topology": {
            "localhost:3310": {
                "address": "localhost:3310",
                "status": "ONLINE",
                "role": "HA",
                "mode": "R/W",
                "leaves": {
                    "localhost:3320": {
                        "address": "localhost:3320",
                        "status": "ONLINE",
                        "role": "HA",
                        "mode": "R/O",
                        "leaves": {}
                    },
                    "localhost:3330": {
                        "address": "localhost:3330",
                        "status": "ONLINE",
                        "role": "HA",
                        "mode": "R/O",
                        "leaves": {}
                    }
                }
            }
        }
    }
}

As the above output demonstrates, status information includes the InnoDB cluster name, topology, PRIMARY, and more.

To get information about the structure of the InnoDB cluster itself, use the cluster.describe() function:

mysql-js> cluster.describe();
{
    "clusterName": "test",
    "adminType": "local",
    "defaultReplicaSet": {
        "name": "default",
        "instances": [
            {
                "name": "localhost:3310",
                "host": "localhost:3310",
                "role": "HA"
            },
            {
                "name": "localhost:3320",
                "host": "localhost:3320",
                "role": "HA"
            },
            {
                "name": "localhost:3330",
                "host": "localhost:3330",
                "role": "HA"
            }
        ]
    }
}

The output from this function shows the structure of the InnoDB cluster including all of its configuration information, and so on.

Once a sandbox instance is running, it is possible to change its status at any time using the following:

You can remove an instance from a cluster at any time should you wish to do so. This can be done with the removeInstance() method, as in the following example:

mysql-js> cluster.removeInstance("192.168.1.1:3306")

If an instance leaves the cluster, for example because it lost connection and did not or could not automatically rejoin the cluster, it may be necessary to rejoin it to the cluster at a later stage. Because the Group Replication configuration is not persisted in the instance's local configuration file, restarting an instance causes it to leave the Replication Group, so it must rejoin to add the instance back.

The command to rejoin an instance to a cluster is cluster.rejoinInstance().

In the case where an instance has been configured using dba.configureLocalInstance(), its Group Replication information is persisted to the configuration file, and will rejoin the cluster automatically. More information on this can be found in the section Configuring the Instance.

When MySQL Router is bootstrapped it records the bootstrap server addresses in its configuration. These servers contain metadata used my MySQL Router in order to route correctly. If any additional instances are added to the cluster after bootstrapping the MySQL Router, they are automatically detected and used for connection routing. If however, all of the original metadata servers go offline for some reason, MySQL Router would no longer be able to route correctly. Consider the following line in a mysqlrouter.conf file:

...
bootstrap_server_addresses=mysql://192.168.56.101:3310,mysql://192.168.56.101:3320,mysql://192.168.56.101:3330
...

There are three original metadata servers specified here. Now if two additional servers (call them D and E) were added, you would have a five instance cluster, and MySQL Router routes to these additional instances as required. If the original metadata instances, A, B and C, stopped unexpectedly and left the cluster, you would be left with only instances D and E running. At this point, instances D and E are still alive and form a quorum. So it should be possible to route calls to them. However, as all original metadata servers are down (instances A, B and C), MySQL Router shuts off all routing. In such a situation you can configure MySQL Router manually.

If you want to remove all information associated with a cluster, you can use the cluster.dissolve() method. This removes all metadata and configuration associated with the cluster. Once you have dissolved the cluster you need to create it again from scratch, using dba.createCluster().

You can automate cluster configuration with scripts. For example:

shell> mysqlsh -f setup-innodb-cluster.js

The contents for an example script file is shown here:

  print('MySQL InnoDB cluster sandbox set up\n');
  print('==================================\n');
  print('Setting up a MySQL InnoDB cluster with 3 MySQL Server sandbox instances.\n');
  print('The instances will be installed in ~/mysql-sandboxes.\n');
  print('They will run on ports 3310, 3320 and 3330.\n\n');

  var dbPass = shell.prompt('Please enter a password for the MySQL root account: ', {type:"password"});

  try {
     print('\nDeploying the sandbox instances.');
     dba.deploySandboxInstance(3310, {password: dbPass});
     print('.');
     dba.deploySandboxInstance(3320, {password: dbPass});
     print('.');
     dba.deploySandboxInstance(3330, {password: dbPass});
     print('.\nSandbox instances deployed successfully.\n\n');

     print('Setting up InnoDB cluster...\n');
     shell.connect('root@localhost:3310', dbPass);

     var cluster = dba.createCluster("devCluster");

     print('Adding instances to the cluster.');
     cluster.addInstance({user: "root", host: "localhost", port: 3320, password: dbPass});
     print('.');
     cluster.addInstance({user: "root", host: "localhost", port: 3330, password: dbPass});
     print('.\nInstances successfully added to the cluster.');

     print('\nInnoDB cluster deployed successfully.\n');
  } catch(e) {
     print('\nThe InnoDB cluster could not be created.\n\nError: ' +
     + e.message + '\n');
}

The cluster.checkInstanceState() function can be used for the following purposes:

Before creating a cluster from remote instances you need to check that the servers are suitably configured. This can be done using the dba.checkInstanceConfiguration() function. For detailed help on this function you can type dba.help('checkInstanceConfiguration').

The dba.checkInstanceConfiguration() function checks if the server instances are valid for InnoDB cluster usage.

The following demonstrates this:

mysql-js> dba.checkInstanceConfiguration('user@139.59.177.10:3306')

Please provide the password for 'user@139.59.177.10:3306':
Validating instance...

The instance '139.59.177.10:3306' is not valid for Cluster usage.

The following issues were encountered:

- Some configuration options need to be fixed.

+----------------------------------+---------------+----------------+--------------------------------------------------+
| Variable                         | Current Value | Required Value | Note                                             |
+----------------------------------+---------------+----------------+--------------------------------------------------+
| binlog_checksum                  | CRC32         | NONE           | Update the server variable or restart the server |
| enforce_gtid_consistency         | OFF           | ON             | Restart the server                               |
| gtid_mode                        | OFF           | ON             | Restart the server                               |
| log_bin                          | 0             | 1              | Restart the server                               |
| log_slave_updates                | 0             | ON             | Restart the server                               |
| master_info_repository           | FILE          | TABLE          | Restart the server                               |
| relay_log_info_repository        | FILE          | TABLE          | Restart the server                               |
| transaction_write_set_extraction | OFF           | XXHASH64       | Restart the server                               |
+----------------------------------+---------------+----------------+--------------------------------------------------+


Please fix these issues , restart the server and try again.

{
  "config_errors": [
    {
      "action": "server_update",
      "current": "CRC32",
      "option": "binlog_checksum",
      "required": "NONE"
    },
    {
      "action": "restart",
      "current": "OFF",
      "option": "enforce_gtid_consistency",
      "required": "ON"
    },
    {
      "action": "restart",
      "current": "OFF",
      "option": "gtid_mode",
      "required": "ON"
    },
    {
      "action": "restart",
      "current": "0",
      "option": "log_bin",
      "required": "1"
    },
    {
      "action": "restart",
      "current": "0",
      "option": "log_slave_updates",
      "required": "ON"
    },
    {
      "action": "restart",
      "current": "FILE",
      "option": "master_info_repository",
      "required": "TABLE"
    },
    {
      "action": "restart",
      "current": "FILE",
      "option": "relay_log_info_repository",
      "required": "TABLE"
    },
    {
      "action": "restart",
      "current": "OFF",
      "option": "transaction_write_set_extraction",
      "required": "XXHASH64"
    }
  ],
  "errors": [],
  "restart_required": true,
  "status": "error"
}
mysql-js>

The report shows the configuration changes required for that instance before it can be added to the cluster.

Once the configuration issues have been identified you can reconfigure your server instance manually. Alternatively, if you can run MySQL Shell directly on the same machine where the instance of MySQL is running, log in to the server and run MySQL Shell on the server. On the server to be configured run:

shell> mysqlsh --log-level=DEBUG3 --uri=root@localhost

The function you use to configure a server for InnoDB cluster use is dba.configureLocalInstance(). This function runs provisioning scripts for you that modify the MySQL server's configuration file.

The dba.configureLocalInstance() function can only configure servers connected to locally. If you try to run dba.configureLocalInstance() remotely you get the following error:

mysql-js> dba.configureLocalInstance('user@139.59.177.10:3306')

Dba.configureLocalInstance: This function only works with local instances (RuntimeError)

If MySQL Shell is started locally, then output will be similar to:

mysql-js> dba.configureLocalInstance('root@localhost:3306')

Please provide the password for 'root@localhost:3306':

Please specify the path to the MySQL configuration file: /etc/mysql/mysql.conf.d/mysqld.cnf
Validating instance...

The configuration has been updated but it is required to restart the server.
{
  "config_errors": [
    {
      "action": "restart",
      "current": "OFF",
      "option": "enforce_gtid_consistency",
      "required": "ON"
    },
    {
      "action": "restart",
      "current": "OFF",
      "option": "gtid_mode",
      "required": "ON"
      },
    {
      "action": "restart",
      "current": "0",
      "option": "log_bin",
      "required": "1"
    },
    {
      "action": "restart",
      "current": "0",
      "option": "log_slave_updates",
      "required": "ON"
    },
    {
      "action": "restart",
      "current": "FILE",
      "option": "master_info_repository",
      "required": "TABLE"
    },
    {
      "action": "restart",
      "current": "FILE",
      "option": "relay_log_info_repository",
      "required": "TABLE"
    },
    {
      "action": "restart",
      "current": "OFF",
      "option": "transaction_write_set_extraction",
      "required": "XXHASH64"
    }
  ],
  "errors": [],
  "restart_required": true,
  "status": "error"
}
mysql-js>

As with dba.checkInstanceConfiguration(), the configuration requirements are identified, but this time the entered configuration file is modified. For the changes to take effect you need to restart the MySQL Server. For example:

shell> sudo service mysql restart

shell.connect({host: 'localhost', port: 3333, user: 'root', password: 'somePwd'});

var cluster = dba.createCluster('devCluster');

// Here, configureLocalInstance makes sure the instance is configured for Group Replication
dba.configureLocalInstance('localhost:3334', {password:'somePwd', mycnfPath:'some path'})
cluster.addInstance('localhost:3334', {password:'somePwd'})

dba.configureLocalInstance('localhost:3335', {password:'somePwd', mycnfPath:'some path'})
cluster.addInstance('localhost:3335', {password:'somePwd'})

// A restart here, would require using rejoin to put the instance back into the cluster
dba.killSandboxInstance(3335);
dba.startSandboxInstance(3335);
cluster.rejoinInstance('localhost:3335', {password:'somePwd'})

// Calling configureLocalInstance again, since the instance is already part of the cluster
// It will persist the Group Replication server variables
dba.configureLocalInstance('localhost:3335', {password:'somePwd', mycnfPath:'some path'})

// On a new restart, the instance automatically joins the Cluster (no need to rejoinInstance)
dba.killSandboxInstance(3335);
dba.startSandboxInstance(3335);

Once the server has restarted, you can use MySQL Shell again to check the configuration:

mysql-js> dba.checkInstanceConfiguration('root@localhost:3306')

Please provide the password for 'root@localhost:3306':
Validating instance...

The instance 'localhost:3306' is valid for Cluster usage
{
  "status": "ok"
}
mysql-js>

Log in to the remote instance and use MySQL Shell to configure the instance automatically and ensure the configuration changes are persisted.

      
shell> mysqlsh --uri user@139.59.177.10:3306

Creating a Session to 'user@139.59.177.10:3306'
Enter password: *********
Classic Session successfully established. No default schema selected.

Now create the cluster:

      
mysql-js> var cluster = dba.createCluster('devCluster');

      A new InnoDB cluster will be created on instance 'user@139.59.177.10:3306'.

      Creating InnoDB cluster 'devCluster' on 'user@139.59.177.10:3306'...
      Adding Seed Instance...

      Cluster successfully created. Use Cluster.addInstance() to add MySQL instances.
      At least 3 instances are needed for the cluster to be able to withstand up to
      one server failure.

First, check the instance configuration:

mysql-js> dba.checkInstanceConfiguration('user@139.59.177.10:3306')
  Please provide the password for 'user@139.59.177.10:3306':
  Validating instance...

  The instance '139.59.177.10:3306' is valid for Cluster usage
  {
    "status": "ok"
  }

You can also check the instance state:

mysql-js> cluster.checkInstanceState('user@139.59.177.10:3306')
  Please provide the password for 'user@139.59.177.10:3306':
  Analyzing the instance replication state...

  The instance '139.59.177.10:3306' is valid for the cluster.
  The instance is fully recoverable.

  {
    "reason": "recoverable",
    "state": "ok"
  }

Check the cluster status:

mysql-js> cluster.status()
  {
    "clusterName": "devCluster",
    "defaultReplicaSet": {
      "name": "default",
      "status": "Cluster is NOT tolerant to any failures.",
      "topology": {}
    }
  }

You need to add two more instances to the cluster to make it tolerant to a server failure.

Check the configuration of the next instance to add to the cluster:

mysql-js> dba.checkInstanceConfiguration('user@139.59.177.11:3306')
  Please provide the password for 'user@139.59.177.10:3306':
  Validating instance...

  The instance '139.59.177.11:3306' is valid for Cluster usage
  {
    "status": "ok"
  }

The instance can now be added into the cluster:

mysql-js> cluster.addInstance("user@139.59.177.11:3306");

  Please provide a password for 'user@139.59.177.11:3306': *****

  A new instance will be added to the InnoDB cluster. Depending on the
  amount of data on the cluster this might take from a few seconds to
  several hours.

  Adding instance 139.59.177.11:3306 to the cluster...

  The instance '139.59.177.11:3306' was successfully added to the
  cluster.

The next instance can now be added into the cluster:

mysql-js> cluster.addInstance("user@139.59.177.12:3306");

  Please provide a password for 'user@139.59.177.12:3306': *****

  A new instance will be added to the InnoDB cluster. Depending on the
  amount of data on the cluster this might take from a few seconds to
  several hours.

  Adding instance 139.59.177.12:3306 to the cluster...

  The instance '139.59.177.12:3306' was successfully added to the
  cluster.

Now recheck cluster status.

When using the createCluster(), addInstance(), and rejoinInstance() methods you can optionally specify a list of approved servers that belong to the cluster, referred to a whitelist. By specifying the whitelist explicitly in this way you can increase the security of your cluster because only servers in the whitelist can connect to the cluster. By default, if not specified explicitly, the whitelist is automatically set to the private network addresses that the server has network interfaces on. To configure the whitelist, specify the servers to add with the ipWhitelist option when using the method. For example:

mysql-js> c.addInstance("root:guidev!@localhost:3320", {ipWhitelist: "10.157.120.0/24, 192.168.1.110"})

This configures the instance to only accept connections from servers at addresses 10.157.120.0/24 and 192.168.1.110.

Using the ipWhitelist option configures the group_replication_ip_whitelist system variable on the instance.

If a instance (or instances) fail, then a cluster can lose its quorum, which is the ability to vote in a new primary. In this case you can re-establish quorum using the method cluster.forceQuorumUsingPartitionOf(), as shown in the following MySQL Shell example:

  // open session to a cluster

mysql-js> cluster = dba.getCluster("devCluster")

  // The cluster lost its quorum and its status shows
  // "status": "NO_QUORUM"

mysql-js> cluster.forceQuorumUsingPartitionOf("localhost:3310")

  Restoring replicaset 'default' from loss of quorum, by using the partition composed of [localhost:3310]

  Please provide the password for 'root@localhost:3310': ******
  Restoring the InnoDB cluster ...

  The InnoDB cluster was successfully restored using the partition from the instance 'root@localhost:3310'.

  WARNING: To avoid a split-brain scenario, ensure that all other members of the replicaset
  are removed or joined back to the group that was restored.

If your cluster suffers from a complete outage, you can ensure it is reconfigured correctly using dba.rebootClusterFromCompleteOutage(). An example of use is as follows:

        
mysql-js> shell.connect('root@localhost:3310');
mysql-js> var cluster = dba.rebootClusterFromCompleteOutage();

This ensures the cluster is correctly reconfigured after a complete outage. It picks the instance the MySQL Shell is connected to as the new seed instance and recovers the cluster based on the existing metadata of that instance.

It is also possible to provide the cluster name as an input parameter:

mysql-js> var cluster = dba.createCluster("devCluster")
  ...
  ...
mysql-js> var cluster = dba.rebootClusterFromCompleteOutage("devCluster");

If this process fails, and the cluster metadata has become badly corrupted, you may need to drop the metadata and create the cluster again from scratch. You can drop the cluster metadata using dba.dropMetaDataSchema().

If changes to the Group Replication configurations are made without using MySQL Shell you need to rescan your cluster. For example, if you create a cluster with three instances, and then without using MySQL Shell you add a new instance to that Group Replication group, the AdminAPI is not aware of that instance. The same would apply if you removed an instance from a Group Replication group without using MySQL Shell. It is necessary to rescan the cluster with cluster.rescan() in such scenarios.

After the command cluster.rescan() has been run, instances are identified that are newly discovered instances. You are prompted to add each of these newly discovered instances into your cluster as required, or you can choose to ignore them.

Nodes that no longer belong to the cluster or which are unavailable are also reported. In this case you are prompted to remove the instance, or you can later attempt to add it back into the cluster using a command such as cluster.rejoin('instancex.example.com:3340').