OPTIONS

Deploy a Geographically Redundant Replica Set

This tutorial outlines the process for deploying a replica set with members in multiple locations. The tutorial addresses three-member sets, four-member sets, and sets with more than four members.

For appropriate background, see Replication and Replica Set Deployment Architectures. For related tutorials, see Deploy a Replica Set and Add Members to a Replica Set.

Overview

While replica sets provide basic protection against single-instance failure, replica sets whose members are all located in a single facility are susceptible to errors in that facility. Power outages, network interruptions, and natural disasters are all issues that can affect replica sets whose members are colocated. To protect against these classes of failures, deploy a replica set with one or more members in a geographically distinct facility or data center to provide redundancy.

Requirements

In general, the requirements for any geographically redundant replica set are as follows:

  • Ensure that a majority of the voting members are within a primary facility, “Site A”. This includes priority 0 members and arbiters. Deploy other members in secondary facilities, “Site B”, “Site C”, etc., to provide additional copies of the data. See Determine the Distribution of Members for more information on the voting requirements for geographically redundant replica sets.
  • If you deploy a replica set with an even number of members, deploy an arbiter on Site A. The arbiter must be on site A to keep the majority there.

For instance, for a three-member replica set you need two instances in a Site A, and one member in a secondary facility, Site B. Site A should be the same facility or very close to your primary application infrastructure (i.e. application servers, caching layer, users, etc.)

A four-member replica set should have at least two members in Site A, with the remaining members in one or more secondary sites, as well as a single arbiter in Site A.

For all configurations in this tutorial, deploy each replica set member on a separate system. Although you may deploy more than one replica set member on a single system, doing so reduces the redundancy and capacity of the replica set. Such deployments are typically for testing purposes and beyond the scope of this tutorial.

This tutorial assumes you have installed MongoDB on each system that will be part of your replica set. If you have not already installed MongoDB, see the installation tutorials.

Procedures

General Considerations

  • Each member of the replica set resides on its own machine and all of the MongoDB processes bind to port 27017 (the standard MongoDB port).

  • Each member of the replica set must be accessible by way of resolvable DNS or hostnames, as in the following scheme:

    • mongodb0.example.net
    • mongodb1.example.net
    • mongodb2.example.net
    • mongodbn.example.net

    You will need to either configure your DNS names appropriately, or set up your systems’ /etc/hosts file to reflect this configuration.

  • Ensure that network traffic can pass between all members in the network securely and efficiently. Consider the following:

    • Establish a virtual private network. Ensure that your network topology routes all traffic between members within a single site over the local area network.
    • Configure authentication using auth and keyFile, so that only servers and processes with authentication can connect to the replica set.
    • Configure networking and firewall rules so that only traffic (incoming and outgoing packets) on the default MongoDB port (e.g. 27017) from within your deployment is permitted.

    For more information on security and firewalls, see Inter-Process Authentication.

  • You must specify the run time configuration on each system in a configuration file stored in /etc/mongodb.conf or a related location. Do not specify the set’s configuration in the mongo shell.

    Use the following configuration for each of your MongoDB instances. You should set values that are appropriate for your systems, as needed:

    port = 27017
    
    bind_ip = 10.8.0.10
    
    dbpath = /srv/mongodb/
    
    fork = true
    
    replSet = rs0
    

    The dbpath indicates where you want mongod to store data files. The dbpath must exist before you start mongod. If it does not exist, create the directory and ensure mongod has permission to read and write data to this path. For more information on permissions, see the security operations documentation.

    Modifying bind_ip ensures that mongod will only listen for connections from applications on the configured address.

    For more information about the run time options used above and other configuration options, see Configuration File Options.

Deploy a Geographically Redundant Three-Member Replica Set

Diagram of a 3 member replica set distributed across two data centers. Replica set includes a priority 0 member.
  1. Start a mongod instance on each system that will be part of your replica set. Specify the same replica set name on each instance. For additional mongod configuration options specific to replica sets, see Replication Options.

    Important

    If your application connects to more than one replica set, each set should have a distinct name. Some drivers group replica set connections by replica set name.

    If you use a configuration file, then start each mongod instance with a command that resembles the following:

    mongod --config /etc/mongodb.conf
    

    Change /etc/mongodb.conf to the location of your configuration file.

    Note

    You will likely want to use and configure a control script to manage this process in production deployments. Control scripts are beyond the scope of this document.

  2. Open a mongo shell connected to one of the hosts by issuing the following command:

    mongo
    
  3. Use rs.initiate() to initiate a replica set consisting of the current member and using the default configuration, as follows:

    rs.initiate()
    
  4. Display the current replica configuration:

    rs.conf()
    

    The replica set configuration object resembles the following

    {
       "_id" : "rs0",
       "version" : 4,
       "members" : [
          {
             "_id" : 1,
             "host" : "mongodb0.example.net:27017"
          }
       ]
    }
    
  1. In the mongo shell connected to the primary, add the remaining members to the replica set using rs.add() in the mongo shell on the current primary (in this example, mongodb0.example.net). The commands should resemble the following:

    rs.add("mongodb1.example.net")
    rs.add("mongodb2.example.net")
    

    When complete, you should have a fully functional replica set. The new replica set will elect a primary.

  1. Make sure that you have configured the member located in Site B (in this example, mongodb2.example.net) as a priority 0 member:

    1. Issue the following command to determine the members array position for the member:

      rs.conf()
      
    2. In the members array, save the position of the member whose priority you wish to change. The example in the next step assumes this value is 2, for the third item in the list. You must record array position, not _id, as these ordinals will be different if you remove a member.

    3. In the mongo shell connected to the replica set’s primary, issue a command sequence similar to the following:

      cfg = rs.conf()
      cfg.members[2].priority = 0
      rs.reconfig(cfg)
      

      When the operations return, mongodb2.example.net has a priority of 0. It cannot become primary.

      Note

      The rs.reconfig() shell method can force the current primary to step down, causing an election. When the primary steps down, all clients will disconnect. This is the intended behavior. While most elections complete within a minute, always make sure any replica configuration changes occur during scheduled maintenance periods.

    After these commands return, you have a geographically redundant three-member replica set.

Check the status of your replica set at any time with the rs.status() operation.

See also

The documentation of the following shell functions for more information:

Refer to Replica Set Read and Write Semantics for a detailed explanation of read and write semantics in MongoDB.

Deploy a Geographically Redundant Four-Member Replica Set

A geographically redundant four-member deployment has two additional considerations:

  • One host (e.g. mongodb4.example.net) must be an arbiter. This host can run on a system that is also used for an application server or on the same machine as another MongoDB process.

  • You must decide how to distribute your systems. There are three possible architectures for the four-member replica set:

    • Three members in Site A, one priority 0 member in Site B, and an arbiter in Site A.
    • Two members in Site A, two priority 0 members in Site B, and an arbiter in Site A.
    • Two members in Site A, one priority 0 member in Site B, one priority 0 member in Site C, and an arbiter in site A.

    In most cases, the first architecture is preferable because it is the least complex.

To deploy a geographically redundant four-member set:

  1. Start a mongod instance on each system that will be part of your replica set. Specify the same replica set name on each instance. For additional mongod configuration options specific to replica sets, see Replication Options.

    Important

    If your application connects to more than one replica set, each set should have a distinct name. Some drivers group replica set connections by replica set name.

    If you use a configuration file, then start each mongod instance with a command that resembles the following:

    mongod --config /etc/mongodb.conf
    

    Change /etc/mongodb.conf to the location of your configuration file.

    Note

    You will likely want to use and configure a control script to manage this process in production deployments. Control scripts are beyond the scope of this document.

  2. Open a mongo shell connected to one of the hosts by issuing the following command:

    mongo
    
  3. Use rs.initiate() to initiate a replica set consisting of the current member and using the default configuration, as follows:

    rs.initiate()
    
  4. Display the current replica configuration:

    rs.conf()
    

    The replica set configuration object resembles the following

    {
       "_id" : "rs0",
       "version" : 4,
       "members" : [
          {
             "_id" : 1,
             "host" : "mongodb0.example.net:27017"
          }
       ]
    }
    
  1. Add the remaining members to the replica set using rs.add() in a mongo shell connected to the current primary. The commands should resemble the following:

    rs.add("mongodb1.example.net")
    rs.add("mongodb2.example.net")
    rs.add("mongodb3.example.net")
    

    When complete, you should have a fully functional replica set. The new replica set will elect a primary.

  2. In the same shell session, issue the following command to add the arbiter (e.g. mongodb4.example.net):

    rs.addArb("mongodb4.example.net")
    
  3. Make sure that you have configured each member located outside of Site A (e.g. mongodb3.example.net) as a priority 0 member:

    1. Issue the following command to determine the members array position for the member:

      rs.conf()
      
    2. In the members array, save the position of the member whose priority you wish to change. The example in the next step assumes this value is 2, for the third item in the list. You must record array position, not _id, as these ordinals will be different if you remove a member.

    3. In the mongo shell connected to the replica set’s primary, issue a command sequence similar to the following:

      cfg = rs.conf()
      cfg.members[2].priority = 0
      rs.reconfig(cfg)
      

      When the operations return, mongodb2.example.net has a priority of 0. It cannot become primary.

      Note

      The rs.reconfig() shell method can force the current primary to step down, causing an election. When the primary steps down, all clients will disconnect. This is the intended behavior. While most elections complete within a minute, always make sure any replica configuration changes occur during scheduled maintenance periods.

    After these commands return, you have a geographically redundant four-member replica set.

Check the status of your replica set at any time with the rs.status() operation.

See also

The documentation of the following shell functions for more information:

Refer to Replica Set Read and Write Semantics for a detailed explanation of read and write semantics in MongoDB.

Deploy a Geographically Redundant Set with More than Four Members

The above procedures detail the steps necessary for deploying a geographically redundant replica set. Larger replica set deployments follow the same steps, but have additional considerations:

  • Never deploy more than seven voting members.
  • If you have an even number of members, use the procedure for a four-member set). Ensure that a single facility, “Site A”, always has a majority of the members by deploying the arbiter in that site. For example, if a set has six members, deploy at least three voting members in addition to the arbiter in Site A, and the remaining members in alternate sites.
  • If you have an odd number of members, use the procedure for a three-member set. Ensure that a single facility, “Site A” always has a majority of the members of the set. For example, if a set has five members, deploy three members within Site A and two members in other facilities.
  • If you have a majority of the members of the set outside of Site A and the network partitions to prevent communication between sites, the current primary in Site A will step down, even if none of the members outside of Site A are eligible to become primary.