MongoDB Replica Set: Achieving High Availability and Scalability

Easy as pie!

There are many articles out there on the internet about setting up a MongoDB replica set. There is a great tutorial here made by MongoDB as well. However, I found most of those explanations to be too complicated for me, or that they work only locally (inside docker). They are good as an example, however you want to have a MongoDB replica set split into three servers. In three data centers.

Therefore, I created a very simple tutorial for myself, with which I ended up being very happy. In case any of you would like to try it out as well, I will share it with you below. I hope you find it helpful.

And from now on, I will try to be as brief as possible.

We will setup a replica set using docker. For the purpose of this article, I will assume you have some basic knowledge of it.

We will secure the communication between the servers by a keyfile.

For the sake of simplicity, I will not explain every command. Or any commands for that matter.

Create a keyfile

Create a text file with random content. Its length should be from 6 to 1024 characters. Keep it simple — the length is more important than including ‘special’ characters. Name it mongo-keyfile (not mongo-keyfile.txt. Otherwise you will need to change some of the lines below.)

Create a certificates for communication

In case you want to encrypt the communication between your client application and MongoDB, you need to have certificates. For self-signed certificates you can follow this section: https://docs.mongodb.com/manual/appendix/security/appendixA-openssl-ca/

Prepare setup scripts

users.js (change the password, please)

db.getSiblingDB("admin").createUser(
{
	"user": "my_admin", "pwd": "generated_long_password",    
	"roles": [        
		{ "role": "userAdminAnyDatabase", "db": "admin" },         
		{ "role": "readWriteAnyDatabase", "db" : "admin"},        
		{ "role": "clusterAdmin", "db" : "admin" },        
		{ "role": "backup", db: "admin" },        
		{ "role": "readWrite", "db" : "admin"},        
		{ "role": "restore", db: "admin" }    
	]  
});

replica.js (put your existing IP address/dns there)

rs.initiate(    
{         
    "_id": "rs0",         
    "members": [ { "_id": 0, "host": "126.20.15:10:27017" }  ]    
})

First server (primary)

Copy the mongo-keyfile file and both scripts (users.js, replice.js) to the directory on the server where you want to install the MongDB. Start the command line and switch to the folder with these files. For better readability, I use ` \` for line breaks (since it can be used in the linux command line)

docker volume create --name mongo_storage_rs0
docker run --name mongo-node-rs0 -v mongo_storage:/data -d mongo
docker exec mongo-node-rs0 bash -c "mkdir /data/keyfile /data/admin"

docker cp mongo-keyfile mongo-node-rs0:/data/keyfile/
docker cp replica.js mongo-node-rs0:/data/admin/
docker cp users.js mongo-node-rs0:/data/admin/

docker cp caTlsCertificate.pem mongo-node-rs0:/data/keyfile/
docker cp tlsCertificate.pem mongo-node-rs0:/data/keyfile/

docker exec mongo-node-rs0 bash -c "chmod 600 /data/keyfile/mongo-keyfile"
docker exec mongo-node-rs0 bash -c "chmod 600 /data/keyfile/caTlsCertificate.pem"
docker exec mongo-node-rs0 bash -c "chmod 600 /data/keyfile/tlsCertificate.pem"
docker exec mongo-node-rs0 bash -c "chown -R mongodb:mongodb /data"
docker stop mongo-node-rs0
docker rm mongo-node-rs0

docker run \
--name mongo-node-rs0 \
--restart always \
-v mongo_storage:/data \
-p 27010:27017 \
-d mongo \
--keyFile /data/keyfile/mongo-keyfile \
--replSet rs0 \
--tlsMode preferTLS \
--tlsCertificateKeyFile /data/keyfile/tlsCertificate.pem \
--tlsCAFile /data/keyfile/caTlsCertificate.pem \
--tlsAllowConnectionsWithoutCertificates \
--tlsAllowInvalidHostnames \
--tlsAllowInvalidCertificates \
--bind_ip_all

docker exec mongo-node-rs0 bash -c "mongo < /data/admin/replica.js"
docker exec mongo-node-rs0 bash -c "mongo < /data/admin/users.js"

Yes, I know. There are quite a lot of commands, but if you create a script from them, this will change to a single command instead. Or you can simply run them. It takes just a few seconds…

(tlsOptions are setup for testing purpose, or for trusted networks.)

Second/Third … server

Again, copy all the files (mongo-keyfile, users.js, replice.js) to the directory in which your command line is located.

docker volume create --name mongo_storage_rs0
docker run --name mongo-node-rs0 -v mongo_storage:/data -d mongo
docker exec mongo-node-rs0 bash -c "mkdir /data/keyfile /data/admin"
docker cp mongo-keyfile mongo-node-rs0:/data/keyfile/
docker exec mongo-node-rs0 bash -c "chmod 600 \/data/keyfile/mongo-keyfile"

docker exec mongo-node-rs0 bash -c "chown -R mongodb:mongodb /data"
docker stop mongo-node-rs0
docker rm mongo-node-rs0

docker run \
--name mongo-node-rs0 \
--restart always \
-v mongo_storage:/data \
-p 27010:27017 \
-d mongo \
--keyFile /data/keyfile/mongo-keyfile \
--replSet rs0 \
--bind_ip_all

Almost the same as the first server. Only there is no setup for users and replica. This can be setup only on the primary server.

Connect secondary servers to the primary one

I use NoSql booster to connect to servers. Connect to the primary server and add the second and the third one (use your valid IP addresses).

rs.add("166.20.15:11:24010")
rs.add("166.20.15:12:24010")

The end of story

And that is it. You are finished and free to do some exercise or spend time with your family. 😎

Or you can checks some tips bellow.

Or you can read our story about cloud security in our Tabidoo systems.

Tips for the next day(s)

You should not use one admin user for admin, to access data, to make a backup…

Use domain names instead of IPs. With cloudflare in nginx, it is very easy and free.

Security by a keyfile is not recommended for production. However, when it comes to the x.509, people find it too complicated and do not use any security instead. Proper security is out of scope of this particular article.

Changing the replica set name of the server is fairly easy and it is well explained here.

cfg = rs.conf()
cfg.members[1].host = "mongodb1.example.net:27017"
rs.reconfig(cfg)

When you connect to the replica, you do not care which server is up — which one is primary. It is usually the job of the library you use. You just need to use the correct connection string on all of the servers.

mongodb://mongodb1.example.com:27317,mongodb2.example.com:27017/?replicaSet=mySet&authSource=authDB

Use at least three servers. Even if you end up with two MongoDBs on the main server.

Establish a backup server in docker (in a file in the hosting system).

docker exec mongo4-single sh -c "exec mongodump \
--authenticationDatabase admin -d putYourDatabaseNameHere \
--username=yourUserName 
--password=yourPassword\--archive" > transfer-backup

Copy your backup to the docker and restore it.

docker cp transfer-backup mongo-node-rs0:/home/transfer-backup

docker exec mongo-node-rs0 sh -c "exec mongorestore \
--drop --writeConcern '{w:0}' --authenticationDatabase admin \
--nsInclude putYourDatabaseNameHere.*  \
--username=yourUserName --password=yourPassword \
--archive=/home/transfer-backup"