Recovering the MongoDB replicaset after a hard server reboot on 32bit systems

If one of your check servers crashes/was hard rebooted and it is running MongoDB you will most likely need to resync the MongoDB replicaset since the 32bit version of MongoDB doesn't use journaling.

First, check to see if your MongoDB instance needs to be resynced, tail the /var/log/mongo/mongod.log and if you see the following you will need to perform a resync.

e.g.

tail -n 100 /var/log/mongo/mongod.log

**************

Unclean shutdown detected.

Please visit http://dochub.mongodb.org/core/repair for recovery instructions.

*************

If you don't see a message about an unclean shutdown then your replica set is fine and doesn't need repaired.

If you do see this message, then ssh to the server that was rebooted and issue the following commands which backs up the existing replicaset data on that server, then removes it so it can be resynced. (If you have recovered this mongodb instance before you will need to use a different backup directory name instead of mongo-bk or remove the old mongo-bk before issuing the following commands).

mv /var/lib/mongo /var/lib/mongo-bk
mkdir /var/lib/mongo
cp /var/lib/mongo-bk/key /var/lib/mongo/.
chown mongod:mongod /var/lib/mongo
service mongod start

Upon starting up mongo, the replicaset will automatically be resynced.
  • 171 Users Found This Useful
Was this answer helpful?

Related Articles

Setting up Kayako v4 Help Desk Alerts

  ServerPing allows you to offer a monitoring with response service by giving customers the...

Adding a SMS Credits Package

Login to your WHMCS Admin Area and go to Setup->Products/Services->Products/Services...

Setting up WHMCS Help Desk Alerts

ServerPing allows you to offer a monitoring with response service by giving customers the option...

Setting up WHMCS to work with the Public Report Page

WHMCS includes the header.tpl and footer.tpl file on every request made from an addon module. In...

Installation Guide

Requirements You will need at least one Cent OS 6 (64 bit recommended) VPS/Dedicated server with...