MongoDB synchronize Development and Production databases

Question

We have a dev server which contains a collection of Objects. The actual accumulation of these objects is an ongoing process, which runs a whole process of labelling, verification, etc. on this local dev server. Once these objects are production ready, they are added to the Production DB, which will from that moment, use them in its calculations.

I'm looking for a way to simply add the delta (new objects) into the production DB, while retaining all the other collections, and older objects in the same collection as is. Until now, we've used MySql, so this process simply involved running DB structure and data synchronization (we used Navicat for that). We're now moving to MongoDB, so this process is a bit more tricky.

I've looked into this, and I think the following solutions do not fit my needs:

1. Dumping the dev DB and loading it in the Production DB using mongodump then mongorestore
2. Running db.copyDatabase - actually replaces the production db with a copy of the dev DB.

Both solutions are problematic, because they actually replace the Production DB, when all I want to do is update objects in an existing collection. Also, going Dev => Production does not fit the Master-Slave topology.

The best thing I could think of is to:

1. Copy the dev DB into a "dev" DB on the production instance.
2. Copy the collection from this dev DB to the actual production DB, and add a predicate that I will only add objects that are non-existent in the production DB, similar to this solution.

I was wondering if anyone has a better solution?

If not, does anyone have a script that could perform this?

Answer 1

You can use the mongoexport tool to export the single collection from your development database. Use it in conjunction with a --query option where you can express a predicate. For example something such as ${ts : {$gt : previous clone time}}.

Then, use mongoimport to import your delta file into production database. Use --upsert and --upsertFields if you have two different logical documents with different _id values, but express the same document

Answer 2

@Orid, thanks for your answer, and sorry for the late reply.

After a lot of research and some trial-error, I decided to use the solution stated in the question (Copy test DB to machine, and then copy the collections one by one). This is because the data I'm using here is static data, that doesn't have a real reason to have a timestamp. Also, I've decided to waive the requirement for "only updating", so for now I'm using mongorestore with --drop

I do all this using this script:

1. The shell script file:

rm -rf dump/;

mongo copyTestDb.js;

for COLLECTION in <Collections>
do
mongodump -d nutrino_copy -c $COLLECTION -o dump
mongorestore -d nutrino -c ${COLLECTION} --drop dump/nutrino_copy/${COLLECTION}.bson
done

1. The js script file:

   db.copyDatabase("<dbName>","<dbName_Copy>","<testMachineUrl>")

Do you think I should use MongoImport instead of MongoRestore?

Answer 3

Check out mongo-sync

---

It's a script I wrote for my self when I had to constantly copy my Local MongoDB database to and from my Production DB for a Project (I know it's stupid).

Once you put your DB details in config.yml, you can start syncing using two simple commands:

./mongo-sync push       # Push DB to Remote
./mongo-sync pull       # Pull DB to Local

If you use it inside some project, it's a good idea to add config.yml to .gitignore

---