MongoDb. Capped collections

Today we’re going to discuss not really popular, but really useful thing in MongoDB – capped collections. What’re they, how and where to use them. Let’s start!

The main thing that you need to know about capped collections at first that’s set of restrictions – they don’t have all the functionality of common collections:

1. No delete operations;
2. All elements in collection should have an equal size (update will be rejected if element has different size than it used to have for example);
3. It works as queue and because of this no index is required;

The large set of restrictions, some of them can be lethal for normal collection usage for obvious reasons, so then exactly can we use such restricted set then? Let’s discuss some specifics in mechanics before make a decision and come to the conclusion.

Mechanic differences provide us following profits:

1. More than 10 000 read/write operations per second (that’s really good performance for MongoDB);
2. Cursor that can be used for tracking of new entry adding to collection;
3. Auto-removing from collection according to FIFO rule than size exceeds given limitations;

I believe now it comes to be more interesting for you, so let’s proof that this functionality is actually used, and the best example at this point is that mongo actually uses it by itself!
Local.startup_log – collection that stores all the MongoDB settings on start is actually a capped one.

Oplog – collection that stores logs about all your operation and sits behind everything that can be done in MongoDB.

So MongoDB uses those collections by itself, that allows it be efficient in many points. I believe that after those examples we definitely need to try this thing by ourselves. But where? In my mind the basic points of usage for such collections can be caching and logging, because for both cases it’s fast and can clean memory by itself.

So let’s do some coding and see how to use it with Node.js. First of all we need to initialize this collection and there’re differences with common collection already, you need to set capped flag to true and specify amount that collection can take from memory (as you remember it’s required for self cleaning and efficient operations upon the data).

let mongodb = require('mongodb');
let MongoClient = mongodb.MongoClient;

MongoClient.connect('mongodb://localhost:27017/capped').then((db) => {
    db.createCollection('errorLogs', {
        capped: true,
        size: 1000000
    });
});

So as soon as we’re done with collection initialization let’s do something with this one. And here we’re going to initialize cursor as was discussed before. Cursor is a special entity that allows us to track adding operations to collection and can be useful for different things, for example, you tracked adding of fatal log error to logs, inside cursor can sit logic that will send email to responsible admin/programmer of this part of code.

let mongodb = require('mongodb');
let MongoClient = mongodb.MongoClient;

MongoClient.connect('mongodb://localhost:27017/capped').then((db) => {
    const collection = db.collection('errorLogs');

    const cursor = collection.find({}, {tailable: true, awaitdata: true});
    const cursorStream = cursor.stream();
    cursorStream.on('data', function (data) {
        console.log('added', data);
    });

    collection.insert({
        type: 'error',
        time: new Date().toISOString(),
        message: 'Something bad happend!'
    });
});

Be free to play with this code and you will understand much better how the capped collections work. Let’s summarize now what’re the differences here:

Now I believe you know the main scope of usage for capped collection, how to initialize and work upon them. Be free to ask questions in comments and offer your best cases of usage. Good luck!

We are looking forward to meeting you on our website blog.soshace.com