Benchmarking MongoDB in a container

The database layer for an application is one of the most crucial part because believe it or not it effects the performance of your application, now with micro-services getting the attention I was just wondering if having a database container will make a difference.

As we have popularly seen most of the containers used are stateless containers that means that they don’t retain the data they generate but there is a way to have stateful containers and that is by mounting a host volume in the container. Having said this there could be an issue with the latency in the database request, I wanted to measure how much will this latency be and what difference will it make if the installation is done natively verses if the installation is done in a container.

I am going to run a simple benchmarking scheme I will make 200 insert request that is write request keeping all other factors constant and will plot the time taken for these request and see what comes out of it.

I borrowed a quick script to do the same from this blog. The script is simple it just uses pymongo the python MongoDB driver to connect to the database and make 200 entries in a random database.


import time
import pymongo
m = pymongo.MongoClient()

doc = {'a': 1, 'b': 'hat'}

i = 0

while (i < 200):

start = time.time()
m.tests.insertTest.insert(doc, manipulate=False, w=1)
end = time.time()

executionTime = (end - start) * 1000 # Convert to ms

print executionTime

i = i + 1

So I went to install MongoDB natively first I ran the above script twice and took the second result into consideration. Once I did that I plotted the graph with value against the number of request. The first request takes time because it requires to make connection and all the over head and the plot I got looked like this.

 

Native
MongoDb Native Time taken in ms v/s Number of request

The graph shows that the first request took about 6 ms but the consecutive requests took way lesser time.

Now it was time I try the same to do it in a container so I did a docker pull mongo and then I mounted a local volume in the container and started the container by

docker run --name some-mongo -v /Users/farhaanbukhsh/mongo-bench/db:/data/db -d mongo

This mounts the volume I specified to /data/db in the container then I did a docker cp of the script and installed the dependencies and ran the script again twice so that file creation doesn’t manipulate the time.

To my surprise the first request took about 4ms but subsequent requests took a lot of time.

Containered
MongoDB running in a container(Time in ms v/s Number of Requests)

 

And when I compared them the time time difference for each write or the latency for each write operation was ​considerable.

MongoDB bench mark
Comparison between Native and Containered MongoDB

I had this thought that there will be difference in time and performance but never thought that it would be this huge, now I am wondering what is the solution to this performance issue, can we reach a point where the containered performance will be as good as native.

Let me know what do you think about it.

Happy Hacking!

Advertisements

The Open Organization

I was recently going through few of the Farnam Street articles, and I landed on the article on how to read a book, where they basically describe how to read a book;  the fact that there are types of books, and the fact that books can, in the words of Francis Bacon “be gulped, some books chewed and others digested.”

This basically signifies the intensity and the level of awareness to have when you are reading a book. I have gulped lots of books, but The Open Organization is one of those, that I wanted to chew on.

I wanted to learn about how you can build an ecosystem where people are free to voice their opinions, where failure is be worn as a badge of honor for trying. This book filled me with thoughts of how would it be like, if an organization is really an Open Organization.

There are a lots of beautiful anecdotes that I came across, and a lot of values that were given in the book to think on.

The book talks about Purpose and Passion. People specially us Millenials,have been spoiled to an extent that we actually don’t run after money but after a purpose, after a problem. We don’t mind working crazy hours and being paid peanuts, but we do care about people, we care about how are we treated, we care about the problem we are after. One of the quotes in the book says Basis of loyalty is a common purpose and not economic dependency. A lot of people I know believe in this. When you unite with an organization which is after the same problem as you, it’s a match made in heaven.

The book talks about Passion, the passion about doing good, making a dent in the universe, but sometimes you realize Universe doesn’t give a damn .

One of the most amazing analogies, is when the book compares a structure of an organization with web architecture which is end to end and not center to end. Where there is no central point of control but there should be a central point of co-ordination. The organization is lead by leaders it selects, where Meritocracy is the idea behind every decision.

The other idea that was completely new to me was the difference between Crowd-sourcing and Open-sourcing.To be honest I had not thought open source to be a business model until the recent past. The thing with the wisdom of the crowd is that it works amazingly well when the work can be easily disagregated and individuals can work in relative isolation. I love the point in the book that says members of the organization should be inspired by the leader and not motivated. Motivation is something they already have and that is the reason they are joining your organization. I love this idea a lot because I have seen people complaining about their employees not being motivated enough. I think that this (lack of inspired leadership) is a reason.

“Great companies don’t hire skilled people and motivate them, they hire already motivated people and inspire them.” – Simon Sinek

I really enjoyed the way the power of purpose is laid out in the book. The other idea was the idea of Meritocracy.  I think of  merit as having an amazing idea and idea being the sole reason for doing a certain action. Better ideas win, they are questioned and deliberated upon and that is how innovation happens in the organization. People debate over it, question it, trash it. People just don’t settle for something to avoid conflict. That very same complacency however is what has creeped into organizations where people don’t debate ideas just to avoid conflict so that everyone remains happy. It was so amazing to read stories where someone thought out of the box and wanted to bring in a new way of doing things and how he convinced everyone that this is the right way of doing things, we ought to give it a try.

This book pushes back on the belief in hierarchy and brings to limelight lateral structure, letting people know that the conventional ways of running an organization might have to change, upgrade as it were, to a newer version.

I got a lot of amazing ideas and to be honest I got to know how a person in an organization should be treated. I was awestruck with the insights in the book. Wish someday I could mould an organization in this way. Theories are always romantic, hope the execution and implementation is beautiful as well.