Benchmarking MongoDB in a container

The database layer for an application is one of the most crucial part because believe it or not it effects the performance of your application, now with micro-services getting the attention I was just wondering if having a database container will make a difference.

As we have popularly seen most of the containers used are stateless containers that means that they don’t retain the data they generate but there is a way to have stateful containers and that is by mounting a host volume in the container. Having said this there could be an issue with the latency in the database request, I wanted to measure how much will this latency be and what difference will it make if the installation is done natively verses if the installation is done in a container.

I am going to run a simple benchmarking scheme I will make 200 insert request that is write request keeping all other factors constant and will plot the time taken for these request and see what comes out of it.

I borrowed a quick script to do the same from this blog. The script is simple it just uses pymongo the python MongoDB driver to connect to the database and make 200 entries in a random database.


import time
import pymongo
m = pymongo.MongoClient()

doc = {'a': 1, 'b': 'hat'}

i = 0

while (i < 200):

start = time.time()
m.tests.insertTest.insert(doc, manipulate=False, w=1)
end = time.time()

executionTime = (end - start) * 1000 # Convert to ms

print executionTime

i = i + 1

So I went to install MongoDB natively first I ran the above script twice and took the second result into consideration. Once I did that I plotted the graph with value against the number of request. The first request takes time because it requires to make connection and all the over head and the plot I got looked like this.

 

Native
MongoDb Native Time taken in ms v/s Number of request

The graph shows that the first request took about 6 ms but the consecutive requests took way lesser time.

Now it was time I try the same to do it in a container so I did a docker pull mongo and then I mounted a local volume in the container and started the container by

docker run --name some-mongo -v /Users/farhaanbukhsh/mongo-bench/db:/data/db -d mongo

This mounts the volume I specified to /data/db in the container then I did a docker cp of the script and installed the dependencies and ran the script again twice so that file creation doesn’t manipulate the time.

To my surprise the first request took about 4ms but subsequent requests took a lot of time.

Containered
MongoDB running in a container(Time in ms v/s Number of Requests)

 

And when I compared them the time time difference for each write or the latency for each write operation was ​considerable.

MongoDB bench mark
Comparison between Native and Containered MongoDB

I had this thought that there will be difference in time and performance but never thought that it would be this huge, now I am wondering what is the solution to this performance issue, can we reach a point where the containered performance will be as good as native.

Let me know what do you think about it.

Happy Hacking!

Advertisements

Debugging Python with Visual Studio Code

Debugging Python with Visual Studio Code

I have started using Visual Studio Code, and to be honest, I feel it’s one of the best IDEs in the market. I’m still a Vimmer; given a chance I still use VIM for small edits or carrying out nifty text transformations. After Vim, the next tool that has really impressed me is VSC; the innovations the team are doing, the utility that it provides is almost a super power.

This post is regarding one of the utilities that I have been using very recently. This is a skill that I have been trying to harness for a long time. For every person who writes code there comes a time where they need to figure out what is going wrong;  there’s a need to debug the code.
The most prominent and well used debugging tools are print statements. To be really honest, it doesn’t feel (to me) quite right to use print statements to debug my code, but that’s the most handy way to figure out the flow and inspect each variable. I’ve tried a lot of debuggers and it alway feels like extra effort to actually take a step up and use them. This could be one of the reasons I might have not used them very intensively. (Although I have used pudb extensively.)

But, with VS Code, the way debugger is integrated in really well. It feels very natural to use it. Recently when I was working on few scripts and was trying to debug them, I went on exploring a little more with the python debugger in VS Code.

So I have this script and I want to run the debugger or it. You hit ctrl + alt + p, this opens the the command drop down, just type debug and you will see the option,  Debug and start debugging.

 

Screenshot 2018-06-24 22.45.31

 

This actually creates a launch.json file in your project. You can put all your configuration in here. We’ll edit the config file as we go; since it is not a Django or Flask project we will use the current file configuration. That looks like this:

{

"name":"Python: Current File",

"type":"python",

"request":"launch",

"program":"${file}"

}

You can set the pythonPath here if you are using a virtual environment, name sets the name of the configuration, type is the type of file, that is being debugged it, and  request can be used to debug it in different ways. Let’s make our configs more customised,
{

"name":"Facebook Achieve Debug",

"type":"python",

"request":"launch",

"program": "${flle}"

}
Screenshot 2018-06-25 00.23.42
If you observe there’s a red dot at line 50.  That is called the breakpoint and that is where the program will stop and you will be able to observe variables and see the flow of the program.
Let’s see what the screen looks like when you do that,
Screenshot 2018-06-25 00.34.34
This is the editor in full flow, you could see the stack that is being followed, you can also go and inspect each variable.
With the debug console (lower right pane) you can even run some code that you want to run or to inspect the same. Now, let us look at the final config and see what is going on.

{

 "name":"Python: Current File",

 "type":"python",

 "request":"launch",

"program":"${file}",

 "pythonPath":"/Users/farhaanbukhsh/.virtualenvs/facebook_archieve/bin/python",

 "args":[

    "--msg",

    "messages"

   ]

}

If you observe I have the pythonPath set to my ​virtualenv and I have one more argument which is args which is the command-line  argument that has to be passed to the script.
I still use print statement sometimes but I have made it  a sure point to start using the debugger as early as possible because, believe it or not, this definitely helps a lot and saves time.

The A/V guy’s take on PyCon Pune

The A/V guy’s take on PyCon Pune

“This is crazy!”, that was my reaction at some point in PyCon Pune. This is one of my first conference where I participated in a lot of things starting from the website to audio/video and of course being the speaker. I saw a lot of aspects of how a conference works and where what can go wrong. I met some amazing people, people who impacted my life , people who I will never forget. I received so much of love and affection that I can never express in words.  So before writing anything else I want to thank each and everyone of you , “Thank you!”.

My experience or association started the time when the PyCon Pune was being conceived Sayan asked me if I could volunteer for Droidcon so that I can learn how to handle A/V for PP,  and our friends at HasGeek were generous enough to let me do that. The experience at Droidcon was crazy, I met a lot of people and made crazy lot of friends. Basically me and Haseeb were volunteering to learn the A/V stuff and Karthik was patient enough to walk us through the whole complex set up, to be very honest I didn’t get the whole picture till now but I some how able to manage. I learned a thing or two about  manning the camera and how much work actually goes to record a conference.

Since I was anyhow going to the conference I thought why not to apply for a talk but somehow I knew I wasn’t going to make it reason being the talks got rejected in a lot of other conferences 😛 . But anyhow being my stubborn self I don’t give up on rejection I gathered all the courage and got Vivek involved and we decided to apply for the talk and to my surprise it got in. This was our first conference talk and it was on one of the projects that we really really love, Pagure.

Since these things happened over a large span of time, by the time conference dates came I have nearly got out of touch with the A/V setup I only have vague idea about what is happening. So Sayan who is a one man army stepped in and he assured me that he will help me with getting the setup ready and we turned again to our friends at HasGeek and they were really humble to help us out this time and also help us with the instruments. We literally had a suitcase full of wires in case things go wrong. We spend around 3 days to up skill ourselves to handle the setup but this time the setup was very simple.

After all this happened and Sayan and Chandan took all the instruments to Pune. I arrived at Pune somewhere around two days before the conference the bus that I took from Bangalore to Pune dropped me somewhere near Telegaun which is near to Mumbai than Pune and I somehow managed to get back to Pune and reached Sayan and Chandan’s house. We were bunking together and there were more people about to come. I took some rest and then we were out , first stop was Reserved Bit , oh I can’t forget this place.

It is a perfect place for geeks and I loved every aspect of it. There I met Siddhesh for the first time we have had conversations over IRC though and met Nisha too. Amazing people the whole experience to travel to Reserved Bit and way back was amazing. We went to the venue to checkout where the camera will be and verify various aspects of the venue. After we came back I started working on the setup and man it was very tough and tricky to gather live feed from the camera.

First of all I was little hesitant to use any proprietary software but then I had no option so we somehow found a windows laptop and tried configuring it but almost everytime either we got a “BLUE SCREEN” or “UPDATES” which annoyed me , the sole reason of using windows was because we had a piece of hardware called capture cards, and the driver for which were not available. After long struggle and a lot of digging done by Siddhesh we got driver for Epiphan capture card for Linux and this was around 12 in the night and we all were still there at Reserved Bit. This gave all of us new hope and then it started we kind of got our minimalistic set up and Siddhesh did a “Compiler talk by Angle Fish” , it was a lot of fun by the time we got it working it was somewhere around 4 in the morning. After all this Sayan and Me actually took a walk back home and picked up Subho on the way. The next day CuriousLearner arrived and then Haseeb , Amit and Gaurav.

We were around 10 people squeezed in a single room but without any discomfort we kind of enjoyed our stay with occasional leg pulling to deep intense tech discussion the whole experience was just terrific. Then comes the actual venue setup that was one crazy thing so the video setup was working with Linux , we had Epiphan capture card working on Kernel version below 4.9 and OBS studio as a recording software. I actually spent a good number of hours to install OBS and downgrading kernel to 4.6 so that Epiphan driver works on at least 6 laptops. When we tried the setup on site and it broke because we didn’t take into account the audio from the mic. All of us were stuck in a state of panic then we realized that we have a mixer with us, but its power cord was left at Reserved Bit . By this time this setup kind of became our conference hack and we wanted it to work so badly. We actually ran back to Reserved Bit spent sometime there since we had some work and then quickly came back to the venue, connected the mixer and after few trial and run it worked.

“YES IT WORKED ” our efforts paid off, we recorded the whole conference using this setup, some of the recordings were a little glitchy and one other hack that we added was we weren’t recording the slides from speaker’s laptop we were doing it manually on our laptops. That means one copy of slide was being played on our laptops and we were recording it accordingly.

Apart from this experience I actually got the opportunity to meet all the keynote speaker the first so I met Nick, Honza, Terri,  John, Steven and Praveen. This was another experience in itself to know them and talk to the Rockstars of the FOSS WORLD.

As a speaker Kushal introduced me as the Speaker who is also the Cameraman for the event and that was may be the first time in a tech conference. Vivek and I have been collaborating over the talk for a long time and we figured out the order in which we need to speak and we spoke accordingly we kind of covered all the things that we wanted to and got a great response from the audience. I attended most of the talks since I was The A/V GUY but I had a huge help from rtnpro he was always there humble and ready to help.

The conference came to an end where Nisha told all the people about the effort that was put in from every person and specially Sayan. After this we had two days of devsprint where we had amazing projects, Vivek and I were mentoring for Pagure and we got a lot of new contributors and quite a number of PRs ( 13 to be precise ), the devsprint was a run away success.

I also got chance to interact with mbuf and man I saw him smile and crack jokes for the first time and it was crazy fun ,  I think it was the dinner after the last day of the conference. One of the most amazing experience was to talk to Haris and yes his name is Haris not Harish. The whole experience was so lovely that I don’t think that it can be better than this.

PS: We fixed my Macbook too

PPS: Video of our talk at PyCon Pune

Hacking on Pagure CI

Hacking on Pagure CI

“Ahaa!” I got a lot of ahaa moments when I was hacking on Pagure CI ,  Pagure CI’s initial draft was laid by lsedlar and I have blogged about it followed by me and Pingou. Pingou has done really amazing work with the flow and refactoring of code to making beautiful api calls.

I had great time hacking around it and got a bunch of learning. Few of the learning are :

  1. Try to do the minimal work in setting up the development environment mock everything that is available for testing.
  2. Think deeply about something when your mentor points it to you.

So the issue I was working on is a long pending one the issue was to attach build ID to all the Jenkins build Pagure was getting . Reason why attaching build id’s are necessary is to distinguish between different builds and to make the link to Jenkins a bit more specific for example if a build fail which was that build.

The first mistake I did was setting up Jenkins on my machine I had it previously but since my machine went under a kernel panic I lost all data related to Jenkins , now Fedora 25 has some packaging issue when installing  Jenkins directly. But anyhow from Jenkins site I got a way to set it up and it worked for me. In the mean while Pingou was pointing it out that I actually don’t need Jenkins instance but I was not able to get him on that and I really feel bad about it.

After setting up Jenkins the other task for me was to configure it , which was really easy because I have done it before and also because it was well documented. For setting up the documentation is fine but for hacking on the CI you need a little less work.

Step 1

Set up REDIS on your machine , you can do that with installing redis using sudo dnf install redis and enable the service using sudo systemctl enable redis and then start the service using sudo systemctl start redis. Along with this you need to add config for redis in default_config.py or which ever config file you are giving to the server using --config. The configuration are well documented in pagure.cfg.sample.

Step 2

Now, copy the pagure-ci-server from pagure-ci directory into the parent directory. Now once you have done that , this step is necessary because this is the service that run for pagure-ci. Now you just have to run pagure-ci-server by python pagure-ci-server.py. Once this started your service will be up and running.

Step 3

Now you just fire up your instance and make a project , have two branches and open a PR form once branch to other, if you get some authentication error that is most probably because you not done the right permission for users to use Jenkins, this is not recommended but you can entirely turn off the security of Jenkins just because you are testing something.

If you have done everything correct you will see the Jenkins flag being attached to the Pull Request.

VERY IMPORTANT NOTE:

All this could be saved if I have just used python-jenkins to fetch a job from Fedora Jenkins instance and send it as a flag to my PR. Thank you Pingou for telling me this hack.

Happy Hacking!

Functional Programming 101

“Amazing!”  that was my initial reaction when I heard and read about functional programming , I am very new to the whole concept so I might go a little off while writing about it so I am open to criticism .  This is basically my understanding about functional programming and why I got hooked to it .

Functional Programming is a concept just like Object Oriented Programming , a lot of people confuse these concept and start relating to a particular language , thing that needs to be clear is languages are tools to implement concepts. There is imperative programming where you tell the machine what to do ? For example

  1. Assign x to y
  2. Open a file
  3. Read a file

While when we specifically talk about FP it is a way to tell how to do things ? The nearest example that I can come up with is SQL query  where you say something like

SELECT  * FROM Something where bang=something and bing=something

Here we didn’t tell what to do but we told how to do it. This is what I got as a gist of functional programming where we divide our task into various functional parts and then we tell how things have to be implemented on the data.

Some of the core concepts that I came across was pure functions and functions treated as first class citizen or first class object . What each term means  lets narrow it down .

Pure functions  is a function whose return value is determined by the input given, the best example of pure functions are Math functions for example Math.sqrt(x) will return the same value for same value of x. Keeping in mind that x will never be altered. Lets go on a tangent and see that how this immutability of x is a good thing, this actually prevents data from getting corrupt.  Okay! That is alot to take in one go, lets understand this with a simple borrowed example from the talk I attended.

We will take example of a simple Library System  now for every library system there should be a book store, the book store here is an immutable data structure now what will happen if I want to add a new book to it ? Since it is immutable I can’t modify it , correct ? So a simple solution to this problem is every time I add or remove a book I will actually deliver a new book store and this new book store will replace the old one. That way I can preserve the old data because hey we are creating a whole new store. This is probably the gist or pros of functional programming.


book_store = ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol"]
def add_book( book_store, book):
    new_book_store = []
    map(lambda old_book: new_book_store.append(old_book), book_store)
    new_book_store.append(book)
    return new_book_store

print add_book(book_store, "Inferno") # ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol", "Inferno"]

print book_store # ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol"]

In the above code you can actually see that a new book store is returned on addition of a new book. This is what a pure function looks like.

Function as first class citizens , I can relate a lot to this because of python where we say that everything is a first class objects. So, basically when we say functions are first class citizen we are implying that functions can be assigned to a variable, passed as a parameter and returned from a function. This is way more powerful then it sounds this bring a lot modular behavior to the software you are writing, it makes the project more organized and less tightly coupled. Which is a good thing in case you want to make quick changes or even feature related big changes.


def find_odd(num):
    return num if(num%2 != 0) else None

def find_even(num):
    return num if(num%2 == 0) else None

def filter_function(number_list, function_filter):
    return [num for num in number_list if(function_filter(num) != None)]

number_list = [1,2,3,4,5,6,7,8,9]
print filter_function(number_list, find_odd) # [1,2,5,7,9]
print filter_function(number_list, find_even) # [2,4,6,8]

In the above code you can see that function is passed as an argument to another function.

I have not yet explored into lambda calculus which I am thinking of getting into . There is a lot more power and beauty in functional programming.  I want to keep this post a quick read so I might cover some code example later, but I really want to demonstrate this code.


def fact(n, acc=1):
    return acc if ( n==1 ) else fact(n-1, n*acc)

where acc=1  this is pure textbook and really beautiful code which calculates factorial of n ,  when it comes to FP it is said To iterate is Human, to recurse is Divine. I will leave you to think more about it, will try to keep writing about things I learn.

Happy Hacking!

KICKSTARTERS

KICKSTARTERS

On 4th of this month we the Jnaapti team planned a small workshop to get 9th and 10th standard kids to get them started with programming . We visited Florence Public School and it was really a pleasant  experience to see such fresh minds exercising their mind.

I was really amazed to watch them apfps_kickstartersproach the problems we gave them, we have a rigid mind and we are really bad at thinking out of the box and that is because we have been taught rather conditioned to go in one direction.

But these kids they never knew this problem so they have a very different approach to these problem and I learnt about 7 new ways to approach the famous N-Queens problem.

Then came my favorite part “LABY”laby its a game which teaches programming in a fun way, so the whole idea is to make an ant to reach the door. There are various obstacles that needs to be handled and you know what you are going to use for that ?  CODE what else but its super fun to do that and we could see that in those kids. I learnt a lot that day like how will you explain conditionals , I am telling you it is not as easy as it looks.

Thanks to Gautham Sir, Shreelaksmi ma’m and Abraar for getting me this opportunity to do something this amazing , hope we will do more of such programs.

Vacation Of Code

Vacation Of Code

This all started when pingou gave me the opportunity to write a utility which helps to user to edit comment on pagure.Now for those who don’t know what pagure is , so pagure is open source equivalent of Github, to be precise :

Pagure is a light-weight git-centered forge based on pygit2. Currently, Pagure offers a web-interface for git repositories, a ticket system and possibilities to create new projects, fork existing ones and create/merge pull-requests across or within projects.

So the utility that was allotted to me was an edit button that privilege the owner of the comment to modify the comment on the pull request. That sounds really simple but writing the whole code and modifying others code are two different things.

I finally started exploring Flask and its feature so Flask is framework  that helps you write servers , Server as the name suggest is use to serve request.I had a brief idea about the type of request but never got the chance to explore it.

There are many types of request that help the client and server to talk to each other  famous ones are POST and GET.

The GET method

The GET method is the method used by the browser to ask the server to send back a given resource: “Hey server, I want to get this resource.” In this case, the browser sends an empty body. Because the body is empty, if a form is sent using this method, the data sent to the server is appended to the URL.

The POST method

The POST method is a little different. It’s the method the browser sends the server to ask for a response that takes into account the data provided in the body of the HTTP request: “Hey server, take a look at this data and send me back an appropriate result.” If a form is sent using this method, the data is appended to the body of the HTTP request.

Source: MDN

So in first attempt I wrote a shabby code and somehow make it work , but as I told it was really shabby so pingou and subho told me to use hidden form technique which is used to trigger certain functions without actually making the user know what is going on . It is more of a hack to get things done your way.The problem I faced was delete comment and edit comment use the same form to send request so both have the same end point hence inside the function that deletes comment I devised a way to distinguish how to differentiate when to edit comment and when to delete it.

The algorithm was simple get comment id , get new comment , replace the old comment by new one and make entry in the database.The code snippet that really helped me to understand things to a new depth was:

Example

Consider the following form:

<form action="http://foo.com" method="get">
  <input name="say" value="Hi">
  <input name="to" value="Mom">
  <button>Send my greetings</button>
</form>
With the GET method, the HTTP request looks like this:
GET /?say=Hi&to=Mom
HTTP/1.1 Host: foo.com
Example

Consider this form (the same one as above):

<form action="http://foo.com" method="post">
  <input name="say" value="Hi">
  <input name="to" value="Mom">
  <button>Send my greetings</button>
</form>

When sent using the POST method, the HTTP request looks like this:

POST / HTTP/1.1
Host: foo.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 13

say=Hi&to=Mom
 My Pull request is still under consideration so you can actually check what I did in pagure. Do let me know what you think about it.