The A/V guy’s take on PyCon Pune

The A/V guy’s take on PyCon Pune

“This is crazy!”, that was my reaction at some point in PyCon Pune. This is one of my first conference where I participated in a lot of things starting from the website to audio/video and of course being the speaker. I saw a lot of aspects of how a conference works and where what can go wrong. I met some amazing people, people who impacted my life , people who I will never forget. I received so much of love and affection that I can never express in words.  So before writing anything else I want to thank each and everyone of you , “Thank you!”.

My experience or association started the time when the PyCon Pune was being conceived Sayan asked me if I could volunteer for Droidcon so that I can learn how to handle A/V for PP,  and our friends at HasGeek were generous enough to let me do that. The experience at Droidcon was crazy, I met a lot of people and made crazy lot of friends. Basically me and Haseeb were volunteering to learn the A/V stuff and Karthik was patient enough to walk us through the whole complex set up, to be very honest I didn’t get the whole picture till now but I some how able to manage. I learned a thing or two about  manning the camera and how much work actually goes to record a conference.

Since I was anyhow going to the conference I thought why not to apply for a talk but somehow I knew I wasn’t going to make it reason being the talks got rejected in a lot of other conferences 😛 . But anyhow being my stubborn self I don’t give up on rejection I gathered all the courage and got Vivek involved and we decided to apply for the talk and to my surprise it got in. This was our first conference talk and it was on one of the projects that we really really love, Pagure.

Since these things happened over a large span of time, by the time conference dates came I have nearly got out of touch with the A/V setup I only have vague idea about what is happening. So Sayan who is a one man army stepped in and he assured me that he will help me with getting the setup ready and we turned again to our friends at HasGeek and they were really humble to help us out this time and also help us with the instruments. We literally had a suitcase full of wires in case things go wrong. We spend around 3 days to up skill ourselves to handle the setup but this time the setup was very simple.

After all this happened and Sayan and Chandan took all the instruments to Pune. I arrived at Pune somewhere around two days before the conference the bus that I took from Bangalore to Pune dropped me somewhere near Telegaun which is near to Mumbai than Pune and I somehow managed to get back to Pune and reached Sayan and Chandan’s house. We were bunking together and there were more people about to come. I took some rest and then we were out , first stop was Reserved Bit , oh I can’t forget this place.

It is a perfect place for geeks and I loved every aspect of it. There I met Siddhesh for the first time we have had conversations over IRC though and met Nisha too. Amazing people the whole experience to travel to Reserved Bit and way back was amazing. We went to the venue to checkout where the camera will be and verify various aspects of the venue. After we came back I started working on the setup and man it was very tough and tricky to gather live feed from the camera.

First of all I was little hesitant to use any proprietary software but then I had no option so we somehow found a windows laptop and tried configuring it but almost everytime either we got a “BLUE SCREEN” or “UPDATES” which annoyed me , the sole reason of using windows was because we had a piece of hardware called capture cards, and the driver for which were not available. After long struggle and a lot of digging done by Siddhesh we got driver for Epiphan capture card for Linux and this was around 12 in the night and we all were still there at Reserved Bit. This gave all of us new hope and then it started we kind of got our minimalistic set up and Siddhesh did a “Compiler talk by Angle Fish” , it was a lot of fun by the time we got it working it was somewhere around 4 in the morning. After all this Sayan and Me actually took a walk back home and picked up Subho on the way. The next day CuriousLearner arrived and then Haseeb , Amit and Gaurav.

We were around 10 people squeezed in a single room but without any discomfort we kind of enjoyed our stay with occasional leg pulling to deep intense tech discussion the whole experience was just terrific. Then comes the actual venue setup that was one crazy thing so the video setup was working with Linux , we had Epiphan capture card working on Kernel version below 4.9 and OBS studio as a recording software. I actually spent a good number of hours to install OBS and downgrading kernel to 4.6 so that Epiphan driver works on at least 6 laptops. When we tried the setup on site and it broke because we didn’t take into account the audio from the mic. All of us were stuck in a state of panic then we realized that we have a mixer with us, but its power cord was left at Reserved Bit . By this time this setup kind of became our conference hack and we wanted it to work so badly. We actually ran back to Reserved Bit spent sometime there since we had some work and then quickly came back to the venue, connected the mixer and after few trial and run it worked.

“YES IT WORKED ” our efforts paid off, we recorded the whole conference using this setup, some of the recordings were a little glitchy and one other hack that we added was we weren’t recording the slides from speaker’s laptop we were doing it manually on our laptops. That means one copy of slide was being played on our laptops and we were recording it accordingly.

Apart from this experience I actually got the opportunity to meet all the keynote speaker the first so I met Nick, Honza, Terri,  John, Steven and Praveen. This was another experience in itself to know them and talk to the Rockstars of the FOSS WORLD.

As a speaker Kushal introduced me as the Speaker who is also the Cameraman for the event and that was may be the first time in a tech conference. Vivek and I have been collaborating over the talk for a long time and we figured out the order in which we need to speak and we spoke accordingly we kind of covered all the things that we wanted to and got a great response from the audience. I attended most of the talks since I was The A/V GUY but I had a huge help from rtnpro he was always there humble and ready to help.

The conference came to an end where Nisha told all the people about the effort that was put in from every person and specially Sayan. After this we had two days of devsprint where we had amazing projects, Vivek and I were mentoring for Pagure and we got a lot of new contributors and quite a number of PRs ( 13 to be precise ), the devsprint was a run away success.

I also got chance to interact with mbuf and man I saw him smile and crack jokes for the first time and it was crazy fun ,  I think it was the dinner after the last day of the conference. One of the most amazing experience was to talk to Haris and yes his name is Haris not Harish. The whole experience was so lovely that I don’t think that it can be better than this.

PS: We fixed my Macbook too

PPS: Video of our talk at PyCon Pune

Hacking on Pagure CI

Hacking on Pagure CI

“Ahaa!” I got a lot of ahaa moments when I was hacking on Pagure CI ,  Pagure CI’s initial draft was laid by lsedlar and I have blogged about it followed by me and Pingou. Pingou has done really amazing work with the flow and refactoring of code to making beautiful api calls.

I had great time hacking around it and got a bunch of learning. Few of the learning are :

  1. Try to do the minimal work in setting up the development environment mock everything that is available for testing.
  2. Think deeply about something when your mentor points it to you.

So the issue I was working on is a long pending one the issue was to attach build ID to all the Jenkins build Pagure was getting . Reason why attaching build id’s are necessary is to distinguish between different builds and to make the link to Jenkins a bit more specific for example if a build fail which was that build.

The first mistake I did was setting up Jenkins on my machine I had it previously but since my machine went under a kernel panic I lost all data related to Jenkins , now Fedora 25 has some packaging issue when installing  Jenkins directly. But anyhow from Jenkins site I got a way to set it up and it worked for me. In the mean while Pingou was pointing it out that I actually don’t need Jenkins instance but I was not able to get him on that and I really feel bad about it.

After setting up Jenkins the other task for me was to configure it , which was really easy because I have done it before and also because it was well documented. For setting up the documentation is fine but for hacking on the CI you need a little less work.

Step 1

Set up REDIS on your machine , you can do that with installing redis using sudo dnf install redis and enable the service using sudo systemctl enable redis and then start the service using sudo systemctl start redis. Along with this you need to add config for redis in default_config.py or which ever config file you are giving to the server using --config. The configuration are well documented in pagure.cfg.sample.

Step 2

Now, copy the pagure-ci-server from pagure-ci directory into the parent directory. Now once you have done that , this step is necessary because this is the service that run for pagure-ci. Now you just have to run pagure-ci-server by python pagure-ci-server.py. Once this started your service will be up and running.

Step 3

Now you just fire up your instance and make a project , have two branches and open a PR form once branch to other, if you get some authentication error that is most probably because you not done the right permission for users to use Jenkins, this is not recommended but you can entirely turn off the security of Jenkins just because you are testing something.

If you have done everything correct you will see the Jenkins flag being attached to the Pull Request.

VERY IMPORTANT NOTE:

All this could be saved if I have just used python-jenkins to fetch a job from Fedora Jenkins instance and send it as a flag to my PR. Thank you Pingou for telling me this hack.

Happy Hacking!

PyCon India 2016

PyCon India 2016

Day 0

“This is awesome!”, this was my first reaction when I boarded my first flight to Delhi. I was having trouble in finding a proper accommodation Kushal, Sayan and Chandan helped me a lot in that part, I finally got honour of  bunking with Sayan , Subho and Rtnpro which I will never forget. So, I landed and directly went to JNU convention center. I met the whole  Red Hat intern gang . It was fun to meet them all. I had proposed Pagure for Dev Sprint and I pulled in Vivek to do the same.

The dev sprint started and there was no sign of Vivek or Saptak, Saptak is FOSSASIA contributor and Vivek  contributes to Pagure with me. Finally it was my turn to talk about Pagure on stage , it was beautiful  the experience and the energy.  We got a lot of young and new contributors and we tried to guide them and make them send at least one PR.  One of them was lucky enough to actually make a PR and it got readily merged.

I met a lot of other contributors and other mentors and each and every project was simply amazing. I wish I could help all of them some day. We also met Paul, who writes code for PyCharm, we had a nice discussion over Vim v/s PyCharm.

Finally the day ended with us Vivek, Sayan , Subho  , Saptak and me going out to grab some dinner. I bunked with Sayan and Subho and we hacked all night. I was configuring my Weechat and was trying all the plugins available and trust me there are a lot of them.

Day 1

I was a session chair in one of the lecture room and it was a crazy experience from learning to write a firmware for a drone, using generators to write multi-threaded program and also to work with salt stack. The food was really good but the line for food was equally “pythonic” as the code should be.

There were a lot of stalls put up and I went to all of them and had a chat with them. My favorite one was PyCharm because Paul promised me to teach me some neat tricks to use PyCharm.

The Redhat and Pyladies booth were also there which also were very informative and they were responsible making people aware about certain social issues and getting women in tech.

We had two keynotes on this day one by BG and the other by VanL and trust me both of the keynotes were so amazing the they make you look technology from a different view point altogether.

One of the amazing part of such conferences are Open Space and Lightning talks. There are few open spaces which I attended and I found them really enthralling. I was waiting for the famous Stair case meeting of Dgplug.  We met Kushal’s mentor, Sartaj and he gave a deep insight in what and why we should contribute to open source. He basically told us that even if one’s code is not used by anyone he will still be writing code for the love of doing it.

After this we went for Dgplug/Volunteers  dinner at BBQ nation, it was an eventful evening 😉 to be modest.

Day 2 

The last day of conference I remember myself wondering how a programming language translates into philosophy and how that philosophy unites a diverse nation like India. The feeling was amazing but I could sense the sadness. The sadness of parting from friends who meet once in an year. I could actually now relate all IRC nicks with their faces. It just brings a lot more on the table.

At last we all went to the humdrum of our normal life with the promise to meet again. But I still wonder how a technology bring comradeship between people from all nook and corners of life. How it relates from a school teacher to a product engineer . T his makes  me feel that this is more than just a programming language , this is that unique medium that unites people and give them power to make things right.

With this thought fhackdroid signs out!

Happy Hacking!

Weechat-Tmux

Weechat-Tmux

Recently I have been to pycon-india (will blog about that too!) there Sayan and Vivek introduced me to weechat which is a terminal based IRC client, from the time I saw Sayan’s weechat configuration I was hooked to it.

The same night I started configuring my weechat , it’s such a beautiful IRC client I was regretting why did I not use it before. It just transforms your terminal into IRC window.

For fedora you need to do:

sudo dnf install weechat

Some of the configuration and plugins you need are :

  1. buffer
  2. notify-send

That’s pretty much it but that doesn’t stop there you can make that client little more aesthetic.  You can set weechat by using their documentation.

The clean design kind of makes you feel happy, plus adding plugin is not at all a pain. In the weechat window you just say /script install buffer.pl and it just installs it in no time.  There are various external plugin in case you want to use them and writing plugin is actually fun , I have not tried that yet.

screenshot-from-2016-09-30-23-02-13

I also use to use bigger font but now I find this size more soothing to eyes. It is because of weechat I got to know or explore about this beautiful tool called tmux ,  because on normal terminal screen weechat lags , what I mean by lag is the keystroke somehow reach after like 5-6 seconds which makes the user experience go bad.  I pinged people on IRC in #weechat channel with the query the community is amazing they helped me to set it up and use it efficiently , they only told me to use tmux or screen . With tmux my session are persistent and without any lag.

To install tmux on fedora:

sudo install tmux

tmux is a terminal multiplexer which means it can extend one terminal screen into many screen . I got to learn a lot of concepts in tmux like session, pane and windows. Once you know these things in tmux its really a funride. Some of the blogs I went through for configuring and using tmux the best I found was hamvoke , the whole series is pretty amazing . So basically my workflow goes for every project I am working on I have a tmux session named after it, which is done by the command:

tmux new-session -s <name_session>

Switching between two session can be done by attach and detach. And I have one constant session running of weechat. I thought I have explored every thing in tmux but that can’t be it , I came to know that there is a powerline for tmux too. That makes it way more amazing so this is how a typical tmux session with powerline looks like.

screenshot-from-2016-09-30-23-31-10

I am kind of loving the new setup and enjoying it. I am also constantly using tmux cheatsheet 😛 because it’s good to look up what else you can do and also I saw various screencast on youtube where  tmux+vim makes things amazing.

Do let me know how you like my setup or how you use it .

Till then, Happy Hacking! 🙂

 

GSoC: Final Submission

GSoC: Final Submission

This summer has been really amazing, I learnt a lot and worked crazy hours it has been a crazy yet amazing ride. I am not going to stop working on open source projects and with Pagure it is something really close to my heart.

There are a few things left but I can conclude that I am able to achieve what I wanted to at the beginning of this program , but there is never a feeling of satisfaction it is just like you want to achieve the best possible and most beautiful solution.

Pagure has CI integration which was one of my major goals to achieve and with the coming release it will be out and will be usable to people. This gives me immense pleasure to say that the foundation of CI was laid by me although Pingou kind of wrote a lot after that but that helped me to learn the depth of thinking one needs to have when you are working on a feature like this. Selection_027

I also worked on Private Repo feature which took more time than expected and it was pretty challenging to achieve , this feature is in  feature branch and it may get merged after it is checked in the staging first. Selection_028

It was so challenging that I got stuck on a data fetching problem from the database , we use Sqlalchemy as ORM in Pagure. I went through a lot of ups and downs at times I was about to give up but then I get some small part of it and Pingou has been so amazing mentor he never spoon fed me instead he asked the right question the moment he ask something the idea bulb use to glow.

I still remember struggling with  Xapian and Whoosh. This was again a very big task and still is , it requires a lot of time to optimize it to a level where it doesn’t slow the site. I gave a lot of time on it but since I few other goals and various issue to solve so I eventually moved on to those just to come back.

Pagure pages is one of the last goals that I worked on recently and there are discussion pending over it.

At a glance I was able to achieve a lot of the big goals on my proposal and still work has to be done, and I will continue to work on achieving various other goals. Few links that I want to share :

Commits made to the master branch

Commits on private-repo branch on pagure 

Pull-request for static page hosting

This kinds of makes me feel happy that I have around 102 commits on the master branch now and I believing I will be working a lot more on Pagure to bring a lot of cool and useful feature to it. In case you have any suggestion feel free to file issues on Pagure.

To be really frank I am not at all sad that GSoC is getting because I have received so much love and inspiration from Fedora Community that contributing to projects has actually become my daily routine the day I don’t commit code, review patches or comment on issues I start feeling something is missing .

And some of my fellow GSoCers said That’s all folks!  😉

Happy Hacking!

 

Docs in Pagure

Docs in Pagure

I took this week to hack on this feature called Docs which gives you the ability to host documentation of your project in Pagure. I have never explored this feature before so I started to hack on it .

This feature is pretty straight forward to use. Once you have your project up and running you can go to Setting of the project and under  Project Option  click on Activate Documentation this will actually activate a Doc tab in the main project. This can be used to host your docs specifically now this is a little tricky because you need to clone and push to a different URL, the docs are maintain in a separate location this is due to security concerns. When you activate the Project option you are provided with a Doc specific URL, you need to push your document or static pages to that URL and automatically any page named as index will be taken as the first page.

Selection_026

You have to click on the more button beside GIT URLs to get your Docs URL and then you are good to go to host your static page.

For people who want to hack on Docs in Pagure you need to pull a few tricks to do that.

First and foremost is you need to get the code from pagure.io and then after setting up Pagure for development, you need to run two servers :

  1. Pagure server
  2. Doc server

The script corresponding to them are  runserver and rundocserver.

So if you have ever hacked on Pagure then you will know that you have to log in make repo  and follow the same steps mentioned above to see the Doc tab.

Under pagure/default_conf.py a new conf key has to be added which is

DOC_APP_URL = 'https://localhost:5001'

This tells Pagure that this instance supports Doc.

Now comes the tricky part, if you need to see Doc there should be a <project_name>.git created in the docs repo which is not there you just need to copy the file from repo directory to docs. Once this is done you need to clone the project repo from docs delete all the files there put the files you want in the static page , we support a lot of formats like md, rst etc. Add, commit and push and voila you will see them in your local instance.

I am actually working on issue 469 in which Ryan has suggested to make docs more specific to static page  hosting with the architecture that docs is based on this is actually a straight forward task but a really beautiful one which need a bit of deliberation on things we want to achieve.  Hope it gave you insight in what I am trying to to do.

More documentation on this can be found in the usage section of Pagure Docs.

Happy Hacking! 🙂

Search for Code in Pagure

Search for Code in Pagure

I was trying to get into code search in Pagure, thing that I land up on got really interesting and amazing.  If you want to have a code searching mechanism in your website you need to look into something called Indexing.

The way search happens in some E-commerce sites like Amazon or be it the search happening on Google, with Google its web scrapping and then indexing on the results. The point being the response time , while you are searching for something you get results in few microseconds.

Now imagine going through such a huge database and going through them in few micro second how much ever power you have but what you need is a clever way to manage it. I was looking at a CS50 video in which Mark Zuckerberg was telling about how he managed his DB, the first architectural design he took was have different MySql instance for different school so that they reduce time taken to search and form relation.

That was a really clever move.

While I was searching for ways to have code search feature on Pagure, I landed up on a pyhton based library called Whoosh. It blew me off with the way it was doing its searches and maintaining the database. I actually looked for a lot for tutorials on how one can understand indexing.

I landed up on Building Search Engines using Python and the way he explained things like N-grams , edge N-grams and how different files store different index words with the frequency and path to documents. I am yet to analyze git grep v/s whoosh.

While I was going through whoosh I saw that it has performance issues and then I started contemplating on the fact that if search is not fast enough then there is no point in having it. I actually looked into HyperKitty I figured out they were using Whoosh before and I assumed even they suffered form performance issues or may be because Django introduce Haystack . As the name suggest you can also use this to find the needle in haystack.

Yeah! you are right, I started looking for Haystack in Flask and I found Flask-whoosh. Again the draw back I had was it use to search through databases and not files, where as my application was to search through files on the system

There came the xapian there are a lot of core concepts involved while using or writing utilities in xapian. I went through the documentation for Xapian. They have covered a lot of concepts and have given examples of it, the bottleneck still persist when it comes to file searching and performance. I found a nice application Building Document Search which might give me some hope but still a lot of work is required there.

The whole concept being you need to do two things on a really high level:

  1. Indexing
  2. Search

Indexing

Indexing is required to go through the each file or record and build something called Index which has the search words filtering  stop words and the new database is build having the frequency and location of the word , this is the most time consuming process.

Search

This comprises of forming a query and searching through the formulated database and return the document in which word or phrase is found.

If you need to see a demo.

Till then Happy Coding an Bingo!