Writing Chuck – Joke As A Service

Writing Chuck – Joke As A Service

Recently I really got interested to learn Go, and to be honest I found it to be a beautiful language. I personally feel that it has that performance boost factor from a static language background and easy prototype and get things done philosophy from dynamic language background.

The real inspiration to learn Go was these amazing number of tools written and the ease with which these tools perform although they seem to be quite heavy. One of the good examples is Docker. So I thought I would write some utility for fun, I have been using fortune, this is a Linux utility which gives random quotes from a database. I thought let me write something similar but let me do something with jokes, keeping this mind I was actually searching for what can I do and I landed up on jokes about Chuck Norris or as we say it facts about him. I landed up on chucknorris.io they have an API which can return different jokes about Chuck, and there it was my opportunity to put something up and I chose Go for it.

JSON PARSING

The initial version of the utility which I put together was way simple, it use to make a GET request stream the data in put in the given format and display the joke. But even with this implementation I learnt a lot of things, the most prominent one was how a variable is exported in Go i.e how can it be made available across scope and how to parse a JSON from a received response to store the beneficial information in a variable.

// Chucknorris is the struct used to unmarshal the JSON response from the URL
type Chucknorris struct {
Category []string `json:"category"`
IconURL string `json:"icon_url"`
ID string `json:"id"`
URL string `json:"url"`
Value string `json:"value"`
}
/* getJokes takes the API url as the parameter and fetch jokes from it,
* here we have assumed it to be of ChuckNorris type
*/
//TODO: Remove the dependency on the hardcoded struct
func getJokes(URL string) (string, error) {
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
return "", fmt.Errorf("No request formed %v", err)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("No response: %v", err)
}
defer resp.Body.Close()
respData, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("Read error")
}
var joke Chucknorris
if err = json.Unmarshal(respData, &joke); err != nil {
return "", fmt.Errorf("Error in unmarsheling, %v", err)
}
return joke.Value, nil
}
view raw initial.go hosted with ❤ by GitHub

Now the mistake I was doing with the above code is I was declaring the fields of the struct with a small letters this caused a problem because although the value get stored in the struct I can’t use them outside the function I have declared it in. I actually took a while to figure it out and it was really nice to actually learn about this. I actually learnt about how to make a GET request and parse the JSON and use the given values.

Let’s walk through the code, the initial part is a struct and I have few fields inside it, the Category field is a slice of string, which can have as many elements as it receives the interesting part is the way you can specify the key from the received JSON how the value of received JSON is stored in the variable or the field of the struct. You can see the json:"categories" that is the way to do it.

With the rest of the code if you see I am making a GET request to the given URL and if the it returns a response it will be res and if it returns an error it will be handled by err. The key part here is how marshaling and unmarshaling of JSON takes place.

This is basically folding and un-folding JSON once that is done and the values are stored to retrieve the value we just use a dot notation and done. There is one more interesting part if you see we passed &joke which if you have a C background you will realize is passing the memory address, pass by reference, is what you are looking at.

This was working good and I was quite happy with it but there were two problems I faced:

  1. The response use to take a while to return the jokes
  2. It doesn’t work without internet

So I showed it to Sayan and he suggested why not to build a joke caching mechanism this would solve both the problems since jokes will be stored internally on the file system it will take less time to fetch and there is no dependency on the internet except the time you are caching jokes.

So I designed the utility in a way that you can cache as may number of jokes as you want you just have to run chuck --index=10 this will cache 10 jokes for you and will store it in a Database. Then from those jokes a random joke is selected and is shown to you.

I learnt to use flag in go and also how to integrate a sqlite3 database in the utility, the best learning was handling files, so my logic was anytime you are caching you should have a fresh set of jokes so when you cache I completely delete the database and create a new one for the user. To do this I need to check of the Database is already existing and if it is then remove it. I landed up looking for the answer on how to do that in Go, there are a bunch of inbuilt APIs which help you to do that but they were misleading for me. There is os.Stat, os.IsExist and os.IsNotExist. What I understood is os.Stat will give me the status of the file, while the other two can tell me if the file exists or it doesn’t, to my surprise things don’t work like that. The IsExist and IsNotExist are two different error wrapper and guess what not of IsExist is not IsNotExist, good luck wrapping your head around it. I eventually ended up answering this on stackoverflow.

After a few iteration of using it on my own and fixing few bugs the utility is ready except the fact that it is missing test cases which I will soon integrate, but this has helped me learn Go a lot and I have something fun to suggest to people. Well, I am open to contribution and hope you will enjoy this utility as much as I do.

Here is a link to chuck!

Give it a try and till then Happy Hacking and Write in GO! 

Featured Image: https://gopherize.me/

The Open Organization

I was recently going through few of the Farnam Street articles, and I landed on the article on how to read a book, where they basically describe how to read a book;  the fact that there are types of books, and the fact that books can, in the words of Francis Bacon “be gulped, some books chewed and others digested.”

This basically signifies the intensity and the level of awareness to have when you are reading a book. I have gulped lots of books, but The Open Organization is one of those, that I wanted to chew on.

I wanted to learn about how you can build an ecosystem where people are free to voice their opinions, where failure is be worn as a badge of honor for trying. This book filled me with thoughts of how would it be like, if an organization is really an Open Organization.

There are a lots of beautiful anecdotes that I came across, and a lot of values that were given in the book to think on.

The book talks about Purpose and Passion. People specially us Millenials,have been spoiled to an extent that we actually don’t run after money but after a purpose, after a problem. We don’t mind working crazy hours and being paid peanuts, but we do care about people, we care about how are we treated, we care about the problem we are after. One of the quotes in the book says Basis of loyalty is a common purpose and not economic dependency. A lot of people I know believe in this. When you unite with an organization which is after the same problem as you, it’s a match made in heaven.

The book talks about Passion, the passion about doing good, making a dent in the universe, but sometimes you realize Universe doesn’t give a damn .

One of the most amazing analogies, is when the book compares a structure of an organization with web architecture which is end to end and not center to end. Where there is no central point of control but there should be a central point of co-ordination. The organization is lead by leaders it selects, where Meritocracy is the idea behind every decision.

The other idea that was completely new to me was the difference between Crowd-sourcing and Open-sourcing.To be honest I had not thought open source to be a business model until the recent past. The thing with the wisdom of the crowd is that it works amazingly well when the work can be easily disagregated and individuals can work in relative isolation. I love the point in the book that says members of the organization should be inspired by the leader and not motivated. Motivation is something they already have and that is the reason they are joining your organization. I love this idea a lot because I have seen people complaining about their employees not being motivated enough. I think that this (lack of inspired leadership) is a reason.

“Great companies don’t hire skilled people and motivate them, they hire already motivated people and inspire them.” – Simon Sinek

I really enjoyed the way the power of purpose is laid out in the book. The other idea was the idea of Meritocracy.  I think of  merit as having an amazing idea and idea being the sole reason for doing a certain action. Better ideas win, they are questioned and deliberated upon and that is how innovation happens in the organization. People debate over it, question it, trash it. People just don’t settle for something to avoid conflict. That very same complacency however is what has creeped into organizations where people don’t debate ideas just to avoid conflict so that everyone remains happy. It was so amazing to read stories where someone thought out of the box and wanted to bring in a new way of doing things and how he convinced everyone that this is the right way of doing things, we ought to give it a try.

This book pushes back on the belief in hierarchy and brings to limelight lateral structure, letting people know that the conventional ways of running an organization might have to change, upgrade as it were, to a newer version.

I got a lot of amazing ideas and to be honest I got to know how a person in an organization should be treated. I was awestruck with the insights in the book. Wish someday I could mould an organization in this way. Theories are always romantic, hope the execution and implementation is beautiful as well.

The A/V guy’s take on PyCon Pune

The A/V guy’s take on PyCon Pune

“This is crazy!”, that was my reaction at some point in PyCon Pune. This is one of my first conference where I participated in a lot of things starting from the website to audio/video and of course being the speaker. I saw a lot of aspects of how a conference works and where what can go wrong. I met some amazing people, people who impacted my life , people who I will never forget. I received so much of love and affection that I can never express in words.  So before writing anything else I want to thank each and everyone of you , “Thank you!”.

My experience or association started the time when the PyCon Pune was being conceived Sayan asked me if I could volunteer for Droidcon so that I can learn how to handle A/V for PP,  and our friends at HasGeek were generous enough to let me do that. The experience at Droidcon was crazy, I met a lot of people and made crazy lot of friends. Basically me and Haseeb were volunteering to learn the A/V stuff and Karthik was patient enough to walk us through the whole complex set up, to be very honest I didn’t get the whole picture till now but I some how able to manage. I learned a thing or two about  manning the camera and how much work actually goes to record a conference.

Since I was anyhow going to the conference I thought why not to apply for a talk but somehow I knew I wasn’t going to make it reason being the talks got rejected in a lot of other conferences 😛 . But anyhow being my stubborn self I don’t give up on rejection I gathered all the courage and got Vivek involved and we decided to apply for the talk and to my surprise it got in. This was our first conference talk and it was on one of the projects that we really really love, Pagure.

Since these things happened over a large span of time, by the time conference dates came I have nearly got out of touch with the A/V setup I only have vague idea about what is happening. So Sayan who is a one man army stepped in and he assured me that he will help me with getting the setup ready and we turned again to our friends at HasGeek and they were really humble to help us out this time and also help us with the instruments. We literally had a suitcase full of wires in case things go wrong. We spend around 3 days to up skill ourselves to handle the setup but this time the setup was very simple.

After all this happened and Sayan and Chandan took all the instruments to Pune. I arrived at Pune somewhere around two days before the conference the bus that I took from Bangalore to Pune dropped me somewhere near Telegaun which is near to Mumbai than Pune and I somehow managed to get back to Pune and reached Sayan and Chandan’s house. We were bunking together and there were more people about to come. I took some rest and then we were out , first stop was Reserved Bit , oh I can’t forget this place.

It is a perfect place for geeks and I loved every aspect of it. There I met Siddhesh for the first time we have had conversations over IRC though and met Nisha too. Amazing people the whole experience to travel to Reserved Bit and way back was amazing. We went to the venue to checkout where the camera will be and verify various aspects of the venue. After we came back I started working on the setup and man it was very tough and tricky to gather live feed from the camera.

First of all I was little hesitant to use any proprietary software but then I had no option so we somehow found a windows laptop and tried configuring it but almost everytime either we got a “BLUE SCREEN” or “UPDATES” which annoyed me , the sole reason of using windows was because we had a piece of hardware called capture cards, and the driver for which were not available. After long struggle and a lot of digging done by Siddhesh we got driver for Epiphan capture card for Linux and this was around 12 in the night and we all were still there at Reserved Bit. This gave all of us new hope and then it started we kind of got our minimalistic set up and Siddhesh did a “Compiler talk by Angle Fish” , it was a lot of fun by the time we got it working it was somewhere around 4 in the morning. After all this Sayan and Me actually took a walk back home and picked up Subho on the way. The next day CuriousLearner arrived and then Haseeb , Amit and Gaurav.

We were around 10 people squeezed in a single room but without any discomfort we kind of enjoyed our stay with occasional leg pulling to deep intense tech discussion the whole experience was just terrific. Then comes the actual venue setup that was one crazy thing so the video setup was working with Linux , we had Epiphan capture card working on Kernel version below 4.9 and OBS studio as a recording software. I actually spent a good number of hours to install OBS and downgrading kernel to 4.6 so that Epiphan driver works on at least 6 laptops. When we tried the setup on site and it broke because we didn’t take into account the audio from the mic. All of us were stuck in a state of panic then we realized that we have a mixer with us, but its power cord was left at Reserved Bit . By this time this setup kind of became our conference hack and we wanted it to work so badly. We actually ran back to Reserved Bit spent sometime there since we had some work and then quickly came back to the venue, connected the mixer and after few trial and run it worked.

“YES IT WORKED ” our efforts paid off, we recorded the whole conference using this setup, some of the recordings were a little glitchy and one other hack that we added was we weren’t recording the slides from speaker’s laptop we were doing it manually on our laptops. That means one copy of slide was being played on our laptops and we were recording it accordingly.

Apart from this experience I actually got the opportunity to meet all the keynote speaker the first so I met Nick, Honza, Terri,  John, Steven and Praveen. This was another experience in itself to know them and talk to the Rockstars of the FOSS WORLD.

As a speaker Kushal introduced me as the Speaker who is also the Cameraman for the event and that was may be the first time in a tech conference. Vivek and I have been collaborating over the talk for a long time and we figured out the order in which we need to speak and we spoke accordingly we kind of covered all the things that we wanted to and got a great response from the audience. I attended most of the talks since I was The A/V GUY but I had a huge help from rtnpro he was always there humble and ready to help.

The conference came to an end where Nisha told all the people about the effort that was put in from every person and specially Sayan. After this we had two days of devsprint where we had amazing projects, Vivek and I were mentoring for Pagure and we got a lot of new contributors and quite a number of PRs ( 13 to be precise ), the devsprint was a run away success.

I also got chance to interact with mbuf and man I saw him smile and crack jokes for the first time and it was crazy fun ,  I think it was the dinner after the last day of the conference. One of the most amazing experience was to talk to Haris and yes his name is Haris not Harish. The whole experience was so lovely that I don’t think that it can be better than this.

PS: We fixed my Macbook too

PPS: Video of our talk at PyCon Pune

Functional Programming 101

“Amazing!”  that was my initial reaction when I heard and read about functional programming , I am very new to the whole concept so I might go a little off while writing about it so I am open to criticism .  This is basically my understanding about functional programming and why I got hooked to it .

Functional Programming is a concept just like Object Oriented Programming , a lot of people confuse these concept and start relating to a particular language , thing that needs to be clear is languages are tools to implement concepts. There is imperative programming where you tell the machine what to do ? For example

  1. Assign x to y
  2. Open a file
  3. Read a file

While when we specifically talk about FP it is a way to tell how to do things ? The nearest example that I can come up with is SQL query  where you say something like

SELECT  * FROM Something where bang=something and bing=something

Here we didn’t tell what to do but we told how to do it. This is what I got as a gist of functional programming where we divide our task into various functional parts and then we tell how things have to be implemented on the data.

Some of the core concepts that I came across was pure functions and functions treated as first class citizen or first class object . What each term means  lets narrow it down .

Pure functions  is a function whose return value is determined by the input given, the best example of pure functions are Math functions for example Math.sqrt(x) will return the same value for same value of x. Keeping in mind that x will never be altered. Lets go on a tangent and see that how this immutability of x is a good thing, this actually prevents data from getting corrupt.  Okay! That is alot to take in one go, lets understand this with a simple borrowed example from the talk I attended.

We will take example of a simple Library System  now for every library system there should be a book store, the book store here is an immutable data structure now what will happen if I want to add a new book to it ? Since it is immutable I can’t modify it , correct ? So a simple solution to this problem is every time I add or remove a book I will actually deliver a new book store and this new book store will replace the old one. That way I can preserve the old data because hey we are creating a whole new store. This is probably the gist or pros of functional programming.


book_store = ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol"]
def add_book( book_store, book):
    new_book_store = []
    map(lambda old_book: new_book_store.append(old_book), book_store)
    new_book_store.append(book)
    return new_book_store

print add_book(book_store, "Inferno") # ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol", "Inferno"]

print book_store # ["Da Vinci's Code", "Angles and Demons", "The Lost Symbol"]

In the above code you can actually see that a new book store is returned on addition of a new book. This is what a pure function looks like.

Function as first class citizens , I can relate a lot to this because of python where we say that everything is a first class objects. So, basically when we say functions are first class citizen we are implying that functions can be assigned to a variable, passed as a parameter and returned from a function. This is way more powerful then it sounds this bring a lot modular behavior to the software you are writing, it makes the project more organized and less tightly coupled. Which is a good thing in case you want to make quick changes or even feature related big changes.


def find_odd(num):
    return num if(num%2 != 0) else None

def find_even(num):
    return num if(num%2 == 0) else None

def filter_function(number_list, function_filter):
    return [num for num in number_list if(function_filter(num) != None)]

number_list = [1,2,3,4,5,6,7,8,9]
print filter_function(number_list, find_odd) # [1,2,5,7,9]
print filter_function(number_list, find_even) # [2,4,6,8]

In the above code you can see that function is passed as an argument to another function.

I have not yet explored into lambda calculus which I am thinking of getting into . There is a lot more power and beauty in functional programming.  I want to keep this post a quick read so I might cover some code example later, but I really want to demonstrate this code.


def fact(n, acc=1):
    return acc if ( n==1 ) else fact(n-1, n*acc)

where acc=1  this is pure textbook and really beautiful code which calculates factorial of n ,  when it comes to FP it is said To iterate is Human, to recurse is Divine. I will leave you to think more about it, will try to keep writing about things I learn.

Happy Hacking!

PyCon India 2016

PyCon India 2016

Day 0

“This is awesome!”, this was my first reaction when I boarded my first flight to Delhi. I was having trouble in finding a proper accommodation Kushal, Sayan and Chandan helped me a lot in that part, I finally got honour of  bunking with Sayan , Subho and Rtnpro which I will never forget. So, I landed and directly went to JNU convention center. I met the whole  Red Hat intern gang . It was fun to meet them all. I had proposed Pagure for Dev Sprint and I pulled in Vivek to do the same.

The dev sprint started and there was no sign of Vivek or Saptak, Saptak is FOSSASIA contributor and Vivek  contributes to Pagure with me. Finally it was my turn to talk about Pagure on stage , it was beautiful  the experience and the energy.  We got a lot of young and new contributors and we tried to guide them and make them send at least one PR.  One of them was lucky enough to actually make a PR and it got readily merged.

I met a lot of other contributors and other mentors and each and every project was simply amazing. I wish I could help all of them some day. We also met Paul, who writes code for PyCharm, we had a nice discussion over Vim v/s PyCharm.

Finally the day ended with us Vivek, Sayan , Subho  , Saptak and me going out to grab some dinner. I bunked with Sayan and Subho and we hacked all night. I was configuring my Weechat and was trying all the plugins available and trust me there are a lot of them.

Day 1

I was a session chair in one of the lecture room and it was a crazy experience from learning to write a firmware for a drone, using generators to write multi-threaded program and also to work with salt stack. The food was really good but the line for food was equally “pythonic” as the code should be.

There were a lot of stalls put up and I went to all of them and had a chat with them. My favorite one was PyCharm because Paul promised me to teach me some neat tricks to use PyCharm.

The Redhat and Pyladies booth were also there which also were very informative and they were responsible making people aware about certain social issues and getting women in tech.

We had two keynotes on this day one by BG and the other by VanL and trust me both of the keynotes were so amazing the they make you look technology from a different view point altogether.

One of the amazing part of such conferences are Open Space and Lightning talks. There are few open spaces which I attended and I found them really enthralling. I was waiting for the famous Stair case meeting of Dgplug.  We met Kushal’s mentor, Sartaj and he gave a deep insight in what and why we should contribute to open source. He basically told us that even if one’s code is not used by anyone he will still be writing code for the love of doing it.

After this we went for Dgplug/Volunteers  dinner at BBQ nation, it was an eventful evening 😉 to be modest.

Day 2 

The last day of conference I remember myself wondering how a programming language translates into philosophy and how that philosophy unites a diverse nation like India. The feeling was amazing but I could sense the sadness. The sadness of parting from friends who meet once in an year. I could actually now relate all IRC nicks with their faces. It just brings a lot more on the table.

At last we all went to the humdrum of our normal life with the promise to meet again. But I still wonder how a technology bring comradeship between people from all nook and corners of life. How it relates from a school teacher to a product engineer . T his makes  me feel that this is more than just a programming language , this is that unique medium that unites people and give them power to make things right.

With this thought fhackdroid signs out!

Happy Hacking!

Weechat-Tmux

Weechat-Tmux

Recently I have been to pycon-india (will blog about that too!) there Sayan and Vivek introduced me to weechat which is a terminal based IRC client, from the time I saw Sayan’s weechat configuration I was hooked to it.

The same night I started configuring my weechat , it’s such a beautiful IRC client I was regretting why did I not use it before. It just transforms your terminal into IRC window.

For fedora you need to do:

sudo dnf install weechat

Some of the configuration and plugins you need are :

  1. buffer
  2. notify-send

That’s pretty much it but that doesn’t stop there you can make that client little more aesthetic.  You can set weechat by using their documentation.

The clean design kind of makes you feel happy, plus adding plugin is not at all a pain. In the weechat window you just say /script install buffer.pl and it just installs it in no time.  There are various external plugin in case you want to use them and writing plugin is actually fun , I have not tried that yet.

screenshot-from-2016-09-30-23-02-13

I also use to use bigger font but now I find this size more soothing to eyes. It is because of weechat I got to know or explore about this beautiful tool called tmux ,  because on normal terminal screen weechat lags , what I mean by lag is the keystroke somehow reach after like 5-6 seconds which makes the user experience go bad.  I pinged people on IRC in #weechat channel with the query the community is amazing they helped me to set it up and use it efficiently , they only told me to use tmux or screen . With tmux my session are persistent and without any lag.

To install tmux on fedora:

sudo install tmux

tmux is a terminal multiplexer which means it can extend one terminal screen into many screen . I got to learn a lot of concepts in tmux like session, pane and windows. Once you know these things in tmux its really a funride. Some of the blogs I went through for configuring and using tmux the best I found was hamvoke , the whole series is pretty amazing . So basically my workflow goes for every project I am working on I have a tmux session named after it, which is done by the command:

tmux new-session -s <name_session>

Switching between two session can be done by attach and detach. And I have one constant session running of weechat. I thought I have explored every thing in tmux but that can’t be it , I came to know that there is a powerline for tmux too. That makes it way more amazing so this is how a typical tmux session with powerline looks like.

screenshot-from-2016-09-30-23-31-10

I am kind of loving the new setup and enjoying it. I am also constantly using tmux cheatsheet 😛 because it’s good to look up what else you can do and also I saw various screencast on youtube where  tmux+vim makes things amazing.

Do let me know how you like my setup or how you use it .

Till then, Happy Hacking! 🙂

 

GSoC: Final Submission

GSoC: Final Submission

This summer has been really amazing, I learnt a lot and worked crazy hours it has been a crazy yet amazing ride. I am not going to stop working on open source projects and with Pagure it is something really close to my heart.

There are a few things left but I can conclude that I am able to achieve what I wanted to at the beginning of this program , but there is never a feeling of satisfaction it is just like you want to achieve the best possible and most beautiful solution.

Pagure has CI integration which was one of my major goals to achieve and with the coming release it will be out and will be usable to people. This gives me immense pleasure to say that the foundation of CI was laid by me although Pingou kind of wrote a lot after that but that helped me to learn the depth of thinking one needs to have when you are working on a feature like this. Selection_027

I also worked on Private Repo feature which took more time than expected and it was pretty challenging to achieve , this feature is in  feature branch and it may get merged after it is checked in the staging first. Selection_028

It was so challenging that I got stuck on a data fetching problem from the database , we use Sqlalchemy as ORM in Pagure. I went through a lot of ups and downs at times I was about to give up but then I get some small part of it and Pingou has been so amazing mentor he never spoon fed me instead he asked the right question the moment he ask something the idea bulb use to glow.

I still remember struggling with  Xapian and Whoosh. This was again a very big task and still is , it requires a lot of time to optimize it to a level where it doesn’t slow the site. I gave a lot of time on it but since I few other goals and various issue to solve so I eventually moved on to those just to come back.

Pagure pages is one of the last goals that I worked on recently and there are discussion pending over it.

At a glance I was able to achieve a lot of the big goals on my proposal and still work has to be done, and I will continue to work on achieving various other goals. Few links that I want to share :

Commits made to the master branch

Commits on private-repo branch on pagure 

Pull-request for static page hosting

This kinds of makes me feel happy that I have around 102 commits on the master branch now and I believing I will be working a lot more on Pagure to bring a lot of cool and useful feature to it. In case you have any suggestion feel free to file issues on Pagure.

To be really frank I am not at all sad that GSoC is getting because I have received so much love and inspiration from Fedora Community that contributing to projects has actually become my daily routine the day I don’t commit code, review patches or comment on issues I start feeling something is missing .

And some of my fellow GSoCers said That’s all folks!  😉

Happy Hacking!

 

Docs in Pagure

Docs in Pagure

I took this week to hack on this feature called Docs which gives you the ability to host documentation of your project in Pagure. I have never explored this feature before so I started to hack on it .

This feature is pretty straight forward to use. Once you have your project up and running you can go to Setting of the project and under  Project Option  click on Activate Documentation this will actually activate a Doc tab in the main project. This can be used to host your docs specifically now this is a little tricky because you need to clone and push to a different URL, the docs are maintain in a separate location this is due to security concerns. When you activate the Project option you are provided with a Doc specific URL, you need to push your document or static pages to that URL and automatically any page named as index will be taken as the first page.

Selection_026

You have to click on the more button beside GIT URLs to get your Docs URL and then you are good to go to host your static page.

For people who want to hack on Docs in Pagure you need to pull a few tricks to do that.

First and foremost is you need to get the code from pagure.io and then after setting up Pagure for development, you need to run two servers :

  1. Pagure server
  2. Doc server

The script corresponding to them are  runserver and rundocserver.

So if you have ever hacked on Pagure then you will know that you have to log in make repo  and follow the same steps mentioned above to see the Doc tab.

Under pagure/default_conf.py a new conf key has to be added which is

DOC_APP_URL = 'https://localhost:5001'

This tells Pagure that this instance supports Doc.

Now comes the tricky part, if you need to see Doc there should be a <project_name>.git created in the docs repo which is not there you just need to copy the file from repo directory to docs. Once this is done you need to clone the project repo from docs delete all the files there put the files you want in the static page , we support a lot of formats like md, rst etc. Add, commit and push and voila you will see them in your local instance.

I am actually working on issue 469 in which Ryan has suggested to make docs more specific to static page  hosting with the architecture that docs is based on this is actually a straight forward task but a really beautiful one which need a bit of deliberation on things we want to achieve.  Hope it gave you insight in what I am trying to to do.

More documentation on this can be found in the usage section of Pagure Docs.

Happy Hacking! 🙂

Search for Code in Pagure

Search for Code in Pagure

I was trying to get into code search in Pagure, thing that I land up on got really interesting and amazing.  If you want to have a code searching mechanism in your website you need to look into something called Indexing.

The way search happens in some E-commerce sites like Amazon or be it the search happening on Google, with Google its web scrapping and then indexing on the results. The point being the response time , while you are searching for something you get results in few microseconds.

Now imagine going through such a huge database and going through them in few micro second how much ever power you have but what you need is a clever way to manage it. I was looking at a CS50 video in which Mark Zuckerberg was telling about how he managed his DB, the first architectural design he took was have different MySql instance for different school so that they reduce time taken to search and form relation.

That was a really clever move.

While I was searching for ways to have code search feature on Pagure, I landed up on a pyhton based library called Whoosh. It blew me off with the way it was doing its searches and maintaining the database. I actually looked for a lot for tutorials on how one can understand indexing.

I landed up on Building Search Engines using Python and the way he explained things like N-grams , edge N-grams and how different files store different index words with the frequency and path to documents. I am yet to analyze git grep v/s whoosh.

While I was going through whoosh I saw that it has performance issues and then I started contemplating on the fact that if search is not fast enough then there is no point in having it. I actually looked into HyperKitty I figured out they were using Whoosh before and I assumed even they suffered form performance issues or may be because Django introduce Haystack . As the name suggest you can also use this to find the needle in haystack.

Yeah! you are right, I started looking for Haystack in Flask and I found Flask-whoosh. Again the draw back I had was it use to search through databases and not files, where as my application was to search through files on the system

There came the xapian there are a lot of core concepts involved while using or writing utilities in xapian. I went through the documentation for Xapian. They have covered a lot of concepts and have given examples of it, the bottleneck still persist when it comes to file searching and performance. I found a nice application Building Document Search which might give me some hope but still a lot of work is required there.

The whole concept being you need to do two things on a really high level:

  1. Indexing
  2. Search

Indexing

Indexing is required to go through the each file or record and build something called Index which has the search words filtering  stop words and the new database is build having the frequency and location of the word , this is the most time consuming process.

Search

This comprises of forming a query and searching through the formulated database and return the document in which word or phrase is found.

If you need to see a demo.

Till then Happy Coding an Bingo!

Setting Postgres For Pagure

Setting Postgres For Pagure

I normally use Sqlite for development because of the ease you get to see your file , browse through it and edit it. Having said that sqlite is good for development and not for production one of the foremost reason being it doesn’t support multi-thread querying.

The other disadvantage was sqlite doesn’t give a damn if you have dangling Foreign Key references, I land up on this problem recently. The way we categorize fork project in Pagure is on the basis of parent_id so if a project has parent_id its a fork and if it doesn’t then its not a fork.

This works out quite well unless recently we figured out a flaw , what if the main project is deleted, the expected behavior is the fork should be accessible but because of the parent_id  dependency the fork was getting inaccessible this was because as you delete the main repo , the FK references with the fork gets modified and becomes Null.

This creates anomaly because now the project is no more a fork , its a main repo and its treated like it which leads to a lot of repo path chaos. The relation of Postgres came here because I was able to have a dangling FK reference here in sqlite but when I try to achieve the same thing in Postgres it throws an integrity error.

Pagure uses Sqlalchemy as the ORM so I just need to set up postgres on my system and provide the URL in pagure/default_config.py  and ORM magic makes all the queries just work.

Setting up Postgres is really easy because of the amazing documentation provided in fedora-wiki . The only thing you need to care or a little tricky about is you need to be a superuser  before you change to user postgres .  So first sudo su and then su - postgres. Then the follow the steps in the wiki and create a user and create a database name pagure.