Setting Postgres For Pagure

Setting Postgres For Pagure

I normally use Sqlite for development because of the ease you get to see your file , browse through it and edit it. Having said that sqlite is good for development and not for production one of the foremost reason being it doesn’t support multi-thread querying.

The other disadvantage was sqlite doesn’t give a damn if you have dangling Foreign Key references, I land up on this problem recently. The way we categorize fork project in Pagure is on the basis of parent_id so if a project has parent_id its a fork and if it doesn’t then its not a fork.

This works out quite well unless recently we figured out a flaw , what if the main project is deleted, the expected behavior is the fork should be accessible but because of the parent_id  dependency the fork was getting inaccessible this was because as you delete the main repo , the FK references with the fork gets modified and becomes Null.

This creates anomaly because now the project is no more a fork , its a main repo and its treated like it which leads to a lot of repo path chaos. The relation of Postgres came here because I was able to have a dangling FK reference here in sqlite but when I try to achieve the same thing in Postgres it throws an integrity error.

Pagure uses Sqlalchemy as the ORM so I just need to set up postgres on my system and provide the URL in pagure/default_config.py  and ORM magic makes all the queries just work.

Setting up Postgres is really easy because of the amazing documentation provided in fedora-wiki . The only thing you need to care or a little tricky about is you need to be a superuser  before you change to user postgres .  So first sudo su and then su - postgres. Then the follow the steps in the wiki and create a user and create a database name pagure.

Private Repo on Pagure

Private Repo on Pagure

One of my proposal for Pagure was to have private repositories. Private repositories are basically repositories which are visible to you and people you give permission to.

To be honest , I thought it would require a few tweaks and I will be good to go, but that wasn’t the case and the insights I got working on this feature was amazing. I fiddle with this project on primarily  three stages. Each stage was a challenge in its own.

The three stages were:

  1. UI
  2.  Database Query
  3. Tests

UI

The UI  was suppose to have a checkbox saying “Private”  and when a user ticks it the existing project becomes private or the new project is private from the time it is conceived.

Achieving this was a joy ride, with flask I just need to make changes in the form and setting page UI and Voilla!

I introduced a column Private in the project table and that was pretty much it. Nice and beautiful.

DATABASE

This was the most challenging part for me , since I have not worked with databases, and this was out of my comfort zone, I actually went back to my database basics to see if I am doing things right.

We in Pagure use Sqlalchemy as the ORM layer, ORM stands for object relation mapper. It basically use to map databases to object-class model of representing data. Sqlalchemy is a really powerful tool.

While figuring out ways to get all admins who can view private projects , I struggled a lot since I was working with a function which forms the core of Pagure so if things go wrong with this function the whole Project will take a hit.

So the challenge was to make minimum changes which are independent so that it doesn’t compromise the existing functionality and yet able to introduce a new one. I struggle to achieve it I failed a lot of time , was working hard to get it working , constantly moving to the board to figure out a solution on paper. Then switching back to my screen to code it out.

I was so desperate to get this working that I even pinged Armin on IRC to ask my doubt about flask and Sqlalchemy.  All this while the best support I got was from my mentor Pingou.

Finally after struggling a lot I got a very beautiful solution and done !

Just when I thought I am done , there comes a question of writing tests. Since I have altered a very major functionality that means I need to test every aspect of it.

Selection_021

Testing

Testing was a herculean task since I have not done a lot of testing, I actually got a lot to learn for starting the DB used for testing is a in-memory DB and not the one used by the app.

The session maintained has to be replicated in a way to use them in the test and how to use pygit to actually initialize a repo with git init and use it.

Towards the end of this PR my development evolve from writing code and testing it , to write the test and then introduce code or write code that pass the test. It has been really amazing working on this feature and hope it will be integrated soon.

I think may be a little more work is required on this feature maybe. It feels really amazing to do this work.

The link to the branch on Pagure.

The link to the current Pull-Request.

Happy Hacking!

Investigating Python

Investigating Python

I have been trying to implement private projects on Pagure, while doing that I was struggling with certain design of a function and while doing that I constantly have to switch between shell, editor and at times browser.

I am not saying it is a bad thing or  a good thing but this lead me to find a debugger I thought that might ease my task of finding what is going wrong and where and it actually helped.

I used a python debugger called pudb. This looks like Turbo C but its a lot more useful. This can be used in one of the two ways:

  1. You can directly debug a script  pudb <your_script_name>.py
  2.  You may need to call certain function when you are working with big projects in that case you can just put import pudb; pu.db

 

The most beautiful thing is, it just pops an IDE out of no where, it gives a deep insight about what code i s doing, how it is doing , what stack it is maintaining and what are the various values of variables.

You can always set breakpoints so that you can investigate about the code,  you actually play a detective. This is actually an important  point in developing in an opensource project since there are a lot of functions doing a lot of Hocus Pocus.

This is one of those tools that might even help you to understand the code base better , it really helped me to design the code better.

This is the script that I am trying to debug, the screen looks like this and ‘n’ can let you to go to the next line which can be investigated using the stacks shown.

Selection_017

These are the few windows that you can navigate to and see what is going on.

Selection_018

Not only this but you can jump between different modules and you can set breakpoints.

Selection_019

 

This might help you get some cool insight about the project.

Happy Hacking !

Pagure CI

Pagure CI

As my GSoC project one of the first goals is getting CI to Pagure. In my previous post I have been blogging about getting Fedmsg to work, Configuring Jenkins and my favorite Poor Man’s CI. Well, Poor Man’s CI evolved into Slightly Richer Man’s CI and now  Pagure CI.

This is how it started I was trying to test how Poor Man’s CI works and how can I actually integrate it, while doing that I hit a deadend when Poor Man’s CI would not respond to local Pagure instance because there is no GIT server running. So one cannot clone git repo with a normal URL, I landed up on a really amazing utility called Git Daemon, it gave me some hope but it didn’t last long, that also has disadvantage that the URL should be a git:// URL.

I somehow thought I needed to set my Pagure as an application and then I can make common clone commands. I tried using that and I am still in the process of doing it , I am still setting gitolite on that front.

While doing that and because of my lazy nature I found a neat hack and it was , my Poor Man’s CI was running on localhost and so was my Jenkins but the I made a repo on pagure.io.  The only thing to understand here is the Fedmsg should listen to all the msgs and Fedmsg plugin should be enabled in the repository on Pagure.

So, recap you have :

1. fedmsg-relay running and listening to all the production messages.

2.  fedmsg-tail  –really-pretty running to check the message is being read.

3. fedmsg-hub running in Poor man’s CI which is consuming the message and configure        to consume new pull request.

Voilla! it worked and there you go I got a build there and the flag is attached to it.

Selection_016

 

After talking to Pierre we arrived a on conclusion that we need a way to either merge the two application or make Pagure talk to Poor Man’s CI. I had a lot of doubts to achieve it, later Pierre gave me “Food for thought ”  which was

“If we are able to gather all information from Pagure and Poor Man’s CI could get that database access would it not do the deal?”

The way PMCI handles it is the consumer written for fedmsg-hubs parses the message received and according to the message received it triggers Jenkins build and does all the work and it just queries a single table.

Pierre wanted no individual UI for PMCI and so we made it a hook , now a person can enable the hook and add the useful information required. I brought the whole consumer as it is from PCMI. Consumers need to get registered in setup.py and dependencies needs to be build up using python setup.py develop which does a lot of magic behind the scenes.

One of the dependencies is python-jenkins library, which provides various functionality. The whole PMCI is reduced to three files in Pagure. The most important being consumer.py , I gathered a lot of functionalities into a lib/pagure_ci.py and made a model in hooks/jenkins_hook.py.  Since I got lot of functionalities in place the only thing then I struggled with was because of my poor understanding of Sqlalchemy. I read up a lot and was able to debug the code and make it work.

 

The most peculiar thing I came across was even the server needs to be run in the virtualenv, since Jenkins library is installed in that. Once that is done its like a train of dominoes , one thing will lead to another . I am trying it to show you diagrammatically.

hook

Step 2: Configuring Jenkins

Step 2: Configuring Jenkins

Jenkins is one of the major part in setting up Poor Man’s CI. Lets look into how Jenkins can be configured and how can we make it automate task.

Jenkins can be downloaded for the OS you are using from their website. After downloading the mistake that I did was using Jenkins with Global Credentials, as pointed by lsedlar on the channel, because of this I was not able to get the “Trigger by URL” option in the project.

Initial configuration is pointed by lsedlar in his blog. I will be covering extra configuration to have it working for local development. First and foremost being the authentication , this can be done by  Manage Jenkins –> Configure Global Security. Selection_013

 

Give Read, View and Discover to anonymous and add another user and give all the permission to that user. You need to restart Jenkins service.

sudo systemctl restart jenkins.service

On web ui jenkins will ask you to sign in ,  create a user with the username you gave all the permission and log in with the user. Now add New Item  and create a Freestyle Project. Now configure the project , click on  “This build is parameterized” and configure it according to Poor Man CI’s. Once that is done, select the option as shown below:

Selection_014

Once that is done you can use this token to trigger build using a POST request. The trick is you need to pass parameters too with the URL. Next thing is you need to tell Jenkins what to do and where to do. Since we are dealing with Jenkins and git we need a local repo or some URL to the git repo. For every operation carrying out in the repository the directory should have the group and user set to jenkins else you cat just put the repo in /var/lib/jenkins.

Download and install Git Plugin for Jenkins. Once that is done you need to point the git plugin to the repository you are going to test.

Selection_015

Once jenkins know where to perform action you need to tell what to perform this is done in the build section of the configuration and select Execute Shell.


if [ -n "$REPO" -a -n "$BRANCH" ]; then
git remote rm proposed || true
git remote add proposed "$REPO"
git fetch proposed
git checkout origin/master
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
git merge --no-ff "proposed/$BRANCH" -m "Merge PR"
fi

We are almost done, the last thing is we need an auth token for the user. Go to Manage Jenkins –> Manager User. Now get the api token for the user. Make sure that branch you are passing as parameter exists in the repository. Lets trigger the build using cuRL.

USER:fhackdroid

API Token: 728507950f65eec1d77bdc9c2b09e14b

Token: BEEFCAFE

BRANCH:checking

curl -X POST http://fhackdroid:728507950f65eec1d77bdc9c2b09e14b@localhost:8080/job/pagureExp/buildWithParameters\?token\=BEEFCAFE\&REPO\=file:///$\{JENKINS_HOME\}/new_one_three\&BRANCH\=checking\&cause\=200

Part 1: Setting Up Fedmsg

Part 1: Setting Up Fedmsg

I was trying to make Poor man’s CI work for local development. Since it has been written in a way that it has been put to productions directly. So I started to talk about it in fedora-admin channel, I got a lot of input from pingou and lsedlar.

Finally, I tried to divide the task into various parts the first one was making fedmsg work on my system locally. Fedmsg is way in which application talk to each other in Fedora infrastructure. A message contains a lot of information about the changes in the application.

Well setting up was a big problem for me because the version of fedmsg being installed was throwing a segment fault whenever it tried running fedmsg-tail. I pinged Sayan regarding this issue, we worked a lot over it finally the solution we found was using virtualenv and installing a specific stable version of fedmsg which was :

        pip install fedmsg==0.16.4

Now, since fedmsg now got properly set up , fedmsg has to take messages from my local Pagure, fedmsg-relay is a service which binds two ports one where the message is being emitted and the other where it has to be listen to.

If a project doesn’t have a fedmsg.d/<somefile>.py  then fedmsg will take inputs from /etc/fedmsg.d/endpoint.py , in my case fedmsg-relay was using the latter location so I modified that file a bit to get message from my local instance.

In a different terminal we can run fedmsg-tail  –really-pretty to see the messages. Any change being done to any repo in local Pagure instance is now being logged.

Selection_010
fedmsg

With these changes I was able to configure in such a way that I can see my local instance emitting messages on fedmsg.

Tip: Using virtualenv wrapper its an amazing tool to manage various virtualenv. And most of the projects comes with a convention that setup.py has install and develop option.

 

GSoC-2016

GSoC-2016

“Woaaahhh! It is accepted” that was me when I saw my proposal got accepted. I had goosebumps, it’s a very big achievement for me to get through GSoC.

This started way back when I went for the last years BangPypers meetup, I met Sayan there , wearing that Dgplug T-shirt, he gave all the links he could for starting with open source.

I enrolled myself for Dgplug Summer Training, and that was it, I learnt so many things and got amazing supportive mentors like Sayan, Kushal and Pierre(pingou).

My connection with fedora projects started from then and there, and cut back now I am into GSoC because of Pagure. This makes me feel so humble and excited , it is just that you need to keep moving and working hard , things will fall in place.

FSMK has been a great support for me, introducing me to FOSS and telling us to spread the movement. Vignesh has been a constant support and motivator for me , he has been someone who actually pushed me beyond my limits to make things work.

Mentors at Jnaapti, Gautham Pai Sir and Shreelakshmi Ma’m, they have been a constant source of inspiration, they have seen  the urge to learn in me and they have been working on me to make me better.

I don’t know what to say, I am just overwhelmed with the result.

I will be spending my summer working on Pagure and making it the best code reviewing system out there.

I will put the proposal link as soon as it is uploaded to fedora-wiki.

EDIT: The link to fedora-wiki proposal .