Showing posts with label open source. Show all posts
Showing posts with label open source. Show all posts
Mar 11, 2011

Have you tried to access Zimbra or Google Calendar from command line? I have. And I couldn't find any normal command line client that would be able to read and write these calendars, display alerts etc. Well there is a googlecl project , but it's specific for Google Calendar and is not using standard WebDav iCal access methods.

Thus I set out to create console application that would fulfil my needs. What are my requirements?:

  • Read/write access to Google Calendar and Zimbra (at least)
  • Multiple remote calendars
  • Working alerts
  • Nice ncurses UI (but also ability to just display some info and quit)
  • Correct handling of timezones
  • Integration with mail client (open ics files received by email)
  • I guess a lot more :-)

I had a look at existing python modules that work on iCalendar, WebDAV and combination of both. There are quite a few of them, but I just didn't like their APIs. They were usually complex and required knowledge of iCal specification. So I decided to create simplified module that would be easy to understand (even if not so powerful).

I named the project pywebcal (yes, unimaginative) and it's now on github. I would LOVE some input. I know it's far from perfect (or complete), but let's see. For now it offers read-only support for Zimbra (Google should work too but I haven't tested in a while).

You can have a look at the example directory that contains one simple example you can run in-place and see if it works :-) I did my best to create proper test cases covering problems with timezones and whatnot, and this helped me quite a lot with recent refactoring. I am now using vobject library as my backend and it is rather nice to use. Plan is to allow access to vobject components so that my simplified API is not preventing some advanced modifications.

Next step is obviously to start working on ncurses application itself. Anyone wants to help?


Share/Save/Bookmark
Dec 10, 2010

Automatic squashing of last git commits

I have written before about my workflow in Fedora. This workflow includes a relatively high number of rebasing where I squash last two commits into one. I use it to quickly refine and test patches. My history then usually looks something like this:
$ git log --format=oneline --decorate=no | head -3
2db8eacd7f7c20be88824caae5f5af16b9520d34 temp
443bb0f019c87f0090bb9da295a019c0eee23729 Add conditional BRs to enable ff merge between f14
b4c602f9f044598544cff3d68710f68b9447ea0f Fix installation of pom files for artifact jars
  
Where 443bb0f is my first attempt at the fix and 2db8eac is a fixed fix :-). Before pushing this into upstream repository I usually squash last two commits to look like this:
$ git log --format=oneline --decorate=no | head -2
f852c3e260d21bbc642f861e6fa6ea62caa7b69b Add conditional BRs to enable ff merge between f14
b4c602f9f044598544cff3d68710f68b9447ea0f Fix installation of pom files for artifact jars
  
I used to do it manually, but I do it often enough it made sense to automate this. So without further ado:
#!/bin/sh

export EDITOR="sed -i '2s/pick/squash/;/# This is the 2nd commit message:/,$ {d}'"

git rebase -i HEAD~2
Save this somewhere inside your $PATH and chmod +x (or set up shell alias). Then just running this will automatically squash last two commits, using HEAD~1 commit message and discarding last commit message. No warranties though :-)

Edit: Benjamin suggested to use "fixup" instead of "squash". This is new thing in git 1.7+. For more information on this see this blogpost.

Share/Save/Bookmark
Nov 11, 2010
I recently encountered weird problems with my network connection at home. Everything worked, but was unbelievably slooooow. Ping showed times of ~30 ms, but I could easily see it took more time for those packets to go there and back.

I took me some time to figure out what was happening. Looking back, checking DNS server should have been one of the first things to do. Seems like first DNS server provided by my provider has been down. That meant that every DNS query timed out and then went to the second DNS which got me my response. For some reason ping did DNS query before every new package being sent. That explains its weird behaviour.

This problem got me to finally install local caching dns. I was thinking about doing it before, but I never got around to do it until now. I always thought it's gonna be a few-hour nightmare. Now I blame my previous experience with bind :-D For simple local caching bind would be overkill, so I chose dnsmasq. Using it was as simple as installing, running dnsmasq and executing
$ echo 'nameserver 127.0.0.1' > /etc/resolv.conf.head

From that point on every resolv.conf file generated by dhcpcd will have my local DNS as first DNS server to try. For this time you can add it there manually. Then you can verify your setup works by running following command twice in a row:
$ dig randomserver.com
First execution should have Query time: XX msec with XX being few tens of miliseconds. Query time for second run should be zero or very close to zero.

Congratulations. You have your very own caching server. Who knows...maybe you'll even notice some improvements in your network connection :-)

Share/Save/Bookmark
Nov 10, 2010

PyQTrailer revisited

Some time ago, I wrote about my little project: apple.com trailer downloader. Apple is still not very open-source friendly as far as its trailer website is concerned. So all points I made in my original post still stand. To my surprise this little project is still alive and kicking, with new ideas for improvements coming and coming :-). What is even more important: it seems that so far no breakage happened due to apple changing something on the web.

Since the first version I released almost 6 months ago several new features appeared. Some of them include:
  • Parallel downloading of trailers
  • Ability to run movie player (mplayer, vlc, etc) without downloading file to HDD
  • Lot of customisation/performance options added
  • Working support for trailer search
  • Localisation support
  • Python 3 support
Latest version (0.5.2) is available in Gentoo repositories already, and should hit Fedora updates in next day or so (this will be delayed due to new package acceptance criteria though). Enjoy.

Share/Save/Bookmark
Oct 20, 2010
As I was recently adding support for Python 3 to my little trailer downloader application that I mentioned before (PyQTrailer) I encountered a strange problem with PyQt4 that only occurred in Python 3.

Let's take this simple python example:
$ python
>>> from PyQt4.QtCore import QString
>>>
That same code snippet doesn't work in python3 interpreter though:
$ python3
>>> from PyQt4.QtCore import QString
Traceback (most recent call last):
  File "", line 1, in 
ImportError: cannot import name QString
>>>
My first instinct was: Bug! Gentoo PyQt4 ebuild was doing something terrible and somehow made PyQt4 unusable in python3 interpreter. Turns out my gut instinct was wrong (once again :-) ).

PyQt4 since version 4.6 changed API of QString and QVariant for python 3.x. For QString this is due to fact that from Python 3.0, string literals are unicode objects (no need for u'unicode' magic anymore). This means that you can use ordinary Python strings in place for QString. But I wanted my QString for something like this:
  ...
  downloadClicked = pyqtSignal((QString, ))
  ...
This snippet creates Qt signals that you can then emit. Question is... How can we update this for Python 3.x? We could probably just replace QString with type(""), but for a change that wouldn't work with Python 2.x. So? Python dynamic nature to the rescue!
Edit: simplified QString definition (thanks Arfrever)
try:
    from PyQt4.QtCore import QString
except ImportError:
    # we are using Python3 so QString is not defined
    QString = str
If we put previous code sample to the beginning of our Python file we can use QString in our code and it will keep working both in Python 3.x and Python 2.x. Case closed dear Watson.

Share/Save/Bookmark

Fedora RPG - Three level badge system?

I stumbled upon one great idea on Fedora Planet. It is nothing other than Fedora RPG!

In short, it's a system to create characters similar as they are in Role-playing games (RPGs) with levels, skill points and more.

You might think it doesn't make sense to give contributors "points" for non-gaming activities but you would be wrong. Most communities have created ways to reward their members this way. Look no further than my favourite stackoverflow.com. It also uses badge and skill point system for various actions on the website. In one of earliest blog posts about how stackoverflow will work, Jeff Atwood shared his vision: three levels of badges (bronze, silver, gold). Each level with unique badges tailored to the purpose of stackoverflow.

I guess Fedora RPG will go a bit further in this regard. I would love to know how it will all turn out and how the levels will work. Let the games begin!


Share/Save/Bookmark
Aug 28, 2010
I have been using Digikam for managing my photos for some time. It's pretty neat software and development is progressing quite fast. You can think of it as Lightroom, just without the neat non-destructive editing. But that is coming too, thanks to Google and another round of Summer of Code participants. But I wouldn't be writing this blog post if Digikam was flawless would I? :-)

First I'll describe my work-flow in few short bullet-points :-)
  • Shoot a LOT of photos
  • Delete almost as many photos
  • Geotag, tag/keyword and rate remaining photos
  • If it was party or something similar upload to Facebook right away
  • Pick few photos and improve them a bit with Gimp (nothing fancy, just crop/levels)
  • Upload all photos to Flickr as a backup/sharing place
Digikam enables me to work like this, except the last point. Why? Because apparently Digikam developers don't think anyone would update their photos to Flickr without first resizing/recompressing them. This  how flickr export dialog looks like this:


See the problem? Even if I don't chose "Resize photos before uploading" Digikam will still re-compress them which is a Really-Bad-Thing(tm) to do with jpeg files. I had some previous experience with Qt3 and even Qt4. so I though it might be a good idea to look into fixing this small annoyance. I will not bore you with the details how I checked out svn repository with git and rest of the stuff. Here is the result:


If you un-check "Send original file (no resizing)" original settings will appear and you can resize/recompress as much as you want (Blasphemy! Madness!). The patch is not flawless, because it won't prevent you from trying to upload RAW files to Flickr, but it's good enough for me :-) It's not even that big, stats are like this:

 flickrexport/flickrtalker.cpp |   18 ++++++++++++------
 flickrexport/flickrtalker.h   |    2 +-
 flickrexport/flickrwidget.cpp |   34 ++++++++++++++++++++++++++--------
 flickrexport/flickrwidget.h   |    5 +++++
 flickrexport/flickrwindow.cpp |    4 ++++
 flickrexport/flickrwindow.h   |    1 +
 6 files changed, 49 insertions(+), 15 deletions(-)

You can download the patch from my Dropbox for now until the bugreport I created some time ago will get sorted out (don't hold your breath too much though). The patch applies cleanly across all versions of kipi-plugins I tried (from 0.8.something to 1.4.0). Happy uploading.



Share/Save/Bookmark
May 10, 2010
I love films. All of them to be exact. I believe you just have to be in the right mood and you would enjoy even few of the worst movies ever made. Even though I am a proponent of open source philosophy, we as a society are obviously not ready to embrace it in entertainment industry just yet.

This is where www.apple.com/trailers comes into play. Apple made great deals with movie studios and you can watch/download newest movie trailers. Well...sort of. Apple employs variety of restrictions which makes this site next to useless on a Linux desktop. It hides links to trailers themselves behind reference files so that when you download with your favorite browser, you will only get small reference file not the trailer itself. And that is after you circumvent user-agent protection. Because apple believes nothing but iTunes/iPad/iOtherAppleStuff should access these trailers. There are scripts around that can make downloading possible for Linux users. I have been using Apple Trailer Download script for Greasemonkey for quite some time, but it always stopped working after some time.

Another opportunity for me I guess. I have been trying to improve my Python-fu for some time so what better way then a small project like this? I started last weekend after I found out Apple actually publishes JSON data of trailers on its site. This made access quite easy from python and is quite error-prone to changes of website itself (as long as Apple doesn't pull whole JSON thingy...but they are actively using it too). Long story short...there are two outputs from this endeavor:
  • pytrailer - python module to simplify access to movies on apple.com/trailers
  • pyqtrailer - Qt4 interface that displays poster, movie information and enables downloading of trailers
You can report bugs on respective websites (there are quite a few now, but basic downloading for hd trailers works). If you want to try it out just running:
# easy_install pyqtrailer
should work as long as you have PyQt4 installed. You can just run pyqtrailer now and you should see something like this:


That's it. I will improve/fix it a bit but don't expect too much :-)

Share/Save/Bookmark
Mar 27, 2010
What better way to celebrate summer solstice, than by making my computer able to hibernate? Since my last post a lot has happened with me. I got a new phone (HTC Hero FTW!), I finished university, went traveling a bit and I also got a new notebook (because the old one died on me). R.I.P. Thinkpad R51, welcome Thinkpad T500. There are several things I could start writing about now. Starting with how great Hero and Android is to use all the way to today's blog post: How to make my computer hibernate?

Linux has had support for hibernating for quite a few years now and although it's not perfect, it usually works out of the box. What it needs however is swap device big enough so that it can store image of memory for hibernating. Now I hit a problem. When I got my new Thinkpad I thought "Hey, I have 4GB of RAM...why would I need a swap?". And even if I REALLY needed more than 4GB RAM I can still create temporary swap by using swapfile. Unfortunately I couldn't make swapfile on LVM work with TuxOnIce. TuxOnIce also has another alternative to swap or swapfile for hibernating: Using filewriter, which is quite similar to swapfile support, I managed to get it to work (after some work, kernel debugging and one small patch to TuxOnIce).

I set FilewriterLocation in hibernate.conf to point to a place where I wanted to store hibernation file and I set the size to 4GB. As instructed in TuxOnIce HOWTO, I then ran
hibernate --no-suspend
to create this image. It created the file as expected, but when it was supposed to tell me settings for bootloader (resume argument) it silently failed. When I tried again, whole computer froze. I was puzzled. How could this happen? I am using Linux so things like this don't happen! But hey, I should be able to figure out what's wrong with it right? I set up my kernel to include netconsole, and ran hibernate again. This time I caught where the bug happened. The output was something like this:

TuxOnIce: No image found.
BUG: unable to handle kernel paging request at 6539207a
IP: [<c10763a6>] toi_attr_store+0x186/0x2a0
*pdpt = 0000000032732001 *pde = 0000000000000000
Oops: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/power/tuxonice/file/target
Modules linked in: netconsole aes_i586 aes_generic radeon ttm drm_kms_helper drm
i2c_algo_bit sco bnep ipt_MASQUERADE iptable_nat nf_nat ipt_LOG nf_conntrack_ip
v4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_t
ables x_tables rfcomm l2cap vboxnetadp vboxnetflt vboxdrv arc4 iwlagn iwlcore ma
c80211 sdhci_pci snd_hda_codec_conexant sdhci pcmcia e1000e uvcvideo mmc_core cf
g80211 snd_hda_intel yenta_socket btusb rsrc_nonstatic tpm_tis pcspkr pcmcia_cor
e videodev v4l1_compat intel_agp wmi agpgart tpm snd_hda_codec tpm_bios video fu
se xfs raid10 raid1 raid0 md_mod scsi_wait_scan sbp2 ohci1394 ieee1394 usbhid uh
ci_hcd usb_storage ehci_hcd usbcore sr_mod sg uvesafb cfbfillrect cfbimgblt cn c
fbcopyarea [last unloaded: microcode]

Pid: 12870, comm: hibernate Not tainted 2.6.33.1-w0rm #16 2082BRG/2082BRG
EIP: 0060:[<c10763a6>] EFLAGS: 00010202 CPU: 0
EIP is at toi_attr_store+0x186/0x2a0
EAX: 00000000 EBX: 36203430 ECX: 00000000 EDX: f231f200
ESI: 65392066 EDI: 00f60062 EBP: f6006331 ESP: f62a7f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process hibernate (pid: 12870, ti=f62a6000 task=f20a0270 task.ti=f62a6000)
Stack:
00000000 fffffff4 00000001 c1790ca0 00000000 f6e8ab64 c16c75a4 f6d1c380
<0> f62a7f64 c114298d 00000015 00000015 b7709000 f21385c0 f6d1c394 c16c75a4
<0> f6ec7ac0 f21385c0 b7709000 00000015 f62a7f8c c10f207c f62a7f98 00000000
Call Trace:
[<c114298d>] ? sysfs_write_file+0x9d/0x100
[<c10f207c>] ? vfs_write+0x9c/0x180
[<c11428f0>] ? sysfs_write_file+0x0/0x100
[<c10f221d>] ? sys_write+0x3d/0x70
[<c1002ccc>] ? sysenter_do_call+0x12/0x22
Code: c7 45 e0 00 00 00 00 3b 5d 08 0f 85 e9 fe ff ff 8b 46 20 85 c0 0f 84 de fe
ff ff ff d0 8b 7d e0 85 ff 8d 76 00 0f 84 d9 fe ff ff <8b> 46 14 31 d2 e8 60 03
05 00 8b 46 10 c7 46 14 00 00 00 00 a8
EIP: [<c10763a6>] toi_attr_store+0x186/0x2a0 SS:ESP 0068:f62a7f14
CR2: 000000006539207a
---[ end trace 124a5ee29ef71277 ]---

So what can we deduce from this bug output? Let's go from the top. Bug name (unable to handle kernel paging request) means that it is likely a memory corruption issue. Someone accessed memory that he was not supposed to. IP tells us that function where the error occurred was toi_attr_store in unknown file, unknown line (I don't have debug information included in kernel). There are other information we can get from that output, but I didn't really need them. Quick search through kernel sources told me that toi_attr_store is a function inside kernel/power/tuxonice_sysfs.c. I scanned the code, learning what approximately it did. Then I placed printk statements thorough the function so that I could approximate where inside the function the code fails. After some time I narrowed it down to following snippet:

if (!result)
result = count;

/* Side effect routine? */
if (result == count && sysfs_data->write_side_effect)
sysfs_data->write_side_effect();

/* Free temporary buffers */
if (assigned_temp_buffer) {
toi_free_page(31,
(unsigned long) sysfs_data->data.string.variable);
sysfs_data->data.string.variable = NULL;
}


Kernel crashed when it tried to call toi_free_page. After a few reboots and printks later I found out that this was just a coincidence, and sysfs_data variable itself became corrupt even before the call to the toi_free_page. Good candidate? Of course: write_side_effect. But what exactly was write_side_effect? This function was passed as an argument, and therefore I wasn't able to easily find out what was real code executed at this point. Time to find out! From my previous debugging attempts I knew code failed while it tried to write location of my resume file into /sys/power/tuxonince/file/target. TuxOnIce code defined handling for string sysfs arguments as such:

#define SYSFS_STRING(_name, _mode, _string, _max_len, _flags, _wse) { \
.attr = {.name = _name , .mode = _mode }, \
.type = TOI_SYSFS_DATA_STRING, \
.flags = _flags, \
.data = { .string = { .variable = _string, .max_length = _max_len } }, \
.write_side_effect = _wse }


I found this macro used inside tuxonice_file.c source code like this:
 
SYSFS_STRING("target", SYSFS_RW, toi_file_target, 256,
SYSFS_NEEDS_SM_FOR_WRITE, test_toi_file_target)


So we found our write_side_effect code inside test_toi_file_target function. In one part this function was calling hex_dump_to_buffer to convert device UUID into hexadecimal string. The call looked like this:
 
hex_dump_to_buffer(fs_info->uuid, 16, 32, 1, buf, 50, 0);


This should convert input (fs_info->uuid) into hexadecimal string and store it inside buf. Author of the original code correctly thought about function adding spaces between bytes and therefore need to have more space in the buffer (argument 50 is telling hex_dump_to_buffer how big is output buffer). Unfortunately that same author declared buf as 33 char array. hex_dump_to_buffer therefore stepped outside the buffer and corrupted memory, causing all the problems. I fixed this bug, and sent a patch to the tuxonice-devel mailing list. As of now, it is already in the git repository ready to be released with next bugfix release of TuxOnIce.

That is everything for today, but as I already noted I am using LVM on my system (except root partition) and also use fbsplash for nice animations while rebooting. I am using initrd for this, and I will have another post on that topic.

Share/Save/Bookmark
Aug 25, 2009
So this year's Google Summer of Code is officially over. Today 19:00 UTC was deadline for sending in evaluations for both mentors and students. Therefore I think some kind of summary what was happening and what I was doing is in order.

I was working on implementing neat idea that would allow previously impossible things for Gentoo users. Original name for the idea was "Tree-wide collision checking and provided files database". You can find it on Gentoo wiki still. I later named the project collagen (as in collision generator). Of course implemented system is quite a bit different from original wiki idea. Some things were added, some were removed. If you want to relive how I worked on my project, you can read my weekly reports at gentoo-soc mailing list (I will not repeat them here). Some information was aggregated also on soc.gentooexperimental.org. As final "pencils down" date approached I created final bugreports of features not present in delivered release (and bugs there were present for that matter). Neither missing features, nor present bugs are a real show-stopper, they mostly affect performance. And more importantly I plan to continue my work on this project and perhaps other Gentoo projects. I guess some research what those projects are is in order :-)

Before GSoC I kind of had an idea how open-source projects work since I've participated with some to a degree. However I underestimated a lot of things, and now I would do them differently. But that's a good thing. I kind of like the idea that no project is a failed one as long as you learn something from it. It reminds me of recent Jeff Atwood's post about Microsoft Bob and other disasters of software engineering. To quote him:
The only truly failed project is the one where you didn't learn anything along the way.
I believe I have learned a lot. I believe that if I started collagen now, it would be much better in the end. And the best thing is that I can still do that. I get to continue my project and learn some more. If I learned anything during my work on collagen it's this:
If you develop something in language without strong type checking CREATE THE DAMN UNIT TESTS! It will make you life later on much easier.
In next episode: Why I think Gmail is corrupting minds of people and why I hate mobile phones



Share/Save/Bookmark
Jul 26, 2009

Technical decorating that makes sense

I have been using Python for several small-ish project in past year or two. In that time I have never found reason to really use decorators. You might have seen them in Java source codes in form of
@deprecated
void getVal()
{
//code
}
Python has same syntax, but because it's scripting language certain things are different (read: more flexible :-) ).

Now to the main thing. Where did I use it? As I was deciding what ORM library/tool to use in my GSoC project, I came to conclusion that I could probably use Django DB backend to work with database. This way I would avoid doing the same thing (ORM) twice. Once for backend and once again for web interface later on. As always, things are not as straightforward as they seem in the beginning. Django is web framework and it's tied with its database backend. In other words, the backend was not created with standalone usage in mind. Some things get a bit hairy because of that. These things are mostly connection management and exception handling.

Stackoverflow to the rescue (once again). There was already a question regarding use of only db part of django framework. Normally function like this in django:


would have to become this:

def add_package(self, name):
reset_queries()
try:
p = Package.objects.filter(name=name)
if len(p) > 0:
return p[0].id
p = Package(name=name)
p.save()
return p.id
except:
_rollback_on_exception()
finally:
close_connection()
Imagine that this exception handling, rollbacks and connection closing would have to be in every function. A bit ugly isn't it? We cannot really use inheritance to our advantage, but we could use metaclass(es). I like look of decorators a bit better, so that's what I used. So final code looks like this:

def dbquery(f):
def newfunc(*args, **kwargs):
reset_queries()
try:
return f(*args, **kwargs)
except Exception, e:
_rollback_on_exception()
raise e
return newfunc

@dbquery
def add_package(self, name):
p = Package.objects.filter(name=name)
if len(p) > 0:
return p[0].id
p = Package(name=name)
p.save()
return p.id
For other functions we could just add simple @dbquery in the beginning and voila, problem solved. Maybe there are even cleaner and/or better ways to do the same thing but at least I finally found non-trivial use for decorators.

Share/Save/Bookmark
Jun 30, 2009
First a bold note. I already have repository on Gentoo infrastructure for working on my GSoC project. Check it out if you want.

Last time I mentioned I won't go into technical details of my GSoC project any more on this blog. For that you can keep an eye on my project on gentooexperimental and/or gentoo mailing lists, namely gentoo-qa and gentoo-soc. But there is one interesting thing I found out while working on Collagen.

One part of my project was automation of creating of chroot environment for compiling packages. For this I created simple shell script that you can see in my repository. I will pick out one line out of previous version of this script:
mount -o bind,ro "$DIR1" "$DIR2"
What does this line do? Or more specifically what should it do? It should create a virtual copy of conents of directory DIR1 inside directory DIR2. Copy in DIR2 should be read-only, that means no creating new files, no changing of files and so on. This command succeeds and we as far as we know everything should work OK right? Wrong!

Command mentioned above actually fails silently. There is a bug in current linux kernels (2.6.30 as of this day). When you execure mount with "-o bind,ro" as arguments, the "ro" part is silently ignored. Unfortunately it is added to /etc/mtab even if it was ignored. Therefore you would not see that DIR2 is writable unless you tried writing to it yourself. Current proper way to create read-only bind mounts is therefore this:
mount -o bind "$DIR1" "$DIR2"
mount -o remount,ro "$DIR2"
There is issue of race conditions with this approach, but in most situations that should not be a problem. You can find more information about read-only bind mounts in LWN article about the topic.



Share/Save/Bookmark
Jun 11, 2009

Reinventing the wheel

It's been a week and something since my last post and again quite a lot of things happened. I had several tasks from my last week:
  • get in touch with gentoo-infra(structure) team
  • improve way build logs were handled
  • look at possible ways to create tinderbox chroot environments
  • make it possible to test all versions of dependencies
I managed to get acquainted with gentoo-infra team a bit, and get a few answers too. Remember last time when I was talking about security issues of pickle module when used over untrusted connection?  That's not an issue anymore apparently since we'll be using encrypted connection in final version. We can consider route between Matchbox and Tinderboxen to be friendly environment.

Few people suggested that I look into few similar project Gentoo, namely catalyst and AutotuA. Main feature of catalyst is release engineering, not testing per se. But it can also create tinderboxes (effectively chroot environments). Perhaps some ideas could be used for my project, but so far it seems that catalyst is not the one and only for me. AutotuA is much more similar to collagen (did I mention that's name of my project yet?). There is master server (web application accepting jobs) and slaves (processing jobs). There were quite a few interesting design decisions (such as keeping jobs in git repository) and some of it I will at least reuse. Integration would be possible, but for now I have a feeling that such integration would be just as complicated as writing my own master/slaves. That is because AutotuA is generic system for jobs and their processing not specific for package compilation and testing. I'll keep both projects in mind during my future endeavours.

As far as build log handling goes, my last POC (proof of concept) code simply grabbed stdout/stderr of whole install process. It also used higher-level interfaces for installing packages in gentoo, I switched to lower APIs because I need to do few things higher-level APIs did not offer. Most of these things had to do with dependency handling. Best way to explain what I have to do is using example. But first a little Gentoo package installation introduction. Package "recipes" called ebuilds reside in so-called portage tree. Most packages have more than one ebuild because there are always older and newer versions supported simultaneously. Each of these package versions has its own set of dependencies, that is other packages that need to be installed for package to compile/run. These dependencies look something like this:
=dev-libs/glib-2*
samba? ( >=net-fs/samba-3.0.0 )
This means that package would need any version of glib-2 library, and if samba feature (USE flag) is enabled then also samba version 3 or higher would be required. My task is to verify that package can be compiled with ALL allowed versions of ALL dependencies. Now the promised example.

Lets assume that we want to install package mc (midnight commander). There are currently 2 versions of app-misc/mc in portage: 4.6.2_pre1 and 4.6.1-r4. List of their dependencies is quite long, but to show you principle I'll use just one dependency, namely sys-libs/ncurses. Version 4.6.2 of mc depends on sys-libs/ncurses and version 4.6.1-r4 depends on >=sys-libs/ncurses-5.2-r5. There are currently 2 versions of sys-libs/ncurses in portage: 5.7 and 5.6-r2. Based on these dependencies it should be possible to install package mc (both versions) with either ncurses-5.7 or 5.6-r2. From this point on there is ping-pong of installing ncurses-5.6-r2, then mc-4.6.1-r4/4.6.2_pre1 followed by uninstalling them all and installing ncurses-5.7 and installing mc-4.6.1-r4/4.6.2_pre1. If mc-4.6.2_pre1 fails to compile with ncurses-5.6-r2 we will know that ebuild needs to be modified with dependency >=sys-libs/ncurses-5.7. All this has to be repeated for every dependency for every version of every package in portage tree. Currently there are 26623 ebuilds in portage tree. Now imagine that some of them will have to be compiled even 30-50 times to test all dependency versions. Good thing we will have dedicated tinderboxes for compiling all those ebuilds.

One more thing for now. Gentoo has project management website based on redmine for all of GSoC students on soc.gentooexperimental.org. From now on I will aggregate all of documentation for my project there. This blog will go on will less technical details and I will link to documentation where needed.




Share/Save/Bookmark
Jun 2, 2009

First commit for GSoC

Recently I finished all of my duties as a student for this term and I could therefore spend the weekend catching up on GSoC (since I am one week behind schedule). In the end it turned out to be pretty productive weekend.

I'll summarize basic architecture without any images (I'll create them later this week probably when everything will settle down). There are two core packages:
  • Matchbox
  • Tinderbox
Matchbox is master server that knows what still needs to be compiled and collects all information. There is always only one Matchbox. There can however be more Tinderboxes. These machines connect to Matchbox and ask for next package to emerge (compile). After emerging package they collect information about files in the package, use flags, emerge environment and error logs from compile phase. This information is then sent back to Matchbox. Tinderbox then asks for another file to emerge. repeat while true.

First thing I did was create basic data model for storing data about compiled packages. What use flags were used, error logs and stuff like that. Lot of things are not in the model, for example information about tinderboxes, but for now this will do. UML diagram is on following picture:


This model should allow efficient storage of data and a lot of flexibility to boot. There can be more versions of the same package (of course) and also packages can change package category (happens quite often). We can also collect different data sets based on USE flags.

With basic data model in place it was time for some serious prototyping :-) Naturally I decided to split implementation into two parts, one for each core modules (more to come later). Matchbox is simple listening server waiting for incoming connections. I wanted to simplify network communication for myself, so I used python module pickle. This module is able to create string representation of classes/functions and basic data types. Because of this I was able to use objects as  network messages. Objects representing Matchbox command set:

class MatchboxCommand(object): pass

class GetNextPackage(MatchboxCommand):
    pass

class AddPackageInfo(MatchboxCommand):
    def __init__(self, package_info):
        self.package_info = package_info

On the other side Tinderbox understands these commands (for now):
class MatchboxReply(object): pass

class GetNextPackageReply(MatchboxReply):
    def __init__(self, package_name, version, use_flags):
        self.package_name = package_name
        self.version = version
        self.use_flags = use_flags

Communication (simplified) goes something like this:
Tinderbox
msg = GetNextPackage()
msg_pickled = pickle.dumps(msg)
sock.sendall(msg_pickled)

Matchbox
data = sock.recv()
command = pickle.loads(data)
if type(command) is GetNextPackage:
        package = get_next_package_to_emerge()
        msg = GetNextPackageReply(package)
        msg_pickled = pickle.dumps(msg)
        sock.sendall(msg_pickled)

There is one BIG caveat to this kind of communication. It is very easy tampered with. This is directly from pickle documentation:

Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

We will have to decide whether to reimplement this part, or trust Gentoo infrastructure. So what do we have for now?
  • Basic communication between Matchbox/Tinderbox
  • Compiling works with file list/emerge environment/stdout/stderr/etc being send back to Matchbox
There is still much more ahead of us:
  • package selection on Matchbox side
  • block resolution on Tinderbox
  • rest of services (web interface, client, etc)
Since GSoC students didn't get git repositories on gentoo servers just yet you can see the code in gentoo-collagen@github. So long and thanks for all the fish (for now)

Share/Save/Bookmark
Apr 11, 2009

Applying for Gentoo project in GSoC

I don't know if you've heard of Google Summer of Code, but most probably yes. Basic description is
Google Summer of Code (GSoC) is a global program that offers student developers stipends to write code for various open source software projects...Through Google Summer of Code, accepted student applicants are paired with a mentor or mentors from the participating projects, thus gaining exposure to real-world software development scenarios and the opportunity for employment in areas related to their academic pursuits. In turn, the participating projects are able to more easily identify and bring in new developers. Best of all, more source code is created and released for the use and benefit of all.
It sure is a great idea and I was always intrigued to participate. This year I finally decided to try it out. Gentoo, my distribution of choice was chosen as one of projects to mentor students. I read the ideas page and some of them seemed pretty interesting. One of them was "Tree-wide collision checking and files database". What was the idea? Basically, Gentoo is a source based distribution so noone knows what files will get installed with a certain package. This can be a problem for quality assurance team (QA). Emerge checks for file collisions before installing so you will never screw up your system, but installing some less known packages can give you a headache. Implementing this idea should help fix this situation and most probably be used for other purposes as well. One of them could be approximation of size of package to be installed. This is common in binary distributions, because they know size of package, but Gentoo has a lot of small "gotcha's" in this department. One of them is USE flags, great way to customize your distribution, but a nightmare for this sort of thing. Good thing emerge is such a great package manager :-). I will not repeat things I've said elsewhere so if you want to read more, you can read my application.

If I get accepted (I should know on April the 20th), there is a long way from where I am to making a great project for Gentoo. Hopefully I'll be able to keep up and help out a bit.



Share/Save/Bookmark
Mar 2, 2009

Power to the masses

Few months back I had a rant about participation in opensource. Things have moved a bit since then. I had a few more commits to the mob branch of gstfs repository on repo.or.cz. More importantly Bob Copeland got in touch with me and one more developer who voiced his interest in the project and offered to hand over gstfs to us. Understandably he doesn't have as much free time to spend on such project as an average university student :-).

During the weekend I created project page, new code repository with my patches included, developer and user mailing lists and few issue tickets. I just hope I will be able to keep up the work on the project at least a bit better than this blog...Fingers crossed.



Share/Save/Bookmark
Jan 13, 2009
First of all...All hail our new overlord. And by overlord I mean year 2009. I hope you all will have a great time. I know I will :-). I didn't write for some time, because I was travelling then I was celebrating holidays with my family and friends. All in all I didn't have so much time to keep my information up to date not to mention doing anything resembling work. That's changing NOW!.

I recently bought new camera (lovely Nikon D90) and also decided I need to backup my previous photos to more than 2 places. I realized you can never have enough backups after a few failed HDDs. So what were the options I was considering?
  • Google's Picasa: 20$/year for 10GB  storage space
  • Flickr: 25$/year for unlimited storage and better sharing/privacy settings, presentation options etc.
I didn't consider other services because...well because I didn't.

Now the issue was...How to upload all of my photos (several gigabytes)? Flickr has client for Windows/MacOS, but not for Linux (The orignal client appears to work through wine though). Kflickr to the rescue! I started uploading photos in no time. But I wouldn't be writing this blog entry if everything went according to plan now would I?

Everything seemed to work, the photos were on the web. I could see them, organize them, tag them...you name it. Then I wanted to download original file from certain photo (for reason I don't remember). How great was my surprise when the file was <1MB in size. The originals I had were ~3 MB. Something rotten in here. The files were obviously recompressed with lower jpeg quality settings before being uploaded. Not all of them were this way though. It seemed like it has something to do with license I used for the files. Power is in the source, Luke so there I was. I wanted to investigate the problem and maybe fix it.

Unfortunately opening Kflickr project files with Kdevelop and trying to debug didn't work. For some reason the gdb was ignoring my breakpoins as if the application was compiled without debugging information. It was however compiled with -g3 (all debugging info). So far I was unable to properly diagnose the orignal bug, but I wrote to author of Kflickr asking for information. Now let's wait.

Share/Save/Bookmark
Nov 14, 2008

Xorg evdev madness

It is really astonishing how easy it is to find topics for blogging when one looks around :)

I recently upgraded my Xorg installation to latest ~x86 version. For Gentoo virgins, this means unstable version, although it is usually considered stable upstream, just integration with other apps can be sometimes problematic. Stable version was really old and had problems with recent kernel versions. I was very happy with the upgrade, which made my 5 year old Thinkpad more alive than ever. I decided to recreate my xorg.conf because most of the stuff that was there was not needed anyway, since XRandR 1.2 is used.

What is my problem then? Well after the upgrade some features of my touchpad stopped working (most notably circular scrolling) and I could not switch between different layouts of my keyboard. First thing I did was of course look at Xorg.0.log. Important part follows:

(II) XINPUT: Adding extended input device "AT Translated Set 2 keyboard" (type: KEYBOARD)
(**) Option "xkb_rules" "base"
(**) AT Translated Set 2 keyboard: xkb_rules: "base"
(**) Option "xkb_model" "evdev"
(**) AT Translated Set 2 keyboard: xkb_model: "evdev"
(**) Option "xkb_layout" "us"
(**) AT Translated Set 2 keyboard: xkb_layout: "us"
(II) config/hal: Adding input device ThinkPad Extra Buttons
(**) ThinkPad Extra Buttons: always reports core events
(**) ThinkPad Extra Buttons: Device: "/dev/input/event3"
(II) ThinkPad Extra Buttons: Found keys
(II) ThinkPad Extra Buttons: Configuring as keyboard
(II) XINPUT: Adding extended input device "ThinkPad Extra Buttons" (type: KEYBOARD)
(**) Option "xkb_rules" "base"
(**) ThinkPad Extra Buttons: xkb_rules: "base"
(**) Option "xkb_model" "evdev"
(**) ThinkPad Extra Buttons: xkb_model: "evdev"
(**) Option "xkb_layout" "us"
(**) ThinkPad Extra Buttons: xkb_layout: "us"

As it happened evdev found additional "keyboards" and IGNORED my layout settings for keyboard. I found few forum posts dealing with the same problem on Gentoo and Arch Linux. I will not go into details, if you really want to know all the crazy solutions people found, read the forums. But easiest solution? Uninstall evdev driver for now if you don't need it (you probably don't). Similar effect could be probably reached by adding Option AutoAddDevices "boolean" to Serverflags section of xorg.conf, however I didn't try this approach.

Share/Save/Bookmark
Oct 25, 2008

GStreamer bug hunting

While I was working on GSTFS (from my previous post) I also managed to stumble on a bug in GStreamer. Of course at the time I didn't know it was a bug since I was GStreamer greenhorn (I actually still am :) ).

What was it about? Well when I was converting mp3 files with pipeline
[source ]! decodebin ! audioconvert ! lame  ! id3v2mux ! [output]
Id3tags were lost in conversion. This was a real bummer for me, since my player fully supports id3v2.4. When I read through the docs, man page and I could not find any solution I did what every self respecting geek would do. I fired up my IRC client and joined #gstreamer on freenode in search for help. After a few advices that didn't work we closed the issue with me filing a bug caused by unknown reason. Originally the bug seemed to be part of demuxing problem with mp3s (ogg files didn't have this problem).

After some time, I finally had a revelation. What made my mp3 files special? Nothing. Except that all of them had ReplayGain tags calculated with mp3gain. I removed the ReplayGain tags, tried the same gstreamer pipeline as before and....Voila! My converted files were not missing id3tags anymore. The bug filed in Bugzilla is not closed yet, but now the developers at least know where to look for it.

Now I have a request for all 2 people reading my blog (including me :) ). If you find a bug in opensource (or free, whatever you want to call it) software, pleeeeeaase report it. Ideally you first make damn sure you are actually not filing PEBKAC so that developers don't waste their time. If you can help with the testing of the fix. If you are not skilled enough you can always go to the IRC channel or send and email and ask if maybe documentation needs clarification or other relatively easy chores need to be done. There's always stuff to do. Just read the post 5 Ways to Contribute to Open Source Projects Without Coding.

Share/Save/Bookmark
Oct 24, 2008

Opensource participation

In my previous post I mentioned project to transparently convert media files uploaded to my mp3 player. At first I wanted to create my own project from scratch. But then I searched around on the net and FUSE wiki and I found Gstreamer filesystem or GSTFS for short.

GSTFS is based on GStreamer multimedia framework to handle media conversions and FUSE to create virtual filesystem. Because of GStreamer, simple changes on command line allow you to do almost any task concerning media files. Conversion of music files, videos, resizing of pictures and more.

I started playing with GSTFS trying to convert my music collection to lower bitrate mp3s. Simple 'cp -R * music/ music_converted/' should have worked. But it didn't. Why? Well GSTFS shows non-converted files as 0-sized files. And cp tried to optimize copying by actually not copying empty files. It doesn't even try to read them. That meant running cp twice since second run would see actual sizes. Even then there is a problem with expiration of file cache so if your music collection is more than a few files, you are out of luck.

And here we come to great advantage of opensource (at least for me). Source code of GSTFS is available, so I fixed this small bug and send a few line patch to original author Bob Copeland. I also asked if he could perhaps create public repository of sources on repo.or.cz. Interestingly enough I was aparently not the only one to ask for it. And so, lo and behold, Git repository of GSTFS is online. You can now find my work on improving GSTFS in mob branch in the repository. Hopefully I will be able to contribute more to this great idea and my code will actually make it into the main branch :)

Share/Save/Bookmark