Ins and Outs of Inputs

Yesterday I promised I'll talk about why I hate mobile phones. Of course I didn't mean all of them. Just the ones I have to deal with. Why? Well my mobile phone kind of died few days ago. I have a Nokia N73 and it's really quite good phone even if it's a bit old by today's standards. You control the phone by using "joystick" kind of thing in the upper part of keyboard. I decided to include image so you don't have to look for it :-)

So this joystick stopped working (even slightest touch would be evaluated as pushing it, therefore it was unusable). I didn't have my backup phone with me, but one friend gave me her battered Siemens S55. So what was the problem? Well I have the same sim card for almost 10 years now. Back then only 100 contacts would fit on it. I have almost 300 contacts in my N73. So how do I get all contacts from one phone to the other? Normally I could just send them through bluetooth, but since I couldn't really control my N73 this was out of question. I was barely able to turn on the bluetooth. I thought that I'll use SyncML interface to get vCards from N73 to my computer and then sync them again to the S55. In the end I kind of did, but boy was that an unpleasant experience!

So what exactly happened? I installed OpenSync libraries and tools and using multisyncgui I created sync group with one side being file-sync plugin and other was syncml-obex-client plugin. Configuraion of file-sync plugin was mostly just changing path to directory where I wanted to sync. Final version looked like this:

<?xml version="1.0"?>
<config>
  <directory>
    <path>/tmp/sync</path>
    <objtype>contact</objtype>
    <objformat>vcard30</objformat>
  </directory>
</config>

Configuration for syncml-obex-client appeared to be much more challenging. It appears that Nokia N73 has two quirks:

• It only talks to SyncML client if it says its name is "PC Suite"
  
• It contains a bug that causes it to freeze after certain amount of data if configuration is not correct
  

First of these quirks is mentioned in almost every tutorial on data synchronization in Linux. However the second one caused me to lose quite some time. My Nokia N73 would freeze after synchronizing approximately 220-240 contacts. To continue working I had to restart the whole phone.In the end I found out that I need to set parameter recvLimit to 10000 in order to synchronize everything. Final setting for syncml-obex-client looks like this:

<?xml version="1.0"?>
<config>
  <type>2</type>
  <bluetooth_address>00:1B:33:3A:D1:37</bluetooth_address>
  <!-- Nokia Obex server channel -->
  <bluetooth_channel>13</bluetooth_channel>
  <interface>0</interface>
  <identifier>PC Suite</identifier>
  <version>1</version>
  <wbxml>1</wbxml>
  <username></username>
  <password></password>
  <usestringtable>1</usestringtable>
  <onlyreplace>0</onlyreplace>
  <onlyLocaltime>0</onlyLocaltime>
  <recvLimit>10000</recvLimit>
  <maxObjSize>0</maxObjSize>
  <database>
     <name>Contacts</name>
     <objtype>contact</objtype>
     <objformat>vcard21</objformat>
  </database>  
</config>

So after all that I was able to get vCards from my N73 to my notebook. For every vCard OpenSync created file in directory /tmp/sync. Now came the interesting part. How to get these vCards to Siemens S55?

Simple Google search on Siemens S55 and synchronization in Linux seemed to suggest that tool most suited to do the job was scmxx. This little app is specialized on certain Siemens phones. According to some manuals it was supposed to be able to upload vCards themselves, however I couldn't get it to work as scmxx was complaining about invalid command line arguments.After some testing I found out that it could access and change sim card phone numbers.

Unfortunately for me, my sim card has limit of 100 phone numbers, each with 14 character identifier (name). This meant I needed to convert vCards from N73 to special format that scmxx used. Mentioned format looked something like this:

1,"09116532168","Jones Rob"
2,"09223344567","Moore John"
...

First column being number of slot that will be overwritten by new information, second column is number and third one name of contact (less than 15 characters).

So I fired up vim and started coding conversion script. It didn't take long and I had my contact in the old-new phone. There are a lot of hard-coded things in that script since I don't plan to ever use it again but you can download it from my dropbox. Consider it public domain, and if anyone asks I didn't have anything to do with it :-)

import os
import re

MAX_CONTACTS=100

class PbEntry(object):

    def __init__(self, name, tel, year, month, day):
        self.name = name
        self.tel = tel
        self.year = year
        self.month = month
        self.day = day

def cmp_pb(e1, e2):
    if e1.year > e2.year:
        return -1
    elif e1.year < e2.year:
        return 1
    else:
        if e1.month > e2.month:
            return -1
        elif e1.month < e2.month:
            return 1
        return 0


telRe = re.compile('TEL(;TYPE=\w+)*:([*#+0-9]+)', re.M)
revRe = re.compile('REV:(\d{4})(\d{2})(\d{2}).*', re.M)
nameRe = re.compile('^N:(.*);(.*);;;', re.M)
def get_entry_from_text(text):
    ret = nameRe.search(text)
    surname = None
    name = None
    tel = None
    rev = None
    if ret:
        surname =  ret.group(1)
        name = ret.group(2)

    ret = telRe.search(text)
    if ret:
        tel =  ret.group(len(ret.groups()))
    
    if surname and name:
        fn = "%s %s" % (surname,name)
    elif surname:
        fn = surname
    else:
       fn = name
    
    if fn:
        ret = re.search('(.{0,14}).*', fn)
        fn = ret.group(1)

    
    ret = revRe.search(text)
    year = ret.group(1)
    month = ret.group(2)
    day = ret.group(3)

    return PbEntry(fn, tel, year, month, day)


entries = []

files = os.listdir('/tmp/sync')
for file in files:
    fh = open('/tmp/sync/%s' % file, 'r')
    content = fh.read()
    entry = get_entry_from_text(content)
    entries.append(entry)

entries = sorted(entries, cmp=cmp_pb)

i = 1
for entry in entries:
    print '%d,"%s","%s"' % (i, entry.tel, entry.name)
    i = i + 1
    if i > MAX_CONTACTS:
        break

I had my share of incompatibilities between mobile phones, computers and other devices. Fortunately most of devices being sold today use open communication protocols for sharing of data (and other stuff). Too bad people had to put so much energy into reverse engineering proprietary solutions in the past. Just ranting about this vendor lock-in could be spread on quite a few pages. Imagine having 300+ contacts and calendar information in your phone of brand X. When you are buying your new phone, you would be able to synchronize your data only if you bought new phone also from brand X. Would that affect your decision? It sure would affect mine.

Now I have a choice. After fixing my old N73 I will start looking into new phone. So far HTC Hero looks pretty cool and reviews are not half bad.

So this year's Google Summer of Code is officially over. Today 19:00 UTC was deadline for sending in evaluations for both mentors and students. Therefore I think some kind of summary what was happening and what I was doing is in order.

I was working on implementing neat idea that would allow previously impossible things for Gentoo users. Original name for the idea was "Tree-wide collision checking and provided files database". You can find it on Gentoo wiki still. I later named the project collagen (as in collision generator). Of course implemented system is quite a bit different from original wiki idea. Some things were added, some were removed. If you want to relive how I worked on my project, you can read my weekly reports at gentoo-soc mailing list (I will not repeat them here). Some information was aggregated also on soc.gentooexperimental.org. As final "pencils down" date approached I created final bugreports of features not present in delivered release (and bugs there were present for that matter). Neither missing features, nor present bugs are a real show-stopper, they mostly affect performance. And more importantly I plan to continue my work on this project and perhaps other Gentoo projects. I guess some research what those projects are is in order :-)

Before GSoC I kind of had an idea how open-source projects work since I've participated with some to a degree. However I underestimated a lot of things, and now I would do them differently. But that's a good thing. I kind of like the idea that no project is a failed one as long as you learn something from it. It reminds me of recent Jeff Atwood's post about Microsoft Bob and other disasters of software engineering. To quote him:

The only truly failed project is the one where you didn't learn anything along the way.

I believe I have learned a lot. I believe that if I started collagen now, it would be much better in the end. And the best thing is that I can still do that. I get to continue my project and learn some more. If I learned anything during my work on collagen it's this:

If you develop something in language without strong type checking CREATE THE DAMN UNIT TESTS! It will make you life later on much easier.

I have been using Python for several small-ish project in past year or two. In that time I have never found reason to really use decorators. You might have seen them in Java source codes in form of

@deprecated
void getVal()
{
//code
}

Python has same syntax, but because it's scripting language certain things are different (read: more flexible :-) ).

Now to the main thing. Where did I use it? As I was deciding what ORM library/tool to use in my GSoC project, I came to conclusion that I could probably use Django DB backend to work with database. This way I would avoid doing the same thing (ORM) twice. Once for backend and once again for web interface later on. As always, things are not as straightforward as they seem in the beginning. Django is web framework and it's tied with its database backend. In other words, the backend was not created with standalone usage in mind. Some things get a bit hairy because of that. These things are mostly connection management and exception handling.

Stackoverflow to the rescue (once again). There was already a question regarding use of only db part of django framework. Normally function like this in django:

def add_package(self, name):
    reset_queries()
    try:
        p = Package.objects.filter(name=name)
        if len(p) > 0:
            return p[0].id
        p = Package(name=name)
        p.save()
        return p.id
    except:
        _rollback_on_exception()
    finally:
        close_connection()

def dbquery(f):
        def newfunc(*args, **kwargs):
            reset_queries()
            try:
                return f(*args, **kwargs)
            except Exception, e:
                _rollback_on_exception()
                raise e
        return newfunc

@dbquery
def add_package(self, name):
    p = Package.objects.filter(name=name)
    if len(p) > 0:
        return p[0].id
    p = Package(name=name)
    p.save()
    return p.id

For other functions we could just add simple @dbquery in the beginning and voila, problem solved. Maybe there are even cleaner and/or better ways to do the same thing but at least I finally found non-trivial use for decorators.

I don't know if you've heard of Google Summer of Code, but most probably yes. Basic description is

Google Summer of Code (GSoC) is a global program that offers student developers stipends to write code for various open source software projects...Through Google Summer of Code, accepted student applicants are paired with a mentor or mentors from the participating projects, thus gaining exposure to real-world software development scenarios and the opportunity for employment in areas related to their academic pursuits. In turn, the participating projects are able to more easily identify and bring in new developers. Best of all, more source code is created and released for the use and benefit of all.

I suppose people who work with computers for a few years saw similar message at least once. Unfortunately it's quite common for hard drives to fail. There is early warninig system that can predict a lot of these misfortunes. It's called S.M.A.R.T. and it is in fact quite smart :-) A lot of HDDs come with this monitoring disabled for (to me) unknown reason. Maybe it's performance reasons, maybe manufacturers don't want users to know their HDDs fail. Aaah...conspiracy theories :)

Enough of being smart though (pun intended). Recently, over 5 years old computer of my parents refused to boot when one of HDDs (320GB WD Caviar) was connected. No matter what I did, Windows wouldn't boot with that HDD connected. The HDD was (still is actually) under warranty, but I really wanted to save the data. Most important files were backed up elsewhere, but my music collection and some movies waiting to be seen were not. I'll skip the boring stuff. Since the computer had other problems my parents decided to buy new one. With 320GB WD Caviar connected even Vista would not boot (old computer was XP).

I made one final attemt to save data. I booted Ubuntu live cd. To my big surprise, Ubuntu did not just "see" the hard drive. It was able to mount it without problems. It didn't even complain. I just backed up the hard drive, did low level format (e.g. dd if=/dev/zero of=/dev/sda bs=1M) and suddenly windows was able to boot without problems. I had one other problematic 80GB Seagate HDD that I remembered and the outcome was the same. Windows was able to see it after low level format. These HDDs were not system HDDs, so even if MBR was corrupted I shouldn't have mattered. I couldn't find anything final on the Internet about this type of HDD "failures" so any info is welcome. S.M.A.R.T. is not complaining so it seems that I have 2 good HDDs in my hands now. Linux saves the day! :-)

Recently ext4 filesystem was marked as stable with release of Linux 2.6.28. Since I like bleeding edge from time to time and backup my files regularly anyway I decided to give it a spin. As far as performance is concerned, I have nothing to report yet, since I haven't been using it that long. But as usuall I found certain annoyance :-)

I was going through my filesystems and converting them one-by-one (after doing one more backup). When it came to /var I hit the wall though. df showed that there is free space (more than 400MB) but tar was telling me there is not enough space on the filesystem to create a directory (ENOSPC). So what was it? I was looking around and finally found the problem. Since the size of /var is only 1GB on my computer, mkfs.ext4 decided I will never use more than ~65000 inodes. Problem is that I have a lot of small files on the filesystem. Ebuilds, git and svn repositories and standard /var stuff. This together meant that I hit the 65000 mark quite easily whithout filling up the filesystem.

Solution to my problem was obvious from this point on. Recreate /var filesystem while manually overriding mkfs.ext4's choice for maximum inode count. Voila, ext4 seems working well from this point on.