Ins and Outs of Inputs

Have you ever said anything you wanted to take back right after you finished the sentence? Well maybe you got lucky and there were only a few people around. But once you put something on the web, it's there forever. Internet doesn't have concept of delete button.

There is always omnipresent cache and archives, so even deleting content from you site doesn't help. This happened recently when Apple pulled biography of their new executive Mark Papermaster from their website, after court barred him from reporting to work in Apple until his lawsuit with IBM is closed. I will not go into details (you can read Ars Technica coverage of the issue) because my point lies elsewhere. You can say what you want, if it is connected to the Internet it is public FULLSTOP

Internet is full of stories where people wanted to hide their humiliations and errors from public by injunctions, lawsuits and whatnot. The end result is almost always Streissand effect. If you read the wiki, there are some nice examples why you should keep your private things private. Once it's out, trying to censor it will only make it worse (the more famous/sexy you are the worse for you). It might be a good time to read guides to privacy right now. I know you are not going to do that anyway, but it is still my dream that once, a new generation will be able to protect their privacy online. Unfortunately anecdotal evidence suggest otherwise.

By the way. Anyone knows a simple list of things to improve your privacy online?

Seriously. According to joint study by security researchers, Storm botnet can create as much as $ 3.5M of revenue per year. It was definitely one of the most ingenious research and analytical papers I have read so far.

In order to measure effectiveness of spam campaigns, researchers joined Storm botnet with bots that were used to conduct MITM attack on Storm itself. These bots changed spam campaigns slightly and redirected targets of spam campaign (users) to servers controlled by researchers. These servers mimicked websites of spammers and counted number of visitors and number of actual victims who fell for the scams and provided their information (credit card number, social security number, etc.). If the results are correct, spam campaigns are effective in less than 0.00001% of cases. This number is indeed extremely low, but if you consider size of the Storm and number of emails that it sends every day, you get to more interesting numbers ranging from $7000 to $9500 of revenue per DAY.

I left out few interesting details so if you have some time, consider reading the whole paper (12 pages).

It is really astonishing how easy it is to find topics for blogging when one looks around :)

I recently upgraded my Xorg installation to latest ~x86 version. For Gentoo virgins, this means unstable version, although it is usually considered stable upstream, just integration with other apps can be sometimes problematic. Stable version was really old and had problems with recent kernel versions. I was very happy with the upgrade, which made my 5 year old Thinkpad more alive than ever. I decided to recreate my xorg.conf because most of the stuff that was there was not needed anyway, since XRandR 1.2 is used.

What is my problem then? Well after the upgrade some features of my touchpad stopped working (most notably circular scrolling) and I could not switch between different layouts of my keyboard. First thing I did was of course look at Xorg.0.log. Important part follows:

As it happened evdev found additional "keyboards" and IGNORED my layout settings for keyboard. I found few forum posts dealing with the same problem on Gentoo and Arch Linux. I will not go into details, if you really want to know all the crazy solutions people found, read the forums. But easiest solution? Uninstall evdev driver for now if you don't need it (you probably don't). Similar effect could be probably reached by adding Option AutoAddDevices "boolean" to Serverflags section of xorg.conf, however I didn't try this approach.

And while it's spinning faster every day (perhaps because of her?) my blog themes are getting cold and old. I wanted to write about many topics but instead I was living my life. Go figure...So first let me just post simple summary of links I found worth reading in past weeks:

There were also few others but just like 2001: A search oddysey they became outdated some time ago.

There is however one article that sparked my interest more than others in past weeks. Title of the article is "Tips for getting started in information security". Why was this interesting to me? I have quite a few feeds in my RSS reader. Some of them are dealing with security, some with more general IT topics, some are just plain fun. My problem is that I like security as much as I like software development. It is however not that easy to find basic-level stuff that is dealing with application security. When I read about attack on Adobe Flash virtual machine my head started turning. I know thing or two about stack, buffer overflows etc. but this is just too much for me now. So I decided I have to change my approach a bit and start catching up on application security. Otherwise I will just turn to one of those old school wannabes that actually know something about everything nut not really everything about something.

Unfortunately I don't suspect I will have much time in upcoming days for blogging but we'll see.

While I was working on GSTFS (from my previous post) I also managed to stumble on a bug in GStreamer. Of course at the time I didn't know it was a bug since I was GStreamer greenhorn (I actually still am :) ).

What was it about? Well when I was converting mp3 files with pipeline

[source ]! decodebin ! audioconvert ! lame  ! id3v2mux ! [output]

Id3tags were lost in conversion. This was a real bummer for me, since my player fully supports id3v2.4. When I read through the docs, man page and I could not find any solution I did what every self respecting geek would do. I fired up my IRC client and joined #gstreamer on freenode in search for help. After a few advices that didn't work we closed the issue with me filing a bug caused by unknown reason. Originally the bug seemed to be part of demuxing problem with mp3s (ogg files didn't have this problem).

After some time, I finally had a revelation. What made my mp3 files special? Nothing. Except that all of them had ReplayGain tags calculated with mp3gain. I removed the ReplayGain tags, tried the same gstreamer pipeline as before and....Voila! My converted files were not missing id3tags anymore. The bug filed in Bugzilla is not closed yet, but now the developers at least know where to look for it.

Now I have a request for all 2 people reading my blog (including me :) ). If you find a bug in opensource (or free, whatever you want to call it) software, pleeeeeaase report it. Ideally you first make damn sure you are actually not filing PEBKAC so that developers don't waste their time. If you can help with the testing of the fix. If you are not skilled enough you can always go to the IRC channel or send and email and ask if maybe documentation needs clarification or other relatively easy chores need to be done. There's always stuff to do. Just read the post 5 Ways to Contribute to Open Source Projects Without Coding.

In my previous post I mentioned project to transparently convert media files uploaded to my mp3 player. At first I wanted to create my own project from scratch. But then I searched around on the net and FUSE wiki and I found Gstreamer filesystem or GSTFS for short.

GSTFS is based on GStreamer multimedia framework to handle media conversions and FUSE to create virtual filesystem. Because of GStreamer, simple changes on command line allow you to do almost any task concerning media files. Conversion of music files, videos, resizing of pictures and more.

I started playing with GSTFS trying to convert my music collection to lower bitrate mp3s. Simple 'cp -R * music/ music_converted/' should have worked. But it didn't. Why? Well GSTFS shows non-converted files as 0-sized files. And cp tried to optimize copying by actually not copying empty files. It doesn't even try to read them. That meant running cp twice since second run would see actual sizes. Even then there is a problem with expiration of file cache so if your music collection is more than a few files, you are out of luck.

And here we come to great advantage of opensource (at least for me). Source code of GSTFS is available, so I fixed this small bug and send a few line patch to original author Bob Copeland. I also asked if he could perhaps create public repository of sources on repo.or.cz. Interestingly enough I was aparently not the only one to ask for it. And so, lo and behold, Git repository of GSTFS is online. You can now find my work on improving GSTFS in mob branch in the repository. Hopefully I will be able to contribute more to this great idea and my code will actually make it into the main branch :)

We all know that, right? Right. Who am I kidding? Most people don't notice the difference between 96kbps 4x re-encoded mp3 and FLAC. Of course it depends on your setup. Headphones, mp3 players. All of it makes your experience better (or worse).

Since I bought my first Koss Porta Pro headphones I realized that there are headphones and HEADPHONES. With some you don't even hear the important parts, while others make you realize how much noise there is in the source :). And so my music collection is now mostly flac, high quality ogg or vbr mp3. Normally I don't care about the size of music files. Storage is not that expensive these days. But I have an old Cowon U3 music player (still cannot find anything better) with only 2GB of memory. Of course that's where size comes into play. What I usually do is convert music to lower bitrates (160 kbps vbr usually) before transfering them to the player. But I don't keep those converted versions around since I don't have that much space lying around :). So I am wasting time chosing files to transfer, then converting them and finally copy them to player.

Manually converting and then transferring files is kind of a bummer though. Now I realized...I can actually program right?! So how about making a FUSE virtual filesystem on top of vfat filesystem on the player. This virtual filesystem would convert music files being copied to the filesystem to specified format in background. Processor speeds are fast enough to do this more-less realtime these days so why not?

How will this affect my workflow? Compare:

Now	Virtual FS
Copy files to temporary directory Convert big files to lower bitrates Copy files to mp3 player	Copy files to mp3 player

So far this is just an idea. I don't know of any other project doing the same thing (there are a few dealing with general data compression but not media files specific).

Expect more to come (just don't expect deadlines :) ).

*EDIT* As it happend there are already FUSE project that do exactly what I had in mind. I guess I should check the page more often. The projects are GSTFS and MP3FS where the first one seems more promising and flexible.

I am pretty sure everyone has seen CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) before. Maybe you didn't know the (full) name but you have encountered it when registering accounts, posting comments or accessing some parts of web. You know, those annoying things that are exercising your ability to recognize distorted words with weird backgrounds.

CAPTCHAs are used to protect against automated attacks. For example automatic registration of new users on Gmail would create great opportunities for spammers. CAPTCHAs are mostly working, even when they get "hacked" from time to time. The biggest problem? They are reaching levels where even humans are having problems reading the letters. I still have nightmares when I remember CAPTCHAs used on RapidShare. Telling cats from dogs was not that easy for me somehow. I am not sure about "hackability" of reCAPTCHA, but as far as usability goes, it's one of the best ones for me. Too bad only a few sites are using it.

The main problem of CAPTCHAs is not the complexity but relay attacks and human solvers from 3rd world countries paid for solving thousands of CAPTCHAs a day. What we really need is CAPTHHA (Completely Automated Public Test to tell Humans and Humans Apart). Computer science is far from being able to tell humans with "clean" intentions from those being paid to get past the defences. One solution would be to issue certificates of "humanity" signed by central authority. You could then ban users that were misusing their certificates. There are of course privacy and security problems with this approach, not to mention financial "issues", so I guess this is not how it's gonna work.  Other approaches have also been tried, but they usually have problems with disabled people. I am certainly interested how Computer Science solves this problem.

I'm slowly starting to feel like Google Fanboy(tm), but Big G has made an interesting announcement recently. Dubbed "Clean Energy 2030", proposal tries to encourage several ways to achieve usage of "clean" energy by year 2030. I suggest you read it, especially if you like sci-fi. Basically they suggest 3 complementary things to do:

Of these, first two seem OK. But electrifying transportation? Especially in US where you cannot buy car with engine less than 2000cc? I will watch closely. I still remember those sci-fi movies that showed flying cars in year 2000 and I am still dissapointed there are almost none.

I would love to see future come sooner, possibly while I'm still alive, but I am a little bit sceptical. Google might chip-in with generous 45$ mil this year, but will goverments follow? I doubt it. Still, hope dies last. I still have this dream of Earthlings being one big nation where it doesn't really matter which part of Earth are you from. It just matters you are not from Qo'noS or Minbar. And this "cheap energy for everyone" initiative reminds me of these dreams. Oh well, one can dream.

I had this in my "almost-finished-near-ready-to-publish" folder for some time already. Past week was again a little crazy in my personal life, so no real time to finish this small piece until now... :)

Do you frequently switch between two computers not connected with local network? If so, I guess you wanted to share data between them at least once before. It used to be a hassle. Now it's easy. Dropbox started public open beta-testing of their service few weeks ago. If you haven't heard of Dropbox here is my little intro. Dropbox is essentially centralized version tracking accessible from anywhere without need to configure anything. You copy files you want to share with other machines to your Dropbox directory and they are automatically uploaded to Dropbox server. If another machine on the other end of the world is running with same Dropbox account, it is automatically synced. If it sounds confusing, I encourage you to read the introduction tour on their website. Free account enables you to use 2GB storage and unlimited bandwith, so it's not that bad. Most of all, it "just works(tm)". And you can later upgrade to Pro versio with 50GB space for $9.99/month or $99.99/year. I am not sure about availability outside US, but I guess that's not gonna be a problem.

You can synchronize files between Windows, MacOS X and Linux machines.There are still a few rough edges, but I guess that's why it's beta :)..It would be really nice if the protocol for communication with Dropbox server was made public, but I guess I am asking for too much. At least the Nautilus interface in Linux is GPLed and there are already alternative "clients" for  retrieving status of your Dropbox account.

Good thing is that you can also share files with the rest of the world. Just like you would with for example Rapidshare account. The difference? No limits on file sizes (so far, as far as I know). I just wonder how will they fight sharing of illegal data.

With services like this privacy is always a concern. You give up certain amount of privacy by uploading your files to 3rd party server. So whatever you do, be sure to encrypt your private files. Happy sharing.

If someone actually read my previous posts (heh), (s)he may have noticed that I quite often link to www.codinghorror.com. It is Jeff Atwood's blog, and I usually find it very exciting to read. His style of writing and ability to convey complex messages in a simple way is my holy grail. And if he is not able to do it himself, he links to other authors A LOT. Instead of repeating same thing that has been said over and over again he just links to proper post made by some other fellow programmer/software engineer. Avoiding duplicity is really one of basic goals of programming. Instead of repeating same code 20 times, just write a function and call it 20 times.

But that is just low-level stuff. Jeff, Joel Spolsky and few others, embarked on an adventure to get rid of duplicity in minds of programmers. Programming is so inherently complex that no one really knows solution to every problem. And don't get me started on optimal solutions. What did you do when you found solution to some programming challenge, or some tricky workaround for problem that was bugging you for weeks? If you have a blog, you could go and post your solution there. Maybe someone would notice. Maybe not. So what did Jeff & Co. do? They created and launched ww.stackowerflow.com. Quoting from about page:

Stack Overflow is a programming Q & A site that's free.

Google released its own Web browser called Chrome few weeks ago and whole web was buzzing with excitement since then. They did it Google style. Everything is neat, clean and simple. And quite a few features are also unique. Google engineers obviously put a lot of thought into scratching their itches with web applications. Javascript engine is fast and whole browser is created around the idea that web is just a place for applications. One of the most touted things about Chrome were its security features. You can read whole account of basic Chrome features on its project page.

In Chrome each tab runs as a separate process communicating with main window through standard IPC. This means that if there is fatal error in handling of some page (malicious or otherwise), other tabs should be unaffected and your half-written witty response to that jerk on the forum will not be lost. Chrome also has other security enhancements, that should make it more secure. I said should. Within few days of Chrome release several security vulnerabilities surfaced, ranging from simply annoying DOS to plain dangerous remote code execution.

What caught my attention was bug that enabled downloading files to user's desktop without user confirmation. It was caused by Googlers using older version of Webkit open source rendering engine in Chrome. Integrating "foreign" software with your application can be tricky, especially if you have to ensure that everything will be working smoothly after the upgrade. In that respect, it is sometimes OK to use older versions of libraries. As long as you fix at least security bugs. People write buggy software. Google engineers included. I am just surprised that they don't have any process that would prevent distribution of software with known security vulnerabilities to the public.

And that is the main problem. Chrome is beta software. Because of this, bugs are to be expected. But Google went public with Chrome in the worst possible way. They included link to Chrome download page on their home page, making hundreds of thousands of people their beta testers. People who have no idea what "beta testing" actually means. They just know that Google has some cool new stuff. So let's try it right? Wrong. Most of us expect our browser to be safe for e-banking, porn and kids (not necessarily in that order). Unfortunately Chrome is not that kind of browser. Yet. I am pretty sure it is gonna be great browser in the future though. But right now Google should put big red sign saying "DANGEROUS" in the middle of Chrome download page.

Until Chrome becomes polished enough for Google to stop calling it "beta", it has no place on desktops of common computer users. Even oh-so-evil Microsoft doesn't show download link for IE8 beta on their main page to promote it. Mentioned issues aside, Chrome really sports few good ideas that other browsers could use as well. Try it out, and you will like it. Then go back to your old browser for the time being.