Feb 21, 2012

DevConf 2012 - "How I Have Seen the Future"

Posted by Stanislav Ochotnicky at 12:15 PM / Labels: , , , / 0 comments
We've had Developer Conference (DevConf) in Brno last weekend and there have been numerous interesting talks and hackfests. You can see the full programme on Fedora wiki.
This year we've had a pleasure to welcome a lot of our colleagues from other Red Hat offices around globe. And they in turn had some of the most interesting talks. I spend most of my time at talks dealing with filesystems, storage and other core components, but there were a few not-so-technical talks that sparked my interest.
Bryn Reeves had two talks, one titled "Supporting the Open Source enterprise" and the other "How to lose data and implicate people". Sadly I had my own lab around fedora-review at the time of the second presentation, but if the first one was any indication the second one must have been great. The talk I've seen was dealing mostly with processes and tools our support uses to help customers deal with problems. And examples. Lots of interesting, fun examples of ingenuity of our engineers when dealing with bugs. Yeah, try replicating customer's setup of few thousand machine grid where the problem occurs. Apparently "git, git, git, git, git, git" is the tool that is saving their lives every day (not surprising).
Other talk that sparked my interest was Lukáš Czerner's "Btrfs - Design, Implementation and the Current Status". While Lukáš is not a Btrfs developer, he is a kernel developer familiar with its internals and the talk contained a lof of technical information I haven't known about before. "The root of the root of the roots" tree must probably be the motto of Btrfs. It looks to me that Btrfs has very powerful abstraction where everything is either a tree or node in a tree, but I guess only time will tell if this abstraction is going to be strong enough for the years to come. I am definitely looking forward to trying Btrfs in a controlled environment (for now).
Another feature that I was drooling over a bit was thin provisioning in LVM that was discussed in a talk given by Edward "Joe" Thornber & Zdeněk Kabeláč. It is a fairly recent feature (first upstream release with support for this was done in January) that allows one to thinly provide LVM volumes. What this means you ask? Well it means you can create 20 GiB volume "pool" that can contain 3 10 GiB thinly provisioned volumes. I.e. these volumes will start small and grow as needed. They will eventually also shrink when space is freed by the underlying filesystem. This of course requires the filesystem to support discard/TRIM commands, but this is not a problem for modern linux filesystems. As I see it thin provisioning teamed with snapshotting will change the way I manage my virtual machines for sure and I can't wait to try it out.
Many people believe Btrfs will take over role of LVM in following years, but the way I see it Btrfs will simplify use cases that LVM is too complex for, while LVM will keep on improving support for more demanding scenarios. Because let's be honest, LVM on desktop is just too darn complicated for an ordinary user/administrator.
I've been to FOSDEM few weeks back, and I have to say that DevConf was smaller, but no less interesting. There are projects or new features that I haven't heard about, but caused some "WOW" monents for me. Definitely looking forward to next year! (and you should try to come too)

Share/Save/Bookmark
Jan 26, 2012

Auto-tag Fedora package git repo from koji builds

Posted by Stanislav Ochotnicky at 5:58 PM / Labels: , , , , / 0 comments
I often browse through various Fedora packages and miss having git tags in package repositories corresponding with builds done in koji. Therefore I created following simple bash script that will create git tags in current git repo (if it's a Fedora package). 
#!/bin/bash

giturl=`fedpkg giturl`
if [ $? -ne 0 ];then
    echo "This doesn't look like fedora package directory"
    exit 1
fi

pkgname=`echo "${giturl}" |\
         sed -e 's|git://pkgs.fedoraproject.org/\(.*\)?.*|\1|'`
# make sure we are up-to-date
git fetch

# go through last 3 releases (incl. rawhide)
for dist in f15 f16 f17;do
    builds=`koji list-tagged "${dist}" "${pkgname}" | \
        grep "${pkgname}" | awk '{print $1}'`
    for build in $builds;do
        # task urls sometimes have ".git" suffix
        git_sha=`koji buildinfo "${build}" | grep '^Task:' | \
            sed -e "s|.*${pkgname}\(\.git\)*:\(.*\))|\2|"`
        version=`echo $build | sed -e "s:${pkgname}-::"`
        echo BUILD: $pkgname\($version\) = $git_sha
        git tag "${version}" "${git_sha}"
    done
done
Enjoy!

Share/Save/Bookmark
Nov 11, 2011

fedora-review - Package reviews made easier

Posted by Stanislav Ochotnicky at 4:15 PM / Labels: , , , , , , / 0 comments
For package to be included in default Fedora repositories, it has to go through process called package review. If you've done a few package reviews you know big chunks of this process are repeated ad-nausea in every review.
There have been quite a few tools aimed at automating and simplifying this process. However they all had one major flaw. They have been designed for reviewing specific class of packages, be it Perl, Python or generic C/C++ packages. Few us us decided to change this.
We used Tim Lauridsen's FedoraReview package as a base for our work and started adding new features and tweaks. Current work has a website on fedorahosted where you'll find all important information. Full feature list would be quite long, but I'll list a few major things:
  • Bugzilla integration
  • Mock integration
  • JSON api for external plugins (more info further down)
  • Several automated tests
The tool runs all checks/tests on spec file and rpms and writes output into a text file. Snippet of the output looks like this: 
Package Review
==============

Key:
- = N/A
x = Check
! = Problem
? = Not evaluated



==== Generic ====
...
[ ]: MUST License field in the package spec file matches the actual license.
[ ]: MUST License file installed when any subpackage combination is installed.
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
        Using both %{buildroot} and $RPM_BUILD_ROOT
...
[x]: SHOULD Spec use %global instead of %define.
...

==== Java ====
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment

Issues:
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
        Using both %{buildroot} and $RPM_BUILD_ROOT
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment
We display only relevant results. In other words if there are no post/postun scriptlets there is no reason to include sanity output checking in the template. This will make more and more sense as we add more checks.

JSON API

So how is it that different people will be able to write additional plugins for this review tool? We provide a relatively simple JSON api through stdin/stdout.
So to create a new check plugin you create a script or program in your language of choice. There is only one requirement:
  • Programming language has to have JSON format support
When the review tool runs your plugin it will send following message on its stdin: 
{
    "supported_api": 1,
    "pkgname": "package name",
    "version": "package version",
    "release": "package release",
    "srpm":"path/to/srpm",
    "spec":{ path: "path/to/spec",
            "text": "spec text with expanded macros"},
    "rpms":[ "path/to/rpm", ...],
    "rpmlint": "rpmlint output",
    "build_dir": "/path/to/src/directory/after/build"
}
When your plugin is done with checks it returns following message by writing it to stdout: 
{
    "command": "results",
    "supported_api": 1,
    "checks":
    [
        {
         "name": "CheckName",
         "url": "URL to guidelines usually",
         "group": "Group for this test.(Java, Perl, etc.)",
         "text": "Check description that shows on review template",
         "deprecates":["DeprecatedTest", ...]
         "type": "MUST"|"SHOULD",
         "result": "pass"|"fail"|"inconclusive",
         "extra_output": "text",
        },
        ...
    ]
}
If the plugin closes stdout without writing anything there, it means there were no relevant automated tests to run and no non-automated tests to include in template for manual evaluation. This is useful so we don't include for example Perl-related test output for Java packages and vice-versa.

Roadmap

While the tool is already usable and soon to be packaged in Fedora, there are still quite a few things we want to improve:
  • Add more functions to API (currently there is just get_section)
  • Automate all automatable tests currently available
  • Get rid of redundant tests (don't duplicate rpmlint)
  • Add more tests of course!
  • Maybe add templating support?
We have currently 3-4 active committers, checks for C/C++, generic, Java, R packages. There is already and example external plugin written in Perl. If you have any improvement ideas, bugreports or just want to tell us we suck because we should have done X...get in touch!

Share/Save/Bookmark
Subscribe to: Posts (Atom)