Ubuntu Developer Week: Getting started with Ubuntu Development : Part II

Tuesday, Jul 13th, 2010

Link

Before going ahead, please bookmark MOTU - Getting started and for more questions you can visit #ubuntu-motu and #ubuntu-packaging

augdawg: fingerprint authent. on lanchpad wont work
dholbach: please follow https://help.launchpad.net/YourAccount/ImportingYourPGPKey very closely and try again or talk to fine people at #launchpad

devildante: Will we fix real bugs?
dholbach: yes

There is a very real sort of bugs, FBTFS bugs and with that a new acronym, FTBFS means Fails To Build From Source

If a package doesn’t build, it can be because all kinds of crazy things. Sometimes the source code is actually broken, sometimes it’s because we lack the right version of some development library etc. It’s something you’ll probably struggle with a couple of times as an aspiring developer :)

So let’s head to http://qa.ubuntuwire.org/ftbfs/ - it shows a list of packages that FTBFS (yes it’s used as a verb) right now. Let’s first have a look at http://launchpadlibrarian.net/50406598/buildlog_ubuntu-maverick-i386.pykickstart_1.74-1_FAILEDTOBUILD.txt.gz which is the compressed log of the build on launchpad

So what you just saw when you ran pbuilder, just on lots of machines in Launchpad. First part is the installation of necessary packages, last part is the deinstallation of packages

So close to the end of the log you’ll find this

dh_usrlocal
dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksvalidator is not a directory
dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksflatten is not a directory
dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksverdiff is not a directory
rmdir: failed to remove `debian/python-pykickstart/usr/local/bin’: Directory not empty
dh_usrlocal: rmdir debian/python-pykickstart/usr/local/bin returned exit code 1

So something with files being installed in usr/local as the packagers, don’t do usr/local - it’s against the rules. usr/local is just for stuff the user installs manually. You can find more about what goes where in the FHS (file hierarchy standard) and the debian policy document.

“upstreams” are the software authors of packages included in ubuntu, they often release their “upstream source” on their own website, it often gets included in debian, where patches are added to make the packages build the debian (and ubuntu) way, and we often inherit the code just like that. The more upstream and debian and ubuntu are in sync the better. Sometimes we make different decisions, but those should be for very good reasons because every deviation means additional work when merging changes later on again. So, when I look at a bug like that I very often check “was it fixed in Debian or upstream already”?

So the package is called pykickstart and let’s check out https://launchpad.net/ubuntu/+source/pykickstart as you can see, it was introduced in maverick and is at version 1.74-1 right now

On the Debian side, we see similar information at http://packages.debian.org/src:pykickstart you can see that ‘sid’, the Debian development version, has 1.75-1 already. Click on the 'sid’ link to get to http://packages.debian.org/source/sid/pykickstart and on the right side it has a link to the debian changelog. So this is a regular debian/ubuntu changelog entry that states what in this package upload was changed. Ss you can see, the maintainer wrote something about a new upstream version and “ * Update debian/rules: pass –buildsystem=python_distutils to avoid ftbfs.” “avoid ftbfs” is like music to my ears - to yours too?

We should get the Debian source and try to build it and see if that works. if you go back to http://packages.debian.org/source/sid/pykickstart - you’ll see a link to a .dsc file http://ftp.de.debian.org/debian/pool/main/p/pykickstart/pykickstart_1.75-1.dsc

When we installed the devscripts package earlier, we got a tool called dget which we’ll use that now to download the source from debian as

dget -xu http://ftp.de.debian.org/debian/pool/main/p/pykickstart/pykickstart_1.75-1.dsc

To build it, you know already what to do:

sudo pbuilder build pykickstart_1.75-1.dsc

this will take a little bit now, so let’s all try to be patient and see how it works out

Questions

Noz3001: Why -“xu”?
dholbach: without -x dget will not unpack the source (you’re right, we could have ignored that - usually it’s very helpful to get the source unpacked immediately to dive right into it) and without -u dget will complain about a missing gpg key, because it can’t confirm the identity of the uploader (you probably won’t have the debian maintainer’s public key in your keyring)

umutuygar: umut@umut-laptop:~/code/ubuntu-classroom$ sudo pbuilder build pykickstart_1.75.dsc
dholbach: did you execute the dget command above in the same directory?

ElPasmo: How you get a sponsor?
dholbach: we’ll get to that in a bit, https://wiki.ubuntu.com/SponsorshipProcess if you’re impatient :)

chilicuil: can I run 2 or more pbuilder instances at the same time?
dholbach: yes, you can use pbuilder-dist (in the ubuntu-dev-tools package) for that and also there’s https://wiki.ubuntu.com/PbuilderHowto

penguin42: We’re past the point of pulling in debian packages for Maverick aren’t we? But if a package ftbfs and the debian one fixes it what happens?
dholbach: I’ll get to that in a sec

Krysis: how to add gpg into keyring?
dholbach: gpg –recv-keys <KEY ID>

umang: What’s the easiest way to find a nice diff of the two source packages (1.74-1 from ubuntu and 1.75-1 from debian)?
dholbach: nice question. You'd apt-get source pykickstart to get the ubuntu version too and then run
debdiff pykickstart_1.74-1.dsc pykickstart_1.75-1.dsc
(probably pipe it through less, filterdiff, etc afterwards)

Alright, pykickstart 1.75-1 from Debian succeeded to build on this end, so what do we do?

Let me explain a bit how Ubuntu and Debian fit in to the big picture of the release schedule I described earlier up until Debian Import Freeze, we automatically sync source packages from Debian and build them in Launchpad, if:

The package in Ubuntu is not modified (usually obvious by reading the version number, 1.74-1 vs. 1.74-1ubuntu1)
If the package is in Debian main (so free software)

So how do we live in this world?

If we introduce a change in Ubuntu, say we go from 1.74-1 to 1.74-1ubuntu1, then Debian introduces 1.75-1 so we need to decide if we can overwrite our changes (we sync the source) or if we need to merge the source manually

It’s very tempting to say “ha, no we can sync from Debian” but it’s INCREDIBLY IMPORTANT to READ THE DIFF as just reading the changelog is not good enough

READ THE DIFF

and TEST

Questions

malcolmci: dholbach: who decides if we should sync with upstream debian or keep ubuntu modification, for example ?
dholbach: That’s the best possible question. That’s the responsibility you have as a developer. You need to make sure in the best way possible that the actual change can be implemented in the way you suggested. If you’re unsure, you can ask lots of other fine people in #ubuntu-devel or #ubuntu-motu or #ubuntu-packaging. Also the sponsoring process (somebody reviews your code or suggestion first) helps with that in the beginning. It’s just important that you do your best in making sure and testing.

malcolmci: ah, so the package maintainer decides?
dholbach: In Ubuntu we don’t have dedicated package maintainers, so we maintain the packages as a big team.

mickstephenson52: would the easiest way to test be toi just install the deb file from debian and see if it works?
dholbach: thanks for getting me back on track :)

So, we have 1.74-1 in maverick and we have 1.75-1 in Debian sid, which means that we did not introduce changes in Ubuntu, so we’re sure that we don’t have to merge things manually.

We just verified that the package builds again in the new version and we’re still before feature freeze in maverick, so requesting a sync (https://wiki.ubuntu.com/SyncRequestProcess) should be fine, once we’re sufficiently sure it’s all good

Installing the .deb and see if it work is a great way

AudioFranky: I would expect that for very essential packages, like glibc, there is more than a single person to decide whether to keep or update?

dholbach: yes, those decisions are made at UDS in a bigger group and usually there’s agreement between people in Ubuntu and Debian who work on the package. When I said that “we don’t have dedicated maintainers” I meant that we have no “big maintainer lock” (somebody OWNS the package), but indeed we have people with a special area of expertise and that’s respected

Alright, the path ahead seems clear: test the package, make sure it’s alright, then engage with the sync request process.

Jere’s some more food for though, you can try it out yourself:

but as I said: be careful and talk to people on #ubuntu-motu or #ubuntu-packaging if you’re unsure. Being unsure is a good thing and knowing when to ask a sign of taking responsibility and that’s valued.

Alright, now I have another interesting case for us:

http://launchpadlibrarian.net/49981844/buildlog_ubuntu-maverick-i386.xpad_4.0-5_MANUALDEPWAIT.txt.gz

the xpad package fails to build too.

Towards the bottom of the log you can see this:

libmagickcore-extra: missing
libmagickcore-extra: does not exist

so this means that one of the packages that is specified as a requirement to build the xpad package is not there. Let’s get the source code and let’s try to fix it. This time we’ll do it differently:

bzr branch lp:ubuntu/xpad

this will not get us the source package (remember the .dsc, .orig.tar.gz and .diff.gz file) but a bazaar branch (revision control system) with all revisions in Ubuntu and Debian. This will take a little bit, but is worth it.

In the case of source packages you just get one revision, in a format that works with all the debian/ubuntu build tools, but we’re slowly moving towards a world where we work with upstreams and debian and distributed version control is just what’s easier to use. I’d like to show you how easy to use it is

Alright, once bzr is done, run

cd xpad

and

less debian/control

Let me explain a bit what this is about:

debian/control contains information about the source package and the resulting .deb (binary) packages. In this case it’s relatively simple, let’s go through the source stanza first.

Source: specifies the name
Section and Priority are used by the package management to have a bit of metadata
Maintainer is the maintainer in Debian
Build-Depends is just what we’re looking for and this is the minimal list of packages necessary to build the package
Standards-Version is the version of the debian policy this complies with and homepage just more metadata

This is all additional info regarding the source: The resulting xpad package is defined in the next stanza

Package: is the package name
“Architecture: any” means that the package needs to get built on every individual architecture available separately
Depends: is the list of dependencies of the resulting package
shlibs:Depends is the list of library packages that contain libraries the binaries in the deb packages are linked against
misc:Depends a few additional Depends that might be useful

architectures are i386, amd64, powerpc, sparc, etc.

If you just put a bunch of python scripts into a package, or just some documentation that is architecture independent, you use “Architecture: all”.

All of these complicated computations are done by scripts in the debhelper package.

Description too is used by the package management tools

So debian/control is a file that contains lots of the definition of the package itself. You can check out the other files in debian/ on your own later on, https://wiki.ubuntu.com/PackagingGuide will help you with that.

bcurtiswx: will we ever mess with the bottom section?
dholbach: we could, for example if a user tells us that there’s a dependency missing or there’s a typo in the package description or something

So to avoid confusion: this example will just work in maverick, it’s a maverick build failure. If you don’t run maverick right now, that’s no big deal, just trust me blindly then.

http://launchpadlibrarian.net/49981844/buildlog_ubuntu-maverick-i386.xpad_4.0-5_MANUALDEPWAIT.txt.gz mentioned that libmagickcore-extra was missing

apt-cache search libmagickcore-extra (on maverick) will reveal that it’s now called libmagickcore3-extra

BeardyGnome13: i guess it’s best to run maverick on a separate machine / VM then?
dholbach: https://wiki.ubuntu.com/UbuntuDevelopment/UsingDevelopmentReleases has all that info

Ok, so we just change libmagickcore-extra to libmagickcore3-extra in debian/control and save the file. If you exit your editor and run bzr diff you’ll see your change. Now please run update-maintainer (script from ubuntu-dev-tools) to update the maintainer field which we were asked to do by our friends at debian to indicate that we made changes in the package)

now please run

dch -i

Also a devscripts tool which deals with all things related to debian/changelog it will automatically fill out some stuff for you. You just need to enter a descriptive message for your changes. I’d go for something like

* debian/control: updated build-depends from libmagickcore-extra to libmagickcore3-extra to fix FTBFS.

It’s important to note what you changed in which files and the more explicit you are about your changes, the easier it will be for you and others later on in understanding what you did. If you close a bug with your fix, please add (LP: #123456). This special syntax will automatically close the bug once it gets merged in.

Now please save the file and run

bzr bd – -S -us -uc

This will build the source package from the branch. (-S -us -uc are arguments for debuild which is used internally) Then you move out of the directory and pbuilder the generated source package.

To test if that works now, this will naturally take a bit. Once you’re happy with all the C A R E F U L tests you did you can go back into the branch, run debcommit and then push the changes to launchpad and request a merge

https://wiki.ubuntu.com/Bugs/HowToFix explains the last bits and please check out https://wiki.ubuntu.com/MOTU/GettingStarted and join #ubuntu-motu and #ubuntu-packaging

As you can see, hugs are a vital ingredient of making Ubuntu ROCK :)

∞10:17 pm: manishsinha11 notes