Before going ahead, please bookmark MOTU - Getting started and for more questions you can visit #ubuntu-motu and #ubuntu-packaging

augdawg: fingerprint authent. on lanchpad wont work

dholbach: please follow https://help.launchpad.net/YourAccount/ImportingYourPGPKey very closely and try again or talk to fine people at #launchpad

devildante: Will we fix real bugs?

dholbach: yes

There is a very real sort of bugs, FBTFS bugs and with that a new acronym, FTBFS means Fails To Build From Source

If a package doesn’t build, it can be because all kinds of crazy things. Sometimes the source code is actually broken, sometimes it’s because we lack the right version of some development library etc. It’s something you’ll probably struggle with a couple of times as an aspiring developer :)

So let’s head to http://qa.ubuntuwire.org/ftbfs/ - it shows a list of packages that FTBFS (yes it’s used as a verb) right now. Let’s first have a look at http://launchpadlibrarian.net/50406598/buildlog_ubuntu-maverick-i386.pykickstart_1.74-1_FAILEDTOBUILD.txt.gz which is the compressed log of the build on launchpad

So what you just saw when you ran pbuilder, just on lots of machines in Launchpad. First part is the installation of necessary packages, last part is the deinstallation of packages

So close to the end of the log you’ll find this

dh_usrlocal

dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksvalidator is not a directory

dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksflatten is not a directory

dh_usrlocal: debian/python-pykickstart/usr/local/bin/ksverdiff is not a directory

rmdir: failed to remove `debian/python-pykickstart/usr/local/bin’: Directory not empty

dh_usrlocal: rmdir debian/python-pykickstart/usr/local/bin returned exit code 1

So something with files being installed in usr/local as the  packagers, don’t do usr/local - it’s against the rules. usr/local is just for stuff the user installs manually. You can find more about what goes where in the FHS (file hierarchy standard) and the debian policy document.

“upstreams” are the software authors of packages included in ubuntu, they often release their “upstream source” on their own website, it often gets included in debian, where patches are added to make the packages build the debian (and ubuntu) way, and we often inherit the code just like that. The more upstream and debian and ubuntu are in sync the better. Sometimes we make different decisions, but those should be for very good reasons because every deviation means additional work when merging changes later on again. So, when I look at a bug like that I very often check “was it fixed in Debian or upstream already”?

So the package is called pykickstart and let’s check out https://launchpad.net/ubuntu/+source/pykickstart as you can see, it was introduced in maverick and is at version 1.74-1 right now

On the Debian side, we see similar information at http://packages.debian.org/src:pykickstart you can see that ‘sid’, the Debian development version, has 1.75-1 already. Click on the ‘sid’ link to get to http://packages.debian.org/source/sid/pykickstart and on the right side it has a link to the debian changelog. So this is a regular debian/ubuntu changelog entry that states what in this package upload was changed. Ss you can see, the maintainer wrote something about a new upstream version and “   * Update debian/rules: pass —buildsystem=python_distutils to avoid ftbfs.” “avoid ftbfs” is like music to my ears - to yours too?

We should get the Debian source and try to build it and see if that works. if you go back to http://packages.debian.org/source/sid/pykickstart - you’ll see a link to a .dsc file http://ftp.de.debian.org/debian/pool/main/p/pykickstart/pykickstart_1.75-1.dsc

When we installed the devscripts package earlier, we got a tool called dget which we’ll use that now to download the source from debian as

dget -xu http://ftp.de.debian.org/debian/pool/main/p/pykickstart/pykickstart_1.75-1.dsc

To build it, you know already what to do:

sudo pbuilder build pykickstart_1.75-1.dsc

this will take a little bit now, so let’s all try to be patient and see how it works out

Questions

Noz3001:  Why -“xu”?

dholbach: without -x dget will not unpack the source (you’re right, we could have ignored that - usually it’s very helpful to get the source unpacked immediately to dive right into it) and without -u dget will complain about a missing gpg key, because it can’t confirm the identity of the uploader (you probably won’t have the debian maintainer’s public key in your keyring)

umutuygar: umut@umut-laptop:~/code/ubuntu-classroom$ sudo pbuilder build pykickstart_1.75.dsc

dholbach: did you execute the dget command above in the same directory?

ElPasmo: How you get a sponsor?

dholbach:  we’ll get to that in a bit, https://wiki.ubuntu.com/SponsorshipProcess if you’re impatient :)

chilicuil: can I run 2 or more pbuilder instances at the same time?

dholbach: yes, you can use pbuilder-dist (in the ubuntu-dev-tools package) for that and also there’s https://wiki.ubuntu.com/PbuilderHowto

penguin42: We’re past the point of pulling in debian packages for Maverick aren’t we? But if a package ftbfs and the debian one fixes it what happens?

dholbach:  I’ll get to that in a sec

Krysis:  how to add gpg into keyring?

dholbach: gpg —recv-keys <KEY ID>

umang: What’s the easiest way to find a nice diff of the two source packages (1.74-1 from ubuntu and 1.75-1 from debian)?

dholbach: nice question. You’d   apt-get source pykickstart   to get the ubuntu version too and then run

debdiff pykickstart_1.74-1.dsc pykickstart_1.75-1.dsc

(probably pipe it through less, filterdiff, etc afterwards)

Alright, pykickstart 1.75-1 from Debian succeeded to build on this end, so what do we do?

Let me explain a bit how Ubuntu and Debian fit in to the big picture of the release schedule I described earlier up until Debian Import Freeze, we automatically sync source packages from Debian and build them in Launchpad, if:

1. The package in Ubuntu is not modified (usually obvious by reading the version number, 1.74-1 vs. 1.74-1ubuntu1)
2. If the package is in Debian main (so free software)

So how do we live in this world?

If we introduce a change in Ubuntu, say we go from 1.74-1 to 1.74-1ubuntu1, then Debian introduces 1.75-1 so we need to decide if we can overwrite our changes (we sync the source) or if we need to merge the source manually

It’s very tempting to say “ha, no we can sync from Debian” but it’s INCREDIBLY IMPORTANT to READ THE DIFF as just reading the changelog is not good enough

READ THE DIFF

and TEST

Questions

malcolmci: dholbach: who decides if we should sync with upstream debian or keep ubuntu modification, for example ?

dholbach: That’s the best possible question. That’s the responsibility you have as a developer. You need to make sure in the best way possible that the actual change can be implemented in the way you suggested. If you’re unsure, you can ask lots of other fine people in #ubuntu-devel or #ubuntu-motu or #ubuntu-packaging. Also the sponsoring process (somebody reviews your code or suggestion first) helps with that in the beginning. It’s just important that you do your best in making sure and testing.

malcolmci: ah, so the package maintainer decides?

dholbach: In Ubuntu we don’t have dedicated package maintainers, so we maintain the packages as a big team.

mickstephenson52: would the easiest way to test be toi just install the deb file from debian and see if it works?

dholbach: thanks for getting me back on track :)

So, we have 1.74-1 in maverick and we have 1.75-1 in Debian sid, which means that we did not introduce changes in Ubuntu, so we’re sure that we don’t have to merge things manually.

We just verified that the package builds again in the new version and we’re still before feature freeze in maverick, so requesting a sync (https://wiki.ubuntu.com/SyncRequestProcess) should be fine, once we’re sufficiently sure it’s all good

Installing the .deb and see if it work is a great way

AudioFranky: I would expect that for very essential packages, like glibc, there is more than a single person to decide whether to keep or update?

dholbach: yes, those decisions are made at UDS in a bigger group and usually there’s agreement between people in Ubuntu and Debian who work on the package. When I said that “we don’t have dedicated maintainers” I meant that we have no “big maintainer lock” (somebody OWNS the package), but indeed we have people with a special area of expertise and that’s respected

Alright, the path ahead seems clear: test the package, make sure it’s alright, then engage with the sync request process.

Jere’s some more food for though, you can try it out yourself:

• http://launchpadlibrarian.net/50981070/buildlog_ubuntu-maverick-i386.xdotool_1:2.20100602.2915-1_FAILEDTOBUILD.txt.gz
• http://launchpadlibrarian.net/50411959/buildlog_ubuntu-maverick-i386.weave_1.3-2_FAILEDTOBUILD.txt.gz

but as I said: be careful and talk to people on #ubuntu-motu or #ubuntu-packaging if you’re unsure. Being unsure is a good thing and knowing when to ask a sign of taking responsibility and that’s valued.

Alright, now I have another interesting case for us:

http://launchpadlibrarian.net/49981844/buildlog_ubuntu-maverick-i386.xpad_4.0-5_MANUALDEPWAIT.txt.gz

the xpad package fails to build too.

Towards the bottom of the log you can see this:

• libmagickcore-extra: missing
• libmagickcore-extra: does not exist

so this means that one of the packages that is specified as a requirement to build the xpad package is not there. Let’s get the source code and let’s try to fix it. This time we’ll do it differently:

bzr branch lp:ubuntu/xpad

this will not get us the source package (remember the .dsc, .orig.tar.gz and .diff.gz file) but a bazaar branch (revision control system) with all revisions in Ubuntu and Debian. This will take a little bit, but is worth it.

In the case of source packages you just get one revision, in a format that works with all the debian/ubuntu build tools, but we’re slowly moving towards a world where we work with upstreams and debian and distributed version control is just what’s easier to use. I’d like to show you how easy to use it is

Alright, once bzr is done, run

cd xpad

and

less debian/control

Let me explain a bit what this is about:

debian/control contains information about the source package and the resulting .deb (binary) packages. In this case it’s relatively simple, let’s go through the source stanza first.

• Source: specifies the name
• Section and Priority are used by the package management to have a bit of metadata
• Maintainer is the maintainer in Debian
• Build-Depends is just what we’re looking for and this is the minimal list of packages necessary to build the package
• Standards-Version is the version of the debian policy this complies with and homepage just more metadata

This is all additional info regarding the source: The resulting xpad package is defined in the next stanza

• Package: is the package name
• “Architecture: any” means that the package needs to get built on every individual architecture available separately
• Depends: is the list of dependencies of the resulting package
• shlibs:Depends is the list of library packages that contain libraries the binaries in the deb packages are linked against
• misc:Depends a few additional Depends that might be useful

architectures are i386, amd64, powerpc, sparc, etc.

If you just put a bunch of python scripts into a package, or just some documentation that is architecture independent, you use “Architecture: all”.

All of these complicated computations are done by scripts in the debhelper package.

Description too is used by the package management tools

So debian/control is a file that contains lots of the definition of the package itself. You can check out the other files in debian/ on your own later on, https://wiki.ubuntu.com/PackagingGuide will help you with that.

bcurtiswx: will we ever mess with the bottom section?

dholbach: we could, for example if a user tells us that there’s a dependency missing or there’s a typo in the package description or something

So to avoid confusion: this example will just work in maverick, it’s a maverick build failure. If you don’t run maverick right now, that’s no big deal, just trust me blindly then.

http://launchpadlibrarian.net/49981844/buildlog_ubuntu-maverick-i386.xpad_4.0-5_MANUALDEPWAIT.txt.gz mentioned that libmagickcore-extra was missing

apt-cache search libmagickcore-extra (on maverick) will reveal that it’s now called libmagickcore3-extra

BeardyGnome13: i guess it’s best to run maverick on a separate machine / VM then?

dholbach: https://wiki.ubuntu.com/UbuntuDevelopment/UsingDevelopmentReleases has all that info

Ok, so we just change libmagickcore-extra to libmagickcore3-extra in debian/control and save the file. If you exit your editor and run    bzr diff    you’ll see your change. Now please run update-maintainer (script from ubuntu-dev-tools) to update the maintainer field which we were asked to do by our friends at debian to indicate that we made changes in the package)

now please run

dch -i

Also a devscripts tool which deals with all things related to debian/changelog it will automatically fill out some stuff for you. You just need to enter a descriptive message for your changes. I’d go for something like

* debian/control: updated build-depends from libmagickcore-extra to libmagickcore3-extra to fix FTBFS.

It’s important to note what you changed in which files and the more explicit you are about your changes, the easier it will be for you and others later on in understanding what you did. If you close a bug with your fix, please add   (LP: #123456). This special syntax will automatically close the bug once it gets merged in.

Now please save the file and run

bzr bd — -S -us -uc

This will build the source package from the branch. (-S -us -uc are arguments for debuild which is used internally) Then you move out of the directory and pbuilder the generated source package.

To test if that works now, this will naturally take a bit. Once you’re happy with all the  C A R E F U L   tests you did you can go back into the branch, run debcommit and then push the changes to launchpad and request a merge

https://wiki.ubuntu.com/Bugs/HowToFix explains the last bits and please check out https://wiki.ubuntu.com/MOTU/GettingStarted and join #ubuntu-motu and #ubuntu-packaging

As you can see, hugs are a vital ingredient of making Ubuntu ROCK :)

The first session of Ubuntu Develop week on “Getting started with Ubuntu Development” just ended. It was taken up by Daniel Holbach. I know reading IRC logs is boring, so here is sum up everything in human readable format. In between you will find quotation, which are actually when Daniel replies to some question asked by a person.

The entire text is given out by Daniel Holbach which I have converted from first person to third person speech.

IRC Channel: #ubuntu-classroom for classes (Freenode) and #ubuntu-classroom-chat for all discussions and asking questions

The whole session is of two parts

• Getting a couple of tools installed and the environment set-up
• Actively having a look at a few bugs and fixing them

Bookmark  MOTU Getting Started and Ubuntu Development: Using Development releases

It is necessary to get the latest development release. It needs not be installed as main system but can be on a Virtual Machine (example Virtualbox) so that we can have access to all the current development libraries, modules and stuff and you can test packages better

Now go and enabled “Source code” and “universe” in System → Software Sources → Ubuntu Software if you havn’t

Questions:

tillux: wouldn’t it make sense to do that now? because you need to download a lot of things/packages etc

dholbach: no, it’d take too much time. It’s fine - you can just copy the instructions later on

umutuygar: What programming language r u going to be using?

dholbach: we’ll see :-) I think for now I’ll stick to packaging basics where we fix a couple of bugs

BeardyGnome13: QUESTION: will this be GNOME-centric?

dholbach: no, not at all

Type this command in terminal

sudo apt-get install —no-install-recommends bzr-builddeb ubuntu-dev-tools fakeroot build-essential gnupg pbuilder debhelper

which will give you a bunch of tools that are going to be useful generally, not just in these examples

The explanation of the packages

• bzr-builddeb pulls in bzr which we’ll use to get the source code for one or two examples
• ubuntu-dev-tools pulls in devscripts which both are incredibly helpful at making repetitive packaging tasks easy
  fakeroot is needed by debuild (in devscripts) to mimic root privileges when installing files into a package
  build-essential pulls in lots of useful very basic build tools like gcc, make, etc
  gnupg is used to sign files in our case (uploads in the future)
  pbuilder is a build tool that builds source in a sane, clean and minimal environment it sets up itself
  debhelper contains scripts that automate lots of the build process in a package

Creating a GPG key

Now let’s move on the get our gpg key which we use it to sign files for upload but more generally it is used to sign and encrypt mails, files or text generally.

We use it to indicate that WE were the last to touch a file and not somebody else that ensures that only people who we know about get to upload packages

To generate a key, please run

 gpg —gen-key

sticking to the defaults is totally fine, for example don’t you need a comment. If you need more info on gpg keys, head to https://help.ubuntu.com/community/GnuPrivacyGuardHowto which talks about everything in more detail.

Enter your name, email address and just stick to the default values for now. It could be that gpg is still sitting there and waiting for more random data to generate your key - that’s expected and fine. Just open another terminal while we carry on, it’ll finish on its own.

If you have your own GPG key already, skip this step.

Setting up pbuilder

Please open an editor and edit the file ~/.pbuilderrc (create if you don’t have it yet). Please add the following content to the file

COMPONENTS=”main universe multiverse restricted”

and save it. Once you’re done, please run

sudo pbuilder create

More about pbuilder:

• It builds packages in a clean and minimal environment
• It keeps your system “clean” (so you don’t install millions of build dependencies on your own system)
• It makes sure the package builds in a minimal, unmodified environment
• Oo you ensure that the package does not just build because you made lots of changes on your system, but the build is reproducable
• You can update package lists (later on) with: sudo pbuilder update
• To build packages you run: sudo pbuilder build package_version.dsc

BeardyGnome13: will the transcript of the classroom be available later?

leak-BebeChooCho: Will there be full log available?

dholbach: Yes. It will be linked from https://wiki.ubuntu.com/UbuntuDeveloperWeek. For the impatient: http://irclogs.ubuntu.com

How pbuilder works:

• It first gets the minimal packages for a base system
• Stores them in a tarball
• whenever you build a package it’ll untar the base tarball, then install whatever your current build requires, then build it, then tear it all down again

somethinginteres: To confirm, the install of pbuilder etc should be done -right now- on our main box?

dholbach: It helps as you can perform the examples on your own machine and have it set up later on too

While gpg and pbuilder are doing their work, lets try out other tools

If you use the bash shell which is default, please edit  ~/.bashrc and at the end of it, please add something like

DEBFULLNAME=”Daniel Holbach”
DEBEMAIL=”daniel.holbach@ubuntu.com”

and save it. PLEASE USE YOUR OWN NAME, THANKS :-)

abhi_nav: I am doing everything in my main day to day usable machine. is it ok? I dont want anything to ruine my cute ubuntu. :)

dholbach: yes, it’ll be fine

once you’re done editing ~/.bashrc, please run   source ~/. bashrc  (it’s only needed once)

tpw_rules87: Should these match the values we put into GPG?

dholbach: yes

OK, with this out of the way, the packaging tools will know you by your name and you don’t need to enter it, for example if you do changelog entries, etc.

Questions:

augdawg: where is the .bashrc file located?

dholbach: ~/.bashrc , so /home/<your user name>/.bashrc

abhi_nav:do i need to write my that email which i was used to create gpgp key some months ago? now I have another email

dholbach: yes, it helps to have the same in there, but you can also add another user id to your gpg key later on. Refer to https://help.ubuntu.com/community/GnuPrivacyGuardHowto for that

malcolmci: is the preferred arch for dev’ing AMD64 or i386 ?

dholbach: every supported architecture of ubuntu will work fine

devildante: Will this session revolve around packaging bugs, or bugs in general?

dholbach: I picked a few packaging bugs, but maybe we’ll do something else later on, let’s see :)

augdawg: what does the ~ mean?

dholbach: my user name is “daniel” , so ~ for me will be expanded to /home/daniel

tillux: for those running on lucid: should pbuilder have been triggered with “—distribution maverick” ?

dholbach: for now that’s fine - if you set up a maverick chroot/virtual-machine/partition/machine later on, you can just repeat the steps. For this sessions it’s irrelevant.

BeardyGnome13: is my gpg key machine-specific?

dholbach: no, you can just copy it to another machine

somethinginteres: PBuilder is downloading for Lucid, should it be downloading for Mavrick or Lucid?

dholbach: see what I just said to tillux above

theneoindian: i hate to ask , but will pbuilder create eat up my b/w .. i’m on a limited b/w connection …

dholbach: you better stop it then and repeat later on on a different connection

By the time  pbuilders and gpgs are done, we keep talking a bit about ubuntu development and how it all works, we’ll make some good use of pbuilder and your shiny new gpg key later on :)

First things first: ubuntu is very special in how it’s produced and how we all work as you know it comes out every 6 months. That means we have a tight release schedule and everything we do and work on is defined by that schedule. Check out https://wiki.ubuntu.com/MaverickReleaseSchedule for the current release schedule for maverick.

The quick version: green means: lots is allowed here, red means: almost nothing is allowed here :)

Long version:

• Toolchain is uploaded for the new release (gcc, binutils, libc, etc.), so the most basic build tools are there
• New changes that happened in the meantime are synced or merged (more on that later on)
• Ubuntu developer summit (uds) happens where features are defined and talked about
• Up until debian import freeze we import source changes from debian semi-automatically
• Up until feature freeze we get new stuff in, work on features, try to make it all work
• If a feature is not half-way there yet by feature freeze, it will likely get deferred to the next release
• From feature freeze on you can see that lots of freezes are added throughout the weeks and you’ll need more and more freeze exceptions for big changes

The focus is clearly: testing, testing, testing and fixing, fixing, fixing.

The more obvious fixes are (a huge 20 million line patch is not obvious) the more obvious the better. The same goes for fixes that are introduced AFTER the release. We just want to fix stuff in …-updates if it’s REALLY URGENT stuff with really simple fixes. So as you can imagine, this means: introduce big changes early in the release cycle, so we can shake out problems over a longer time.

devildante: On the release schedule, I see a column named “Work Item Iteration”

dholbach: Ah, yes. If you check out http://people.canonical.com/~pitti/workitems/maverick/all-ubuntu-10.10.html you will see a graph and loads of text. This is a list of work items defined by specifications written at the ubuntu developer summit.  Work items are specific pieces of work that somebody at UDS decided to work on during the cycle. It’s more a tool for the managers at Canonical to keep their people busy. I mean… make sure we’re on track ;-)

Questions:

marceau: (I’ve asked this during the Userday as well but am wondering as to your input) Do you feel that the fast release cycle of Ubuntu might be hindering it’s progress? It seems a lot of time is spent in freezing and testing rather than working out new features.

dholbach: Not at all - I think it’s a good thing - it forces us to stay focused and make sure that stuff gets done. If we don’t get a feature done this cycle, it’ll be next cycle. Also it’s important for everybody else who works with the project, as it’s very easily predictable.

Daniel on how to get your stuff in:

When I spoke about gpg you noticed that I said that only people who “we know” get to upload packages directly. This means that as a new contributor you will have to work with sponsors who basically review your work and upload it for you. Once you did that a couple of times and they recognise you and your good work, you can apply for ubuntu developer membership and you can ask the people you’ve worked with for comments on your application.

It’s really not very complicated, you basically set up a wiki page with your contributions, ask for comments and submit for an irc meeting of the developer membership board and you’re done. No need to learn a secret handshake, send me money or anything else. It’s contributions and good work that counts. It helps if you’re a team player :)

Questions:

augdawg: what does an ubuntu developer membership get you?

dholbach: https://wiki.ubuntu.com/UbuntuDevelopers will tell you :). The most obvious thing is that you can upload changes yourself

prayii: But will sending you money help the process? *wink*

dholbach: unfortunately I’m not on the Developer Membership Board

tech2077: so when your a member do you get the ability to fix bugs somewhat more independently, trying to word it right

dholbach: It’s a question of trust: once you demonstrated that you do good work, you can get upload rights. Same goes for Ubuntu membership. People get an @ubuntu.com email address when they can be recognised for their good work in Ubuntu

marceau: what is the expected knowledge level of people taking part in this week’s sessions?

dholbach: I guess it will differ from session to session: in this session it’s good enough to be curious and have played with a terminal before and have a knack for making things work again :)

somethinginteres: is it the case that being an Ubuntu dev means being a programming vet or are there plenty of opportunity for n00bs to learn and contribute something..?

dholbach: there are lots of opportunities to learn, which is why we’re doing this whole thing :)

umang: If I’d like to contribute to both Ubuntu and Debian, which should I use?

dholbach: you can contribute to Debian by using Ubuntu and collaborating with Debian developers. It totally depends on you what you use and what you work on.

Now assuming pbuilder setup is complete, try getting the source of hello package

apt-get source hello

and run

sudo pbuilder build hello_*.dsc

this will kick off the build

abhi_nav: what .dsc stands for?

dholbach: great question. What “apt-get source” downloads is: (in most cases) an .orig.tar.gz file, which contains the source code of the software in a tarball released by the software authors. In some cases a .diff.gz file with changes that were applied to make the package build in the Ubuntu/Debian way and a .dsc file with metadata like checksums and the like it’s really not that important right now.

Once the build is done, you can check out the contents of /var/cache/pbuilder/result as it will contain the built hello .deb file :)

Now dealing with GPG. If you run

gpg —fingerprint your.name@email.com

It will print out something like:

pub   1024D/059DD5EB 2007-09-29

      Key fingerprint = 3E5C 0B3C 8987 79B9 A504  D7A5 463A E59D 059D D5EB

uid                  Daniel Holbach <dh@mailempfang.de>

…

In this case, the key ID is 059DD5EB , now please run

gpg —send-keys <KEY ID>

It will upload your _public_ portion of the key to keyservers which will sync the key among them. Once that’s done, you can tell Launchpad, which we use for all development about your shiny new gpg key - https://launchpad.net/people/+me/+editpgpkeys (you need a Launchpad account for this)

https://help.launchpad.net/YourAccount/ImportingYourPGPKey should help you if you run into any trouble

omid_: gpg: no keyserver known (use option —keyserver)

dholbach: try:     gpg —keyserver keyserver.ubuntu.com —send-keys <KEY ID>

TO BE CONTINUED IN PART II