Tasks

When you go about building and installing a software package there are a number of tasks that you generally follow with most software packages. You probably need to start out by downloading the source code, then unpacking the source code. Maye you need to apply some patches for some reason. Then you might run the configure script of the package, perhaps passing it some options to configure it to your liking. The you might run "make install" to install the software. If your actually going to make some packages, such as .deb or .rpm, then you'd have additional tasks you'd perform to make them.

You find that building things in OpenEmbedded works in a similar way - there are a number of tasks that are executed in a predefined order for each recipe. Any many of the tasks correspond to those listed above like "download the source". In fact you've probably already seen some of the names of these tasks - bitbake displays them as they are processed:

~%> bitbake lzo
NOTE: Psyco JIT Compiler (http://psyco.sf.net) not available. Install it to increase performance.
NOTE: Handling BitBake files: \ (4541/4541) [100 %]
NOTE: Parsing finished. 4325 cached, 0 parsed, 216 skipped, 0 masked.
NOTE: build 200705041709: started

OE Build Configuration:
BB_VERSION     = "1.8.2"
OE_REVISION    = ""
TARGET_ARCH    = "sh4"
TARGET_OS      = "linux"
MACHINE        = "titan"
DISTRO         = "erouter"
DISTRO_VERSION = "0.1-20070504"
TARGET_FPU     = ""

NOTE: Resolving missing task queue dependencies
NOTE: preferred version 2.5 of glibc not available (for item virtual/sh4-linux-libc-for-gcc)
NOTE: Preparing Runqueue
NOTE: Executing runqueue
NOTE: Running task 208 of 226 (ID: 11, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_fetch)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_fetch: started
NOTE: package lzo-1.08-r14: task do_fetch: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 209 of 226 (ID: 2, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_unpack)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_unpack: started
NOTE: Unpacking /home/lenehan/devel/oe/sources/lzo-1.08.tar.gz to /home/lenehan/devel/oe/build/titan-glibc-25/tmp/work/lzo-1.08-r14/
NOTE: package lzo-1.08-r14: task do_unpack: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 216 of 226 (ID: 3, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_patch)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_patch: started
NOTE: package lzo-1.08-r14: task do_patch: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 217 of 226 (ID: 4, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_configure)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_configure: started
NOTE: package lzo-1.08-r14: task do_configure: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 218 of 226 (ID: 12, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_qa_configure)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_qa_configure: started
NOTE: Checking sanity of the config.log file
NOTE: package lzo-1.08-r14: task do_qa_configure: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 219 of 226 (ID: 0, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_compile)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_compile: started
NOTE: package lzo-1.08-r14: task do_compile: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 220 of 226 (ID: 1, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_install)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_install: started
NOTE: package lzo-1.08-r14: task do_install: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 221 of 226 (ID: 5, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_package)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_package: started
NOTE: DO PACKAGE QA
NOTE: Checking Package: lzo-dbg
NOTE: Checking Package: lzo
NOTE: Checking Package: lzo-doc
NOTE: Checking Package: lzo-dev
NOTE: Checking Package: lzo-locale
NOTE: DONE with PACKAGE QA
NOTE: package lzo-1.08-r14: task do_package: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 222 of 226 (ID: 8, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_package_write)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_package_write: started
Packaged contents of lzo-dbg into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo-dbg_1.08-r14_sh4.ipk
Packaged contents of lzo into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo1_1.08-r14_sh4.ipk
NOTE: Not creating empty archive for lzo-doc-1.08-r14
Packaged contents of lzo-dev into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo-dev_1.08-r14_sh4.ipk
NOTE: Not creating empty archive for lzo-locale-1.08-r14
NOTE: package lzo-1.08-r14: task do_package_write: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 223 of 226 (ID: 6, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_populate_staging)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_populate_staging: started
NOTE: package lzo-1.08-r14: task do_populate_staging: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 224 of 226 (ID: 9, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_qa_staging)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_qa_staging: started
NOTE: QA checking staging
NOTE: package lzo-1.08-r14: task do_qa_staging: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 225 of 226 (ID: 7, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_distribute_sources)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_distribute_sources: started
NOTE: package lzo-1.08-r14: task do_distribute_sources: completed
NOTE: package lzo-1.08: completed
NOTE: Running task 226 of 226 (ID: 10, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_build)
NOTE: package lzo-1.08: started
NOTE: package lzo-1.08-r14: task do_build: started
NOTE: package lzo-1.08-r14: task do_build: completed
NOTE: package lzo-1.08: completed
NOTE: Tasks Summary: Attempted 226 tasks of which 213 didn't need to be rerun and 0 failed.
NOTE: build 200705041709: completed

Note

The output may look different depending on the version of bitbake being used, and some tasks are only run when specific options are enabled in your distribution. The important point to note is that the various tasks are being run and bitbake shows you each time it starts and completes a task.

So there's a set of tasks here which are being run to generate the final packages. And if you'll notice that every recipe runs through the same set of tasks (ok I'll admit that it is possible that some additional tasks could be run for some recipes, but we'll talk about that later). The tasks that you'll need to be most familiar with are:

fetch

The fetch task is responsible for fetching any source code that is required. This means things such as downloading files and checking out from source control repositories such as git or svn.

unpack

The unpack task is responsible for extracting files from archives, such as .tar.gz, into the working area and copying any additional files, such as init scripts, into the working area.

patch

The patch task is responsible for applying any patches to the unpacked source code

configure

The configure task takes care of the configuration of the package. Running a configure script ("./configure ") is probably the form of configuration that is most recognised but it's not the only configuration system that exists.

compile

The compile task actually compiles the software. This could be as simple as running make.

populate_staging (stage)

The populate_staging task (stage is an alternate, easier to type name, that can be used to refer to this task) is responsible for making available libraries and headers (if any) that may be required by other packages to build. For example if you compile zlib then it's headers and the library need to be made available for other applications to include and link against.

Note

This is different to the install task in that this is responsible for making available libraries and headers for use during build on the development host. Therefore it's libraries which normal have to stage things while applications normally don't need to. The install task on the other hand is making files available for packaging and ultimately installation on the target.

install

The install task is responsible for actually installing everything. Now this needs to install the software into the destination directory, D. This directory won't actually be a part of the final package though. In other words if you install something into ${D}/bin then it will end up in the /bin directory in the package and therefore on the target.

package

The package task takes the installed files and splits them into separate directories under the ${WORKDIR}/install directory, one per package. It moves the files for the destination directory, ${D}, that they were installed in into the appropriate packages subdirectory. Usually there will be a main package a separate documentation (-doc), development (-dev) and debugging packages (-dbg) for example.

package_write

The package_write task is responsible for taking each packages subdirectory and creating any actual installation package, such as .ipk, .deb or .rpm. Currently .ipk is the only fully supported packing format although .deb packages are being actively worked on. It should be reasonably easy for an experienced OpenEmbedded developer to add support for any other packaging formats they might required.

Note

You'll notice that the bitbake output had tasks prefixed with do_, as in do_install vs install. This is slightly confusing but any task x is implemented via a function called do_x in the class or recipe where it is defined. See places refer to the tasks via their name only and some with the do prefix.

You will almost certainly notice tasks beyond these ones - there are various methods available to insert additional tasks into the tasks sequence. As an example the insane.bbclass, which performs various QA checks, does these checks by inserting a new task called qa_configure between the configure and compile tasks and another new task called qa_staging between populate_staging and build tasks. The former validates the result of the configure task and the late the results of the populate_staging task.

To determine the full list of tasks available for a specific recipe you can run bitbake on the recipe and asking it for the full list of available tasks:

~%> bitbake -b packages/perl/perl_5.8.8.bb -c listtasks
NOTE: package perl-5.8.8: started
NOTE: package perl-5.8.8-r11: task do_listtasks: started
do_fetchall
do_listtasks
do_rebuild
do_compile
do_build
do_populate_staging
do_mrproper
do_fetch
do_configure
do_clean
do_package
do_unpack
do_install
do_package_write
do_distribute_sources
do_showdata
do_qa_configure
do_qa_staging
do_patch
NOTE: package perl-5.8.8-r11: task do_listtasks: completed
NOTE: package perl-5.8.8: completed
~%> 

If your being observant you'll note that listtasks is in fact a task itself, and that the -c option to bitbake allows you to explicitly run specific tasks. We'll make use of this in the next section when we discuss working with a recipe.