Packaging: Defining packages and their contents

A bitbake recipe is a set of instructions from creating one, or more, packages for installation on the target device. Typically these are .ipkg or .deb packages (although bitbake itself isn't associated with any particular packaging format).

By default several packages are produced automatically without any special action required on the part of the recipe author. The following example of the packaging output from the helloworld example above shows this packaging in action:

[NOTE: package helloworld-0.1-r0: task do_package_write: started
NOTE: Not creating empty archive for helloworld-dbg-0.1-r0
Packaged contents of helloworld into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/helloworld_0.1-r0_sh4.ipk
Packaged contents of helloworld-doc into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/helloworld-doc_0.1-r0_sh4.ipk
NOTE: Not creating empty archive for helloworld-dev-0.1-r0
NOTE: Not creating empty archive for helloworld-locale-0.1-r0
NOTE: package helloworld-0.1-r0: task do_package_write: completed

We can see from above that the packaging did the following:

There are several things happening here which are important to understand:

  1. There is a default set of packages that are considered for creation. This set of packages is controlled via the PACKAGES variable.

  2. For each package there is a default set of files and/or directories that are considered to belong to those packages. The documentation packages for example include anything found /usr/share/doc. The set of files and directories is controlled via the FILES_ variables.

  3. By default packages that contain no files are not created and no error is generated. The decision to create empty packages or not is controlled via the ALLOW_EMPTY variable.

Philosophy

Separate packaging, where possible, is of high importance in OpenEmbedded. Many of the target devices have limited storage space and RAM and giving distributions and users the option of not installing a part of the package they don't need allows them to reduce the amount of storage space required.

As an example almost no distributions will include documentation or development libraries since they are not required for the day to day operation of the device. In particular if your package provides multiple binaries, and it would be common to only use one or the other, then you should consider separating them into separate packages.

By default several groups of files are automatically separate out, including:

dev

Any files required for development. This includes header files, static libraries, the shared library symlinks required only for linking etc. These would only ever need to be installed by someone attempt to compile applications on the target device. While this does happen it is very uncommon and so these files are automatically moved into a separate package

doc

Any documentation related files, including man pages. These are files which are of informational purposes only. For many embedded devices there is no way for the user to see any of the documentation anyway, and documentation can consume a lot of space. By separating these out they don't take any space by default but distributions and/or users may choose to install them if they need some documentation on a specific package.

locale

Locale information provides translation information for packages. Many users do not require these translations, and many devices will only want to provide them for user visible components, such as UI related items, and not for system binaries. By separating these out it is left up to the distribution or users to decide if they are required or not.

Default packages and files

The defaults package settings are defined in conf/bitbake.conf and are suitable for a lot of recipes without any changes. The following list shows the default values for the packaging related variables:

PACKAGES

This variable lists the names of each of the packages that are to be generated.

PACKAGES = "${PN}-dbg ${PN} ${PN}-doc ${PN}-dev ${PN}-locale"

Note that the order of packages is important: the packages are processed in the listed order. So if two packages specify the same file then the first package listed in packages will get the file. This is important when packages use wildcards to specify their contents.

For example if the main package, ${PN}, contains /usr/bin/* (i.e. all files in /usr/bin), but you want /usr/bin/tprogram in a separate package, ${PN}-tpackage, you would need to either ensure that ${PN}-tpackage is listed prior to ${PN} in PACKAGES or that FILES_${PN} was modified to not contain the wildcard that matches /usr/bin/tprogram.

Note that the -dbg package contains the debugging information that has been extracted from binaries and libraries prior to them being stripped. This package should always be the first package in the package list to ensure that the debugging information is correctly extracted and moved to the package prior to any other packaging decisions being made.

FILES_${PN}

The base package, this includes everything needed to actually run the application on the target system.

FILES_${PN} = "\
    ${bindir}/* \
    ${sbindir}/* \
    ${libexecdir}/* \
    ${libdir}/lib*.so.* \
    ${sysconfdir} \
    ${sharedstatedir} \
    ${localstatedir} \
    /bin/* \
    /sbin/* \
    /lib/*.so* \
    ${datadir}/${PN} \
    ${libdir}/${PN}/* \
    ${datadir}/pixmaps \
    ${datadir}/applications \
    ${datadir}/idl \
    ${datadir}/omf \
    ${datadir}/sounds \
    ${libdir}/bonobo/servers"
FILES_${PN}-dbg

The debugging information extracted from non-stripped versions of libraries and executable's. OpenEmbedded automatically extracts the debugging information into files in .debug directories and then strips the original files.

FILES_${PN}-dbg = "\
    ${bindir}/.debug \
    ${sbindir}/.debug \
    ${libexecdir}/.debug \
    ${libdir}/.debug \
    /bin/.debug \
    /sbin/.debug \
    /lib/.debug \
    ${libdir}/${PN}/.debug"
FILES_${PN}-doc

Documentation related files. All documentation is separated into it's own package so that it does not need to be installed unless explicitly required.

FILES_${PN}-doc = "\
    ${docdir} \
    ${mandir} \
    ${infodir} \
    ${datadir}/gtk-doc \
    ${datadir}/gnome/help"
FILES_${PN}-dev

Development related files. Any headers, libraries and support files needed for development work on the target.

FILES_${PN}-dev = "\
    ${includedir} \
    ${libdir}/lib*.so \
    ${libdir}/*.la \
    ${libdir}/*.a \
    ${libdir}/*.o \
    ${libdir}/pkgconfig \
    /lib/*.a \
    /lib/*.o \
    ${datadir}/aclocal"
FILES_${PN}-locale

Locale related files.

FILES_${PN}-locale = "${datadir}/locale"

Wildcards

Wildcards used in the FILES variables are processed via the python function fnmatch. The following items are of note about this function:

  • /

    /*: This will match all files and directories in the dir - it will not match other directories.

  • /

    /a*: This will only match files, and not directories.

  • /dir: will include the directory dir in the package, which in turn will include all files in the directory and all subdirectories.

Note that the order of packages effects the files that will be matched via wildcards. Consider the case where we have three binaries in the /usr/bin directory and we want the test program in a separate package:

/usr/bin/programa /usr/bin/programb /usr/bin/test

So we define a new package and instruct bitbake to include /usr/bin/test in it.

FILES-${PN}-test = "${bindir}/test"
PACKAGES += "FILES-${PN}-test"

When the package is regenerated no ${PN}-test package will be created. The reason for this is that the PACKAGES line now looks like this:

{PN}-dbg ${PN} ${PN}-doc ${PN}-dev ${PN}-locale ${PN}-test

Note how ${PN} is listed prior to ${PN}-test, and if we look at the definition of FILES-${PN} it contains the ${bindir}/* wildcard. Since ${PN} is first it'll match that wildcard are be moved into the ${PN} package prior to processing of the ${PN}-test package.

To achieve what we are trying to accomplish we have two options:

  1. Modify the definition of ${PN} so that the wildcard does not match the test program.

    We could do this for example:

    FILES-${PN} = "${bindir}/p*"

    So now this will only match things in the bindir that start with p, and therefore not match our test program. Note that FILES-${PN} contains a lot more entries and we'd need to add any of the other that refer to files that are to be included in the package. In this case we have no other files, so it's safe to do this simple declaration.

  2. Modify the order of packages so that the ${PN}-test package is listed first.

    The most obvious way to do this would be to prepend our new package name to the packages list instead of appending it:

    PACKAGES =+ "FILES-${PN}-test"

    In some cases this would work fine, however there is a problem with this for packages that include binaries. The package will now be listed before the -dbg package and often this will result in the .debug directories being included in the package. In this case we are explicitly listing only a single file (and not using wildcards) and therefore it would be ok.

    In general it's more common to have to redefine the entire package list to include your new package plus any of the default packages that you require:

    PACKAGES = "${PN}-dbg ${PN}-test ${PN} ${PN}-doc ${PN}-dev ${PN}-locale"

Checking the packages

During recipe development it's useful to be able to check on exactly what files went into each package, which files were not packaged and which packages contain no files.

One of easiest method is to run find on the install directory. In the install directory there is one subdirectory created per package, and the files are moved into the install directory as they are matched to a specific package. The following shows the packages and files for the helloworld example:

~/oe%> find tmp/work/helloworld-0.1-r0/install
tmp/work/helloworld-0.1-r0/install
tmp/work/helloworld-0.1-r0/install/helloworld-locale
tmp/work/helloworld-0.1-r0/install/helloworld-dbg
tmp/work/helloworld-0.1-r0/install/helloworld-dev
tmp/work/helloworld-0.1-r0/install/helloworld-doc
tmp/work/helloworld-0.1-r0/install/helloworld-doc/usr
tmp/work/helloworld-0.1-r0/install/helloworld-doc/usr/share
tmp/work/helloworld-0.1-r0/install/helloworld-doc/usr/share/doc
tmp/work/helloworld-0.1-r0/install/helloworld-doc/usr/share/doc/helloworld
tmp/work/helloworld-0.1-r0/install/helloworld-doc/usr/share/doc/helloworld/README.txt
tmp/work/helloworld-0.1-r0/install/helloworld
tmp/work/helloworld-0.1-r0/install/helloworld/usr
tmp/work/helloworld-0.1-r0/install/helloworld/usr/bin
tmp/work/helloworld-0.1-r0/install/helloworld/usr/bin/helloworld
~/oe%>

The above shows that the -local, -dbg and -dev packages are all empty, and the -doc and base package contain a single file each. Uses "-type f" option to find to show just files will make this clearer as well.

In addition to the install directory the image directory (which corresponds to the destination directory, D) will contain any files that were not packaged:

~/oe%> find tmp/work/helloworld-0.1-r0/image
tmp/work/helloworld-0.1-r0/image
tmp/work/helloworld-0.1-r0/image/usr
tmp/work/helloworld-0.1-r0/image/usr/bin
tmp/work/helloworld-0.1-r0/image/usr/share
tmp/work/helloworld-0.1-r0/image/usr/share/doc
tmp/work/helloworld-0.1-r0/image/usr/share/doc/helloworld
~/oe%>

In this case all files were packaged and so there are no left over files. Using find with "-type f" makes this much clearer:

~/oe%> find tmp/work/helloworld-0.1-r0/image -type f
~/oe%>

Messages reading missing files are also display by bitbake during the package task:

NOTE: package helloworld-0.1-r0: task do_package: started
NOTE: the following files were installed but not shipped in any package:
NOTE:   /usualdir/README.txt
NOTE: package helloworld-0.1-r0: task do_package: completed

Except in very unusual circumstances there should be no unpackaged files left behind by a recipe.

Excluding files

There's no actual support for explicitly excluding files from packaging. You could just leave them out of any package, but then you'll get warnings (or errors if requesting full package checking) during packaging which is not desirable. It also doesn't let other people know that you've deliberately avoided packaging the file or files.

In order to exclude a file totally you should avoid installing it in the first place during the install task.

In some cases it may be easier to let the package install the file and then explicitly remove the file and the end of the install task. The following example from the samba recipe shows the removal of several files that get installed via the default install task generated by the autotools class. By using do_install_append these commands and run after the autotools generated install task:

do_install_append() {
    ...
    rm -f ${D}${bindir}/*.old
    rm -f ${D}${sbindir}/*.old
    ...
}

Debian naming

A special debian library name policy can be applied for packages that contain a single shared library. When enabled packages will be renamed to match the debian policy for such packages.

Debian naming is enabled by including the debian class via either local.conf or your distributions configuration file:

INHERIT += "debian"

The policy works by looking at the shared library name and version and will automatically rename the package to . For example if the package name (PN) is foo and the package ships a file named libfoo.so.1.2.3 then the package will be renamed to libfoo1 to follow the debian policy.

If we look at the lzo_1.08.bb recipe, currently at release 14, it generates a package containing a single shared library :

~oe/build/titan-glibc-25%> find tmp/work/lzo-1.08-r14/install/
tmp/work/lzo-1.08-r14/install/lzo
tmp/work/lzo-1.08-r14/install/lzo/usr
tmp/work/lzo-1.08-r14/install/lzo/usr/lib
tmp/work/lzo-1.08-r14/install/lzo/usr/lib/liblzo.so.1
tmp/work/lzo-1.08-r14/install/lzo/usr/lib/liblzo.so.1.0.0

Without debian naming this package would have been called lzo_1.08-r14_sh4.ipk (and the corresponding dev and dbg packages would have been called lzo-dbg_1.08-r14_sh4.ipk and lzo-dev_1.08-r14_sh4.ipk). However with debian naming enabled the package is renamed based on the name of the shared library, which is liblzo.so.1.0.0 in this case. So the name lzo is replaced with liblzo1:

~oe/build/titan-glibc-25%> find tmp/deploy/ipk/ -name '*lzo*'  
tmp/deploy/ipk/sh4/liblzo1_1.08-r14_sh4.ipk
tmp/deploy/ipk/sh4/liblzo-dev_1.08-r14_sh4.ipk
tmp/deploy/ipk/sh4/liblzo-dbg_1.08-r14_sh4.ipk

Some variables are available which effect the operation of the debian renaming class:

LEAD_SONAME

If the package actually contains multiple shared libraries then one will be selected automatically and a warning will be generated. This variable is a regular expression which is used to select which shared library from those available is to be used for debian renaming.

DEBIAN_NOAUTONAME_

If this variable is set to 1 for a package then debian renaming will not be applied for the package.

AUTO_LIBNAME_PKGS

If set this variable specifies the prefix of packages which will be subject to debian renaming. This can be used to prevent all of the packages being renamed via the renaming policy.

Empty packages

By default empty packages are ignored. Occasionally you may wish to actually created empty packages, typically done when you want a virtual package which will install other packages via dependencies without actually installing anything itself. The ALLOW_EMPTY variable is used to control the creation of empty packages:

ALLOW_EMPTY

Controls if empty packages will be created or not. By default this is "0" and empty packages are not created. Setting this to "1" will permit the creation of empty packages (packages containing no files).