Intro
Headers
Instructions
Macros
Commands
This Application Note is about resolving in OSGi. The OSGi Framework has always used a resolver to wire a given set of bundles together, ensuring that only valid wires are made. However, the same OSGi resolver can also be used to select a set of bundles from a much larger set. This application note discusses this secondary usage.
The resolver model is based on technology developed in OSGi since 2006 with RFC-0112 Bundle Repository. This RFC laid out a model that was gradually implemented in the OSGi specifications and for which many tools were developed. Resolving automates a task that is mostly done manually today.
The pain that the resolver solves is well known to anybody that builds application from modules, the so called assembly process. All developers know the dreadful feeling of having to drag in more and more dependencies to get rid of Class Not Found errors when the application starts (or after running for an hour). For many, Maven was a significant improvement because it made this process so much easier because dependencies in Maven are transitive. If you depend on some artefact, then this artefact drags in its own dependencies to the runtime.
Unfortunately, the Maven dependency model has limitations for a number of reasons. Though it undeniably has become easier to get a result, the result is often littered with unnecessary or wrong artefacts. Often application developers have no real understanding what is in their runtime.
These are the causes:
The result is that all too often the class path of the final application is littered with never used byte codes and/or overlapping packages that came from artefacts with different versions. If the customer could see the mess the average class path is they would run away screaming.
Establishing the perfect class path for an application is a process for which the human mind is extremely badly suited and the transitive model of Maven has too many sharp edges. The resolver provides an alternative model that focuses on looking at the whole solution space instead of having to live with the (sometimes arbitrary) decisions of developers or of deeply nested transitive dependencies.
The design of the resolving model is quite simple. It consists of the following entities:
String, Integer, Double, and Long.Resolving in this App note is therefore the process of constructing an application out of resources. Resolving takes a list of initial requirements, a description of the system capabilities, and one or more repositories. It will use the list of initial requirements to find resources in the repository that provide the required capabilities. Clearly, these resources have their own requirements, retrieving applicable resources is therefore a recursive process. A resolver will find either a solution consisting of a set of resources where all requirements are satisfied or that there is no solution.
It is important to realise that a resource and its capabilities and requirements are descriptions. They provide a formal representation of an external artefact. Since these formal representations can be read by a computer, we can calculate a closure of resources that, when installed together, have only resources where all their mandatory requirements are satisfied by the resources in the closure.
Although the OSGi specifications started out with a set of headers that each had their own semantics, over time the specification migrated fully to the more simple and formal model of Resources, Capabilities, and Requirements. Since the function of the legacy headers were still needed, it was necessary to map these legacy headers to the formal model. This resulted in a number of OSGi core namespaces.
osgi.wiring.identity – Bundle-SymbolicName header.osgi.wiring.bundle – Bundle-SymbolicName and Require-Bundle header.osgi.wiring.package – Import-Package and Export-Package headers.osgi.wiring.host – Bundle-SymbolicName and Fragment-Host header.osgi.ee – Bundle-RequiredExecutionEnvironment header.osgi.native – Bundle-NativeCode header.osgi.content – Provides the URL and checksum to download the corresponding artefact. (This namespace is defined in the compendium Repository specification.)Each namespace defines the names of the properties and their semantics. For the OSGi namespaces, there are classes like org.osgi.framework.namespace.IdentityNamespace that contain the details of a namespace.
To make the model more clear let’s take a closer look to a simple bundle com.example.bundle that exports package com.example.pe and imports a package com.example.pi. When the bundle is installed it will require that some other bundle, or the framework, provides package com.example.pi. We can describe this bundle then as follows:
Resource for bundle com.example.bundle
capabilities:
osgi.wiring.identity; osgi.wiring.identity=com.example.bundle
osgi.wiring.bundle; osgi.wiring.bundle=com.example.bundle
osgi.wiring.host; osgi.wiring.host=com.example.bundle
osgi.wiring.package; osgi.wiring.package=com.example.pe
requirements:
osgi.ee; filter:="(&(osgi.ee=JavaSE)(version=1.8))"
osgi.wiring.package; filter:='(osgi.wiring.package=com.example.pi)'
Clearly, an Export-Package: com.example.pe is a bit easier to read than the corresponding capability, let alone the filter in the requirement. This is especially true when the versions are taken into account, with the assertion of the version range the filters become quite unreadable. Clearly, if this had to be maintained by human developers then a model like Maven would be far superior. Fortunately, almost all of the metadata that is needed to make this work does not require much extra work from the developer. For example, the bnd tool that is available in Maven, Gradle, Eclipse, Intellij and others can easily analyse a JAR file and automatically generate the requirements and capabilities based on the source code.
In certain cases it is necessary to provide requirements and capabilities that bnd cannot infer. However, Manifest annotations support in bnd can be used to define an annotation that will add parameterised requirements to the resource.
Really, when you use an adequately appointed toolchain (like Bndtools) none of this gibberish is visible to normal developers unless they have a special case. The gibberish is left to the tools that prefer gibberish over natural language.
For a bundle to be a good citizen in a resolution it suffices to follow the standard rules of good software engineering. However, since there is so much software out there that does not follow these rules, a short summary.
java.* which is an implicit import). Therefore, a developer must alway be aware of this trade off.To see the requirements and capabilities of a bundle Bndtools has a special Resolution view. This shows the requirements and capabilities of a selected JAR file or a bnd.bnd file.
This view uses a number of icons to represent the requirements/capabilities. You can hover over them to see further details.
– osgi.service.
– osgi.identity
– osgi.wiring.package
– osgi.ee
– osgi.extender
– osgi.bundleDevelopers that are keen on develping good bundles should pay close attention to this pane.
The previous diagram adds the resolver to the earlier diagram of the basic model. It adds the following entities:
An important variable in the resolving process is effective which defines the effectiveness under which the resolve operation is performed. The resolver will only look at requirements that it deems effective. The default effectiveness is resolve. The effectiveness active is a convention commonly used for situations that do not need to be resolved by the OSGi framework but are relevant in using the resolver for assembling applications.
A repository is a collection of resources. This could be Maven Central or it could be a single bundle. That said, neither is a good idea in practice. The repository is the scope of the resolution. Only resources in the repository can be selected. Although the naive idea is to make the scope as large as possible to let the computer do the work, it is much better to curate the repository.
The are many reasons why you need a curated repository but basically it comes down is the GIGO principle: garbage is garbage out. Another reason is running time. The resolver gets overwhelmed quickly when there are too many alternatives for a requirement. Resolving is an NP complete problem and this means that the resolution time quickly becomes very long when a lot of alternatives need to be examined.
The bnd toolchain provides OSGi Repository access for many popular repository formats.
pom.xml. It integrates fully with the local ~/.m2 repository.Since bnd has an extensive library to parse bundles and generate the resources it is relatively easy to parse bundles in other ways. For example, there is a Maven plugin that can generate an OSGi index.
A repository generally represents a release of a software product. For example, in OSGi enRoute, the OSGi Alliance has provided a base API bundle. This bundle contains all the APIs of OSGi core, OSGi compendium, and an OSGi enRoute release. There is also an OSGi enRoute distro for each release. The distro is a repository that provides access to implementations for the base API. The OSGi enRoute distro is maintained in a Maven POM. This POM lists the exact versions of a release. Many developers publish an OSGi Repository XML to allow the resolver access to the metadata of their bundles, for example Knopflerfish produces an OSGi XML Repository with all their bundles.
This model is in contrast with Maven Central and most other Maven repositories. These repositories are designed to contain everything that was ever released. An OSGi repository is more the content of a specific release. Since it generally only contains a specific release, it will disallow the resolver to use any unwanted resources.
A natural repository is the bnd workspace since a workspace is a collection of related projects. Since these projects are build together it is trivial to keep them compatible. The metadata burden can be mitigated by sharing metadata between these related projects. For example, in bnd all bundles share the same version. Although this might sometimes release unchanged bundles under a newer version, the only cost is a bit of disk space. A small price to pay for an otherwise very error prone manual process. The Gradle build can release to a Nexus repository and automatically generate an index that can be used as repository.
By far the best way to get experience with the resolver is using the bnd’s bndrun files. Bndtools provides a friendly user interface that makes it easy to use the OSGi Resolver in an interactive way. You can play along with the following explanations by creating an OSGi enRoute Application project from the templates.
A bndrun describes the runtime configuration for an application. This can either be a standalone executable JAR or an application that should be hosted in a container like a Java EE server or Karaf. In this app note, the target is an executable JAR unless otherwise noted.
If you double click on the osgi.enroute.examples.resolver.application.bndrun file (or whatever name you picked) it opens a Run pane.
>
Even though most lists are hidden, the pane is already quite overwhelming. So let’s go through the GUI and explain it one by one.
Remember that a resolution is for a specific system, in our case an executable JAR. In the Core Runtime pane we specify the OSGi framework that will be used as well as the execution environment. The execution environment is from a list of Java VM versions and OSGi specifications.
The execution environment is used to calculate the system capabilities. That is, the system is treated as a single resource that provides the capabilities of the Framework as well as the capabilities of the OSGi defined execution environments. The system resource is always included in the resolution but can of course never be downloaded, it is the target environment.
The Browse Repos is a list of the resources that are found in the active repositories. A search field makes it easy to find specific resources. For example, if you type in gogo it will list all Gogo bundles.
One or more resources from the Browse Repos list can be selected and then dragged to the Run Requirements list to the left. This adds an identity requirement to the set of initial requirements. You can also drag a resource to the Run Blacklist and Run Bundles lists.
This is the main list to watch. It contains the set of initial requirements given to the resolver. The GUI makes it possible to add identity requirements and remove listed requirements. The easiest way is to use drag and drop form the Browse Repos list but it is also possible to use the green +, which opens a dialog from which bundles can be added directly from the selected repositories.
There are some more panes that are useful but they will be handled in diagnosing problems. For reference, a short introduction to these panes.
bndrun file to standalone. A standalone bndrun file has no relation to the workspace it resides in and establishes its own repositories.Run, Debug, and Export functions always work from the -runbundles list.Taking the initial requirement it is possible to resolve by clicking on the Resolve button. This will show a rather large dialog window with the resolution.
This dialog window is divided in three main parts:
If some of the listed optional resources are desired then they can be selected. Pressing the Update and Resolve button will restart the resolver but now with the selected optional resources as mandatory.
If the Finish button is pressed then the current required resources list is converted to the list of Run Bundles. You can inspect them at the right bottom of the bndrun editor window.
After a successful resolve you can either Run, Debug, or Export the bndrun file.
So far the ideal process of happy resolves and satisfied bundles has been described. It is now necessary to leave this rosy world and descend into the world of failed resolves. Unfortunately, the provided diagnostic information when a resolve fails is quite low.
When a resolve fails it returns a cause but more often than not this is not the real cause. This is not some shortcoming from the current resolver but a fundamental logical problem. The simplest form of a resolution is if you have for example 3 numbers 1,4,8. You need to find the numbers that sum to 10 using only addition and subtraction. If you try out all the combination then you find that no combination works. A failure report could be that -3 is not available because the last tried permutation was 1+4+8. Clearly, before that permutation many other numbers were missing as well, the missing -3 just happened to be the last one …
That said, there are a number of scenarios where the resolver does give a hint where the problem is.
This rather obscure message indicates that the resolver tries to include a api bundle that was made unresolvable.
In OSGi enRoute it is recommended to include the API package of a service in the provider bundle. Since a provider has a very tight relation to the API there is no loss of flexibility by including the package. For this reason, OSGi enRoute templates make the API bundle compile only. Each API has a special Require-Capability header:
Require-Capability: \
compile-only
This header creates a requirement that cannot be satisfied. There is nothing special with compile-only, it is just an unused namespace. It could also have been foo-bar.
In the resolver, you will see the following error chain:
Unable to resolve <<INITIAL>> version=null:
missing requirement osgi.enroute.examples.resolver.missingapi.provider
-> Unable to resolve osgi.enroute.examples.resolver.missingapi.provider version=1.0.0.201710041250:
missing requirement osgi.enroute.examples.resolver.missingapi.api; version=[1.0.0,1.1.0)
-> Unable to resolve osgi.enroute.examples.resolver.missingapi.api version=1.0.0.201710041249:
missing requirement false]]
Note: Unfortunately, the output is blurred by a misguided attempt to make the output more concise. Because of this, the distinction between a bundle and a package is not very clear. Sadly you can only see the difference between a requirement for a bundle and a package by looking at the version. If this is a range then it is a package and if it is a version with a timestamp it is a bundle. (This works most of the time.) This is a bug in bnd and should be corrected.
As indicated, you really need to understand that the diagnostic is just the last path the resolver took. The osgi.enroute.examples.resolver.missingapi.provider bundle tries to find a provider for the package osgi.enroute.examples.resolver.missingapi.api and has found the osgi.enroute.examples.resolver.missingapi.api bundle. However, this API bundle has the impossible to satisfy compile-only requirement.
That said, it is better to look at the Missing Requirements list since this reports quite nicely what is missing.
The icon and the text more clearly indicate that it cannot resolve the osgi.enroute.examples.resolver.missingapi.api bundle due to the compile-only requirement.
Exporting the API package from the provider bundle will correct this case.
Many developers compile against the compendium bundle to get the OSGi service API packages. Although compiling against an API bundle has advantages, using the compendium bundle in runtime is evil. Since the compendium bundle aggregates a large number of API packages it will have the tendency to unnecessarily constrain the versions of different APIs. That is, it blocks you from using newer APIs.
In OSGi enRoute the base API bundle is similar. For this reason it is also made compile-only. However, the compendium bundle is a perfectly valid bundle.
To make bundles that should not be used at runtime not resolvable there is the Run Blacklist list on the bndrun editor. This list contains bundles that should never be included in a resolution.
For example, we create a bundle that implements the Wire Admin service.
public class MyService implements WireAdmin {
...
}
This will cause a requirement for an exported package org.osgi.service.wireadmin. Since the OSGi enRoute does not provide an implementation in its distro this API is not available from the OSGi base API. We therefore add the OSGi compendium bundle to the -buildpath. This then compiles fine.
However, the resolve will then drag in the OSGi compendium bundle.
If we look at the reasons when we select the compendium bundle we see that also the Configuration Admin imports from the compendium even though it actually might provide a higher version. (This maybe understandable but a really bad practice.)
To get rid of the compendium bundle we can drag and drop it to the Run Blacklist window. Any requirement in the Run Blacklist list will automatically exclude all bundles that are selected by that requirement.
Sometimes the resolver can complain about a missing requirement but you are sure that it is in the repository. The first thing is to try to isolate the problem. Almost any problem can be solved if you remove the redundant parts. Quite often developers are trying to debug this situation in a complex large bndrun file and then get overwhelmed.
Just create a new bndrun file and only add the bundle you think should provide the resource to the Run Requires list, keep only one requirement, and then resolve. If this resolves fine then at least you know that that bundle can potentially resolve.
However, often you find that even on its own it does not resolve. In most case the error message and the Reasons list provide sufficient information to understand why it does not resolve.
It is still a mystery, try checking the Run Blacklist list. If it is not there, it might be time to raise a bug.
So far this App Note only visited the graphic user interface (GUI). However, bnd always keeps all information in simple properties files that can also edited as text. In the Run editor (that edits bndrun files) you can also select the Source view. Not all features of a bndrun file can be manipulated through the GUI. This section therefore shows what is in the source and it can be manipulated.
Notice that most instructions are merge properties. That is, bnd will first find all properties that start with the instruction name and merge their values together. For example, if you set -runrequires, -runrequires.foo, and -runrequires.bar bnd will use the combination of these properties. The order is the sorting order of the names used.
-distro option, it contains additional capabilities that are not in the distro files.The problem with metadata is that it can also be wrong. Especially legacy bundles lack the proper metadata to inform the resolver that they provide a service or require an implementation of a specification. This is not a problem in runtime since these requirements are generally not used by the Framework, they are designed for using the resolver in selecting a closure of bundles. However, in certain cases adding a capability or a requirement to a bundle in the repository would simplify things. Clearly it is possible to wrap the legacy bundle but this is extremely cumbersome and can create confusion down the line.
Therefore, another method is to use the augments that the bnd repositories support. Augments can add capabilities and requirements to existing bundles in a repository. However, it can only do this for the interactive resolve process. When the OSGi Framework resolves a number of bundles it will never takes augments into account.
There are two different ways to add the augments.
bndrun file. This is an ad-hoc mechanism that is normally a last resort. For non-standalone bndrun files you can also add augments in the cnf/ext directory since any bnd file in there will add its properties to the bndrun file. It is of course also possible to include file in any bnd/bndrun file.bnd.augment capability. Such a resource has a file with properties that describe the augment for that repository. This is a good way when you have to curate a repository and need to fixup some legacy bundles. This resource can also add to the blacklist. See the OSGi enRout distro for an example.In both cases the augments are described using the standard OSGi/bnd syntax. The syntax is not a beauty since it stretches what you can do with the OSGi header format. However, augmenting should really be a last resort so maybe it is not really bad that the syntax is cumbersome.
You can add an augment with the -augment instruction in the bndrun file.
-augment PARAMETER ( ',' PARAMETER ) *
Augmenting is adding additional capabilities and requirements. When bnd resolves a project or bndrun file, it will read these instructions. Since -augment is a merge property you can also use additional keys like -augment.xyz.
The key of the PARAMETER is used for matching the Bundle Symbolic Name. It can contain the * wildcard character to match multiple bundles. The bundle symbolic name must be allowed as a value in a filter it is therefore not a globbing expression.
The following directives and attribute are architected:
[<version>,∞). The version range can be prefixed with an @ for a consumer range (to the next major) or a provider range (to the next minor) when the @ is a suffix of the version. The range can restrict the augmentation to a limited set of bundles.capability: directive specifies a Provide-Capability instruction, this will therefore likely have to be quoted to not confuse bnd with embedded comma’s. Any number of clauses can be specified in a capability directive by separating the clauses with a comma. (This is where the syntax gets stretched.)requirement: directive specifies a Require-Capability instruction similar to the capability: directive.To augment the repositories during a resolve, bnd will find all bundles that match the bundle symbolic name and fall within the defined range. If no range is given, only the bundle symbolic name will constrain the search. Each found bundle will then be decorated with the capabilities and requirements defined in the capability: and requirement: directive.
For example, we need to provide an extender capability to a bundle with the bundle symbolic name com.example.prime with version [1.2,1.3). In that case add the following instruction to the bndrun file.
-augment.prime = \
com.example.prime; \
capability:='osgi.extender; \
osgi.extender=some.extender; \
version:Version=1.2@'
The capability: and requirement: directives follow all the rules of the Provide-Capability and Require-Capability headers respectively. For the resolver, it is as if these headers were specified in their manifests. Since these headers can contain semicolons and commas they must be quoted. bnd will allow double quotes inside normal quotes and vice versa when it is necessary to nest quotes.
A workspace setup with bnd will generally provide a good start. However, when you need to grandfather in a lot of bundles from Maven Central then it is likely that you will need to spend some time to augment these bundles. This can be a depressing task since you’ll find out how messy the world is. However, experience shows that once the repository is resolvable, maintaining it has very little overhead. Better, it tends to signal probems very early in the development process.
A good example of a curated repository is the OSGi enRoute Distro. If you’re responsible for a repository it might help to take a good look at the augments.bndrun file.
There are a number of (rudimentary) functions in the command line version of bnd that might be useful. Unfortunately, the commands currently assume an OSGi repository.
The enroute model tends to guide you to resolve an executable. Meanwhile with openliberty, WebSphere Liberty, Karaf, Liferay, etc. you are deploying into an existing container which already has a lot of capabilities. The crux of the issue becomes resolving only what you need to deploy. What you need at that point is a way to find out what the container already provides in a format which can be used during resolve time.
Currently the way to do that is to create a distro of the target container. This distro is a JAR file which provides all the capabilities that the target container provides at one point in time. It includes the capabilities of all currently installed bundles. It also contains all capabilities provided by the system bundle which may have been configured by framework properties. It is an aggregate view of all the capabilities available in the framework contained in a single JAR.
bnd remote distro -o container-5.6.7.jar container 5.6.7container-5.6.7.jar created in 2. and place it into the the directory containing the bndrun file that is used to resolve your deployment jars.in the bndrun file add:
-distro: ${.}/container-5.6.7.jar;version=file
What you do with those resolved bundles is dependent on the goal of the developer. They can be directly installed in the existing container, or they could be assembled into a package format native to the container, or possibly a subsystem.
What you need to bear in mind is that the distro needs to be re-created each time the target container changes in any significant way, otherwise it won’t reflect the true capabilities of the system needed to resolve against.
Creating applications from reusable models is a goal that the software industry has been trying to achieve for a long time. To a certain extent, the Maven dependency model provides this model. However, it also is a model where dependencies are not strictly managed and much is left to chance.
The resolver model provides an alternative (working inside maven if so desired) to establish a class path that optimises the whole application class path instead of just slavishly following transitively dependencies. Experience shows that organisations that use the resolver have a much better grip of what is actually running in their runtime. Not only minimises this runtime errors, it generally also makes it easier to migrate and evolve the code base.
Converting an existing build into a resolve based build can be daunting but the efforts are worth it. For bnd users that use the workspace model the advantages will flow freely.