Intro
Headers
Instructions
Macros
Commands
OSGi is arguably one of the best specifications in the Java world. It is a no compromise specification for a component framework that is based on what we today know are the best practices. It extends the type safety first model of Java to hold true over time when the modules morph into sometimes unrecognizable shapes. It provides a solid foundation to build applications that can run anywhere, the original promise of Java; offering these impressive features while remaining as simple as possible, albeit no simpler.
For some, the previous paragraph may come as a surprise because OSGi has had its share of people complaining about it. Surprisingly, the biggest complaint is often the Class Not Found Exception, which is always a perfect sign that people try to push a round peg in a too small square hole, and with all their might. You only see those exceptions when you’re not doing engineering but when you are hacking. If you run head on into the walls that OSGi installs and it is giving you a headache, then just look around and find the elegant and easy to use doors: services.
Though this is all true, I do not claim that OSGi is trivial to use; triviality has a way to clash with large system that must evolve over many years. The software profession has a brutal industry that lures us with the siren song of a ‘few hours work’ to devour us while trying to main gigantic hairballs. As Fred Brooks already said so many years ago in his seminal book ‘The Mythical Man Month’, there is no silver bullet. Even OSGi will require hard work to build evolvable systems. And though we cannot make building complex systems easy, bnd can at least make it easier (and considerably more fun).
When we started building the OSGi specifications around 1998 it became clear we needed metadata to describe the bundles. We chose the manifest since this was already a well defined resource in the JAR, why invent the wheel? Actually, it was Anselm Baird, a Sun employee at the time, who had come up with this idea in his Java Embedded Server (JES). Basically, the manifest was a property file with benefits. We still consider this choice for embedded metadata an excellent idea. Looking at Maven where the metadata is separated over a surprising number of files. It is clear that the reduction in the number of files and potential errors caused by out-of-sync metadata is quite large. My favorite example of this additional complexity is when the SHA1 files necessary in Maven were also signed, an easy mistake but induced by unnecessary complexity. An OSGi bundle is a completely self described artifact.
So who got bnd started? Well, when we wrote our bundles in the dawn of the new millenium we quickly discovered that transient dependencies, well, eh, sucked. Several of us had great experience in Object Oriented design, and though objects clearly had become mainstream they frustratingly had not provided the level of reusability that some had dreamed of. As Brian Foote shows, we thought we were building shiny castles but in the end built a big ball of mud. Where our predecessors of the structured programming era always hammered on low coupling and high cohesion we had ignored their wise lessons and got ourselves tangled up in a nice mess. Trying to reuse a class too often dragged in more unwanted stuff than the cat does in the whole year.
The advantage of Java was that it had found an elegant solution to the scourge of transient dependencies: Java interfaces. An interface elegantly broke transitive dependencies while maintaining type safety at the cost of an indirection. A cost anyone should gladly bear for hairballs are quite distasteful. Surprisingly, while Java offered us this elegant solution to break transitive dependencies, most people in our industry were blindly committing themselves to the same route, just on another level, which actually would make the consequences even worse.
What we discovered was that we could use the idea of interfaces in an object oriented environment also for modules. An interface is an instance of contract based programming and we found out that we could use the same between modules. The idea behind the OSGi service is that you do not let modules directly interact but that you use reified conduits between the modules so that you keep the modules, which are implementations, unconnected. What better concept in Java to express this contract between modules is there but the concept of a package?
Originally devised as the Java modules, Java packages are heavily underappreciated in the Java world. Even highly expert users sometimes only appreciate the namescoping of the package, brushing over any accessibility issues arising out of the use of multiple classloaders. They are wrong, a package is a perfect module. That said, it is a perfect encapsulation to describe the contract that governs the collaboration between modules. An interface has a too small granularity for such a collaboration contract since it often requires additional interfaces like listeners and also helper classes. Packages, although they are too fine grained for deployment modules, are eminently suitable to define a contract to given the collaboration between deployment modules.
So we had come up with the surprisingly well fitting dependency model: ‘Look Ma, No Coupling!’ But then using it became quite painful because it required a lot of tedious house keeping, tracking which packages you were using, of what version. Just bluntly expressing your dependency on an implementation sometimes felt as attractive as the next shot of heroin for a junkie. Now, housekeeping and chores in general are quite low on the list of reasons to give meaning to live for most software developers, and we were no exception. Not only are we in general horrible at chores, we tend to make an amazing amount of mistakes doing them. Which, in general is not good software engineering. Fortunately, we largely live in a virtual world where it is easy to let the computer do the chores. And this is the raison d’etre of bnd: take the chores out of OSGi and focus on the fun parts. And maybe even more important, ‘Do Not Repeat Yourself’.
The early version of bnd was called btool and was incepted at Ericsson, where it was used to automate the generation of the manifest in the ebox project. Then, in 2001, btool was used to automate the build process of the specifications, the reference implementations, and the test suites at the OSGi Alliance. One day there was a bit of confusion about the pedigree of the btool, and it was showing its age; it was time for a rewrite and bnd was born.
The name came from bundle, then removing the vowels, and cutting it to three characters to reduce the typing on the command line. The pronunciation: we actually have no clue, whatever your preference is. B AND D, bind, whatever. You want to write it as bnd, Bnd, or BND, be our guest. We prefer to write it like bnd since we love the symmetry.
Over the years bnd gained a lot of power because we needed a tool at the OSGi Alliance that could not just create good bundles, it also was heavily used in the test cases that required bad bundles. Of course this was an excellent test range for bnd and this synergy is still true today.
In 2003 Eclipse decided to adopt the OSGi specifications and this generated some interesting discussions. To the OSGi crowd it was crystal clear that packages were the future, the Eclipse crowd, coming from a traditional transitive direct hard module dependency world, this future was more fuzzy. We lost, and Require-Bundle and Fragment-Host are the souvenirs of that battle. However, this was not the only disagreement, unfortunately we also quarreled about how to develop bundles (sorry, plugins) and there we lost again with PDE and P2 the landmarks remembering us of this lost fight. Our experience with bnd had taught us that one should not try to maintain the manifest by hand, just like one should not try to write class files with a hex editor. The manifest is a readable format but it was never intended to be human writable except for emergencies. To simplify the runtime (important in an embedded world) we never made the manifest easy to write. For example, when you specify the packages you want to export it is very useful to use wildcards to include a number of sub-packages. However, in runtime this would force the framework to traverse all classes to make the list of total packages of which it could then select the packages based on the wildcards. There is also an inevitable duplication of information as well as derived information which makes it impossible to not create evil redundancy.
Unfortunately, the PDE guys insisted on ‘manifest first’. In their model, the manifest drives the building of the artifact instead of representing the output of a build process. That is, if you require a bundle in your manifest, then PDE will place this on the classpath of the bundle you’re building. If you import a package in the manifest, PDE will find a bundle exporting that package and place it on the classpath (praying there is only one bundle that exports it in the target).
Inside the OSGi Alliance we were in a bind (pun intended) because PDE was not suitable for our own build. We used bnd in ant (it was also an ant plugin) but we missed the joy of Eclipse. This caused bnd to develop a split personality. Originally it was a JAR generator based on a small recipe but for our build we needed project & workspace concepts. We toyed with the idea to split it into a bld and a bnd tool but in the end even we committed the sin against modularity of low cohesion and kept these two tools in one out of laziness.
Obviously we also developed PDE envy because Eclipse was actually awfully nice except for it. After toying with the idea of using the Eclipse metadata (.classpath) and finding out that this was only possible if we included the complete Eclipse IDE in an offline build, we embarked on developing an Eclipse plugin. Just not a good idea. So we reversed the model, and developed a library that had its own internal, uncoupled, model of a build. You see, the problem is that most build environments are quite pedantic and strongly optimized for their primary goal. Ant, maven, gradle, et. al. are just not easy to use inside an IDE because they are very stream driven: start, process, stop. In this model there is very little incentive to optimize incremental building and event notifications for important changes. An IDE is the reverse, it is start, build, build, build, …, build, build, stop. For performance, it is crucial to optimize the building out of incremental changes to keep the IDE responsive. It is also crucial to send out events when important things happen. So we decided to pursue the middle ground: a model of projects and workspaces that was as uncoupled of the real world as possible but providing the hooks to use it in all popular build tools, either command line tools or IDEs. This is rather well captured in the expression: “The one tool that bnd’s them all”.
Then one day we heard that Neil Bartlett also had started a plugin called bndtools. He had used the open source library of bnd and started to work on creating a pleasant to use friendly environment. Interestingly, he had created a continuous builder for bnd so that every save operation automatically build the bundles, something we had considered in our plugin but had been too afraid to do out of performance fear. Neil, however, was developing a lot of stuff that was already in bnd, he only used the JAR packager and manifest generator. After beating him up, which even took a special trip to the UK where we worked for 8 hours in a hotel lounge, he surrendered and thus our long term fruitful collaboration was born.
Currently bnd(tools) is managed through a Github organization called bndtools and has its home site at https://bndtools.org. It is split in a number of repositories: bndtools, bnd, bndtools.rt and a number of associated support repositories. Though the collaboration between bndtools and bnd is awfully close, we are fanatic about separating the projects, our goal is still to allow Jetbrains to use bnd in IntelliJ without forcing them to eat any Eclipsisms. And it is always fun when hear someone claiming to successfully integrate bndlib in a product or tool, raving how uncoupled it was.
What you are now reading is the manual of bnd, which is the result of a tremendous amount of work, and will be a work in progress for a long time to be. Though much of bnd’s development was indirectly supported by the OSGi Alliance it is and always will be an open source project. This gave the authors the license to scratch their itches and not worry too much about documenting the nifty things they developed to scratch. Though the most important aspects were documented, it was sparse and not overly well organized. Though we always hoped someone with a gift for documenting would come along, fall in love with bnd, and write the perfect documentation, this somehow failed. After 16 years, we find it is time to take up this task ourselves, still praying that we will get support from bnd’s surprisingly large (to some) group of users, don’t hesitate. This is the reason this manual is a github repositories. Contributing is trivial, clone the bndtools/bnd.manual repo, edit the markdown text (you can even do this on the github web), save it, and create a pull request. Don’t (always) ask what bnd can do for you, ask what you can do for bnd’s users …
Not sure why we wrote such a long introduction, the facebook generation seems have an attention span of 5 lines, so we are probably among ourselves dear reader, congratulations on your stamina!