Paul Hammant's Blog:
Benchmarking a Java/Maven monorepo
Maven is the build technology that’s dominant in the Java community, but also has a ‘love to hate’ aspect to it. If you’re in Java development you’re going to encounter it again and again, so on that basis, I’ll discuss a public monorepo that uses it - that of the excellent Jooby web micro-framework
Note: I found this technology after a production usage of SparkJava that was cool, but left me with dissatisfaction because of the static nature of it. I went as far as to complete a refactoring to make it non-static in nature that was also mostly backwards compatible. The author probably felt that it was wrong in a number of ways. Anyway, then I found Jooby as I was trying to stay way from SpringBoot. Apart from anything else, Jooby uses a Monorepo for the organization of its dependent integration modules.
Cloning Jooby
git clone git@github.com:jooby-project/jooby.git
That takes 20 seconds and delivers a .git
folder that is 46MB in size, and a checkout that is 61MB.
Building it.
There are three runs, of mvn install
below, each with different parameters. Each takes a different elapsed time and
consumes disk space differently. Importantly, all these builds are from the project root.
In order to test the impact on the acquisition of JARs from Maven central that go into a local artifact cache, I’m
doing a rm -rf ~/.m2/repository/
before each mvn
command below.
Jooby itself but no modules that depend on Jooby
mvn clean install -pl jooby -am
This gets a total of 268 jars from Maven central consuming 54MB of disk space and takes 1m 41s to complete. Much of
those 268 are plugins for maven itself. After the build, there is an additional 6MB of items in target/
folders. The results in tabular form:
Jars from Maven Central | 268 |
Disk space for those Jars | 54MB |
Build time | 1m 41s |
Temp target/ folder space |
6MB |
Jooby’s MongoDB integration module
Well and the things Jooby-Mongo depends on, but no modules that depend on it.
mvn clean install -pl jooby-mongodb -am
The results:
Jars from Maven Central | 286 |
Disk space for those Jars | 61MB |
Build time | 2m 10s |
Temp target/ folder space |
6.6MB |
If you cd into jooby-mongo
and run straight mvn clean install
from there, the build time is just 17 seconds. At least, if the dependencies are already cached. Recursive build systems
like Maven allow you to build that way, as an option. It turns out the -am -pl
business is lesser known known, though, than
straight mvn install
after traversing to the directory/module you’re interested in.
Building everything
Yup, Jooby itself and all dependant modules including Jooby-Mongo (again).
mvn clean install
Perhaps only Jooby committers themselves are interested in this build. The results:
Jars from Maven Central | 809 |
Disk space for those Jars | 347MB |
Build time | 13m 30s |
Temp target/ folder space |
88.7MB |
Working in an IDE
The pinnacle of Java IDEs is Intellij IDEA, of course.
Mvn idea:idea
seems to work, but it doesn’t obey the -pl and -am flags. That was the official Maven plugin for Intellij
IDEA, but it is abandoned now.
I raised a feature request with the excellent JetBrains folks, as they have a competing (and recommended) way of loading Maven projects into the IDE. Until that’s implemented all the modules are going to be shown, regardless of the intention of the developer.
Asking Jooby’s author Edgar Espina
Edgar sharing his thinking on motivations, pros/cons, reflecting back now he’s accomplished it:
It was easy to start with… especially when I need to make a new release. All the modules are updated to follow the release, this simplifies a lot my work as framework developer but also helps to framework users. On upgrades, they just changed a single property/value.
Cons?
Today it is very very hard to just see the number of sub-modules in GitHub. It could be considered a GitHub UX issue too, but also I think Jooby has two many modules and they need to be moved to their own repository. Having modules in their own repository helps when someone need to change something… They just need to checkout the sub-project and not the entire repo.
The code coverage project was also hard to implement… It has references to ALL the existing modules. I need this for producing a single JaCoCo (code coverage)report. JaCoCo doesn’t work well on multi-module project :(
Conclusion / My opinion
Edgar’s monorepo setup gives him a bunch of nebulous benefits without sacrificing build speed and disk space given Maven’s build options. Maven’s command line is a little bit cryptic, though, specialist knowledge has to be shared.
Consistency is one of the nebulous benefits. While the wordsmith Emerson is generally right when he says “A foolish consistency is the hobgoblin of little minds …”, in this case, he’s wrong, it was worth it.
Expanding/Contracting monorepos
You may be interested in a previous article of mine on Jooby’s monorepo - specifically, a modification to it that allowed it to expanded and contract (like Google’s) based on the intention of the developer who’s about to do work.
Comments formerly in Disqus, but exported and mounted statically ...
Fri, 07 Apr 2017 | Markus Kohler |
Hi Paul, | |
Fri, 07 Apr 2017 | paul_hammant |
Generally we're better composing applications in fluent Java than annotations. I grant you annotations are way better than XML for composition (thank goodness Spring moved away from that), but that is not enough. Take a look at https://github.com/jooby-pr.... Then compare that to https://github.com/jcsantos.... Specifically only the URL mapping grammar. The Java web frameworks have begun to get out of the way. For bits and pieces of the PicoContainer project ten years ago was already into embedded Jetty, and weaving something that was reminiscent enough of what we're looking at now. That said SparkJava was the first production-ready tech I played with in the Java space, in this style. When moving away from it, I was pretty sure it was not going to be SprintBoot that I settled on. Discovering Jooby was like finding something that I wish I'd made myself. Moreover, looking at it's own source, made me think I've not such a good Java programer after all. |