Paul Hammant's Blog: Further Experiments With Expanding/Contracting Monorepos
This builds on Maven In A Google Style Monorepo from a week ago. This time:
- a bigger repo
- still only 0.0007% of the number of files of Google’s Trunk’s HEAD revision
- and only 0.0005% of the history in Google’s monorepo
- a full python expand/contract script that understands Maven, its module hierarchy, as well as Git’s sparse-checkouts
Oh yes, still Maven for this article. Sorry to Maven haters - it is a good example of a depth-first recursive build technology that is in wide use in the enterprise. And it is regular enough for its XML to be processed in Python. This tech would have to be modified for Gradle or Ant, but it is the same idea.
Buck and Bazel are more like Google’s in-house Blaze technology (which is a directed graph build system). Someone else can turn Bazel back into Blaze for monorepo nirvana. Maybe using this know-how.
A Bigger Test Repository
I recently discovered the very pleasant Jooby web framework for Java. Its own Github repo has many Maven subprojects in it, so it is perfect for my continuing experiments.
Take a look at Jooby’s repo itself, and a 700K full PDF version I made of the module dependencies. The full version of that “zoomed in” DOT graph above.
My fork of that repo is where I have completed the expand/contract experiment. There’s a single commit too that shows what I’ve done to make this expandable/contractable.
The experiment on that repo
Clone my fork.
git clone git@github.com:paul-hammant-fork/jooby-monorepo-experiment.git
cd jooby-monorepo-experiment
Open up the pom.xml file in IntelliJ IDEA or Eclipse. Perhaps like so:
# Mac
open pom.xml -a /Applications/IntelliJ\ IDEA.app
Yay, lots of modules are evident after a build:
$ mvn install -DskipTests
...
[INFO] Reactor Summary:
[INFO]
[INFO] jooby-project ...................................... SUCCESS [ 7.371 s]
[INFO] jooby .............................................. SUCCESS [01:37 min]
[INFO] route specification module ......................... SUCCESS [ 13.904 s]
[INFO] executor module .................................... SUCCESS [ 2.111 s]
[INFO] jackson module ..................................... SUCCESS [ 1.383 s]
[INFO] gson module ........................................ SUCCESS [ 1.355 s]
[INFO] handlebars module .................................. SUCCESS [ 2.427 s]
[INFO] freemarker module .................................. SUCCESS [ 2.252 s]
[INFO] pebble module ...................................... SUCCESS [ 1.895 s]
[INFO] jade module ........................................ SUCCESS [ 2.147 s]
[INFO] caffeine module .................................... SUCCESS [ 1.858 s]
[INFO] guava-cache module ................................. SUCCESS [ 2.099 s]
[INFO] jdbc module ........................................ SUCCESS [ 1.569 s]
[INFO] jooq module ........................................ SUCCESS [ 1.389 s]
[INFO] netty module ....................................... SUCCESS [ 10.840 s]
[INFO] hibernate 5.x module ............................... SUCCESS [ 8.319 s]
[INFO] hibernate 4.x module ............................... SUCCESS [ 5.950 s]
[INFO] jooby-dist ......................................... SUCCESS [ 0.315 s]
[INFO] undertow module .................................... SUCCESS [ 5.703 s]
[INFO] servlet module ..................................... SUCCESS [ 2.106 s]
[INFO] jetty module ....................................... SUCCESS [ 4.282 s]
[INFO] quartz module ...................................... SUCCESS [ 1.986 s]
[INFO] jdbi module ........................................ SUCCESS [ 2.267 s]
[INFO] hot reload module .................................. SUCCESS [ 1.842 s]
[INFO] jedis module ....................................... SUCCESS [ 2.127 s]
[INFO] mongodb module ..................................... SUCCESS [ 1.476 s]
[INFO] RxJava module ...................................... SUCCESS [ 3.983 s]
[INFO] mongodb-rx module .................................. SUCCESS [ 12.098 s]
[INFO] morphia module ..................................... SUCCESS [ 2.798 s]
[INFO] camel module ....................................... SUCCESS [ 5.303 s]
[INFO] Elasticsearch module ............................... SUCCESS [ 1.624 s]
[INFO] hibernate validator module ......................... SUCCESS [ 2.563 s]
[INFO] jooby archetype generator .......................... SUCCESS [ 7.064 s]
[INFO] assets module ...................................... SUCCESS [ 4.102 s]
[INFO] maven plugins ...................................... SUCCESS [ 20.021 s]
[INFO] ehcache module ..................................... SUCCESS [ 3.154 s]
[INFO] pac4j module ....................................... SUCCESS [ 21.848 s]
[INFO] swagger module ..................................... SUCCESS [ 3.858 s]
[INFO] raml module ........................................ SUCCESS [ 5.045 s]
[INFO] spymemcached module ................................ SUCCESS [ 1.718 s]
[INFO] aws module ......................................... SUCCESS [ 3.896 s]
[INFO] commons email module ............................... SUCCESS [ 1.631 s]
[INFO] jongo module ....................................... SUCCESS [ 1.505 s]
[INFO] flyway module ...................................... SUCCESS [ 1.246 s]
[INFO] hazelcast module ................................... SUCCESS [ 4.455 s]
[INFO] ebean module ....................................... SUCCESS [ 3.561 s]
[INFO] akka module ........................................ SUCCESS [ 3.536 s]
[INFO] metrics module ..................................... SUCCESS [ 5.628 s]
[INFO] jooby-querydsl ..................................... SUCCESS [ 39.016 s]
[INFO] sitemap module ..................................... SUCCESS [ 14.788 s]
[INFO] rx-jdbc module ..................................... SUCCESS [ 2.368 s]
[INFO] banner module ...................................... SUCCESS [ 11.513 s]
[INFO] reactor module ..................................... SUCCESS [ 9.560 s]
[INFO] whoops module ...................................... SUCCESS [ 9.002 s]
[INFO] gradle plugin ...................................... SUCCESS [ 5.478 s]
[INFO] couchbase module ................................... SUCCESS [ 7.060 s]
[INFO] cassandra module ................................... SUCCESS [ 5.754 s]
[INFO] scanner module ..................................... SUCCESS [ 5.769 s]
[INFO] xss-csl module ..................................... SUCCESS [ 0.762 s]
[INFO] xss-unbescape module ............................... SUCCESS [ 0.707 s]
[INFO] crash shell module ................................. SUCCESS [ 10.459 s]
[INFO] thymeleaf module ................................... SUCCESS [ 1.150 s]
[INFO] j2v8 module ........................................ SUCCESS [ 0.508 s]
[INFO] nodejs module ...................................... SUCCESS [ 0.752 s]
[INFO] jshint module ...................................... SUCCESS [ 0.801 s]
[INFO] jscs module ........................................ SUCCESS [ 1.252 s]
[INFO] less.js module ..................................... SUCCESS [ 0.951 s]
[INFO] less4j module ...................................... SUCCESS [ 1.532 s]
[INFO] csslint module ..................................... SUCCESS [ 0.706 s]
[INFO] ng-annotate module ................................. SUCCESS [ 0.676 s]
[INFO] uglify.js module ................................... SUCCESS [ 1.123 s]
[INFO] clean-css module ................................... SUCCESS [ 1.019 s]
[INFO] yui-compressor module .............................. SUCCESS [ 0.890 s]
[INFO] google-closure compiler module ..................... SUCCESS [ 1.018 s]
[INFO] sass module ........................................ SUCCESS [ 0.958 s]
[INFO] require.js module .................................. SUCCESS [ 0.997 s]
[INFO] babeljs module ..................................... SUCCESS [ 0.880 s]
[INFO] rollup module ...................................... SUCCESS [ 0.708 s]
[INFO] svg-sprites module ................................. SUCCESS [02:16 min]
[INFO] svg-symbol module .................................. SUCCESS [ 1.633 s]
[INFO] autoprefixer module ................................ SUCCESS [ 0.981 s]
[INFO] coverage report .................................... SUCCESS [ 43.310 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:40 min
[INFO] Final Memory: 199M/551M
[INFO] ------------------------------------------------------------------------
To try out a contract operation, go into the checkout on your workstation and do:
python mr/checkout.py jooby-jooq
Kick off that build from root again:
$ mvn install -DskipTests
...
[INFO] Reactor Summary:
[INFO]
[INFO] jooby-project ...................................... SUCCESS [ 5.848 s]
[INFO] jooby .............................................. SUCCESS [ 20.239 s]
[INFO] jdbc module ........................................ SUCCESS [ 7.172 s]
[INFO] jooq module ........................................ SUCCESS [ 2.323 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.715 s
[INFO] Final Memory: 31M/276M
[INFO] ------------------------------------------------------------------------
That checkout.py script calculated the modules needed (jooq, jdbc, jooby and the parent pom) and then
contracted the repo’s checkout. The build, as you can see, is 14x quicker than the full build.
Intellij flickers modestly for a few seconds then shows a dialog:
You should hit ‘yes’ when this is offered after an expansion or contraction of the monorepo.
Changing your own repo to be expandable/contractable
Firstly clone the repo, cd into it, and kick off the Maven build as you would normally.
Then copy the mr folder from my repo into your repo’s checkout
Next run these commands:
mvn com.github.ferstl:depgraph-maven-plugin:aggregate -Dincludes=org.jooby
# replace org.jooby with the group name for all the modules in the
# repo you're wanting to make expandable/contractable.
mv mv target/dependency-graph.dot mr/
find . -name pom.xml -type f | while read a; do n=$(echo $a | sed -e 's/pom.xml/pom-template.xml/'); git mv $a $n; done
find . -name pom-template.xml > mr/all_poms.txt
echo "pom.xml" >> .gitignore
git add .
git commit -m "now monorepo capable"
If you know your module names, the checkout.py will do the right thing for you. You can do more that
one module (no commas). If you don’t specify any module names after checkout.py then it’s going to
expand the checkout back to all modules. For my repo both the expansion and the contraction take about 5 seconds.
More?
On Monorepos
trunkbaseddevelopment.com/monorepos/
On the expandable/contractable tooling Google made for themselves and kept secret for years
trunkbaseddevelopment.com/expanding-contracting-monorepos/