Multi-module build systems per language

Monorepo build styles

We’re going to introduce two anemic applications DirectedGraphBuildSystemsAreCool and MonoreposRule. All they do is print to STDOUT then exit.

Components that the applications use

The components involved in the monorepos-rule application are categorized into several modules, each representing a different type of phonetic sound. These include:

• Vowels: Fundamental sounds in speech, represented by classes like A, E, I, O, and U.
• Nasal: Sounds produced with airflow through the nose, such as M and N.
• Voiceless: Sounds produced without vocal cord vibration, including P and T.
• Sonorants: Sounds produced with a relatively open airflow, including L and R.
• Fricatives: Sounds produced by forcing air through a narrow channel, like S.
• Labiodental: Sounds produced with the lower lip against the upper teeth, such as V and W.
• Glides: Sounds that involve a gliding motion of the articulators, like H, J, and Y.
• Sibilants: Hissing sounds produced by directing a stream of air with the tongue towards the sharp edge of the teeth, including Q, X, and Z.
• Velar: Sounds produced with the back of the tongue against the soft palate, such as G and K.
• Voiced: Sounds produced with vocal cord vibration, including B and D.

The vowelbase component is a unique part of the system that bridges Java and Rust using JNI (Java Native Interface). It consists of:

• Java: The VowelBase class, which loads the native library and provides a method to print strings wrapped in parentheses.
• JNI: Facilitates the interaction between Java and Rust, allowing Java to call native methods implemented in Rust.
• Rust: Implements a native method printString, which outputs the pass-in string to the console wrapped in parentheses. As with much of this app, this is a gratuitous use of Java-invoking-Rust.

Graph of Class Dependencies

Below is a representation of the class dependencies in the project. This graph illustrates the relationships between different components, showing which classes depend on others.

DOT Graph

https://dreampuf.github.io/GraphvizOnline/?engine=dot#digraph%20ClassDependencies%20%7B%0A%20%20%20%20subgraph%20cluster_components%20%7B%0A%20%20%20%20%20%20%20%20label%20%3D%20%22components%22%3B%0A%20%20%20%20%20%20%20%20style%20%3D%20%22dotted%22%3B%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_vowelbase%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22vowelbase%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22libvowelbase.so%22%20%5Bshape%3Drectangle%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22VowelBase%22%20-%3E%20%22libvowelbase.so%22%20%20%5Blabel%3D%22has%20a%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22VowelBaseLibraryUnZipper%22%20-%3E%20%22libvowelbase.so%22%20%20%5Blabel%3D%22unzips%22%5D%3B%0A%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_vowels%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22vowels%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22A%22%20-%3E%20%22VowelBase%22%20%5Blabel%3D%22is%5Cna%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22I%22%20-%3E%20%22VowelBase%22%20%5Blabel%3D%22is%5Cna%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22E%22%20-%3E%20%22VowelBase%22%20%5Blabel%3D%22is%5Cna%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22O%22%20-%3E%20%22VowelBase%22%20%5Blabel%3D%22is%5Cna%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22U%22%20-%3E%20%22VowelBase%22%20%5Blabel%3D%22is%5Cna%22%5D%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_nasal%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22nasal%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22M%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22N%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_fricatives%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22fricatives%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22F%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22S%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_voiceless%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22voiceless%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22P%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22T%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_consonants%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22consonants%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22C%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_sonorants%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22sonorants%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22L%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22R%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20subgraph%20cluster_velar%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22velar%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22G%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22K%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_voiced%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22voiced%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22B%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22D%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20subgraph%20cluster_glides%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22glides%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22H%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22J%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22Y%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20subgraph%20cluster_labriotental%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22labriotental%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22V%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22W%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20subgraph%20cluster_sibilants%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22sibilants%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22Q%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22X%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22Z%22%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%0A%20%20%20%20B%20-%3E%20V%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20S%20-%3E%20F%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20T%20-%3E%20Q%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20%23V%20-%3E%20W%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20Y%20-%3E%20J%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20G%20-%3E%20K%20%5Bcolor%3D%22transparent%22%5D%3B%0A%0A%0A%20%20%20%20subgraph%20cluster_key%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20label%20%3D%20%22Key%22%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22Rust%22%20%5Bshape%3Drectangle%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22libvowelbase.so%22%20-%3E%20Java%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22libvowelbase.so%22%20-%3E%20Rust%20%5Bcolor%3D%22transparent%22%5D%3B%0A%20%20%20%20%7D%0A%0A%20%20%20%20subgraph%20cluster_applications%20%7B%0A%20%20%20%20%20%20%20%20label%20%3D%20%22applications%22%3B%0A%20%20%20%20%20%20%20%20style%20%3D%20%22dotted%22%3B%0A%20%20%20%20%20%20%20%20subgraph%20cluster_MonoreposRule%20%7B%0A%20%20%20%20%20%20%20%20label%20%3D%20%22monorepos_rule%22%3B%0A%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22E%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22O%22%20%5Blabel%3D%22has%5Cn3%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22U%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22M%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22N%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22P%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22L%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22R%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%22MonoreposRule%22%20-%3E%20%22S%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22red%22%5D%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%0A%20%20%20%20%20%20%20%20subgraph%20cluster_DirectedGraphBuildSystemsAreCool%20%7B%0A%20%20%20%20%20%20%20%20label%20%3D%20%22directed_graph_build_systems_are_cool%22%3B%0A%0A%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22D%22%20%5Blabel%3D%22has%5Cn3%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22I%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22R%22%20%5Blabel%3D%22has%5Cn3%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22E%22%20%5Blabel%3D%22has%5Cn4%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22C%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22T%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22G%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22A%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22P%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22H%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22B%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22U%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22L%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22S%22%20%5Blabel%3D%22has%5Cn3%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22Y%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22M%22%20%5Blabel%3D%22has%5Cna%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%22DirectedGraphBuildSystemsAreCool%22%20-%3E%20%22O%22%20%5Blabel%3D%22has%5Cn2%22%20color%3D%22blue%22%5D%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%7D%0A

You can see that neither app does not use two of the component libs: sibilants and labiodental. Each app only needs the letters in its own class name.

Depth-first recursive

That is Java’s Maven for our discussion. In a typical Maven build, modules can be built from the root of the project, ensuring all dependencies are resolved and built in the right sequence. Alternatively, if modules are already present in the local Maven repository (~/.m2/repository), Maven can pull these pre-built modules, potentially speeding up the build process by avoiding recompilation. We’re going to ignore that alternate way of working for now.

Maven prefers to encode its build instructions and dependencies in an XML formal, One per module typically. Maven prefers to traverse each module to its full depth of submodules before traversing to the next module at sibling level to understand the entire implicit graph. Intelligence from that graph means Maven reorders modules to make sure that the most-depended-on ones are compiled ad tested first. That is a “depth-first” visitation of modules. At some level if Maven thinks the compilation order is not important, the order could resemble the original depth-first visitation.

A First Maven Build: Full Clean, Compilation, Testing and Jar creation

Calculated order of module traversal

./quieter-mvn clean package

(this shel script is just a mvn clean package invocation with fewer lines of output)

Output:

<Rust compilation output>
clean for components
clean for components-vowel-base
rust compile for components-vowel-base
java compile for components-vowel-base
java tests compile for components-vowel-base
test for components-vowel-base
jar for components-vowel-base
copy-resources for components-vowel-base
clean for components-vowels
java compile for components-vowels
java tests compile for components-vowels
test for components-vowels
jar for components-vowels
clean for components-nasal
java compile for components-nasal
java tests compile for components-nasal
test for components-nasal
jar for components-nasal
clean for components-voiceless
java compile for components-voiceless
java tests compile for components-voiceless
test for components-voiceless
jar for components-voiceless
clean for components-sonorants
java compile for components-sonorants
java tests compile for components-sonorants
test for components-sonorants
jar for components-sonorants
clean for components-fricatives
java compile for components-fricatives
java tests compile for components-fricatives
test for components-fricatives
jar for components-fricatives
clean for monorepo-sim
clean for applications
clean for monorepos-rule
java compile for monorepos-rule
java tests compile for monorepos-rule
test for monorepos-rule
jar for monorepos-rule
uberJar for monorepos-rule
clean for components-voiced
java compile for components-voiced
java tests compile for components-voiced
test for components-voiced
jar for components-voiced
clean for components-consonants
java compile for components-consonants
java tests compile for components-consonants
test for components-consonants
jar for components-consonants
clean for components-velar
java compile for components-velar
java tests compile for components-velar
test for components-velar
jar for components-velar
clean for components-glides
java compile for components-glides
java tests compile for components-glides
test for components-glides
jar for components-glides
clean for components-labiodental
java compile for components-labiodental
java tests compile for components-labiodental
test for components-labiodental
jar for components-labiodental
clean for directed-graph-build-systems-are-cool
java compile for directed-graph-build-systems-are-cool
java tests compile for directed-graph-build-systems-are-cool
test for directed-graph-build-systems-are-cool
jar for directed-graph-build-systems-are-cool
uberJar for directed-graph-build-systems-are-cool
clean for components-sibilants
java compile for components-sibilants
java tests compile for components-sibilants
test for components-sibilants
jar for components-sibilants

(78 steps)

All in all, that was about 11 seconds on my middle-of-the-pack AMD cpu desktop. It’d be about the same if clean was done in one command, then package was done in a follow-up.

A Second Maven Build: Incremental Compilation and Testing, because of no ‘clean’ step

In the second build, Maven efficiently skips recompilation as they have not changed. Maven detects changes by comparing timestamps of source files with previously compiled classes. This works for the Rust sources too.

./quieter-mvn package

Maven still executes tests of modules regardless of whether the module sources have changed. That’s because a linked dependency may have changed.

<Rust compilation output>
rust compile for components-vowel-base
java compile for components-vowel-base
java tests compile for components-vowel-base
test for components-vowel-base
jar for components-vowel-base
copy-resources for components-vowel-base
java compile for components-vowels
java tests compile for components-vowels
test for components-vowels
jar for components-vowels
java compile for components-nasal
java tests compile for components-nasal
test for components-nasal
jar for components-nasal
java compile for components-voiceless
java tests compile for components-voiceless
test for components-voiceless
jar for components-voiceless
java compile for components-sonorants
java tests compile for components-sonorants
test for components-sonorants
jar for components-sonorants
java compile for components-fricatives
java tests compile for components-fricatives
test for components-fricatives
jar for components-fricatives
java compile for monorepos-rule
java tests compile for monorepos-rule
test for monorepos-rule
jar for monorepos-rule
uberJar for monorepos-rule
java compile for components-voiced
java tests compile for components-voiced
test for components-voiced
jar for components-voiced
java compile for components-consonants
java tests compile for components-consonants
test for components-consonants
jar for components-consonants
java compile for components-velar
java tests compile for components-velar
test for components-velar
jar for components-velar
java compile for components-glides
java tests compile for components-glides
test for components-glides
jar for components-glides
java compile for components-labiodental
java tests compile for components-labiodental
test for components-labiodental
jar for components-labiodental
java compile for directed-graph-build-systems-are-cool
java tests compile for directed-graph-build-systems-are-cool
test for directed-graph-build-systems-are-cool
jar for directed-graph-build-systems-are-cool
uberJar for directed-graph-build-systems-are-cool
java compile for components-sibilants
java tests compile for components-sibilants
test for components-sibilants
jar for components-sibilants

(61 steps as no ‘clean’ goal specified)

All in all, that was about 5.5 seconds on my middle-of-the-pack AMD cpu desktop. That’s half the elapsed time from the clean run because of the incremental nature of compilation

Unfortunately, the compile and surefire (testing) Maven plugins are still entered causing them to be listed above, but the skip of the actual do-compile and run-tests did happen, just within the Maven plugin.

Slightly faster with Maven options -pl and -am*

Focussing on the MonoReposRule app.

$ ./quieter-mvn package -pl applications/monorepos_rule -am
<Rust compilation output>
rust compile for components-vowel-base
java compile for components-vowel-base
java tests compile for components-vowel-base
test for components-vowel-base
jar for components-vowel-base
copy-resources for components-vowel-base
java compile for components-vowels
java tests compile for components-vowels
test for components-vowels
jar for components-vowels
java compile for components-nasal
java tests compile for components-nasal
test for components-nasal
jar for components-nasal
java compile for components-voiceless
java tests compile for components-voiceless
test for components-voiceless
jar for components-voiceless
java compile for components-sonorants
java tests compile for components-sonorants
test for components-sonorants
jar for components-sonorants
java compile for components-fricatives
java tests compile for components-fricatives
test for components-fricatives
jar for components-fricatives
java compile for monorepos-rule
java tests compile for monorepos-rule
test for monorepos-rule
jar for monorepos-rule
uberJar for monorepos-rule

These maven options are project list (-pl) and also make (-am). They allow for more efficient builds by targeting specific modules and their dependencies. The -pl option specifies the modules to build, while -am ensures that all required dependencies are also built. The former is the local-path not the Group:Artifact:Version that Maven normally counts as canonical. This approach helps in skipping unrelated modules, thus reducing build time.

Note components-sibilants (and others) were not a dep for the MonoreposRule app, so Maven decided to skip those completely. In this instance, Maven still visited all modules however deep to build understanding, but decided subset the larger list to support the more focused intention.

Maven has a “Reactor” facility. The Reactor is the part of Maven that schedules the sequence of module builds based on dependencies and orchestrates the execution of lifecycle phases - such as compile, test, and package (which typically produces a JAR) — within each module.

Sequence Diagram:

https://mermaid.live/edit#pako:eNqVV82O2zYQfhWChyIBvI4d_6wtBDl0iwBZYHPYADkUvnAl2mYqkS5JOXEX22Pfoa_XJ-mQ-iVN_axPGs73zYy-GVLmM45FQnGEFf0zpzymvzFykCTbcQS_E5GaxexEuEaPlMRaSEQUeiBnyj88yY_l2jXYIh7pSdRwdEe5liRFZtVw38QkPtLEAP5-h1IRkzS9vL0O9e1XoqhBfRM_aHrzZKx7ciboF_SYK12EEtlJcEgQ4huaqgOoQcIXoqBMwNuHEfFZTFOqyhSlMUj7KriQ8GBptTFI-yRZTDQ7U8trrEHig0kB0qvHPLVyOguWTk6n1MYTHAIUIaC_7Ew0RU6rS-PmYzUBEUqopjJjnMLTSR9v9kwWzZE0zqWCGpGQCZVI7FEmEsipqhxNNNvrCJ2ZYhp5BVhfscYF2JIdjtpEK0l3RB4EOlCjKHrKeXwEpyngv3_-NTUptJciQ3cSgqkpE0UoS24yS5goZGRkKQ0CMvIHRVMVZn83Y-mw631w00LlnOnpdwJbiSfoSLJYUlUsQJmUJ7ANmVGnKwPUDy_ZV6XMeYHqfgdI5zrrTk6L9YS60pu1QMfsnipbVjgLoNs7i2rna1Hvv3yGDd16-UsT4QoM0oeB1t8CDvWigo1tRl-aUEN8oN8S3-81pXL3daU42IJtsUdX3RWvG9ZZrNnHBj8gWoV6zQCHMgT08mCeWp7X1apy9khVvvH18JandZdSNaDqSmm63MFRa5Cv2vo92YITF4RfzV0Q5Sp6De0dQ0ejRt36o9albg0o1mvT5Q6o20a-Rt2-bAF1O-Ceuh0oV902qK3rlbKePr6-zce_S-AGUTga26MPaOxAXyNyf8KAzJ0ET-hOnCu1A-uZYV8oX2rnv5LzsatDOBDnEL_mO2JdvC-dh_U-eG_gPzSH7FDr236imyRALM7NkdXVZ8E4fDPj4_CtRo0jOJAQpXegfXRgpscP-ZhSAqM-QPMGfgDtjv0YcP5E5X1FaG0F955Q7wU_ZM9mqm8KO44n-CBZgiMtczrBGdwQiDHxs2HtsD7SjO5wBI8J3ZM81Tu84y9AgzvL70JkFVOK_HDE0Z6kCqz8lECi8qZar0rTF3kncq5x9H61sUFw9Ix_4mg7m27er9fLxWKxmm02y-0EX3A0X0zn6-XsdrParG_nq-1y9TLBf9m08-l8O1vO5uvbxWK-3q5W6wkmuRZfLzyuiqIJg_d8KO7P9hr98j9oaT5x (may 8)

TODO: add svg of the same

A note on our test application and component dependencies

The modules we have here are not representative of enterprise module sizes. Each of the java modules are just a few tiny classes. In reality, there could be many 100’s of sources per module and compilation for each could be many tens of seconds. The same is true of the tests for each module and the execution of tests (our repeated testPlaceholder() testmethod if quite poor really).

Discussion

The depth-first recursive nature of Maven’s build process ensures that all dependencies are resolved and built in the correct order. The incremental build capabilities of Maven, combined with options like -pl and -am, help in optimizing the build process by avoiding unnecessary recompilation and retesting, thus saving time and resources. Many times though, devs would do an initial mvn install from the root, and Maven would visit every module in a depth-first recursive manner. For Google’s many thousands of applications and libraries with high automated test coverage, that could take days on a single dev workstation and is never ever done even with Blaze given the checkout of all source from their trunk was 90 GB in 2012, and no dev workstation ever did that “all source” action.

Maven remains a historically dominant build tool in the Java ecosystem. At least for established Java enterprises it does. Microsoft .NET enterprises would consider MsBuild the dominant tech for multi-module backend builds.

Sequence Diagram for a directed graph build system

Same as for the depth-first recursive Maven, Making the fat-jar of the MonreposRule application, with ALL tests running for all pertinent sources:

rm -rf target
./javatests/applications/monorepos_rule/.testAndDist.sh

Mermaid Sequence Diagram:

https://mermaid.live/edit#pako:eNqVVslu2zAQ_RWBhyIpHNW7VaHwoS0KNEBycIAeCl8YaWwzlUiVpNy4gf89XGxtlmxaJ85w3uPjcIbiG4pYDChEAv7mQCP4TvCa43RJPfVlmEsSkQxT6T0wyjhkTCzyBDws6o4vz3x-E7E0xTS-Swg1DpxlCYmwJIzenhL-4GZuC0KzldaRKmMUqGwBPmKBE40xg4vhT1qnGphlCuMi7BcjESQgDKwwHGD_IDlg9MgN8BULk9N7vMWlx2DvH396TviFJljkQjYICvCnhDxzzHeKxNLgSCUcS6gfpZ2rue7m8_J8Qs_XlCQBX2y8LcGeL0FIoawGbYmxE6XdILR8hK69jLPY0yXZgqhJUipsSAyt67XvwlSM-wZMuPWZYcnQqfgYd1HsgbtdZ1Gm7loLiPUXZp2tU3c1_qL2ylrt-ot-cddfQKy_MOtsnfqr8Rf1V9bq0q979xrxOv6oRI-PJLoNr-TRkAqVNqtsizpdF8OiQbFocJSZ5PrOOElnByzFf8AifMHOxbbnvVNcHSqULvWXUUsoeAY0Vr-m3ZmcVLfzou_Qru1Uz7ZbXttBCqc1HMvPVsvJHXcK_uhHCVb_oGYW7DXjHl-2tzumbKlrMCYJVwLMybhgvBupdiGITtdtWw01GFrrqIWk2f8NmvMHfx1Yd747GF4hyqUGmxvDCWNa9AVzb8VZ6uHEPJGsZp1O9d76UDZvpSxrNKiHUuApJrF6HL7p2CWSG0hhiUI1jGGF80Qu0ZLuVSjOJXva0QiFkufQQ5zl6w0KVzgRysqzWPEfXpbHEPV4QeEbekXhcDr1R9PBKOhPBv1gMAt6aIfCwbDvj8fD8TCYjSeT2WgS7HvoP2OKYOQPJpNRoGYG0-Es6M-mPQQxkYw_2MesedP20Jpr9QdF-vD5N5ZTicLPZvnfhkxP798BOK_S8Q

TODO: add svg of the same

rm -rf target
./javatests/applications/monorepos_rule/.testAndDist.sh --test-deps-too

Mermaid Sequence Diagram:

https://mermaid.live/edit#pako:eNqVV8ty2yAU_RWGRSfpOIof8aOaThZNp51kmiycmS463mAJ26QIVEBJ3Ey2_YZ-X7-kgGw9sBRkrQCdcy4cLq8XGPEYwxBK_CvDLMKfCVoLlCwY0F-KhCIRSRFT4JYzLnDK5TyjGCBZb_i4FJcnEU8SxOIzSphtQGlKSYQU4ez0UPAbWUqjc07JEqw4jbEwJMLAV6KAUT7kfBFW7xFbZlnbh085w0w1BLtDElHDsQUv_N6MTRdsmKLipX3nJMIUS0srKh1oT5juOKbUjfAJSTsPN-gRlS2We3N3DTrx50ZgnknlCBRkMzkCia0WyWVQpA1HCtenP_9Xazq7vCznJwSBkSQUB3IDHgkCgcJSSV1zZEtO_qOsO4K5HmFrkAoeA5PGOcPk1ZkDfsgYUcEDEkDnJ9igJBI6vGnoHMX0txKllYGfcZSpPaNpFDWbtDM5JMaNHjQ7a7O4u6kWnrfZYqngdXGH8xnYKuvY5uAazdpjvD7thtVsUbFqu9tUUPL2olpX81pWwfps84Zw7GvBN9pYxXqtrAy92c5iN-tuZ0HJ24tqXc1rZwXrs9MbwrGzBd9oZxXrtbMy9DY7zUZ_jJcGv–JKe9FzJ59pI6hVKRMtao2r8u1KczB7hfjupnilQJ8BaoqV0isOVhjfYym2VJfBYA5XGKcygBca7co3ZoGtUEKPPGMxmBpj5-V4Im9Fshzoc-mc_Dvz999sFLf6XI5yYbjJlMrLUE_cc4IJH8L2zzNe-QBs06Vul_61qRDaHqKWayvWts3pqA6nAdzvrcNp5pK7d1ryhvpi1GuvxztX3xvyx-sPQfesup2qA5LLl8hBwfuIfl9EFGkL2nuVOQHT3d8ucN255TbyDEca8KRBJseXTjgROlRSGLsOm1KZEehMZkbRNw9z5Hpln0OyZeEx8V0UtJDbkxQD8duL6a7dk_TO57Z3_KhmlnQ75h35cZTyeaaDOzBBIsEkVg_1F4MdgHVBid4AUNdjPEKZVQt4IK9aijKFL_fsgiGSmS4BwXP1hsYrhCVupalsdbfvfL2EP0ogOELfIbhcDIJRpPBaNYfD_qzwXTWg1sYDob94OJieDGcTS_G4-loPHvtwd-ca4FRMBiPRzP9ZzAZTmf96aQHcUwUF7f5w9K-L3twLUzvdz0yOSOueMYUDD_Y8D-smPn9-h-xAw-q

TODO: add svg of the same

Versus the depth-first recursive Maven, there’s no scheduler looking at the source-base up front.

Google Style Monorepo - directed acyclic graph

I’m not going to use Bazel (bazel.build (Google’s open-sourcing of Blaze) or Nx (the newer popular monorepo build technology nx.dev). I’m going to use an imperfect system of shell scripts to illustrate only two aspects of the Blaze I recall from my nearly two years in Google’s Test Mercenaries team:

1. a leaf-first directed (acyclic) graph approach builds in a monorepo
2. a checking of changed-or-not of sources to skip individual steps

I accept that my shell-script alternate to Blaze/Bazel/Buck/Nx is crude and repetitive versus those. It is more teachable, though.

Cloning the simulation repo

git clone --no-checkout git@github.com:paul-hammant/monorepo-sim-preso2.git monorepo-sim-preso2b
cd monorepo-sim-preso2b
git sparse-checkout init --cone
git sparse-checkout add shared-build-scripts
git checkout
./shared-build-scripts/gcheckout.sh add javatests/applications/monorepos_rule

./javatests/applications/monorepos_rule/.tests.sh

First build

Picking the contrived MonoreposRule app again:

rm -rf target
./javatests/applications/monorepos_rule/.testAndDist.sh

Output:

New build:
java/components/fricatives/.compile.sh: compiling prod code
java/components/nasal/.compile.sh: compiling prod code
java/components/sonorants/.compile.sh: compiling prod code
java/components/voiceless/.compile.sh: compiling prod code
rust/components/vowelbase/.compile.sh: compiling prod code
   Compiling proc-macro2 v1.0.86
   Compiling unicode-ident v1.0.13
   Compiling same-file v1.0.6
   Compiling thiserror v1.0.64
   Compiling memchr v2.7.4
   Compiling bytes v1.7.2
   Compiling jni-sys v0.3.0
   Compiling log v0.4.22
   Compiling cesu8 v1.1.0
   Compiling walkdir v2.5.0
   Compiling jni v0.20.0
   Compiling combine v4.6.7
   Compiling quote v1.0.37
   Compiling syn v2.0.79
   Compiling thiserror-impl v1.0.64
   Compiling vowelbase v0.1.0 (/home/paul/scm/monorepo-sim-preso2/directed_acyclic_graph_modular_monorepo/rust/components/vowelbase)
    Finished release [optimized] target(s) in 5.38s
java/components/vowelbase/.compile.sh: compiling prod code
java/components/vowels/.compile.sh: compiling prod code
java/applications/monorepos_rule/.compile.sh: compiling prod code
javatests/applications/monorepos_rule/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.M(O)N(O)R(E)P(O)SR(U)L(E)
Time: 0.021

OK (1 test)

Making applications/monorepos_rule distribution jar
To run this application, do:  java -Djava.library.path=. -jar ./target/applications/monorepos_rule/bin/monorepos-rule.jar

This skipped all the component tests. It did run the tests for the app, MonoreposRule though. For the .dist.sh scripts in the javatests/ folder it also runs the co-located .tests.sh script and the .dist.sh for the same module from the java folder. I could have run ./java/applications/monorepos_rule/.dist.sh instead causing no tests to run.

Build time is the same as the mvn package -pl applications/monorepos_rule -am invocation for the depth-first_recursive_modular_monorepo branch (with Maven is that wasn’t obvious).

Running it again

A lot of step-skipping results:

New build:
java/components/fricatives/.compile.sh: skipping compilation of prod code (not changed)
java/components/nasal/.compile.sh: skipping compilation of prod code (not changed)
java/components/sonorants/.compile.sh: skipping compilation of prod code (not changed)
java/components/voiceless/.compile.sh: skipping compilation of prod code (not changed)
rust/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowels/.compile.sh: skipping compilation of prod code (not changed)
java/applications/monorepos_rule/.compile.sh: skipping compilation of prod code (not changed)
javatests/applications/monorepos_rule/.tests.sh: skipping compilation of test code (not changed)
Making applications/monorepos_rule distribution jar
To run this application, do:  java -Djava.library.path=. -jar ./target/applications/monorepos_rule/bin/monorepos-rule.jar

The build scripts worked out that nothing had changed since the previous invocation.

Building the other application

Command:

# Don't delete the 'target' folder to see what happens
./javatests/applications/directed_graph_build_systems_are_cool/.testAndDist.sh

Output has some skipping because I build monorepos_rule just before:

New build:
java/components/consonants/.compile.sh: compiling prod code
java/components/fricatives/.compile.sh: skipping compilation of prod code (not changed)
java/components/glides/.compile.sh: compiling prod code
java/components/nasal/.compile.sh: skipping compilation of prod code (not changed)
java/components/sonorants/.compile.sh: skipping compilation of prod code (not changed)
java/components/velar/.compile.sh: compiling prod code
java/components/voiced/.compile.sh: compiling prod code
java/components/voiceless/.compile.sh: skipping compilation of prod code (not changed)
rust/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowels/.compile.sh: skipping compilation of prod code (not changed)
java/applications/directed_graph_build_systems_are_cool/.compile.sh: compiling prod code
javatests/applications/directed_graph_build_systems_are_cool/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.D(I)R(E)CT(E)DGR(A)PHB(U)(I)LDSYST(E)MS(A)R(E)C(O)(O)L
Time: 0.028

OK (1 test)

Making applications/directed_graph_build_systems_are_cool distribution (jar)
To run this application, do:  java -Djava.library.path=. -jar ./target/applications/directed_graph_build_systems_are_cool/bin/directed-graph-build-systems-are-cool.jar

The fat-jar that contains all the deps for the directed-graph-build-systems-are-cool app, and none of the ones that monorepos_rule needed that were not in common with the former. You can see in this build that the common components were skipped as they had already been compiled

Incidentally, launching one of our apps shows its contrived nature:

$ java -Djava.library.path=. -jar ./target/applications/directed_graph_build_systems_are_cool/bin/directed-graph-build-systems-are-cool.jar
libvowelbase.so extracted successfully.
main() .. DirectedGraphBuildSystemsAreCool instance created:
D(I)R(E)CT(E)DGR(A)PHB(U)(I)LDSYST(E)MS(A)R(E)C(O)(O)L
DirectedGraphBuildSystemsAreCool inst toString():
DirectedGraphBuildSystemsAreCool{d=class components.voiced.D, i=class components.vowels.I, r=class components.sonorants.R, e=class components.vowels.E, c=class components.consonants.C, t=class components.voiceless.T, e2=class components.vowels.E, d2=class components.voiced.D, g=class components.velar.G, r2=class components.sonorants.R, a=class components.vowels.A, p=class components.voiceless.P, h=class components.glides.H, b=class components.voiced.B, u=class components.vowels.U, i2=class components.vowels.I, l=class components.sonorants.L, d3=class components.voiced.D, s=class components.fricatives.S, y=class components.glides.Y, s2=class components.fricatives.S, t2=class components.voiceless.T, e3=class components.vowels.E, m=class components.nasal.M, s3=class components.fricatives.S, a2=class components.vowels.A, r3=class components.sonorants.R, e4=class components.vowels.E, c2=class components.consonants.C, o=class components.vowels.O, o2=class components.vowels.O, l2=class components.sonorants.L}

That is all they do: print to stdout then exit

Other build-tech features for this demo project

Test Dependencies Too (-tdt or –test-deps-too)

If I add -dtt to .dist.sh or .tests.sh, the build will do the tests for the dependent modules too

$ rm -rf target
$ ./javatests/applications/monorepos_rule/.tests.sh 
... output omitted ....
$ ./javatests/applications/monorepos_rule/.tests.sh --test-deps-too
New build:
java/components/fricatives/.compile.sh: skipping compilation of prod code (not changed)
javatests/components/fricatives/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.FS
Time: 0.004

OK (1 test)

java/components/nasal/.compile.sh: skipping compilation of prod code (not changed)
javatests/components/nasal/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.MN
Time: 0.003

OK (1 test)

java/components/sonorants/.compile.sh: skipping compilation of prod code (not changed)
javatests/components/sonorants/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.LR
Time: 0.004

OK (1 test)

java/components/voiceless/.compile.sh: skipping compilation of prod code (not changed)
javatests/components/voiceless/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.PT
Time: 0.003

OK (1 test)

rust/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowelbase/.compile.sh: skipping compilation of prod code (not changed)
java/components/vowels/.compile.sh: skipping compilation of prod code (not changed)
javatests/components/vowels/.tests.sh: compiling test code and executing tests
JUnit version 4.12
.(A)(E)(I)(O)(U)
Time: 0.009

OK (1 test)

java/applications/monorepos_rule/.compile.sh: skipping compilation of prod code (not changed)
javatests/applications/monorepos_rule/.tests.sh: skipping compilation of test code (not changed)

Comparing features

In a larger company’s monorepo, where the intention is to share code at the source level and minimize reliance on a binary repository like Nexus or Artifactory, the company may want DAG build systems like Buck, Bazel, or Nx, with these features:

1. Incremental Builds: Buck, Bazel, and Nx are designed to optimize incremental builds. They track file changes and only rebuild the parts of the codebase that are affected by those changes. This is crucial in large monorepos to reduce build times significantly.
2. Remote Caching and Execution: These systems support remote caching and execution, which allows build artifacts to be shared across different machines (say teammates in an delivery team). This can drastically speed up builds by reusing previously built artifacts and distributing the build workload across a cluster of machines.
3. Hermetic Builds: They ensure that builds are hermetic, meaning they produce the same output given the same input, regardless of the environment. This is achieved by isolating the build environment and managing dependencies explicitly.
4. Parallel Execution: Buck, Bazel, and Nx can execute build steps in parallel, leveraging multi-core processors to speed up the build process. This is particularly beneficial in large codebases with many independent modules.
5. Fine-Grained Dependency Management: These tools allow for fine-grained dependency management, where dependencies are specified at a more granular level. This helps in minimizing the build scope and reducing unnecessary recompilation.
6. Cross-Language Support: They support multiple programming languages and can handle complex build scenarios involving different languages. This is important in a monorepo where different teams might use different technologies.
7. Reproducibility: By using a directed acyclic graph (DAG) approach, these systems ensure that builds are reproducible. This is essential for debugging and ensuring consistency across different environments.
8. Scalability: Buck, Bazel, and Nx are designed to scale with the size of the codebase. They can handle large numbers of modules and dependencies efficiently, which is crucial in a large monorepo.
9. Custom Build Rules: They allow for the creation of custom build rules, enabling teams to define specific build logic that suits their needs. This flexibility is important in a diverse development environment.
10. Integration with CI/CD: These tools integrate well with continuous integration and continuous deployment (CI/CD) systems, facilitating automated testing and deployment processes.
11. Community and Ecosystem: Buck, Bazel, and Nx have active communities and ecosystems, providing plugins, extensions, and support for various use cases. This can be beneficial for staying up-to-date with best practices and leveraging community contributions.

The tech I’ve made on a 0-5 score, has

1. 3: mine’s imperfect as it uses timestamps rather than a hash of source file content, and doesn’t delve into deps (binary or in-repo sources) at all.
2. 0
3. 0
4. 0
5. 3: Bazel can depend on listed individual sources, not all in a directory/module
6. 3: any languae could be added to shell scripts, but I didn’t even complete the job here for the Rust module. Specifically Cargo is getting the deps from public not within a repo-coupled system (Java’s deps are in lib/’)
7. 2
8. 4: At some point I would regret the verbose build scripts with unchanging sections, vs something tith library functions and a more compact way of declaring other-module dependencies.
9. 2: doable, but I didn’t show any
10. 4: I don’t think there’s a reason that CI would find this tech problematic - maybe the reporting needs to be made better.
11. 0

More on fine-grained dependency management

There are some additional aspects that tools like Buck, Bazel, and Nx provide that enhance fine-grained dependency management:

1. Explicit Dependency Declarations: These tools require explicit declarations of dependencies, which helps in understanding and managing the relationships between different parts of the codebase more clearly.
2. Granularity at the file Level (as mentioned): While my approach manages dependencies at the module level, these tools can manage dependencies at the file level - “Module Abc needs ../../modules/foo/bar/Baz.java only and none of the rest of that dir”, allowing for even more precise builds.
3. Graph Visualization: They often provide tools to visualize the dependency graph, which can be useful for understanding complex relationships in large codebases, though a Python script could make that I guess.
4. Expanding and contracting of the checkout (Google has that with their in-house Piper-based monorepo).