Maven Central Meta Model

A few years ago I made a proof of concept crawler in Python for “maven central” that slurped a list of all released artifacts into YAML documents and checked what it found into Git on GitHub. One document per group/artifact. Core Maven developer, Hervé Boutemy made a more scalable solution and deployed it in Apache infrastructure, checking its output into Git and publishing that on GitHub.

“Jetty All” is a fat jar for the well known Jetty web server. The YAML for it from Hervé’s script looks like:

%YAML 1.1
---
groupId: org.eclipse.jetty.aggregate
artifactId: jetty-all
versioning:
  latest: 11.0.0.beta1
  release: 11.0.0.beta1
  versions:
  - 7.0.0.M3
  - 7.0.0.M4
  - 7.0.0.RC0
  - 7.0.0.RC1
  - 7.0.0.RC2
  - 7.0.0.RC3
  - 7.0.0.RC4
  - 7.0.0.RC5
  - 7.0.0.RC6
  - 7.0.0.v20091005
  - 7.0.1.v20091125
  - 7.0.2.RC0
  - 7.0.2.v20100331
  - 7.1.0.RC0
  - 7.1.0.RC1
  - 7.1.0.v20100505
  - 7.1.1.v20100517
  - 7.1.2.v20100523
  - 7.1.3.v20100526
  - 7.1.4.v20100610
  - 7.1.5.v20100705
  - 7.1.6.v20100715
  - 7.2.0.RC0
  - 7.2.0.v20101020
  - 7.2.1.v20101111
  - 7.2.2.v20101205
  - 7.3.0.v20110203
  - 7.3.1.v20110307
  - 7.4.0.RC0
  - 7.4.0.v20110414
  - 7.4.1.v20110513
  - 7.4.2.v20110526
  - 7.4.3.v20110701
  - 7.4.4.v20110707
  - 7.4.5.v20110725
  - 7.5.0.RC0
  - 7.5.0.RC1
  - 7.5.0.RC2
  - 7.5.0.v20110901
  - 7.5.1.v20110908
  - 7.5.2.v20111006
  - 7.5.3.v20111011
  - 7.5.4.v20111024
  - 7.6.0.RC0
  - 7.6.0.RC1
  - 7.6.0.RC2
  - 7.6.0.RC3
  - 7.6.0.RC4
  - 7.6.0.RC5
  - 7.6.0.v20120127
  - 7.6.1.v20120215
  - 7.6.2.v20120308
  - 7.6.3.v20120416
  - 7.6.4.v20120524
  - 7.6.5.v20120716
  - 7.6.6.v20120903
  - 7.6.7.v20120910
  - 7.6.8.v20121106
  - 7.6.9.v20130131
  - 7.6.10.v20130312
  - 7.6.11.v20130520
  - 7.6.12.v20130726
  - 7.6.13.v20130916
  - 7.6.14.v20131031
  - 7.6.15.v20140411
  - 7.6.16.v20140903
  - 7.6.17.v20150415
  - 7.6.18.v20150929
  - 7.6.19.v20160209
  - 7.6.20.v20160902
  - 7.6.21.v20160908
  - 8.0.0.M0
  - 8.0.0.M1
  - 8.0.0.M2
  - 8.0.0.M3
  - 8.0.0.RC0
  - 8.0.0.v20110901
  - 8.0.1.v20110908
  - 8.0.2.v20111006
  - 8.0.3.v20111011
  - 8.0.4.v20111024
  - 8.1.0.RC0
  - 8.1.0.RC1
  - 8.1.0.RC2
  - 8.1.0.RC4
  - 8.1.0.RC5
  - 8.1.0.v20120127
  - 8.1.1.v20120215
  - 8.1.2.v20120308
  - 8.1.3.v20120416
  - 8.1.4.v20120524
  - 8.1.5.v20120716
  - 8.1.6.v20120903
  - 8.1.7.v20120910
  - 8.1.8.v20121106
  - 8.1.9.v20130131
  - 8.1.10.v20130312
  - 8.1.11.v20130520
  - 8.1.12.v20130726
  - 8.1.13.v20130916
  - 8.1.14.v20131031
  - 8.1.15.v20140411
  - 8.1.16.v20140903
  - 8.1.17.v20150415
  - 8.1.18.v20150929
  - 8.1.19.v20160209
  - 8.1.20.v20160902
  - 8.1.21.v20160908
  - 8.1.22.v20160922
  - 8.2.0.v20160908
  - 9.0.0.RC1
  - 9.0.0.RC2
  - 9.0.0.v20130308
  - 9.0.1.v20130408
  - 9.0.2.v20130417
  - 9.0.3.v20130506
  - 9.0.4.v20130625
  - 9.0.5.v20130815
  - 9.0.6.v20130930
  - 9.0.7.v20131107
  - 9.1.0.v20131115
  - 9.1.1.v20140108
  - 9.1.2.v20140210
  - 9.1.3.v20140225
  - 9.1.4.v20140401
  - 9.1.5.v20140505
  - 9.1.6.v20160112
  - 9.2.0.M0
  - 9.2.0.M1
  - 9.2.0.RC0
  - 9.2.0.v20140526
  - 9.2.1.v20140609
  - 9.2.2.v20140723
  - 9.2.3.v20140905
  - 9.2.4.v20141103
  - 9.2.5.v20141112
  - 9.2.6.v20141205
  - 9.2.7.v20150116
  - 9.2.8.v20150217
  - 9.2.9.v20150224
  - 9.2.10.v20150310
  - 9.2.11.M0
  - 9.2.11.v20150529
  - 9.2.12.M0
  - 9.2.12.v20150709
  - 9.2.13.v20150730
  - 9.2.14.v20151106
  - 9.2.15.v20160210
  - 9.2.16.v20160414
  - 9.2.17.v20160517
  - 9.2.18.v20160721
  - 9.2.19.v20160908
  - 9.2.20.v20161216
  - 9.2.21.v20170120
  - 9.2.22.v20170606
  - 9.2.23.v20171218
  - 9.2.24.v20180105
  - 9.2.25.v20180606
  - 9.2.26.v20180806
  - 9.2.27.v20190403
  - 9.2.28.v20190418
  - 9.2.29.v20191105
  - 9.2.30.v20200428
  - 9.3.0.M0
  - 9.3.0.M1
  - 9.3.0.M2
  - 9.3.0.RC0
  - 9.3.0.RC1
  - 9.3.0.v20150612
  - 9.3.1.v20150714
  - 9.3.2.v20150730
  - 9.3.3.v20150827
  - 9.3.4.RC0
  - 9.3.4.RC1
  - 9.3.4.v20151007
  - 9.3.5.v20151012
  - 9.3.6.v20151106
  - 9.3.7.RC0
  - 9.3.7.RC1
  - 9.3.7.v20160115
  - 9.3.8.v20160314
  - 9.3.9.M0
  - 9.3.9.M1
  - 9.3.9.v20160517
  - 9.3.10.M0
  - 9.3.10.v20160621
  - 9.3.11.M0
  - 9.3.11.v20160721
  - 9.3.12.v20160915
  - 9.3.13.M0
  - 9.3.13.v20161014
  - 9.3.14.v20161028
  - 9.3.15.v20161220
  - 9.3.16.v20170120
  - 9.3.17.RC0
  - 9.3.17.v20170317
  - 9.3.18.v20170406
  - 9.3.19.v20170502
  - 9.3.20.v20170531
  - 9.3.21.M0
  - 9.3.21.RC0
  - 9.3.21.v20170918
  - 9.3.22.v20171030
  - 9.3.23.v20180228
  - 9.3.24.v20180605
  - 9.3.25.v20180904
  - 9.3.26.v20190403
  - 9.3.27.v20190418
  - 9.3.28.v20191105
  - 9.3.29.v20201019
  - 9.4.0.M0
  - 9.4.0.M1
  - 9.4.0.RC0
  - 9.4.0.RC1
  - 9.4.0.RC2
  - 9.4.0.RC3
  - 9.4.0.v20161208
  - 9.4.0.v20180619
  - 9.4.1.v20170120
  - 9.4.1.v20180619
  - 9.4.2.v20170220
  - 9.4.2.v20180619
  - 9.4.3.v20170317
  - 9.4.3.v20180619
  - 9.4.4.v20170414
  - 9.4.4.v20180619
  - 9.4.5.v20170502
  - 9.4.5.v20180619
  - 9.4.6.v20170531
  - 9.4.6.v20180619
  - 9.4.7.RC0
  - 9.4.7.v20170914
  - 9.4.7.v20180619
  - 9.4.8.v20171121
  - 9.4.8.v20180619
  - 9.4.9.v20180320
  - 9.4.10.RC0
  - 9.4.10.RC1
  - 9.4.10.v20180503
  - 9.4.11.v20180605
  - 9.4.12.RC0
  - 9.4.12.RC1
  - 9.4.12.RC2
  - 9.4.12.v20180830
  - 9.4.13.v20181111
  - 9.4.14.v20181114
  - 9.4.15.v20190215
  - 9.4.16.v20190411
  - 9.4.17.v20190418
  - 9.4.18.v20190429
  - 9.4.19.v20190610
  - 9.4.20.v20190813
  - 9.4.21.v20190926
  - 9.4.22.v20191022
  - 9.4.23.v20191118
  - 9.4.24.v20191120
  - 9.4.25.v20191220
  - 9.4.26.v20200117
  - 9.4.27.v20200227
  - 9.4.28.v20200408
  - 9.4.29.v20200521
  - 9.4.30.v20200611
  - 9.4.31.v20200723
  - 9.4.32.v20200930
  - 9.4.33.v20201020
  - 9.4.34.v20201102
  - 9.4.35.v20201120
  - 10.0.0-alpha0
  - 10.0.0.alpha1
  - 10.0.0.alpha2
  - 10.0.0.beta0
  - 10.0.0.beta1
  - 11.0.0-alpha0
  - 11.0.0.beta1
  lastUpdated: 20201123164237

Well, it does today at least. Source: https://github.com/hboutemy/mcmm-yaml/blob/master/org/eclipse/jetty/aggregate/jetty-all.yaml

The order of the releases within is approximately the order of releases in real life. It is complicated though - did 9.4.7.RC0 come after or before 9.4.7.v20180619? The Jetty people know what that means for them, but the same question could be answered differently for other teams doing fine-grained versioning. We have tools like dependabot that could use this YAML meta-model info for upgrade tips. It seems to me that it’d be nice to have additional info. If there were an adjacent resource .supported-versions that had a list of resources the Jetty tea think are current, that’d be great:

9.4.*

The implication is that as a team you could depend on the latest of the 9.4.x series and be OK for bugs even if you were behind on features like “the servlet spec” (Jetty 10 adopted servlet spec v4 - see the wikipedia page for Jetty .

Maybe that also gets copied into the YAML document as it updates:

%YAML 1.1
---
groupId: org.eclipse.jetty.aggregate
artifactId: jetty-all
versioning:
  latest: 11.0.0.beta1
  release: 11.0.0.beta1
  versions:
  // snip
  - 9.4.15.v20190215
  // snip
  - 9.4.34.v20201102  
  - 9.4.35.v20201120 Supported
  // snip
  - 11.0.0.beta1 
  lastUpdated: 20201123164237

Note that 11.0.0.beta1 is marked as latest and release elsewere in the document.

Maybe vulnerabilities could also be indicated in the YAML doc:

%YAML 1.1
---
groupId: org.eclipse.jetty.aggregate
artifactId: jetty-all
versioning:
  latest: 11.0.0.beta1
  release: 11.0.0.beta1
  versions:
  // snip
  - 9.4.15.v20190215 Vulns: CVE-2019-10247
  // snip
  - 9.4.35.v20201120
  // snip
  - 11.0.0.beta1
  lastUpdated: 20201123164237

Perhaps vulnerabilities are similarly co-located with the YAML files, and their info is copied in as the update script runs. Rather than a database or any hot querying of cvedetails.com.

Big corporations would pull changes daily (git pull) and run their own scripts based on what was found. Crude internal dependabot functionality perhaps, but anything else that can be added after GAV creation (not known at the time of creation).

Hervé has been running the updates for it ever since: https://github.com/hboutemy/mcmm-yaml. Well, his bot is, as he has better things to do. The repo size is 85MB today. Individuals and companies could use it where Hervé has it, but it could disappear at any moment. What’d be great would be if it were adopted officially. Interesting too, that since launching this cron job, Hervé joined Sonatype the company that leads the management of Maven Central.

My obsolete prototype - https://github.com/paul-hammant/mcmm - only partially populated with GAVs and with an inferior format to Hervé’s YAML.

← Previous Archive Next →

Published

December 6^th, 2020

Reads:

Paul Hammant's Blog: Maven Central Meta Model

Published