Paul Hammant's Blog: Joe Walnes On Successful Open Source

Joe on Github :
And on Twitter :

I’ve been telling a story for years to colleagues and friends who want to make open source. Joe Walnes had a particular strategy for making open source successful, with the minimal personal effort. He’s not lazy of course, he just wants to start early with a “pull in worthy collaborators” strategy. I figured it was time to ask Joe how well my mythic story stacks up to reality. I’ve also asked Joe’s collaborators on the open source pieces he started. Everyone here was interviewed in the last couple of weeks.

Anyway, this article is a bit long, as it contains case studies and quotes from others. Only read it if you’re intrigued about Joe’s way. Or perhaps you have something that you want to me more prominent, and could do with some pointers. Note that there are other ways to make successful or popular open source too. I’m just focussing on Joe’s way here.

Joe’s Modus Operandi

1. Secretly scratching that itch

  1. Work on itch in secret
  2. Flesh out what that looks like
  3. Fail & Start over are all good
  4. TDD is often a reality for when your progress beyond “spikes”

TDD will give you a dozen epiphanies about your solution, and lead to a better codebase in the end, but some of the tests that Joe wrote historically are more like integration tests than unit ones: see later. These days Joe is in two minds about the efficacy of test extremism.

Sometimes new OSS project propositions carry an “unbelievable” reaction. Joe’s solution is to not propose them, and instead develop them privately. At least, up to a point ..

2. Presenting it to busy peers

  1. Hey “take a look at this”
  2. “No it only took a few hours at the weekend” (optional small lie)
  3. “yes you’re right, we could use this in our current project”

The selling should be implicit at this stage: Tread softly and let the person you’re speaking to have the epiphanies. Sure, you may have had those thoughts already, but their enthusiasm is so much better when they have the thoughts. Slide past the fact that they were initially ambivalent, or critical :)

3. Getting others involved

  1. Release whenever you can. Especially after a bug fix
  2. (related) Try not to delay a release because you have a backlog
  3. Always leave unimplemented features for others to be attracted to
  4. Try to be bug-free though
  5. Make it easy for others to build from scratch after a checkout (have a solid README)
  6. Have tests as worthy fragments of example code - these are sometimes integration tests
  7. Hope that the potential collaborators will clone a working test to start their feature
  8. Implore potential collaborators to contribute patches (pull requests) that are implementation and tests
  9. Rewrite/Refactor as you speedily consume patches
  10. Suggest the collaborator joins the developer mail list

A big mistake, Joe feels, would be to try to own everything on a roadmap. That ends up being off-putting to potential collaborators.

4. Handing it to new owners

Joe’s libraries tend towards stability, and thus appear to get less and less commits and releases over time.

At some point, it seems natural to have someone else take over the management of the project, and Joe take back seat. Hopefully they are already interested and active on the mail-list.

Joe’s Maxims

  • Small libraries are beautiful.
  • Do one thing well, etc.

Case studies

QDox

On the project site it says “QDox is a high speed, small footprint parser for extracting class/interface/method definitions from source files complete with JavaDoc @tags. It is designed to be used by active code generators or documentation tools”. It does more than that really - it gives you an object model of all source bar method bodies.

QDox has lots of integration tests that masquerade as unit tests in that they execute in less that 1ms. Here’s a good integration-style test from MethodsTest

public void testSupportsJava5VarArgsParameter() {
        JavaMethod javaMethod = buildMethod("void doStuff(AThing param1, BThing... param2);");

        JavaParameter standardParam = javaMethod.getParameterByName("param1");
        JavaParameter varArgsParam = javaMethod.getParameterByName("param2");

        assertFalse("param1 should NOT be var args", standardParam.isVarArgs());
        assertTrue("param2 should be var args", varArgsParam.isVarArgs());
    }

When QDox came out it was subtly competing with XDoclet. That was very much the gorilla in the room for extracting “doclet tags”. We can barely remember now, but there was a huge community effort to avoid the molasses-like nature of Sun’s EJB 2.0, and Doclet tags were part of that. Aslak Hellesoy had just joined ThoughtWorks, and Joe had to present this little library to him given it’s disruptiveness for XDoclet’s existing code.

Aslak Hellesøy’s thoughts on using Qdox inside XDoclet (back in 2004):

“My first thoughts were: Wow this thing is really fast. When I looked at the code I was amazed how small and simple it was.

I felt a little stupid for having written something similar (xjavadoc) that was a lot slower and a lot more code, but I was happy to come across QDox because I could learn something from it. It didn’t take me long to decide to ditch my own code and improve Joe’s instead. IIRC I made some improvements to the parser and some to the API. It was pretty easy to do since the structure Joe had put in place was easy to work with.

Note: Aslak is more famous for Cucumber these days - and has a company around it.

Since 2010 Robert Scholte leads QDox development presently. He comments:

I discovered QDox due to an issue with the gwt-maven-plugin, which I had to use to generate the async-interfaces. These interfaces were missing information when generics were used. The root cause was QDox: although it could parse sources with generics, that information was simply ignored. This looked like a really interesting issue, so I took the challenge to fix it.

Instead of starting with a simple fix I chose one of the hardest open issues. However, the code was that good to read that it took me a relatively small amount of time to implement it. I enjoyed the work a lot and it felt better to donate it back rather than maintaining my own fork. I knew I could make a lot of developers happy with it, so why keep it to myself? I uploaded my patch just a couple of days after the release of QDox 1.7, which was the first release after 1.5 years. I didn’t want to wait that long, so I had a look at the open issues and recognized several of them. So my plan became: drive that QDox-team crazy by overloading them with a lot of patches. Instead I retrieved an invitation to join to team and apply the patches myself.

QDox-1.x is already good, but it could still be better. To make it easier to use I wanted to hide most of the business logic methods by using interfaces. Sure, QDox would become a little bit bigger, but the definition of ‘lightweight’ has changed a bit between 2005 and 2013.

Even though the architecture has changed, a lot of Joe’s original code could still be used. Together with this huge change QDox-2.x came with JDK7 support, classloader order control and other improvements and new features. The first milestone has been released in august 2013 and already seem to be very stable.

It was said that Annotations would make QDox useless, but the project is still active. For instance it is a perfect fit within the Apache Maven ecosystem, where you can generate and enrich your code as part of the build based on the Java-sources(!).

The time it takes me to maintain this project is right now quite low due to the very low number of open issues. This could mean 2 things: Either QDox is never used or it does what it should do. And I’m positive it’s not the first. Let’s say that Joe did a great job here.

XStream

On the project site it says “XStream is a simple library to serialize objects to XML and back again”, which is about right. Joe started it in 2003.

Like QDox, XStream has lots of integration tests that masquerade as unit tests. It is by copying these, that I would make my own contributions to XStream - it suits my top-down thinking and allows me to be guided by tests still.

Here’s one test from TreeMapAndTreeSetTest that shows the simplicity and easy-for-newbies nature of integration tests the Joe’s way:

public void testTreeMapWithComparator() {
        TreeMap map = new TreeMap(new MyComparator());
        map.put("benny", "hill");
        map.put("joe", "walnes");

        String expected = "" +
                "<tree-map>\n" +
                "  <comparator class=\"my-comparator\">\n" +
                "    <something>stuff</something>\n" +
                "  </comparator>\n" +
                "  <entry>\n" +
                "    <string>benny</string>\n" +
                "    <string>hill</string>\n" +
                "  </entry>\n" +
                "  <entry>\n" +
                "    <string>joe</string>\n" +
                "    <string>walnes</string>\n" +
                "  </entry>\n" +
                "</tree-map>";

        TreeMap result = (TreeMap) assertBothWays(map, expected);
        assertEquals(MyComparator.class, result.comparator().getClass());
    }

Joe made XStream in private and presented it during the working day at the ThoughtWorks energy-trader client we were all working at (2003). He said he’d made it in a few hours on one weekend (a lie!). Up to that moment in time there were only SAX and DOM parsers for XML in Java-land, and the idea alone to make a reflection-esqe serializer/deserializer might have been peer rejected. Thus Joe made it to the point of demo, instead of talking it up as vaporware.

Since 2006 Jörg Schaible has very professionally run the XStream project, and the only observed problem being vacations that Jörg takes away from the computer, when others step up and monitor the mail lists at least.

Jörg’s thoughts:

The first time I looked at XStream was because it was used for XML configuration support in NanoContainer (part of PicoContainer 1.x). I worked with other Java/XML libraries before, but XStream’s ease of use and clean structure made it compelling. Soon I used it in other projects, too.

When I started to report issues and provide patches, Joe asked me quite immediately to become committer, also because of some reputation gained with other Codehaus projects.

Over the next months I helped to implement some major features and fixed a lot of corner cases. Nevertheless I was really surprised by Joe when he proposed me as new leader of the project 15 months later.

XStream benefits still from Joe’s original design proposing a lot of extension points and most of them are used heavily. However, it seems the niche of it is narrowing, looking at the number of postings in the mailing lists and other committers going silent. With JAXB support in the Java runtime and XML often replaced by more lightweight formats, it’s importance is dwindling.

Websocketd

Joe’s WebSocketd is thing, that he feels could have a long life. WebSocketd is:

It is not a library, it’s a program that allows you to build websocket servers in any programming language. It takes STDOUT output from command line application (that’s where the ‘any language’ comes in), and essentially streams it out over websockets to consumers.

  • Github: 3000+ stars, 150+ forks
  • No mailing list, for now. Questions/requests get opened as tickets, which has more of a workflow than a mail-list
  • Strong emphasis on doing one thing well and staying relatively small
  • Implemented in Go but end users don’t need to know it, in the same way mysql users don’t need to know C
  • Applies old techniques (e,g CGI, unix ‘pipes’ philosophy) to a modern web world (e.g. single page apps with streaming websockets)
  • There is an inherent design tradeoff: ease of use, simplicity and decoupling over high scalability
  • As of May 2014, it’s the OpenSource dog food Joe eat the most (he says), although he could be unwittingly using XStream a lot in other tools.

Alex Sergeyev is taking over leadership of Websocketd, and says:

I have many projects that I “star” on github. I starred websocketd after this tweet.

Since I had some experience with websocket apps before, I did not think it’ll be long until I’ll try it. One of our apps that uses websocket protocol is featured in this post by Tom Daly and I: On The Journey For The Perfect Network Health Dashboard.

Anyway… I looked at repo again in February, realized that there are features I did not notice before (CGI and static HTML) …. and, instead of using it for websocket, I got to use it for little CGI webserver that I need to control minimally (I use it for boot-time (iPXE) configuring small experimental rack of servers.

My use case (being a weird one) run into some troubles and I started to change few things that I needed the most :) Then I realized that I like it and can do more. I started to look at the issue list.

Then (in April) I finally got to plug it into one of the websocket apps. My experience was positive as I expected, much less code to reason about, easy to benchmark and so on.

I asked, “What were your first impressions of Websocketd”?, and Alex responded:

I liked the idea, I checked Joe’s profile and I starred it :.

How did you get started with contributing to Websocketd, as opposed to just using it?

We have good open source-friendly culture at work, you are expected to communicate to open source authors and give them feedback. This is bottom line what helps. Patches are always hard to find time to (even when O.K. by the culture) but giving someone friendly feedback is crucial. I was always in the area of giving such advice and sharing ideas of implementation with maintainers, but since I’m big fan of golang and coding is not what I do much lately, I really wanted to practice and that’s what I’m doing with websocketd now.

Do you have any pleasure or displeasure now you’re leading the team?

Participating in 3000 star project is definitely a new feeling :)

I’m somewhat surprised how passive users are but it’s not qualifies for displeasure.

Do you have any other thoughts, re “the Joe way”?

I agree with Joe that the project could have interesting future. It’s dead simple and therefore very easy to adopt. Many of my friends also interested. They do not have use for it right now but IMO this is how things get wider adoption. Knowing and liking is the first step.

Lastly, the catchy byline from the main site “Like inetd, but for WebSockets. Turn any application that uses STDIO/STDOUT into a WebSocket server.” Was that one of the things that help differentiate Websocketd for you ? Are tag-lines important?

It’s a good tagline. I like inetd parallels but I don’t know if it added attention or not, those things are subconscious for all of us :.

Update (Jan, 2019): Infamy via http://n-gate.com/hackernews/2019/01/31/ ->

Sitemesh

Sitemesh is verging on a disproof of this theory, as it happens. It remains very useful, but has no lead.

Sitemesh has been one of the hushed legend pieces of Java web programming since the end of the 90’s. This is the longest lasting piece of the OpenSymphony suite that bought Mike Cannon Brookes, Scott Farquhar (co-owners of Atlassian), Pat Lightbody, Hani Suleiman (the Bile Blog), Ara Abrahamian, and Joe together. Pat went on to steer the sibling ‘WebWork’ piece to be Struts2 at Apache. It was the other lasting piece from OpenSymphony, but Pat has also had two startups in the Selenium space, that he’s gone on to sell. Namely BrowserMob and HostedQA.

Anyway, SiteMesh is a Servlet Filter that decorates arbitrary pages in a pipeline way. For most users, it takes a base page, and adds a header, footer and a left navigation. It does so without anyone hand-snipping HTML for the sake of server side includes (which predated it). Nowadays things like Python’s Flask go so much further.

Joe Says:

“SiteMesh is a counter example. This is a project I’ve no longer got any time or motivation to work on. I’ve made many attempts to hand it over (and I’m currently attempting it again) but it’s just never quite worked and the user community has suffered. Possible reasons: complicated code, very old system (15 years!), poor docs and website, annoying to test on many old containers, lack of documented vision, v2 vs v3 fragmentation, general lack of love makes it an unappealing project.

Perhaps Sitemesh is less of a library and more of a framework for people to feel they can’t cut in on it’s dance, so to speak. As a consequence, there’s no post-Joe leader for the project. The technology is still hugely successful, and used in new projects every day, so maybe it’s complete enough as is.

Conclusion

Joe makes demos of ideas, instead of talking then up (vaporware), he does the work quietly. He then demos it, and allows others to extrapolate a larger/longer life for the piece.

Updates

May 28: Add QDox code snippet.

Published

May 26th, 2014
Reads:

Tags