From [The] ‘Basics of the Unix Philosophy’ , Doug Micilroy is credited with saying:

“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”

I think that the “text strings” sentence should be changed to:

“Write programs to handle object streams, but fallback to text streams when object steams can’t be passed to the receiving program, because that is lowest common denominator.”

Transit & msgpack

Cognitect who are general experts with Datomic and Clojure blogged about a Transit which is a “format and set of libraries for conveying values between applications written in different programming languages”. It comes with examples and has a number of key benefits that are bullet pointed at the top of the entry. There are other technologies in the same space (Protobufs, Thrift, Cap’n Proto, Avro, Sereal), but I like this one for now. Msgpack is the format.

I think this type of thing could work well in a Unix pipe setting.

An example

Today, to shows files in the current directory touched in July, and sort them by size descending, you would do:

ls -l | grep "Jul" | sort -nr -k5

Size is the fifth ‘field’. Field is a loose concept of course, as this is all strings. With a pervasive and easy to link to library for transforms to a structure, a future alternative could be:

ls -l | grep month:July | sort size:desc

Fallback

You’d want to fall back to text, if the receiver definitely could not handle transit/msgpack. Unix command shells already optionally allow “ANSI colors” quite well. For example ‘git diff’ will give ANSI colored output, but when you pipe to a text file, the colors are gone unless you use Aha to make colored HTML output (obviously no longer plain text). Aha declares to the terminal that it can process ANSI Select Graphic Rendition (SGR) codes within text streams.

Why couldn’t objects and object arrays be handled the same way?

Who’s going to do the work?

Who’s going to do the work upgrading unix commands to have that optional (or preferred) output? There is no way to short-cut all unix command maintainers from having to at least consume patches. It would take a few tight examples, and the ball would start rolling rolling.

One Msgpack/Transit specific command that could be made would be an equivalent to Sponge to operate pipewise on a file in-situ. See the Sponge notes to get why it’s needed.

Another command would transit/msgpack via sys-in and pass on plain text via sys-out. It’d allow some msgpack-only commands disobeying unix tenants to co-exist in a world where test is still needed for inputs. Of course upgraded terminals themselves might do some special presentation for arrays or objects by default.

Why do this?

Because of PowerShell on Windows, which kicks ass. It presently allows piping of objects/arrays of objects between execution nodes. This is not even a new feature, and we want that in Unix-land too.

Spending 2 mins in Clojure’s REPL

The Clojure REPL would be a good place to prototype the ideas, seeing as it is romantically close to Cognitect’s roots. It is not a terminal, but is a good surrogate for one while playing with ideas and having cheap implementations.

user=> ls -l | grep "Jul" | sort -nr -k5

CompilerException java.lang.RuntimeException: Unable to resolve symbol: ls in this context, compiling:(NO_SOURCE_PATH:0:0) 
CompilerException java.lang.RuntimeException: Unable to resolve symbol: -l in this context, compiling:(NO_SOURCE_PATH:0:0) 
CompilerException java.lang.RuntimeException: Unable to resolve symbol: | in this context, compiling:(NO_SOURCE_PATH:0:0) 
CompilerException java.lang.RuntimeException: Unable to resolve symbol: grep in this context, compiling:(NO_SOURCE_PATH:0:0) 
"Jul"
CompilerException java.lang.RuntimeException: Unable to resolve symbol: | in this context, compiling:(NO_SOURCE_PATH:0:0) 
#<core$sort clojure.core$sort@610773bd>
CompilerException java.lang.RuntimeException: Unable to resolve symbol: -nr in this context, compiling:(NO_SOURCE_PATH:0:0) 
CompilerException java.lang.RuntimeException: Unable to resolve symbol: -k5 in this context, compiling:(NO_SOURCE_PATH:0:0)

OK, this is going to be more than two minute’s work.

Task #1, the REPL is going to need to understand the pipe char in this context…

Object arrays are not streams

That’s true, but the notes for Transit suggests that streams are doable too:

… There is no enclosing element required at the top level. Thus Transit is suitable for streaming and interactive applications. A use of transit might be a stream or file containing a series of elements …