At presentations related to archives or archiving, there is the inevitable moment when “the repository diagram” will be shown — cylindrical “silo’s” of data and meta-data servers woven by the thread-like arrows of various protocols into a fabric of services and applications. The (implicit) message of showing the diagrams is something like “ok the system is a bit complex, but at least it’s modular.”
Image search for “repository”, result #1:
But modularity is not enough, and can in fact form chains of dependence within a system. A system, no matter how “light” and compartmentalized, becomes in effect heavy and monolithic when the divisions are essentially dependent on each other. As with the Titanic, despite the redudancies and compartmentalization, when the boat sinks, everything goes down together.
In contrast, a system that is both modular, and flat — in the sense that the various components can operate independently, perhaps in radically different situtations or kinds of use, exhibits a powerful kind of “lightweight” modularity, and one that fits the usage model of the pipeline.
In computer programming, “robustness” is taught as a strategy in relation to modularization: make modules perform even better that is defined by a requirement specification; given “erroneous” or “malformed” data, a robust module should exhibit a “graceful degradation”, and still perform in a tolerable way thus not decreasing the overall system’s stability.
The pipeline goes a step further from classical robustness to a kind of promiscous robustness. Modules are small independently-usable tools, whose interfaces are defined broadly to the specification of very specific use cases, and typically use an “interface” of (human) legible text, and often supporting flexibility of accepted formatting grounded in an understanding of the practices of writing. Perhaps counter-intuitively, defining a tool to a very specific task often opens up a kind of splendid envelope of abstraction that actually enables the tool to be used in contexts that do not need to be pre-defined or “designed for” from the start. In this way the pipeline tool opens itself up to unexpected uses.