* It’s still early days; this project is a pre-alpha minimum-viable-product – features are missing, the query system needs optimising, etc. This document describes both where ion is, and also what it’s been designed for – any features that are still upcoming have been marked with an asterisk.
ion is a universal media processing pipeline. ion’s core purpose in life is to take an input folder and produce an indexed and optimised version of that folder, optionally publishing it, while responding to changes in the input by rebuilding only what has changed. ion can act as an invisible content management system, or a generator for websites, apps, ebooks, audiobooks, and perhaps, one day, even realbooks.
If you want to know more, then one advantage of ion being entirely self-contained is that it (the source below) can be pasted to a language model to ask it questions about ion, like: was ion written by a human or by a machine, what is the purpose of ion, what is its aim, where is it in its development, is it quickly becoming one of the greatest content management systems currently in existence, etc, etc. This is useful because ion is currently basically undocumented.
Intuitive content management; familiar document-oriented design that works across formats, from Markdown to Word, with an integrated graph query language. ion can watch or synchronise with virtually all types of storage; local, remote, and cloud. This means a user can manage their production-grade website solely from their Dropbox app, or iCloud, or Amazon S3, or just from their local drive.
Parallel build system; across cores and machines*, indexing, filtering, optimising, and publishing potentially massive and disparate collections of information, aiming eventually to support inputs the size of Wikipedia, and perhaps even the Common Crawl. Eventually, ion aims to be able to take snapshots of the web and consolidate them down into a single coherent form. Imagine the entire web, collapsed down into a single website, or offline app, even for low-power devices such as watches and audio players. For that, the build system would need to be able to act across clusters.
Generated ahead-of-time; optimising a single video properly requires deriving more than a hundred variations of that file, for the different formats, resolutions, and bitrates. For big inputs this can grow to be a massive cost, which is why ion has been designed to spend this cost only once, up-front, then only rebuilding what has changed, and recycling* everything else. ion does this for every input type, from documents, to images, videos, objects, maps, data, and even AI filters.
Optimising content server; ion focuses on generating everything ahead-of-time, but it is not solely static; ion also runs a server for displaying the optimised output as a website, while running a receiver for a rate-limited* and spam-filtered* inbox. At the same time, ion continuously watches the input for changes – when the file system supports it, or otherwise it spins at an interval – and then ion automatically rebuilds the output. A high-performance transceiver has been custom-written in C for this project, by hand, and it is currently being integrated - until then, a basic shell server is standing in for demo purposes. The server is C89, zero-allocation, and based on libuv; for those that know, this is the gold standard.
Standing on the shoulders of giants; ion made the odd choice of being a shell script. The advantage is that it sits on the ancient foundation of Unix, necessarily inheriting its philosophy: an old, boring, trusted, standardised, and widely available ground from which ion reaches out to the best-in-class for the task at hand, including pandoc, rclone, ffmpeg, vips, esbuild, parallel, and more. This gives it an unsurpassable feature-set, quality, and format support, with no vendor lock-in or separate data entry step; just a standard folder of files, optimised.
Scale, performance, and security; currently, generally, the generation phase is slow, and the production phase is fast – that is, slow to build, fast to run. It has been designed so that, in time, the performance limitations of its shell nature can be optimised away, namely with C. For security, ion is as static as possible, with a small runtime footprint and as few moving parts as possible. ion contains thorough, almost paranoid, type checking and internal validation, for an unusually rock-solid* shell script – perhaps to compensate for feelings of insecurity regarding its shell script nature. ion is only superficially a shell script though – the shell layer negotiates with the host, while acting as a self-extracting archive for various Lua filters and C scripts.
Portable, offline-first, and extensible; ion’s output is fully self-contained, enabling it to work with or without a connection, with full support for third-party development, including JS/CSS document components, Lua document filters, and build scripts. Soon, ion should get the ability to monitor its host: a lone hypervisor, able to drop down on nearly any machine, adapting to the environment, making use of what’s available – from an old mac to a modern data-centre – responding to changes from the user, and keeping them updated with status reports, all through the file system.
International and multilingual; built from the ground-up with translations in mind: document translations can be linked*, directions are considered, every word and symbol can be translated, and even the query language’s word order can be altered, from subject-verb-object to verb-subject-object, for example.
Open-source, mostly. Currently, commercial use will usually require a licence.
This project began as the scaffolding for a nutrition calculator, with the goal of balancing the often conflicting ideals of health and resource usage. Initially, the goal was to be able to reverse engineer Huel (sorry Huel) by importing a large amount of nutrition data, particular the USDA FoodData, using a constraint solver to find the inputs, and then using the imported statistics to determine its health charistics as well as its costs economically and ecologically, particularly land use, for both individuals and for large groups of people.
This turns out to be an interesting challenge – the answer of how much land does something like Huel use depends on your personal requirements mixed with the capabilities of the land in question – the answer is different for an Indian woman that wants to eat organically than it is for a British man that wants to eat cheaply; the personal requirements, land yields, and ecological impacts vary significantly; the resource usage changes completely across supply chains (or trees, or graphs, properly speaking), and so the complexity of this question spirals out of control very quickly. A human diet can be modelled as a supply graph, but the statistics needed to answer these questions lie in many varied sources, from government nutrition databases, to information buried in the tables of messy and obscure documents. This was the crucible that designed ion. Achieving this goal ended up requiring a new approach to content management, and so ion was born.
With this as the foundation stone, ion became a fairly capable data hub, or node. This was the first stage. These nodes are designed to connect with others, to share information as a mesh network. This is to enable a broader goal – beyond consolidating the web and making it work offline, the ultimate goal of ion is to sit as the foundation for an open federation of interconnected franchises.
The view is that the world needs a safety net. As a thought experiment, imagine an empire, like a fast-food franchise, except it’s an open, decentralised network of non-profit, non-loss, nutritionally-balanced restaurants and hotels that receive and redistribute the fundamentals. This is the primary reference project.
At the heart of a restaurant is the menu, and the ticketing system; a menu is an inventory; a ticket is a subset of that inventory; a recipe, and even ingredients are inventories; porridge is a collection of oats; oats are collections of nutrients; proteins are collections of amino acids; blogs are collections of articles; invoices are collections of products and services; everything is an inventory.
A network of nutritionally-balanced kitchens seems then to be an accountancy problem, and this is where ion aims to sit, as the scaffolding that supports the inventory component; a node that indexes, balances, and connects inventory components together, so that they may form, in the real world, a decentralised safety net.
Until then, it also makes nice websites (coming soon), and you can watch its development from here.