Tools for Thought need a standard for data interoperability. It should be org-mode

What are Tools for Thought

Tools for Thought is a general term for software applications that help us capture, process, store, and retrieve all of the stuff we need to absorb in our information-overloaded lives. The Tools For Thought (#TfT) space is exploding, with new products becoming available every day. Journaling tools, task managers, outliners, and networked note-taking apps are all tools for thought. Second brain app is another term used for these kinds of tools.

Why do Tools for Thought need data interoperability

No one tool does it all. People need to be able to use multiple productivity and personal knowledge management tools (heres Paco’s opinion). People also want to share subsets of their second brain content with others. Tools in the space continually appear, evolve and disappear. For these and other reasons there should be a base level of interoperability between tools in the #TfT space.

Interoperability needs an understanding of the things we’re describing and the language or format we’re using to do so. For #TfTs those things include paragraphs of marked up text, outline hierarchies, dates, tasks, tags, lists, tables, pointers to external resources (eg bookmarks) and internal links/relationships. Formats include XML, JSON and just plain text.

Options for an interop format

Over time there have been many efforts to standardize how we exchange information, and information about information, (semantic web, XML/WSDL, RDF, OWL etc), but none successfully addressing the semantics required to describe the set of #TfT entities above.

Markdown was defined in 2004 as a human readable markup language, it has been adopted by a number of note taking and Zettlekasten tools such as Obsidian and Bear. It is an obvious option for an interop format. But it only covers a subset of the semantics and has been extended inconsistently across applications, so much so that there is now an effort (CommonMark) to re-standardize the variants. Markdown does not have good developer support and has other deficiencies as outlined by @kmelve here.

Dave Winer has advocated using OPML, a long-standing XML-based hierarchy/outline description. But again, small semantic footprint.

Most tools use proprietary formats and app-specific APIs, but there are efforts underway to define new models. Portable Text and atJSON are JSON specifications targeted at the space. There are also a number of other build-something-new proposals. xkcd N+1 standards

While there are plenty of interesting options, XML and JSON based formats all have the disadvantage of not being human readable, and anything new is going to take time to evolve. So is there something existing that fits the bill? Turns out there is, it’s Org-mode!

Org-mode for interop

Org-mode, developed in 2003, was “designed for notes, planning, and authoring” as a module inside the emacs text editor. It is both an application running inside emacs, controlled with keyboard commands, and a plain text markup language for describing content. It has broad adoption within the emacs community and has been continuously refined as a personal text-based #TfT by productivity and process savants for decades. In its current state org-mode syntax can model all the listed aspects needed for #TfT interop. In addition it is at least as human readable as markdown, see Voit.

There are an increasing number of tools using org-mode outside emacs. Org-mode has strong developer tool support, and there are a number of parsers available. Rather than reinvent the wheel, build something proprietary, or adopt an inferior solution, I recommend companies in the #TfT space look at adopting Org for interoperability between tools and, ideally, as a user-accessible data store.


org-mode for #TfT interop

Why don’t #TfTs already use org-mode

Well actually some do! Obviously the original org-mode/emacs combo qualifies as a #TfT, but in addition the popular LogSeq app reads and writes Org, as does BrainTool. That said, Org, living in the self-contained emacs ecosystem, has an enormously steep learning curve for the outsider and so has never been widely known. Plus, until recently the only definition of org syntax was embodied in the emacs lisp code running the Org application (as noted above this is no longer the case).

What would it look like to use org-mode for interop

From an end user perspective using Org for #TfT interop would look like having a repository of plain text files stored somewhere and giving various applications access to those files to perform their specific function. A ‘task’ created in my journaling app should then show up in my task manager. When I mark it complete in the task manager, it should also show as complete when I revisit my journal (perhaps with annotations as to when it was completed). See this demo for an example of the same tasks being usable across BrainTool, emacs/org and Logseq.

Org-mode covers a lot of ground and not every application needs all of those entity types. The minimal bar for an application to be ‘Org compliant’ is that it read and write plain text without screwing up any embedded Org markup. From this perspective any plain text editor is level zero compliant and could be used to edit Org-based #TfT content.

Karl Voit has proposed ‘OrgDown’ which is a model for defining increasing subsets of Org as conforming to different levels of compliance. Thus far only an OD-1 level is defined. Personally I would prefer to see a more granular enumerated list of OrgDown entity types for which applications can demonstrate support. See the addendum below for one take at such a list.

Have thoughts? Drop a comment on the https://twitter.com/tconfrey/status/1521934272617136131

Addendum: Proposed Org syntax components

OD-itemDescription
OD-0Read and write plain text preserving any markup.
OD-Outlines Nested outlines/headers with paragraph text.
OD-Markup Basic text markup (bold, italic etc).
OD-ListsAndCheckboxes Bulleted and numbered lists. Lists of checkbox items.
OD-EscapesAndComments Sections not subject to interpretation as markup. Meta comments not intended to be interpreted by an application.
OD-Tables Basic table formatting.
OD-Links 1) One-directional
2) Bidirectional.
OD-Tasks 1) Basic todo/done toggle
2) Configurable lifecycle.
OD-Tags Tag assignment to outline headers with inheritance along the outline hierarchy.
OD-Properties Name value pairs of meta-data properties associated with an outline or file. Useful for app-specific content.

Org-Mode tools outside Emacs

Org-Mode Parsers

29 Apr 2022 - Written by Tony

Share on: