A core ingredient of collaboration is a shared issue tracking environment. When I went to design issue tracking for Manganin, I already had some requirements in mind.
- The backing data for issues should be stored in Git, and in a way that is playing to the strengths of the tool, not fighting against it. Git a powerful tool to manage versioning, but it isn't the obvious choice of database. Bolting on Postgres would be the standard method, but introduces a whole class of vendor lock-in issues, introduces dependencies, and complicates deployment.
- Working with issues via the command line, and for periods without internet, needs to be ergonomic without installing additional tooling. This is what I mean when I talk about "local-first" as a core principle of Manganin. You should be able to access all core functionality on your own device, without internet access or complicated setup steps. The workflow should integrate seamlessly with proven tools that you already know (and have lovingly configured). An open source tool that has had the efforts of passionate engineers poured into it over decades is going to be better than a crappy electron app whipped up in a weekend to do the same thing with a proprietary tech stack.
I soon realized that having both of these while maintaining usability was more difficult than it seemed. This devlog explains my approach to issue tracking that meets these requirements, and documents some of my (many) failures along the way.
Issues alongside code
The naive solution is to just have a .issues/ directory in the project root, with each file within representing an issue. I thought this was neat because the issues were tied with the code—every branch has its own issue state, so you close an issue in the same commit as fixing it. But once I followed this benefit to its logical conclusion, I realized that it is also the fatal flaw. When a new issue is opened (on the main branch), to which active branches does it apply? Should you rebase constantly to keep up with new issues? Oh no-
Rebasing
Issue churn generally happens with a much higher frequency than actual code commits. Every time someone opens, edits, closes, reopens an issue, you have to pull. This would get old pretty fast once the project achieved any kind of velocity.
Special refs
Git stores tracking information in refs. Branches are stored in refs/heads, tags are typically enumerated in refs/tags, and so on. I learned about a trick at the Recurse Center: you can point to arbitrary data with unconventionally named refs to store data in Git, in a way that is completely opaque to (but still faithfully propagated by) repository hosts such as GitHub. You can put issue tracking information in here so that it doesn't clutter up the source tree but is still stored and cloned with the repo. Each issue is assigned an autoincrementing integer and put in a ref addressed by that index, such as refs/issues/12.
The problem is, this makes issues massively annoying to edit locally. Using this method for actual local development would probably require downloading a separate tool to manage this complexity. Git was not designed to be used in this manner, and it shows in the ergonomics. There are a lot of layers of complexity wrapping what is essentially just a small text file. More layers means more chances for things to go wrong and more unneeded redundancy of information—there are 4 different IDs that have to be created that essentially refer to a single issue.
Process to edit an issue (technical)
Git addresses stored objects with object IDs, or OIDs. We need to store some data in Git's database, then point to it with a ref so that it can be discovered by other commands or tools, and to prevent it from being garbage collected. First, we will use `git hash-object -w` to store some data in Git's database. This will give us the OID of the "blob", which is just some data. Then, we need to turn the blob into a tree, and the tree into a commit. Then, we point a ref to that commit by calling `git update-ref`. If this command isn't run, no refs actually point to the new objects we've made. That way, if a step before this fails and the process can't be completed, nothing actually changes, and all of the objects we've set up will eventually be GC'd. Atomicity!# Returns the OID of the newly written blob
git hash-object -w {new contents}
# The previous step gave us the OID of a "blob"
# If there's anything else in the tree, it will need to be re-added
echo "100644 blob {issue_oid}\tissue.txt\n" | git mktree
# Notice the similarity in this command to `git commit`
git commit-tree -m " " {tree_oid}
git update-ref refs/issues/{id} {commit_oid}
There are additional problems with this method, particularly surrounding avoiding conflicts and race conditions with other people editing the same issues. Implementing this method raised questions around what should be done about conflicts and invalid data, and what exactly an issue ID represents.
Keep it simple, stupid
I took a step back for a couple weeks to think. I had preconceived notions of what issue tracking should be, formed from working with existing software built for SQL backends. What would it look like if I forgot all of that, and tried to work with Git? The solution seems obvious in retrospect: just work in the way I want to work, and build a tool that facilitates that workflow.
Trying to force Git to work like a relational database results in massively overcomplicating things. If you're going to have Git track changes in something distinct from your source code, it should be stored distinctly. Instead of trying to force two disparate things to be stored together, why not just store them apart in the most ergonomic way, and use tooling to bridge the gap between them?
Manganin now uses a separate repo for storing issue data. Whenever a new repo is created, a hidden sister repo is also made for tracking issues. You don't see it in the list of repositories on the frontends. Instead, the data inside is parsed and presented in a manner similar to other forges: a list of issue titles and their bodies. The repo is cloneable from a special path, so not everyone who clones the codebase needs to download all of the issue tracking data.
The issues are simply stored transparently as files. Every part of the file is semantic: the issue title is the name of the file (you would be surprised by what constitutes a valid path). The contents of the file store the issue body. Workflows like this are ubiquitous, so Git, the filesystem, and other tools support it well. I use FZF in Neovim for my normal issue-browsing and triage workflow.
There's no need for an autoincrementing integer for issue IDs. The filesystem guarantees unique filenames. If the filename fails to be a sufficient identifier—I have yet to come across a use case that it doesn't satisfy—then the Git OID can be used.
A shorter letter
It takes time to refine a complex idea into a simple one. Finding the most elegant solution requires trial and error. The result should be a system that feels so obvious that it's impossible to perceive the effort that went into it. The ideal tool is one that stays out of your way so that you can do your best work.
The refinement process is iterative. As I continue to use Manganin for issue tracking, I note pain points. These are places where subtle tooling can smooth out the workflow. The key to making it better, and not more obtuse, is to start with what the user experience should be, then design architecture to facilitate it.