In my last post, I crowed a lot about how ready I was to start working on DJDNS, and things were finally lined up for a quick path to a working proof of concept. Actually starting work on the project was a bit of a reality check, so I’d like to document the challenges I’m running into, as much to organize my own thoughts as to be open about the development process (and no, I don’t count simply using Github and its issue tracker and keeping the public repo up-to-date as “enough openness” in isolation).
The primary issue is that I did a lot of formalization work regarding EJTP identities as part of the DEJE project, instead of EJTP itself, originally thinking that the code would be DEJE-specific. It ended up being very generic and valuable, establishing a standard for storing identity information for use in EJTP applications. So the biggest thing in the way is that I have to migrate the code out of DEJE, into EJTP.
This isn’t necessarily easy. Identities are fairly well hooked into DEJE, and not at all in EJTP. The churn on both sides will be pretty big, especially secondary goals. For example, identities being a standard part of EJTP means writing a few helper scripts for managing idcache files. On the DEJE side, there’s already lots of problems of ambiguity regarding what’s accepted where - some functions take string nicknames, others only accept Identity-class instances, and some have extra logic to handle both. It’s not even documented, really, what accepts what - and that’s icky and insane. So part of the cleanup will involve making definite, documented choices of where to take nicknames, and where to take objects.
Much less necessary, and part of the ultimate goals of the 1.0 release, but I suspect any serious usage is going to run into the limits of the simplistic routing mechanisms used to determine where a message ought to be delivered to/through. Most use cases will likely be able to be patched over with a BounceJack, which tells the router that it’s a given jack type (say, email), but forwards all traffic through another jack object (say, https). So when the router needs to send something it doesn’t have a jack for, you can initialize a
BounceJack("udp4", my_https_jack) and fold the traffic through any medium as you see fit.
Ultimately, though, that’s a hacky patch, and what we really need is a more thought-out and powerful model inspired by Linux’s packet routing mechanisms (or at least, the nice parts). And that’s going to take some time to work out how I want to do it.
Develop a standard way of loading documents from scratch
One of the sillier things that made it into the current design of the code (and believe me, my work always has a plethora of terrible ideas that are abandoned anywhere from 0-70% of their implementation being written), was the idea that third-party projects using DEJE would roll their own mechanisms to agree on the initial state of a document and provide it to others. Once you have a starting point, DEJE can guarantee your safety from there on out. Yay.
Except that it’s retarded to put that responsibility on third parties, as I immediately realized when I started writing a third party application. I found myself wistfully looking at my call to an imaginary Owner.load_document function, wishing it existed, and realizing that writing one was basically a necessity. That doesn’t make it easy, but it’s a lot easier and more logical to tackle a hard thing once now, than a thousand times down the road.
So far, my thinking is to use a hash-based system. I’m already identifying documents by arbitrary names, and was getting into the creeping problems of needing more and more data as part of the name - arbitrary aesthetic component plus reliable bootstrapping source plus this plus that - and using hashes just makes sense. You can then pull the initial version out of a DHT, with as much rich metadata as you want as part of the document itself. You have your starting ground that you can then update based on queries of the listed participants.
This does run into the problem of participant drift - people who were participants at the time the snapshot was put on the DHT, may not be participants now, and may be malicious. Being able to forge consensus and trick your client down the wrong path is really unlikely, so you’re pretty much safe from forged information (even if you’re knocking on the bad guy’s door looking for your cup of sugar), but that doesn’t mean anyone you ask will provide you with new, accurate updates.
So an important principle is that much like a car, or ancient chinese teapots, some things have to be maintained through a bit of regular exercise. Every once in awhile, you ought to freshen up your documents, make sure they’re all up to date, so you don’t get left behind in drift. Documents designed for high availability will probably never experience this problem, but small documents being completely handed off to a different set of maintainers (and the old maintainers not hosting the bridge blocks that made it happen) can leave a person high and dry, and in need of resolving the situation manually by finding a more recent DHT snapshot or a block archive through alternative means, like a forum or IRC chat. The block archive is preferable since it’s difficult to forge and can guarantee the accuracy of the change of ownership step by step - always nice security guarantees.
I ran into all the above troubles almost immediately when I started working on DJDNS, during a car ride back from a vacation with friends and family. I expected to encounter bugs and deficiencies in the relatively untested and immature foundation libraries, but it surprised me how quickly I ran into brick walls. And I will almost certainly run into more once I get these things straightened out, which is especially exhausting to think about when I consider that I have pretty much the whole thing still to write, and have barely written a line of open source code since closing my laptop lid to save power around 6 hours into the aforementioned long ride home.
The good news is, while I will probably have to do most of the stuff I’ve talked about here on my own, being that I have specific intentions for how I want them done, most of the complications I run into from there can probably be solved by other people, being paid to do so.
I freaking love this site, although it’s still very glitchy and not so popular yet. I’m planning to get lots of exposure and introduce other developers to my code, by funding parts of development that I don’t feel like doing myself. That means there are other people familiar with the way this stuff works on the inside, I can be lazy sometimes without breaking the entire progress of an important project (as the sole maintainer of many personal projects, this is a familiar burden already), I’ll be financially supporting freelance FOSS, and I’ll probably get a bit of free maintenance out of it as well, or at least respect and publicity.
Whenever I work on the DJDNS stack, I am constantly looking for opportunities to outsource my first ticket to freelance FOSS coders. I’ve mostly been held back by the immaturity of the libraries. But EJTP, thanks to its bounty of long-term planned enhancements and relative stability (even if that’s analogous to being the thinnest person on Day 1 of The Biggest Loser), has a lot of potential for this, and I’d like to post bounties for some of the simpler changes that I simply haven’t got around to doing. Such as throwing an exception instead of silently overwriting jacks, or gzip message compression support, which can developed fairly orthogonal to any priority or architectural work I need to do myself.
It’s very exciting to be contributing this way for the first time, and I hope that others find it a positive experience as well, but if not, I’m open to any criticism that makes next time better.
Oh, and the video…
I plan to do a screencast via Hangouts on Air as soon as DJDNS is a semi-working installable thing, so that everyone can help test it. I may or may not jump on that sooner for the more fundamental libraries, depending on how much difficulty people end up having contributing. So if you want to help develop but run into problems getting things set up as-is, let me know via email or reddit, since that’s what’s going to inform when I do my screencast, and what I’ll focus on.