I’m Quinn, and I’m one of the researchers who’s been working on RhizomeDB.
In the spirit of trying to work in the open, I want to start tracking some brief weekly notes about what I’ve been doing with the project. Feel free to chime in if any of this sounds interesting to you: the implementation is still early, so the repo is private for now, but we’re happy to open up some of what we’re doing to interested collaborators!
December 12 - 23:
Started work on the Rust implementation of Rhizome!
Implemented the lattice and timestamp types needed for incremental view maintenance
Implemented a simple semi-naive relational algebra machine, based on Souffle’s design
Replaced Ustr with string-interner, due to a bug impacting the wasm32-unknown-unknown target
Currently working on supporting arbitrary query sinks and sources, for connecting long-running queries to the outside world (ingesting data from IPFS, pushing results to Javascript, etc)
Hi @Quinn this is super interesting. I have a similar background. I started working on LSD (Leapsight Semantic Dataspace (Video here) in 2010 and had a first working version by 2013. RDF model with bi-temporal (validity) and probabilistic extensions (veracity) using Leaplog (Datalog w/extensions using top-down evaluation and support for distributed queries) all implemented in Erlang on top of Riak Core. My next step was to implement a bottom-up evaluation based on Timely/Differential Dataflow (at that point in time the reference implementation was Naiad off course).
Although I have not worked on LSD since 2016 this is a pending chapter which I hope to resume soon. But pretty much interested in what you are doing and would love to know more.
January 9 - 13:
This week was largely spent exploring the async ecosystem in Rust, and what bridging that with Javascript in the browser looks like, using WebAssembly.
I did extend the VM to support registering sources and sinks, implemented using futures::stream::Stream and futures::sink::Sink. Theoretically I should be able to stream facts into and out of the system over anything that implements those traits, including types from Tokio that implement AsyncWrite or AsyncRead, like sockets or files, but I haven’t spent too much time experimenting with some of those options.
The other big focus of mine right now is tying things together into a cross-target async runtime, and I’ll likely be taking a ton of inspiration from Prokio for that. I’m keeping things simple to start though, and just breaking the project up into three crates: rhizome-core, rhizome-wasm, and rhizome-tokio, so that I’m not spending too much effort worrying about conditional compilation or feature detection.
Beyond that, I’ve extended the Datalog parser + compiler to support specifying input schemas, which are then used for validation and scheduling sources.
January 30 - February 3:
I’ve been traveling for the last three weeks, so I haven’t been great about posting updates, but there’s been a lot of exciting progress!
I wrapped Rhizome in a cross-target asynchronous runtime, that supports running async tasks under both WebAssembly, using single-threaded promises, and natively, using multiple threads.
Using that, I wrote a runtime for the query engine that handles registering sources and sinks, and handling incoming + outgoing facts over them. That gave me a good foundation to spend some time experimenting with how the system actually gets used.
The outcome of that experimentation is examples of querying over generic async iterators in Javascript, JSON documents, websockets, and IPFS, using WebNative. I ended up tying a lot of that together into a small Rhizome implementation of a browser based todo app, that uses IPFS for persistence for and websockets for realtime collaboration:
I need to spend some time cleaning up a lot of that code and verifying the assumptions I made in putting together those examples, so I’ll likely be spending at least early next week doing a lot of testing and refactoring.
