Geoffrey Litt (PhD student at MIT) @geoffreylitt and Peter van Hardenberg (researcher at Ink and Switch) @pvh will be presenting about Cambria.
Cambria is an experimental Typescript library for effectively managing schema change in distributed systems. It aims to allow developers to express relationships between schemas using bidirectional lenses, and to avoid mixing compatibility code into application logic.
The Project Cambria article https://inkandswitch.github.io/cambria/ is already here in the forum and is important back ground reading:
Plus other ink-and-switch work.
Video
Slides on Pitch:
Resources
Github: https://github.com/inkandswitch/cambria
Local links to crdt related content
Via @maparent, Marc Shapiro, Nuno PreguiƧa, Carlos Baquero, Marek Zawirski. Conflict-free Replicated Data Types. [Research Report] RR-7687, 2011, pp.18. ffinria-00609399v1f https://hal.inria.fr/inria-00609399v1/document
Chat Log
| Time | User | Message |
|---|---|---|
| 00:35:53 | Marc-Antoine Parent: | asymptomatically typeless  |
| 00:36:20 | Marc-Antoine Parent: | asymptotically, sorry |
| 00:37:56 | Jess Martin: | reminds me of this article on software longevity called ā100 year computerā: https://thedorkweb.substack.com/p/the-100-year-computer |
| 00:38:29 | Jess Martin: | without schema migration support, software āossifiesā over time, which makes it really hard to write software that can run for 100 years without needing to be completely rewritten from scratch |
| 00:38:36 | Jess Martin: | makes it hard to build āevolutionaryā systems |
| 00:39:11 | Boris Mann: | I see breaking API changes as being evolutionary pressure |
| 00:39:31 | Peter: | the name ācambriaā is no accident |
| 00:39:35 | Boris Mann: | SHOULD we break APIs more often? If no one screams, no one cares. If a client breaks, it means it doesnāt have sustainable maintenance. |
| 00:39:59 | Boris Mann: | Mainly this sucks for users who arenāt devs. |
| 00:41:37 | Brent Anderson: | Having options to lift this above the storage layer is so important for stateless systems. Is that one of the key insights to Cambria? |
| 00:42:24 | Brian Cloutier: | Maybe we should break APIs more often but itās tantamount to abandoning maintainerless software |
| 00:42:41 | Peter: | shout out to jim pick on the call here who helped us explore an early version of this |
| 00:42:53 | Jim Pick: | /me waves |
| 00:43:34 | Jess Martin: | Mann: feels like using a blowtorch to light a candle.  I agree that schema migrations are a signal that evolution is happening! But if preserving forwards/backwards compatibility is possible and easy, why shouldnāt we? |
| 00:43:51 | Benjamin Bollen: | are there constraints on this graph of lenses? eg. a global directional flow / foliation |
| 00:44:32 | Brooklyn Zelenka: | Theyāre bidirectional, so the arrows always point both ways |
| 00:44:54 | Boris Mann: | @J Martin: yes, for sure. Iām thinking a lot about people supporting maintainers in a more wholistic way these days. I am all for using approaches like this to contain breakage. |
| 00:44:58 | Benjamin Bollen: | right, but it could be circular / contain loops |
| 00:45:01 | Brooklyn Zelenka: | Itās also a graph, so itās not ājustā server, you can get very forked schemata |
| 00:45:05 | Peter: | loops are fine |
| 00:45:07 | Brooklyn Zelenka: | Yeah, totally |
| 00:45:12 | Peter: | we do a Dijkstra search to find a path |
| 00:45:22 | Benjamin Bollen: | nice |
| 00:45:39 | Peter: | Geoffrey will, Iām sure return to this, but not every migration is necessarily reversible. You will still need an escape hatch now and again. |
| 00:45:42 | Brent Anderson: | Somewhat out of scope with schema migration: What about CRDT compaction? If weāre accumulating a pile of CRDT deltas and sharing them between peers, what options exist for preventing the massive accumulation of past events? |
| 00:45:52 | Peter: | Letās save that for the end, Brent |
| 00:46:00 | Jess Martin: | āYAML. we donāt love it, either.ā Heard that one before. |
| 00:46:07 | James Conkling: | anyone have a good CRDT for Dummies resource? |
| 00:46:30 | Peter: | happy to have a CRDT sidebar at the end |
| 00:46:38 | Boris Mann: | @James: thatās a good thing to look for ā I think we have some things at the āpapersā level in the Fission forum, but would be good to have more resources and samples. |
| 00:46:39 | Marc-Antoine Parent: | I got a lot out of this: https://hal.inria.fr/inria-00609399v1/document |
| 00:47:43 | James Conkling: | Iām going to have to drop early, unfortunately, so wonāt be able to join a sidebar at the end. But will definitely follow up on the above resources. Thanks all! |
| 00:47:55 | Jess Martin: | hmmmā¦ this really goes beyond compatibility. this makes possible different representations of core data. |
| 00:48:20 | Peter: | check out automerge (our crdt), y-js (the other best crdt out there). also crdt.tech by martin kleppmann |
| 00:48:23 | Jess Martin: | can represent the data in the user-facing application as it most naturally should be represented as appropriate to context  |
| 00:48:25 | Boris Mann: | @James if you registered, Iāll send out email with a link to the video and any resources we gather |
| 00:48:37 | karlicoss: | Maybe a bit of an overkill at this point, but perhaps dependent types or some other kind of formal verification could be used to verify that the lenses are reversible (I guess the most important properly?). This might allow imposing less structure and assumptions on lenses |
| 00:49:45 | karlicoss: | s/properly/property |
| 00:49:55 | Jess Martin: | doesnāt go through the āmigration stackā like in Stripe impl |
| 00:50:10 | Jim Pick: | might be a fun tool for mapping schemas between proprietary silos for cross-silo sync tooling |
| 00:50:12 | Steven: | Amazing  |
| 00:50:31 | Brian Cloutier: | For the push/pull distinction, it feels possible to extend this to work over more than JSON objects. You might write a lense which transforms one sequence of RPC calls into another sequence. |
| 00:51:08 | Marc-Antoine Parent: | it had schemas but not a good schema migration story |
| 00:51:24 | Jess Martin: | someone should capture a quote |
| 00:51:25 | James Conkling: | @Boris Thanks! Pretty sure I registered. If not: |
| 00:52:58 | Dylan Steck: | Peter, that was key ā a key to semantic web technologies being using different schemas to relate to other peopleās schema. Itās a system like this that is going to help āun-siloā data and build out some of the core linked data concepts without cloning the Internet(like Blockstack) |
| 00:54:41 | Idan Gazit: | Yes, Kafka, the datastore known for its administrative ergonomics |
| 00:55:23 | Idan Gazit: | This sounds an awful lot like āStore UTC, convert at displaytime" |
| 00:55:30 | Peter: | absolutely! |
| 00:55:33 | Idan Gazit: | (In a good way!) |
| 01:03:39 | Boris Mann: | āNice squigglesā |
| 01:03:45 | Boris Mann: | Well, thatās going to be a t-shirt |
| 01:06:34 | Brent Anderson: | So are lenses always associative, or is it possible for a lens to be commutative? |
| 01:06:58 | James Conkling: | how do you get loops in the schema graph? |
| 01:07:08 | Idan Gazit: | āPRs not mergeable without tests, docs, lensesā |
| 01:07:15 | Peter: | the lenses are not directional |
| 01:07:18 | Helder S Ribeiro: | So how dows the schema for representing the lens graph evolve? |
| 01:07:31 | Boris Mann: | Idan: yaaas |
| 01:07:33 | Peter: | haha, so we tried writing lenses for that because we changed the format at one point |
| 01:07:49 | Peter: | in brief, writing lenses for your lenses is possible but Lovecraftian to try and read and think about |
| 01:08:11 | Peter: | impossible geometries. unthinkable complexities. horrors that cause the human mind to shrink in terror. |
| 01:08:15 | Helder S Ribeiro: | āwho lenses the lensmenā |
| 01:08:35 | Peter: | also, more practically, we were missing a couple of lenses needed to do it and didnāt get to them during project implementation |
| 01:08:35 | Benjamin Bollen: | have you considered āversionsā or ānames of nodesā to be derived from the type |
| 01:08:51 | Marc-Antoine Parent: | morphism->functor->natural transformationā¦ same idea, easy, right? |
| 01:08:57 | Antranig Basman: | Sadly, the world requires us to think the unthinkable |
| 01:09:23 | Peter: | fun fact, Haskell lenses were inspired by the academic research that we built on |
| 01:10:01 | Marc-Antoine Parent: | Yes, had met Pierceās lens through Unison ages ago! |
| 01:10:23 | Dillon Kearns: | Relevant talk: Evergreen Elm https://www.youtube.com/watch?v=4T6nZffnfzg. Uses the same concept of migrating data. And itās being used in production in https://lamdera.app/ (announcement talk: https://www.youtube.com/watch?v=nSrucNcwlA8) |
| 01:13:56 | James Conkling: | Iāve got to drop. Thanks Geoffrey and Peter and all. This is giving me a lot to chew on. |
| 01:14:40 | Jess Martin: | also, knowledge representation. |
| 01:14:45 | Jess Martin: | we suck at that, too |
| 01:15:35 | Jess Martin: | mapping across applications is the more exciting application! or mapping across use cases/views. we can represent knowledge in a stronger form, then allow for manipulating and interacting with the knowledge using appropriate ālensesā on the data |
| 01:15:38 | Idan Gazit: | Apologies! I also have to drop, right in the middle of a proper diatribe that Iām 100% into |
| 01:15:41 | Jess Martin: | separate the tools from the data |
| 01:16:06 | Marc-Antoine Parent: | Letās face it, interop is opposed by business models also. But I agree, some of us want interop. |
| 01:16:29 | Jess Martin: | hereās that PVH quote as a tweet: https://twitter.com/jessmartin/status/1359556952322408451 |
| 01:16:33 | Jess Martin: | |
| 01:17:12 | karlicoss: | Hopefully with Cambria it would be possible to adversarily interop with such businesses |
| 01:17:37 | Helder S Ribeiro: | amen to that |
| 01:17:59 | Marc-Antoine Parent: | indeed |
| 01:18:32 | Helder S Ribeiro: | advop all the things |
| 01:20:57 | Brooklyn Zelenka: | FWIW weāve been exploring lenses + datalog (early work, but still) |
| 01:21:08 | David K: | Nice |
| 01:23:02 | Jess Martin: | On Prolog: ārummage through that corpse in the alley and steal anything we can useā |
| 01:23:32 | Boris Mann: | J Martin: welcome to the harvest PVH quotes channel! |
| 01:24:11 | Gyuri Lajos: | https://twitter.com/TrailHub1/status/1359499797007327232 |
| 01:26:04 | David K: |  |
| 01:27:12 | Tanishq Kancharla: | Do people want to drop their twitters? Mine is @moonriseTK. |
| 01:27:36 | Jess Martin: | something like general knowledge vs context-specific knowledge |
| 01:28:12 | Boris Mann: | Good idea! Please do ā Iām @bmann, @FissionCodes is the company account |
| 01:28:23 | Marc-Antoine Parent: | @maparent |
| 01:28:29 | Brent Anderson: | @brentjanderson or @bja@mastodon.social |
| 01:28:29 | Marc-Antoine Parent: | sorry @ma_parent |
| 01:28:45 | Antranig Basman: | Iām sure saw a āraise handā control in other Zoom meetings but I donāt seem to see one here : P |
| 01:28:45 | Boris Mann: | @bmann@social.coop on Masto, good call Brent! |
| 01:28:48 | Gyuri Lajos: | @TrailHub1 |
| 01:28:58 | Brian Cloutier: | @bmc_ |
| 01:29:08 | Ian: | @ianopolous is me |
| 01:29:36 | Boris Mann: | @Antranig ā under Reactions at bottom of your screen |
| 01:29:36 | karlicoss: | @karlicoss |
| 01:30:08 | Brian Cloutier: | I really hope the pvh rant was recorded, that was some inspirational stuff |
| 01:31:04 | Boris Mann: | Who can do me a dubstep remix of Peter Rant? |
| 01:31:19 | Boris Mann: | Iām also down with a Sea Shanty version |
| 01:31:27 | James Walker: | I was gonna sayā¦ shouldnāt it be a sea shanty ? |
| 01:32:15 | Helder S Ribeiro: | (hands waving, doing buzzword arithmetic) would lens graphs and ipld relate in some useful way? |
| 01:32:18 | Brian Cloutier: | The Cambriaman |
| 01:33:20 | Jess Martin: | digging Antranigās office lighting vibe. very film noir. |
| 01:35:06 | Antranig Basman: | Itās just the evening : P |
| 01:35:27 | Peter: | @helder: possibly. Weāve talked to them a bit ā I think they compose well |
| 01:35:35 | Marc-Antoine Parent: | Q: am I right that you built the schema transformation language while assuming the schema language (JsonSchema, I guessā¦) |
| 01:37:09 | Benjamin Bollen: | right, so now itās a DAG of bidirectional lenses (which still have primary direction āsource - targetā) |
| 01:37:16 | Benjamin Bollen: | so there is a foliation |
| 01:37:23 | Antranig Basman: | It is actually an edit transformation language rather than a schema transformation language |
| 01:37:45 | Boris Mann: | Topics tagged crdt |