Notes on Data Interoperability and Schemas

Developers
#1

A continuation of Where does data live on your filesystem?
Inspired by the Cambria talk Project Cambria Overview with Geoffrey Litt and Peter van Hardenberg

Goal:
Easily use different apps for the same data.

Webnative SDK

To encourage data reuse in apps, we figured the webnative SDK would benefit from common paths such as:

playlistPath = webnative.common.audio.music.playlists(
  "Feel Good Songs.m3u"
)

With that reasoning, would it make sense to have the schema in the SDK as well?

webnative.common.audio.music.playlists.schema

Or does our original idea make more sense? Using the schema of a specific app.

/public/Apps/Published/APP_NAME.fission.app/schemas/playlist.json

I’m not sure. I think we should support this as well, we can’t possibly add all schemas to the sdk.

Changes to the schema

Assuming the schema never changes, we’d have to account for schema changes somehow. We could do that by using GitHub - inkandswitch/cambria: Schema evolution with bi-directional lenses. lenses.

Tricky thing here is, where do we store these lenses? Different applications will have different needs, and thus different lenses. What if we stored the lenses, or json patches, along side the data?

That way apps can revert the data back to the common schema, or perhaps a common lens (I think the cambria lens graphs make this possible). Apply their lenses, and then store the data and the lenses they used.

3 Likes
#2

This is a recurring topic in remoteStorage as well:

My way of doing it would be:

  1. assume that the notion of schema is not static or cannot be “common”, that there will always be inconsistencies (between multiple apps, between different versions of the same app, between representational differences like JSON vs RDF)
  2. that those inconsistencies extend to “common paths” too
  3. apps can declare lenses for translating their custom format and custom paths to a more common ones (there does not need to be consensus on which ‘common ones’), perhaps as part of manifest.json
  4. use something like Cambria to re-interpret paths and formats at runtime
2 Likes