File System Access API support in Webnative

Developers ODD SDK
1

From discussion with @gotjoshua and issue here Implement File System Access for high performance user experience. · Issue #398 · fission-codes/webnative · GitHub

I think File System Access API, while Chrome only, covers a lot of interesting use cases that fit with the Webnative model of user owned data.

Can we store WNFS formatted data on disk? Yes. But maybe it should be stored un-encrypted and then encrypted when synced?

@gotjoshua if you can share some examples of how you see this working it would be helpful.

2

sure… gladly!

1.) immediate write to local file system in the spirit of local first

  • note File System Access can be ponyfilled for non-chromium browsers
  • my web app data can be simply browsed and manipulated via my local OS
    • apps can also become sensitive to manipulation that did not originate in app

2.) “lazy” sync to *local-ipfs which in turn lazy syncs to global ipfs pins

  • the whole encrypting, hashing, cid calculation, local caching and global announcing needs to happen in a well throttled manner in worker threads

3.) Assuming multiple agents/devices will access the same directories and files: each app instance will need to actively poll the remote ipfs directories and do intelligent conflict resistant syncing in the background

4.) *Various levels of “local” : my webnative user account could be configured to “use” several levels of local…

  • local FS (via File System Access or ponyfill)
  • local in-browser ipfs (via js-ipfs)
  • local native ipfs node
  • LAN node(s) - (native nodes available within my firewalled NAT LAN situation)
  • WAN node(s) - specific private or public nodes

< admittedly slightly out of scope >
5.) Not exactly sure about technical details on this part, but: It would be epic if agents/peers could somehow publish their availability for serverless p2p content addressable folder sync.
</ admittedly slightly out of scope >

3

I’m looking into these two libs so far:

anyone know of others worth considering?

4

Great, thanks for the thoughts on this.

Brave browser (and a few other things plus the IPFS Companion browser extension) have IPFS built in. How to expose that to a browser context is a thing that’s in progress.

js-IPFS in browser has some issues — but so does native IPFS itself which is why we’re investigating more efficient sync / transfer.

CAR files and using HTTP transport to IPFS nodes is what the IPFS community is looking at.

What you call lazy sync is part of this — say your file system is 1GB but only 2MB has changed. IPFS does some of this effectively with blocks, but there are ways to optimize this.

To keep things encrypted we need to build a WNFS aware desktop client. Rust version of WNFS is progress that could handle this. Could still use Kubo (go-ipfs) for sync.

Keeping things unencrypted on local disk could be done and yeah for certain use cases just being able to browse on (or some sort of transparent encrypt / decrypt).

On 3, yes we are looking at how to include CRDTs for conflict resolution.

IPFS protocol itself already takes care of this. It will discover and connect to other nodes and/or be given certain peer addresses.

Peer connecting to eg a Fission server or Web3Storage for persistence / permanent storage with Filecoin would be a call to make to pin/save
. This gets done by Webnative SDK today — we want to add support for Web3Storage and others.

There are parts of this that could be interesting to experiment with.

  1. I think IPFS from the browser / injecting IPFS is one interesting area. It doesn’t need file system access and keeps things in browser, but could also work with eg IPFS Desktop app. Let me see where I can find this discussion happening.

  2. WNFS on the desktop. Lots of different pieces here on what this should do, but rs-wnfs and rs-ucan get us there.

You would need a Webnative desktop client to communicate the CID of your WNFS filesystem root / do conflict resolution.

I think File System Access is interesting but the state of the IPFS protocol and the lag between components from browser to local to local IPFS to remote IPFS and other clients — and I’m not even clear that local file changes can trigger a call back to the browser.

I think public data only would be where to start with an experiment. And I’d love an even more specific example of something to do. Eg photo gallery / photo syncing.

Let’s talk more about it during our call this coming week!