Debugging webnative traffic

icidasset · January 22, 2021, 10:51pm

Webnative stores data locally and on the fission ipfs gateway.
If everything works correctly, all the data you work with regularly should be available locally.
If you make changes on other devices, this data will come from the fission gateway.

Monitor

To monitor the traffic coming in and going out you can do the following:

Open the shared worker console, or regular worker console in case of Safari.
Chromium based browsers: chrome://inspect/#workers
Firefox: about:debugging#/runtime/this-firefox (scroll to shared workers at the bottom)
Safari: Bottom right corner in the dev console, select “Marx” worker.
Type in await ipfs.bitswap.stat()
Look at the stats, some are BigNum instances, so you might need to call .toNumber() (eg. stats.blocksReceived.toNumber()

Want list

If an app/filesystem is stuck loading, it could be that a certain CID isn’t available on the Fission cluster. To find out which CID it’s try to load, look at the stats.wantlist (using the stats variable from above).

Local Data

Ideally when you add some data to your filesystem and reload the app, there should be no incoming data stream. To see if that’s the case, follow the steps laid out above and then check if blocksReceived is zero.

expede · February 2, 2021, 4:03am

I wonder if it would be helpful to add a config setting to WebNative that turns off various sources. It would help us isolate problems communicating with the remote server by turning off the local worker, and vice versa.

Thoughts?

matheus23 · February 2, 2021, 2:02pm

Just tried to do this with an infinite spinner case. For me, ipfs is not defined in the worker scope. I couldn’t find a self.ipfs, only a self.Ipfs (with a capital I). Do I have to enable some setting?

icidasset · February 8, 2021, 10:20pm

I think you can already do this by setting your own js-ipfs instance.

const ipfs = await Ipfs.create()
await ipfs.swarm.connect( fission_node_address )

webnative.ipfs.set( ipfs )

And then you can configure your js-ipfs instance to only keep data in memory by setting the Datastore config property. So if you’d refresh, you’d get a clean slate. That said, I haven’t tested this.

If that’s not enough, we could add a config setting to disable ipfs pinning entirely. If you don’t want to wait, I think you can already do this by making a javascript proxy.

const proxy = new Proxy(ipfs, {
  pin: {
    add: () => {} // don't pin
  }
})

webnative.ipfs.set( proxy )

(you’d also need to do this for add)

icidasset · February 8, 2021, 10:26pm

ipfs should be there. Maybe the ipfs instance hadn’t started yet?

matheus23 · March 1, 2021, 9:00am

Had another case of “infinite spinner” today. This time in chromium when I newly linked my oldest web account (matheus23-drive).

I’m able to debug it this time!

🚀 Started IPFS node
🪐 Connected to /dns4/production-ipfs-cluster-eu-north-1-node0.runfission.com/tcp/4003/wss/p2p/12D3KooWDTUTdVJfW7Rwb6kKhceEwevTatPXnavPwkfZp2A6r1Fn
🪐 Connected to /dns4/production-ipfs-cluster-eu-north-1-node1.runfission.com/tcp/4003/wss/p2p/12D3KooWRwbRrSN2cPAKz4yt1vxBFdh53CpgWjSFK5hZPkzHHz5h
🪐 Connected to /dns4/production-ipfs-cluster-us-east-1-node0.runfission.com/tcp/4003/wss/p2p/12D3KooWFSAbpiAeKHnVyqMqrdvAtu8C3veePHi36bZGNM2qv42q
🪐 Connected to /dns4/production-ipfs-cluster-us-east-1-node1.runfission.com/tcp/4003/wss/p2p/12D3KooWNntMEXRUa2dNgkQsVgzao6zGSYxm1oAs83YtRy6uBuxv
🪐 Connected to /dns4/node.fission.systems/tcp/4003/wss/p2p/QmVLEz2SxoNiFnuyLpbXsH6SvjPTrHNMU88vCQZyhgBzgw
const stats = await ipfs.bitswap.stat()
stats.wantlist[0].toBaseEncodedString()
"bafkreidxqm643dzot3xjpyu62jxni4mtzebd6zmg55lnzr5uwmj65hr4he"

Trying to fetch this CID from the fission ipfs gateway times out: ipfs.runfission.com/ipfs/bafkreidxqm643dzot3xjpyu62jxni4mtzebd6zmg55lnzr5uwmj65hr4he

The account loads in firefox, but not in chromium. I have both open at the same time on the same machine. This leads me to believe that the individual js-ipfs peers aren’t connecting to each other, is that right?

I’ll try 2 things next:

Try to connect the js-ipfs peers to each other
Try to somehow re-upload my filesystem from firefox and see whether it then loads in chromium.

matheus23 · March 1, 2021, 11:22am

I’ve done more debugging, first, some corrections. I got confused about where I’m logged into what (I’ve got 3 browsers on my laptop and another one on my phone and then production and staging accounts on all of them ).

Both firefox and chromium behaved the same
Drive opened up fine, but Flatmate didn’t load (in both browsers)

It just came down to the CID I posted earlier missing. Of course Drive works fine, because it only fetches the directory I’m in and it turns out that it was struggling to fetch the Flatmate app directory. And of course webnative in the Flatmate app then gets stuck in a loading state when trying to fetch the same directory as the root of the system.

When I delete the Flatmate directory in Drive (my Flatmate data is gone and) it loads fine.

I guess the next step would be figuring out why data goes missing. In theory we shouldn’t be updating the dnslink unless we fetched all data, right?

I’ll try getting into a corrupted state again intentionally now. Maybe this was just a residual issue of some ipfs-cluster problems.

matheus23 · March 1, 2021, 12:11pm

It doesn’t seem to be a residual issue of ipfs-cluster. I’m able to reproduce the issue on production today.

I ran lots of .publish-es by changing stuff in the Flatmate app on both my phone and laptop at the same time. At some point, their data got out of sync, but both were still loading fine. Looking at the console on the phone, this made sense:

📓 DNSLink is outdated (1 newer local entry), using local CID: bafybeibrbnaheqojyju45gyohejge6x43p6yedxoqdzjs2ftzs4w7qvpoy

However, that message didn’t go away with more .publish calls.

It did go away, when I did some actual filesystem changes and called .publish:

📓 Adding to the CID ledger: bafybeibxrkjpbuxfkj63ky5bc4ophxzweztmllqvb7tub2a77qinqae52y
🚀 Updating your DNSLink: bafybeibxrkjpbuxfkj63ky5bc4ophxzweztmllqvb7tub2a77qinqae52y
🚀 DNSLink updated: bafybeibxrkjpbuxfkj63ky5bc4ophxzweztmllqvb7tub2a77qinqae52y

Then, my laptop was able to fetch the newer version as well:

📓 DNSLink is newer: bafybeibxrkjpbuxfkj63ky5bc4ophxzweztmllqvb7tub2a77qinqae52y

However, the laptop now got stuck at webnative.initialise.

It had this hash in the wantlist: bafkreiaj3frv75q7wy5ei4d3ck5znqslquul63a43xh4zwjkrh4j6nkxom

I checked my phone, it could actually fetch that CID:

async function toArray(generator) {
  let chunks = [];
  for await (const chunk of generator) {
    console.log("got another item");
    chunks.push(chunk);
  }
  return chunks;
}

toArray(ipfs.cat("bafkreiaj3frv75q7wy5ei4d3ck5znqslquul63a43xh4zwjkrh4j6nkxom")).then(x => console.log("done"))
got another item
done

I took the data from ipfs.cat (from my phone) and manually (via copy-paste) ipfs.added it to my laptop.

(I think I just invented a new transport type: Transport on OSI layer 8 )

This got me one step further. Webnative would now successfully webnative.initialise. This leads me to believe that the private/Apps/matheus23-test/Flatmate directory was able to load. However, it gets stuck on reading the state.json file within it.

So I did the manual “look for what hash it wants, go to the other node, ipfs.cat, then copy the data over to the other node and ipfs.add it” again.

As soon as I finished doing that with two more hashes, it loaded fine.

Here are my takeaways:

It’s possible to get into a state where the sever ipfs cluster doesn’t store all data behind the dnslink by doing lots of (conflicting?) publishes
The data can’t be fetched from one device to the next via js-ipfs. The nodes don’t discover each other for some reason.

gyuri-lajos · March 1, 2021, 3:19pm

At the height of the “Mountain of Pain” I had gone up to the twelve accounts like gyuri12 and 5 browsers, chrome, chromium, brave, opera, dolphin, two laptops and and Android. Switching between browser profiles came to the rescue in the end.
Thankfully those days are over, and as I mirror everything I write in localforage, data loss can be avoided. I did not need to rely on that for over a month.
The more I use it the better more valuable it gets.
I rely on OrbitDb and its CRDT to exchange notes that I write, hot data,
and intend to save them in Fission WebNative on demand and or through automatic archiving. This way synchronization is fast and reliable. Never had data loss due to OrbitDb.
For read only files like pdfs and images and other static warm/cold data fission sync seems just fine.

icidasset · March 2, 2021, 2:17pm

Thanks for all the testing!

The data can’t be fetched from one device to the next via js-ipfs. The nodes don’t discover each other for some reason.

Yes, that’s a limitation of browsers and/or js-ipfs. Browser nodes cannot find each other, unless you introduce more centralisation by adding a signaling server. That’s why we’re talking with the IPFS team to fix this.