Fear and Loathing and IPFS File Metadata Tricks

Fission is working on defining a Fission File System spec (very work in progress in our whitepaper) that adds some convenience features on top of IPFS, along with support for our encrypted files implementation.

Here’s a neat trick that we came up with while working on metadata:

https://ipfs.runfission.com/ipfs/QmaxBR8djuHiQ5DcDH92rsVAjaeLFNnT8awcLKiGE5SiEx

If you follow that link, you should see this image loaded:

However, we’ve also attached a link within that IPFS entry (which is the same boris.fission.name directory that we use as our default example public content):

https://ipfs.runfission.com/ipfs/QmaxBR8djuHiQ5DcDH92rsVAjaeLFNnT8awcLKiGE5SiEx/embedded

How does this work? CIDs for images are actually DAG Nodes pointing to a bunch of chunks of data that make up the image. We put an extra link in there to point to more data:

{
  "data":"CAIY65c8IICAECCAgBAggIAQIOuXDA==",
  "links": [
    {"Cid":{"/":"QmYDARDj3A4sTVMkSC8wf7KjXo6MhzwLwHvoHnKzKVTAP6"},"Name":"","Size":262158},
{"Cid":{"/":"QmeG8Rw9YEZx8zw93wTyC5NgRemjufaT9HaWQvvp27cThf"},"Name":"","Size":262158},
{"Cid":{"/":"QmdrQ5FNZN6DbMuC7yMnH5iWpk667eW99sHEQWFkzVHTXz"},"Name":"","Size":262158},
{"Cid":{"/":"QmS51h7vQPvRxH1hQjMKhREXX22DaqgYPNZAi3jJvVqXL5"},"Name":"","Size":199673},
{"Cid":{"/":"QmWKst5WVNTPfMsSFCQEJYBJLEsUZfghqrKXJBVU4EuA76"},"Name":"embedded","Size":1234}
  ]
} 

The image still loads fine. This is a technique we’re looking at to store metadata alongside files broadly – with some specific interesting things to be done with images, like storing multiple different sizes along with the original image.

Problem 1: Doesn’t work for files smaller than the chunk size (32K)

Say we have very short content, like hello world, which by itself looks like this:

https://ipfs.runfission.com/ipfs/QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o

And then add the same /embedded link:

{
  "data":"CAISDGhlbGxvIHdvcmxkChgM",
  "links":
    [
       {"Cid":{"/":"QmWKst5WVNTPfMsSFCQEJYBJLEsUZfghqrKXJBVU4EuA76"},"Name":"embedded","Size":1234}
    ]
}

It doesn’t work / display correctly:

https://ipfs.runfission.com/ipfs/QmZMgruHmTJv6MQSvq5RZhrskfcKwFooqmDpet78jdQcst

The /embedded path does still work though:

https://ipfs.runfission.com/ipfs/QmZMgruHmTJv6MQSvq5RZhrskfcKwFooqmDpet78jdQcst/embedded

Screenshot of that path:

IPFS / Gateway Issue

What we’ve identified is that if the IPFS gateway sees links in a node (what would otherwise be a single, non-chunked data node like a short piece of text or data), then it doesn’t interpret it as a content node.

Is this a bug? We think so, since it would be consistent to work in both cases.

Any chance you can share the code you’re using to do this?

1 Like

@alanshaw manually with go-ipfs—

ipfs add huntersthompson.jpg

ipfs dag get QmeMu9rLCG1eSEYTYtDw5EDpJbmw8qJPTJszPsLSC4ZNKt

manually add embbedded link

ipfs dag put -f dag-pb

paste

{"data":"CAIY65c8IICAECCAgBAggIAQIOuXDA==","links":[{"Cid":{"/":"QmYDARDj3A4sTVMkSC8wf7KjXo6MhzwLwHvoHnKzKVTAP6"},"Name":"","Size":262158},{"Cid":{"/":"QmeG8Rw9YEZx8zw93wTyC5NgRemjufaT9HaWQvvp27cThf"},"Name":"","Size":262158},{"Cid":{"/":"QmdrQ5FNZN6DbMuC7yMnH5iWpk667eW99sHEQWFkzVHTXz"},"Name":"","Size":262158},{"Cid":{"/":"QmS51h7vQPvRxH1hQjMKhREXX22DaqgYPNZAi3jJvVqXL5"},"Name":"","Size":199673},{"Cid":{"/":"QmWKst5WVNTPfMsSFCQEJYBJLEsUZfghqrKXJBVU4EuA76"},"Name":"embedded","Size":1234}]} 

Note that you cannot ipfs get or ipfs cat nodes with this extra link:

$ ipfs get QmaxBR8djuHiQ5DcDH92rsVAjaeLFNnT8awcLKiGE5SiEx
Saving file(s) to QmaxBR8djuHiQ5DcDH92rsVAjaeLFNnT8awcLKiGE5SiEx
 962.98 KiB / 962.98 KiB [==========================================================================================] 100.00% 0s
Error: archive/tar: write too long
$ ipfs cat QmaxBR8djuHiQ5DcDH92rsVAjaeLFNnT8awcLKiGE5SiEx > img.jpg
Error: found Directory node in unexpected place

I think the gateway is ignoring the error and should probably be adding a HTTP trailer (as it does on other cases for stream errors) but I don’t see one being added.

This is an interesting hack though!

1 Like

This is not the exact same gateway bug, but rather an issue that @ellttBen found and documented that is similar:

https://github.com/ipfs/go-ipfs/issues/7190