Sharing/browsing large volumes of offline content

[OtisByron]

This is a more of a thought experiment than anything else. The scenario I have in mind while thinking about this is hosting a local Wikipedia instance using kiwix on the same computer as a NomadNet node, or on another computer on the same local network (i.e. not requiring internet access). I was initially imagining an experience similar to Lynx or other text-based web browsers. I believe this would require external link support (see discussion #34) as well as parsing html into micron on-the-fly. Maybe there is a simpler way to achieve this?

In general using NomadNet's file sharing support would make more sense with how the system is structured today. However, the zim files used by kiwix are quite large (the text-only version of Wikipedia is over 12GB) and quite binary (not easily parsed). Even so, they are the best thing I have found so far for sourcing and locally hosting large volumes of web content (Wikipedia, Project Gutenberg Library, StackExchange, etc). If anyone knows of a better alternative please share!

Ideally there would be a better system for compiling content. I have yet to experiment with this, but one idea is to setup a local kiwix server hosting the desired zim content and then use wget to mirror from the local kiwix server (since the restructuring for offline use is already done in the zim file content). This would generate the individual files needed to share over RNS. Additional parsing of links would be required to point to adjacent content on the node (possibly using the wget --base option?). You would lose the search capabilities of kiwix as well, which would be a significant loss. Maybe the above concept of creating a mirror of individual files plus basic file content search capability in NomadNet for searching/filtering files on a hosting node would be a good compromise? Hosted file content search would be a new feature for NomadNet as far as I know.

I'm interested if others have thoughts on how best to achieve such a system. The answer may be that libraries are centralized within each community for a reason and this case isn't any different, meaning that it would be best for someone in each community to take on the responsibility of hosting large volumes of content for use locally instead of trying to distribute it. Or maybe creating a RNS browser should be a separate application outside of NomadNet all together (possibly related to discussion #12 on RRTP?).

[samuk]

I don't think LoRa is well suited to this kind of data transfer personally. Getting short messages across a multi-hop mesh is challenging enough IMHO.

I've been playing with piratebox a bit here on an Orangepi, with a view to adding a LoRa shield to that once it's working as a file share device & SSB server.

Some degree of sharing SSB messages over the LoRa interface might be possible, although that's a bit of an open question

The content I've pulled together is here: https://rosettaphone.org/ it includes the IWS.apk and shareviahttps.apk so there is the possibility of using android devices as hotspots or further sharing the content Android<>Android.

[OtisByron]

@samuk thanks for the feedback and for linking some of the other relevant discussions. You are likely correct that traditional "browsing" doesn't have a place in a bandwidth limited environment. That feels like the right answer considering how much congestion it would cause on the network.

It would seem that there is still a need for sharing content in an offline and preferably distributed way to enable learning. Given that RNS has support for passing resources it seems like it could be useful for content distribution as discussed in #22. A dedicated resource distribution network with only higher throughput interfaces configured is interesting. I was coming at this from a different direction, but the more I think about it we are indeed talking about the same problem.

I really like what you're doing with Rosetta Phone. I'm assuming you are already familiar with Kolibri which has similar goals.

[samuk]

Kolibri is a great project, they are probably one of the best for an institutional setting, schools particularly. Also looking at https://internet-in-a-box.org/ but I find their Ansible approach a bit confusing/scary. Also, there's https://worldpossible.org/ do a hotspot thing, but no captive portal.

A lot of this type of content is Attribution-Non-Commercial (NC) which is fine, but my personal interest is in devices/SDcards that can be sold. Not that I'm personally interested in selling them, but I'd like local partners to be able to.

Just for completeness, I'll also mention https://othernet.is/ who sends Wikipedia pages, news & weather from Space. They actually do use LoRa, but not in the ISM bands which are typically available terrestrially.

[4c3e]

Content browsing and file sharing is the primary goal of RRTP, which is built on RNS Link and Resource API's in order to handle data compression + encryption in an efficient manner. Obviously downloading 12GB over a multi-hope LoRa connection would take quite a while, but searching for individual text files a few kb's in size would be doable even over the slowest LoRa connections.

I wrote a demo of a basic RRTP server, and a text-based browser: https://github.com/4c3e/rrtp-demo-1
And am working on a RRTP request library to make requesting resources over RNS very easy for app developers: https://github.com/4c3e/rrtprequest

NOTE: these demos are already out of date because the RRTP spec is still very much under construction.

Once RRTP becomes more stable, users could host an RRTP proxy server for the Kiwix wikipedia instance. The proxy server would take RRTP requests, translate them into a request that the local kiwix server can handle, then forward the response over RNS to the client. The cleaner solution would be to create something like Kiwix, but as a native RRTP server, which would be a great project to work on at some point 😄

[4c3e]

Also, with RRTP, (and HTTP of course) the response header specifies a MIME type which gives browsers a hint about how to handle the incoming data. NomadNet's browser and/or any independent browsers can support different MIME types however they want. One fantastic example of this is bombadillo which supports a number of different protocols and MIME types. If you request an https link in bombadillo it warns the users and can be configured to optionally open firefox/lynx at that link.

[OtisByron]

@4c3e thanks the details info. I haven't had a chance to check out the demo yet but I plan to. This sounds quite interesting.

[4c3e]

no problem, feel free to reach out if you have any questions. The fundamental idea is just to document a standardized format of resource transfer (something that NomadNet already does quite well!), to allow for many different clients/servers to connect and interact with each other.

[markqvist]

I think it's an OK assumption that libraries are mostly centralised in communities (but should be very easy to replicate and make redundant so they spread to anyone who wants to host them), and I like the idea of being able to serve ZIMs directly to NomadNet clients. In this case I think search functionality is pretty essential, and it is actually almost possible right now.

I have a kinda-working proof-of-content thingy that serve content from ZIMs directly to the NomadNet browser, but it is an awful hackjob that was more of an experiment to see what would work. Search can happen, but only "statically" (via a predefined link, serving a "predefined" search) since there is not yet any support for inputting and sending data to the page request handlers.

I think there is two ways to go about this. Either we make it possible via the NomadNet browser, which require something like the following:

• Get to satisfactory version of the RRTP spec that @4c3e is working on, and change the NomadNet implementation to follow this spec. We need to also formalise a way of "posting" data with a request. Reticulum already supports this (the data argument), we just need to settle on how to do it in RRTP.
• Change the NomadNet node server and browser a bit, so it follows the RRTP spec.
• Add a way to handle very basic forms with input fields in NomadNet and the micron markup language. This will allow us to send search queries and other basic stuff.
• Either we can add a feature directly to NomadNet for reading ZIMs in a directory, or we could write a preprocessor script that gets executed by NomadNets page server which handles the lookup in ZIMs and translation to micron format.

Or we could just build a small standalone server and client only for the purpose of connecting to and browsing ZIMs served over RNS. I think I'm more in favor of getting the transfer spec standardised, and integrating it into NomadNet though. I think that makes most sense in the long run.

[jphastings]

Hi folks; I'm only just getting up to speed in this community, so I'm lacking a lot of context on what currently exists — my apologies if I'm retreading ground!

The problem of widely distributing large, identical content is something that IPFS has put some effort into (see https://en.wikipedia-on-ipfs.org/) — the code that creates the bundles of data is in distributed-wikipedia-mirror. Reticulum + LXMF seem like an excellent combination link/transport layer for the IPFS protocols. I'd be interested in researching where IPFS's content-addressed retrieval would fit into Reticulum's ecosystem.

When it comes to search, IPLD (a method for linking data in an extensible format, used by IPFS) is flexible enough to allow a search index to be stored in such a way that you only need retrieve the parts that you need to get the content-hash of the data/page you want, then requesting the data behind that hash from your connected peers. Folks have already experimented with storing a lunr.js search index as a retrievable file that can be used in a JS-based retrievable homepage with solid success (though I've yet to find examples of people breaking apart the index file in a suitable way when large amounts of data are indexed).

Help me direct my thoughts & experience in these spaces to help out here?

[4c3e]

Hello and welcome! IPFS, and specifically the IPFS wikipedia project seem like they could possibly be very useful for the problem presented in the OP. My initial impression is that adding reticulum as a transport layer to IPFS makes the most sense to accomplish this. In fact I think it would help a lot with increasing the resiliency of IPFS networks.

The problem that would need investigating is how IPFS performs in delay tolerant, very low bandwidth environments. When dealing with packet radio, it's not uncommon to encounter transfer speeds of 200 bytes/s, (SSH handshake took @markqvist 40 seconds at 15km almost line-of-sight with LoRa). Because of these physical-layer limitations, everything built with RNS need to be extremely cautious of network bandwidth. I have no clue how IPFS works under the hood, but some modifications might be needed in order to cut down on bandwidth usage. In general, RNS always chooses to use extra memory and "off-air" computation whenever possible to avoid using bandwidth whenever possible.

Besides IPFS, it might be interesting to experiment with DHT's on RNS as a way to save bandwidth for LXMF propagation nodes. For example instead of syncing entire message content, nodes could just sync message hash values, and store a reference to the node they received the hash from. Then when a client requests a message hash from a propagation node it could forward the client to the propagation node that had the message content. Intuitively this seems like it would save network bandwidth, but might not work as well in situations where propagation nodes aren't always online.

[jphastings]

I think you're right; the existing content replication protocol used by IPFS (BitSwap) seems likely optimised for mid- to high-bandwidth networks, but thankfully IPFS is modular and there seems to be explicit research into high latency systems too (see GraphSync).

Either way, it should be entirely reasonable to make use of existing IPFS code for content addressing and DAG structuring and use anything specific to reticulum for discovery & transfer (eg. something built on top of LXMF over Reticulum).

Right now, "sneakernets" are supported with Content Addressable aRchive (CAR) transport, where someone would verbally ask for specific subset of the wikipedia, then receive the structured data — eg. carried on a USB drive — in a format that can be readily understood by a (disconnected) IPFS node; given this ultra high latency approach is supported (albeit, without any digital node discovery!), it seems like IPFS is a suitable platform to leverage 😊

[jphastings]

I had a poke around in the graphsync-go library and docs; there's an interface which we could fill with a client talking over Reticulum (by communicating with a local rnsd instance — perhaps it could use LXMF?).

Given that GraphSync is designed with high latency/low bandwidth in mind, and optimises for minimum data transfer by doing heavy lifting in the requestor/responder (rather than IPFS' current default data transfer protocol, BitSwap, which optimises for computation simplicity at the expense of network use) — this feels like it could be a good place to start; what do you think?

[4c3e]

Thanks so much for all the references, looking forward to spending some time reading through all this.

For a quick sanity check, is the basic idea that RNS as a graphsync interface would allow peers on the RNS network to hold pieces of a content graph (i.e. sections of Wikipedia) and sync/request other parts of the content graph from other peers on the RNS? If I'm understanding that rough idea correctly, then I think adding an RNS interface to the go impl would be a great way forward. I couldn't find a graphsync python lib, so there will be a bit of friction to get the go lib communicating to python.

One idea that I'll hopefully get to eventually would be to have a python process that runs a local TCP server and listens to text commands coming in from client connections, then forward these commands/data over RNS. This would mean that a go RNS library could be built just by maintaining a client connection to this socket instead of totally re-implementing RNS protocol and transports. This is the basic idea behind i2p's SAM Library. I2P has one main Java impl, and one partial C++ impl, the rest just use this SAM API to build libraries.

For now I'll spend some time going deeper into the documentation you linked.

Additional question:: Would graphsync-over-RNS allow for internet connected RNS instances to forward graphsync requests to IPFS?

[samuk]

I don't have much to usefully add. I did bring up IPFS here: #45

[4c3e]

Another relevant protocol for this: https://hypercore-protocol.org/ , https://github.com/hypercore-protocol/hyperdrive

Not sure how it compares to IPFS, but there is a nice looking browser to go along with this protocol: https://beakerbrowser.com/

Edit: just kidding beaker browser is defunct beakerbrowser/beaker#1944 (comment)

Modern alternative seems to be https://agregore.mauve.moe/

[4c3e]

Also here's a TUI for browsing wikipedia files: https://github.com/NerdyPepper/taizen
Might be able to use some of it for creating an interface for a wikipedia app on RNS.