Example from dat cookbook doesn't produce any files

[cafca]

I am trying to run the second node.js example from the dat cookbook where mirror-folder is used to copy all files from a dat to a local folder. I copied the code from the example verbatim to a file index.js. When I run

$ node index.js dat://....

a folder dat-download is created but it doesn't contain any files. The console shows:

pv@pv:dat-test$ node mirror.js dat://2714774d6c464dd12d5f8533e28ffafd79eec23ab20990b5ac14de940680a6fe
done downloading!

I tried several different dat addresses other than the rotonde-client address, none of which worked. The other example from the cookbook, where only one specific file is downloaded does work.

I also installed the mirror-folder lib separately and ran npm test, which produced no errors.

Which steps could I take to find out why this is not working?

I'm on osx 10.13.4

pv@pv:dat-test$ node -v
v9.11.1
pv@pv:dat-test$ npm -v
6.0.0

[aral]

It’s not producing any files as / is being skipped. To reproduce, add:

  progress.on('skip', (source) => {
    console.log(source.name, 'skipped')
  })

And you’ll see: / skipped

I’m looking into why this is now.

[aral]

Have just confirmed that the problem exists on latest master and that tests on latest master are all passing.

[aral]

Right, this is a timing-related issue: the mirror is starting before the archive has synced. Code below delays execution and works to illustrate the problem:

const path = require('path')
const ram = require('random-access-memory')
const hyperdrive = require('hyperdrive')
const discovery = require('hyperdiscovery')
const mirror = require('mirror-folder')
const mkdirp = require('mkdirp')

const link = process.argv[2]

const key = link.replace('dat://', '')

const dir = path.join(process.cwd(), 'dat-download')
mkdirp.sync(dir)

const archive = hyperdrive(ram, key)

archive.ready(function (error) {
  if (error) throw error
  console.log('key', archive.key.toString('hex'))
  const swarm = discovery(archive)
  swarm.on('connection', (peer, type) => {
    console.log(`Connected to peer (${type.type})`)
  })

  setTimeout(function () {
    console.log('Starting to mirror.')
    const progress = mirror({name: '/', fs: archive}, dir, (error) => {
      if (error) throw error
      console.log('Done downloading')
    })
    progress.on('put', (source) => {
      console.log(source.name, 'downloaded')
    })
    progress.on('skip', (source) => {
      console.log(source.name, 'skipped')
    })
  }, 2000)
})

With that, the console output is as expected:

key 7c10637cbc7497b002dec935415879cf5a4f666bb30fbe8269a95ca5600ba0a0
Connected to peer (tcp)
Connected to peer (tcp)
Starting to mirror.
/ skipped
/aral.jpg downloaded
/dat.json downloaded
/index.html downloaded
Done downloading

Compared to the console output without the setInterval, which is:

key 7c10637cbc7497b002dec935415879cf5a4f666bb30fbe8269a95ca5600ba0a0
Starting to mirror.
/ skipped
Done downloading
Connected to peer (tcp)

The question is what’s the correct way to wait for sync before kicking off the mirror operation? Will look into that next.

[aral]

Right, so if you start mirroring when the archive emits the content event, it seems to work (and makes sense to me, at least semantically, if I’m correct to assume that the content event means “content is ready/synced” – see holepunchto/hyperdrive#219).

I’ll put in a PR for the docs with the code below.

@i’d appreciate it if someone from the dev team can review the code to make sure that this is the recommended way of doing things.

const path = require('path')
const ram = require('random-access-memory')
const hyperdrive = require('hyperdrive')
const discovery = require('hyperdiscovery')
const mirror = require('mirror-folder')
const mkdirp = require('mkdirp')

const link = process.argv[2]

const key = link.replace('dat://', '')

const dir = path.join(process.cwd(), 'dat-download')
mkdirp.sync(dir)

const archive = hyperdrive(ram, key)

archive.ready((error) => {
  if (error) throw error
  const swarm = discovery(archive)
  swarm.on('connection', (peer, type) => {
    console.log(`Connected to peer (${type.type})`)
  })
})

archive.on('content', () => {
  console.log('Archive’s content is ready.')
  console.log('Starting to mirror…')
  const progress = mirror({name: '/', fs: archive}, dir, (error) => {
    if (error) throw error
    console.log('Done mirroring the archive’s content to disk.')
  })
  progress.on('put', (source) => {
    console.log(`  Mirrored: ${source.name}`)
  })
  progress.on('skip', (source) => {
    console.log(`  Skipped:  ${source.name}`)
  })
})

Output:

Connected to peer (tcp)
Connected to peer (tcp)
Archive’s content is ready.
Starting to mirror…
  Skipped:  /
  Skipped:  /aral.jpg
  Skipped:  /dat.json
  Skipped:  /index.html
Done mirroring the archive’s content to disk.

[aral]

Unless mirror-folder should work in the ready event handler, this issue should be moved to https://github.com/datproject/docs.

[mafintosh]

nice catch and investigation @aral - i think this is a bug here. i'll dig into it tmw :)

[cosmicespresso]

Just a reminder that the official documentation still has the older code: https://docs.datproject.org/diy-dat which skips /.

[martinheidegger]

/ping