plugin idea: YA Bitcoin backend plugin

[darosior]

We could have a plugin that completely reproduce bcli's behaviour (hit bitcoind) but which would in addition fall back to other block sources if bitcoind can't serve that block (for example if it's been pruned already). Here is an example use case of nix-bitcoin.

We could fall back to a randomly-picked explorer from a list, cross-check its hash with bitcoind and serve it to lightningd.

There could also be a fallback to the Blockstream satellite :).

[jsarenik]

There is no cross-reference, so adding it: https://github.com/clightning4j/btcli4j

[cdecker]

I had something like this for a while, as a --bitcoin-cli argument, that would run against the pruned bitcoind, except for blocks which it'd fetch from a blockexplorer:

#!/usr/bin/env python3
from urllib import request
import subprocess
import sys
import json

cmd = sys.argv
cmd[0] = 'bitcoin-cli'
args = [c for c in cmd[1:] if c[:1] != '-']

if len(args) == 2 and args[0] == 'getblock':
    # Download from blockstream.info instead
    url = 'https://blockstream.info/api/block/{hash}/txids'.format(hash=args[1])
    r = request.Request(url, headers={'Accept': 'application/json'})
    content = json.loads(request.urlopen(r).read().decode('ASCII'))
    json.dump({'tx': content}, sys.stdout, indent=2)

elif len(args) == 3 and args[0] == 'getblock' and args[2] == 'false':
    try:
        out = subprocess.check_output(cmd).decode('ASCII')
        sys.stdout.write(out)
        sys.exit(0)
    except subprocess.CalledProcessError as exc:
        #sys.stderr.write("Could not retrieve block from local bitcoind, falling back to online explorers")
        pass
    url = 'https://blockexplorer.com/api/rawblock/{hash}'.format(hash=args[1])
    r = request.Request(url, headers={
        'Accept': 'application/json',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
    })
    content = json.loads(request.urlopen(r).read().decode('ASCII'))
    sys.stdout.write(content['rawblock'])
    sys.stdout.write('\n')

else:
    try:
        out = subprocess.check_output(cmd).decode('ASCII')
        sys.stdout.write(out)
    except subprocess.CalledProcessError as exc:
        sys.exit(exc.returncode)

Can't remember how well this was working, and today I'd probably use blockstream.info as backend, but thought I'd share it here

[jsarenik]

@cdecker Amazing! Thank you! An eye-opener for me. So here is my take in POSIX-shell compatible with busybox:

#!/bin/sh

exec </dev/null
ADD="/$(echo $* | grep -o "testnet\|signet")"
URL="https://mempool.space$ADD"

bcli() {
  test "$TEST_GETBLOCK" = 1 && return 1
  bitcoin-cli $*
}

getesplora() {
  END=$1; shift
  #wget -q -O- --header='Accept: application/json' "$URL/api/block/$2/$END"
  curl -s -o- -H='Accept: application/json' "$URL/api/block/$2/$END"
}

ARGS=$(echo $* | sed 's/-[^ ]\+ //')
OPTS=$(echo $* | tr ' ' '\n' | grep '^-')
shift $(echo $OPTS | wc -w)
test "$1" = "getblock" && {
  ARGN=$(echo $ARGS | wc -w)
  if test "$ARGN" = 2
  then
    bcli $OPTS $ARGS 2>/dev/null \
      || { printf '{"tx":'; getesplora txids $ARGS; printf '}'; }
  elif test "$ARGN" = 3
  then
    bcli $OPTS $ARGS 2>/dev/null \
      || { getesplora raw $ARGS | od -v -A n -t x1 \
             | tr -d ' \n'; echo; }
  fi
} || bitcoin-cli $OPTS $ARGS

Works also with -testnet and -signet (though tested only on the former so far).

Recap: all one needs to do is to add --bitcoin-cli PATH/TO/bcli-esplora.sh. Simply export TEST_GETBLOCK=1 before running lightningd --bitcoin-cli ... in order to test esplora-only-getblock-mode (still needs local pruned bitcoind for hello world etc.).

NOTE: The txids part was not tested yet (manually yes, but how to trigger lightningd to ask for it? @cdecker ).