Startup crash when Deploying to Heroku plus question connecting to Umbrel lnd with onion

[trbouma]

I tried deploying to Heroku and connecting to my Umbrel node. I got my LND onion address from the file./home/umbrel/umbrel/tor/data/lnd-grpc/hostname

I also created the HEX cert and macaroon as per your instructions. To create the macaroon I had to go into the lnd contain - docker exec -it lnd /bin/bash and execute:

lncli bakemacaroon onchain:read address:read invoices:read address:write invoices:write

So I think I got all of the env variables correct...

But when I deploy to Heroku - everything builds ok, but crashes on startup - I get these messages in the log

panic: exec: "tor": executable file not found in $PATH
goroutine 1 [running]:
main.main()
/tmp/build_55ed8a5d/lnme.go:101 +0x1556
Process exited with status 2
State changed from starting to crashed
Build succeeded

The buildpack is installed ok (https://github.com/iamashks/heroku-buildpack-tor-proxy.git)

When I restart - same thing. I am using a free dyno (do I need a paid one?)

Do you know:

1. Why is it crashing upon startup? Is it my config info that is causing the crash
2. can I acutally connect to my umbrel lnd via the lnd address above.

Thanks,

Tim

[bumi]

thanks for reporting this! I am not very experienced with the Heroku tor setup, so this is super helpful!

the easy one is question 2: YES, that's what the idea of LnMe is.

can you go go into your dyno (heroku run bash) and do:
$which tor
$ls -al "${HOME}/tor/bin (edit: I had a typo in this one)

[bumi]

can you try this branch: https://github.com/bumi/lnme/tree/configurable-tor-path
and set a LNME_TOR_EXE_PATH environment variable to ~/tor/bin/tor or /app/tor/bin/tor

//cc @crc32 do you have an idea here?

[bumi]

maybe we actually also should embed tor directly. (as mentioned here: https://github.com/cretz/bine#readme )
maybe somebody is willing to try this?

[trbouma]

When I /bin/bash via the console I get this running log which I have to ctrl-c out of

Oct 28 15:02:39.622 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Oct 28 15:02:39.622 [notice] Read configuration file "/app/tor/etc/tor/torrc".
Oct 28 15:02:39.624 [notice] Opening Socks listener on 127.0.0.1:9050
Oct 28 15:02:39.624 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050
Oct 28 15:02:39.624 [notice] Opening Control listener on 127.0.0.1:9051
Oct 28 15:02:39.624 [notice] Opened Control listener connection (ready) on 127.0.0.1:9051
Oct 28 15:02:39.000 [notice] Bootstrapped 0% (starting): Starting
Oct 28 15:02:39.000 [notice] Starting with guard context "default"
Oct 28 15:02:40.000 [notice] Bootstrapped 5% (conn): Connecting to a relay

Once in the shell - doing which tor, I get nothing (meaning not in path?)
others work ok - eg. which pwd returns /usr/bin/pwd

as for ls -al "${HOME}/tor/bin
I get nothing
but ls -al /app/tor/bin I get

drwx------ 2 u14020 dyno 4096 Oct 28 12:14 .
drwx------ 5 u14020 dyno 4096 Oct 28 12:14 ..
-rwx------ 1 u14020 dyno 637 Oct 28 12:14 run_tor_proxy
-rwx------ 1 u14020 dyno 15737408 Oct 28 12:14 tor
-rwx------ 1 u14020 dyno 2574032 Oct 28 12:14 tor-gencert
-rwx------ 1 u14020 dyno 2206744 Oct 28 12:14 tor-print-ed-signing-cert
-rwx------ 1 u14020 dyno 811664 Oct 28 12:14 tor-resolve
-rwx------ 1 u14020 dyno 1375 Oct 28 12:14 torify

So I think everything is there. It seems like there might be a typo in the scripts somewhere that prevents tor from being found.

[bumi]

thanks!
it seems somehow the tor command is not in the $PATH variable and thus it can not be found.
can you try to use that new branch and set the LNME_TOR_EXE_PATH variable?

Just push that branch to your existing heroku app. (e.g. git push heroku configurable-tor-path:master)

[trbouma]

Will do later today!

[trbouma]

Almost there! Site is running, but fails on invoice request. Looks like I need to generate valid cert. See log below. Otherwise, everything else seems to be working. Advise on how to generate the TLS cert for the node would be great!

2021-10-28T16:10:52.432715+00:00 app[web.1]: 2021/10/28 16:10:52 Adding invoice: memo=Tim value=100
2021-10-28T16:10:52.432836+00:00 app[web.1]: 2021/10/28 16:10:52 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, umbrel.local, unix, unixpacket, bufconn, not gvcnwhe7p2xenumcusu7hv3eiunuc53krnglcy5ovmgm6oo5gdxglgyd.onion"

[bumi]

great!

then you are missing the TLS certificate from your LND node OR (likely) you need to tell LND your onion address - somewhere in the config - I am not exactly sure where right now.

[bumi]

was it tlsextradomain ?
we should document that. maybe you can make a PR for the readme when you got it working?

[trbouma]

Yeah, I need to find or generate something. That'll be the homework for tonight. The TLS cert is for 'umbrel.local' but i need one for lnd node which has it own onion address, not the web on for umbrel.local. It can be generated with openssl, but I need to understand how TLS fits into LND. I also need to understand the docker network to make nothing else is in the way.

Otherwise everything else looks like it is working.

[bumi]

ok, but you used the LND tls.cert file? (something like: ~/umbrel/lnd/tls.cert)
you don't need to create one. LND does that for you.

[trbouma]

I think that TLS cert is bound to 'umbrel.local' - when a request is coming from heroko - it is looking for a cert that is bound to the onion address. That is what I discern from the error messge below. So i think I need to generate a cert that is bound to the onion address then it will work.

2021/10/28 17:09:17 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, umbrel.local, unix, unixpacket, bufconn, not gvcnwhe7p2xenumcusu7hv3eiunuc53krnglcy5ovmgm6oo5gdxglgyd.onion"

[bumi]

LND will do that for you or do you have some proxy in front of LND?
I am pretty sure it is a LND config to tell it the onion domain.

[vbrekher]

can you try this branch: https://github.com/bumi/lnme/tree/configurable-tor-path and set a LNME_TOR_EXE_PATH environment variable to ~/tor/bin/tor or /app/tor/bin/tor

//cc @crc32 do you have an idea here?

I had the same problem with path. This solution helped, thanks!

But now I am getting a new error:

Error while dialing socks connect tcp 127.0.0.1:33700->lnnod....onion:9735: unknown error host unreachable"
Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing context deadline exceeded"

[bumi]

@vbrekher what did you set as LNME_LND_ADDRESS is that address reachable when you try to open it in a tor browser for example?

the :9735 indicates that you used the lightning port there, but you should use the LND gRPC port there - typically :10009

[trbouma]

I am pretty sure it's a problem with the certificate. For fun, I tried to set at :9735 and I get this message:

Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing socks connect tcp 127.0.0.1:44930->gvcnwhe7p2xenumcusu7hv3eiunuc53krnglcy5ovmgm6oo5gdxglgyd.onion:9735: unknown error connection refused"

Which is expected, because there is nothing going on at :9735

When set to :10009, I get this message

2021/10/29 11:28:08 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, umbrel.local, unix, unixpacket, bufconn, not gvcnwhe7p2xenumcusu7hv3eiunuc53krnglcy5ovmgm6oo5gdxglgyd.onion"

So, it llooks like heroku is communicating and the certiificate is being enaged on the umbrel node, but the handshake fails because the onion address is not in the certificate. (as it is only umbrel.local, unix, unixpacket, bufconn). We likely need to raise an issue with umbrel to see how we can add the grpc address as part of the certificate.

I know I can communicate with my umbrel node because I can access the dashboard through onion no problem at all.

Anyway, I am really keen to get this working, because I want to add this to my twitter profile, so any help is appreciated!

[bumi]

2021/10/29 11:28:08 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, umbrel.local, unix, unixpacket, bufconn, not gvcnwhe7p2xenumcusu7hv3eiunuc53krnglcy5ovmgm6oo5gdxglgyd.onion"

that error is perfectly fine. And afaik means that you did not tell LND about your onion address (tlsextradomain).
Delete your tls.cert file (needs to be deleted for LND to re-create it), add the onion address in the lnd.conf as tlsextradomain, then LND generates a new tls.cert on the next restart and you can use that to configure LnMe.

Not sure if umbrel exposes the LND gRPC interface by default on the onion address. Maybe you can check that with umbrel?

[trbouma]

OK, I will try this, but will consult the umbrel experts first as it sounds dangerous. Do I need to restart my node to regenerate the TLS cert? In what file do I add the tlsextradomain?
Thankks agaain, tthis has been tottally fun.

[trbouma]

So I've tried on a test lnd node I have on another machine to regenerate the certs and it seems to work. So I will do on my umbrel when I feel comfortable, likely in the next day or so. The final question I have - can an onion address be a valid tlsextradomain?

[bumi]

Yes, best is to ask in the Umbrel forum what's the best way to make the LND gRPC interface available. (because I think that actually should directly work. is that also the onion address that you seen when you go to connect wallet and select "lnd grpc" in umbrel?

[bumi]

I looked at umbrel and you should be able to get the certificate also from the "Connect Wallet" -> "Other" -> "lndconnect grpc (tor)" entry. But there the certificate is base64 encoded and we need it hex encoded. (also the macaroon).

that tor URL in that lndconnect string should actually already be properly configured in the lnd.conf - BUT it seems it is not? so is this maybe a umbrel bug? let me know what you hear from them.

[trbouma]

OK cool - we are getting there. A couple of fruitful avenues to investigate. Thanks!

[trbouma]

On second thought, could you add this an environment parameter in your app and parse accordingly? - If it is non-null use that info to do the connect^ More work on your app, but it saves me a lot of scary work on my side

[bumi]

yep, that's a good idea. that should be possible.
do you want to create an issue?
I currently do not have much time, but maybe somebody can pick it up.

[crc32]

Sorry, just seeing this now. Looking through my Heroku config vars, I see that I did add Tor to the path: /usr/local/bin:/usr/bin:/bin:/app/bin:/app/tor/bin

That may be the solution?

I must have forgotten to add that to the instructions, very sorry.

Another thing to take note of is how the SSL cert works. I use a proxy that handles it.

[trbouma]

Ok thanks - I will try to deploy the latest tonight to see if I can connect. If not successful, I will raise an issue for an enhancement request to see if it is possible to add a variable that takes the lndconnect gRPC from Umbrel and then you figure it out from there. Thanks again for your support - I am really excited to get this working!

[bumi]

@crc32 ah yes, that would work, too. wondering why the buildpack actually does not do that. maybe we should make a PR there?
but for now setting a LNME_TOR_EXE_PATH also works.

@trbouma I tried it with an umbrel and I have the same issue. I also could not connect via Zeus. I am really wondering if that is an umbrel issue (or missing default config setting) - Sadly I don't know umbrel much.

[crc32]

RE: Umbrel, not sure. I have it running on Raspiblitz (Zeus too). I do have Zeus connecting to Umbrel, but I do recall it was a bear to get it to work (and I have no idea what I did to get it working - it was awhile ago).

[trbouma]

Some good progress. I added the onion address to the lnd.conf with the tlsextradomain and regenerated the tls.cert and tls.key by restarting lnd. It looks like I am getting through but now there is some sort of timeout - deadline exceeded. Log entry below

2021-10-30T00:34:53.991544+00:00 app[web.1]: 2021/10/30 00:34:53 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing context deadline exceeded"

So it might be something to do with latency in the onion network - is there a parameter that can be changed?

for more context in the log see below:

2021-10-30T00:33:39.441066+00:00 heroku[router]: at=info method=GET path="/" host=lnme-yow3.herokuapp.com request_id=a6bf647a-7804-45d1-a0df-35eb451f1ef4 fwd="174.93.52.10" dyno=web.1 connect=0ms service=1ms status=200 bytes=7138 protocol=https
2021-10-30T00:33:46.825167+00:00 heroku[router]: at=info method=POST path="/v1/invoices" host=lnme-yow3.herokuapp.com request_id=6a91eada-d4bb-41aa-b8ef-3c653caf442e fwd="174.93.52.10" dyno=web.1 connect=0ms service=1ms status=500 bytes=385 protocol=https
2021-10-30T00:33:46.824888+00:00 app[web.1]: 2021/10/30 00:33:46 Adding invoice: memo=tim value=100
2021-10-30T00:33:46.824938+00:00 app[web.1]: 2021/10/30 00:33:46 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing context deadline exceeded"
2021-10-30T00:34:42.954462+00:00 heroku[router]: at=info method=GET path="/" host=lnme-yow3.herokuapp.com request_id=693d65de-0859-4764-bd3d-5805cd03f78d fwd="174.93.52.10" dyno=web.1 connect=0ms service=1ms status=200 bytes=7138 protocol=https
2021-10-30T00:34:50.843816+00:00 app[web.1]: 2021/10/30 00:34:50 Adding invoice: memo=test value=125
2021-10-30T00:34:53.991255+00:00 heroku[router]: at=info method=POST path="/v1/invoices" host=lnme-yow3.herokuapp.com request_id=c5a275fb-e342-43f6-886a-26a24cb107a2 fwd="174.93.52.10" dyno=web.1 connect=0ms service=3149ms status=500 bytes=385 protocol=https
2021-10-30T00:34:53.991544+00:00 app[web.1]: 2021/10/30 00:34:53 Error creating invoice: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing context deadline exceeded"

[trbouma]

Update. It worked!

[bumi]

it is now working for you, isn't it?

[trbouma]

Kinda. It is working, with the website, but the lightning address stopped working. I have to troubleshoot on my side, as my lnpay.ca cert stopped working inexplicably after a few days. It is also seems trbouma@lnme-yow3.herokuapp.com stopped working as well.

[trbouma]

I tested trbouma@lnme-yow3.herokuapp.com 3 days ago, and it is now not working. But the web app is fine. Tested a few minutes ago.

[bumi]

for lnpay.ca I'd assume it is a DNS issue.
on the heroku domain I've just sent you a few sats.

[crc32]

I also just sent you a few sats to the trbouma@lnme-yow3.herokuapp.com. I tried with Breez wallet at first, but it failed. Ended up going through from my own node, so it could just be a LN liquidity issue.

[trbouma]

@crc32 looks like 10 sats from you came through ok. Not exactly sure what the issue is. I will continue to troubleshoot on my side and keep you posted. We should keep this issue open until we understand what's going on completely.

[crc32]

Only 10? I sent 100 (at 12:48:31 MST). If you want me to send you the payment details, feel free to DM me on telegram (@arceris_btc), and I'll send them to you. It also could be useful to have those from me to help figure this out too.

[trbouma]

I stand corrected. 100 sats came through at 2:48pm ET

[crc32]

Ahh, ok. Just got concerned for a sec!

[trbouma]

I got it all working again. My registrar is a bit goofy. I have to 'park the domain' so I can add the CNAME, let that take, do the heroku cert refresh, then switch to third-party hosting (which then I can no longer access CNAME). It seems to work as I just paid myself 11 sats with trbouma@lnpay.ca

[trbouma]

All good - just got anyname@lnpay.direct working as well!

[bumi]

this can be closed, now?

[trbouma]

Yes, it can be closed @bumi . Thanks for all your help!