Circular not starting with cln 0.12.0 and allow-deprecated-apis=false

[daywalker90]

lightning-cli plugin start /home/bitcoin/circular/circular
{
   "code": -3,
   "message": "/home/bitcoin/circular/circular: exited before replying to init"
}

[devastgh]

lightning-cli plugin start /home/bitcoin/circular/circular
{
   "code": -3,
   "message": "/home/bitcoin/circular/circular: exited before replying to init"
}

Obviously, since it's not prepared for the big msat migration.... I don't run 0.12 yet, but i expect it starts with deprecated apis true. I'm pretty sure glightning is not prepared for that at all... that option is for developers to test their code/plugins for future compatibility, not really for end users to play with.

[daywalker90]

yes it works with the default allow-deprecated-apis=true

[BTCBellyButton]

I was trying your plugin with cln 0.12.0 but I get this slightly different error:

lightning-cli plugin start ~/.lightning/plugins.bak/circular/circular 
{
   "code": -3,
   "message": "/mnt/md0/C-Lightning/plugins.bak/circular/circular: exited before replying to getmanifest"
}

I used to use rebalance.py but it doesn't work with the latest version...

Something seems wrong with this v0.12.0 on the plugins front (see: ElementsProject/lightning#5551)

[giovannizotta]

Hi @BTCBellyButton, please look at #10 (comment)

This could be caused by your node not replying to the plugin initialization in 60s, which is the limit CLN allows for replying to the init function. Circular has to make some expensive operations (in terms of time) to get started, mainly it has to call listchannels. If you are on a Raspi 4 or some low-power hardware it might take too long. Does your node run on not-so-powerful hardware?

[BTCBellyButton]

Does your node run on not-so-powerful hardware?

Hi @giovannizotta, no. It runs on a dedicated HP laptop with 12GB RAM and it returns that error message almost instantly... :(

I never had a problem with other plugins until the v0.12.0...

[giovannizotta]

Interesting, let's do some checks @BTCBellyButton:

• is the file executable? chmod +x /path/to/circular
• do you have allow-deprecated-apis=true?
• in the circular directory (which is inside your lightningdir) there should be a file named stderr-123456.log. There should be some clue there about what's wrong.

If you still have problems feel free to join the Telegram group which you find at the bottom of the README, it should be easier to talk there

[BTCBellyButton]

is the file executable?

Yes: -rwxrwxr-x 1 go go 12405306 Sep 27 18:36 circular*

do you have allow-deprecated-apis=true?

It should be the default according to https://lightning.readthedocs.io/lightning-cli.1.html?highlight=allow-deprecated-apis#options.
Unless it changed with the new CLN?

lightning-cli: -allow-deprecated-apis=true: unrecognized option

file named stderr-123456.log.

There's no log. Note that the plugin don't seem to even load. I get the error trying to start it with lightning-cli plugin start ~/.lightning/plugins.bak/circular/circular

[BTCBellyButton]

FYI ElementsProject/lightning#5638 a lot of problems with the new releases of CLN and plugins, apparently.
I'll join the telegram group.

[giovannizotta]

It should be the default according to https://lightning.readthedocs.io/lightning-cli.1.html?highlight=allow-deprecated-apis#options.
Unless it changed with the new CLN?

Yes it should be the default, I was saying that just in case you were running false. It is a lightningd flag tho (you set it when you start the node), you do not set it with lightning-cli afaik.

Have you built the plugin yourself or are you using a binary from the release? It would probably be helpful to run in debug mode to find out what's wrong. There is an example in the readme for how to run circular in debug mode (but you need to load it at startup in order to do that, can't do it dynamically)

[BTCBellyButton]

Have you built the plugin yourself or are you using a binary from the release?

I am using the binary.

First I explicitly added --allow-deprecated-apis=true:

CGroup: /system.slice/lightningd.service
             ├─1146991 /usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --log-file=/mnt/md0/C-Lightning/log --pid-file=/run/lightningd/lightningd.pid --allow-deprecated-apis=true

But that didn't help.

Then I tried to run it in debug mode by adding

plugin=/home/go/.lightning/plugins.bak/circular/circular --circular-graph-refresh=5 --circular-peer-refresh=60 --circular-liquidity-refresh=120 --circular-save-stats=false --log-level=debug:plugin-circular

to the CLN config file but this is the result when restarting lightnind.service:

sudo systemctl restart lightningd.service 
Job for lightningd.service failed because the control process exited with error code.
See "systemctl status lightningd.service" and "journalctl -xe" for details.

Details:

Sep 28 09:26:15 BTCPayServer systemd[1]: Starting C-Lightning daemon...
-- Subject: A start job for unit lightningd.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit lightningd.service has begun execution.
-- 
-- The job identifier is 204344.
Sep 28 09:26:15 BTCPayServer lightningd[1150266]: lightningd: /mnt/md0/C-Lightning/config line 27: plugin: Failed to register /home/go/.lightning/plugins.bak/circular/circular --circular-graph-refresh=5 --circular-peer-refresh=60>
Sep 28 09:26:15 BTCPayServer systemd[1]: lightningd.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit lightningd.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Sep 28 09:26:15 BTCPayServer systemd[1]: lightningd.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit lightningd.service has entered the 'failed' state with result 'exit-code'.
Sep 28 09:26:15 BTCPayServer systemd[1]: Failed to start C-Lightning daemon.
-- Subject: A start job for unit lightningd.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit lightningd.service has finished with a failure.
-- 
-- The job identifier is 204344 and the job result is failed.

[daywalker90]

Then I tried to run it in debug mode by adding

plugin=/home/go/.lightning/plugins.bak/circular/circular --circular-graph-refresh=5 --circular-peer-refresh=60 --circular-liquidity-refresh=120 --circular-save-stats=false --log-level=debug:plugin-circular

to the CLN config file but this is the result when restarting lightnind.service:

That's not how you do it. It should be like this in the config file:

plugin=/home/go/.lightning/plugins.bak/circular/circular
circular-graph-refresh=5
circular-peer-refresh=60
circular-liquidity-refresh=120
circular-save-stats=false
log-level=debug:plugin-circular

[BTCBellyButton]

With

Same result:

sudo systemctl restart lightningd.service 
Job for lightningd.service failed because the control process exited with error code.
See "systemctl status lightningd.service" and "journalctl -xe" for details.

But this is weird:

Sep 28 09:55:42 BTCPayServer lightningd[1155908]: lightningd: /mnt/md0/C-Lightning/config line 28: plugin: Failed to register /home/go/.lightning/plugins.bak/circular/circular : No such file or directory
Sep 28 09:55:42 BTCPayServer systemd[1]: lightningd.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit lightningd.service has exited.

"No such file or directory" ?!?

Maybe this v0.12.0 version messed up the plugin management... Someone reported problems also with backup.py...
I tend to believe CLN has a bug that is not evident IF you run a plugin at startup via the config file AND until you actually try to use it OR if you use the "plugins directory" method and in that case CLN 0.12.0 doesn't even start (my open issue on CLN).

[daywalker90]

hmm idk about all that but what i do is i don't have the plugin= line in my config. Instead i have the --plugin option with the path in my systemd service file.

[BTCBellyButton]

I can try and move it to the systemd file but I don't think it will make a difference...beside having to reload the daemon...
Let's see.

[BTCBellyButton]

So I put it in lightnind.service:

CLN starts but the plugin doesn't so you can't notice unless you go and try to use it or run lightning-cli plugin list

Sep 28 10:41:46 BTCPayServer systemd[1]: Starting C-Lightning daemon...
-- Subject: A start job for unit lightningd.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit lightningd.service has begun execution.
-- 
-- The job identifier is 206585.
Sep 28 10:41:46 BTCPayServer lightningd[1164978]: /mnt/md0/C-Lightning/plugins.bak/circular/circular: 1: Syntax error: end of file unexpected
Sep 28 10:42:13 BTCPayServer systemd[1]: Started C-Lightning daemon.
-- Subject: A start job for unit lightningd.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit lightningd.service has finished successfully.
-- 
-- The job identifier is 206585.

LOG:

2022-09-28T14:41:46.676Z INFO    plugin-circular: Killing plugin: exited before replying to getmanifest
2022-09-28T14:41:59.503Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2022-09-28T14:42:13.227Z INFO    plugin-autoclean: autocleaning every 1800 seconds
2022-09-28T14:42:13.230Z INFO    lightningd: --------------------------------------------------
2022-09-28T14:42:13.230Z INFO    lightningd: Server started with public key 0228cc784d015731cc0c5dc2163c8bb856f59feb3234ce813edeccf2e26733c32d, alias BitcoinBellyButton (color #0228cc) and lightningd 0.12.0
2022-09-28T14:42:14.800Z UNUSUAL plugin-plugin.js: --- Starting the cl-rest server ---
2022-09-28T14:42:14.800Z UNUSUAL plugin-plugin.js: --- cl-rest api server is ready and listening on port: 3001 ---
2022-09-28T14:42:14.800Z UNUSUAL plugin-plugin.js: --- cl-rest doc server is ready and listening on port: 4001 ---

I am pretty sure this complete lack of consistency in the behavior depending on how you try to start a plugin ("lightningd.service" vs "plugin directory" vs "config file" is due to a bug in CLN 0.12.0...and it's not just yours...

[daywalker90]

oh for me it works. The non-default developer setting allow-deprecated-apis=false was the topic of this issue.

[BTCBellyButton]

Maybe as a developer you have some different setting/dependency... I don't know...
I started having problems with plugins since I tried to upgrade to this release...