clnrest: refactoring

doc/developers-guide/app-development/rest.md

@@ -3,15 +3,16 @@ title: "REST APIs"
 slug: "rest"
 hidden: false
 createdAt: "2023-09-05T09:54:01.784Z"
-updatedAt: "2023-09-05T09:54:01.784Z"
+updatedAt: "2023-10-13T09:54:01.784Z"
 ---
 
 # CLNRest
 
-CLNRest is a lightweight Python-based core lightning plugin that transforms RPC calls into a REST service. By generating REST API endpoints, it enables the execution of Core Lightning's RPC methods behind the scenes and provides responses in JSON format.
-
-A complete documentation for the REST interface is available at [REST API REFERENCE](ref:get_list_methods_resource). 
+CLNRest is a lightweight Python-based built-in Core Lightning plugin (from v23.08) that transforms RPC calls into a REST service. 
+It also broadcasts Core Lightning notifications to listeners connected to its websocket server. By generating REST API endpoints, 
+it enables the execution of Core Lightning's RPC methods behind the scenes and provides responses in JSON format.
 
+An online demo for the REST interface is available at [REST API REFERENCE](ref:get_list_methods_resource).
 
 > 📘 Pro-tip
 > 
@@ -34,22 +35,34 @@ A complete documentation for the REST interface is available at [REST API REFERE
 
 ## Installation
 
-Install required packages with `pip install json5 flask flask_restx gunicorn pyln-client flask-socketio gevent gevent-websocket` or `pip install -r requirements.txt`.
+The plugin is built-in with Core Lightning but its python dependencies are not, and must be installed separately.
+Install required packages with `pip install -r plugins/clnrest/requirements.txt`.
+
+Note: if you have the older c-lightning-rest plugin, you can configure Core Lightning with `disable-plugin=clnrest.py` option 
+to avoid any conflict with this one. Of course, you could use this one instead!
 
-Note: if you have the older c-lightning-rest plugin, you can use `disable-plugin clnrest.py` to avoid any conflict with this one.  Of course, you could use this one instead!
 
 ## Configuration
 
 If `rest-port` is not specified, the plugin will disable itself.
 
 - --rest-port: Sets the REST server port to listen to (3010 is common)
+
 - --rest-protocol: Specifies the REST server protocol. Default is HTTPS.
+
 - --rest-host: Defines the REST server host. Default is 127.0.0.1.
-- --rest-certs: Defines the path for HTTPS cert & key. Default path is same as RPC file path to utilize gRPC's client certificate. If it is missing at the configured location, new identity will be generated.
+
+- --rest-certs: Defines the path for HTTPS cert & key. Default path is same as RPC file path to utilize gRPC's client certificate.
+If it is missing at the configured location, new identity will be generated.
+
 - --rest-csp: Creates a whitelist of trusted content sources that can run on a webpage and helps mitigate the risk of attacks. 
-Default CSP is set as `default-src 'self'; font-src 'self'; img-src 'self' data:; frame-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline';`.
-Example CSP: `rest-csp=default-src 'self'; font-src 'self'; img-src 'self'; frame-src 'self'; style-src 'self'; script-src 'self';`.
-- --rest-cors-origins:   Define multiple origins which are allowed to share resources on web pages to a domain different from the one that served the web page. Default is `*` which allows all origins. Example to define multiple origins:
+Default CSP:
+`default-src 'self'; font-src 'self'; img-src 'self' data:; frame-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline';`
+Example CSP:
+`rest-csp=default-src 'self'; font-src 'self'; img-src 'self'; frame-src 'self'; style-src 'self'; script-src 'self';`.
+
+- --rest-cors-origins: Define multiple origins which are allowed to share resources on web pages to a domain different from the 
+one that served the web page. Default is `*` which allows all origins. Example to define multiple origins:
 
 ```
 rest-cors-origins=https://localhost:5500
@@ -60,21 +73,31 @@ rest-cors-origins=https?://127.0.0.1:([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65
 
 ## Server
 
-With the default configurations, the Swagger user interface will be available at https://127.0.0.1:3010/. The POST method requires `rune` header for authorization.
+With the default configurations, the Swagger user interface will be available at https://127.0.0.1:3010/. 
+The POST method requires `rune` header for authorization.
 
-- A new `rune` can be created via [createrune](https://docs.corelightning.org/reference/lightning-createrune) or the list of existing runes can be retrieved with [listrunes](https://docs.corelightning.org/reference/lightning-listrunes) command.
+- A new `rune` can be created via [createrune](https://docs.corelightning.org/reference/lightning-createrune) or the list of 
+existing runes can be retrieved with [listrunes](https://docs.corelightning.org/reference/lightning-listrunes) command.
 
-Note: in version v23.08, a parameter `Nodeid` was required to be the id of the node we're talking to (see `id (pubkey)` received from [getinfo](https://docs.corelightning.org/reference/lightning-getinfo) ).  You can still send this for backwards compatiblity, but it is completely ignored.
+Note: in version v23.08, a parameter `Nodeid` was required to be the id of the node we're talking to (see `id (pubkey)` received 
+from [getinfo](https://docs.corelightning.org/reference/lightning-getinfo)). You can still send this for backwards compatiblity, 
+but it is completely ignored.
 
 ### cURL
 Example curl command for POST will also require a `rune` header like below:
-    `curl -k -X POST 'https://127.0.0.1:3010/v1/getinfo' -H 'Rune: <node-rune>'`
+    `curl -k -X POST 'https://localhost:3010/v1/getinfo' -H 'Rune: <node-rune>'`
 
 With `-k` or `--insecure` option curl proceeds with the connection even if the SSL certificate cannot be verified.
 This option should be used only when testing with self signed certificate.
 
 ## Websocket Server
-Websocket server is available at `https://127.0.0.1:3010`. clnrest queues up notifications received for a second then broadcasts them to listeners.
+Websocket server is available at `https://127.0.0.1:3010`. clnrest queues up notifications received for a second 
+then broadcasts them to all listeners. 
+
+This websocket server requires a `rune` with at least `readonly` access for authorization. The default method used
+for current validation is `listclnrest-notifications`. User can either provided a rune with minimum `readonly`
+access or can create a new special purpose rune, only for websocket validation, with restrictions='[["method=listclnrest-notifications"]]'.
+The client will only receive notifications if `rune`, provided in headers, allows it.
 
 ### Websocket client examples
 

doc/getting-started/getting-started/installation.md

@@ -27,7 +27,7 @@ You will need some Python packages if you want to use clnrest.  Unfortunately th
 
 ```
 sudo apt-get install python3-json5 python3-flask python3-gunicorn
-pip3 install --user flask_restx pyln-client
+pip3 install --user flask-cors flask_restx pyln-client flask-socketio gevent gevent-websocket
 ```
 
 If you're on a different distribution or OS, you can compile the source by following the instructions from [Installing from Source](<>).

plugins/clnrest/clnrest.py

@@ -60,7 +60,6 @@ def broadcast_from_message_queue():
             msg = msgq.get()
             if msg is None:
                 return
-            plugin.log(f"Emitting message: {msg}", "debug")
             socketio.emit("message", msg)
         # Wait for a second after processing all items in the queue
         time.sleep(1)
@@ -81,8 +80,8 @@ def handle_message(message):
 def ws_connect():
     try:
         plugin.log("Client Connecting...", "debug")
-        is_valid_rune = verify_rune(plugin, request)
-
+        rune = request.headers.get("rune", None)
+        is_valid_rune = verify_rune(plugin, rune, "listclnrest-notifications", None)
         if "error" in is_valid_rune:
             # Logging as error/warn emits the event for all clients
             plugin.log(f"Error: {is_valid_rune}", "info")
@@ -137,13 +136,14 @@ def set_application_options(plugin):
     else:
         cert_file = Path(f"{CERTS_PATH}/client.pem")
         key_file = Path(f"{CERTS_PATH}/client-key.pem")
-        if not cert_file.is_file() or not key_file.is_file():
-            plugin.log(f"Certificate not found at {CERTS_PATH}. Generating a new certificate!", "debug")
-            generate_certs(plugin, CERTS_PATH)
         try:
+            if not cert_file.is_file() or not key_file.is_file():
+                plugin.log(f"Certificate not found at {CERTS_PATH}. Generating a new certificate!", "debug")
+                generate_certs(plugin, REST_HOST, CERTS_PATH)
             plugin.log(f"Certs Path: {CERTS_PATH}", "debug")
         except Exception as err:
             raise Exception(f"{err}: Certificates do not exist at {CERTS_PATH}")
+
         # Assigning only one worker due to added complexity between gunicorn's multiple worker process forks
         # and websocket connection's persistance with a single worker.
         options = {
@@ -164,8 +164,8 @@ def __init__(self, app, options=None):
         from utilities.shared import REST_PROTOCOL, REST_HOST, REST_PORT
         self.application = app
         self.options = options or {}
-        plugin.log(f"REST server running at {REST_PROTOCOL}://{REST_HOST}:{REST_PORT}", "info")
         super().__init__()
+        plugin.log(f"REST server running at {REST_PROTOCOL}://{REST_HOST}:{REST_PORT}", "info")
 
     def load_config(self):
         config = {key: value for key, value in self.options.items()
@@ -216,7 +216,7 @@ def on_any_notification(request, **kwargs):
         # A plugin which subscribes to shutdown is expected to exit itself.
         sys.exit(0)
     else:
-        msgq.put(str(kwargs))
+        msgq.put(kwargs)
 
 
 try:

plugins/clnrest/utilities/generate_certs.py

@@ -5,98 +5,65 @@
 from cryptography.hazmat.primitives import serialization, hashes
 from cryptography.hazmat.primitives.asymmetric import ec
 import datetime
+from utilities.shared import validate_ip4
 
 
-def generate_ca_cert(certs_path):
-    # Generate CA Private Key
-    ca_private_key = ec.generate_private_key(ec.SECP256R1())
+def save_cert(entity_type, cert, private_key, certs_path):
+    """Serialize and save certificates and keys.
+    `entity_type` is either "ca", "client" or "server"."""
+    with open(os.path.join(certs_path, f"{entity_type}.pem"), "wb") as f:
+        f.write(cert.public_bytes(serialization.Encoding.PEM))
+    with open(os.path.join(certs_path, f"{entity_type}-key.pem"), "wb") as f:
+        f.write(private_key.private_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PrivateFormat.PKCS8,
+            encryption_algorithm=serialization.NoEncryption()))
 
-    # Generate CA Public Key
-    ca_public_key = ca_private_key.public_key()
 
-    # Generate CA Certificate
-    ca_subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, u"cln Root REST CA")])
+def create_cert_builder(subject_name, issuer_name, public_key, rest_host):
+    list_sans = [x509.DNSName("cln"), x509.DNSName("localhost")]
+    if validate_ip4(rest_host) is True:
+        list_sans.append(x509.IPAddress(ipaddress.IPv4Address(rest_host)))
 
-    ca_cert = (
+    return (
         x509.CertificateBuilder()
-        .subject_name(ca_subject)
-        .issuer_name(ca_subject)
-        .public_key(ca_public_key)
+        .subject_name(subject_name)
+        .issuer_name(issuer_name)
+        .public_key(public_key)
         .serial_number(x509.random_serial_number())
         .not_valid_before(datetime.datetime.utcnow())
         .not_valid_after(datetime.datetime.utcnow() + datetime.timedelta(days=10 * 365))  # Ten years validity
-        .add_extension(x509.SubjectAlternativeName([x509.DNSName(u"cln"), x509.DNSName(u'localhost'), x509.IPAddress(ipaddress.IPv4Address(u'127.0.0.1'))]), critical=False)
-        .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
-        .sign(ca_private_key, hashes.SHA256())
+        .add_extension(x509.SubjectAlternativeName(list_sans), critical=False)
     )
 
-    # Create the certs directory if it does not exist
-    os.makedirs(certs_path, exist_ok=True)
-
-    # Serialize CA certificate and write to disk
-    with open(os.path.join(certs_path, "ca.pem"), "wb") as f:
-        f.write(ca_cert.public_bytes(serialization.Encoding.PEM))
-
-    # Serialize and save the private key to a PEM file (CA)
-    with open(os.path.join(certs_path, "ca-key.pem"), "wb") as f:
-        f.write(ca_private_key.private_bytes(
-            encoding=serialization.Encoding.PEM,
-            format=serialization.PrivateFormat.PKCS8,
-            encryption_algorithm=serialization.NoEncryption()
-        ))
-
-    return ca_subject, ca_private_key
-
-
-def generate_client_server_certs(certs_path, ca_subject, ca_private_key):
-    # Generate Server and Client Private Keys
-    server_private_key = ec.generate_private_key(ec.SECP256R1())
-    client_private_key = ec.generate_private_key(ec.SECP256R1())
 
-    # Generate Server and Client Public Keys
-    server_public_key = server_private_key.public_key()
-    client_public_key = client_private_key.public_key()
-
-    # Generate Server and Client Certificates
-    for entity_type in ["server", "client"]:
-        public_key = server_public_key if entity_type == "server" else client_public_key
+def generate_cert(entity_type, ca_subject, ca_private_key, rest_host, certs_path):
+    # Generate Key pair
+    private_key = ec.generate_private_key(ec.SECP256R1())
+    public_key = private_key.public_key()
 
+    # Generate Certificates
+    if isinstance(ca_subject, x509.Name):
         subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, f"cln rest {entity_type}")])
-
+        cert_builder = create_cert_builder(subject, ca_subject, public_key, rest_host)
+        cert = cert_builder.sign(ca_private_key, hashes.SHA256())
+    else:
+        ca_subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, u"cln Root REST CA")])
+        ca_private_key, ca_public_key = private_key, public_key
+        cert_builder = create_cert_builder(ca_subject, ca_subject, ca_public_key, rest_host)
         cert = (
-            x509.CertificateBuilder()
-            .subject_name(subject)
-            .issuer_name(ca_subject)
-            .public_key(public_key)
-            .serial_number(x509.random_serial_number())
-            .not_valid_before(datetime.datetime.utcnow())
-            .not_valid_after(datetime.datetime.utcnow() + datetime.timedelta(days=10 * 365))  # Ten years validity
-            .add_extension(x509.SubjectAlternativeName([x509.DNSName(u"cln"), x509.DNSName(u'localhost'), x509.IPAddress(ipaddress.IPv4Address(u'127.0.0.1'))]), critical=False)
+            cert_builder
+            .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
             .sign(ca_private_key, hashes.SHA256())
         )
 
-        # Serialize Server and Client certificates and write to disk
-        with open(os.path.join(certs_path, f"{entity_type}.pem"), "wb") as f:
-            f.write(cert.public_bytes(serialization.Encoding.PEM))
-
-    # Serialize Private Keys (Server)
-    with open(os.path.join(certs_path, "server-key.pem"), "wb") as f:
-        f.write(server_private_key.private_bytes(
-            encoding=serialization.Encoding.PEM,
-            format=serialization.PrivateFormat.PKCS8,
-            encryption_algorithm=serialization.NoEncryption()
-        ))
-
-    # Serialize Private Keys (Client)
-    with open(os.path.join(certs_path, "client-key.pem"), "wb") as f:
-        f.write(client_private_key.private_bytes(
-            encoding=serialization.Encoding.PEM,
-            format=serialization.PrivateFormat.PKCS8,
-            encryption_algorithm=serialization.NoEncryption()
-        ))
+    os.makedirs(certs_path, exist_ok=True)
+    save_cert(entity_type, cert, private_key, certs_path)
+    return ca_subject, ca_private_key
 
 
-def generate_certs(plugin, certs_path):
-    ca_subject, ca_private_key = generate_ca_cert(certs_path)
-    generate_client_server_certs(certs_path, ca_subject, ca_private_key)
+def generate_certs(plugin, rest_host, certs_path):
+    ca_subject, ca_private_key = generate_cert("ca", None, None, rest_host, certs_path)
+    generate_cert("client", ca_subject, ca_private_key, rest_host, certs_path)
+    generate_cert("server", ca_subject, ca_private_key, rest_host, certs_path)
     plugin.log(f"Certificates Generated!", "debug")

plugins/clnrest/utilities/rpc_plugin.py

@@ -8,4 +8,4 @@
 plugin.add_option(name="rest-host", default="127.0.0.1", description="REST server host", opt_type="string", deprecated=False)
 plugin.add_option(name="rest-port", default=None, description="REST server port to listen", opt_type="int", deprecated=False)
 plugin.add_option(name="rest-cors-origins", default="*", description="Cross origin resource sharing origins", opt_type="string", deprecated=False, multi=True)
-plugin.add_option(name="rest-csp", default="default-src 'self'; font-src 'self'; img-src 'self' data:; frame-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline';", description="Content security policy (CSP) for the server", opt_type="string", deprecated=False, multi=True)
+plugin.add_option(name="rest-csp", default="default-src 'self'; font-src 'self'; img-src 'self' data:; frame-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline';", description="Content security policy (CSP) for the server", opt_type="string", deprecated=False, multi=False)

plugins/clnrest/utilities/rpc_routes.py

@@ -23,7 +23,7 @@ def get(self):
             return response
 
         except Exception as err:
-            plugin.log(f"Error: {err}", "error")
+            plugin.log(f"Error: {err}", "info")
             return json5.loads(str(err)), 500
 
 
@@ -37,25 +37,25 @@ class RpcMethodResource(Resource):
     def post(self, rpc_method):
         """Call any valid core lightning method (check list-methods response)"""
         try:
-            is_valid_rune = verify_rune(plugin, request)
+            rune = request.headers.get("rune", None)
+            rpc_method = request.view_args.get("rpc_method", None)
+            rpc_params = request.form.to_dict() if not request.is_json else request.get_json() if len(request.data) != 0 else {}
 
-            if "error" in is_valid_rune:
-                plugin.log(f"Error: {is_valid_rune}", "error")
-                raise Exception(is_valid_rune)
+            try:
+                is_valid_rune = verify_rune(plugin, rune, rpc_method, rpc_params)
+                if "error" in is_valid_rune:
+                    plugin.log(f"Error: {is_valid_rune}", "error")
+                    raise Exception(is_valid_rune)
 
-        except Exception as err:
-            return json5.loads(str(err)), 401
+            except Exception as err:
+                return json5.loads(str(err)), 401
 
-        try:
-            if request.is_json:
-                if len(request.data) != 0:
-                    payload = request.get_json()
-                else:
-                    payload = {}
-            else:
-                payload = request.form.to_dict()
-            return call_rpc_method(plugin, rpc_method, payload), 201
+            try:
+                return call_rpc_method(plugin, rpc_method, rpc_params), 201
+
+            except Exception as err:
+                plugin.log(f"Error: {err}", "info")
+                return json5.loads(str(err)), 500
 
         except Exception as err:
-            plugin.log(f"Error: {err}", "error")
-            return json5.loads(str(err)), 500
+            return f"Unable to parse request: {err}", 500