Webhooks cann't trigger AppAPI events

[PatrickPromitzer]

I looked through the documentation, and can't find a solution.

Problem

The AppAPI docker image in the example is using an secret for connecting to the docker image.
Without adding a auth_method, a webhook can be registered but raise an error after the first call.

# will not be called at all
nc.webhooks.register(
    http_method="POST",
    uri="/basic_file_checker",
    event="OCP\\Files\\Events\\Node\\NodeWrittenEvent"
)

# will raise a "401 Unauthorized" error
nc.webhooks.register(
    http_method="POST",
    uri="http://127.0.0.1:2981/basic_file_checker",
    event="OCP\\Files\\Events\\Node\\NodeWrittenEvent"
)

I can't find an example for a webhook call with an auth_method (or guess the name of the auth_method)

Steps/Code to Reproduce

same setup as #352

Python script in
lib/main.py

import pathlib
import time
import traceback
import requests

from contextlib import asynccontextmanager
from fastapi import FastAPI, BackgroundTasks

from nc_py_api import NextcloudApp
from nc_py_api.ex_app import AppAPIAuthMiddleware, LogLvl, run_app, set_handlers, nc_app
from fastapi import Depends
from typing import Annotated

LOG_LOCAL_HOST_STRING = "127.0.0.1"
WEBHOOK_HOST = "http://127.0.0.1:2981"


@asynccontextmanager
async def lifespan(app: FastAPI):
    set_handlers(app, enabled_handler)
    yield


APP = FastAPI(lifespan=lifespan)
APP.add_middleware(AppAPIAuthMiddleware)


@APP.post("/basic_file_checker")
async def basic_file_checker(
        files: dict,
        # files: ActionFileInfoEx,
        nc: Annotated[NextcloudApp, Depends(nc_app)],
        background_tasks: BackgroundTasks,
):
    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"ID": "basic_file_checker", "called": "true"})
    # background_tasks.add_task(file_event_handler, files, nc)


def enabled_handler(enabled: bool, nc: NextcloudApp) -> str:
    print(f"enabled={enabled}")

    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"ID": "enabled_handler", "enabled": enabled})
    try:
        if enabled:
            event_list = [
                #"OCP\\Files\\Events\\Node\\BeforeNodeWrittenEvent",
                "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
                "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
            ]
            for event_item in event_list:
                nc.webhooks.register(
                    http_method="POST",
                    uri=f"{WEBHOOK_HOST}/basic_file_checker",
                    event=event_item
                )
            #nc.events_listener.register(
            #    event_type="node_event",
            #    callback_url="/basic_file_checker"
            #    # event_subtypes=[
            #    #    "NodeWrittenEvent",
            #    #    "NodeCreatedEvent",
            #    #    "NodeTouchedEvent",
            #    #    "NodeRenamedEvent",
            #    #    "NodeCopiedEvent",
            #    #    "NodeDeletedEvent",
            #    # ]
            #)
            # https://docs.nextcloud.com/server/latest/developer_manual/basics/events.html#ocp-files-events-node-nodecreatedevent
            # FileChecker.create_nc_tags(nc=nc)
            wh_list = nc.webhooks.get_list()
            wh_list = [i.uri for i in wh_list]
            response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"id": "webhook", "uri": wh_list})
        nc.log(LogLvl.INFO, f"App enabled: {enabled}")
    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": error_string})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")

    return ""


if __name__ == "__main__":
    pathlib.Path("temp").mkdir(parents=True, exist_ok=True)
    run_app(
        "main:APP",
        log_level="trace",
    )

after that, make an edit in a file.

[PatrickPromitzer]

Note:

nc_py_api/CHANGELOG.md

Line 9 in 1e667d4

- ExApps: `nc_py_api.ex_app.events_listener.EventsListener` was removed in favor of `nc_py_api.webhooks`. #348

nc_py_api.ex_app.events_listener.EventsListener was the workaround to this problem.

[bigcat88]

webhooks_listener code that calls ExApp is located here:

https://github.com/nextcloud/server/blob/28df049b995608d20ae01f8a4f322bc46c9a5d4d/apps/webhook_listeners/lib/BackgroundJobs/WebhookCall.php#L61-L80

[bigcat88]

I can check this later with your example(will try today, but can't promise), we currently use this only in flow ExApp, but it's code is hard to read.

[PatrickPromitzer]

I checked my code and the nextcloud responses.
I don't know what I changed, but for unknown reasons it works now.

Here the code.

import pathlib
import time
import traceback
import requests

from contextlib import asynccontextmanager
from fastapi import FastAPI, BackgroundTasks

from nc_py_api import NextcloudApp
from nc_py_api.ex_app import AppAPIAuthMiddleware, LogLvl, run_app, set_handlers, nc_app
from fastapi import Depends
from typing import Annotated

LOG_LOCAL_HOST_STRING = "127.0.0.1"
WEBHOOK_HOST = ""  # "http://127.0.0.1:2981"

possible_note_events = [
    "OCA\\Forms\\Events\\FormSubmittedEvent",
    "OCA\\Tables\\Event\\RowAddedEvent",
    "OCA\\Tables\\Event\\RowDeletedEvent",
    "OCA\\Tables\\Event\\RowUpdatedEvent",
    "OCP\\Calendar\\Events\\CalendarObjectCreatedEvent",
    "OCP\\Calendar\\Events\\CalendarObjectDeletedEvent",
    "OCP\\Calendar\\Events\\CalendarObjectMovedEvent",
    "OCP\\Calendar\\Events\\CalendarObjectMovedToTrashEvent",
    "OCP\\Calendar\\Events\\CalendarObjectRestoredEvent",
    "OCP\\Calendar\\Events\\CalendarObjectUpdatedEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeCreatedEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeTouchedEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeWrittenEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeReadEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeDeletedEvent",
    "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
    "OCP\\Files\\Events\\Node\\NodeTouchedEvent",
    "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
    "OCP\\Files\\Events\\Node\\NodeDeletedEvent",
    "OCP\\Files\\Events\\Node\\NodeCopiedEvent",
    # "OCP\\Files\\Events\\Node\\NodeRestoredEvent",
    "OCP\\Files\\Events\\Node\\NodeRenamedEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeCopiedEvent",
    # "OCP\\Files\\Events\\Node\\BeforeNodeRestoredEvent",
    "OCP\\Files\\Events\\Node\\BeforeNodeRenamedEvent",
    "OCP\\SystemTag\\MapperEvent"
]


def file_event_handler(event_data: dict, nc: NextcloudApp):
    try:
        pass

    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": str(error_string)})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")
        raise e


@asynccontextmanager
async def lifespan(app: FastAPI):
    set_handlers(app, enabled_handler)
    yield


APP = FastAPI(lifespan=lifespan)
APP.add_middleware(AppAPIAuthMiddleware)


@APP.post("/basic_file_checker")
async def basic_file_checker(
        event_data: dict,
        # files: ActionFileInfoEx,
        nc: Annotated[NextcloudApp, Depends(nc_app)],
        background_tasks: BackgroundTasks,
):
    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                             json={"ID": "basic_file_checker", "called": "true", "data": event_data})
    background_tasks.add_task(file_event_handler, event_data, nc)


def enabled_handler(enabled: bool, nc: NextcloudApp) -> str:
    print(f"enabled={enabled}")

    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                             json={"ID": "enabled_handler", "enabled": enabled})
    try:
        if enabled:
            event_list = [
                #"OCP\\Files\\Events\\Node\\BeforeNodeCreatedEvent",
                "OCP\\Files\\Events\\Node\\NodeCreatedEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeTouchedEvent",
                "OCP\\Files\\Events\\Node\\NodeTouchedEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeWrittenEvent",
                "OCP\\Files\\Events\\Node\\NodeWrittenEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeReadEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeDeletedEvent",
                "OCP\\Files\\Events\\Node\\NodeDeletedEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeCopiedEvent",
                "OCP\\Files\\Events\\Node\\NodeCopiedEvent",

                #"OCP\\Files\\Events\\Node\\BeforeNodeRenamedEvent",
                "OCP\\Files\\Events\\Node\\NodeRenamedEvent",
            ]
            for event_item in event_list:
                if not event_item.startswith("OCP\\Files\\Events\\Node\\"):
                    continue
                nc.webhooks.register(
                    http_method="POST",
                    uri=f"{WEBHOOK_HOST}/basic_file_checker",
                    event=event_item
                )
            # nc.events_listener.register(
            #    event_type="node_event",
            #    callback_url="/basic_file_checker"
            #    # event_subtypes=[
            #    #    "NodeWrittenEvent",
            #    #    "NodeCreatedEvent",
            #    #    "NodeTouchedEvent",
            #    #    "NodeRenamedEvent",
            #    #    "NodeCopiedEvent",
            #    #    "NodeDeletedEvent",
            #    # ]
            # )
            # https://docs.nextcloud.com/server/latest/developer_manual/basics/events.html#ocp-files-events-node-nodecreatedevent
            # FileChecker.create_nc_tags(nc=nc)
            wh_list = nc.webhooks.get_list()
            wh_list = [i.__dict__ for i in wh_list]
            response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                                     json={"id": "webhook list", "data": wh_list})
        nc.log(LogLvl.INFO, f"App enabled: {enabled}")
    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": error_string})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")

    return ""


if __name__ == "__main__":
    pathlib.Path("temp").mkdir(parents=True, exist_ok=True)
    run_app(
        "main:APP",
        log_level="trace",
    )

[PatrickPromitzer]

I found the problem again @bigcat88

The webhook call has no permissions

def file_event_handler(event_data: dict, nc: NextcloudApp):
    try:
        nc_file = nc.files.by_id(str(event_data["event"]["node"]["id"]))

    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": str(error_string)})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")
        raise e

Traceback (most recent call last):
  File "/app/lib/main.py", line 53, in file_event_handler
    nc_file = nc.files.by_id(str(event_data["event"]["node"]["id"]))
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/files.py", line 65, in by_id
    result = self.find(req=["eq", "fileid", file_id])
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/files.py", line 86, in find
    return lf_parse_webdav_response(self._session.cfg.dav_url_suffix, webdav_response, request_info)
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/_files.py", line 345, in lf_parse_webdav_response
    return _parse_records(dav_url_suffix, _webdav_response_to_records(webdav_res, info), response_type)
                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/_files.py", line 349, in _webdav_response_to_records
    check_error(webdav_res, info=info)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/_exceptions.py", line 65, in check_error
    raise NextcloudException(status_code, reason=codes(status_code).phrase, info=info)
nc_py_api._exceptions.NextcloudException: [401] Unauthorized <find: , [], >
[401] Unauthorized <find: , [], >

[bigcat88]

The webhook call has no permissions

Do not understand how that is possible.

NextcloudApp class uses AppAPI auth which should have admin permissions..

Will take a look at this

[bigcat88]

I modifed your example a bit, and I see that basic_file_checker is called, but Auth for it to be executed is not passed, so looks like Nextcloud does not send the correct AppAPI auth headers.

Is I understand that this is the issue?

---

Edited: if yes, that this is not an issue.

webhooks_listeners has specific check condition:

if ($exAppId !== null && str_starts_with($webhookUri, '/'))

uri should start from / - only in that case AppAPI auth will be added.

This is done for several reasons(from memory, afaik):

1. ExApp can register webhook target endpoint to be outside of ExApp - if url starts from {WEBHOOK_HOST} like in your example - in that case we do not add AppAPI auth as we are not sure that it is needed and we do not want it's auth values to be exposed.
2. Only AppAPI knows and controls the true host of ExApp(for example for new HaRP system - ExApps can listen on unix-socket), so to be things more simpler ExApp just specify relative route from it's root for webhook and AppAPI/webhook_listeners should know on which host it is to send.

[PatrickPromitzer]

I modifed your example a bit, and I see that basic_file_checker is called, but Auth for it to be executed is not passed, so looks like Nextcloud does not send the correct AppAPI auth headers.
Is I understand that this is the issue?

It is some kind of auth problem, but I don't know from where.

uri should start from / - only in that case AppAPI auth will be added.

The check for the "/" at the start makes sense.

For that reason, I changed the WEBHOOK_HOST variable to "" in my second example, which should make the uri /basic_file_checker

To be sure. I tested it with this code for the register and got the same error.

nc.webhooks.register(
    http_method="POST",
    uri="/basic_file_checker",
    event=event_item
)

Here the whole cleaned up code.

import pathlib
import traceback
import requests

from contextlib import asynccontextmanager
from fastapi import FastAPI, BackgroundTasks

from nc_py_api import NextcloudApp
from nc_py_api.ex_app import AppAPIAuthMiddleware, LogLvl, run_app, set_handlers, nc_app
from fastapi import Depends
from typing import Annotated


LOG_LOCAL_HOST_STRING = "127.0.0.1"


def file_event_handler(event_data: dict, nc: NextcloudApp):
    try:
        nc_file = nc.files.by_id(str(event_data["event"]["node"]["id"]))

    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": str(error_string)})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")
        raise e


@asynccontextmanager
async def lifespan(app: FastAPI):
    set_handlers(app, enabled_handler)
    yield


APP = FastAPI(lifespan=lifespan)
APP.add_middleware(AppAPIAuthMiddleware)


@APP.post("/basic_file_checker")
async def basic_file_checker(
        event_data: dict,
        # files: ActionFileInfoEx,
        nc: Annotated[NextcloudApp, Depends(nc_app)],
        background_tasks: BackgroundTasks,
):
    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                             json={"ID": "basic_file_checker", "called": "true", "data": event_data})
    background_tasks.add_task(file_event_handler, event_data, nc)


def enabled_handler(enabled: bool, nc: NextcloudApp) -> str:
    print(f"enabled={enabled}")

    response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                             json={"ID": "enabled_handler", "enabled": enabled})
    try:
        if enabled:
            event_list = [
                "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
                "OCP\\Files\\Events\\Node\\NodeTouchedEvent",
                "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
                "OCP\\Files\\Events\\Node\\NodeDeletedEvent",
                "OCP\\Files\\Events\\Node\\NodeCopiedEvent",
                "OCP\\Files\\Events\\Node\\NodeRenamedEvent",
            ]
            for event_item in event_list:
                if not event_item.startswith("OCP\\Files\\Events\\Node\\"):
                    continue
                nc.webhooks.register(
                    http_method="POST",
                    uri="/basic_file_checker",
                    event=event_item
                )
            wh_list = nc.webhooks.get_list()
            wh_list = [i.__dict__ for i in wh_list]
            response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string",
                                     json={"id": "webhook list", "data": wh_list})
        nc.log(LogLvl.INFO, f"App enabled: {enabled}")
    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        response = requests.post(f"http://{LOG_LOCAL_HOST_STRING}:2998/log_string", json={"error": error_string})
        nc.log(LogLvl.ERROR, f"Error: {error_string}")

    return ""


if __name__ == "__main__":
    pathlib.Path("temp").mkdir(parents=True, exist_ok=True)
    run_app(
        "main:APP",
        log_level="trace",
    )

Traceback (most recent call last):
  File "/app/lib/main.py", line 53, in file_event_handler
    nc_file = nc.files.by_id(str(event_data["event"]["node"]["id"]))
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/files.py", line 65, in by_id
    result = self.find(req=["eq", "fileid", file_id])
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/files.py", line 86, in find
    return lf_parse_webdav_response(self._session.cfg.dav_url_suffix, webdav_response, request_info)
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/_files.py", line 345, in lf_parse_webdav_response
    return _parse_records(dav_url_suffix, _webdav_response_to_records(webdav_res, info), response_type)
                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/files/_files.py", line 349, in _webdav_response_to_records
    check_error(webdav_res, info=info)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/nc_py_api/_exceptions.py", line 65, in check_error
    raise NextcloudException(status_code, reason=codes(status_code).phrase, info=info)
nc_py_api._exceptions.NextcloudException: [401] Unauthorized <find: , [], >
[401] Unauthorized <find: , [], >

[bigcat88]

Please ,post the content of oc_webhook_listeners table.

I guess the field app_id maybe is not filled in it for some reason, or maybe it contains old records.

[PatrickPromitzer]

Content of

wh_list = nc.webhooks.get_list()
wh_list = [i.__dict__ for i in wh_list]

[
    {
        "_raw_data": {
            "id": 404,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    },
    {
        "_raw_data": {
            "id": 405,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeTouchedEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    },
    {
        "_raw_data": {
            "id": 406,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    },
    {
        "_raw_data": {
            "id": 407,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeDeletedEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    },
    {
        "_raw_data": {
            "id": 408,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeCopiedEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    },
    {
        "_raw_data": {
            "id": 409,
            "appId": "basic_file_checker",
            "userId": null,
            "httpMethod": "POST",
            "uri": "/basic_file_checker",
            "event": "OCP\\Files\\Events\\Node\\NodeRenamedEvent",
            "eventFilter": [],
            "userIdFilter": "",
            "headers": null,
            "authMethod": "none",
            "authData": null
        }
    }
]

[bigcat88]

thanks, I will try to reproduce this today

[bigcat88]

I triaged this in a little bit more than a hour...

Exception 401 come from Nextcloud/AppAPI from WebDAV part. It does not like when the user is not set.
As webhooks_listeners executes notifications from the background jobs, it does not set the user in AppAPI authentication, and in the ExApp upon receiving hook, the NextcloudApp class is with empty user field.

To make it work now you can add such code:

def file_event_handler(event_data: dict, nc: NextcloudApp):
    nc.set_user(event_data["user"]["uid"])  # **Add this line**
    try:
        nc_file = nc.files.by_id(str(event_data["event"]["node"]["id"]))
        print(nc_file)
    except Exception as e:
        error_string = 'Exception:\n', traceback.format_exc() + "\n" + str(e)
        nc.log(LogLvl.ERROR, f"Error: {error_string}")
        raise e

Feel free to create issue(potential bug or feature request from blank issue) in AppAPI repo regarding this, maybe it can be fixed/improved, as here we see that user gets filled:

https://github.com/nextcloud/server/blob/0dc971189badaf050fa5048e391c70fb15171b6f/apps/webhook_listeners/lib/Listener/WebhooksEventListener.php#L36-L46

but here

https://github.com/nextcloud/server/blob/0dc971189badaf050fa5048e391c70fb15171b6f/apps/webhook_listeners/lib/BackgroundJobs/WebhookCall.php#L71-L80

is used value from the filter from that user for which webhook is registered(which is null for the global hooks from your example).

[PatrickPromitzer]

I checked the solution and it works.
Thank you for your help.

I can add a simple version of my code to the example for future reference.