You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When attempting to use CVAT's auto annotation feature, the UI shows the following error:
Error: Inference status for the task 17 is failed. requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://cvat-nuclio-dashboard:8070/api/function_invocations
This is likely arising due to the X-Nuclio-Function-Namespace being set incorrectly for invocation calls, as shown by these logs from the Nuclio dashboard:
In this, we can see that the request to get the function that succeeded (200) has the X-Nuclio-Function-Namespace header set to cvat which is the correct namespace. In the unsuccessful (404) request to invoke the function, the X-Nuclio-Function-Namespace header is set to nuclio, which doesnt exist in my Kubernetes cluster.
If the function is deployed to the cvat namespace (for discovery by CVAT), and the nuclio namespace (for invocation), the auto annotation succeeds. If the function is only deployed to the nuclio namespace, no models are visible in CVAT.
I am unable to provide source code for the function as this is proprietary but it is just a simple function that decodes an image from Base64 before passing it through a model loaded via MLFlow and returning the predictions.
Due to upgrading from CVAT 1.6, I have made some config changes to fit the deployment into our existing infra (as detailed here):
Using an external Postgres instance
Using an external Redis instance
Using a pre-existing PVC
Disabling the default ingress and using a pre-existing one, updated with the routes from here. This terminates our TLS.
Expected Behaviour
When using auto annotation, calls to invoke the Nuclio function use the correct X-Nuclio-Function-Namespace header and don't 404.
Current Behaviour
When using auto annotation, calls to invoke the Nuclio function don't use the correct X-Nuclio-Function-Namespace header, leading to a 404.
Possible Solution
Set the X-Nuclio-Function-Namespace header to the same value used for discovering functions (see logs from the Nuclio dashboard in the summary).
I have also got this to work be deploying the function twice - once in the cvat namespace so it can be discovered, and once in the nuclio namespace for the actual invocation, but this is more of a temporary fix.
Context
Unable to run auto annotation on tasks without deploying the same function twice in separate namespaces.
The `CVAT_NUCLIO_FUNCTION_NAMESPACE` needs to be defined consistently in
order for Nuclio integration to work. Currently, it's set to `cvat` for
the main CVAT server process, but not for any other CVAT process (which
means it defaults to `nuclio` in those processes). Since it's the
annotation worker process that actually invokes the Nuclio functions,
the invocation fails.
Fix it by synchronizing the Nuclio environment variables across all
backend deployments. Technically, I think only the server and annotation
worker deployments need these variables, but since they're accessed by
`cvat/settings/base.py` in every process that loads Django, define them
everywhere to be sure.
Fixes#5626.
…t-ai#5917)
The `CVAT_NUCLIO_FUNCTION_NAMESPACE` needs to be defined consistently in
order for Nuclio integration to work. Currently, it's set to `cvat` for
the main CVAT server process, but not for any other CVAT process (which
means it defaults to `nuclio` in those processes). Since it's the
annotation worker process that actually invokes the Nuclio functions,
the invocation fails.
Fix it by synchronizing the Nuclio environment variables across all
backend deployments. Technically, I think only the server and annotation
worker deployments need these variables, but since they're accessed by
`cvat/settings/base.py` in every process that loads Django, define them
everywhere to be sure.
Fixescvat-ai#5626.
My actions before raising this issue
Summary
When attempting to use CVAT's auto annotation feature, the UI shows the following error:
This is likely arising due to the
X-Nuclio-Function-Namespace
being set incorrectly for invocation calls, as shown by these logs from the Nuclio dashboard:In this, we can see that the request to get the function that succeeded (200) has the
X-Nuclio-Function-Namespace
header set tocvat
which is the correct namespace. In the unsuccessful (404) request to invoke the function, theX-Nuclio-Function-Namespace
header is set tonuclio
, which doesnt exist in my Kubernetes cluster.If the function is deployed to the
cvat
namespace (for discovery by CVAT), and thenuclio
namespace (for invocation), the auto annotation succeeds. If the function is only deployed to thenuclio
namespace, no models are visible in CVAT.Steps to Reproduce (for bugs)
I'm running CVAT and Nuclio on Kubernetes via Helm. These are the Helm charts used.
This is the command used to install CVAT and Nuclio.
Here is my
values.override.yaml
:To deploy my Nuclio function, I am using the following commands:
I am unable to provide source code for the function as this is proprietary but it is just a simple function that decodes an image from Base64 before passing it through a model loaded via MLFlow and returning the predictions.
Due to upgrading from CVAT 1.6, I have made some config changes to fit the deployment into our existing infra (as detailed here):
Expected Behaviour
When using auto annotation, calls to invoke the Nuclio function use the correct
X-Nuclio-Function-Namespace
header and don't 404.Current Behaviour
When using auto annotation, calls to invoke the Nuclio function don't use the correct
X-Nuclio-Function-Namespace
header, leading to a 404.Possible Solution
Set the
X-Nuclio-Function-Namespace
header to the same value used for discovering functions (see logs from the Nuclio dashboard in the summary).I have also got this to work be deploying the function twice - once in the
cvat
namespace so it can be discovered, and once in thenuclio
namespace for the actual invocation, but this is more of a temporary fix.Context
Unable to run auto annotation on tasks without deploying the same function twice in separate namespaces.
Your Environment
git log -1
): fb4af9cdocker version
(e.g. Docker 17.0.05): N/AMany thanks!
The text was updated successfully, but these errors were encountered: