APIExport consumer can't wildcard list/watch its schemas until the first APIBinding is created

[ncdc]

Describe the bug
A controller using APIExports can't successfully start up and wildcard list/watch the APIs it's exporting until the first APIBinding is created.

To Reproduce
Steps to reproduce the behavior:

1. Create an APIResourceSchema
2. Create an APIExport that exports the APIResourceSchema
3. Try to do a wildcard list/watch through the APIExport virtual workspace, such as

https://$server/services/apiexport/root:default:andy/controller-runtime-example-data.my.domain/clusters/*/apis/data.my.domain/v1alpha1/widgets

4. Get a 404

In a controller-runtime app, you'd see something like this:

1.6541008964721239e+09  ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "Widget.data.my.domain", "error": "no matches for kind \"Widget\" in version \"data.my.domain/v1alpha1\""}

Expected behavior
The controller should be able to start listing/watching its APIs even without any APIBindings for it

Additional context
None

[sttts]

workaround: create an APIBinding in the same workspace to itself (we do the same in compute workspaces).

[sttts]

cc @davidfestal

[ncdc]

I'm working on a fix

[ncdc]

Would it make sense to create a new controller for apiresourceschema to create bound CRDs?

[ncdc]

@sttts ^?

[ncdc]

Actually I want to go back and look at the vw and see if I can fix it there

[sttts]

I don't think we want a copy of every schema in every cluster.

[sttts]

Actually I want to go back and look at the vw and see if I can fix it there

Also my intuition.

[sttts]

there is a problem though with the vw solution: what about watches? They have to start and then eventually show objects when the first APIBinding is created on that shard. That's tricky. We could have a fake storage for identity based requests always returning empty, or a watch without events. When an APIBinding is added, we would have to terminate the old fake storage (including the fake watch) and swich over to the bound CRD based storage. Does that make sense?

[ncdc]

Yeah, I'll look into it

[ncdc]

Our custom apiextensions logic to get CRDs looks for a CRD matching <identity, group, resource>, where the index data is populated from APIBindings. This is why you get a 404 when doing a wildcard list/watch (when no APIBindings exist that point to the APIExport).

[ncdc]

Chatted with @sttts and here's what we want to do:

1. Because APIExports will have to be replicated from their source shard to all other shards that have APIBindings for them, we don't want to automatically create a bound CRD as soon as an APIExport is created
2. Instead, we only want to preserve the current behavior of only creating the bound CRD when the first APIBinding is created, AND
3. We only populate APIExport.status.virtualWorkspaceURLs if there is at least 1 APIBinding for an APIExport. This means moving the URL management into the APIBinding controller
4. Controllers using the APIExport virtual workspace must wait until there is at least 1 virtualWorkspaceURL entry before proceeding to list/watch exported resource (this means there is at least 1 APIBinding)
   1. They should probably (must?) terminate when the list is empty (restarting would being the waiting process again)

[sttts]

cc @p0lyn0mial

[stevekuznetsov]

DX of a crash-looping controller until an APIBinding exists is ... not ideal? :)

[stevekuznetsov]

Or even ops-X - that will likely trigger alerts on any/all clusters

[sttts]

They shouldn't be crash-looping, but just notice the URL list is empty and keep watching it until some shard (in their topology partition) shows up.

[stevekuznetsov]

I was responding to:

They should probably (must?) terminate when the list is empty (restarting would being the waiting process again)

We likely want some user-facing way to expose this waiting state, ultimately.

[ncdc]

We can adjust that bit to not do the termination

[sttts]

Just sit there and wait until a shard appears, like a controller not seeing any object in a watch.

[p0lyn0mial]

I think the same issue applies to the kcp server, not only to the vw server.

Because APIExports will have to be replicated from their source shard to all other shards that have APIBindings for them, we don't want to automatically create a bound CRD as soon as an APIExport is created

is this due to some performance issue? having a single CRD object per shard doesn't sound too heavy.

btw: do we want to change the reply from the server to something like 404: you need to create an APIBinding first to use this api (if the export exists, otherwise we would return plain 404) ?

[ncdc]

I think the same issue applies to the kcp server, not only to the vw server.

Yes, this is for APIExports in general, so it's not specific to the vw server.

is this due to some performance issue? having a single CRD object per shard doesn't sound too heavy.

It's 1 CRD per APIResourceSchema. An APIExport can export 1..n APIResourceSchemas.

btw: do we want to change the reply from the server to something like 404: you need to create an APIBinding first to use this api (if the export exists, otherwise we would return plain 404) ?

Maybe?

[sttts]

is this due to some performance issue? having a single CRD object per shard doesn't sound too heavy.

There can be many. And there is just no need to tell controllers to list/watch shards without bindings.

[sttts]

also creating APIExports could create a storm of storage creations and list/watches. Doing that on demand when a binding shows up is much much better.

[sttts]

And we can safely assume that only very very few APIExports will actually be bound to more than one workspace.

[ncdc]

@sttts & I chatted a couple of weeks ago about what to do about this. We decided that the APIExport status should only get virtual workspace URL(s) set if and only if there is at least 1 APIBinding referencing the APIExport.