[Connectors][API] Update connectors with enabledAfterImport flag

[ymao1]

In order to support import/export of connectors, we need to mark a connector as enabled/disabled after import. This issue addresses the API portion of this. A separate issue has been created for addressing the UI portion.

1. Add enabledAfterImport flag (or something similar) to RawAction type
2. Update connectors APIs to handle the new flag - create and update should just set enabledAfterImport to true
3. Add migration to add enabledAfterImport: true to all existing connectors

[ymao1]

Should we be able to disable preconfigured connectors? My feeling is no. If it's preconfigured and users want to move to a different cluster, they can copy the kibana.yml configuration instead of importing/exporting.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

Should we be able to disable preconfigured connectors? My feeling is no. If it's preconfigured and users want to move to a different cluster, they can copy the kibana.yml configuration instead of importing/exporting.

Yeah, that seems correct. The secrets for preconfigured connectors are either in the config as well as the other params, or can be in a a keystore. So they already have a mechanism for "import/export".

[pmuellr]

A couple of sharp edges to watch out for when adding new fields the connector saved objects:

• we need to figure out if we want the new fields to be excluded from AAD; by default they are included in AAD, but can be explicitly marked to be excluded. When excluded, this allows us to do partial updates on just those fields, without having have all the values and re-encrypt everything

• generally, "empty" values of saved object fields should be null and not allowed to be undefined, since allowing undefined values doesn't allows us to "delete" the values later (considered a partial update, which then causes subsequent AAD issues)

[pmuellr]

Update built in connectors secrets schemas so fields are nullable

That's going to be complicated. We already have a mix of secrets which are nullable, and some are not. How would we know then if a particular secret is "really" nullable or not?

Could we just set the secrets field itself to be null? I think elasticsearch allows us to do that at runtime. We could then change the places where we call the config/secrets validation code to check the enabled flag to see if we should bother running the secrets validator or not.

... sorry, starting to ramble below, but given the number of APIs we're adding, now wondering if there is some "simpler" story ...

I'm kinda wondering if we even need to support specific enable/disable APIs. During export, we would presumably not export the secrets, but then we can't do an import of those using existing APIs as the validation would fail because there are no secrets (except for server log and the index connector since they don't have secrets :-)).

We'd need a special import API that would allow connectors to be created but not validate their secrets. Those connectors would then fail if you ever tried to run them (we validate before running), and require that the customer update the connector to add the secrets back.

What we'd be missing is a clear signal that there are connectors which need their secrets added back, since we don't really have a way of showing that now. But maybe that's just some hidden field, which we set on the special import API, and can show an indication in the alerts list (and make it sortable/etc) - I guess this is the justImported flag I referenced in the meta issue - not neccessarily "hidden", but read-only from an API point of view (and of course it would need a better name)

It does make me wonder in general how we will ensure that server log and the index connector will get exported and then imported without requiring any user interaction! Interesting edge case!

Also I'm wondering if we have use cases where we would want to enable/disable connectors independent of import/export - we've talked about it before, but I'm not sure we have any real requirements for it. Having to deal with justImported AND enabled - is that good or bad? If it's all clumped in a single enabled flag, is it important for us to be able to tell the difference - so that we can tell the customer the difference? Perhaps it depends on whether a disable API specifically nulls out the secrets or not - if it doesn't - which would be nice so that you don't have to re-enter them when re-enabling the connector, then having them as separate things seems like it makes sense. But maybe secrets == null is enough of an indicator of justImported? Maybe we don't need a specific field?

[ymao1]

Also I'm wondering if we have use cases where we would want to enable/disable connectors independent of import/export - we've talked about it before, but I'm not sure we have any real requirements for it.

My thinking behind this was since the workflow user imports connectors that are disabled and are directed to the connectors list to re-enable and re-enter secrets, each connector in the list would have some sort of enable switch (like the rules list does currently). If there is an enable, then there probably should be a disable :)

However, I see your point that this is a lot of API changes. Maybe if we consider the re-enable workflow more of prompting the user to edit their connector instead of re-enabling it, we can get away with just updating the update API instead of introducing the concept of explicit enable/disable.

I do think it would be useful to have a disable by ID API (maybe a connector is erroring; user could disable it while troubleshooting to avoid littering the logs with errors and then reenable later without having to go through all the alerts to figure out which ones used the connector, which they would need to do if they deleted the connector). But maybe this isn't something that needs to happen now?