removing a failing system and integration will still cause a request to fail

[Roger-Ethyca]

Bug Description

even after removing a failing system and integration will still cause a request to fail

Steps to Reproduce

1. create a system from add a system
2. click on the integrations tab and add an integration (i used domo which will fail connection test once creds are added)
3. add a new system and integration (i used friendbuy nextgen)
4. process a access your data request submitted through the privacy center (it will error on domo)
5. go to view systems and remove the domo system
6. reprocess the request

Expected behavior

the request still errors on domo

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

• Version:
• OS:
• Python Version:
• Docker Version:

Additional context

Add any other context about the problem here.

[adamsachs]

after doing some initial investigation, it's worth noting that this isn't even specifically tied to reprocessing an existing privacy request. this seems to impact any new, future privacy request too.

this implies a bigger impact to users, and also a different root cause technically - it doesn't have anything to do with request reprocessing, it's some artifact from the deleted system/integration being left around generally, not something tied to a particular request state.

[adamsachs]

on first glance, it looks like the problem here is that both the connectionconfig and the datasetconfig records are left in tact even after deleting the system (the ctl_systems record is properly removed).

in request_runner_service.py we pretty basically just query for all DatasetConfig and ConnectionConfig records to create our execution graph.

at this point it's probably worth clarifying expectations, since i do notice that the "old" connection still exists in the "connection manager" screen. do we expect deleting a system to effectively also delete its integration, or should users have to go into the connection manager and explicitly delete the integration? if it's the former, then i think we'll need to update the system delete operation to remove these "integration" artifacts from the db (either by making multiple API calls from the FE or adjusting the API on the BE.

[rsilvery]

They won't be able to use the connection manager anymore so, yes, deleting a systems should delete the connection/integration.

[Roger-Ethyca]

moving to done