Consider whether to remove analytics data from API Umbrella when deleting ProxyBackend in Apinf

[bajiat]

When deleting a ProxyBackend from Apinf, the backend is also deleted from API Umbrella. However, API Umbrella does not delete analytics data related to the backend, perhaps by design.

Discussion

Consider the pros and cons of deleting related analytics data from API Umbrella. E.g. might the data be useful to a system administrator? Determine why API Umbrella does not delete this data. Is it by design, or perhaps an oversight?

Decide whether we align with the design of API Umbrella. If not,

• can/should the problem be solved upstream?
• is it necessary to produce a solution directly in our project?

Original post

From @Nazarah on January 30, 2017 13:37

Reproduction Steps:

1. Sign in to https://apika.digipalvelutehdas.fi/
2. Add an API with proper URL
3. Go to proxy tab in API profile and provide Proxy Base Path and API Base path.
4. Make a call to the API.
5. Go to Dashboard and select the added API from the dropmenu. Remember the no. of request.
6. Go to API profile and delete the API.
7. After confirming the deletion, add the same API. (includng the proxy base path and API base path)
8. go to Dashboard and select the API from dropmenu.

Result:

Dashboard shows the number of requests made before the API was deleted and readded.

Expected Result

API details should also be deleted from proxy on deleting an API.
So that when the same API is added again, Dashboard should show fresh history (of 0 or requests made after added, etc.) than the previous history of requests.

Environment

Chrome, Windows 8

Copied from original issue: Digipalvelutehdas/APIKA#324

[bajiat]

This should be verified in staging and nightly, not on apika server.

[bajiat]

@marla-singer Would you be interested in this task?

[bajiat]

From @marla-singer on February 6, 2017 8:18

@bajiat Yes, I'd . Estimated

[marla-singer]

@bajiat Researched this bug.

1. It was reproduced on nightly.apinf.io . I used proxy "Nightly Umbrella"

2. The API is deleted from proxy. I added API with name 00 Music with proxy base path alternative and repeated steps.

Screens

Before deleting

The api-umbrella had information about my API

After deleting

The api-umbrella didn't have information about my API

But Analytics section still has information about requests

[marla-singer]

@brylie As I understand the Analytics section has data from ElasticSearch. How do we delete information from ElastciSearch?

[brylie]

This is probably an issue we should discuss with @GUI in the API Umbrella platform.

In a nutshell, the Elasticsearch analytics are not 'linked' to a specific user or API Backend. Rather, they are grouped by prefix (as you note in your message), since API Umbrella allows flexible rules for API Backend prefixing.

[brylie]

Here is a related upstream issue, where @GUI mentions the limitations of Elastic analytics:

• Filter analytics data by API Backend?

It seems like @GUI would be open to a pull request that would add Backend ID to the analytics storage:

If you're looking to add the backend ID to the analytics storage, I think the main areas involved would be storing the matched API backend on an ngx.ctx variable, then adding that to the log message data, and then ensuring that gets pushed onto the data sent to elasticsearch.

cc: @bajiat @ccsr

[philippeluickx]

Suggestion during standup: Instead of deleting APIs, we can only "archive" them. This way we ensure they remain "in the system" and are unique. Historical data will still be relevant then.

[brylie]

I like the idea of leaving the historical analytics data in-tact. It seems simpler in the short-term, and may be useful for the System Administrator to have access to all analytics data, even for 'deleted' backends.

[bajiat]

This task is under research. See issue #2115

[marla-singer]

Suggestion during standup: Instead of deleting APIs, we can only "archive" them. This way we ensure they remain "in the system" and are unique. Historical data will still be relevant then.

I don't mind about archiving. However, it needs to be thought detail because now we don't have any features for that. For archive data, we must do it not only in ElasticSearch but on API-umbrella side and our side: storage all data about used frontend_prefix. And if we decide to implement it then we need to use every profit from that.