fix loading remote definitions via cron

[pedroslopez]

What

fixes #21401

After #21073 moved the definitions provider to be injected as a singleton, the behavior of only fetching definitions in the RemoteDefinitionProvider on class initialization meant that we were no longer fetching the latest definitions each time the cron ran.

How

Refactor the RemoteDefinitionsProvider to let use leverage micronaut caching to determine when to re-fetch the data. Each getter now calls the method to fetch the data, but we're caching the results of the method for 15s on airbyte-cron.

[evantahler]

👍

Just to check my understanding - in Cloud, there will be a PR to call definitionsProvider.loadDefinitions()?

[gosusnp]

I am wondering if we could leverage micronaut caching here instead of requiring the client to decide when to actually load the definitions. I added comments trying to highlight what this could look like.
edit: link to micronaut doc

The other missing piece would be updating the application.yml with:

micronaut:
  caches:
    remoteDefinitions: 
      charset: 'UTF-8'
      expire-after-write: 1min

@jdpgrailsdev, do you think that would be a valid approach to manage how frequently we refresh definitions?

[gosusnp]

Since local definitions are read from the resources, we could annotate here with @PostConstruct

[gosusnp]

We do not need this call here.

[gosusnp]

Add a @cACHEpUT("remoteDefinitions") annotation here instead of @PostConstruct

all the getters in this provider should call loadDefinitions()

[jdpgrailsdev]

I am wondering if we could leverage micronaut caching here instead of requiring the client to decide when to actually load the definitions. I added comments trying to highlight what this could look like. edit: link to micronaut doc

The other missing piece would be updating the application.yml with:

micronaut:
  caches:
    remoteDefinitions: 
      charset: 'UTF-8'
      expire-after-write: 1min

@jdpgrailsdev, do you think that would be a valid approach to manage how frequently we refresh definitions?

@gosusnp Yes, but I think it is a little more involved. The Micronaut Cache abstraction is just that -- an abstraction over the creation and management of an in memory cache, such as one you could create using Guava, Ehcache or Caffeine. You would still need to mark the methods that will cache their return values with the @Cachable annotation. This is also not a distributed cache, so its possible that different instances of the application may return different results depending on when they were called even with caching.

[pedroslopez]

Ok! Moved to using micronaut to handle the caching and reverted to the previous interface. Set to cache for 15s (cron runs every 30s).

[gosusnp]

@jdpgrailsdev, I agree, it is a restriction from having an in-memory cache, I don't think it is a blocker for the current use case. The goal is to refresh frequently and I don't think we want to have concurrent updates so it should remain restricted to a single instance.

[github-actions]

File	Coverage [96.63%]	🍏
RemoteDefinitionsProvider.java	98.91%	🍏
LocalDefinitionsProvider.java	94.85%	🍏

Total Project Coverage	26.59%	🍏