docs(rfc): Improve Time & Flow of the core Job Deployment & other areas

[arinda-arif]

RFC to improve the time taken to do:

• Job deployment
• Job creation
• Job deletion
• Replay
• Backup

[coveralls]

Pull Request Test Coverage Report for Build 1897022334

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 43 unchanged lines in 6 files lost coverage.
• Overall coverage increased (+0.5%) to 72.399%

---

Files with Coverage Reduction	New Missed Lines	%
api/handler/v1beta1/error_helper.go	2	57.14%
service/errors.go	2	89.36%
service/namespace_service.go	2	95.65%
service/project_service.go	2	93.94%
service/secret_service.go	3	86.96%
api/handler/v1beta1/runtime.go	32	68.32%

Totals
Change from base Build 1839481496:	0.5%
Covered Lines:	5115
Relevant Lines:	7065

---

💛 - Coveralls

[kushsharma]

I see we are trying to cache the upstream job dependency in db at optimus level instead of plugin level(bq2bq does in-memory cache at the moment) but one problem with this approach that I see is upstream dependencies can change without even modifying the job. For example in case bq2bq, if your query uses a view, that view could have been updated externally which indirectly updated this job's upstream tables. So unless we invalidate the job hash in optimus db, optimus will remain oblivious to this change.

[arinda-arif]

@kushsharma @sravankorumilli how about Optimus provides the ability to do dependency resolution for all jobs in a project? It can be a separate command, that can be run before the job deployment, on-demand, or can be called in deploy --all. Or we can also make it a flag.

The point is there will be 2 abilities, resolve the dependencies for all jobs when needed, and also only the modified ones.

[sravankorumilli]

If it is outside of Optimus then, it is not suggested but as @arinda-arif mentioned we can have a mechanism of refresh all jobs, which computes the dependencies & priorities of all jobs, either it can be automated in the pipeline or it can be used by admins for fixing any errors.

For the resources being managed within Optimus, it is fair point to automatically infer the dependencies, so we need to have a relationship b/w resources & jobs, through which we can do the dependency resolution for the jobs for which these views are the sources, it is just that we need to keep the design clean with this change.

[arinda-arif]

Summary of discussion

1. Deployment process will not be failed when there is a failing dependency resolution. We will fetch the existing job dependency from the table and proceed with the deployment. The list of failing jobs will be shown in the response. There will be metrics added and alert.
2. Sources and destination will be computed and stored before doing the dependency resolution (when saving the job). The source and destination will be already stored regardless of whether the dependency resolution and compilation succeed or not. It will not be a problem if any issue is happening in some job. Jobs and resources already integrated through these 2 fields.
3. Check and remove caching in transformers, to avoid giving wrong dependencies
4. Ability to deploy multiple namespaces in 1 request. This is helpful to do clean deployment per project and avoid stale dependencies when doing it 1 by 1.

[arinda-arif]

As discussed, this is the expected behavior of jobs and resources. This behavior is the guideline on what to build and improve.

Job Creation / Modification

• Sources & Destination Resource Urns to be identified & persisted
• Dependency resolution for the current job & the jobs which have source as the new jobs destination.
• Priority Resolution for all jobs
• Compilation & deployment for all jobs
• Whether we support multiple jobs writing to a single destination or not? - Validation to not support this.

Job Deletion

• Identify any dependent jobs & delete only if there are no dependencies.
• Soft Delete can be avoided & rather store in a separate table for audit as it avoids reusing the names (TBD)
• No Need for Priority Resolution or compilation, but we have deleted it from the scheduler storage.

Refresh all Jobs at namespace or at project level (Deploy)

• Diff of modified/created/deleted jobs at the namespace level
• Compute sources & destination & persist in DB
• Dependency Resolution & Persist
• PriorityResolution, Compilation & Deployment only once.

Clean Refresh all jobs (Clean Deploy)

• Identify deleted Jobs
• We do the rest all steps for all(dependency resolution, priority resolution, deployment)
• Side effect the plugins will be upgraded.

Upgrade Jobs at namespace, project (upgrading plugins)

• Can use Clean Refresh

Resource Creation

• On Resource Creation it doesn’t affect any dependencies, rather we can check if any job can be created without resource being available as an optional validation to avoid missing resource creation.

Refresh all Resources at namespace or at project level.

• Diff of modified/created/deleted resources at the namespace level
• Dependency Resolution
• Priority Resolution & Job Deployment

Clean Refresh all resources

• Consider all resources as new/modified we just update all resources.
• Dependency Resolution, Priority Resolution & Deployment for all.

[arinda-arif]

Optimus job deploy –namespace –project –clean
Will do clean refresh all jobs

Optimus resource deploy –namespace –project –clean –resources=(resource1, resourc_2)
Will do deployment only for specific resources

Optimus job deploy –namespace –project –jobs=(job_name1,job_name2)
Will do deployment only for specific jobs

Optimus job/resource delete/upgrade –job-names
Will delete or upgrade jobs.