#458 removal of gaian/ranger/derby/virtualization connectors/services

[planetf1]

Signed-off-by: Nigel Jones nigel.l.jones+git@gmail.com

• Docker build for atlas, range (base and customization) are removed
• Information view OMAS removed
• Gaian authentication plugin removed
• Gaian Impersonation module removed
• Gaian-ranger plugin remove
• Gaiab data store connector removed
• Security sync services removed
• Security sync connector removed
• Virtualization services removed
• CODEOWNERS , THIRD_PARTY.md has references to removed components removed
• Developer guidelines references to ranger, atlas, gaian, derby updates/removed appropriately
• Server chassis - removed unused components
• Open-metadata packages have some artifacts (as listed below) removed as dependencies
• Build.gradle/settings.gradle/pom.xml - unused dependencies removed (management)
• Download of gaian/derby libraries/creation of local maven repo removed
• Connector configuration factory updated to remove refs to deleted components
• View generator connectors removed (derby/gaian)
• Admin services - removed refs to security sync/tag, virtualizer config/service definition
• Removed security officer services (governance service)
• Updated website docs for connector catalog inc. glossary
• Removed sample docs on information view
• Tested so far: gradle build (inc. FVT), maven build (inc. FVT)

Still to consider

• Security Officer Services - could be removed?
• Data platform OMAS - remove?

Open for review....

Fixes: #458

[mandy-chessell]

It is ok to remove the governance server modules:

• security-server-services
• security-sync-services

Removing obsolete admin services structures (SecurityOfficerConfig, SecuritySyncConfig, VirtualizationConfig, Data PlatformServicesConfig, DiscoveryEngineServices, StewardshipEngineServicesConfig) could break backward compatibility - however I would be in favour of the clean up since they have been deprecated for a while. If we remove these structures, and they are included in a server's configuration document then the first time our code reads the JSON and sotres it, these sections will be removed.

[mandy-chessell]

Also need to update ContentOrganization.md in top level directory

[planetf1]

It is ok to remove the governance server modules:

• security-server-services
• security-sync-services

Removing obsolete admin services structures (SecurityOfficerConfig, SecuritySyncConfig, VirtualizationConfig, Data PlatformServicesConfig, DiscoveryEngineServices, StewardshipEngineServicesConfig) could break backward compatibility - however I would be in favour of the clean up since they have been deprecated for a while. If we remove these structures, and they are included in a server's configuration document then the first time our code reads the JSON and sotres it, these sections will be removed.

• OK I will remove them
• I wasn't certain of the admin services, but it was my understanding a later server would just skip over unknown content, so shouldn't case a problem. One thought I have is whether they might be used in their current form for future work (at least security officer/sync/virtualization) but on balance I think we agree to remove ...
• suggest a new PR for cleanup of the other services a) data platform b) discovery/stewardship

[planetf1]

@mandy-chessell many thanks for the review

[planetf1]

@lpalashevski @mandy-chessell When trying to remove data platform OMAS, I note that the cassandra metadata extractor module is dependent on data platform OMAS.

I need to either

• leave data platform OMAS in situ
• Remove data platform omas and the cassandra metadata extractor (or have a prereq of this change on another, which I'd prefer not to)

[planetf1]

AdditionalChanges

• Definition for Information View OMAS restored (but see question above)
• additional doc updates
• more cleanup of SecurityOfficerConfig (Admin Services)

Having reviewed the changes some more I propose:

• any additional cleanup/doc improvements to stewardship, discovery etc is outside the scope of this PR - we can do a followup. I'd like to keep this focussed on the ranger/atlas/gaian change - easier to review, test
• removal of data platform OMAS is done alongside the Cassandra metadata extractor change in a separate PR
• Security Officer OMAS needs checking due to it's current assumed default schema name of 'gaian'

[planetf1]

As a TBD I will update the release notes once we're generally happy with the proposal (likely in this PR)

[mandy-chessell]

It is ok to remove the governance server modules:

• security-server-services
• security-sync-services

Removing obsolete admin services structures (SecurityOfficerConfig, SecuritySyncConfig, VirtualizationConfig, Data PlatformServicesConfig, DiscoveryEngineServices, StewardshipEngineServicesConfig) could break backward compatibility - however I would be in favour of the clean up since they have been deprecated for a while. If we remove these structures, and they are included in a server's configuration document then the first time our code reads the JSON and sotres it, these sections will be removed.

* OK I will remove them

* I wasn't certain of the admin services, but it was my understanding a later server would just skip over unknown content, so shouldn't case a problem. One thought I have is whether they might be used in their current form for future work (at least security officer/sync/virtualization) but on balance I think we agree to remove ...

* suggest a new PR for cleanup of the other services a) data platform b) discovery/stewardship

These services have already been folded into the integration services running in the intrgration daemon so no longer needed.

[mandy-chessell]

Just noticed an update needed to ContentOrganization.md - removal of governance-engine-plugins under adapters

[planetf1]

Just noticed an update needed to ContentOrganization.md - removal of governance-engine-plugins under adapters

done.

[planetf1]

Opened up followup items:

• Remove admin services config for deprecated Stewardship engine, discovery engine #5318 - cleanup of discovery/data stewardship in admin services
• Remove Data Platform OMAS #5319 - removal of data engine omas once cassandra metadata extractor is removed.

[planetf1]

As a TBD I will update the release notes once we're generally happy with the proposal (likely in this PR)

Now added an initial readme entry for release notes

[planetf1]

Opened #5320 to track gaian reference in Security Officer OMAS

[planetf1]

@mandy-chessell SecurityManagerEventType.java contains:

    NEW_INFORMATION_VIEW_EVENT          (1,  "New View Event", "An new information view asset has been added to one of the access point zones."),

(Data Platform has a similar event - but this will be addressed in #5319 most easily)

I think this should be removed. There is no other handling of it. However it may be that we want to reuse it in future?

@mandy-chessell Can you also clarify the requirement to leave Information View OMAS in the AccessServiceDescription.java file? (presuming the same logic would apply to data platform omas in future). We also have SECURITY_OFFICER_SERVICES & SECURITY_SYNC_SERVICES where the same question applies (GovernanceServicesDescription.java)

[planetf1]

Some other observations that I'm looking for insight on. None directly affect this PR, and I think all relate to the base support for security tags which is seen across many interfaces & beans.

• the securitytagconnector (security officer connector) remains - this will support future security tag mechanisms including potentially an updated ranger implementation
• Asset Owner has support for setting/retrieving security tags for an asset
• Security Officer OMAS has support for detecting changes to security tags (the checks differ from asset owner?)
• The OpenMetadataAPIGenericHandler (setClassificationInRepository)refers to security tags in the comments when updating classifications - which is one possible route, but the handler also gets used in other contexts such as when classifying glossary terms (similar in ReferenceableBuilder.java & AssetReferenceable.java).
• Text in various places refers to atlas/ranger as examples - as these are descriptive have left them in
• Type references for information views etc remain in-situ (and are really part of the model and not directly associated with IV omas etc)

[planetf1]

@yevgenmar I note that in AnalyticsArtifactHandler.java in analytics modeling, you comment 'Note: use resolver from InformationView asset'. From the code I presume you mean 'from Analytics Asset' or similar. I can update if needed (v. minor)

[planetf1]

Any more comments on this? I am going to propose it's inclusion on the team call 2021-06-08 unless I get multiple positive reviews before then

[planetf1]

Approved in developer call 2021-06-08