Support for Salesforce table description ingestion

[kwgdaig]

Is your feature request related to a problem? Please describe.
For Salesforce, which is a Database service, there is no function to extract the description assigned to an object (table).
Even if the object (table) description is written in Salesforce, it must be manually added to OpenMetadata.Even if the description is included in Salesforce, it must be manually added to OpenMetadata.

Describe the solution you'd like
We would like to extract object (table) descriptions in the Salesforce Metadata ingestion and automatically import them into OpenMetadata. This feature eliminates the need for double management of table descriptions in Salesforce and OpenMetadata. In addition, the automatic inclusion of decsription will improve the rate at which descriptions are assigned to tables and increase the organization's understanding of the data.

Additional context
In Salesforce, it is possible to optionally assign a description to an object, as shown in the attached image.

[kwgdaig]

FYI

I have investigated how to extract the description of a salesforce object.
REST API used in this code cannot get the object description, so we need to use another API. I think that we need to use Tooling API to access the metadata as the other API since EntityDefinition document on Tooling API includes the object description.
In simple-salesforce package currently used in OpenMetadata, it seems that Tooling API can be used with toolingexecute function.

I have confirmed that the object description can be extracted by using the following code.

code:

self.client.toolingexecute("""query/?q=SELECT+DeveloperName,QualifiedApiName,Description+FROM+EntityDefinition+LIMIT+100""")

retuls:

{
	'size': 100,
	'totalSize': 100,
	'done': True,
	'queryLocator': None,
	'entityTypeName': 'EntityDefinition',
	'records': [{
		'attributes': {
			'type': 'EntityDefinition',
			'url': '/services/data/v42.0/tooling/sobjects/EntityDefinition/01***********'
		},
		'DeveloperName': '******Info',
		'QualifiedApiName': '***_******__c',
		'Description': None
	}, {
		'attributes': {
			'type': 'EntityDefinition',
			'url': '/services/data/v42.0/tooling/sobjects/EntityDefinition/01*********'
		},
		'DeveloperName': '******',
		'QualifiedApiName': '******__******__c',
		'Description': '****** for ******.'
	}, {
...
	}]
}

Reference document:

• Metadata that can be obtained with sObject Describe
  • https://developer.salesforce.com/docs/atlas.ja-jp.api_rest.meta/api_rest/resources_sobject_describe.htm
• List of Salesforce APIs
  • https://help.salesforce.com/s/articleView?id=sf.integrate_what_is_api.htm&type=5

[pmbrull]

hi @kwgdaig since we don't have access to enterprise salesforce instance, would you be able to contribute this? Happy to guide you. Thanks

[kwgdaig]

@pmbrull Yes, I would like to contribute to it. Cloud you assign me to this?

[ayush-shah]

Hello @kwgdaig i have assigned the issue to you, let me know if you need help with the same..we can communicate via https://slack.open-metadata.org/

Find me under Ayush Shah or ask on #support / #contributor channel and we will be available to help

[pmbrull]

@kwgdaig do you have any updates here? WE'd need help since we don't have the enterprise service to test 🙏

[kwgdaig]

Hi @pmbrull -san, I have fixed the code and created a PR, but will check as some tests are failing.