fix(ingestion/spark): Platform instance and column level lineage fix

[treff7es]

• Add platform instance fix
• Add incremental column-level lineage fix
• Add option to set platform instance and env per platform
• Add option to lowercase urns

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Summary by CodeRabbit

• New Features

  • Introduced a new configuration setting to handle lowercase dataset URNs, enhancing flexibility in dataset identification.
  • Added utility methods to the Spark agent for extracting paths from various RDD types in Apache Spark.

• Bug Fixes

  • Corrected logic for processing downstream elements to improve data lineage accuracy.

• Documentation

  • Updated README to reflect new default settings and newly introduced configuration options.

• Tests

  • Added new test cases to validate platform-specific configurations and the handling of lowercase dataset URNs.

[coderabbitai]

Walkthrough

The recent updates consist of adding the capability to convert dataset URNs to lowercase, enhancing configuration handling for environment and platform instances, updating downstream processing logic, and modifying default configuration settings for improved flexibility and usability.

Changes

File Path	Change Summary
.../openlineage-converter/.../DatahubOpenlineageConfig.java	Added a new boolean field lowerCaseDatasetUrns with a default value false.
.../openlineage-converter/.../OpenLineageToDataHub.java	Added imports for FabricType, PathSpec, and DatahubUtils. Implemented logic for lowercase dataset URNs and utilizing PathSpec for configurations.
.../openlineage-converter/.../DatahubJob.java	Modified logic in processDownstreams to swap downstream and upstream variables.
.../openlineage-converter/.../PathSpec.java	Changed env field to an Optional<String> with a default value of Optional.empty().
.../spark-lineage-beta/README.md	Updated default value of spark.datahub.patch.enabled to false. Added new settings: spark.datahub.metadata.dataset.lowerCaseUrns and spark.datahub.disableSymlinkResolution.
.../spark-lineage-beta/.../SparkConfigParser.java	Enhanced configuration handling for lowercase dataset URNs.
.../spark-lineage-beta/.../OpenLineageEventToDatahubTest.java	Added imports and usage of PathSpec in new test methods.
.../spark/agent/util/RddPathUtils.java	Utility methods to extract paths from various types of RDD nodes in Apache Spark.

Poem

In code's vast fields, we play today,
With datasets lowercase, in a bright array.
The configs now with care we tune,
To dance with Spark beneath the moon.
Downstream flows we guide anew,
A rabbit's dream in code come true. 🌟🐇

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 8

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 80e5a64 and 5f904d4.

Files selected for processing (8)

• metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/config/DatahubOpenlineageConfig.java (1 hunks)
• metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/converter/OpenLineageToDataHub.java (4 hunks)
• metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/dataset/DatahubJob.java (1 hunks)
• metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/dataset/PathSpec.java (1 hunks)
• metadata-integration/java/spark-lineage-beta/README.md (2 hunks)
• metadata-integration/java/spark-lineage-beta/src/main/java/datahub/spark/conf/SparkConfigParser.java (5 hunks)
• metadata-integration/java/spark-lineage-beta/src/test/java/datahub/spark/OpenLineageEventToDatahubTest.java (2 hunks)
• metadata-integration/java/spark-lineage-beta/src/test/resources/ol_events/redshift_mixed_case_lineage_spark.json (1 hunks)

Additional context used

LanguageTool

metadata-integration/java/spark-lineage-beta/README.md

[grammar] ~184-~184: After ‘it’, use the third-person verb form “coalesces”.
Context: ... | | | Normally it coalesce and send metadata at the onApplicationE...

(IT_VBZ)

---

[uncategorized] ~185-~185: Did you mean: “By default,”?
Context: ...rwrites existing Dataset lineage edges. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[uncategorized] ~186-~186: Did you mean: “By default,”?
Context: ...this to true to lowercase dataset urns. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[style] ~187-~187: ‘prefer to have’ might be wordy. Consider a shorter alternative.
Context: ...use dataset symlink (for example if you prefer to have the s3 location instead of the Hive tab...

(EN_WORDINESS_PREMIUM_PREFER_TO_HAVE)

---

[uncategorized] ~187-~187: Did you mean: “By default,”?
Context: ...s3 location instead of the Hive table). By default it is disabled. ...

(BY_DEFAULT_COMMA)

Markdownlint

metadata-integration/java/spark-lineage-beta/README.md

189-189: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)

---

346-346: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

348-348: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

349-349: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

349-349: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

345-345: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

---

350-350: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

Additional comments not posted (22)

metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/dataset/PathSpec.java (1)

17-17: Good use of Optional for env field.

Using Optional for the env field improves the handling of potentially absent values and makes the code more robust.

metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/config/DatahubOpenlineageConfig.java (1)

36-36: New configuration option for lowercasing dataset URNs.

The addition of the lowerCaseDatasetUrns field adds flexibility to the configuration, allowing users to specify whether dataset URNs should be lowercased.

metadata-integration/java/spark-lineage-beta/src/test/resources/ol_events/redshift_mixed_case_lineage_spark.json (1)

1-147: Well-formed JSON for Spark job lineage.

The JSON representation of the Spark job with detailed lineage information for inputs and outputs is well-formed and comprehensive.

metadata-integration/java/spark-lineage-beta/src/main/java/datahub/spark/conf/SparkConfigParser.java (1)

45-46: Consistent addition of lowercasing functionality for dataset URNs.

The new constant DATASET_LOWERCASE_URNS and the corresponding method isLowerCaseDatasetUrns are well-integrated and consistent with the configuration changes.

Also applies to: 157-157, 252-262, 273-274, 351-354

metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/dataset/DatahubJob.java (1)

283-287: Bug fix: Corrected handling of downstream datasets.

The change to swap the use of downstream and upstream variables in the processDownstreams method ensures the correct handling of downstream datasets.

metadata-integration/java/spark-lineage-beta/README.md (3)

351-351: LGTM!

This line looks good to me.

---

352-352: LGTM!

This line looks good to me.

---

353-353: LGTM!

This line looks good to me.

metadata-integration/java/spark-lineage-beta/src/test/java/datahub/spark/OpenLineageEventToDatahubTest.java (6)

16-16: LGTM!

The import statement for PathSpec looks good to me.

---

23-23: LGTM!

The import statement for Optional looks good to me.

---

603-636: LGTM!

The test method testProcessRedshiftOutputWithPlatformInstance looks good to me.

---

638-681: LGTM!

The test method testProcessRedshiftOutputWithPlatformSpecificPlatformInstance looks good to me.

---

683-722: LGTM!

The test method testProcessRedshiftOutputWithPlatformSpecificEnv looks good to me.

---

724-756: LGTM!

The test method testProcessRedshiftOutputLowercasedUrns looks good to me.

metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/converter/OpenLineageToDataHub.java (8)

7-7: LGTM! Import added for FabricType.

This import is necessary for handling environment configurations.

---

61-61: LGTM! Import added for PathSpec.

This import is necessary for managing platform instances and environments.

---

62-62: LGTM! Import added for DatahubUtils.

This import is necessary for utility functions related to DataHub.

---

Line range hint 115-115: LGTM! Method signature updated for convertOpenlineageDatasetToDatasetUrn.

The additional parameter allows the method to use configuration settings for namespace and dataset name transformations.

---

199-216: LGTM! Added handling for environment configurations using PathSpec.

The code iterates through PathSpec to get the environment configuration for the specified platform and handles invalid values with a warning log.

However, ensure that the PathSpec configuration is correctly set in the codebase.

---

194-197: LGTM! Added code to create dataset URNs using platform instance and environment.

The code uses DatahubUtils.createDatasetUrn to create dataset URNs with the specified platform, platform instance, dataset name, and environment.

However, ensure that the DatahubUtils.createDatasetUrn function is correctly implemented in the codebase.

Verification successful

---

The DatahubUtils.createDatasetUrn function is correctly implemented.

The createDatasetUrn method constructs a DatasetUrn using the provided platform, platform instance, dataset name, and fabric type. This ensures that the URNs are correctly formatted based on the configuration.

• Location:
  • metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/utils/DatahubUtils.java

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the implementation of `DatahubUtils.createDatasetUrn`.

# Test: Search for the function implementation. Expect: Occurrences of `createDatasetUrn` in the codebase.
rg --type java $'createDatasetUrn'

Length of output: 1476

---

Script:

#!/bin/bash
# Extract the implementation of `createDatasetUrn` from `DatahubUtils`.

ast-grep --lang java --pattern $'class DatahubUtils {
  $$$
  public static DatasetUrn createDatasetUrn($_, $_, $_, $_) {
    $$$
  }
  $$$
}'

Length of output: 143

---

Script:

#!/bin/bash
# Extract the implementation of `createDatasetUrn` from `DatahubUtils` using rg to get more context.

rg --context 10 'public static DatasetUrn createDatasetUrn' metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/utils/DatahubUtils.java

Length of output: 969

---

157-161: LGTM! Added handling for lowercase dataset URNs.

The code checks the configuration and converts the namespace and dataset name to lowercase if enabled.

However, ensure that the lowerCaseDatasetUrns configuration is correctly set in the codebase.

Verification successful

---

LGTM! Added handling for lowercase dataset URNs.

The code checks the configuration and converts the namespace and dataset name to lowercase if enabled. The configuration lowerCaseDatasetUrns is correctly set and used in the codebase.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the usage of `lowerCaseDatasetUrns` configuration.

# Test: Search for the configuration usage. Expect: Occurrences of `lowerCaseDatasetUrns` in the codebase.
rg --type java $'lowerCaseDatasetUrns'

Length of output: 575

---

Script:

#!/bin/bash
# Description: Locate and display the implementation of `SparkConfigParser.isLowerCaseDatasetUrns`.

# Find the file containing the isLowerCaseDatasetUrns method.
file_path=$(rg --files-with-matches --type java 'isLowerCaseDatasetUrns' | head -n 1)

# Display the implementation of the method.
if [[ -n "$file_path" ]]; then
  rg --context 10 'isLowerCaseDatasetUrns' "$file_path"
else
  echo "Method isLowerCaseDatasetUrns not found."
fi

Length of output: 1154

---

Script:

#!/bin/bash
# Description: Locate and display the implementation of `SparkConfigParser.isLowerCaseDatasetUrns`.

# Find the file containing the SparkConfigParser class.
file_path=$(rg --files-with-matches --type java 'class SparkConfigParser' | head -n 1)

# Display the implementation of the isLowerCaseDatasetUrns method in the SparkConfigParser class.
if [[ -n "$file_path" ]]; then
  rg --context 10 'isLowerCaseDatasetUrns' "$file_path"
else
  echo "SparkConfigParser class not found."
fi

Length of output: 2164

---

218-232: LGTM! Added handling for platform instance configurations using PathSpec.

The code iterates through PathSpec to get the platform instance configuration for the specified platform and uses the common platform instance if not present.

However, ensure that the PathSpec configuration is correctly set in the codebase.

Verification successful

---

Verified: The PathSpec configuration is correctly set and utilized within the codebase.

The DatahubOpenlineageConfig class contains the necessary configuration for PathSpec, and the getPlatformInstance method in OpenLineageToDataHub.java correctly uses this configuration to fetch the platform instance.

• DatahubOpenlineageConfig.java:

  • Contains Map<String, List<PathSpec>> pathSpecs and method getPathSpecsForPlatform(String platform).

• PathSpec.java:

  • Defines fields for alias, platform, environment, path specification list, and platform instance.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the usage of `PathSpec` configuration.

# Test: Search for the configuration usage. Expect: Occurrences of `PathSpec` in the codebase.
rg --type java $'PathSpec'

Length of output: 47867

---

Script:

#!/bin/bash
# Description: Verify the configuration and usage of `PathSpec` within `DatahubOpenlineageConfig`.

# Search for the implementation details of `DatahubOpenlineageConfig` class.
ast-grep --lang java --pattern 'class DatahubOpenlineageConfig { $$$ }'

# Search for the implementation details of `PathSpec` class.
ast-grep --lang java --pattern 'class PathSpec { $$$ }'

Length of output: 131

---

Script:

#!/bin/bash
# Description: Verify the configuration and usage of `PathSpec` within `DatahubOpenlineageConfig`.

# Search for the definition of `DatahubOpenlineageConfig` class.
rg --type java 'class DatahubOpenlineageConfig'

# Search for the definition of `PathSpec` class.
rg --type java 'class PathSpec'

Length of output: 390

---

Script:

#!/bin/bash
# Description: Display the contents of `DatahubOpenlineageConfig` and `PathSpec` classes to verify their configurations and usages.

# Display the contents of `DatahubOpenlineageConfig.java`.
cat metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/config/DatahubOpenlineageConfig.java

# Display the contents of `PathSpec.java`.
cat metadata-integration/java/openlineage-converter/src/main/java/io/datahubproject/openlineage/dataset/PathSpec.java

Length of output: 2707

[coderabbitai]

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 5f904d4 and ec55767.

Files selected for processing (1)

• metadata-integration/java/spark-lineage-beta/README.md (7 hunks)

Additional context used

LanguageTool

metadata-integration/java/spark-lineage-beta/README.md

[grammar] ~184-~184: After ‘it’, use the third-person verb form “coalesces”.
Context: ... | | | Normally it coalesce and send metadata at the onApplicationE...

(IT_VBZ)

---

[uncategorized] ~185-~185: Did you mean: “By default,”?
Context: ...rwrites existing Dataset lineage edges. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[uncategorized] ~186-~186: Did you mean: “By default,”?
Context: ...this to true to lowercase dataset urns. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[style] ~187-~187: ‘prefer to have’ might be wordy. Consider a shorter alternative.
Context: ...use dataset symlink (for example if you prefer to have the s3 location instead of the Hive tab...

(EN_WORDINESS_PREMIUM_PREFER_TO_HAVE)

---

[uncategorized] ~187-~187: Did you mean: “By default,”?
Context: ...s3 location instead of the Hive table). By default it is disabled. ...

(BY_DEFAULT_COMMA)

Markdownlint

metadata-integration/java/spark-lineage-beta/README.md

189-189: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)

---

346-346: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

348-348: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

349-349: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

349-349: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

345-345: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

---

350-350: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

Additional comments not posted (6)

metadata-integration/java/spark-lineage-beta/README.md (6)

27-27: LGTM!

The configuration instruction for spark.jars.packages is correct.

---

35-35: LGTM!

The configuration instruction for spark-submit is correct.

---

44-44: LGTM!

The configuration instruction for Amazon EMR is correct.

---

59-59: LGTM!

The configuration instruction for Notebooks is correct.

---

82-82: LGTM!

The configuration instruction for Standalone Java Applications is correct.

---

162-162: LGTM!

The configuration options for the Spark agent are correct.

[coderabbitai]

Fix grammatical and punctuation errors.

There are minor grammatical and punctuation issues in the descriptions.

- By default it is disabled.
+ By default, it is disabled.

- Set this to true to lowercase dataset urns. By default it is disabled.
+ Set this to true to lowercase dataset URNs. By default, it is disabled.

- Set this to true if you prefer to have the s3 location instead of the Hive table. By default it is disabled.
+ Set this to true if you prefer using the S3 location instead of the Hive table. By default, it is disabled.

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	| spark.datahub.patch.enabled | | false | Set this to true to send lineage as a patch, which appends rather than overwrites existing Dataset lineage edges. By default it is disabled. |
	| spark.datahub.metadata.dataset.lowerCaseUrns | | false | Set this to true to lowercase dataset urns. By default it is disabled. |
	| spark.datahub.disableSymlinkResolution | | false | Set this to true if you prefer not use dataset symlink (for example if you prefer to have the s3 location instead of the Hive table). By default it is disabled. |
	| spark.datahub.patch.enabled | | false | Set this to true to send lineage as a patch, which appends rather than overwrites existing Dataset lineage edges. By default, it is disabled. |
	| spark.datahub.metadata.dataset.lowerCaseUrns | | false | Set this to true to lowercase dataset URNs. By default, it is disabled. |
	| spark.datahub.disableSymlinkResolution | | false | Set this to true if you prefer using the S3 location instead of the Hive table. By default, it is disabled. |

Tools

LanguageTool

[uncategorized] ~185-~185: Did you mean: “By default,”?
Context: ...rwrites existing Dataset lineage edges. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[uncategorized] ~186-~186: Did you mean: “By default,”?
Context: ...this to true to lowercase dataset urns. By default it is disabled. ...

(BY_DEFAULT_COMMA)

---

[style] ~187-~187: ‘prefer to have’ might be wordy. Consider a shorter alternative.
Context: ...use dataset symlink (for example if you prefer to have the s3 location instead of the Hive tab...

(EN_WORDINESS_PREMIUM_PREFER_TO_HAVE)

---

[uncategorized] ~187-~187: Did you mean: “By default,”?
Context: ...s3 location instead of the Hive table). By default it is disabled. ...

(BY_DEFAULT_COMMA)

[coderabbitai]

Actionable comments posted: 0

Outside diff range and nitpick comments (3)

metadata-ingestion/docs/sources/databricks/README.md (1)

11-11: Remove extra blank line.

Remove the extra blank line to maintain proper formatting.

- 

Tools

Markdownlint

11-11: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)

metadata-integration/java/spark-lineage-beta/README.md (2)

186-187: Fix grammatical and punctuation errors.

There are minor grammatical and punctuation issues in the descriptions.

- Set this to true to lowercase dataset urns. By default, it is disabled.
+ Set this to true to lowercase dataset URNs. By default, it is disabled.

- Set this to true if you prefer to have the s3 location instead of the Hive table. By default, it is disabled.
+ Set this to true if you prefer using the S3 location instead of the Hive table. By default, it is disabled.

---

347-358: Fix formatting issues.

There are minor formatting issues in the changelog.

- ## Changelog
+ 
+ ## Changelog
- ### Version 0.2.12
+ 
+ ### Version 0.2.12
- - Add option to lowercase dataset URNs
+ 
+ - Add option to lowercase dataset URNs

<details>
<summary>Tools</summary>

<details>
<summary>Markdownlint</summary><blockquote>

347-347: Expected: dash; Actual: plus
Unordered list style

(MD004, ul-style)

---

347-347: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)

---

348-348: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

350-350: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

353-353: null
Multiple headings with the same content

(MD024, no-duplicate-heading)

---

347-347: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

---

351-351: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

</blockquote></details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**

<details>
<summary>Commits</summary>

Files that changed from the base of the PR and between ec55767086cd61c692461505e6744e4b524c7a82 and 4db4ff8ba2e6005995a594c3b5c72526c17de3a5.

</details>


<details>
<summary>Files selected for processing (3)</summary>

* metadata-ingestion/docs/sources/databricks/README.md (1 hunks)
* metadata-integration/java/spark-lineage-beta/README.md (7 hunks)
* metadata-integration/java/spark-lineage-beta/src/main/java/io/openlineage/spark/agent/util/RddPathUtils.java (1 hunks)

</details>










<details>
<summary>Additional context used</summary>

<details>
<summary>Markdownlint</summary><blockquote>

<details>
<summary>metadata-ingestion/docs/sources/databricks/README.md</summary><blockquote>

11-11: Expected: 1; Actual: 2
Multiple consecutive blank lines

(MD012, no-multiple-blanks)

</blockquote></details>
<details>
<summary>metadata-integration/java/spark-lineage-beta/README.md</summary><blockquote>

347-347: Expected: dash; Actual: plus
Unordered list style

(MD004, ul-style)

---

347-347: Expected: 0 or 2; Actual: 1
Trailing spaces

(MD009, no-trailing-spaces)

---

345-345: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

348-348: Expected: 1; Actual: 0; Above
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

350-350: Expected: 1; Actual: 0; Below
Headings should be surrounded by blank lines

(MD022, blanks-around-headings)

---

353-353: null
Multiple headings with the same content

(MD024, no-duplicate-heading)

---

344-344: null
Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

---

347-347: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

---

351-351: null
Lists should be surrounded by blank lines

(MD032, blanks-around-lists)

</blockquote></details>

</blockquote></details>

</details>
<details>
<summary>Additional comments not posted (11)</summary><blockquote>

<details>
<summary>metadata-ingestion/docs/sources/databricks/README.md (1)</summary><blockquote>

`14-14`: **Approved: Addition is helpful.**

The addition of the recommendation to use the Spark agent for real-time activity and lineage tracking is helpful and aligns with the document's purpose.

</blockquote></details>
<details>
<summary>metadata-integration/java/spark-lineage-beta/src/main/java/io/openlineage/spark/agent/util/RddPathUtils.java (7)</summary><blockquote>

`29-40`: **Approved: Method is well-structured.**

The `findRDDPaths` method is well-structured and uses streams effectively to find paths from RDDs.

---

`42-53`: **Approved: Method is straightforward and correct.**

The `UnknownRDDExtractor` method correctly logs unknown RDD classes at the debug level and returns an empty stream.

---

`56-67`: **Approved: Method correctly extracts paths from HadoopRDDs.**

The `HadoopRDDExtractor` method correctly extracts paths from HadoopRDDs and maps them using `PlanUtils.getDirectoryPath`.

---

`70-80`: **Approved: Method is efficient.**

The `MapPartitionsRDDExtractor` method is efficient and leverages recursion to extract paths from MapPartitionsRDDs.

---

`83-107`: **Approved: Method correctly handles different Spark versions.**

The `FileScanRDDExtractor` method correctly handles different Spark versions and extracts paths from FileScanRDDs.

---

`110-146`: **Approved: Method correctly extracts paths from ParallelCollectionRDDs.**

The `ParallelCollectionRDDExtractor` method correctly extracts paths from ParallelCollectionRDDs and logs relevant debug information.

---

`149-154`: **Approved: Method is straightforward and handles exceptions.**

The `parentOf` method is straightforward and correctly handles exceptions by returning null.

</blockquote></details>
<details>
<summary>metadata-integration/java/spark-lineage-beta/README.md (3)</summary><blockquote>

`27-27`: **Approved: Configuration instructions are clear.**

The configuration instructions for spark-submit are clear and provide necessary details for configuring the Spark agent.

---

`44-44`: **Approved: Configuration instructions are clear.**

The configuration instructions for Amazon EMR are clear and provide necessary details for configuring the Spark agent on Amazon EMR.

---

`59-59`: **Approved: Configuration instructions are clear.**

The configuration instructions for Notebooks are clear and provide necessary details for configuring the Spark agent in Notebooks.

</blockquote></details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->