Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark: Fix historyUrl format #2741

Merged
merged 1 commit into from
May 30, 2024
Merged

Spark: Fix historyUrl format #2741

merged 1 commit into from
May 30, 2024

Conversation

dolfinus
Copy link
Contributor

Problem

spark_applicationDetails facet now contains value like http://history.server/application/app123. But it should be http://history.server/history/app123.

Solution

One-line summary:

Fix historyUrl format in spark_applicationDetails.

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

@boring-cyborg boring-cyborg bot added area:integration/spark area:tests Testing code language:java Uses Java programming language labels May 29, 2024
@dolfinus dolfinus force-pushed the bugfix/spark-history-url branch 4 times, most recently from 32e4e0d to 09688c5 Compare May 29, 2024 14:37
@dolfinus
Spark: Fix historyUrl format
e88f9b3 
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
@dolfinus dolfinus force-pushed the bugfix/spark-history-url branch from 09688c5 to e88f9b3 Compare May 29, 2024 14:37
@dolfinus dolfinus marked this pull request as ready for review May 29, 2024 15:03
@pawel-big-lebowski pawel-big-lebowski merged commit bf3e327 into OpenLineage:main May 30, 2024
34 checks passed
@dolfinus dolfinus deleted the bugfix/spark-history-url branch May 30, 2024 09:57
ngorchakova pushed a commit to ngorchakova/OpenLineage that referenced this pull request Jun 11, 2024
@dolfinus @ngorchakova
Spark: Fix historyUrl format (OpenLineage#2741)
844f3c3 
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>
harels pushed a commit that referenced this pull request Jun 11, 2024
@ngorchakova @pawel-big-lebowski
@dolfinus @kacpermuda @mobuchowski @dependabot @JDarDagran @tnazarew
Register GCP common job facet (#2740)
e2390d5 
* Register GCP common job facet

Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* [SPARK] verify jar content after build (#2698)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Apply prettier for json

Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Ignore registry.json files by generator

Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Spark: Fix historyUrl format (#2741)

Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Add Atlan as OpenLineage contributor (#2742)

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* [SPARK] fix drop table for Spark 3.4 and higher (#2745)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* spark: make sure debug logging is guarded when it can cause function call (#2744)

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Bump org.assertj:assertj-core from 3.25.3 to 3.26.0 in /client/java (#2747)

Bumps [org.assertj:assertj-core](https://github.com/assertj/assertj) from 3.25.3 to 3.26.0.
- [Release notes](https://github.com/assertj/assertj/releases)
- [Commits](assertj/assertj@assertj-build-3.25.3...assertj-build-3.26.0)

---
updated-dependencies:
- dependency-name: org.assertj:assertj-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* [SPARK] fix NPE in column level lineage (#2749)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Update the name for facet;
Update procedure to publish facets to documentation: ignore registry.json fileds

Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* alias: allow self-recursive aliases (#2753)

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* fix changelog (#2759)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* [SPARK] refactor OpenLineageRunEventBuilder (#2754)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Bump mypy to 1.10. (#2760)

Use attr scope so that attributes named `field` do not break static checks.

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* Dataset host resolver feature (#2720)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

* remodeled transformation type (#2756)

* generated python client, backward compatibility fixed

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

Update class after #2760 fix.

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>

* move required fields into transformation object

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

Co-authored-by: Jakub Dardzinski <kuba0221@gmail.com>

* update python classes

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* type changed to DIRECT|INDIRECT

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* add changelog

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* change deprecation and fix changelog

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* add deprecated field info to changelog

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* fix redact_fields for Transformation

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

* updated generated python class

Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>

---------

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Co-authored-by: Jakub Dardzinski <kuba0221@gmail.com>
Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>

---------

Signed-off-by: Natalia Gorchakova <ngorchakova@google.com>
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Co-authored-by: pawel.leszczynski <leszczynski.pawel@gmail.com>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: Kacper Muda <mudakacper@gmail.com>
Co-authored-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jakub Dardzinski <kuba0221@gmail.com>
Co-authored-by: tnazarew <tomasz.nazarewicz@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pawel-big-lebowski pawel-big-lebowski pawel-big-lebowski approved these changes

Assignees
No one assigned
Labels
area:integration/spark area:tests Testing code language:java Uses Java programming language
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dolfinus @pawel-big-lebowski