Allow instrumented Spark trace linked to Openlineage originated context #7450

yiliangzhou · 2024-08-15T21:26:08Z

What Does This Do

If Openlineage context (parent job namespace, job name, run id) is detected in the Spark configuration, the Spark instrumentation generates the TraceID and SpanId for the root Span in a certain way such that it matches the upstream's cue, allowing the Trace from upstream task and the Spark trace linked together in Datadog.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Squash your commits prior merging or merge using GitHub's Squash and merge
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

… if present

…ineage context is present

…ontext

pr-commenter · 2024-08-20T19:22:15Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	liangzhou.yi/airflow-to-spark-lineage
git_commit_date	1724427865	1724432989
git_commit_sha	`62eb0c2`	`dbf8e55`
release_version	1.39.0-SNAPSHOT~62eb0c29c3	1.39.0-SNAPSHOT~dbf8e55f45

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1724436179	1724436179
ci_job_id	615609270	615609270
ci_pipeline_id	42708259	42708259
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.557 s) : 0, 1557441
Total [baseline] (14.339 s) : 0, 14338778
Agent [candidate] (1.551 s) : 0, 1550988
Total [candidate] (14.22 s) : 0, 14219821
section appsec
Agent [baseline] (1.738 s) : 0, 1737696
Total [baseline] (14.416 s) : 0, 14415581
Agent [candidate] (1.742 s) : 0, 1741811
Total [candidate] (14.426 s) : 0, 14425679
section iast
Agent [baseline] (1.717 s) : 0, 1717318
Total [baseline] (14.838 s) : 0, 14837590
Agent [candidate] (1.72 s) : 0, 1719632
Total [candidate] (14.889 s) : 0, 14888675
section profiling
Agent [baseline] (1.861 s) : 0, 1861448
Total [baseline] (14.682 s) : 0, 14682104
Agent [candidate] (1.864 s) : 0, 1863880
Total [candidate] (14.581 s) : 0, 14581208

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.557 s	-
Agent	appsec	1.738 s	180.255 ms (11.6%)
Agent	iast	1.717 s	159.877 ms (10.3%)
Agent	profiling	1.861 s	304.008 ms (19.5%)
Total	tracing	14.339 s	-
Total	appsec	14.416 s	76.803 ms (0.5%)
Total	iast	14.838 s	498.812 ms (3.5%)
Total	profiling	14.682 s	343.326 ms (2.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.551 s	-
Agent	appsec	1.742 s	190.823 ms (12.3%)
Agent	iast	1.72 s	168.644 ms (10.9%)
Agent	profiling	1.864 s	312.892 ms (20.2%)
Total	tracing	14.22 s	-
Total	appsec	14.426 s	205.859 ms (1.4%)
Total	iast	14.889 s	668.854 ms (4.7%)
Total	profiling	14.581 s	361.387 ms (2.5%)

gantt
    title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (998.134 ms) : 0, 998134
BytebuddyAgent [candidate] (993.596 ms) : 0, 993596
GlobalTracer [baseline] (455.712 ms) : 0, 455712
GlobalTracer [candidate] (454.724 ms) : 0, 454724
AppSec [baseline] (72.722 ms) : 0, 72722
AppSec [candidate] (72.026 ms) : 0, 72026
Remote Config [baseline] (838.94 µs) : 0, 839
Remote Config [candidate] (820.659 µs) : 0, 821
Telemetry [baseline] (9.921 ms) : 0, 9921
Telemetry [candidate] (9.731 ms) : 0, 9731
section appsec
BytebuddyAgent [baseline] (1.009 s) : 0, 1008825
BytebuddyAgent [candidate] (1.013 s) : 0, 1013223
GlobalTracer [baseline] (447.524 ms) : 0, 447524
GlobalTracer [candidate] (449.108 ms) : 0, 449108
AppSec [baseline] (235.562 ms) : 0, 235562
AppSec [candidate] (234.952 ms) : 0, 234952
Remote Config [baseline] (753.111 µs) : 0, 753
Remote Config [candidate] (759.404 µs) : 0, 759
Telemetry [baseline] (12.657 ms) : 0, 12657
Telemetry [candidate] (11.287 ms) : 0, 11287
IAST [baseline] (24.732 ms) : 0, 24732
IAST [candidate] (24.769 ms) : 0, 24769
section iast
BytebuddyAgent [baseline] (1.148 s) : 0, 1147976
BytebuddyAgent [candidate] (1.15 s) : 0, 1149683
GlobalTracer [baseline] (435.751 ms) : 0, 435751
GlobalTracer [candidate] (436.177 ms) : 0, 436177
AppSec [baseline] (71.059 ms) : 0, 71059
AppSec [candidate] (73.903 ms) : 0, 73903
Remote Config [baseline] (749.999 µs) : 0, 750
Remote Config [candidate] (744.462 µs) : 0, 744
Telemetry [baseline] (11.695 ms) : 0, 11695
Telemetry [candidate] (10.811 ms) : 0, 10811
IAST [baseline] (30.09 ms) : 0, 30090
IAST [candidate] (28.19 ms) : 0, 28190
section profiling
ProfilingAgent [baseline] (151.196 ms) : 0, 151196
ProfilingAgent [candidate] (151.089 ms) : 0, 151089
BytebuddyAgent [baseline] (984.728 ms) : 0, 984728
BytebuddyAgent [candidate] (987.246 ms) : 0, 987246
GlobalTracer [baseline] (585.511 ms) : 0, 585511
GlobalTracer [candidate] (585.32 ms) : 0, 585320
AppSec [baseline] (73.785 ms) : 0, 73785
AppSec [candidate] (73.832 ms) : 0, 73832
Remote Config [baseline] (911.412 µs) : 0, 911
Remote Config [candidate] (890.697 µs) : 0, 891
Telemetry [baseline] (9.528 ms) : 0, 9528
Telemetry [candidate] (9.417 ms) : 0, 9417
Profiling [baseline] (151.271 ms) : 0, 151271
Profiling [candidate] (151.161 ms) : 0, 151161

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.554 s) : 0, 1553782
Total [baseline] (11.883 s) : 0, 11883367
Agent [candidate] (1.557 s) : 0, 1557160
Total [candidate] (11.817 s) : 0, 11817159
section iast
Agent [baseline] (1.718 s) : 0, 1718330
Total [baseline] (12.511 s) : 0, 12510728
Agent [candidate] (1.719 s) : 0, 1719347
Total [candidate] (12.54 s) : 0, 12540460
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.72 s) : 0, 1719631
Total [baseline] (12.466 s) : 0, 12466455
Agent [candidate] (1.718 s) : 0, 1717898
Total [candidate] (12.464 s) : 0, 12464250
section iast_TELEMETRY_OFF
Agent [baseline] (1.715 s) : 0, 1715378
Total [baseline] (12.503 s) : 0, 12502590
Agent [candidate] (1.712 s) : 0, 1711637
Total [candidate] (12.479 s) : 0, 12478545

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.554 s	-
Agent	iast	1.718 s	164.548 ms (10.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.72 s	165.849 ms (10.7%)
Agent	iast_TELEMETRY_OFF	1.715 s	161.597 ms (10.4%)
Total	tracing	11.883 s	-
Total	iast	12.511 s	627.361 ms (5.3%)
Total	iast_HARDCODED_SECRET_DISABLED	12.466 s	583.088 ms (4.9%)
Total	iast_TELEMETRY_OFF	12.503 s	619.223 ms (5.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.557 s	-
Agent	iast	1.719 s	162.187 ms (10.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.718 s	160.738 ms (10.3%)
Agent	iast_TELEMETRY_OFF	1.712 s	154.476 ms (9.9%)
Total	tracing	11.817 s	-
Total	iast	12.54 s	723.301 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	12.464 s	647.092 ms (5.5%)
Total	iast_TELEMETRY_OFF	12.479 s	661.386 ms (5.6%)

gantt
    title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (995.913 ms) : 0, 995913
BytebuddyAgent [candidate] (997.881 ms) : 0, 997881
GlobalTracer [baseline] (454.77 ms) : 0, 454770
GlobalTracer [candidate] (455.731 ms) : 0, 455731
AppSec [baseline] (72.436 ms) : 0, 72436
AppSec [candidate] (72.661 ms) : 0, 72661
Remote Config [baseline] (822.884 µs) : 0, 823
Remote Config [candidate] (830.734 µs) : 0, 831
Telemetry [baseline] (9.798 ms) : 0, 9798
Telemetry [candidate] (9.88 ms) : 0, 9880
section iast
BytebuddyAgent [baseline] (1.149 s) : 0, 1148848
BytebuddyAgent [candidate] (1.15 s) : 0, 1150048
GlobalTracer [baseline] (435.991 ms) : 0, 435991
GlobalTracer [candidate] (436.406 ms) : 0, 436406
AppSec [baseline] (72.73 ms) : 0, 72730
AppSec [candidate] (71.043 ms) : 0, 71043
IAST [baseline] (29.062 ms) : 0, 29062
IAST [candidate] (29.216 ms) : 0, 29216
Remote Config [baseline] (741.232 µs) : 0, 741
Remote Config [candidate] (785.791 µs) : 0, 786
Telemetry [baseline] (10.899 ms) : 0, 10899
Telemetry [candidate] (11.727 ms) : 0, 11727
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (1.15 s) : 0, 1149879
BytebuddyAgent [candidate] (1.149 s) : 0, 1148576
GlobalTracer [baseline] (436.072 ms) : 0, 436072
GlobalTracer [candidate] (436.176 ms) : 0, 436176
AppSec [baseline] (74.548 ms) : 0, 74548
AppSec [candidate] (73.78 ms) : 0, 73780
IAST [baseline] (28.237 ms) : 0, 28237
IAST [candidate] (29.254 ms) : 0, 29254
Remote Config [baseline] (734.232 µs) : 0, 734
Remote Config [candidate] (734.337 µs) : 0, 734
Telemetry [baseline] (10.08 ms) : 0, 10080
Telemetry [candidate] (9.225 ms) : 0, 9225
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (1.147 s) : 0, 1146952
BytebuddyAgent [candidate] (1.143 s) : 0, 1143402
GlobalTracer [baseline] (435.846 ms) : 0, 435846
GlobalTracer [candidate] (435.404 ms) : 0, 435404
AppSec [baseline] (70.251 ms) : 0, 70251
AppSec [candidate] (71.81 ms) : 0, 71810
IAST [baseline] (30.48 ms) : 0, 30480
IAST [candidate] (29.299 ms) : 0, 29299
Remote Config [baseline] (780.532 µs) : 0, 781
Remote Config [candidate] (737.755 µs) : 0, 738
Telemetry [baseline] (11.011 ms) : 0, 11011
Telemetry [candidate] (10.886 ms) : 0, 10886

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-08-23T17:19:07	2024-08-23T17:25:57
git_branch	master	liangzhou.yi/airflow-to-spark-lineage
git_commit_date	1724427865	1724432989
git_commit_sha	`62eb0c2`	`dbf8e55`
release_version	1.39.0-SNAPSHOT~62eb0c29c3	1.39.0-SNAPSHOT~dbf8e55f45
start_time	2024-08-23T17:18:54	2024-08-23T17:25:44

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1724434303	1724434303
ci_job_id	615609271	615609271
ci_pipeline_id	42708259	42708259
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.191 µs) : 350, 390
.   : milestone, 370,
iast (487.51 µs) : 465, 510
.   : milestone, 488,
iast_FULL (553.61 µs) : 532, 576
.   : milestone, 554,
iast_GLOBAL (517.94 µs) : 494, 542
.   : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (485.119 µs) : 463, 508
.   : milestone, 485,
iast_INACTIVE (447.176 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (477.253 µs) : 456, 499
.   : milestone, 477,
tracing (440.191 µs) : 420, 461
.   : milestone, 440,
section candidate
no_agent (365.662 µs) : 346, 385
.   : milestone, 366,
iast (480.7 µs) : 458, 503
.   : milestone, 481,
iast_FULL (558.255 µs) : 537, 579
.   : milestone, 558,
iast_GLOBAL (523.341 µs) : 500, 546
.   : milestone, 523,
iast_HARDCODED_SECRET_DISABLED (493.79 µs) : 472, 516
.   : milestone, 494,
iast_INACTIVE (443.176 µs) : 423, 464
.   : milestone, 443,
iast_TELEMETRY_OFF (465.545 µs) : 445, 486
.   : milestone, 466,
tracing (447.574 µs) : 427, 468
.   : milestone, 448,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.191 µs [350.491 µs, 389.891 µs]	-
iast	487.51 µs [464.973 µs, 510.048 µs]	117.319 µs (31.7%)
iast_FULL	553.61 µs [531.709 µs, 575.51 µs]	183.419 µs (49.5%)
iast_GLOBAL	517.94 µs [494.239 µs, 541.64 µs]	147.749 µs (39.9%)
iast_HARDCODED_SECRET_DISABLED	485.119 µs [462.67 µs, 507.567 µs]	114.928 µs (31.0%)
iast_INACTIVE	447.176 µs [426.021 µs, 468.331 µs]	76.985 µs (20.8%)
iast_TELEMETRY_OFF	477.253 µs [455.816 µs, 498.69 µs]	107.062 µs (28.9%)
tracing	440.191 µs [419.554 µs, 460.827 µs]	70.0 µs (18.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.662 µs [346.063 µs, 385.26 µs]	-
iast	480.7 µs [458.346 µs, 503.054 µs]	115.039 µs (31.5%)
iast_FULL	558.255 µs [537.142 µs, 579.369 µs]	192.594 µs (52.7%)
iast_GLOBAL	523.341 µs [500.231 µs, 546.451 µs]	157.679 µs (43.1%)
iast_HARDCODED_SECRET_DISABLED	493.79 µs [471.632 µs, 515.947 µs]	128.128 µs (35.0%)
iast_INACTIVE	443.176 µs [422.697 µs, 463.656 µs]	77.515 µs (21.2%)
iast_TELEMETRY_OFF	465.545 µs [444.637 µs, 486.453 µs]	99.883 µs (27.3%)
tracing	447.574 µs [427.031 µs, 468.118 µs]	81.913 µs (22.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.359 ms) : 1340, 1379
.   : milestone, 1359,
appsec (1.737 ms) : 1714, 1761
.   : milestone, 1737,
appsec_no_iast (1.715 ms) : 1691, 1739
.   : milestone, 1715,
iast (1.479 ms) : 1456, 1502
.   : milestone, 1479,
profiling (1.541 ms) : 1514, 1567
.   : milestone, 1541,
tracing (1.469 ms) : 1446, 1493
.   : milestone, 1469,
section candidate
no_agent (1.351 ms) : 1332, 1370
.   : milestone, 1351,
appsec (1.722 ms) : 1699, 1745
.   : milestone, 1722,
appsec_no_iast (1.735 ms) : 1712, 1759
.   : milestone, 1735,
iast (1.48 ms) : 1457, 1503
.   : milestone, 1480,
profiling (1.486 ms) : 1462, 1511
.   : milestone, 1486,
tracing (1.455 ms) : 1430, 1480
.   : milestone, 1455,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.359 ms [1.34 ms, 1.379 ms]	-
appsec	1.737 ms [1.714 ms, 1.761 ms]	377.827 µs (27.8%)
appsec_no_iast	1.715 ms [1.691 ms, 1.739 ms]	355.908 µs (26.2%)
iast	1.479 ms [1.456 ms, 1.502 ms]	119.502 µs (8.8%)
profiling	1.541 ms [1.514 ms, 1.567 ms]	181.07 µs (13.3%)
tracing	1.469 ms [1.446 ms, 1.493 ms]	109.935 µs (8.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.332 ms, 1.37 ms]	-
appsec	1.722 ms [1.699 ms, 1.745 ms]	371.174 µs (27.5%)
appsec_no_iast	1.735 ms [1.712 ms, 1.759 ms]	384.471 µs (28.5%)
iast	1.48 ms [1.457 ms, 1.503 ms]	128.88 µs (9.5%)
profiling	1.486 ms [1.462 ms, 1.511 ms]	135.397 µs (10.0%)
tracing	1.455 ms [1.43 ms, 1.48 ms]	104.541 µs (7.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	liangzhou.yi/airflow-to-spark-lineage
git_commit_date	1724427865	1724432989
git_commit_sha	`62eb0c2`	`dbf8e55`
release_version	1.39.0-SNAPSHOT~62eb0c29c3	1.39.0-SNAPSHOT~dbf8e55f45

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1724434922	1724434922
ci_job_id	615609272	615609272
ci_pipeline_id	42708259	42708259
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.454 ms) : 1443, 1466
.   : milestone, 1454,
appsec (2.231 ms) : 2196, 2265
.   : milestone, 2231,
iast (1.976 ms) : 1933, 2019
.   : milestone, 1976,
iast_GLOBAL (2.033 ms) : 1989, 2078
.   : milestone, 2033,
profiling (1.856 ms) : 1822, 1890
.   : milestone, 1856,
tracing (1.834 ms) : 1801, 1866
.   : milestone, 1834,
section candidate
no_agent (1.462 ms) : 1450, 1473
.   : milestone, 1462,
appsec (2.225 ms) : 2190, 2260
.   : milestone, 2225,
iast (1.966 ms) : 1924, 2008
.   : milestone, 1966,
iast_GLOBAL (2.027 ms) : 1983, 2070
.   : milestone, 2027,
profiling (1.854 ms) : 1820, 1888
.   : milestone, 1854,
tracing (1.847 ms) : 1814, 1880
.   : milestone, 1847,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.454 ms [1.443 ms, 1.466 ms]	-
appsec	2.231 ms [2.196 ms, 2.265 ms]	776.106 µs (53.4%)
iast	1.976 ms [1.933 ms, 2.019 ms]	521.768 µs (35.9%)
iast_GLOBAL	2.033 ms [1.989 ms, 2.078 ms]	578.628 µs (39.8%)
profiling	1.856 ms [1.822 ms, 1.89 ms]	401.604 µs (27.6%)
tracing	1.834 ms [1.801 ms, 1.866 ms]	379.452 µs (26.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.462 ms [1.45 ms, 1.473 ms]	-
appsec	2.225 ms [2.19 ms, 2.26 ms]	763.008 µs (52.2%)
iast	1.966 ms [1.924 ms, 2.008 ms]	504.634 µs (34.5%)
iast_GLOBAL	2.027 ms [1.983 ms, 2.07 ms]	564.707 µs (38.6%)
profiling	1.854 ms [1.82 ms, 1.888 ms]	392.403 µs (26.8%)
tracing	1.847 ms [1.814 ms, 1.88 ms]	385.238 µs (26.4%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~dbf8e55f45, baseline=1.39.0-SNAPSHOT~62eb0c29c3
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.399 s) : 15399000, 15399000
.   : milestone, 15399000,
appsec (14.973 s) : 14973000, 14973000
.   : milestone, 14973000,
iast (18.808 s) : 18808000, 18808000
.   : milestone, 18808000,
iast_GLOBAL (17.948 s) : 17948000, 17948000
.   : milestone, 17948000,
profiling (15.92 s) : 15920000, 15920000
.   : milestone, 15920000,
tracing (14.964 s) : 14964000, 14964000
.   : milestone, 14964000,
section candidate
no_agent (15.503 s) : 15503000, 15503000
.   : milestone, 15503000,
appsec (14.898 s) : 14898000, 14898000
.   : milestone, 14898000,
iast (18.826 s) : 18826000, 18826000
.   : milestone, 18826000,
iast_GLOBAL (17.917 s) : 17917000, 17917000
.   : milestone, 17917000,
profiling (15.395 s) : 15395000, 15395000
.   : milestone, 15395000,
tracing (15.002 s) : 15002000, 15002000
.   : milestone, 15002000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.399 s [15.399 s, 15.399 s]	-
appsec	14.973 s [14.973 s, 14.973 s]	-426.0 ms (-2.8%)
iast	18.808 s [18.808 s, 18.808 s]	3.409 s (22.1%)
iast_GLOBAL	17.948 s [17.948 s, 17.948 s]	2.549 s (16.6%)
profiling	15.92 s [15.92 s, 15.92 s]	521.0 ms (3.4%)
tracing	14.964 s [14.964 s, 14.964 s]	-435.0 ms (-2.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.503 s [15.503 s, 15.503 s]	-
appsec	14.898 s [14.898 s, 14.898 s]	-605.0 ms (-3.9%)
iast	18.826 s [18.826 s, 18.826 s]	3.323 s (21.4%)
iast_GLOBAL	17.917 s [17.917 s, 17.917 s]	2.414 s (15.6%)
profiling	15.395 s [15.395 s, 15.395 s]	-108.0 ms (-0.7%)
tracing	15.002 s [15.002 s, 15.002 s]	-501.0 ms (-3.2%)

…context is present

PerfectSlayer

Few minor comments from Language Platform POV.
You might want a review from Integrations and Data Modelling too

...tation/spark/src/main/java/datadog/trace/instrumentation/spark/OpenlineageParentContext.java

.../testFixtures/groovy/datadog/trace/instrumentation/spark/OpenlineageParentContextTest.groovy

…and runId being a valid UUID

PerfectSlayer

LGTM! Thanks for taking into account my comments 👍

amarziali · 2024-08-23T07:28:16Z

Removing IDM since there does not seem to have any changes on our owned files

yiliangzhou · 2024-08-23T15:55:01Z

/merge

dd-devflow · 2024-08-23T15:55:10Z

🚂 MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

dd-devflow · 2024-08-23T19:55:12Z

⚠️ MergeQueue: This merge request was unqueued

This merge request was unqueued

If you need support, contact us on Slack #devflow!

PerfectSlayer · 2024-08-26T09:13:54Z

Did the bot just created a bunch of label on the repo by himself? 🤔

yiliangzhou added 5 commits August 6, 2024 17:10

add OpenlineageContext to capture the parent context from Openlineage…

1f7e31b

… if present

create parent span id from airflow task run for OpenLineageParentContext

6fcfb34

Merge branch 'master' into liangzhou.yi/airflow-to-spark-lineage

31c2ac8

Creat root span's trace id and span id for spark application if openl…

46d882e

…ineage context is present

add tests for OpenlineageParentContext

66f3390

yiliangzhou changed the title ~~Liangzhou.yi/airflow to spark lineage~~ Allow instrumented Spark trace linked to Openlineage originated context Aug 16, 2024

yiliangzhou requested a review from kr-igor August 16, 2024 21:02

yiliangzhou added 2 commits August 20, 2024 14:40

ensure all openlineage context present when we capture it as parent c…

32116f9

…ontext

Merge branch 'master' into liangzhou.yi/airflow-to-spark-lineage

0dc4674

yiliangzhou marked this pull request as ready for review August 20, 2024 18:43

yiliangzhou requested a review from a team as a code owner August 20, 2024 18:43

kr-igor approved these changes Aug 20, 2024

View reviewed changes

Use SHA-256 in generating trace id and root span id when Openlineage …

0784808

…context is present

yiliangzhou requested review from a team, kr-igor and smola and removed request for a team August 20, 2024 22:11

PerfectSlayer approved these changes Aug 22, 2024

View reviewed changes

create OpenlineageParentContext only all required fields are present …

fdeb1fb

…and runId being a valid UUID

yiliangzhou requested a review from PerfectSlayer August 22, 2024 20:44

Merge branch 'master' into liangzhou.yi/airflow-to-spark-lineage

33038c8

yiliangzhou requested a review from a team August 22, 2024 21:02

PerfectSlayer approved these changes Aug 23, 2024

View reviewed changes

amarziali removed the request for review from a team August 23, 2024 07:27

dd-devflow bot added the mergequeue-status: waiting label Aug 23, 2024

Merge branch 'master' into liangzhou.yi/airflow-to-spark-lineage

dbf8e55

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Aug 23, 2024

update test to avoid codenarcTestFixtures complaints

252431c

yiliangzhou merged commit 6326167 into master Aug 23, 2024

yiliangzhou deleted the liangzhou.yi/airflow-to-spark-lineage branch August 23, 2024 21:19

github-actions bot added this to the 1.39.0 milestone Aug 23, 2024

nikita-tkachenko-datadog added the inst: apache spark Apache Spark instrumentation label Sep 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow instrumented Spark trace linked to Openlineage originated context #7450

Allow instrumented Spark trace linked to Openlineage originated context #7450

Uh oh!

yiliangzhou commented Aug 15, 2024 •

edited

Loading

Uh oh!

pr-commenter bot commented Aug 20, 2024 •

edited

Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer left a comment

Uh oh!

amarziali commented Aug 23, 2024

Uh oh!

yiliangzhou commented Aug 23, 2024

Uh oh!

dd-devflow bot commented Aug 23, 2024

Uh oh!

dd-devflow bot commented Aug 23, 2024

Uh oh!

PerfectSlayer commented Aug 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Allow instrumented Spark trace linked to Openlineage originated context #7450

Allow instrumented Spark trace linked to Openlineage originated context #7450

Uh oh!

Conversation

yiliangzhou commented Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

amarziali commented Aug 23, 2024

Uh oh!

yiliangzhou commented Aug 23, 2024

Uh oh!

dd-devflow bot commented Aug 23, 2024

Uh oh!

dd-devflow bot commented Aug 23, 2024

Uh oh!

PerfectSlayer commented Aug 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yiliangzhou commented Aug 15, 2024 •

edited

Loading

pr-commenter bot commented Aug 20, 2024 •

edited

Loading