Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework Profile tool to not require Spark to run and process files faster #3161

Merged
merged 246 commits into from
Aug 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
246 commits
Select commit Hold shift + click to select a range
97a31d6
Start redesign profiling tool
tgravescs Jul 19, 2021
fa25376
comment stuff out
tgravescs Jul 19, 2021
1eff270
comment out
tgravescs Jul 19, 2021
89bc2f1
comment out tests
tgravescs Jul 19, 2021
1e42bcf
print
tgravescs Jul 19, 2021
f23bbac
debug
tgravescs Jul 19, 2021
44b8c60
fix print:
tgravescs Jul 19, 2021
3363646
remove sparkSession usage
tgravescs Jul 19, 2021
0051455
test fixes
tgravescs Jul 19, 2021
292e34a
add printing
tgravescs Jul 20, 2021
96aa46a
add writer
tgravescs Jul 20, 2021
26d2ae3
add schema
tgravescs Jul 20, 2021
8c73f93
add appindex
tgravescs Jul 20, 2021
cd68f48
error
tgravescs Jul 20, 2021
aa4293c
add rapids jars output
tgravescs Jul 20, 2021
3f399d4
add app index
tgravescs Jul 20, 2021
f79e96e
print executor info
tgravescs Jul 21, 2021
50552d7
print job info
tgravescs Jul 21, 2021
56461b1
remove extra
tgravescs Jul 21, 2021
6ec4872
sort
tgravescs Jul 21, 2021
5ead8c2
fix cast
tgravescs Jul 21, 2021
5cc8095
print data source info
tgravescs Jul 21, 2021
cc0e7ac
add in properties
tgravescs Jul 21, 2021
a302f85
change to startswith
tgravescs Jul 21, 2021
967adf6
show failed tasks
tgravescs Jul 21, 2021
65b4820
fix substr
tgravescs Jul 21, 2021
6719eeb
add in failed stages and jobs
tgravescs Jul 21, 2021
2725024
fix truncate
tgravescs Jul 21, 2021
8ae3eaf
fix truncate
tgravescs Jul 21, 2021
a2b26a1
list removed executors
tgravescs Jul 21, 2021
d7bc81c
call
tgravescs Jul 21, 2021
79f783e
chagne to ms from date
tgravescs Jul 21, 2021
e127cd8
long
tgravescs Jul 21, 2021
44ff351
list unsupported ops
tgravescs Jul 21, 2021
9b29d3f
comments
tgravescs Jul 22, 2021
4bd0950
debug
tgravescs Jul 22, 2021
801ace6
print sql plan metrics
tgravescs Jul 22, 2021
7bc954f
fixes
tgravescs Jul 22, 2021
f25df90
debug
tgravescs Jul 22, 2021
43e1fc1
log
tgravescs Jul 22, 2021
d79e252
add driver accums
tgravescs Jul 22, 2021
278ae9e
debug
tgravescs Jul 22, 2021
91bd59f
add driver accums
tgravescs Jul 22, 2021
73492d5
debug
tgravescs Jul 22, 2021
d45dd8d
leave off no settings
tgravescs Jul 22, 2021
e61f43a
fix 0L
tgravescs Jul 22, 2021
4ae12d1
remove debug
tgravescs Jul 22, 2021
3fc4aba
remove log
tgravescs Jul 22, 2021
3967b94
add in task metrics
tgravescs Jul 27, 2021
91ae9e5
fix type
tgravescs Jul 27, 2021
d53db05
fix sort
tgravescs Jul 27, 2021
41cadca
more cols
tgravescs Jul 27, 2021
ef1a90c
fix missing
tgravescs Jul 27, 2021
9f132a6
debug
tgravescs Jul 27, 2021
ec399f3
fix flatmap
tgravescs Jul 27, 2021
0661fb0
fix order
tgravescs Jul 27, 2021
cbc97c1
remove
tgravescs Jul 27, 2021
85de497
add duration
tgravescs Jul 27, 2021
166f07e
fix stage dur
tgravescs Jul 27, 2021
5a0ebb2
sort desc
tgravescs Jul 27, 2021
5c59de2
change average
tgravescs Jul 27, 2021
7ebcc33
fix job
tgravescs Jul 27, 2021
9a7dc9f
sql metrics agg
tgravescs Jul 27, 2021
ed70b09
debug
tgravescs Jul 27, 2021
1aa0883
fix dup columns
tgravescs Jul 27, 2021
1bae96b
add appid
tgravescs Jul 27, 2021
acbfd75
fix sorting
tgravescs Jul 27, 2021
5ccb342
redo sql agg
tgravescs Jul 27, 2021
47dcbcd
debug
tgravescs Jul 27, 2021
060537e
fix sqlid comparison
tgravescs Jul 27, 2021
516643d
debug
tgravescs Jul 27, 2021
92f67bb
try
tgravescs Jul 27, 2021
40e7795
fix
tgravescs Jul 27, 2021
338bce4
empty
tgravescs Jul 27, 2021
8d4bc1c
nonempty
tgravescs Jul 27, 2021
9e65076
fix cpu ratio
tgravescs Jul 27, 2021
4fe621b
sql duration and exec time percent
tgravescs Jul 27, 2021
4da9b0b
fix
tgravescs Jul 27, 2021
5cd131e
null
tgravescs Jul 27, 2021
448662e
fix to string
tgravescs Jul 27, 2021
dd3e9f2
fix header
tgravescs Jul 27, 2021
251866d
fix spelling
tgravescs Jul 27, 2021
b909f20
skew
tgravescs Jul 27, 2021
90b1f49
fixes
tgravescs Jul 27, 2021
ba50122
fix
tgravescs Jul 27, 2021
0e563b3
format
tgravescs Jul 27, 2021
662b885
fix
tgravescs Jul 27, 2021
7686b9f
fix
tgravescs Jul 27, 2021
7036a01
generate dot
tgravescs Jul 28, 2021
cb56f1e
fix
tgravescs Jul 28, 2021
5d5f0a9
fix generate dot and cleanup
tgravescs Jul 28, 2021
d9382f4
Enable timeline
tgravescs Jul 28, 2021
4b515ff
compare
tgravescs Jul 28, 2021
2d3dda8
fix imports
tgravescs Jul 28, 2021
acb03b5
comment out compare
tgravescs Jul 28, 2021
c937427
more compare
tgravescs Jul 28, 2021
219c721
fix empty
tgravescs Jul 28, 2021
9312896
debug
tgravescs Jul 28, 2021
8b5ae49
debug
tgravescs Jul 28, 2021
002c7c7
debug
tgravescs Jul 28, 2021
4d16ac3
fix
tgravescs Jul 28, 2021
37346ee
na to null
tgravescs Jul 28, 2021
4af7c9b
compare
tgravescs Jul 28, 2021
dae0709
add headers
tgravescs Jul 28, 2021
519c552
cleanup
tgravescs Jul 28, 2021
7f2d99f
compare
tgravescs Jul 28, 2021
71acf4b
multi-thread and cleanup
tgravescs Jul 28, 2021
d69c8df
Merge remote-tracking branch 'origin/branch-21.08' into profileRedesign
tgravescs Jul 28, 2021
e514fb2
add Profiler class
tgravescs Jul 28, 2021
b14fde7
fix empty string
tgravescs Jul 28, 2021
1a217d9
fix returns
tgravescs Jul 28, 2021
3655502
fix creating threadpool
tgravescs Jul 28, 2021
49fe625
fix
tgravescs Jul 28, 2021
285b8bc
cchange what is parallel
tgravescs Jul 28, 2021
3d9380c
one file per event log
tgravescs Jul 28, 2021
d6c4d44
fix empty string
tgravescs Jul 29, 2021
1a8e414
mopre empty
tgravescs Jul 29, 2021
e0e47fe
don't turn string to long
tgravescs Jul 29, 2021
c18c681
sort by appindex
tgravescs Jul 29, 2021
c5d9d4c
fix typo
tgravescs Jul 29, 2021
491e0bd
cleanup
tgravescs Jul 29, 2021
de8bdd0
fix
tgravescs Jul 29, 2021
60ac41a
more fixes
tgravescs Jul 29, 2021
567f869
fix
tgravescs Jul 29, 2021
359c3e4
fix test
tgravescs Jul 29, 2021
a484336
fix
tgravescs Jul 29, 2021
eb39098
fix size
tgravescs Jul 29, 2021
073e808
enable tests
tgravescs Jul 30, 2021
fe6a996
fixes
tgravescs Jul 30, 2021
070530e
fix test
tgravescs Jul 30, 2021
8c6c4ad
try
tgravescs Jul 30, 2021
adb4431
test
tgravescs Jul 30, 2021
7c3b609
debug
tgravescs Jul 30, 2021
86b260d
to string
tgravescs Jul 30, 2021
a4bef0b
try again
tgravescs Jul 30, 2021
87f5019
fix syntax
tgravescs Jul 30, 2021
4b29706
executor prof result
tgravescs Jul 30, 2021
b9ad7eb
sql
tgravescs Jul 30, 2021
0cc83f4
add test
tgravescs Jul 30, 2021
6a4ce4d
fix
tgravescs Jul 30, 2021
916beff
try
tgravescs Jul 30, 2021
d17a4a5
remove sql no value
tgravescs Jul 30, 2021
500a835
revert
tgravescs Jul 30, 2021
0089fcc
more convert
tgravescs Jul 30, 2021
f0c56ad
more
tgravescs Jul 30, 2021
8c57fc8
fix
tgravescs Jul 30, 2021
cb0b31a
longs
tgravescs Jul 30, 2021
66b0b7d
more
tgravescs Jul 30, 2021
a7510d5
more tests
tgravescs Jul 30, 2021
33da7d3
fix
tgravescs Jul 30, 2021
4b38e0f
fix copy and paste
tgravescs Jul 30, 2021
c493b10
test
tgravescs Jul 30, 2021
a3b6ea9
fix
tgravescs Jul 30, 2021
138157e
debug
tgravescs Jul 30, 2021
6431d29
debug
tgravescs Jul 30, 2021
30da075
debug
tgravescs Jul 30, 2021
0a0299c
remove debug
tgravescs Jul 30, 2021
284f35c
debug
tgravescs Jul 30, 2021
8fa1b4b
debug
tgravescs Jul 30, 2021
4fa5848
convert Analysis
tgravescs Aug 2, 2021
65259ad
fix longs
tgravescs Aug 2, 2021
30b1959
debug
tgravescs Aug 2, 2021
cfc2b1d
print exception
tgravescs Aug 2, 2021
a3f7f49
fix header
tgravescs Aug 2, 2021
19a07f6
fix null
tgravescs Aug 2, 2021
787867c
count duplicate
tgravescs Aug 2, 2021
6f98656
count all attempts
tgravescs Aug 2, 2021
ce4a605
handle only job end
tgravescs Aug 2, 2021
4ad9c84
debug
tgravescs Aug 2, 2021
0cb8ea7
fix
tgravescs Aug 2, 2021
7bfc577
update
tgravescs Aug 2, 2021
1d6eb88
fix
tgravescs Aug 2, 2021
291f77e
update
tgravescs Aug 2, 2021
3056210
debug
tgravescs Aug 2, 2021
cbb09e0
add missing stages
tgravescs Aug 2, 2021
192c24e
enable analysis tests
tgravescs Aug 2, 2021
00174df
test
tgravescs Aug 2, 2021
7415ae7
fix
tgravescs Aug 2, 2021
5d6f412
update expected
tgravescs Aug 2, 2021
d382f73
convert healthcheck
tgravescs Aug 2, 2021
fb91504
fix typo
tgravescs Aug 2, 2021
201a0b7
truncate
tgravescs Aug 2, 2021
7f007c9
update result
tgravescs Aug 2, 2021
d5bd781
Merge branch 'profileRedesign' of github.com:tgravescs/spark-rapids i…
tgravescs Aug 2, 2021
8e5e2cc
node desc truncate
tgravescs Aug 2, 2021
3d0b78e
add compare suite
tgravescs Aug 2, 2021
a286975
update
tgravescs Aug 2, 2021
4297c33
more tests
tgravescs Aug 2, 2021
e90e83e
fix
tgravescs Aug 2, 2021
1a494b2
fix
tgravescs Aug 2, 2021
b04c8b2
compare count stages
tgravescs Aug 2, 2021
a4eaac3
fix test
tgravescs Aug 2, 2021
44d346a
close
tgravescs Aug 2, 2021
b66fc3b
update log file index
tgravescs Aug 2, 2021
4854f29
track driver info
tgravescs Aug 2, 2021
24a67d9
count driver memory
tgravescs Aug 2, 2021
6b16fd0
remove debug
tgravescs Aug 2, 2021
99e8b32
handle driver for local mode
tgravescs Aug 2, 2021
3385355
add test
tgravescs Aug 2, 2021
cb85be7
fix test
tgravescs Aug 2, 2021
9a27a0e
add more tests
tgravescs Aug 2, 2021
e2e188e
fix test
tgravescs Aug 2, 2021
6272653
more test
tgravescs Aug 2, 2021
f81bc4b
fix incompat
tgravescs Aug 2, 2021
efbcdbc
fix + log
tgravescs Aug 2, 2021
a4bc9ac
debug
tgravescs Aug 2, 2021
7d0bd98
show more
tgravescs Aug 2, 2021
80eaaaf
more
tgravescs Aug 2, 2021
d21dbdc
more
tgravescs Aug 2, 2021
16b3303
test
tgravescs Aug 3, 2021
0389356
test
tgravescs Aug 3, 2021
cb01c2f
fix
tgravescs Aug 3, 2021
5d3d4c6
fix
tgravescs Aug 3, 2021
8caa601
long
tgravescs Aug 3, 2021
7bb1b2f
test
tgravescs Aug 3, 2021
8b2e9d1
fix
tgravescs Aug 3, 2021
8d8592a
fix task max
tgravescs Aug 3, 2021
d9b54a8
debug driver
tgravescs Aug 3, 2021
8e5b36f
driver accums
tgravescs Aug 3, 2021
4bf0bf2
cleanup
tgravescs Aug 3, 2021
38a3d07
debug
tgravescs Aug 3, 2021
c1d329a
debug:
tgravescs Aug 3, 2021
004a1c1
fix
tgravescs Aug 3, 2021
dc2b8e7
fix
tgravescs Aug 3, 2021
3e2c23d
cleanup
tgravescs Aug 3, 2021
794f60e
cleanup
tgravescs Aug 3, 2021
f7b6d91
cleanup
tgravescs Aug 3, 2021
170d623
fixies
tgravescs Aug 3, 2021
f51e094
Update docs
tgravescs Aug 4, 2021
c2c0d61
more doc updates
tgravescs Aug 4, 2021
3882e42
compare mode error
tgravescs Aug 4, 2021
04a27ad
more docs
tgravescs Aug 4, 2021
a4e85e7
Merge remote-tracking branch 'origin/branch-21.10' into profileRedesign
tgravescs Aug 4, 2021
fbf8c3e
fix merge issue
tgravescs Aug 4, 2021
4088eaf
cleanup
tgravescs Aug 4, 2021
f406eb2
fix test
tgravescs Aug 4, 2021
2c9ed5d
Merge remote-tracking branch 'origin/branch-21.10' into profileRedesign
tgravescs Aug 5, 2021
862d886
update test
tgravescs Aug 6, 2021
816319c
change log file name
tgravescs Aug 6, 2021
6e981e5
test fixes
tgravescs Aug 6, 2021
928d51e
Merge branch 'profileRedesign' of github.com:tgravescs/spark-rapids i…
tgravescs Aug 6, 2021
d49204e
cleanup
tgravescs Aug 6, 2021
f379ae8
remove comment
tgravescs Aug 6, 2021
b52bd4a
fix schema
tgravescs Aug 6, 2021
c1c2803
make appIndex 1
tgravescs Aug 6, 2021
c67853c
comment app index
tgravescs Aug 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
331 changes: 185 additions & 146 deletions docs/additional-functionality/qualification-profiling-tools.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ object ShimLoader extends Logging {
def getSparkVersion: String = {
// hack for databricks, try to find something more reliable?
if (SPARK_BUILD_USER.equals("Databricks")) {
SPARK_VERSION + "-databricks"
SPARK_VERSION + "-databricks"
} else {
SPARK_VERSION
}
Expand Down

Large diffs are not rendered by default.

Loading