Skip to content

[SPARK-33906][WEBUI] Fix the bug of UI Executor page stuck due to undefined peakMemoryMetrics #30920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
baohe-zhang wants to merge 3 commits into from
Closed

[SPARK-33906][WEBUI] Fix the bug of UI Executor page stuck due to undefined peakMemoryMetrics #30920

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c3ab7a1
Check if the peakMemoryMetrics is undefined before accessing it
Dec 24, 2020
e7154fd
Update display text to N/A when peakMemoryMetrics is undefined
Dec 25, 2020
2f7b9be
Update display text to 0.0 B / 0.0 B
Dec 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 56 additions & 20 deletions core/src/main/resources/org/apache/spark/ui/static/executorspage.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,38 +414,74 @@ $(document).ready(function () {
},
{
data: function (row, type) {
if (type !== 'display')
return row.peakMemoryMetrics.JVMHeapMemory;
else
return (formatBytes(row.peakMemoryMetrics.JVMHeapMemory, type) + ' / ' +
formatBytes(row.peakMemoryMetrics.JVMOffHeapMemory, type));
var peakMemoryMetrics = row.peakMemoryMetrics;
if (typeof peakMemoryMetrics !== 'undefined') {
if (type !== 'display')
return peakMemoryMetrics.JVMHeapMemory;
else
return (formatBytes(peakMemoryMetrics.JVMHeapMemory, type) + ' / ' +
formatBytes(peakMemoryMetrics.JVMOffHeapMemory, type));
} else {
if (type !== 'display') {
return 0;
Copy link
Member

@sarutak sarutak Dec 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to return "" rather than 0 like done here.

Could you update the similar logic in the previous PR too?

Copy link
Member

@gengliangwang gengliangwang Dec 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually in https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/stagepage.js#L501, it also returns 0.
If the column is invisible, it seems fine to return anything.

Copy link
Member

@sarutak sarutak Dec 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that returns 0 but I wonder it's impossible for us to tell that "the metrics is not updated yet" from "that the metrics is actually 0".

} else {
return '0.0 B / 0.0 B';
}
}
}
},
{
data: function (row, type) {
if (type !== 'display')
return row.peakMemoryMetrics.OnHeapExecutionMemory;
else
return (formatBytes(row.peakMemoryMetrics.OnHeapExecutionMemory, type) + ' / ' +
formatBytes(row.peakMemoryMetrics.OffHeapExecutionMemory, type));
var peakMemoryMetrics = row.peakMemoryMetrics;
if (typeof peakMemoryMetrics !== 'undefined') {
if (type !== 'display')
return peakMemoryMetrics.OnHeapExecutionMemory;
else
return (formatBytes(peakMemoryMetrics.OnHeapExecutionMemory, type) + ' / ' +
formatBytes(peakMemoryMetrics.OffHeapExecutionMemory, type));
} else {
if (type !== 'display') {
return 0;
} else {
return '0.0 B / 0.0 B';
}
}
}
},
{
data: function (row, type) {
if (type !== 'display')
return row.peakMemoryMetrics.OnHeapStorageMemory;
else
return (formatBytes(row.peakMemoryMetrics.OnHeapStorageMemory, type) + ' / ' +
formatBytes(row.peakMemoryMetrics.OffHeapStorageMemory, type));
var peakMemoryMetrics = row.peakMemoryMetrics;
if (typeof peakMemoryMetrics !== 'undefined') {
if (type !== 'display')
return peakMemoryMetrics.OnHeapStorageMemory;
else
return (formatBytes(peakMemoryMetrics.OnHeapStorageMemory, type) + ' / ' +
formatBytes(peakMemoryMetrics.OffHeapStorageMemory, type));
} else {
if (type !== 'display') {
return 0;
} else {
return '0.0 B / 0.0 B';
}
}
}
},
{
data: function (row, type) {
if (type !== 'display')
return row.peakMemoryMetrics.DirectPoolMemory;
else
return (formatBytes(row.peakMemoryMetrics.DirectPoolMemory, type) + ' / ' +
formatBytes(row.peakMemoryMetrics.MappedPoolMemory, type));
var peakMemoryMetrics = row.peakMemoryMetrics;
if (typeof peakMemoryMetrics !== 'undefined') {
if (type !== 'display')
return peakMemoryMetrics.DirectPoolMemory;
else
return (formatBytes(peakMemoryMetrics.DirectPoolMemory, type) + ' / ' +
formatBytes(peakMemoryMetrics.MappedPoolMemory, type));
} else {
if (type !== 'display') {
return 0;
} else {
return '0.0 B / 0.0 B';
}
}
}
},
{data: 'diskUsed', render: formatBytes},
Expand Down