Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Tensor zero point sparsity #1304

Closed
wants to merge 10 commits into from
Closed

WIP: Tensor zero point sparsity #1304

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b12d72f
Update onednn.js
lutzroeder Jun 1, 2024
99c0381
Merge branch 'main' of https://github.com/kylesayrs/netron into main
kylesayrs Jun 13, 2024
1b2cf28
Merge branch 'main' of https://github.com/lutzroeder/netron into main
kylesayrs Jun 13, 2024
bf425ca
Merge branch 'main' of https://github.com/lutzroeder/netron into main
kylesayrs Jun 15, 2024
8b46d9a
Merge branch 'main' of https://github.com/lutzroeder/netron into main
kylesayrs Jun 21, 2024
932b101
WIP: zero point
kylesayrs Jun 21, 2024
2a5eb19
implement the rest of the op types
kylesayrs Jun 21, 2024
effbf09
Update TorchScript test files (#1067)
lutzroeder Jul 8, 2024
dda6828
Merge branch 'main' of https://github.com/lutzroeder/netron into tens…
kylesayrs Jul 8, 2024
fa41bde
fix lints
kylesayrs Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions source/base.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -585,6 +585,7 @@ base.Tensor = class {
this.type = tensor.type;
this.layout = tensor.type.layout;
this.stride = tensor.stride;
this._zero = tensor.zero || null;
base.Tensor.dataTypes = base.Tensor.dataTypeSizes || new Map([
['boolean', 1],
['qint8', 1], ['qint16', 2], ['qint32', 4],
Expand Down Expand Up @@ -613,6 +614,10 @@ base.Tensor = class {
return this._data;
}

get zero() {
return this._zero;
}

get metrics() {
return this._tensor.metrics;
}
Expand Down
42 changes: 41 additions & 1 deletion source/onnx.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,8 +482,9 @@

onnx.Tensor = class {

constructor(context, tensor, category) {
constructor(context, tensor, category, zero) {
this._category = category || null;
this._zero = zero || null;
if (tensor.indices && tensor.values) {
this._name = tensor.values.name || '';
this._type = context.createTensorType(tensor.values.data_type, tensor.dims, 'sparse');
Expand Down Expand Up @@ -620,6 +621,14 @@
}
}

setZero(zero) {
this._zero = zero;
}

get zero() {
return this._zero;
}

get name() {
return this._name;
}
Expand Down Expand Up @@ -1301,6 +1310,37 @@
node = new onnx.Node(this, node, inputs, outputs);
this._nodes.push(node);

const setZero = (inputs, weightName, zeroName) => {
const weight = inputs.find(input => input.name == weightName);

Check failure on line 1314 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected parentheses around arrow function argument

Check failure on line 1314 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected '===' and instead saw '=='

Check failure on line 1314 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected parentheses around arrow function argument

Check failure on line 1314 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected '===' and instead saw '=='
const zero = inputs.find(input => input.name == zeroName);

Check failure on line 1315 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected parentheses around arrow function argument

Check failure on line 1315 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected '===' and instead saw '=='

Check failure on line 1315 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected parentheses around arrow function argument

Check failure on line 1315 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected '===' and instead saw '=='
if (
weight && weight.value.length === 1 && weight.value[0].initializer &&
zero && zero.value.length === 1 && zero.value[0].initializer
) {
weight.value[0].initializer.setZero(zero.value[0].initializer);
}
}

Check failure on line 1322 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Missing semicolon

Check failure on line 1322 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Missing semicolon
switch (op_type) {

Check failure on line 1323 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected a default case

Check failure on line 1323 in source/onnx.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected a default case
Copy link
Author

@kylesayrs kylesayrs Jun 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order for a tensor to know its zero, it must get the sibling initializer to it on a quantized node

The onnx format doesn't allow for easy lookup from initializer to node or from initializer to initializer, so it's best to setZero after onnx.Tensor has been constructed.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An initializer is a parameter to a node. A node should not modify an initializer.

Copy link
Author

@kylesayrs kylesayrs Jun 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code happens within onnx.Context.Graph, so it is the onnx context graph which is modifying initializers. Setting the zero point from onnx.Context.Graph seems like the best option?

Attempting to force the zero point to be set during initializer construction seems like a bad idea. Initializers must know the context of their parent node in order to know their zero point.

  1. Looking up the parent node of each initializer is O(N*M) if N is the number of nodes and M is the number of tensors, which seems like unacceptable runtime.
  2. Attempting to create a parent node lookup table requires that table to be filled outside of onnx.Tensor (ie, in some place like onnx.Context.Graph), which is similar to the current implementation. It also requires all nodes to be ingested into the lookup table before tensors are constructed, which also seems like an unacceptable solution.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initializer has no zero point. Only the context of the operation or result of the operation does.

x = a + z0
y = a + z1

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lutzroeder The zero point is needed by the initializer/tensor to correctly calculate the sparsity metric. Without the zero point, the sparsity metric for quantized weights is meaningless. It would be nice if the sparsity metric for quantized tensors reflected the true number of zeros that would be skipped by a sparsity-aware inference engine.

Copy link
Author

@kylesayrs kylesayrs Jun 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lutzroeder Sorry, let me see if I can summarize what you're suggesting

  1. You're saying that calculating sparsity and flops is equivalent to built in runtime execution or partial execution?

While it's true that these calculations are somewhat similar to runtime execution, I don't consider them to be the same since calculating sparsity and flops can be done much faster and can be done lazily/piece-wise behind expanders as previously suggested.

I agree that work will need to be done to support different formats. Metrics calculations may go unsupported for some formats. I'm not sure if I understand the alternatives though.

  1. You're suggesting an alternative could be to have metrics handled by the runtime itself and results written to a "metrics" file.

When you refer to runtime itself, I assume you mean ORT or other 3rd party runtimes? Would the intention be to have the user run the model outside of Netron? The problem would obviously be a non-standardization/absence of metrics output files. Even if there was a sparsity-aware runtime for each format which outputted metrics files, these metrics files would be in different formats and we would have the same problem as 1 where we now have to do work to support all runtime output formats.

  1. You're suggesting a third alternative of attaching to a runtime.

This part I do not understand. This seems similar to the job of a profiler specific to the runtime? How would Netron know the internals of how a runtime performed on a network? What would it look like for the user to "attach a runtime"?

I truly apologize if I am missing something obvious

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. These would be the harder questions that need better answers. It isn't clear what the right path is.

sparsity and flops can be done much faster

Are these a few exceptions or the beginning of discovering that many useful metrics require deeper runtime integration or crossing over into profiling.

The problem would obviously be a non-standardization/absence of metrics output files.

Which metrics output file formats and tools do exist. How are these problems solved today?

Metrics calculations may go unsupported for some formats. I'm not sure if I understand the alternatives though.

UX approach isn't clear. How can users intuitively understand what's happening. The sparsity is not the sparsity of this tensor, it's the result of an operation. Some formats tweaking source tensors and others not is confusing at best. Users wouldn't be able to tell without reading the source.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lutzroeder I have implemented sparsity and ops calculations in the past for ONNX here: neuralmagic/sparsezoo#173. The only metric which was somewhat runtime specific was the flops calculations for conv.

If we choose not to calculate skipped padding for conv ops, then we don't need to iterate through tensor values at all and can calculate ops from the raw number of zeros. This makes all common onnx operations one-pass linear time to calculate metrics for.

In terms of performance, I suspect that calculating most individual tensor metrics will be negligible, especially if they're behind expanders. Calculating metrics for all tensors/nodes so that they can be searched will likely take some time no matter which approach is taken. I believe users would be willing to endure some wait time for these metrics if Netron

  • Provides a loading bar in large tensor/ metrics search cases
  • Lazily load metrics behind expanders
  • Refuses to calculate tensors with size above a certain threshold
  • Computes metrics in a separate thread to avoid UI slowdown and enable the option to cancel the search

Giving users the option to use these metrics in exchange for some wait time seems like a valuable feature.

I think 2 and 3 are viable, but having users use an external engine or attach a runtime will likely be slower than if Netron calculates the metrics directly, not to mention more inconvenient.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just because Netron supports 1 doesn't mean it can't support 2 and/or 3 down the line. Indeed I think Netron should support 2 in some capacity. All of these options require format-specific implementations, at least with 1, more of the logic is shared.

Copy link
Owner

@lutzroeder lutzroeder Jul 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, trying to summarize:

This pull request should not merge as modifying the source tensors is running the computation and storing the result in the source tensor which will lead to conflicts. It would not be transparent to users that the source tensor was modified and confusing as this happens in some cases but not others. A better way to model this might be sparsity of the operation (e.g. can Node have a sparsity).

This leads to next level of questions how the computation is performed and tensor access is needed to perform these computations. Not sure if these issues need to be solved immediately towards making metrics a minimum viable feature or if this can be more exploratory. There seem to be more basic changes like lazy expansion for tensor properties and basic metrics like min, max, stddev. Also, now that some effort went into tensor properties, would tensor visualizations for example be more valuable and broadly applicable than handling zero point in special operators in a single framework.

On the explorations side, is it possible to generalize metrics that need tensor data and computation to avoid implementations for each framework or is that too far fetched? If the later, will this lead to more of an execution layer that needs to be implemented for each framework (similar question for shape inference) and should this code be always loaded. Starting down this route will create the expectation that this gets implemented and maintained more broadly. e.g. how to guarantee these "partial execution implementations" are more than prototypes adding maintenance scope. Other routes worth exploring would be driving metric implementations via external tools or the runtime and serializing to a metrics file (which might be also desirable for performance metrics) or attaching to Python runtimes to avoid duplicate execution implementations and enabling live execution.

case 'ConvInteger':
setZero(inputs, 'w', 'w_zero_point');
break;

case 'MatMulInteger':
setZero(inputs, 'A', 'a_zero_point');
setZero(inputs, 'B', 'b_zero_point');
break;

case 'QLinearConv':
setZero(inputs, 'w', 'w_zero_point');
// bias zero point is always 0
break;

case 'QLinearMatMul':
setZero(inputs, 'a', 'a_zero_point');
setZero(inputs, 'b', 'b_zero_point');
break;
}

// const path = (node.name || '').split('/');
// path.pop();
// this.group(path.join('/')).get('').push(node);
Expand Down
33 changes: 17 additions & 16 deletions source/view.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3235,10 +3235,11 @@

view.TensorSidebar = class extends view.ObjectSidebar {

constructor(context, value, tensor) {
constructor(context, value, tensor, metrics) {
super(context);
this._value = value;
this._tensor = tensor || new base.Tensor(value.initializer);
this._metrics = metrics;
}

render() {
Expand Down Expand Up @@ -3288,22 +3289,19 @@
}
}
}
/*
// Metrics
if (value.initializer) {
if (!tensor.empty) {
if (!this._metrics) {
this._metrics = new metrics.Tensor(this._tensor);
}
this.addHeader('Metrics');
const metrics = this._metrics.metrics;
for (const metric of metrics) {
const value = metric.type === 'percentage' ? `${(metric.value * 100).toFixed(1)}%` : metric.value;
this.addProperty(metric.name, [value]);
}
///* TODO
if (!tensor.empty) {
if (!this._metrics) {
this._metrics = new metrics.Tensor(this._tensor);
}
this.addHeader('Metrics');
const metricsArguments = this._metrics.metrics;
for (const metric of metricsArguments) {
const value = metric.type === 'percentage' ? `${(metric.value * 100).toFixed(1)}%` : metric.value;
this.addProperty(metric.name, [value]);
}
}
*/
//*/
}
};

Expand Down Expand Up @@ -4862,6 +4860,9 @@
this._metrics = Array.from(tensor.metrics || []);
const keys = new Set(this._metrics.map((metrics) => metrics.name));
if (!keys.has('sparsity')) {
// TODO: support channel-wise zeros

Check failure on line 4863 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Unexpected 'todo' comment: 'TODO: support channel-wise zeros'

Check failure on line 4863 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Unexpected 'todo' comment: 'TODO: support channel-wise zeros'
const zero = tensor.zero != null ? tensor.zero.values[0] : null;

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Unexpected negated condition

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Use '===' to compare with null

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (macos-latest) 

Expected '!==' and instead saw '!='

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Unexpected negated condition

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Use '===' to compare with null

Check failure on line 4864 in source/view.js

View workflow job for this annotation

GitHub Actions / Build (ubuntu-latest) 

Expected '!==' and instead saw '!='

let zeros = 0;
let parameters = 0;
const stack = [data];
Expand All @@ -4872,7 +4873,7 @@
stack.push(element);
}
} else {
zeros += data === 0 || data === 0n || data === '';
zeros += zero != null ? data === zero : data === 0 || data === 0n || data === '';
parameters += 1;
}
}
Expand Down
Loading