Add specification for computing the dot product (linalg: vecdot)

[kgryte]

This PR

• adds the specification for computing the dot product.

• currently deviates from current behavior across array libraries. Follows more closely to NumPy's inner.

• comparisons for API signatures can be found here.

• attempts to eliminate overloaded behavior found in the NumPy API. Namely, when

  • x1 and x2 are 1D: NumPy computes the dot product.
  • x1 and x2 are mixed 1D and ND: NumPy operates on the stack, computing the dot product over the last axis.
  • x1 and x2 are 2D: NumPy performs matrix multiplication.
  • x1 and x2 are ND: NumPy performs a tensor contraction using the second-to-last and the last axes, respectively.

  This proposal seeks to limit computation to strictly the dot product and limit to a single axis for >1D.

• adds axis keyword to specify dimension over which to compute the dot product. (Note: this is similar to MATLAB's dot).

• renames the API as vecdot to avoid user confusion regarding expected behavior when transitioning from, e.g., np.dot to using vecdot.

Background

Overview of the various NumPy APIs for computing the dot product:

	dot	inner	vdot	multi_dot	tensordot	matmul	einsum
1D	dot product	dot product	dot product	dot product	dot product1	dot product	dot product2
2D	matrix multiplication	dot product	(flattens)	dot product	dot product1	matrix multiplication	dot product2
ND	tensor contraction	dot product (cols vs cols)	(flattens)	--	dot product1	matrix multiplication	dot product 2
stacks	N	Y	N	N	Y	Y	Y
axes	N	N	N	N	Y	N	Y
complex conjugation	N	N	Y	N	--	N	--

¹ can specify arbitrary axes for tensor contraction as a tuple of sequences.
² can specify an arbitrary axis for computing a sum.

Notes

• NumPy et al perform different quantities (dot product vs matrix multiplication) depending on the input argument shapes. This stems, at least in part, from legacy concerns dating prior to the introduction of matmul.

• Pytorch limits computation to 1D vectors. From their docs (latest master):

  Unlike NumPy’s dot, torch.dot intentionally only supports computing the dot product of two 1D tensors with the same number of elements.

• For reference, Julia limits computing the dot product over vectors. Extension to multi-dimensional arrays is supported so long as the dot product operation is defined on the elements. BLAS routines for computing the dot product are limited to strided vectors, with complex conjugation for complex dtypes.

Other Considerations

• An alternative specification could limit computing the dot product to one of the last two dimensions and treat >2D arrays as stacks.
• Once complex dtypes are added to the standard, I'd advocate for complex conjugation, rather than having a separate API (e.g., vdot).
• If an axis keyword argument is supported, then, similar to statistical reduction APIs (mean, max, min), we could consider supporting a keepdims keyword argument to retain the reduced dimension.

[shoyer]

@kgryte could you add batching vs tensor-product for leading axes to this table? or maybe that's what you mean by "Stacks"?

[asmeurer]

I think another thing that is missing from the table is the behavior when the inputs have mixed dimensionality, e.g., 1D x 2D.

[kgryte]

@shoyer Correct. In the table, stacks is equivalent to batching.

[kgryte]

Following @leofang's suggestion, I've renamed the API to inner_dot to avoid user confusion with np.dot behavior.

[shoyer]

I wonder if vecdot might be a good name (short for "vector dot product"), mirroring matmul ("matrix multiply").

inner_dot sounds a little redundant.

[rgommers]

I wonder if vecdot might be a good name (short for "vector dot product"), mirroring matmul ("matrix multiply").

I like this. There's also a precedent for the "vec" part, scipy.sparse.linalg uses matvec, matmat, etc. as LinearOperator methods.

[kgryte]

I've renamed the proposed API to vecdot as discussed here and in the previous (3/25/2021) meeting.

[kgryte]

This PR has been open for some time without further comment and resolves issues discussed above. Will merge and we can submit follow-up PRs should any further issues/concerns arise.