Don't wrap round

[dpsanders]

This seems like a gotcha (indeed it caught us out when manipulating images):

julia> x = N0f8(0.5)
0.502N0f8

julia> x + x
0.0N0f8

[timholy]

#41. FWIW this happens in most image-processing frameworks, including Matlab (which uses saturating arithmetic) and Python (which uses overflow arithmetic like us). That said, I am tempted to blaze a new trail to make this better.

There was an epic discussion in #143. We seem close to re-addressing this shortly, with the following proposals:

• No supported arithmetic, convert immediately to float
• Overflow (wrap-around) arithmetic like it is now
• Saturating arithmetic like Matlab
• Checked arithmetic (you'll get an error from that operation)
• Widening arithmetic like in WIP/RFC/Scary: allow signed Normed numbers #143 (aside from supporting signed arithmetic, the key element of that proposal was a 2-fold widening, but no further widening from that point forward---the types are marked as "fully widened" and henceforth use overflow arithmetic)

Each proposal has its strengths and weaknesses. I'd be curious to hear your thoughts. I know @kimikage is planning to solicit input on discourse, but since you posted it...

[dpsanders]

Thanks. I guess we were expecting it to saturate, but I see that it's a much harder (or insoluble) problem than it initially seems.

One option would be to implement +(:saturate, x , y) or +(Saturate(), x, y) etc., but of course this is pretty cumbersome. (And I guess have plain + throw and error and inform you about these options.)

[timholy]

It's not impossible, it's just not obvious it's a good idea. Stefan has posted on the evils of saturating artihmetic. (E.g., x + x - x != x for many x.)

[timholy]

Also much of it is already implemented (by @kimikage), see

FixedPointNumbers.jl/src/FixedPointNumbers.jl

Lines 218 to 287 in 595f7a7

	# saturating arithmetic
	saturating_neg(x::X) where {X <: FixedPoint} = X(~min(x.i - true, x.i), 0)
	saturating_neg(x::X) where {X <: FixedPoint{<:Unsigned}} = zero(X)
	saturating_abs(x::X) where {X <: FixedPoint} =
	X(ifelse(signbit(abs(x.i)), typemax(x.i), abs(x.i)), 0)
	saturating_add(x::X, y::X) where {X <: FixedPoint} =
	X(x.i + ifelse(x.i < 0, max(y.i, typemin(x.i) - x.i), min(y.i, typemax(x.i) - x.i)), 0)
	saturating_add(x::X, y::X) where {X <: FixedPoint{<:Unsigned}} = X(x.i + min(~x.i, y.i), 0)
	saturating_sub(x::X, y::X) where {X <: FixedPoint} =
	X(x.i - ifelse(x.i < 0, min(y.i, x.i - typemin(x.i)), max(y.i, x.i - typemax(x.i))), 0)
	saturating_sub(x::X, y::X) where {X <: FixedPoint{<:Unsigned}} = X(x.i - min(x.i, y.i), 0)
	saturating_mul(x::X, y::X) where {X <: FixedPoint} = clamp(float(x) * float(y), X)
	saturating_fdiv(x::X, y::X) where {X <: FixedPoint} =
	clamp(floattype(X)(x.i) / floattype(X)(y.i), X)
	# checked arithmetic
	checked_neg(x::X) where {X <: FixedPoint} = checked_sub(zero(X), x)
	function checked_abs(x::X) where {X <: FixedPoint}
	abs(x.i) >= 0 || throw_overflowerror_abs(x)
	X(abs(x.i), 0)
	end
	function checked_add(x::X, y::X) where {X <: FixedPoint}
	r, f = Base.Checked.add_with_overflow(x.i, y.i)
	z = X(r, 0) # store first
	f && throw_overflowerror(:+, x, y)
	z
	end
	function checked_sub(x::X, y::X) where {X <: FixedPoint}
	r, f = Base.Checked.sub_with_overflow(x.i, y.i)
	z = X(r, 0) # store first
	f && throw_overflowerror(:-, x, y)
	z
	end
	function checked_mul(x::X, y::X) where {X <: FixedPoint}
	z = float(x) * float(y)
	typemin(X) - eps(X)/2 <= z < typemax(X) + eps(X)/2 || throw_overflowerror(:*, x, y)
	z % X
	end
	function checked_fdiv(x::X, y::X) where {T, X <: FixedPoint{T}}
	y === zero(X) && throw(DivideError())
	z = floattype(X)(x.i) / floattype(X)(y.i)
	if T <: Unsigned
	z < typemax(X) + eps(X)/2 || throw_overflowerror(:/, x, y)
	else
	typemin(X) - eps(X)/2 <= z < typemax(X) + eps(X)/2 || throw_overflowerror(:/, x, y)
	end
	z % X
	end
	# default arithmetic
	const DEFAULT_ARITHMETIC = :wrapping
	for (op, name) in ((:-, :neg), (:abs, :abs))
	f = Symbol(DEFAULT_ARITHMETIC, :_, name)
	@eval begin
	$op(x::X) where {X <: FixedPoint} = $f(x)
	end
	end
	for (op, name) in ((:+, :add), (:-, :sub), (:*, :mul))
	f = Symbol(DEFAULT_ARITHMETIC, :_, name)
	@eval begin
	$op(x::X, y::X) where {X <: FixedPoint} = $f(x, y)
	end
	end
	/(x::X, y::X) where {X <: FixedPoint} = checked_fdiv(x, y) # force checked arithmetic

[kimikage]

We should eventually decide on a final course of action on the overflow issue, but we haven't made any significant progress in the last six months at least. 😭

Apart from the overflow issues, I was unhappy with the implementation of arithmetic (e.g. ::Normed * ::Normed). So I decided to support checked, wrapping and saturating arithmetic as part of the improvement of arithmetic implementations.

After I complete the implementation of checked arithmetic (cf. #41 (comment)), I'm going to open a thread in discourse to decide which is the default arithmetic (on a per-method basis) in v0.9.0.

That decision in v0.9.0 will only be the first step and should not be the final goal. However, I believe it is useful to have real working codes in order to determine the goal. Benchmarking is extremely important because overflow solutions are closely related to speed and memory performance.

[kimikage]

Incidentally, I "personally" disagree with making saturating arithmetic the default. However, I think there is a rationale for using saturation arithmetic for Gray{N0f8} and RGB{N0f8}, not N0f8.

I'm also preparing to experiment with the three types of arithmetic for ColorVectorSpace. https://github.com/kimikage/CheckedColorArithmetic.jl

[kimikage]

@timholy, I would appreciate it if you could reply to JuliaMath/CheckedArithmetic.jl#6 when you have a time.

[timholy]

As you think about the options, let me make a pitch again for the type of solution in #143, since you haven't mentioned it as something you're actively considering and because I do think it's one of the more attractive options.

The naive motivation is the following: if UInt8 + UInt8 might overflow, why not just return a UInt64? Well, that's a lot bigger and eats up a lot more memory, a serious negative if you're processing large images. OK, so how about a UInt16? The problem there is that when you then go to add a UInt16 to anything else, do you return a UInt32? That quickly gets out of hand---it makes arithmetic essentially non-inferrable.

So the core idea in #143 is to split into two worlds: "representation types" and "computed types." "Representation types" are what might be created, say, by reading an image from a file---they are used for your typical "starting image." Computed types are produced as the result of any math operation (well, any operation that tries to stay in the world of Fixed* numbers). Continuing the UInt analogy, instead of UInt8 + UInt8 -> UInt16, we instead have UInt8 + UInt8 -> WUInt16, where W marks it as a computed type. The key proposal is that representation types widen to computed types, but that computed types are "terminal" and used wrap-around (overflow) arithmetic. What that means is that if you're playing around at the command line and add two images and then send the result to a viewer, you should get exactly what you expect; if we make the computed type signed, then the same is true of the difference between two images. That latter is especially important: in image processing, we look at the difference image all the time, and making that both easy and accurate seems like an important goal.

The biggest strength of the proposal is that naive operations "just work," which is true of only one of the other proposals, having all math operations immediately convert to float. Unlike the latter, it also keeps resulting values relatively compact. The weaknesses of the proposal are that it effectively doubles the number of types we have to support, and it could lull people into false complacency: my proposed strategy is great for interactive exploration where you compute the sum or difference of a pair of images, but if you sum 1000 UInt8-represented images you need to allocate the result using a safer type.

Why don't other languages/frameworks do it this way? Because they don't have the types---they are limited to machine numbers, and so they can't distinguish a terminally-widened WUInt16 from a representational UInt16. But we don't have such limitations.

On balance I think having things "just work" for interactive exploration, while not compromising performance and accuracy, is probably the most important goal. Hence my two favorite options (in the abstract, without any practical experience) are "all math immediately converts to float" and "limited doubling of size" (aka, #143). I'm less fond of the saturating, "simple" wrap-around, or checked arithmetic options, which are the main ones you have been promoting. Hence I wanted to remind you of these options.

[johnnychen94]

Unsure how much effort is needed to achieve this, from the user's side I like the idea of splitting it apart to representation types and computation types. If I get the idea correctly, the representation types are mostly exclusive to IO package developers and are transparent to normal users.

Haven't taken a time to experiment with the new checked-math functions developed by @kimikage, though.

[timholy]

Yes, the idea is that we want to be able to represent any "input" image without conversion; reinterpret is fine, I'm talking about the bit-level description here. This is particularly important if your image is a 1TB file that you import via memory-mapping: you must be able to represent the meaning of that file using its native bit layout. But once you're working with it in memory you're free to choose different types for the result of math operations.

[kimikage]

The fundamental problem is that what is a "representation type" and what is a "computed type" depend on the use case. For me, N0f8 is sometimes a representation type and sometimes a computed type.

If you think it's an exception, please consider the case of ::N0f8 * ::N0f8. This can cause a loss of accuracy, but not an overflow. If you convert N0f8 to a wider type, you lose the optimization opportunity described above.

In fact, there must be both cases where you want to widen and cases where you don't want to widen. The two requirements are completely antithetical and there is no universal solution. The "neutral" approach is to let the user specify the way. We can add widening_* methods, but users can still convert the types manually.

[timholy]

Sure, but in cases where widening isn't necessary then you don't have to do it. Maybe instead of calling them "representation" and "computed" we could call them "plain" and "marked." "plain" types can widen, if the operation requires it to avoid overflow; "marked" types cannot.

[kimikage]

You seem to be addressing only a single operation. I'm illustrating a case like (a::N0f8 - b::N0f8) * c::N0f8. How do I clear the "marked"?

Edit:
Of course, this is about regressions, so I'm assuming that a - b is working correctly (even if it's wrapped-around). (1 - b) * c is actually used in interpolation etc.

Edit2:

Edit: you can always use % N0f8 for interpolation, or better just write that as complement(b).

Yes, I know the complement. However, it doesn't seem to me that calling additional functions intentionally is much different than leaving it up to the user to write float(x) or N24f8(x). In some cases, just widening 8-bit types into 16-bit types doesn't "just work".

[timholy]

You don't. The answer might be negative. But in any case, once marked, always marked.

Edit: you can always use % N0f8 for interpolation, or better just write that as complement(b).