Skip to content

Commit

Permalink
Merge pull request #383 from enthought/tst/upcast-issue-win32-x86
Browse files Browse the repository at this point in the history
MAINT: Upcast to int_ instead of int64 to avoid bincount issue.
  • Loading branch information
jvkersch authored Oct 31, 2017
2 parents c57ec89 + fb9437a commit 7747ad4
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions chaco/tests/test_image_plot.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,12 @@ def calculate_rms(image_result, expected_image):
"""
# calculate the per-pixel errors, then compute the root mean square error
num_values = np.prod(expected_image.shape)
# Cast to int64 to reduce likelihood of over-/under-flow.
abs_diff_image = abs(np.int64(expected_image) - np.int64(image_result))
# Images may be e.g. 8-bit unsigned integer; upcast to default integer size
# (32 or 64 bit) to reduce likelihood of over-/under-flow.
abs_diff_image = abs(np.int_(expected_image) - np.int_(image_result))

histogram = np.bincount(abs_diff_image.ravel(), minlength=256)
sum_of_squares = np.sum(histogram * np.arange(len(histogram))**2)
sum_of_squares = np.sum(np.int64(histogram) * np.arange(len(histogram))**2)
rms = np.sqrt(float(sum_of_squares) / num_values)
return rms

Expand Down

0 comments on commit 7747ad4

Please sign in to comment.