Problem with number format #889
-
Dear all, Do you know why this is happening? Is there a possibility to change the expression before I give it to the "trim"-method? My code looks for example as follows:
Thank you for your help. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
Hi Michael, There are two problems here. This first is that you're using a 32 bit version of Python and the number 36028797018963968 cannot be represented by a 32 bit signed integer. The second problem is Nutils not telling you this, but truncating the number to the least significant 32 bits silently, which is 0 in this particular case. The second problem is fixed in master (#890). The first problem is fixed by appending a dot, which makes it a floating point number, reducing the number to something that can be represented by a 32 bit signed integer, or switching to a 64 bit build of Python (which is the default on all platforms but Windows). Best, Joost |
Beta Was this translation helpful? Give feedback.
Hi Michael,
There are two problems here. This first is that you're using a 32 bit version of Python and the number 36028797018963968 cannot be represented by a 32 bit signed integer. The second problem is Nutils not telling you this, but truncating the number to the least significant 32 bits silently, which is 0 in this particular case. The second problem is fixed in master (#890). The first problem is fixed by appending a dot, which makes it a floating point number, reducing the number to something that can be represented by a 32 bit signed integer, or switching to a 64 bit build of Python (which is the default on all platforms but Windows).
Best, Joost