Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace strtold with non locale dependent version #337

Closed
wants to merge 14 commits into from
Closed

replace strtold with non locale dependent version #337

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
60bba02
Merge branch 'release/2.0.6'
nlohmann Oct 15, 2016
22f3f88
new strtod/ld/f function
whackashoe Oct 16, 2016
8877ad7
remove whitespace loop in strtox
whackashoe Oct 16, 2016
201e1f3
remove parse check for strtox
whackashoe Oct 16, 2016
b7a978f
remove endptr parameter from strtox
whackashoe Oct 16, 2016
7f5b228
remove unused errno setting from strtox
whackashoe Oct 16, 2016
973ee97
remove unnecessary str pointer and rename arg to be clearer
whackashoe Oct 16, 2016
7dcc8a5
continuing work on our own strtold - reduced to 3 failing assertions
whackashoe Oct 21, 2016
7cc60cc
nl_isdigit can be made constexpr
whackashoe Oct 21, 2016
dec60cb
remove unneeded branch
whackashoe Oct 21, 2016
babb65d
different neg style for consistency
whackashoe Oct 21, 2016
308082a
make negative_exp and plus_or_minus const
whackashoe Oct 23, 2016
66a4c06
use not over !
whackashoe Oct 23, 2016
bc53b79
add additional comments
whackashoe Oct 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 122 additions & 43 deletions src/json.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8800,64 +8800,143 @@ class basic_json
return result;
}

/*!
@brief parse floating point number
// non locale aware isdigit
// Microsoft in 1252 codepage and others may classify additional single-byte characters as digits using std::isdigit
constexpr bool nl_isdigit(const char c) const
{
return c >= '0' and c <= '9';
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
/*!
@brief parse string to floating point number

@param[in] type the @ref number_float_t in use
This function is a partial reimplementation of the strtold in order to meet needs of JSON number

@param[in,out] endptr recieves a pointer to the first character after
the number
@param[in] str the string we will parse

@return the floating point number
*/
long double str_to_float_t(long double* /* type */, char** endptr) const
long double strtojnum(const char *str) const
{
return std::strtold(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
}
long double result = 0;
char cp = *str;
int exp = 0; // exponent
{
const bool negative_sign = cp == '-';

/*!
@brief parse floating point number
if (cp == '-' or cp == '+')
{
++str;
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
// read in fractional part of number, until an 'e' is reached.
// count digits after decimal point.
while (nl_isdigit(cp = *str))
{
result = result * 10 + (cp - '0');
++str;
}

@param[in] type the @ref number_float_t in use
if (cp == '.')
{
while (nl_isdigit(cp = *++str))
{
result = result * 10 + (cp - '0');
--exp;
}
}

@param[in,out] endptr recieves a pointer to the first character after
the number
// if negative number, reverse sign
if (negative_sign)
{
result *= -1;
}
}

@return the floating point number
*/
double str_to_float_t(double* /* type */, char** endptr) const
{
return std::strtod(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
}
// read in explicit exponent and calculate real exponent.
if (*str == 'e' or *str == 'E')
{
cp = *++str;
bool negative_exp = cp == '-'; // read in exponent sign (+/-)

/*!
@brief parse floating point number
bool plus_or_minus = false;
if (cp == '-' or cp == '+')
{
cp = *++str;
plus_or_minus = true;
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
int count = 0; // exponent calculation
if (! nl_isdigit(cp))
{
if (plus_or_minus)
{
*--str;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is there a '*' here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a look! @nlohmann want to incorp these into your feature branch?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes no sense. Please have a look at https://github.com/nlohmann/json/blob/feature/strtold/src/json.hpp.re2c#L8139 where I started an overworked version of this PR.

}

@param[in] type the @ref number_float_t in use
*--str;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above.

goto skip_loop;
}

@param[in,out] endptr recieves a pointer to the first character after
the number
while (nl_isdigit(cp))
{
constexpr int imax = std::numeric_limits<int>::max();

@return the floating point number
*/
float str_to_float_t(float* /* type */, char** endptr) const
{
return std::strtof(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
if ((imax - std::abs(exp) - (cp - '0')) / 10 > count)
{
count *= 10;
count += cp - '0';
}
else
{
count = imax - exp;
break;
}

cp = *++str;
}
skip_loop:
exp += negative_exp ? -count : count;
}

// adjust number by powers of ten specified by format and exponent.
constexpr std::array<long double, 9> powerof10 = {
{1.e1L, 1.e2L, 1.e4L, 1.e8L, 1.e16L, 1.e32L, 1.e64L, 1.e128L, 1.e256L}
};

if (exp > std::numeric_limits<long double>::max_exponent10)
{
constexpr long double inf = std::numeric_limits<long double>::infinity();
result = (result < 0) ? -inf : inf;
}
else if (exp < std::numeric_limits<long double>::min_exponent10)
{
result = 0.0L;
}
else if (exp < 0)
{
exp *= -1;

for (std::size_t count = 0; exp; ++count, exp >>= 1)
{
if (exp & 1)
{
result /= powerof10[count];
}
}
}
else
{
for (std::size_t count = 0; exp; ++count, exp >>= 1)
{
if (exp & 1)
{
result *= powerof10[count];
}
}
}

return result;
}

/*!
Expand Down Expand Up @@ -8958,8 +9037,8 @@ class basic_json
}
else
{
// parse with strtod
result.m_value.number_float = str_to_float_t(static_cast<number_float_t*>(nullptr), NULL);
// convert string by json number format to floating point
result.m_value.number_float = strtojnum(reinterpret_cast<typename string_t::const_pointer>(m_start));
}

// save the type
Expand Down
165 changes: 122 additions & 43 deletions src/json.hpp.re2c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8097,64 +8097,143 @@ class basic_json
return result;
}

/*!
@brief parse floating point number
// non locale aware isdigit
// Microsoft in 1252 codepage and others may classify additional single-byte characters as digits using std::isdigit
constexpr bool nl_isdigit(const char c) const
{
return c >= '0' and c <= '9';
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
/*!
@brief parse string to floating point number

@param[in] type the @ref number_float_t in use
This function is a partial reimplementation of the strtold in order to meet needs of JSON number

@param[in,out] endptr recieves a pointer to the first character after
the number
@param[in] str the string we will parse

@return the floating point number
*/
long double str_to_float_t(long double* /* type */, char** endptr) const
long double strtojnum(const char *str) const
{
return std::strtold(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
}
long double result = 0;
char cp = *str;
int exp = 0; // exponent
{
const bool negative_sign = cp == '-';

/*!
@brief parse floating point number
if (cp == '-' or cp == '+')
{
++str;
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
// read in fractional part of number, until an 'e' is reached.
// count digits after decimal point.
while (nl_isdigit(cp = *str))
{
result = result * 10 + (cp - '0');
++str;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't str be incremented in the while condition just like in the loop below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we may not want to increment if nl_isdigit fails so it must be in body.

}

@param[in] type the @ref number_float_t in use
if (cp == '.')
{
while (nl_isdigit(cp = *++str))
{
result = result * 10 + (cp - '0');
--exp;
}
}

@param[in,out] endptr recieves a pointer to the first character after
the number
// if negative number, reverse sign
if (negative_sign)
{
result *= -1;
}
}

@return the floating point number
*/
double str_to_float_t(double* /* type */, char** endptr) const
{
return std::strtod(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
}
// read in explicit exponent and calculate real exponent.
if (*str == 'e' or *str == 'E')
{
cp = *++str;
bool negative_exp = cp == '-'; // read in exponent sign (+/-)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

negative_exp could be const bool


/*!
@brief parse floating point number
bool plus_or_minus = false;
if (cp == '-' or cp == '+')
{
cp = *++str;
plus_or_minus = true;
}

This function (and its overloads) serves to select the most approprate
standard floating point number parsing function based on the type
supplied via the first parameter. Set this to @a
static_cast<number_float_t*>(nullptr).
int count = 0; // exponent calculation
if (! nl_isdigit(cp))
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use not rather than !

{
if (plus_or_minus)
{
*--str;
}

@param[in] type the @ref number_float_t in use
*--str;
goto skip_loop;
}

@param[in,out] endptr recieves a pointer to the first character after
the number
while (nl_isdigit(cp))
{
constexpr int imax = std::numeric_limits<int>::max();

@return the floating point number
*/
float str_to_float_t(float* /* type */, char** endptr) const
{
return std::strtof(reinterpret_cast<typename string_t::const_pointer>(m_start), endptr);
if ((imax - std::abs(exp) - (cp - '0')) / 10 > count)
{
count *= 10;
count += cp - '0';
}
else
{
count = imax - exp;
break;
}

cp = *++str;
}
skip_loop:
exp += negative_exp ? -count : count;
}

// adjust number by powers of ten specified by format and exponent.
constexpr std::array<long double, 9> powerof10 = {
{1.e1L, 1.e2L, 1.e4L, 1.e8L, 1.e16L, 1.e32L, 1.e64L, 1.e128L, 1.e256L}
};

if (exp > std::numeric_limits<long double>::max_exponent10)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting here, there a now comments in the code. Please briefly describe each conditional and loop.

{
constexpr long double inf = std::numeric_limits<long double>::infinity();
result = (result < 0) ? -inf : inf;
}
else if (exp < std::numeric_limits<long double>::min_exponent10)
{
result = 0.0L;
}
else if (exp < 0)
{
exp *= -1;

for (std::size_t count = 0; exp; ++count, exp >>= 1)
{
if (exp & 1)
{
result /= powerof10[count];
}
}
}
else
{
for (std::size_t count = 0; exp; ++count, exp >>= 1)
{
if (exp & 1)
{
result *= powerof10[count];
}
}
}

return result;
}

/*!
Expand Down Expand Up @@ -8255,8 +8334,8 @@ class basic_json
}
else
{
// parse with strtod
result.m_value.number_float = str_to_float_t(static_cast<number_float_t*>(nullptr), NULL);
// convert string by json number format to floating point
result.m_value.number_float = strtojnum(reinterpret_cast<typename string_t::const_pointer>(m_start));
}

// save the type
Expand Down