Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make std::filesystem::path conversion to/from UTF-8 encoded string explicit #4631

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

risa2000
Copy link

This PR makes the default conversions of std::filesystem::path from/to JSON UTF-8 encoded string explicit.
This is a follow up on #4271.

Original implementation relied on the platform locale to be UTF-8 and used std::string as std::filesystem::path conversion source/target. This did not work on Windows (where the implicit locale is using "multibyte" encoding) and may potentially trigger a similar issue on platforms which do not have the locale set to UTF-8.

Proposed change uses explicit conversion functions from std library to UTF-8 encoded string. Unfortunately, the UTF-8 handling of std library changed from C++17 to C++20 so the code uses different codepaths for corresponding C++ standards.

This code has been inspired by:
https://github.com/alf-p-steinbach/C---how-to---make-non-English-text-work-in-Windows/blob/main/microlibs/cppm/stdlib_workarounds/fs_path.hpp#L19

@coveralls
Copy link

coveralls commented Jan 26, 2025

Coverage Status

coverage: 99.186%. remained the same
when pulling 008c50b on risa2000:patch-fspath-conversions
into e90c860 on nlohmann:develop.

@risa2000 risa2000 force-pushed the patch-fspath-conversions branch from 61a26dd to a395539 Compare January 26, 2025 19:41
@github-actions github-actions bot added L and removed M labels Jan 26, 2025
…plicit.

Signed-off-by: Richard Musil <risa2000x@gmail.com>
@risa2000 risa2000 force-pushed the patch-fspath-conversions branch from a395539 to 008c50b Compare January 27, 2025 06:16
@risa2000
Copy link
Author

I am bit confused about how make amalgamate works. It seems to arbitrary change formatting on some files, even when I do only a small change (e.g. unit-conversions.cpp) and sometimes it looks like those changes are different each time I run it?

I could not run it on Windows (seems like Makefile is not compatible with nmake) so I run it on macOS.

@nlohmann
Copy link
Owner

I am bit confused about how make amalgamate works. It seems to arbitrary change formatting on some files, even when I do only a small change (e.g. unit-conversions.cpp) and sometimes it looks like those changes are different each time I run it?

asytle and the amalgamation is deterministic. There should not be different results after subsequent calls.

I could not run it on Windows (seems like Makefile is not compatible with nmake) so I run it on macOS.

It's based on Python. The calls the Makefile makes are:

python3 -mvenv tools/astyle/venv
tools/astyle/venv/bin/pip3 install --quiet --upgrade pip
tools/astyle/venv/bin/pip3 install --quiet -r tools/astyle/requirements.txt
tools/amalgamate/amalgamate.py -c tools/amalgamate/config_json_fwd.json -s . --verbose=yes
tools/astyle/venv/bin/astyle --project=tools/astyle/.astylerc include/nlohmann/adl_serializer.hpp include/nlohmann/byte_container_with_subtype.hpp include/nlohmann/detail/abi_macros.hpp include/nlohmann/detail/conversions/from_json.hpp include/nlohmann/detail/conversions/to_chars.hpp include/nlohmann/detail/conversions/to_json.hpp include/nlohmann/detail/exceptions.hpp include/nlohmann/detail/hash.hpp include/nlohmann/detail/input/binary_reader.hpp include/nlohmann/detail/input/input_adapters.hpp include/nlohmann/detail/input/json_sax.hpp include/nlohmann/detail/input/lexer.hpp include/nlohmann/detail/input/parser.hpp include/nlohmann/detail/input/position_t.hpp include/nlohmann/detail/iterators/internal_iterator.hpp include/nlohmann/detail/iterators/iter_impl.hpp include/nlohmann/detail/iterators/iteration_proxy.hpp include/nlohmann/detail/iterators/iterator_traits.hpp include/nlohmann/detail/iterators/json_reverse_iterator.hpp include/nlohmann/detail/iterators/primitive_iterator.hpp include/nlohmann/detail/json_custom_base_class.hpp include/nlohmann/detail/json_pointer.hpp include/nlohmann/detail/json_ref.hpp include/nlohmann/detail/macro_scope.hpp include/nlohmann/detail/macro_unscope.hpp include/nlohmann/detail/meta/call_std/begin.hpp include/nlohmann/detail/meta/call_std/end.hpp include/nlohmann/detail/meta/cpp_future.hpp include/nlohmann/detail/meta/detected.hpp include/nlohmann/detail/meta/identity_tag.hpp include/nlohmann/detail/meta/is_sax.hpp include/nlohmann/detail/meta/std_fs.hpp include/nlohmann/detail/meta/type_traits.hpp include/nlohmann/detail/meta/void_t.hpp include/nlohmann/detail/output/binary_writer.hpp include/nlohmann/detail/output/output_adapters.hpp include/nlohmann/detail/output/serializer.hpp include/nlohmann/detail/string_concat.hpp include/nlohmann/detail/string_escape.hpp include/nlohmann/detail/string_utils.hpp include/nlohmann/detail/value_t.hpp include/nlohmann/json.hpp include/nlohmann/json_fwd.hpp include/nlohmann/ordered_map.hpp include/nlohmann/thirdparty/hedley/hedley.hpp include/nlohmann/thirdparty/hedley/hedley_undef.hpp tests/abi/config/config.hpp tests/abi/config/custom.cpp tests/abi/config/default.cpp tests/abi/config/noversion.cpp tests/abi/diag/diag.cpp tests/abi/diag/diag.hpp tests/abi/diag/diag_off.cpp tests/abi/diag/diag_on.cpp tests/abi/inline_ns/use_current.cpp tests/abi/inline_ns/use_v3_10_5.cpp tests/abi/main.cpp tests/benchmarks/src/benchmarks.cpp tests/cmake_add_subdirectory/project/main.cpp tests/cmake_fetch_content/project/main.cpp tests/cmake_fetch_content2/project/main.cpp tests/cmake_import/project/main.cpp tests/cmake_import_minver/project/main.cpp tests/cmake_target_include_directories/project/Bar.cpp tests/cmake_target_include_directories/project/Bar.hpp tests/cmake_target_include_directories/project/Foo.cpp tests/cmake_target_include_directories/project/Foo.hpp tests/cmake_target_include_directories/project/main.cpp tests/cuda_example/json_cuda.cu tests/src/fuzzer-driver_afl.cpp tests/src/fuzzer-parse_bjdata.cpp tests/src/fuzzer-parse_bson.cpp tests/src/fuzzer-parse_cbor.cpp tests/src/fuzzer-parse_json.cpp tests/src/fuzzer-parse_msgpack.cpp tests/src/fuzzer-parse_ubjson.cpp tests/src/make_test_data_available.hpp tests/src/test_utils.hpp tests/src/unit-32bit.cpp tests/src/unit-algorithms.cpp tests/src/unit-allocator.cpp tests/src/unit-alt-string.cpp tests/src/unit-assert_macro.cpp tests/src/unit-binary_formats.cpp tests/src/unit-bjdata.cpp tests/src/unit-bson.cpp tests/src/unit-byte_container_with_subtype.cpp tests/src/unit-capacity.cpp tests/src/unit-cbor.cpp tests/src/unit-class_const_iterator.cpp tests/src/unit-class_iterator.cpp tests/src/unit-class_lexer.cpp tests/src/unit-class_parser.cpp tests/src/unit-class_parser_diagnostic_positions.cpp tests/src/unit-comparison.cpp tests/src/unit-concepts.cpp tests/src/unit-constructor1.cpp tests/src/unit-constructor2.cpp tests/src/unit-convenience.cpp tests/src/unit-conversions.cpp tests/src/unit-custom-base-class.cpp tests/src/unit-deserialization.cpp tests/src/unit-diagnostic-positions-only.cpp tests/src/unit-diagnostic-positions.cpp tests/src/unit-diagnostics.cpp tests/src/unit-disabled_exceptions.cpp tests/src/unit-element_access1.cpp tests/src/unit-element_access2.cpp tests/src/unit-hash.cpp tests/src/unit-inspection.cpp tests/src/unit-items.cpp tests/src/unit-iterators1.cpp tests/src/unit-iterators2.cpp tests/src/unit-iterators3.cpp tests/src/unit-json_patch.cpp tests/src/unit-json_pointer.cpp tests/src/unit-large_json.cpp tests/src/unit-locale-cpp.cpp tests/src/unit-merge_patch.cpp tests/src/unit-meta.cpp tests/src/unit-modifiers.cpp tests/src/unit-msgpack.cpp tests/src/unit-no-mem-leak-on-adl-serialize.cpp tests/src/unit-noexcept.cpp tests/src/unit-ordered_json.cpp tests/src/unit-ordered_map.cpp tests/src/unit-pointer_access.cpp tests/src/unit-readme.cpp tests/src/unit-reference_access.cpp tests/src/unit-regression1.cpp tests/src/unit-regression2.cpp tests/src/unit-serialization.cpp tests/src/unit-testsuites.cpp tests/src/unit-to_chars.cpp tests/src/unit-type_traits.cpp tests/src/unit-ubjson.cpp tests/src/unit-udl.cpp tests/src/unit-udt.cpp tests/src/unit-udt_macro.cpp tests/src/unit-unicode1.cpp tests/src/unit-unicode2.cpp tests/src/unit-unicode3.cpp tests/src/unit-unicode4.cpp tests/src/unit-unicode5.cpp tests/src/unit-user_defined_input.cpp tests/src/unit-windows_h.cpp tests/src/unit-wstring.cpp tests/src/unit.cpp single_include/nlohmann/json.hpp single_include/nlohmann/json_fwd.hpp docs/mkdocs/docs/examples/*.cpp

@risa2000
Copy link
Author

After I pushed my PR, I noticed here when checking the diff that in my PR the amalgamation changed unit-conversions.cpp format in a way it obscured the change. So I run make amalgamate again and it seemed to change the formatting again, but still making changes on many more places. So I am giving up - I do not know how to keep the original format while running make amalgamate. Probably formatting changed meanwhile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants