Suggestion to handle duplication key properties in object-level

[Omar-Medina]

Dear nLohmann-Json Team,

Thank you once again for providing the amazing library.

I'm currently using the SAX interface to detect if any key-property within an object-level is duplicated.
To clarify, the term "object level" does not include any existing sub-object levels. Here are some examples to illustrate:
Example of duplication:
{"a":"s","a":"c"} // Key-property "a" is duplicated.
Example without duplication:
{"a":"s","z":{"a":"c"}} // Key-property "a" is not duplicated.

I have noticed that some JSON schema validator libraries do not report a failure when a key within an object level is duplicated,
which conforms to the standard. However, in my case, I want to catch such failures in JSON configurations.

See please below used code snip below.

In addtion. It was neccesary to modify (quick and dirty) the nlohmann library, so that the following mesages are provided, when a duplication has veen found.

{"elm":[{"str":{"name":"sd","name":"xz","name":"xz"}}]}
===
duplicate key 'name' found/defined near at line=0, column=27
duplicate key 'name' found/defined near at line=0, column=39

The modification consist of extension of the nlohmann::json::json_sax_t with a new function.
void position(nlohmann::detail::position_t position).

this allows to get the currently position of the Json-file during the parsing.

nLohman version : 3.11.3
JSON_DIAGNOSTIC_POSITIONS is not required.

patch for nLohmann/json.hpp:

--- MODIFIED\nLohmann\json.hpp	2025-02-12 15:32:20.677902700 +0100
+++ ORIGINAL\single_include\nlohmann\json.hpp	2023-11-28 22:36:31.000000000 +0100
@@ -6772,12 +6772,6 @@
                              const std::string& last_token,
                              const detail::exception& ex) = 0;
 
-    /*!
-    @brief to provide the currently position
-    @param[in] position    the position in the input where the error occurs
-    */
-    virtual void position(nlohmann::detail::position_t position) = 0;
-
     json_sax() = default;
     json_sax(const json_sax&) = default;
     json_sax(json_sax&&) noexcept = default;
@@ -6936,10 +6930,6 @@
         return false;
     }
 
-    void position(nlohmann::detail::position_t position)
-    {
-    }
-
     constexpr bool is_errored() const
     {
         return errored;
@@ -7190,10 +7180,6 @@
         return false;
     }
 
-    void position(nlohmann::detail::position_t position)
-    {
-    }
-
     constexpr bool is_errored() const
     {
         return errored;
@@ -12371,7 +12357,6 @@
         {
             if (!skip_to_state_evaluation)
             {
-                sax->position(m_lexer.get_position()); // publish/update the position
                 // invariant: get_token() was called before each iteration
                 switch (last_token)
                 {

Demo Class

  using nsBSW::btUint32 = std::uint32_t;
  using nsBSW::btBoolean = bool;
  using nsBSW::btVectorStrings = std::vector<std::string>;
  using nsBSW::btMapVectorStringsToUint32 = std::map<std::vector<std::string>, std::uint32_t>;
  using nsBSW::btString = std::string;

  class JsonDuplicateKeyValidator  : public nlohmann::json::json_sax_t
  {
    /***************************************
     *        BEGIN variables BEGIN        *
     ***************************************/

    /***************************************
     *          public variables           *
     ***************************************/
    public:

    /***************************************
     *          private variables          *
     ***************************************/
    private:
    nsBSW::btUint32 mLevel;           /*!< To manage the level */
    nsBSW::btBoolean mErrorFound;      /*!< flag to manage a detected error during the parsing */
    nlohmann::detail::position_t pos; /*!< To manage the position inside the JSON file */
    nsBSW::btVectorStrings events;    /*!< To manage the finding */
    nsBSW::btMapVectorStringsToUint32 mKeysInObjectLevel; /*!< to manage the keys that belong to json-object in certatin level */

    /***************************************
     *         protected variables         *
     ***************************************/
    protected:

    /***************************************
     *          END variables END          *
     ***************************************/

    /*=====================================*/

    /***************************************
     *         BEGIN methods BEGIN         *
     ***************************************/

    /***************************************
     *            public methods           *
     ***************************************/
    public:
    /*! constructor */
    JsonDuplicateKeyValidator(void) :
      mLevel{0U},
      mErrorFound{false}
    {
      pos.chars_read_current_line = 0;
      pos.chars_read_total = 0;
      pos.lines_read = 0;
    } /* constructor void */

    /*! destructor */
    virtual ~JsonDuplicateKeyValidator(void)
    {
      mKeysInObjectLevel.clear();
    } /* destructor */

    /***************************
     * Group of SAX event handler functions
     ***************************/
    /* \brief SAX event handler functions.
     *   \detail These functions are part of the SAX (Simple API for XML) interface for handling different JSON events.
     *           They are called by the JSON parser for various JSON elements such as null, boolean, integer, unsigned,
     *           float, string, array start, array end, and binary data.
     *  \retval true Always returns true, indicating the event was successfully handled. */
    bool null() override { return true; }
    bool boolean(bool val) override { return true; }
    bool number_integer(nlohmann::json::number_integer_t val) override { return true; }
    bool number_unsigned(nlohmann::json::number_unsigned_t val) override { return true; }
    bool number_float(nlohmann::json::number_float_t val, const string_t& s) override { return true; }
    bool string(nlohmann::json::string_t& val) override { return true; }
    bool start_array(std::size_t elements) override { return true; }
    bool end_array() override { return true; }
    bool binary(nlohmann::json::binary_t& val) override { return true; }

    /***************************
     * bool start_object
     ***************************/
    /*! \brief Handles the start of a JSON object.
     *  \detail Increments the level counter and clears any existing keys at the current level.
     *  \param[in] elements The number of elements in the object.
     *  \retval true Always returns true.
     */
    bool start_object(std::size_t elements) override
    {
      mLevel++;
      /* clear level */
      if (mKeysInObjectLevel.contains(mLevel) == true)
      {
        mKeysInObjectLevel[mLevel].clear();
        mKeysInObjectLevel.erase(mLevel);
      }
      return true;
    } /* start_object */

    /***************************
     * bool end_object
     ***************************/
    /*! \brief Handles the end of a JSON object.
     *  \detail Decrements the level counter.
     *  \retval true Always returns true.
     */
    bool end_object() override
    {
      mLevel--;
      return true;
    } /* end_object */

    /***************************
     * bool key
     ***************************/
    /*! \brief Handles JSON keys and checks for duplicates.
     *  \detail Collects all duplicate keys inside an object and logs an event if found.
     *  \param[in] val The JSON key as a string.
     *  \retval true Always returns true.
     */
    bool key(nlohmann::json::string_t& val) override
    {
      bool retVal = true;
      /* collect all duplicate keys inside an object */
      if (mKeysInObjectLevel.find(mLevel) != mKeysInObjectLevel.end())
      {
        const auto& vec = mKeysInObjectLevel[mLevel];
        auto it = std::find(vec.begin(), vec.end(), val);
        /* Check if the key is already present in the object level */
        if (it != vec.end())
        {
          /* append detection . it will be provide later , when the IsValid fuction is called */
          events.push_back(nsBSW::formaT("duplicate key '%1%' found/defined near at line=%2%, column=%3%",
                           val,
                           pos.lines_read,
                           pos.chars_read_current_line));
        }
        else
        {
          /* append key to the level */
          mKeysInObjectLevel[mLevel].push_back(val);
        }
      }
      else
      {
        /* append key to the level */
        mKeysInObjectLevel[mLevel].push_back(val);
      }
      return retVal;
    } /* key */

    /***************************
     * bool parse_error
     ***************************/
    /*! \brief Handles parse errors by logging the error details and setting the error flag.
     *  \detail This function is called when a parse error occurs. It logs the error details
     *          including line, column, position, last token, and the exception message.
     *  \param[in] position The position in the input where the error occurred.
     *  \param[in] last_token The last token processed before the error.
     *  \param[in] ex The exception object containing error details.
     *  \retval false Always returns false to indicate a parse error occurred.
     */
    bool parse_error(std::size_t position, const std::string& last_token, const nlohmann::json::exception& ex) override
    {
      bool ret = false;
      events.push_back(nsBSW::formaT("parse_error near at line=%1%, column=%2%, position=%3%, last_token=%4%,\nexception=%5%",
                       pos.lines_read,
                       pos.chars_read_current_line,
                       position,
                       last_token,
                       ex.what()));
      mErrorFound = true;
      return ret;
    } /* parse_error */

    /***************************
     * void position
     ***************************/
    /*! \brief Updates the position object with the new position.
     *  \detail This function sets the internal position object to the provided position.
     *  \param[in] position The new position to be set.
     */
    void position(nlohmann::detail::position_t position)
    {
      pos = position;
    } /* position */

    /***************************
     * bool IsValid
     ***************************/
    /*! \brief Collects all findings and checks for duplicate keys in the JSON file.
     *  \detail This function checks if the JSON file contains duplicate keys and logs the events.
     *  \param[out] log The log string containing recorded events.
     *  \param[out] errorFound Boolean indicating if any error was found.
     *  \retval false JSON file contains at least a duplicate key and it is not valid.
     *  \retval true  JSON file does not contain any duplicate key.
     */
    bool IsValid(nsBSW::btString &log, nsBSW::btBoolean &errorFound)
    {
      nsBSW::btBoolean retV = (events.size() == 0 ? true : false);
      errorFound = mErrorFound;
      /* output the recorded events */
      log += nsBSW::FuncLib::ConcatenateStrings(events,nsBSW::ConstExpr::strEOL.data());

      return retV;
    } /* IsValid */

    /***************************************
     *           private methods           *
     ***************************************/
    private:

    /***************************************
     *          protected methods          *
     ***************************************/
    protected:

    /***************************************
     *           END methods END           *
     ***************************************/
  }; /* class JsonDuplicateKeyValidator */

How to use:

    nsBSW::btBoolean ret = false;
    nsBSW::btBoolean retx = false;
    nsBSW::btBoolean errorFound = false;
    nsBSW::btString msg;

        JsonDuplicateKeyValidator sec;
        /* sax_parse returns true if the parsing of the file is successful */
        /* sax_parse returns false if the parsing of the file fails */
        retx = nlohmann::json::sax_parse(f_json, &sec, nlohmann::json::input_format_t::json, true, false);
        /* sec.IsValid returns true if the parsing of the file is successful */
        /* sec.IsValid returns false if the parsing of the file fails */
        ret = sec.IsValid(msg, errorFound);

Your library has been incredibly helpful, and I appreciate the hard work you put into it.

Omar

[nlohmann]

Adding a function to the SAX interface would be a breaking change to existing implementations. I am further not convinced that adding this functionality is of great benefit, because, as you mentioned, JSON does not forbid duplicate keys.

[Omar-Medina]

Dear Mr. Lohmann,

Thank you for your prompt reply and for maintaining such a valuable library.

I understand the concerns regarding making modifications that could break existing implementations. Rest assured, I am not requesting any changes to the nLohmann library. I previously suggested an enhancement and it was rightly declined. I have accepted this decision.

While the JSON standard allows for duplicate keys at the object level, there are instances where it is necessary to check for such duplications. The nlohmann::json::json_sax_t interface seems ideal for this purpose, as it allows for duplication checks before the actual parsing takes place, serving as a preliminary step.

For example, your library currently takes the last found key.
In the following sample:
{"elm":[{"str":{"name":"Freedom","name":"Ugly"}}]}
Such a configuration could generate problems, particularly in sensitive contexts, such as when deciding the name of a "child."

If a configuration JSON file is used in a security or safety context and contains duplicate keys, it is highly recommended to check for such issues. While the standard permits such cases, it falls upon developers to ensure these issues are addressed.

The software should serve us, and in this case, my modification of your library is used privately within my own context.
In addtion , for large configuration files, knowing the location of any duplications can be very helpful. And this is my approach. Nothingselse.

If you find my post unsuitable for your concept, please feel free to remove it, or I can remove it myself.

Thank you once again for your time and for creating such an outstanding library.

Kind regards,
Omar

[nlohmann]

All fine.