Dumping and Loading Nested Maps/Sets

[atom-moyer]

Hey Greg,

I would like to make an unordered unique multimap phmap::flat_hash_map<Key, phmap::flat_hash_set<Value>> where Key and Value are trivially copyable. As one would expect when I try to dump the outer map using phmap::BinaryOutputArchive ar_out( filename.c_str() ); dict.dump( ar_out ); I get an error that the inner set is not trivially copyable.

What would you suggest for serializing?

Also, side question: Is this how you would make a multimap?

Thanks,
Adam

[greg7mdp]

Hi Adam,

Maybe you could dump it as:

ar_out.dump(dict.size());
for (const auto& [k, v] : dict) {
    ar_out.dump(k);
    v.dump(ar_out);
}

and do the opposite when reading (read size, call reserve, and then read n [k, v] pairs and populate the dict).

Let me think about the multimap question.

greg

[atom-moyer]

That worked great!

Here is the code that I ended up implementing for people in the future:

    void dump ( const std::string & filename ) {
        phmap::BinaryOutputArchive ar_out ( filename.c_str() );

        ar_out.dump( dict.size() );
        for ( auto & [k, v] : dict ) {
            ar_out.dump( k );
            v.dump( ar_out );
        }
    }


    void load ( const std::string & filename ) {
        phmap::BinaryInputArchive ar_in ( filename.c_str() );

        size_t size; Key k; phmap::flat_hash_set<Value> v;

        ar_in.load( &size );
        dict.reserve( size );

        for ( size_t idx = 0; idx < size; idx++ ) {
            ar_in.load( &k );
            v.load( ar_in );

            dict.insert_or_assign( k, v );
        }
    }

Notice that I had to remove the const in the dump for loop because v.dump( ar_out ) must not be implemented as a const member function. You might want to add that.

Adam

[greg7mdp]

Nice. I'll do the const change and add this as an example.
I would do the following change though to avoid copying the newly loaded hash_set:

    void load ( const std::string & filename ) {
        phmap::BinaryInputArchive ar_in ( filename.c_str() );

        size_t size;

        ar_in.load( &size );
        dict.reserve( size );

        for ( size_t idx = 0; idx < size; idx++ ) {
            Key k;
            phmap::flat_hash_set<Value> v;

            ar_in.load( &k );
            v.load( ar_in );

            dict.insert_or_assign( std::move(k), std::move(v) );
        }
    }

[greg7mdp]

I added the example dump_nested.cc. Thanks! - greg

[mr-eyes]

Hi @greg7mdp

It seems that the example in https://github.com/greg7mdp/parallel-hashmap/blob/master/examples/dump_nested.cc is not working anymore to dump TriviallyCopyable types.

[greg7mdp]

Thanks for the note, @mr-eyes , I'll have a look this weekend.

[greg7mdp]

@mr-eyes sorry for that, I fixed the issue and even made the example nicer :-)

[mr-eyes]

@greg7mdp Thank you so much!
I have been using the parallel-hashmap for almost all the projects I am working on; Much appreciation 🙌

[greg7mdp]

@mr-eyes - thank you! It is always great to hear from happy users.

[greg7mdp]

@mr-eyes if you are using c++20, you may want to switch to the gtl library, it has all the stuff from phmap and more.

[DJuego]

Hi, I've been following this library for a while now, and although I haven't used it, I'm glad it exists, thanks for the great work! However I was unaware of the existence of the gtl library instead.

Based on your comment, could you confirm that it is an updated superset of the phpmap library, with the only limitation of requiring compatibility with c++20?

Thank you very much!

DJuego

[greg7mdp]

it is an updated superset of the phpmap library, with the only limitation of requiring compatibility with c++20?

yes, exactly!