Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using of memory pool in hash table decreases the scalabiity #36

Open
wangli1426 opened this issue May 26, 2014 · 1 comment
Open

Using of memory pool in hash table decreases the scalabiity #36

wangli1426 opened this issue May 26, 2014 · 1 comment

Comments

@wangli1426
Copy link
Contributor

We use boost::pool in hash table to resolve the memory leak caused by fragment.
As boost::pool is not thread-safe, we have to use a spin-lock on the memory pool. This becomes the performance bottleneck when multiple thread are allocating hash buckets.

To solve this performance problem, I will first consider to use multiple pools in single hash table, such that one pool will only be used by a single thread. I expect this enhancement could hopefully improve the scalability.

@wangli1426
Copy link
Contributor Author

The following is the comparison between the scalability of using one pool and multiple pool.

Number of thread All threads sharing one pool Each thread has a pool
1 1 1
2 1.5 1.6
3 1.9 2
4 2 2.5

The test is made by generating data and inserting into hash table. The configures are:

    const unsigned nbuckets=1024*1024;
    const unsigned tuple_size=32;
    const unsigned bucketsize=1024-8;
    const unsigned long data_size_in_MB=256;

The inserting code is like the following:

    while(arg.buffer->getBlock(*fetched_block)){
        void* tuple;
        BlockStreamBase::BlockStreamTraverseIterator* it=fetched_block->createIterator();
        while(tuple=it->nextTuple()){
            unsigned bn=arg.schema->columns[0].operate->getPartitionValue(tuple,arg.hash);
            void* new_tuple_in_hashtable=(*arg.hash_table)->atomicAllocate(bn,arg.tid);
            arg.schema->copyTuple(tuple,new_tuple_in_hashtable);
        }
    }

@wangli1426 wangli1426 removed their assignment Nov 22, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant