Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-27805 The chunk created by mslab may cause memory fragement and… #5193

Merged
merged 2 commits into from
Jul 27, 2023
Merged

HBASE-27805 The chunk created by mslab may cause memory fragement and… #5193

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
838c7d9
HBASE-27805 The chunk created by mslab may cause memory fragement and…
bsglz Apr 20, 2023
87e0692
fix review issues
bsglz Jul 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions hbase-common/src/main/resources/hbase-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -755,8 +755,12 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.hregion.memstore.mslab.chunksize</name>
<value>2097152</value>
<description>The maximum byte size of a chunk in the MemStoreLAB. Unit: bytes</description>
<value>2096128</value>
<description>
The maximum byte size of a chunk in the MemStoreLAB.
Use 2047k as the default because 2m may cause memory fragmentation in some cases.
Unit: bytes
</description>
</property>
<property>
<name>hbase.regionserver.offheap.global.memstore.size</name>
Expand Down
2 changes: 1 addition & 1 deletion hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public interface MemStoreLAB {
String MSLAB_CLASS_NAME = "hbase.regionserver.mslab.class";

String CHUNK_SIZE_KEY = "hbase.hregion.memstore.mslab.chunksize";
int CHUNK_SIZE_DEFAULT = 2048 * 1024;
int CHUNK_SIZE_DEFAULT = 2047 * 1024;
String INDEX_CHUNK_SIZE_PERCENTAGE_KEY = "hbase.hregion.memstore.mslab.indexchunksize.percent";
float INDEX_CHUNK_SIZE_PERCENTAGE_DEFAULT = 0.1f;
String MAX_ALLOC_KEY = "hbase.hregion.memstore.mslab.max.allocation";
Expand Down
3 changes: 2 additions & 1 deletion src/main/asciidoc/_chapters/offheap_read_write.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@ of off-heap memory that should be used by MSLAB (e.g. `25` would result in 25MB
MaxDirectMemorySize property (see <<hbase.offheapsize>> for more on _HBASE_OFFHEAPSIZE_). The default value of
`hbase.regionserver.offheap.global.memstore.size` is 0 which means MSLAB uses onheap, not offheap, chunks by default.

`hbase.hregion.memstore.mslab.chunksize` controls the size of each off-heap chunk. Default is `2097152` (2MB).
`hbase.hregion.memstore.mslab.chunksize` controls the size of each off-heap chunk. Default is `2096128` (that's 2047KB, why not 2MB?
see link:https://issues.apache.org/jira/browse/HBASE-27805[HBASE-27805] for more detail).

When a Cell is added to a MemStore, the bytes for that Cell are copied into these off-heap buffers (if `hbase.regionserver.offheap.global.memstore.size` is non-zero)
and a Cell POJO will refer to this memory area. This can greatly reduce the on-heap occupancy of the MemStores and reduce the total heap utilization for RegionServers
Expand Down