Skip to content

Commit 38d0aad

Browse files
sergey-senozhatskyakpm00
sergey-senozhatsky
authored and
akpm00
committed
zsmalloc: turn zspage order into runtime variable
Some use-cases and/or data patterns may benefit from larger zspages. Currently the limit on the number of physical pages that are linked into a zspage is hardcoded to 4. Higher limit changes key characteristics of a number of the size clases, improving compactness of the pool and redusing the amount of memory zsmalloc pool uses. For instance, the huge size class watermark is currently set to 3264 bytes. With order 3 zspages we have more normal classe and huge size watermark becomes 3632. With order 4 zspages huge size watermark becomes 3840. Commit #1 has more numbers and some analysis. This patch (of 6): zsmalloc has 255 size classes. Size classes contain a number of zspages, which store objects of the same size. zspage can consist of up to four physical pages. The exact (most optimal) zspage size is calculated for each size class during zsmalloc pool creation. As a reasonable optimization, zsmalloc merges size classes that have similar characteristics: number of pages per zspage and number of objects zspage can store. For example, let's look at the following size classes: class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable .. 94 1536 0 0 0 0 0 3 0 100 1632 0 0 0 0 0 2 0 .. Size classes torvalds#95-99 are merged with size class torvalds#100. That is, each time we store an object of size, say, 1568 bytes instead of using class torvalds#96 we end up storing it in size class torvalds#100. Class torvalds#100 is for objects of 1632 bytes in size, hence every 1568 bytes object wastes 1632-1568 bytes. Class torvalds#100 zspages consist of 2 physical pages and can hold 5 objects. When we need to store, say, 13 objects of size 1568 we end up allocating three zspages; in other words, 6 physical pages. However, if we'll look closer at size class torvalds#96 (which should hold objects of size 1568 bytes) and trace get_pages_per_zspage(): pages per zspage wasted bytes used% 1 960 76 2 352 95 3 1312 89 4 704 95 5 96 99 We'd notice that the most optimal zspage configuration for this class is when it consists of 5 physical pages, but currently we never let zspages to consists of more than 4 pages. A 5 page class torvalds#96 configuration would store 13 objects of size 1568 in a single zspage, allocating 5 physical pages, as opposed to 6 physical pages that class torvalds#100 will allocate. A higher order zspage for class torvalds#96 also changes its key characteristics: pages per-zspage and objects per-zspage. As a result classes torvalds#96 and torvalds#100 are not merged anymore, which gives us more compact zsmalloc. Let's take a closer look at the bottom of /sys/kernel/debug/zsmalloc/zram0/classes: class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable ... 202 3264 0 0 0 0 0 4 0 254 4096 0 0 0 0 0 1 0 ... For exactly same reason - maximum 4 pages per zspage - the last non-huge size class is torvalds#202, which stores objects of size 3264 bytes. Any object larger than 3264 bytes, hence, is considered to be huge and lands in size class torvalds#254, which uses a whole physical page to store every object. To put it slightly differently - objects in huge classes don't share physical pages. 3264 bytes is too low of a watermark and we have too many huge classes: classes from torvalds#203 to torvalds#254. Similarly to class size torvalds#96 above, higher order zspages change key characteristics for some of those huge size classes and thus those classes become normal classes, where stored objects share physical pages. We move huge class watermark with higher order zspages. For order 3, huge class watermark becomes 3632 bytes: class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable ... 202 3264 0 0 0 0 0 4 0 211 3408 0 0 0 0 0 5 0 217 3504 0 0 0 0 0 6 0 222 3584 0 0 0 0 0 7 0 225 3632 0 0 0 0 0 8 0 254 4096 0 0 0 0 0 1 0 ... For order 4, huge class watermark becomes 3840 bytes: class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable ... 202 3264 0 0 0 0 0 4 0 206 3328 0 0 0 0 0 13 0 207 3344 0 0 0 0 0 9 0 208 3360 0 0 0 0 0 14 0 211 3408 0 0 0 0 0 5 0 212 3424 0 0 0 0 0 16 0 214 3456 0 0 0 0 0 11 0 217 3504 0 0 0 0 0 6 0 219 3536 0 0 0 0 0 13 0 222 3584 0 0 0 0 0 7 0 223 3600 0 0 0 0 0 15 0 225 3632 0 0 0 0 0 8 0 228 3680 0 0 0 0 0 9 0 230 3712 0 0 0 0 0 10 0 232 3744 0 0 0 0 0 11 0 234 3776 0 0 0 0 0 12 0 235 3792 0 0 0 0 0 13 0 236 3808 0 0 0 0 0 14 0 238 3840 0 0 0 0 0 15 0 254 4096 0 0 0 0 0 1 0 ... TESTS ===== 1) ChromeOS memory pressure test ----------------------------------------------------------------------------- Our standard memory pressure test, that is designed with the reproducibility in mind. zram is configured as a swap device, lzo-rle compression algorithm. We captured /sys/block/zram0/mm_stat after every test and rebooted device. Columns per (Documentation/admin-guide/blockdev/zram.rst) orig_data_size mem_used_total mem_used_max pages_compacted compr_data_size mem_limit same_pages huge_pages ORDER 2 (BASE) 10353639424 2981711944 3166896128 0 3543158784 579494 825135 123707 10168573952 2932288347 3106541568 0 3499085824 565187 853137 126153 9950461952 2815911234 3035693056 0 3441090560 586696 748054 122103 9892335616 2779566152 2943459328 0 3514736640 591541 650696 119621 9993949184 2814279212 3021357056 0 3336421376 582488 711744 121273 9953226752 2856382009 3025649664 0 3512893440 564559 787861 123034 9838448640 2785481728 2997575680 0 3367219200 573282 777099 122739 ORDER 3 9509138432 2706941227 2823393280 0 3389587456 535856 1011472 90223 10105245696 2882368370 3013095424 0 3296165888 563896 1059033 94808 9531236352 2666125512 2867650560 0 3396173824 567117 1126396 88807 9561812992 2714536764 2956652544 0 3310505984 548223 827322 90992 9807470592 2790315707 2908053504 0 3378315264 563670 1020933 93725 10178371584 2948838782 3071209472 0 3329548288 548533 954546 90730 9925165056 2849839413 2958274560 0 3336978432 551464 1058302 89381 ORDER 4 9444515840 2613362645 2668232704 0 3396759552 573735 1162207 83475 10129108992 2925888488 3038351360 0 3499597824 555634 1231542 84525 9876594688 2786692282 2897006592 0 3469463552 584835 1290535 84133 10012909568 2649711847 2801512448 0 3171323904 675405 750728 80424 10120966144 2866742402 2978639872 0 3257815040 587435 1093981 83587 9578790912 2671245225 2802270208 0 3376353280 545548 1047930 80895 10108588032 2888433523 2983960576 0 3316641792 571445 1290640 81402 First, we establish that order 3 and 4 don't cause any statistically significant change in `orig_data_size` (number of bytes we store during the test), in other words larger zspages don't cause regressions. T-test for order 3: x order-2-stored + order-3-stored +-----------------------------------------------------------------------------+ |+ + + + x x + x x + x+ x| | |________________________AM__|_________M_____A____|__________| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 9.8384486e+09 1.0353639e+10 9.9532268e+09 1.0021519e+10 1.7916718e+08 + 7 9.5091384e+09 1.0178372e+10 9.8074706e+09 9.8026344e+09 2.7856206e+08 No difference proven at 95.0% confidence T-test for order 4: x order-2-stored + order-4-stored +-----------------------------------------------------------------------------+ | + | |+ + x +x xx x + ++ x x| | |__________________|____A____M____M____________|_| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 9.8384486e+09 1.0353639e+10 9.9532268e+09 1.0021519e+10 1.7916718e+08 + 7 9.4445158e+09 1.0129109e+10 1.001291e+10 9.8959249e+09 2.7947784e+08 No difference proven at 95.0% confidence Next we establish that there is a statistically significant improvement in `mem_used_total` metrics. T-test for order 3: x order-2-usedmem + order-3-usedmem +-----------------------------------------------------------------------------+ |+ + + x ++ x + xx x + x x| | |_________________A__M__|____________|__A________________| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 2.9434593e+09 3.1668961e+09 3.0256497e+09 3.0424532e+09 73235062 + 7 2.8233933e+09 3.0712095e+09 2.9566525e+09 2.9426185e+09 84630851 Difference at 95.0% confidence -9.98347e+07 +/- 9.21744e+07 -3.28139% +/- 3.02961% (Student's t, pooled s = 7.91383e+07) T-test for order 4: x order-2-usedmem + order-4-usedmem +-----------------------------------------------------------------------------+ | + x | |+ + + x ++ x x * x x| | |__________________A__M__________|_____|_M__A__________| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 2.9434593e+09 3.1668961e+09 3.0256497e+09 3.0424532e+09 73235062 + 7 2.6682327e+09 3.0383514e+09 2.8970066e+09 2.8814248e+09 1.3098053e+08 Difference at 95.0% confidence -1.61028e+08 +/- 1.23591e+08 -5.29272% +/- 4.0622% (Student's t, pooled s = 1.06111e+08) Order 3 zspages also show statistically significant improvement in `mem_used_max` metrics. T-test for order 3: x order-2-maxmem + order-3-maxmem +-----------------------------------------------------------------------------+ |+ + + x+ x + + + x x x x| | |________M__A_________|_|_____________________A___________M____________| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 3.3364214e+09 3.5431588e+09 3.4990858e+09 3.4592294e+09 80073158 + 7 3.2961659e+09 3.3961738e+09 3.3369784e+09 3.3481822e+09 39840377 Difference at 95.0% confidence -1.11047e+08 +/- 7.36589e+07 -3.21017% +/- 2.12934% (Student's t, pooled s = 6.32415e+07) Order 4 zspages, on the other hand, do not show any statistically significant improvement in `mem_used_max` metrics. T-test for order 4: x order-2-maxmem + order-4-maxmem +-----------------------------------------------------------------------------+ |+ + + x x + + x + * x x| | |_______________________A___M________________A_|_____M_______| | +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 3.3364214e+09 3.5431588e+09 3.4990858e+09 3.4592294e+09 80073158 + 7 3.1713239e+09 3.4995978e+09 3.3763533e+09 3.3554221e+09 1.1609062e+08 No difference proven at 95.0% confidence Overall, with sufficient level of confidence order 3 zspages appear to be beneficial for these particular use-case and data patterns. Rather expectedly we also observed lower numbers of huge-pages when zsmalloc is configured with order 3 and order 4 zspages, for the reason already explained. 2) Synthetic test ----------------------------------------------------------------------------- Test untars linux-6.0.tar.xz and compiles the kernel. zram is configured as a block device with ext4 file system, lzo-rle compression algorithm. We captured /sys/block/zram0/mm_stat after every test and rebooted VM. orig_data_size mem_used_total mem_used_max pages_compacted compr_data_size mem_limit same_pages huge_pages ORDER 2 (BASE) 1691807744 628091753 655187968 0 655187968 59 0 34042 34043 1691803648 628089105 655159296 0 655159296 60 0 34043 34043 1691795456 628087429 655151104 0 655151104 59 0 34046 34046 1691799552 628093723 655216640 0 655216640 60 0 34044 34044 ORDER 3 1691787264 627781464 641740800 0 641740800 59 0 33591 33591 1691795456 627794239 641789952 0 641789952 59 0 33591 33591 1691811840 627788466 641691648 0 641691648 60 0 33591 33591 1691791360 627790682 641781760 0 641781760 59 0 33591 33591 ORDER 4 1691807744 627729506 639627264 0 639627264 59 0 33432 33432 1691820032 627731485 639606784 0 639606784 59 0 33432 33432 1691799552 627725753 639623168 0 639623168 59 0 33432 33433 1691820032 627734080 639746048 0 639746048 61 0 33432 33432 Order 3 and order 4 show statistically significant improvement in `mem_used_total` metrics. T-test for order 3: x order-2-usedmem-comp + order-3-usedmem-comp +-----------------------------------------------------------------------------+ |++ x| |++ x| |AM A| +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 4 6.551511e+08 6.5521664e+08 6.5518797e+08 6.5517875e+08 29795.878 + 4 6.4169165e+08 6.4178995e+08 6.4178176e+08 6.4175104e+08 45056 Difference at 95.0% confidence -1.34277e+07 +/- 66089.8 -2.04947% +/- 0.0100873% (Student's t, pooled s = 38195.8) T-test for order 4: x order-2-usedmem-comp + order-4-usedmem-comp +-----------------------------------------------------------------------------+ |+ x| |+ x| |++ x| |A| A| +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 4 6.551511e+08 6.5521664e+08 6.5518797e+08 6.5517875e+08 29795.878 + 4 6.3960678e+08 6.3974605e+08 6.3962726e+08 6.3965082e+08 64101.637 Difference at 95.0% confidence -1.55279e+07 +/- 86486.9 -2.37003% +/- 0.0132005% (Student's t, pooled s = 49984.1) Order 3 and order 4 show statistically significant improvement in `mem_used_max` metrics. T-test for order 3: x order-2-maxmem-comp + order-3-maxmem-comp +-----------------------------------------------------------------------------+ |++ x| |++ x| |AM A| +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 4 6.551511e+08 6.5521664e+08 6.5518797e+08 6.5517875e+08 29795.878 + 4 6.4169165e+08 6.4178995e+08 6.4178176e+08 6.4175104e+08 45056 Difference at 95.0% confidence -1.34277e+07 +/- 66089.8 -2.04947% +/- 0.0100873% (Student's t, pooled s = 38195.8) T-test for order 4: x order-2-maxmem-comp + order-4-maxmem-comp +-----------------------------------------------------------------------------+ |+ x| |+ x| |++ x| |A| A| +-----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 4 6.551511e+08 6.5521664e+08 6.5518797e+08 6.5517875e+08 29795.878 + 4 6.3960678e+08 6.3974605e+08 6.3962726e+08 6.3965082e+08 64101.637 Difference at 95.0% confidence -1.55279e+07 +/- 86486.9 -2.37003% +/- 0.0132005% (Student's t, pooled s = 49984.1) This test tends to benefit more from order 4 zspages, due to test's data patterns. Data patterns that generate a considerable number of badly compressible objects benefit from higher `huge_class_size` watermark, which is achieved with order 4 zspages. Link: https://lkml.kernel.org/r/20221024161213.3221725-1-senozhatsky@chromium.org Link: https://lkml.kernel.org/r/20221024161213.3221725-2-senozhatsky@chromium.org Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org> Cc: Minchan Kim <minchan@kernel.org> Cc: Nitin Gupta <ngupta@vflare.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
1 parent eae5270 commit 38d0aad

File tree

2 files changed

+55
-30
lines changed

2 files changed

+55
-30
lines changed

‎include/linux/zsmalloc.h

+13
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,19 @@ enum zs_mapmode {
3333
*/
3434
};
3535

36+
#define ZS_PAGE_ORDER_2 2
37+
#define ZS_PAGE_ORDER_3 3
38+
#define ZS_PAGE_ORDER_4 4
39+
40+
/*
41+
* A single 'zspage' is composed of up to 2^N discontiguous 0-order (single)
42+
* pages. ZS_MAX_PAGE_ORDER defines upper limit on N, ZS_MIN_PAGE_ORDER
43+
* defines lower limit on N. ZS_DEFAULT_PAGE_ORDER is recommended value.
44+
*/
45+
#define ZS_MIN_PAGE_ORDER ZS_PAGE_ORDER_2
46+
#define ZS_MAX_PAGE_ORDER ZS_PAGE_ORDER_4
47+
#define ZS_DEFAULT_PAGE_ORDER ZS_PAGE_ORDER_2
48+
3649
struct zs_pool_stats {
3750
/* How many pages were migrated (freed) */
3851
atomic_long_t pages_compacted;

‎mm/zsmalloc.c

+42-30
Original file line numberDiff line numberDiff line change
@@ -74,12 +74,7 @@
7474
*/
7575
#define ZS_ALIGN 8
7676

77-
/*
78-
* A single 'zspage' is composed of up to 2^N discontiguous 0-order (single)
79-
* pages. ZS_MAX_ZSPAGE_ORDER defines upper limit on N.
80-
*/
81-
#define ZS_MAX_ZSPAGE_ORDER 2
82-
#define ZS_MAX_PAGES_PER_ZSPAGE (_AC(1, UL) << ZS_MAX_ZSPAGE_ORDER)
77+
#define ZS_MAX_PAGES_PER_ZSPAGE (_AC(1, UL) << ZS_MAX_PAGE_ORDER)
8378

8479
#define ZS_HANDLE_SIZE (sizeof(unsigned long))
8580

@@ -124,10 +119,8 @@
124119
#define ISOLATED_BITS 3
125120
#define MAGIC_VAL_BITS 8
126121

127-
#define MAX(a, b) ((a) >= (b) ? (a) : (b))
128-
/* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */
129-
#define ZS_MIN_ALLOC_SIZE \
130-
MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS))
122+
#define ZS_MIN_ALLOC_SIZE 32U
123+
131124
/* each chunk includes extra space to keep handle */
132125
#define ZS_MAX_ALLOC_SIZE PAGE_SIZE
133126

@@ -141,12 +134,10 @@
141134
* determined). NOTE: all those class sizes must be set as multiple of
142135
* ZS_ALIGN to make sure link_free itself never has to span 2 pages.
143136
*
144-
* ZS_MIN_ALLOC_SIZE and ZS_SIZE_CLASS_DELTA must be multiple of ZS_ALIGN
145-
* (reason above)
137+
* pool->min_alloc_size (ZS_MIN_ALLOC_SIZE) and ZS_SIZE_CLASS_DELTA must
138+
* be multiple of ZS_ALIGN (reason above)
146139
*/
147140
#define ZS_SIZE_CLASS_DELTA (PAGE_SIZE >> CLASS_BITS)
148-
#define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \
149-
ZS_SIZE_CLASS_DELTA) + 1)
150141

151142
enum fullness_group {
152143
ZS_EMPTY,
@@ -230,12 +221,16 @@ struct link_free {
230221
struct zs_pool {
231222
const char *name;
232223

233-
struct size_class *size_class[ZS_SIZE_CLASSES];
224+
struct size_class **size_class;
234225
struct kmem_cache *handle_cachep;
235226
struct kmem_cache *zspage_cachep;
236227

237228
atomic_long_t pages_allocated;
238229

230+
u32 num_size_classes;
231+
u32 min_alloc_size;
232+
u32 max_pages_per_zspage;
233+
239234
struct zs_pool_stats stats;
240235

241236
/* Compact classes */
@@ -523,15 +518,15 @@ static void set_zspage_mapping(struct zspage *zspage,
523518
* classes depending on its size. This function returns index of the
524519
* size class which has chunk size big enough to hold the given size.
525520
*/
526-
static int get_size_class_index(int size)
521+
static int get_size_class_index(struct zs_pool *pool, int size)
527522
{
528523
int idx = 0;
529524

530-
if (likely(size > ZS_MIN_ALLOC_SIZE))
531-
idx = DIV_ROUND_UP(size - ZS_MIN_ALLOC_SIZE,
525+
if (likely(size > pool->min_alloc_size))
526+
idx = DIV_ROUND_UP(size - pool->min_alloc_size,
532527
ZS_SIZE_CLASS_DELTA);
533528

534-
return min_t(int, ZS_SIZE_CLASSES - 1, idx);
529+
return min_t(int, pool->num_size_classes - 1, idx);
535530
}
536531

537532
/* type can be of enum type class_stat_type or fullness_group */
@@ -591,7 +586,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
591586
"obj_allocated", "obj_used", "pages_used",
592587
"pages_per_zspage", "freeable");
593588

594-
for (i = 0; i < ZS_SIZE_CLASSES; i++) {
589+
for (i = 0; i < pool->num_size_classes; i++) {
595590
class = pool->size_class[i];
596591

597592
if (class->index != i)
@@ -777,13 +772,13 @@ static enum fullness_group fix_fullness_group(struct size_class *class,
777772
* link together 3 PAGE_SIZE sized pages to form a zspage
778773
* since then we can perfectly fit in 8 such objects.
779774
*/
780-
static int get_pages_per_zspage(int class_size)
775+
static int get_pages_per_zspage(struct zs_pool *pool, int class_size)
781776
{
782777
int i, max_usedpc = 0;
783778
/* zspage order which gives maximum used size per KB */
784779
int max_usedpc_order = 1;
785780

786-
for (i = 1; i <= ZS_MAX_PAGES_PER_ZSPAGE; i++) {
781+
for (i = 1; i <= pool->max_pages_per_zspage; i++) {
787782
int zspage_size;
788783
int waste, usedpc;
789784

@@ -1410,7 +1405,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp)
14101405

14111406
/* extra space in chunk to keep the handle */
14121407
size += ZS_HANDLE_SIZE;
1413-
class = pool->size_class[get_size_class_index(size)];
1408+
class = pool->size_class[get_size_class_index(pool, size)];
14141409

14151410
/* class->lock effectively protects the zpage migration */
14161411
spin_lock(&class->lock);
@@ -1959,7 +1954,7 @@ static void async_free_zspage(struct work_struct *work)
19591954
struct zs_pool *pool = container_of(work, struct zs_pool,
19601955
free_work);
19611956

1962-
for (i = 0; i < ZS_SIZE_CLASSES; i++) {
1957+
for (i = 0; i < pool->num_size_classes; i++) {
19631958
class = pool->size_class[i];
19641959
if (class->index != i)
19651960
continue;
@@ -2108,7 +2103,7 @@ unsigned long zs_compact(struct zs_pool *pool)
21082103
struct size_class *class;
21092104
unsigned long pages_freed = 0;
21102105

2111-
for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
2106+
for (i = pool->num_size_classes - 1; i >= 0; i--) {
21122107
class = pool->size_class[i];
21132108
if (class->index != i)
21142109
continue;
@@ -2152,7 +2147,7 @@ static unsigned long zs_shrinker_count(struct shrinker *shrinker,
21522147
struct zs_pool *pool = container_of(shrinker, struct zs_pool,
21532148
shrinker);
21542149

2155-
for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
2150+
for (i = pool->num_size_classes - 1; i >= 0; i--) {
21562151
class = pool->size_class[i];
21572152
if (class->index != i)
21582153
continue;
@@ -2199,6 +2194,22 @@ struct zs_pool *zs_create_pool(const char *name)
21992194
if (!pool)
22002195
return NULL;
22012196

2197+
pool->max_pages_per_zspage = 1U << ZS_MIN_PAGE_ORDER;
2198+
/* min_alloc_size must be multiple of ZS_ALIGN */
2199+
pool->min_alloc_size = (pool->max_pages_per_zspage << PAGE_SHIFT) >>
2200+
OBJ_INDEX_BITS;
2201+
pool->min_alloc_size = max(pool->min_alloc_size, ZS_MIN_ALLOC_SIZE);
2202+
2203+
pool->num_size_classes =
2204+
DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - pool->min_alloc_size,
2205+
ZS_SIZE_CLASS_DELTA) + 1;
2206+
2207+
pool->size_class = kmalloc_array(pool->num_size_classes,
2208+
sizeof(struct size_class *),
2209+
GFP_KERNEL | __GFP_ZERO);
2210+
if (!pool->size_class)
2211+
goto err;
2212+
22022213
init_deferred_free(pool);
22032214
rwlock_init(&pool->migrate_lock);
22042215

@@ -2213,17 +2224,17 @@ struct zs_pool *zs_create_pool(const char *name)
22132224
* Iterate reversely, because, size of size_class that we want to use
22142225
* for merging should be larger or equal to current size.
22152226
*/
2216-
for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
2227+
for (i = pool->num_size_classes - 1; i >= 0; i--) {
22172228
int size;
22182229
int pages_per_zspage;
22192230
int objs_per_zspage;
22202231
struct size_class *class;
22212232
int fullness = 0;
22222233

2223-
size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
2234+
size = pool->min_alloc_size + i * ZS_SIZE_CLASS_DELTA;
22242235
if (size > ZS_MAX_ALLOC_SIZE)
22252236
size = ZS_MAX_ALLOC_SIZE;
2226-
pages_per_zspage = get_pages_per_zspage(size);
2237+
pages_per_zspage = get_pages_per_zspage(pool, size);
22272238
objs_per_zspage = pages_per_zspage * PAGE_SIZE / size;
22282239

22292240
/*
@@ -2307,7 +2318,7 @@ void zs_destroy_pool(struct zs_pool *pool)
23072318
zs_flush_migration(pool);
23082319
zs_pool_stat_destroy(pool);
23092320

2310-
for (i = 0; i < ZS_SIZE_CLASSES; i++) {
2321+
for (i = 0; i < pool->num_size_classes; i++) {
23112322
int fg;
23122323
struct size_class *class = pool->size_class[i];
23132324

@@ -2327,6 +2338,7 @@ void zs_destroy_pool(struct zs_pool *pool)
23272338
}
23282339

23292340
destroy_cache(pool);
2341+
kfree(pool->size_class);
23302342
kfree(pool->name);
23312343
kfree(pool);
23322344
}

0 commit comments

Comments
 (0)