feat: Implement into_memory for memory manager

[bitdivine]

Motivation

When a memory manager is created, the constructor (init()) consumes the undertlying memory and there is no way of getting it back.

A workaround is to clone a reference to the memory and keep the reference around, however that opens the possibility that the underlying memory is changed without going via the memory manager. It would be better if the memory manager could be made to surrender the underlying memory.

Changes

• Add into_inner() to MemoryManager (and to the inner data structure).

Tests

• A unit test is included to demonstrate that memory can be returned.

[ielashi]

@bitdivine Can you share what your use-case is?

[bitdivine]

@ielashi Sure thing: I am migrating from raw memory to stable structures and back again. Migrating back involves getting the raw memory back from the managed memory - or keeping a second reference. With a wrapper that keeps both managed memory and a second pointer, I also found that this simplifies unit testing schema upgrades. Restarting the "canister" in tests is then just a matter of getting the raw memory back, and that raw memory is then loaded in the upgraded canister. I don't need an entire replica for this. This was not the original motivation but was a nice bonus! :-)

[bitdivine]

It looks as if CI failed because it could not find a comment in the benchmark output. The comment logic in CI was not present in main when I made the fork so I have pulled in the latest changes from main. Hopefully that will fix it.

No such luck. Here is the error:

[ielashi]

It looks as if CI failed because it could not find a comment in the benchmark output. The comment logic in CI was not present in main when I made the fork so I have pulled in the latest changes from main. Hopefully that will fix it.

No such luck. Here is the error:

@bitdivine The issue was actually more complex to fix than excepted, but it's finally resolved. PR LGTM! :)

[github-actions]

canbench 🏋 (dir: .)

No significant performance changes detected ✅

./canbench_results.yml is up to date ✅

---------------------------------------------------

Benchmark: btreemap_insert_blob_4_1024
  total:
    instructions: 601.82 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 123 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_4_1024_v2
  total:
    instructions: 685.72 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 92 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_8_1024
  total:
    instructions: 720.47 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 183 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_8_1024_v2
  total:
    instructions: 811.30 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 138 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_16_1024
  total:
    instructions: 802.98 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 215 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_16_1024_v2
  total:
    instructions: 900.36 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 161 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_32_1024
  total:
    instructions: 852.44 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 230 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_32_1024_v2
  total:
    instructions: 944.16 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 173 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_64_1024
  total:
    instructions: 1.12 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 245 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_64_1024_v2
  total:
    instructions: 1.22 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 183 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_128_1024
  total:
    instructions: 1.38 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 260 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_128_1024_v2
  total:
    instructions: 1.51 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 195 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_256_1024
  total:
    instructions: 1.91 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 292 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_256_1024_v2
  total:
    instructions: 2.10 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 219 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_512_1024
  total:
    instructions: 2.99 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 351 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_512_1024_v2
  total:
    instructions: 3.24 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 263 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_4
  total:
    instructions: 4.79 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 235 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_4_v2
  total:
    instructions: 5.19 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 176 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_8
  total:
    instructions: 4.88 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 237 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_8_v2
  total:
    instructions: 5.27 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 178 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_16
  total:
    instructions: 4.89 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 241 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_16_v2
  total:
    instructions: 5.28 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 181 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_32
  total:
    instructions: 4.91 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 239 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_32_v2
  total:
    instructions: 5.30 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 180 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_64
  total:
    instructions: 4.93 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 250 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_64_v2
  total:
    instructions: 5.32 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 188 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_128
  total:
    instructions: 4.93 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 262 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_128_v2
  total:
    instructions: 5.32 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 196 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_256
  total:
    instructions: 4.94 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 292 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_256_v2
  total:
    instructions: 5.33 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 219 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_512
  total:
    instructions: 5.03 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 348 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_1024_512_v2
  total:
    instructions: 5.43 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 261 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_u64_u64
  total:
    instructions: 438.06 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 7 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_u64_u64_v2
  total:
    instructions: 508.14 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 6 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_u64_blob_8
  total:
    instructions: 426.45 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 7 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_u64_blob_8_v2
  total:
    instructions: 492.82 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 5 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_8_u64
  total:
    instructions: 416.17 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 6 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_blob_8_u64_v2
  total:
    instructions: 514.62 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 4 pages (no change)

---------------------------------------------------

Benchmark: btreemap_insert_10mib_values
  total:
    instructions: 99.51 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 32 pages (no change)

---------------------------------------------------

Benchmark: btreemap_iter_count_small_values
  total:
    instructions: 10.12 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_iter_count_10mib_values
  total:
    instructions: 497.38 K (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_4_1024
  total:
    instructions: 616.57 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_4_1024_v2
  total:
    instructions: 716.54 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_8_1024
  total:
    instructions: 805.39 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_8_1024_v2
  total:
    instructions: 921.13 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_16_1024
  total:
    instructions: 994.05 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_16_1024_v2
  total:
    instructions: 1.13 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_32_1024
  total:
    instructions: 1.07 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_32_1024_v2
  total:
    instructions: 1.20 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_64_1024
  total:
    instructions: 1.38 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_64_1024_v2
  total:
    instructions: 1.51 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_128_1024
  total:
    instructions: 1.69 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_128_1024_v2
  total:
    instructions: 1.83 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_256_1024
  total:
    instructions: 2.31 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_256_1024_v2
  total:
    instructions: 2.46 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_512_1024
  total:
    instructions: 3.60 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_512_1024_v2
  total:
    instructions: 3.75 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_u64_u64
  total:
    instructions: 620.59 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_u64_u64_v2
  total:
    instructions: 725.62 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_u64_blob_8
  total:
    instructions: 597.01 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_u64_blob_8_v2
  total:
    instructions: 693.17 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_8_u64
  total:
    instructions: 538.73 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_remove_blob_8_u64_v2
  total:
    instructions: 672.71 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_4_1024
  total:
    instructions: 242.79 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_4_1024_v2
  total:
    instructions: 324.66 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_8_1024
  total:
    instructions: 282.19 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_8_1024_v2
  total:
    instructions: 363.31 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_16_1024
  total:
    instructions: 361.11 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_16_1024_v2
  total:
    instructions: 445.04 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_32_1024
  total:
    instructions: 405.06 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_32_1024_v2
  total:
    instructions: 488.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_64_1024
  total:
    instructions: 649.03 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_64_1024_v2
  total:
    instructions: 732.96 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_128_1024
  total:
    instructions: 878.06 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_128_1024_v2
  total:
    instructions: 976.05 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_256_1024
  total:
    instructions: 1.38 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_256_1024_v2
  total:
    instructions: 1.48 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_512_1024
  total:
    instructions: 2.37 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_512_1024_v2
  total:
    instructions: 2.47 B (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_u64_u64
  total:
    instructions: 234.87 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_u64_u64_v2
  total:
    instructions: 298.87 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_u64_blob_8
  total:
    instructions: 232.00 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_u64_blob_8_v2
  total:
    instructions: 290.96 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_8_u64
  total:
    instructions: 252.25 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap_get_blob_8_u64_v2
  total:
    instructions: 334.30 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_4
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_8
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 1 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_16
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 2 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_32
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 5 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_64
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 9 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_blob_128
  total:
    instructions: 3.78 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 19 pages (no change)

---------------------------------------------------

Benchmark: vec_insert_u64
  total:
    instructions: 5.85 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 1 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_4
  total:
    instructions: 5.79 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_8
  total:
    instructions: 7.03 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_16
  total:
    instructions: 9.79 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_32
  total:
    instructions: 10.62 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_64
  total:
    instructions: 15.31 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_blob_128
  total:
    instructions: 20.90 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_get_u64
  total:
    instructions: 6.27 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: memory_manager_baseline
  total:
    instructions: 1052 (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 8000 pages (no change)

---------------------------------------------------

Benchmark: memory_manager_overhead
  total:
    instructions: 4.54 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 8320 pages (no change)

---------------------------------------------------

Benchmark: memory_manager_grow
  total:
    instructions: 285.29 M (no change)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 32.00 K pages (no change)

---------------------------------------------------

[bitdivine]

@ielashi Thank you! I do not have permission to merge but CI is green.