Memory Usage Explosion by zstd #675
Replies: 6 comments 11 replies
-
Decompressing takes memory. Decompressing 9000x takes huge amount of memory. |
Beta Was this translation helpful? Give feedback.
-
|
Alternatively content must be encoded with a very small window, but that requires you to be in control of the encoding as well. You can enforce a max decoder window, but it will of course fail decoding if that cannot be satisfied. |
Beta Was this translation helpful? Give feedback.
-
|
I switched to using https://github.com/DataDog/zstd, and it works fine. So, the memory issue is specific to this implementation of zstd. In the reduce step, the loader has to read through all the map files because each file has sorted data. So, it's sort of like 9000 streams that the reducer has to read to achieve global sorting. With datadog zstd, each stream takes around 1 MiB, so the whole thing can still easily fit in memory. |
Beta Was this translation helpful? Give feedback.
-
|
I am still unable to reproduce the issue. Are you calling I added a benchmark that tests allocations on Once you have Read from it, it will keep the buffer around until you call |
Beta Was this translation helpful? Give feedback.
-
|
@manishrjain A single stream should be fine.
I cannot see how it will not end up allocating at least "window size" after a few reads. |
Beta Was this translation helpful? Give feedback.
-
|
Here's a few files from the map output: https://link.storjshare.io/jubw4annrrb7kpdzh3rj7efaztoa/zstd |
Beta Was this translation helpful? Give feedback.
-
|
I did a benchmark tests with various window sizes:
Dissecting the 32KB window 64KB allocated in ensureblock, this is windows size + max block size. 32KB is history between calls and 32KB are output from next block. When Window Size gets above 2MB, it will switch from 2x Window size to Window Size + 2 MB in low-mem mode. BENCHMARK SOURCE
I cannot get any numbers from the cgo code, since it doesn't register as Go Heap allocations. Therefore you should double check if you are actually observing memory usage on the OS level or just monitoring the Go heap. I cannot see how it can be significantly (more than 50%) better, since it would at least need to keep the window and decoders in memory for each between calls. Reduce your window size when encoding your data. That is the only way to reduce memory use when streaming so many inputs. As a sidenote, it seems your content compresses better with a 32KB window than any of the others. |
Beta Was this translation helpful? Give feedback.
-
|
With DataDog, Go heap only shows 2.23GB at zstd.NewReader Here's the image from htop -- resident memory usage is around 37 GB. And that also includes around 12 GB of Go memory and 4 GB of jemalloc (so, 16 GB in use, but Go always keeps more than needed).
As you figured out, the map file names are just an artifact of the past when we were using gzip, but they're all zstd. Need to change the naming. I'll try out with 32 KB window size -- that's interesting. I also wonder if Cgo based zstd is better at memory management because of jemalloc / tcmalloc v/s the Go version. We (at Dgraph) definitely saw a huge difference in our programs in terms of memory usage once we switched to jemalloc. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, GOGC=100 will of course allocate double allocs since it is in Go heap, but I expect you've taken that into account. The cgo allocs are made in C-code, so not reflected in Go heap, and GOGC percentage doesn't apply. It is not possible to stream without keeping window and decoders in memory between calls, so they will for sure allocate. I would expect them to keep buffers allocated between calls, so worst case this package uses 2x of what the cgo uses, plus GOGC overhead. |
Beta Was this translation helpful? Give feedback.
-
|
I am also facing the same issue. I am trying to decompress Reddit zstd archives from pushshift.io and they require a 2gb window size. It works fine with this repo, but memory usage goes upto 4gb just for decompression. |
Beta Was this translation helpful? Give feedback.
-
|
The code you link has Also note that Go allocates 2x the amount of memory. Google A window size of 2GB is an extreme choice, so the choice on the encoder side to use that is what is causing this. I suggest if you want to actively work with this data that you recompress it. The For example Using these settings is for archival only, and not really for "daily use". |
Beta Was this translation helpful? Give feedback.
-
|
@klauspost Thank you for the suggestion. I will recompress it. |
Beta Was this translation helpful? Give feedback.
-
|
If you are worried about size, sorting the JSON keys will make the output smaller. Here is a Go program I wrote that will sort the files: package main
import (
"flag"
"fmt"
"io"
"log"
"os"
"sort"
"sync"
"github.com/minio/simdjson-go"
)
func main() {
flag.Parse()
if !simdjson.SupportedCPU() {
log.Fatal("Unsupported CPU")
}
f := os.Stdin
if flag.NArg() > 0 {
var err error
f, err = os.Open(flag.Arg(0))
if err != nil {
log.Fatalf("Failed to load file: %v", err)
}
defer f.Close()
}
const processors = 4
results := make(chan simdjson.Stream, processors)
reuse := make(chan *simdjson.ParsedJson, 1000)
simdjson.ParseNDStream(f, results, reuse)
// Order is not important, so we process results concurrently
var wg sync.WaitGroup
wg.Add(processors)
for i := 0; i < processors; i++ {
go func() {
defer wg.Done()
var o *simdjson.Object
var e *simdjson.Elements
buf := make([]byte, 1024)
for r := range results {
if r.Error != nil {
if r.Error == io.EOF {
return
}
log.Fatal(r.Error)
}
var err error
err = r.Value.ForEach(func(i simdjson.Iter) error {
o, err = i.Object(o)
if err != nil {
return nil
}
e, err = o.Parse(e)
if err != nil {
return nil
}
sort.Slice(e.Elements, func(i, j int) bool {
return e.Elements[i].Name < e.Elements[j].Name
})
buf, err = e.MarshalJSONBuffer(buf[:0])
if err != nil {
return err
}
fmt.Fprintln(os.Stdout, string(buf))
return nil
})
if err != nil {
log.Fatal(err)
}
reuse <- r.Value
}
}()
}
wg.Wait()
}Set "processors =1" if order is important. Example: |
Beta Was this translation helpful? Give feedback.
-
|
@klauspost Wow this is great!! I don't even need all the fields from the JSON. So it makes it much smaller than what it is originally. Also as a Golang newbie there are so many things I learn from this example. Thanks so much! |
Beta Was this translation helpful? Give feedback.
-
|
Added a short test: But it revealed a bug in simdjson: minio/simdjson-go#82 |
Beta Was this translation helpful? Give feedback.
-
Hey @klauspost ,
I just switched to using zstd for compression for Outserv boot (/ Dgraph bulk) loader. My map phase produces ~9000 files, which are read concurrently by the reduce phase.
I was using snappy before, and it was working just fine. When I try to read this via zstd, my memory usage just blows up (using 90 GiB).
I'm opening my reader like so:
dec, err := zstd.NewReader(fd, zstd.WithDecoderConcurrency(1), zstd.WithDecoderLowmem(true))So, trying my best to decrease memory usage. Suggestions?
P.S. Tangential, but perhaps consider using jemalloc for such big allocations instead of Go memory.
Beta Was this translation helpful? Give feedback.
All reactions