Slice out of bounds

[ghost]

I wanted to test this db but I got this error:

panic: runtime error: slice bounds out of range [:1073742336] with length 1073741824

goroutine 1 [running]:
github.com/akrylysov/pogreb/fs.mmap(0xc00008c038, 0x40000200, 0x80000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        .../github.com/akrylysov/pogreb/fs/os_windows.go:32 +0x259
github.com/akrylysov/pogreb/fs.(*osfile).Mmap(0xc000068c90, 0x40000200, 0x200, 0x200)
        .../github.com/akrylysov/pogreb/fs/os.go:100 +0x6e
github.com/akrylysov/pogreb.(*file).append(0xc00004f140, 0xc0001b6800, 0x200, 0x200, 0x0, 0x0, 0x0)
        .../github.com/akrylysov/pogreb/file.go:45 +0xc7
github.com/akrylysov/pogreb.(*dataFile).writeKeyValue(0xc00004f140, 0xc000089eb0, 0x8, 0x8, 0xc000089eb0, 0x8, 0x8, 0x3ffffe00, 0x0, 0x0)
        .../github.com/akrylysov/pogreb/datafile.go:44 +0x1a7
github.com/akrylysov/pogreb.(*DB).put(0xc00004f110, 0xc95a802f, 0xc000089eb0, 0x8, 0x8, 0xc000089eb0, 0x8, 0x8, 0x0, 0x0)
        .../github.com/akrylysov/pogreb/db.go:432 +0x260
github.com/akrylysov/pogreb.(*DB).Put(0xc00004f110, 0xc000089eb0, 0x8, 0x8, 0xc000089eb0, 0x8, 0x8, 0x0, 0x0)
        .../github.com/akrylysov/pogreb/db.go:366 +0x171
main.main()
        .../main.go:27 +0x1b3
exit status 2

Code:

package main

import (
	"encoding/binary"
	"github.com/akrylysov/pogreb"
	"log"
	"time"
)

func main() {
	db, err := pogreb.Open("pogreb.test", nil)
	if err != nil {
		log.Fatal(err)
		return
	}
	defer db.Close()

	start := time.Now()
	var pk [8]byte
	for i := uint64(1); i <= 10000000; i++ {
		binary.BigEndian.PutUint64(pk[:], i)
		if err := db.Put(pk[:], pk[:]); err != nil {
			panic(err)
		}
	}

	log.Println("put 10M: ", time.Now().Sub(start).String())
}

I think the db needs to do automatic fsync when it reaches 1gb file?

[akrylysov]

Hi! What is your OS version? Is it 32 or 64 bit?

[ghost]

win 10 pro x64. i tried to insert only 10M values, that should not be issue on any platform.

[ghost]

I have added sync after every 100 000 Put() operations and immediately when the file got to 1 GiB I got the same error. Not to mention that it took forever(very slow Put() performance). So I think the issue might be the storage driver/format that pogreb uses. Even on older 32bit Windows the limit was 2 GiB so this is not an OS or Go issue but this library's itself.

[akrylysov]

@nonpcnpc Sync is not required and expected to be slow because it forces the OS to commit the file buffers to the underlying device.

The "slice bounds out of range" was fixed in version 0.8.3, thanks for the bug report.

[ghost]

Great, will have a look. Though I would still like to know why it is so incredibly slow to write 10M records? Badger takes 54 seconds, Bolt 4 minutes and BitCask 3 minutes. Pogreb takes so long that I stopped measuring(+it always crashed).

[ghost]

Looks like the bug was fixed BUT after 40 minutes and 4.3GiB file i stopped the script. This is simply unusable library for me.

[akrylysov]

@nonpcnpc I managed to reproduce the issue, write performance on Windows in much lower compared to Linux or Mac, I'll need to investigate that #25.

[ghost]

i tried this with WSL and I had to stop it after 10 minutes and 5gb file. the write performance is simply not there and the file is humongous. even if the read performance somehow beats bolt, it is not worth it. i will stick with badger for writes and bolt for reads.

[akrylysov]

High disk space utilization will be addressed in Pogreb 0.9 #24.