Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go1.16 Encode/Marshal maybe OOM when high QPS for sonic v1.10.0~v1.11.6 #634

Closed
liuq19 opened this issue May 21, 2024 · 0 comments
Closed

Comments

@liuq19
Copy link
Collaborator

liuq19 commented May 21, 2024

There may be OOM when using sonic Encode/Marshal in too many parallel goroutines.

The problem is caused by #486 .

Reproduce code

package issue_test

import (
    "strings"
    "testing"

    "sync"

    "github.com/bytedance/sonic"
    "github.com/bytedance/sonic/option"
    "github.com/stretchr/testify/assert"
)

func marshalSingle() {
    var m = map[string]interface{}{
        "1": map[string]interface{} {
            `"`+strings.Repeat("a", int(option.DefaultEncoderBufferSize) - 38)+`"`: "b",
            "1": map[string]int32{
                "b": 1658219785,
            },
        },
    }
    _, err := sonic.Marshal(&m)
    if err != nil {
        panic("err")
    }
}

type zoo foo

func (z *zoo) MarshalJSON() ([]byte, error) {
    marshalSingle()
    return sonic.Marshal((*foo)(z))
}

type foo bar

func (f *foo) MarshalJSON() ([]byte, error) {
    marshalSingle()
    return sonic.Marshal((*bar)(f))
}

type bar int

func (b *bar) MarshalJSON() ([]byte, error) {
    marshalSingle()
    return sonic.Marshal(int(*b))
}

 func TestEncodeOOM(t *testing.T) {
    wg := &sync.WaitGroup{}
    N := 10000
    for i:=0; i<N; i++ {
        wg.Add(1)
        go func (wg *sync.WaitGroup)  {
            defer wg.Done()
            var z zoo
            _, err := sonic.Marshal(&z)
            assert.NoError(t, err)
        }(wg)
    }
    wg.Wait()
 }
@liuq19 liuq19 changed the title Go1.16 Encode/Marshal OOM when high QPS Go1.16 Encode/Marshal OOM when high QPS for sonic v1.10.0~v1.11.6 May 21, 2024
@liuq19 liuq19 changed the title Go1.16 Encode/Marshal OOM when high QPS for sonic v1.10.0~v1.11.6 Go1.16 Encode/Marshal maybe OOM when high QPS for sonic v1.10.0~v1.11.6 May 21, 2024
@liuq19 liuq19 closed this as completed May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant