panic unprotected error in call to lua api (not enough memory) #120

lynnf87 · 2017-03-13T09:05:34Z

test mysql
panic unprotected error in call to lua api (not enough memory)

akopytov · 2017-03-13T09:09:17Z

Can you provide more details please? Knowing the platform, sysbench version and command line arguments would be helpful. Thanks!

lynnf87 · 2017-03-13T10:59:03Z

--test=select.lua mysql --num-threads=4096
system centos 6.5 64bit memory 64G sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta2)
memory is ok and less than 4096 no error
Thanks

akopytov · 2017-03-13T11:21:16Z

How much memory is actually available for sysbench? Depending on the contents of the Lua script, sysbench with 4096 threads can allocate about 2 GB of memory, mostly for LuaJIT. I would monitor memory usage with a lower numbers of threads when sysbench doesn't crash and then do the math.

lynnf87 · 2017-03-13T11:50:12Z

about 30G available when --max-time >300 then crash

akopytov · 2017-03-13T11:53:07Z

So it must be either a memory leak in sysbench itself, or garbage collection in LuaJIT not working properly. Can you share the contents of select.lua to me to debug further?

akopytov · 2017-03-19T06:40:10Z

Closing, feel free to reopen if can provide more details.

luckywhu · 2017-10-10T05:24:48Z

I encounter the same problem with sysbench 1.1.0
and i try different version of sysbench and have the same problem

"PANIC: unprotected error in call to Lua API (not enough memory)"

/local/sysbench/bin/sysbench --test=/local/sysbench/share/sysbench/oltp_insert.lua --time=100 --mysql-host=ip --mysql-port=5714 --auto-inc=0 --mysql-db=sbtest --mysql-user=sbtest --mysql-password=sbtest --max-time=100 --table-size=200000000 --tables=10 --max-requests=0 --report-interval=1 --num-threads=4096 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
WARNING: --max-time is deprecated, use --time instead
sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 4096
Report intermediate results every 1 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 1s ] thds: 4096 tps: 251639.38 qps: 251639.38 (r/w/o: 0.00/251639.38/0.00) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 2s ] thds: 4096 tps: 201786.34 qps: 201786.34 (r/w/o: 0.00/201786.34/0.00) lat (ms,95%): 29.19 err/s: 0.00 reconn/s: 0.00
[ 3s ] thds: 4096 tps: 169315.59 qps: 169315.59 (r/w/o: 0.00/169315.59/0.00) lat (ms,95%): 40.37 err/s: 0.00 reconn/s: 0.00
[ 4s ] thds: 4096 tps: 278786.23 qps: 278785.23 (r/w/o: 0.00/278785.23/0.00) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 5s ] thds: 4096 tps: 279187.76 qps: 279166.76 (r/w/o: 0.00/279166.76/0.00) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
PANIC: unprotected error in call to Lua API (not enough memory)

it seem's this was caused by LUAJIT and build sysbench with system LUAJIT solve this problem

export LUAJIT_CFLAGS=/usr/local/lua/include && export LUAJIT_LIBS=/usr/local/lua/lib/libluajit.so && ./configure --prefix=/usr/local/sysbench110 --with-mysql-includes=/usr/local/5714/include --with-mysql-libs=/u01/mysql/lib   --with-system-luajit=system --with-extra-ldflags=-all-static && make -j 64 && make install

i think there is something wrong with the bunded LUAJIT

akopytov · 2017-10-10T05:35:16Z

@luckywhu thanks for the info. It looks like the sysbench version you were using had LuaJIT 2.1.0-beta2, while the latest sysbench bundles LuaJIT 2.1.0-beta3. System LuaJIT is usually 2.0.

Can you please try the latest sysbench with LuaJIT 2.1.0-beta3 and see if it makes any difference?

But generally, LuaJIT 2.1 is expected to require more memory per interpreter state. So using sysbench with that many threads is not a good idea. I need to spend some time to look at possible solutions.

luckywhu · 2017-10-10T06:05:52Z

@akopytov
the same problem with sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

 /usr/local/sysbench110/bin/sysbench --test=/usr/local/sysbench110/share/sysbench/oltp_insert.lua --time=100 --mysql-host=ip --mysql-port=5714 --auto-inc=0 --mysql-db=sbtest --mysql-user=sbtest --mysql-password=sbtest --max-time=100 --table-size=200000000 --tables=10 --max-requests=0 --report-interval=1 --num-threads=4096 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
WARNING: --max-time is deprecated, use --time instead
sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 4096
Report intermediate results every 1 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 1s ] thds: 4096 tps: 196540.10 qps: 196540.10 (r/w/o: 0.00/196540.10/0.00) lat (ms,95%): 39.65 err/s: 0.00 reconn/s: 0.00
[ 2s ] thds: 4096 tps: 180564.11 qps: 180567.11 (r/w/o: 0.00/180567.11/0.00) lat (ms,95%): 40.37 err/s: 0.00 reconn/s: 0.00
[ 3s ] thds: 4096 tps: 148465.46 qps: 148459.46 (r/w/o: 0.00/148459.46/0.00) lat (ms,95%): 54.83 err/s: 0.00 reconn/s: 0.00
[ 4s ] thds: 4096 tps: 127646.85 qps: 127649.85 (r/w/o: 0.00/127649.85/0.00) lat (ms,95%): 30.81 err/s: 0.00 reconn/s: 0.00
[ 5s ] thds: 4096 tps: 214476.10 qps: 214476.10 (r/w/o: 0.00/214476.10/0.00) lat (ms,95%): 33.12 err/s: 0.00 reconn/s: 0.00
[ 6s ] thds: 4096 tps: 233825.59 qps: 233822.59 (r/w/o: 0.00/233822.59/0.00) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 7s ] thds: 4096 tps: 240646.51 qps: 240645.51 (r/w/o: 0.00/240645.51/0.00) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
PANIC: unprotected error in call to Lua API (not enough memory)

but sysbench 1.1.0 (using system LuaJIT 2.1.0-beta2) is ok

/usr/local/sysbench110/bin/sysbench --test=/usr/local/sysbench110/share/sysbench/oltp_insert.lua --time=10  --mysql-port=5714 --auto-inc=0 --mysql-db=sbtest --mysql-user=sbtest --mysql-password=sbtest --max-time=10 --table-size=200000000 --tables=10 --max-requests=0 --report-interval=1 --num-threads=4096 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
WARNING: --max-time is deprecated, use --time instead
sysbench 1.1.0 (using system LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 4096
Report intermediate results every 1 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 1s ] thds: 4096 tps: 168474.72 qps: 168529.07 (r/w/o: 0.00/168529.07/0.00) lat (ms,95%): 32.53 err/s: 0.00 reconn/s: 0.00
[ 2s ] thds: 4096 tps: 235535.95 qps: 235450.95 (r/w/o: 0.00/235450.95/0.00) lat (ms,95%): 44.17 err/s: 0.00 reconn/s: 0.00
[ 3s ] thds: 4096 tps: 146205.61 qps: 148482.43 (r/w/o: 0.00/148482.43/0.00) lat (ms,95%): 48.34 err/s: 0.00 reconn/s: 0.00
[ 4s ] thds: 4096 tps: 193337.73 qps: 191117.60 (r/w/o: 0.00/191117.60/0.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 5s ] thds: 4096 tps: 209165.73 qps: 209373.81 (r/w/o: 0.00/209373.81/0.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00
[ 6s ] thds: 4096 tps: 197161.54 qps: 197604.54 (r/w/o: 0.00/197604.54/0.00) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 7s ] thds: 4096 tps: 202487.28 qps: 201778.28 (r/w/o: 0.00/201778.28/0.00) lat (ms,95%): 33.12 err/s: 0.00 reconn/s: 0.00
[ 8s ] thds: 4096 tps: 212776.17 qps: 213006.17 (r/w/o: 0.00/213006.17/0.00) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00
[ 9s ] thds: 4096 tps: 202269.36 qps: 202035.36 (r/w/o: 0.00/202035.36/0.00) lat (ms,95%): 35.59 err/s: 0.00 reconn/s: 0.00
[ 10s ] thds: 4096 tps: 206041.76 qps: 206389.73 (r/w/o: 0.00/206389.73/0.00) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            0
        write:                           1969995
        other:                           0
        total:                           1969995
    transactions:                        1969995 (189903.16 per sec.)
    queries:                             1969995 (189903.16 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

Throughput:
    events/s (eps):                      189903.1614
    time elapsed:                        10.3737s
    total number of events:              1969995

Latency (ms):
         min:                                  1.82
         avg:                                 20.81
         max:                               1522.05
         95th percentile:                     33.72
         sum:                            40996187.08

Threads fairness:
    events (avg/stddev):           480.9558/29.78
    execution time (avg/stddev):   10.0088/0.11

akopytov · 2017-10-10T06:09:58Z

@luckywhu thanks. I wonder if --luajit-cmd=off would also fix the OOM problem.

luckywhu · 2017-10-10T06:24:24Z

different error message

WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
WARNING: --max-time is deprecated, use --time instead
sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 4096
Report intermediate results every 1 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 1s ] thds: 4096 tps: 122270.15 qps: 122271.08 (r/w/o: 0.00/122271.08/0.00) lat (ms,95%): 31.94 err/s: 0.00 reconn/s: 0.00
[ 2s ] thds: 4096 tps: 211198.13 qps: 211197.13 (r/w/o: 0.00/211197.13/0.00) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 3s ] thds: 4096 tps: 221095.18 qps: 221095.18 (r/w/o: 0.00/221095.18/0.00) lat (ms,95%): 30.81 err/s: 0.00 reconn/s: 0.00
[ 4s ] thds: 4096 tps: 232495.49 qps: 232495.49 (r/w/o: 0.00/232495.49/0.00) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 5s ] thds: 4096 tps: 212889.10 qps: 212887.10 (r/w/o: 0.00/212887.10/0.00) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 6s ] thds: 4096 tps: 225397.33 qps: 225399.33 (r/w/o: 0.00/225399.33/0.00) lat (ms,95%): 27.66 err/s: 0.00 reconn/s: 0.00
[ 7s ] thds: 4096 tps: 218666.57 qps: 218666.57 (r/w/o: 0.00/218666.57/0.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
FATAL: `thread_run' function failed: not enough memory
Error in my_thread_global_end(): 13 threads didn't exit

akopytov · 2017-10-10T06:25:32Z

@luckywhu interesting. thanks!

helloEson · 2018-06-21T09:04:08Z

Hi @akopytov I have met the same problem, below is the detail.

sysbench oltp_insert --threads=4096 --events=100000000 --time=120 --report-interval=5 --debug=off --tables=16 --mysql_storage_engine=innodb --mysql-host=172.16.8.101 --mysql-port=3307 --mysql-user=sysbench --mysql-password=12345 --mysql-db=platform --histogram=on --db-ps-mode=disable --skip_trx=on --non_index_updates=0 --delete_inserts=10 --index_updates=10 --percentile=90 run | tee 5k_90
sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 4096
Report intermediate results every 5 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 5s ] thds: 4096 tps: 39019.65 qps: 39019.65 (r/w/o: 0.00/39019.65/0.00) lat (ms,90%): 204.11 err/s: 0.00 reconn/s: 0.00
[ 10s ] thds: 4096 tps: 47634.30 qps: 47634.30 (r/w/o: 0.00/47634.30/0.00) lat (ms,90%): 189.93 err/s: 0.00 reconn/s: 0.00
[ 15s ] thds: 4096 tps: 53600.14 qps: 53600.14 (r/w/o: 0.00/53600.14/0.00) lat (ms,90%): 183.21 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 4096 tps: 42813.10 qps: 42813.10 (r/w/o: 0.00/42813.10/0.00) lat (ms,90%): 176.73 err/s: 0.00 reconn/s: 0.00
[ 25s ] thds: 4096 tps: 41712.77 qps: 41712.77 (r/w/o: 0.00/41712.77/0.00) lat (ms,90%): 139.85 err/s: 0.00 reconn/s: 0.00
PANIC: unprotected error in call to Lua API (not enough memory)

sysbench oltp_read_only --threads=5000 --events=100000000 --time=120 --report-interval=5 --debug=off --tables=16 --mysql_storage_engine=innodb --table_size=1000000 --mysql-host=172.16.8.101 --mysql-port=3307 --mysql-user=sysbench --mysql-password=12345 --mysql-db=platform --histogram=on --db-ps-mode=disable --distinct_ranges=0 --sum_ranges=0 --skip_trx=on --simple_ranges=0 --index_updates=0 --delete_inserts=0 --non_index_updates=0 --point_selects=10 --percentile=90 run | tee 5k_90
sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 5000
Report intermediate results every 5 second(s)
Initializing random number generator from current time


Initializing worker threads...

FATAL: error loading module 'oltp_common' from file '/usr/local/share/sysbench/oltp_common.lua':
        not enough memory

cyckax · 2019-05-24T10:06:23Z

I still have this kind of questions. I only need to generate 200 threads, not as much as 4096...

cyckax · 2019-05-24T10:31:43Z

but I init 1000 tables。./sysbench --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=*** --mysql-password=**** --mysql-db=sbtest --tables=1000 --table_size=500000 --rand-type=uniform --threads=250 --time=604800 --report-interval=10 --db-ps-mode=disable --percentile=95 --skip_trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062 lua/oltp_read_write.lua run

gmcalist · 2020-02-12T18:59:20Z

From my testing at large numbers of threads this is luajit hitting the 2GB memory limit. When I switched out the luajit that is included with the current version of sysbench with this one that has a patch to fix the memory limit I can now run more than 2K threads. https://github.com/openresty/luajit2

dimitrik-fr · 2020-02-13T21:22:19Z

Tried to do the same, finally got Sysbench compiled with this luajit2, but getting error on probe test: FATAL: 'sysbench.cmdline.call_command' function failed: ./lua/oltp_common.lua:222: attempt to call global 'sb_rand' (a nil value)

any idea what could go wrong ?

Rgds, -Dimitri

dimitrik-fr · 2020-02-14T18:13:37Z

Ok, seems to be the false error -- I did not pay attention to update my Lua scripts on the host where I've tested the new Sysbench binary ;-)) -- now seems to work, test in progress!

However, I also saw the claim that the latest LuaJIT-2.1 has also the fix for 2GB RAM limit, so I'm curious now to try both and see if the problem is solved for both, let's see..

Rgds, -Dimitri

dimitrik-fr · 2020-02-17T16:58:38Z

So far, I can confirm the latest code of LuaJIT-2.1 has also fix for 2GB limit -- makes sense to upgrade the bundled LuaJIT in Sysbench to the latest code as well ! -- any objections ?

Rgds, -Dimitri

cyckax · 2020-02-19T03:17:13Z

So far, I can confirm the latest code of LuaJIT-2.1 has also fix for 2GB limit -- makes sense to upgrade the bundled LuaJIT in Sysbench to the latest code as well ! -- any objections ?

Rgds, -Dimitri

how to upgrade the the bundled LuaJIT in Sysbench to the latest code？

cyckax · 2020-02-19T12:02:29Z

From my testing at large numbers of threads this is luajit hitting the 2GB memory limit. When I switched out the luajit that is included with the current version of sysbench with this one that has a patch to fix the memory limit I can now run more than 2K threads. https://github.com/openresty/luajit2

Hi, I also replace the luajit this version, but it still note me "PANIC: unprotected error in call to Lua API (not enough memory)"

how do you do the replace activity?

dimitrik-fr · 2020-02-19T19:32:07Z

To upgrade to the latest LuaJIT-2.1 :

download LuaJIT-2.1 from https://github.com/LuaJIT/LuaJIT
replace in Sysbench source tree ./third_party/luajit/luajit by the tree from LuaJIT-2.1
autogen, configure, compile as usual
enjoy new Sysbench binary ;-))

Rgds, -Dimitri

akopytov added the need feedback label Mar 13, 2017

akopytov closed this as completed Mar 19, 2017

akopytov added the question label Oct 10, 2017

lhy1024 mentioned this issue Oct 19, 2021

update luajit to luajit2 #433

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic unprotected error in call to lua api (not enough memory) #120

panic unprotected error in call to lua api (not enough memory) #120

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

akopytov commented Mar 19, 2017

luckywhu commented Oct 10, 2017 •

edited

Loading

akopytov commented Oct 10, 2017

luckywhu commented Oct 10, 2017

akopytov commented Oct 10, 2017

luckywhu commented Oct 10, 2017

akopytov commented Oct 10, 2017

helloEson commented Jun 21, 2018 •

edited

Loading

cyckax commented May 24, 2019

cyckax commented May 24, 2019

gmcalist commented Feb 12, 2020

dimitrik-fr commented Feb 13, 2020

dimitrik-fr commented Feb 14, 2020 •

edited

Loading

dimitrik-fr commented Feb 17, 2020

cyckax commented Feb 19, 2020

cyckax commented Feb 19, 2020

dimitrik-fr commented Feb 19, 2020

panic unprotected error in call to lua api (not enough memory) #120

panic unprotected error in call to lua api (not enough memory) #120

Comments

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

lynnf87 commented Mar 13, 2017

akopytov commented Mar 13, 2017

akopytov commented Mar 19, 2017

luckywhu commented Oct 10, 2017 • edited Loading

akopytov commented Oct 10, 2017

luckywhu commented Oct 10, 2017

akopytov commented Oct 10, 2017

luckywhu commented Oct 10, 2017

akopytov commented Oct 10, 2017

helloEson commented Jun 21, 2018 • edited Loading

cyckax commented May 24, 2019

cyckax commented May 24, 2019

gmcalist commented Feb 12, 2020

dimitrik-fr commented Feb 13, 2020

dimitrik-fr commented Feb 14, 2020 • edited Loading

dimitrik-fr commented Feb 17, 2020

cyckax commented Feb 19, 2020

cyckax commented Feb 19, 2020

dimitrik-fr commented Feb 19, 2020

luckywhu commented Oct 10, 2017 •

edited

Loading

helloEson commented Jun 21, 2018 •

edited

Loading

dimitrik-fr commented Feb 14, 2020 •

edited

Loading