Runtime errors about buffer overflows

[Y4NG333]

Hello, after compiling according to the readme file, when I run the fastafs list command, there is a problem of detecting buffer overflow. Have you encountered this situation? If this happens, how to solve this problem? Thank you very much.

[yhoogstrate]

Hi @Y4NG333 ,

Sorry for the late response. Github didn't send me an e-mail for this issue, i will go after this. No, I haven't seen such an error before. Could you:

• copy the exact error
• run: ls -als ~/.local/share/fastafs/ ; cat ~/.local/share/fastafs/index

[Y4NG333]

It's okay, thanks for your reply.The specific error is shown in Figure 1.
When I enter the fastafs list command, I get a buffer overflow error.
Figure 2 is the situation in the index.

[yhoogstrate]

Thank you!

I suspect conversion did something weird. The error presumably takes places reading your compressed file. If there are no legal issues regarding your test file, would you mind sendin me an e-mail with the plain test fasta and the fastafs archive (~/.local/share/fastafs/test.fastafs.gz)? My e-mail adress is y.hoogstrate at [g][m][a][i][l]<dot>[c][o][m].

Thank you so much for reporting.

[Y4NG333]

OK, it's been emailed to you.

[yhoogstrate]

Thanks @Y4NG333,

I received your e-mail. During the day I will be doing non-fastafs releated data analysis. I will try to make time for this soon after!

[Y4NG333]

OK, thanks a lot!

[yhoogstrate]

Hi @Y4NG333 ,

I've been testing but I can't reproduce it. Your files are byte identical to mine, so it must be the fastafs binary behaving differently on your system than on mine. Which git commit did you compile (git log | head -n 10)?

Could you run make check? One this will for sure give an error @ one of the tests because I changed buffer size.
I'm a bit puzzled by the error, but I will try my very best to resolve this.

Youri

[Y4NG333]

Thanks @yhoogstrate ,

When I run make check command, it shows that there is a problem with a part of the test, as shown in the following figure:

I downloaded the code to my linux system through the git clone command, and then followed the readme file:
#!/bin/bash
cmake -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON .
make "$@" -j nproc
sudo make install
to complete the compilation.

[yhoogstrate]

This is interesting. Could you copy the output of ./bin/test_cache and ./bin/test_fastafs.

I suspect there is something going wrong with integer conversion or something. I will probably make a branch with all kind of debug messages, ask you to re-compile and copy the output of test again.

[Y4NG333]

OK, thanks a lot! I sent the file to your email.

[yhoogstrate]

@Y4NG333 Could you also send me he output when running those binaries directly from the command line ($./bin/test_cache and $./bin/test_fastafs). I seem to have a much older system and I can't run your binaries due to different glibc versions.

[yhoogstrate]

Hi again,

My appologies for all this disturbance. I have created a branch with more debug options for me.

Could you do the following from the main fastafs directory:

make clean
git fetch
git checkout -b debug_buff_overflow origin/debug_buff_overflow
./build-debug.sh
./bin/test/test_fastafs > debug-messages.txt

Please send me the debug messages.

If you know how to use gdb, could you please run this?
gdb bin/test/test_fastafs
(gdb) type: run
(gdb) type: quit

and copy all the uitput after run into a file.

[Y4NG333]

The specific debugging information has been sent to you, thank you very much for your help.

[yhoogstrate]

Oh no, thank you!! Are you running ubuntu 21.04 or 21.10? I will also setup a VM.

[Y4NG333]

it seems to be 20.04

[yhoogstrate]

Thanks. I found a compile bug for 21.10 and resolved it.
Anyway, I am closing in on where the error appears at your instance. I suspect something with a function names realpath.

Could you run:

git pull
make clean
./build-debug.sh
./bin/test/test_fastafs > debug-messages.txt

[Y4NG333]

Seems like there are no errors?

[yhoogstrate]

What!? and without piping the output to the debug-messages file?

juist ./bin/test/test_fastafs

[Y4NG333]

When I enter the code ./bin/test/test_fastafs, the file is as follows: (Because the content is too long, I can't take a screenshot, I will send you a copy of the entire file)
001.txt

[yhoogstrate]

I am totally lost. The code only contains additional printing messages and somehow it now works.
Did you change something in your VM or VM settings or your host OS that could be related to memory, filenames or file access?

[yhoogstrate]

Hi @Y4NG333,

I haven't received an update from you anymore. Did you get the master branch working without issues?

Youri

[Y4NG333]

Hi，I'm so sorry to reply to you now, after receiving your reply, I haven't used a virtual machine anymore, but chose to connect to a server remotely to run the code, but the error of buffer overflow still exists.

When I use the make check command, the errors appear to be the same as before.

[yhoogstrate]

Hi again,

Hmm, if the debug branch never errors out and the main branch does, I honestly don't know/understand what is causing this.
Could you compile and run the debug branch a few more times?

make clean
git checkout debug_buff_overflow
git pull
./build-debug.sh

./bin/test/test_fastafs
./bin/test/test_fastafs > debug-messages.txt

./bin/test/test_fastafs
./bin/test/test_fastafs > debug-messages.2.txt

Otherwise maybe we need to do a teams or zoom session?

[yhoogstrate]

Found it, checking the version of FASTAFS or UCSC 2bit went 4 bytes out of bound. Thanks @Y4NG333 . Finally I could reproduce it because of your bug report!!! I found it by usingVMs of other distro's.