Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pdns_server --list-modules dumps core on OpenBSD #2422

Closed
paultheweird opened this issue Apr 10, 2015 · 33 comments
Closed

pdns_server --list-modules dumps core on OpenBSD #2422

paultheweird opened this issue Apr 10, 2015 · 33 comments

Comments

@paultheweird
Copy link

When running pdns_server --list-modules I get a core dump on OpenBSD. This is PowerDNS 3.4.3 (port not updated in OpenBSD ports tree yet):

[weerd@pom] $ pdns_server --list-modules         
Apr 10 18:10:49 Reading random entropy from '/dev/urandom'
Apr 10 18:10:49 Loading '/usr/local/lib/powerdns/pdns/libgeobackend.so'
Apr 10 18:10:49 Loading '/usr/local/lib/powerdns/pdns/libgsqlite3backend.so'
Modules available:
geo
gsqlite3
pdns_server(3822) in free(): error: chunk is already free 0xb065ba27d00
Abort trap (core dumped) 

Running gdb on the core file I get the following backtrace:

(gdb) bt
#0  0x00000b061c1b991a in kill () at <stdin>:2
#1  0x00000b061c1f30d9 in abort () at /usr/src/lib/libc/stdlib/abort.c:53
#2  0x00000b061c1ef7f0 in wrterror (msg=0xb061c2fc476 "chunk is already free", p=0xb065ba27d00)
    at /usr/src/lib/libc/stdlib/malloc.c:281
#3  0x00000b061c1ef8e6 in find_chunknum (d=Variable "d" is not available.
) at /usr/src/lib/libc/stdlib/malloc.c:1005
#4  0x00000b061c1f0aff in ofree (p=0xb065ba27d00) at /usr/src/lib/libc/stdlib/malloc.c:1240
#5  0x00000b061c1f0cee in free (ptr=0xb065ba27d00) at /usr/src/lib/libc/stdlib/malloc.c:1282
#6  0x00000b0374e75bba in GeoBackend::queueNSRecords () from /usr/local/sbin/pdns_server
#7  0x00000b061c1d5d49 in __cxa_finalize (dso=0x0) at /usr/src/lib/libc/stdlib/atexit.c:141
#8  0x00000b061c1c7f1e in exit (status=99) at /usr/src/lib/libc/stdlib/exit.c:57
#9  0x00000b0374f42384 in main () from /usr/local/sbin/pdns_server
Current language:  auto; currently asm
@fobser
Copy link
Contributor

fobser commented Apr 11, 2015

With the geo backend disabled and the bind backend enabled I see a similar crash:

$ pdns_server --list-modules
Apr 11 12:01:41 Reading random entropy from '/dev/urandom'
Apr 11 12:01:41 Loading '/usr/local/lib/powerdns/pdns/libbindbackend.so'
Apr 11 12:01:41 Loading '/usr/local/lib/powerdns/pdns/libgsqlite3backend.so'
Modules available:
bind
gsqlite3
Bus error (core dumped)

(gdb) bt
#0  0x00001b17a3d7bb36 in Bind2Backend::safeRemoveBBDomainInfo ()
   from /usr/local/sbin/pdns_server
#1  0x00001b1a18c8ef59 in __cxa_finalize (dso=0x0)
    at /usr/src/lib/libc/stdlib/atexit.c:142
#2  0x00001b1a18c6ac4e in exit (status=99)
    at /usr/src/lib/libc/stdlib/exit.c:57
#3  0x00001b17a3e57ef4 in main () from /usr/local/sbin/pdns_server
(gdb)

With only the sqlite backend enabled --list-modules works fine.
A limited test of the bind backend shows that it's working fine, this seems to be only an issue with --list-modules.

@cmouse
Copy link
Contributor

cmouse commented Apr 11, 2015

Any chance you could run valgrind on this?

@fobser
Copy link
Contributor

fobser commented Apr 11, 2015

On 04/11/15 13:13, Aki Tuomi wrote:

Any chance you could run valgrind on this?

Not at this time, openbsd doesn't have valgrind, yet.

@cmouse
Copy link
Contributor

cmouse commented Apr 11, 2015

@fobser
Copy link
Contributor

fobser commented Apr 12, 2015

Oh, heh, I didn't know he valgrind to work yet.
This is with the geo backend enabled again:

$ valgrind /usr/local/sbin/pdns_server --list-modules 
==7972== Memcheck, a memory error detector
==7972== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==7972== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==7972== Command: /usr/local/sbin/pdns_server --list-modules
==7972== 
==7972== Invalid read of size 8
==7972==    at 0x72D83B6: pthread_self (rthread.c:279)
==7972==    by 0x72D8464: _rthread_dl_lock (rthread.c:669)
==7972==    by 0x72D8050: _rthread_init (rthread.c:214)
==7972==    by 0x72D83C6: pthread_self (rthread.c:276)
==7972==    by 0x72DB13A: _rthread_mutex_lock (rthread_sync.c:88)
==7972==    by 0x72D7374: pthread_once (rthread_once.c:24)
==7972==    by 0x5272102: __gthread_once (gthr-default.h:575)
==7972==    by 0x5272102: std::locale::_S_initialize() (locale_init.cc:265)
==7972==    by 0x527240C: std::locale::locale() (locale_init.cc:214)
==7972==    by 0x52576C0: basic_streambuf (streambuf:462)
==7972==    by 0x52576C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==7972==    by 0x52576C0: std::ios_base::Init::Init() (ios_init.cc:89)
==7972==    by 0x583B3BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==7972==    by 0x583B416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==7972==    by 0x57F7BFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==7972==  Address 0x10 is not stack'd, malloc'd or (recently) free'd
==7972== 
==7972== 
==7972== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==7972==  Access not within mapped region at address 0x10
==7972==    at 0x72D83B6: pthread_self (rthread.c:279)
==7972==    by 0x72D8464: _rthread_dl_lock (rthread.c:669)
==7972==    by 0x72D8050: _rthread_init (rthread.c:214)
==7972==    by 0x72D83C6: pthread_self (rthread.c:276)
==7972==    by 0x72DB13A: _rthread_mutex_lock (rthread_sync.c:88)
==7972==    by 0x72D7374: pthread_once (rthread_once.c:24)
==7972==    by 0x5272102: __gthread_once (gthr-default.h:575)
==7972==    by 0x5272102: std::locale::_S_initialize() (locale_init.cc:265)
==7972==    by 0x527240C: std::locale::locale() (locale_init.cc:214)
==7972==    by 0x52576C0: basic_streambuf (streambuf:462)
==7972==    by 0x52576C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==7972==    by 0x52576C0: std::ios_base::Init::Init() (ios_init.cc:89)
==7972==    by 0x583B3BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==7972==    by 0x583B416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==7972==    by 0x57F7BFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==7972==  If you believe this happened as a result of a stack
==7972==  overflow in your program's main thread (unlikely but
==7972==  possible), you can try to increase the size of the
==7972==  main thread stack using the --main-stacksize= flag.
==7972==  The main thread stack size used in this run was 4194304.

valgrind: m_coredump/coredump-elf.c:816 (make_elf_coredump): Assertion 'VG_(lseek)(core_fd, phdrs[idx].p_offset, VKI_SEEK_SET) == phdrs[idx].p_offset' failed.

host stacktrace:

==7972==    at 0x3803B343: ???
==7972==    by 0x8030BEFDF: ???
==7972==    by 0x38054882: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==7972==    at 0x72D83B6: pthread_self (rthread.c:279)
==7972==    by 0x72D8464: _rthread_dl_lock (rthread.c:669)
==7972==    by 0x72D8050: _rthread_init (rthread.c:214)
==7972==    by 0x72D83C6: pthread_self (rthread.c:276)
==7972==    by 0x72DB13A: _rthread_mutex_lock (rthread_sync.c:88)
==7972==    by 0x72D7374: pthread_once (rthread_once.c:24)
==7972==    by 0x5272102: __gthread_once (gthr-default.h:575)
==7972==    by 0x5272102: std::locale::_S_initialize() (locale_init.cc:265)
==7972==    by 0x527240C: std::locale::locale() (locale_init.cc:214)
==7972==    by 0x52576C0: basic_streambuf (streambuf:462)
==7972==    by 0x52576C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==7972==    by 0x52576C0: std::ios_base::Init::Init() (ios_init.cc:89)
==7972==    by 0x583B3BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==7972==    by 0x583B416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==7972==    by 0x57F7BFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==7972==    by 0x57E8D18: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)

@cmouse
Copy link
Contributor

cmouse commented Apr 12, 2015

Can you compile the code again without removing debugging symbols and see what gdb thinks. To be sure, you can add -ggdb to the CFLAGS and CXXFLAGS.

@paultheweird
Copy link
Author

After updating the port to put the modules in /usr/local/lib/pdns/ (as opposed to /usr/local/lib/powerdns/pdns/), the crash seems to be gone:

[weerd@pom] $ pdns_server --list-modules
Apr 14 08:57:15 Reading random entropy from '/dev/urandom'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libgsqlite3backend.so'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libbindbackend.so'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libgeobackend.so'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libldapbackend.so'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libgmysqlbackend.so'
Apr 14 08:57:15 Loading '/usr/local/lib/pdns/libgpgsqlbackend.so'
Modules available:
bind
geo
gmysql
gpgsql
gpgsql2
gsqlite3
ldap
[weerd@pom] $ 

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

Did you update PKGLIBDIR when moving the modules to the other directory?

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

Or module-dir in powerdns config

@paultheweird
Copy link
Author

The alternative location was set with --libdir="${PREFIX}/lib/powerdns" as a configure flag (where PREFIX defaults to /usr/local/ in the OpenBSD ports tree).

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

Can you test if the crash occurs if you define "module-dir= /usr/local/lib/powerdns/pdns" in pdns.conf?

@paultheweird
Copy link
Author

It doesn't:

[weerd@pom] $ sudo vi pdns.conf                     
[weerd@pom] $ pdns_server --list-modules            
Apr 14 10:05:28 Reading random entropy from '/dev/urandom'
Apr 14 10:05:28 Unable to open module directory '/usr/local/lib/powerdns/pdns'
Modules available:
[weerd@pom] $ sudo mkdir -p /usr/local/lib/powerdns/pdns
[weerd@pom] $ sudo cp /usr/local/lib/pdns/* /usr/local/lib/powerdns/pdns/
[weerd@pom] $ pdns_server --list-modules                                 
Apr 14 10:05:55 Reading random entropy from '/dev/urandom'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libbindbackend.so'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libgeobackend.so'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libgmysqlbackend.so'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libgpgsqlbackend.so'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libgsqlite3backend.so'
Apr 14 10:05:55 Loading '/usr/local/lib/powerdns/pdns/libldapbackend.so'
Modules available:
bind
geo
gmysql
gpgsql
gpgsql2
gsqlite3
ldap
[weerd@pom] $ 

So I guess that rules out the module-dir part?

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

did you recompile it to original problem state first? can you run grep PKGLIBDIR in some Makefile?

@paultheweird
Copy link
Author

Oh, sorry, no I didn't. Let me do that now. Will report in a few minutes.

@paultheweird
Copy link
Author

Confirmed that a binary with the /usr/local/lib/powerdns/pdns path configured and that same directory configured in the pdns.conf module-dir option still crashes:

[weerd@pom] $ grep ^module-dir /etc/pdns/pdns.conf 
module-dir=/usr/local/lib/powerdns/pdns
[weerd@pom] $ pdns_server --list-modules          
Apr 14 10:18:18 Reading random entropy from '/dev/urandom'
Apr 14 10:18:18 Loading '/usr/local/lib/powerdns/pdns/libgsqlite3backend.so'
Apr 14 10:18:18 Loading '/usr/local/lib/powerdns/pdns/libbindbackend.so'
Apr 14 10:18:18 Loading '/usr/local/lib/powerdns/pdns/libgeobackend.so'
Modules available:
bind
geo
gsqlite3
pdns_server(23216) in free(): error: chunk is already free 0x6f99a2c9640
Abort trap (core dumped) 
[weerd@pom] $ 

Also, here's the grep for PKGLIBDIR:

[weerd@pom] $ grep -r PKGLIBDIR /usr/ports/pobj/powerdns-3.4.3/pdns-3.4.3/pdns/Makefile
/usr/ports/pobj/powerdns-3.4.3/pdns-3.4.3/pdns/Makefile:        -DPKGLIBDIR=\"$(pkglibdir)\" -DLOCALSTATEDIR=\"$(socketdir)\" \
[weerd@pom] $ 

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

um, can you grep again for pkglibdir?

@paultheweird
Copy link
Author

pkglibdir = $(libdir)/pdns

and

libdir = /usr/local/lib/powerdns

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

interesting that it crashes when you change that. i can't find any references in the code for PKGLIBDIR other than as default value for module-dir. did you compile with -ggdb, can you use gdb to get us a stack trace?

@paultheweird
Copy link
Author

I'd have to look into that as I'm not 100% sure about the details. Will get back to you later today or tomorrow. Thanks for your interest so far!

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

make clean
./configure <stuff here> CXXFLAGS="-O2 -Wall -ggdb" CFLAGS="-O2 -Wall -ggdb"
make
gdb program
r <parameter>
<after crash>
bt

@paultheweird
Copy link
Author

(gdb) bt
#0  0x00000163f7b8d9ba in kill () at <stdin>:2
#1  0x00000163f7bc7839 in abort () at /usr/src/lib/libc/stdlib/abort.c:53
#2  0x00000163f7b9c1b0 in wrterror (msg=0x163f7cceeb6 "chunk is already free", p=0x163a8dc7800)
    at /usr/src/lib/libc/stdlib/malloc.c:281
#3  0x00000163f7b9c2a6 in find_chunknum (d=Variable "d" is not available.
) at /usr/src/lib/libc/stdlib/malloc.c:1010
#4  0x00000163f7b9d4cf in ofree (p=0x163a8dc7800) at /usr/src/lib/libc/stdlib/malloc.c:1245
#5  0x00000163f7b9d6be in free (ptr=0x163a8dc7800) at /usr/src/lib/libc/stdlib/malloc.c:1287
#6  0x000001613037b53a in __tcf_6 () at basic_string.h:238
#7  0x00000163f7bc3f59 in __cxa_finalize (dso=0x0) at /usr/src/lib/libc/stdlib/atexit.c:142
#8  0x00000163f7b9fc4e in exit (status=99) at /usr/src/lib/libc/stdlib/exit.c:57
#9  0x0000016130467604 in main (argc=2, argv=0x7f7fffff0a08) at receiver.cc:519
Current language:  auto; currently asm
(gdb) 

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

Can you run valgrind once more? Not that I expect it to reveal anything new.

@fobser
Copy link
Contributor

fobser commented Apr 14, 2015

I don't quite get why valgrind outputs ??? for boost, it's build with debug symbols
and not stripped.

[florian@openbsd-build:~]$ valgrind pdns_server --list-modules
==27696== Memcheck, a memory error detector
==27696== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==27696== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==27696== Command: pdns_server --list-modules
==27696==
==27696== Invalid read of size 8
==27696== at 0x5D813B6: pthread_self (rthread.c:279)
==27696== by 0x5D81464: _rthread_dl_lock (rthread.c:669)
==27696== by 0x5D81050: _rthread_init (rthread.c:214)
==27696== by 0x5D813C6: pthread_self (rthread.c:276)
==27696== by 0x5D8413A: _rthread_mutex_lock (rthread_sync.c:88)
==27696== by 0x5D80374: pthread_once (rthread_once.c:24)
==27696== by 0x4C9F102: __gthread_once (gthr-default.h:575)
==27696== by 0x4C9F102: std::locale::_S_initialize() (locale_init.cc:265)
==27696== by 0x4C9F40C: std::locale::locale() (locale_init.cc:214)
==27696== by 0x4C846C0: basic_streambuf (streambuf:462)
==27696== by 0x4C846C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==27696== by 0x4C846C0: std::ios_base::Init::Init() (ios_init.cc:89)
==27696== by 0x71E13BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==27696== by 0x71E1416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==27696== by 0x719DBFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==27696== Address 0x10 is not stack'd, malloc'd or (recently) free'd
==27696==
==27696==
==27696== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==27696== Access not within mapped region at address 0x10
==27696== at 0x5D813B6: pthread_self (rthread.c:279)
==27696== by 0x5D81464: _rthread_dl_lock (rthread.c:669)
==27696== by 0x5D81050: _rthread_init (rthread.c:214)
==27696== by 0x5D813C6: pthread_self (rthread.c:276)
==27696== by 0x5D8413A: _rthread_mutex_lock (rthread_sync.c:88)
==27696== by 0x5D80374: pthread_once (rthread_once.c:24)
==27696== by 0x4C9F102: __gthread_once (gthr-default.h:575)
==27696== by 0x4C9F102: std::locale::_S_initialize() (locale_init.cc:265)
==27696== by 0x4C9F40C: std::locale::locale() (locale_init.cc:214)
==27696== by 0x4C846C0: basic_streambuf (streambuf:462)
==27696== by 0x4C846C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==27696== by 0x4C846C0: std::ios_base::Init::Init() (ios_init.cc:89)
==27696== by 0x71E13BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==27696== by 0x71E1416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==27696== by 0x719DBFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==27696== If you believe this happened as a result of a stack
==27696== overflow in your program's main thread (unlikely but
==27696== possible), you can try to increase the size of the
==27696== main thread stack using the --main-stacksize= flag.
==27696== The main thread stack size used in this run was 4194304.

valgrind: m_coredump/coredump-elf.c:816 (make_elf_coredump): Assertion 'VG_(lseek)(core_fd, phdrs[idx].p_offset, VKI_SEEK_SET) == phdrs[idx].p_offset' failed.

host stacktrace:
==27696== at 0x3803B343: ???
==27696== by 0x8030BEFDF: ???
==27696== by 0x38054882: ???

sched status:
running_tid=1

Thread 1: status = VgTs_Runnable
==27696== at 0x5D813B6: pthread_self (rthread.c:279)
==27696== by 0x5D81464: _rthread_dl_lock (rthread.c:669)
==27696== by 0x5D81050: _rthread_init (rthread.c:214)
==27696== by 0x5D813C6: pthread_self (rthread.c:276)
==27696== by 0x5D8413A: _rthread_mutex_lock (rthread_sync.c:88)
==27696== by 0x5D80374: pthread_once (rthread_once.c:24)
==27696== by 0x4C9F102: __gthread_once (gthr-default.h:575)
==27696== by 0x4C9F102: std::locale::_S_initialize() (locale_init.cc:265)
==27696== by 0x4C9F40C: std::locale::locale() (locale_init.cc:214)
==27696== by 0x4C846C0: basic_streambuf (streambuf:462)
==27696== by 0x4C846C0: stdio_sync_filebuf (stdio_sync_filebuf.h:72)
==27696== by 0x4C846C0: std::ios_base::Init::Init() (ios_init.cc:89)
==27696== by 0x71E13BA: __static_initialization_and_destruction_0(int, int) (iostream:77)
==27696== by 0x71E1416: global constructors keyed to libs_serialization_src_xml_oarchive.cpp_1FD5A050_A88192F8 (xml_oarchive.cpp:33)
==27696== by 0x719DBFD: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)
==27696== by 0x718ED18: ??? (in /usr/local/lib/libboost_serialization-mt.so.5.1)

@cmouse
Copy link
Contributor

cmouse commented Apr 14, 2015

Wonder if this can be due to our hardening features... can you provide the CXXFLAGS, CFLAGS and LDFLAGS from Makefile?

@paultheweird
Copy link
Author

Sorry for the slow reply.

CXXFLAGS = -pthread -O2 -Wall -ggdb
CFLAGS = -O2 -Wall -ggdb
LDFLAGS = -pthread -L/usr/local/lib

I would suspect the OpenBSD hardening features before PowerDNS hardening features (since this seems an OpenBSD-only problem). Happy to be proven wrong though :)

@cmouse
Copy link
Contributor

cmouse commented Apr 15, 2015

Those are not used here, it seems, unless the compiler itself adds some stuff by default.

@paultheweird
Copy link
Author

On OpenBSD, binaries are built with several hardening features by default (pie and stack protection come to mind).

@rubenk
Copy link
Contributor

rubenk commented May 5, 2015

This is fixed by 44ac597

@ahupowerdns
Copy link
Contributor

did you test it? I didn’t.

@rubenk
Copy link
Contributor

rubenk commented May 5, 2015

I did.

@ahupowerdns
Copy link
Contributor

ahupowerdns commented May 5, 2015 via email

@rubenk
Copy link
Contributor

rubenk commented May 5, 2015

Is this worth backporting to 3.x?

@Habbie
Copy link
Member

Habbie commented Jun 16, 2015

geobackend is gone, closing this

@Habbie Habbie closed this as completed Jun 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants