Wrong return value in janus_mutex_trylock for PTHREAD implementation

[BogdanovKirill]

Bug: contract of janus_mutex_trylock depends on compilation settings (pthread enabled/disabled)

janus-gateway/src/mutex.h

Line 37 in eae2947

#define janus_mutex_trylock_nodebug(a) { ret = pthread_mutex_trylock(a); }

Ассording to manual for pthread_mutex_trylock :
"If successful, the pthread_mutex_lock() and pthread_mutex_unlock() functions shall return zero; otherwise, an error number shall be returned to indicate the error."

It means that you should logically invert result value:
ret = !pthread_mutex_trylock(a);

Also you could compare with GLIB implementation:

janus-gateway/src/mutex.h

Line 87 in eae2947

#define janus_mutex_trylock_nodebug(a) { ret = g_mutex_trylock(a); }

Ассording to manual for g_mutex_trylock :
Tries to lock mutex. If mutex is already locked by another thread, it immediately returns FALSE. Otherwise it locks mutex and returns TRUE.

[lminiero]

Good catch. It looks like we currently only use it once, in the Streaming plugin, where we assume the Glib logic you pasted is the correct one. As such, yes, as you suggest we should invert the pthread definition. I'll do that shortly.

[BogdanovKirill]

@lminiero
It seems that your fix is not enough. Check here:

janus-gateway/src/mutex.h

Line 39 in eae2947

#define janus_mutex_trylock_debug(a) { JANUS_PRINT("[%s:%s:%d:trylock] %p\n", __FILE__, __FUNCTION__, __LINE__, a); ret = pthread_mutex_trylock(a); }

[lminiero]

Oh right, the debug/nodebug variants... will fix them too 👍