segfault error in Cholesky decomposition

[lbelzile]

Reported by Liangliang Zhang.
***caught segfault *** cause 'memory not mapped' in .cholpermGGE
Affects both Windows and Mac OSX.

[lbelzile]

Running with valgrind and MWE gives

==5914== Invalid read of size 4
==5914==    at 0x1D289CFF: cholperm(arma::Mat<double>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>) (in /home/leo/R/x86_64-pc-linux-gnu-library/3.6/TruncatedNormal/libs/TruncatedNormal.so)
==5914==    by 0x1D27C021: _TruncatedNormal_cholperm (in /home/leo/R/x86_64-pc-linux-gnu-library/3.6/TruncatedNormal/libs/TruncatedNormal.so)
==5914==    by 0x4978065: ??? (in /usr/lib/R/lib/libR.so)
==5914==    by 0x498326F: Rf_eval (in /usr/lib/R/lib/libR.so)
==5914==    by 0x498515F: ??? (in /usr/lib/R/lib/libR.so)
==5914==    by 0x4986004: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==5914==    by 0x497BC85: ??? (in /usr/lib/R/lib/libR.so)
==5914==    by 0x498326F: Rf_eval (in /usr/lib/R/lib/libR.so)
==5914==    by 0x498515F: ??? (in /usr/lib/R/lib/libR.so)
==5914==    by 0x4986004: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==5914==    by 0x497BC85: ??? (in /usr/lib/R/lib/libR.so)
==5914==    by 0x498326F: Rf_eval (in /usr/lib/R/lib/libR.so)
==5914==  Address 0xfffffffe12267554 is not stack'd, malloc'd or (recently) free'd
==5914== 

 *** caught segfault ***
address 0xfffffffe12267554, cause 'memory not mapped'

[lbelzile]

Source of the bug: caused by a degenerate truncated normal model in which some lower bounds are equal to upper bounds (up to numerical precision). The variance of the individual truncated components is negative and this causes the segfault (probably via log(0)).

Two fixes implemented:

• arguments in the two Cholesky are checked to ensure lb < ub (and check is now TRUE by default).
• in rtmvnorm and rtmvt, if the model is degenerate (because of equal truncation bounds), the routine fixes these values to the bound and passes on a sub-problem with the conditional normal or Student.

These are typically indicative of problems in the call, so a warning is thrown.