Error running GPP models with specific knot spacing

[elgabbas]

Hello,

I receive the following messages (warnings?) very frequently when I use the Hmsc-hpc extension (on a Windows server and Linux HPC).

....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:141: RuntimeWarning: divide by zero encountered in divide
  W12 = d12 / rLPar["alphapw"][:,0,None,None]
....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:144: RuntimeWarning: divide by zero encountered in divide
  W22 = d22 / rLPar["alphapw"][:,0,None,None]
....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:144: RuntimeWarning: invalid value encountered in divide
  W22 = d22 / rLPar["alphapw"][:,0,None,None]

....\hmsc-venv\Lib\site-packages\scipy\sparse\linalg\_dsolve\linsolve.py:663: SparseEfficiencyWarning: CSR matrix format is required. Converting to CSR matrix.
  warn('CSR matrix format is required. Converting to CSR matrix.',

....\hmsc-venv/lib/python3.10/site-packages/hmsc/updaters/updateEta.py:172: SparseEfficiencyWarning: CSR matrix format is required. Converting to CSR matrix.
  eta = spsolve_triangular(LiUEta.transpose(), mu1 + np.random.normal(dtype(0), dtype(1), size=[nf*nu]), lower=False)

Should I care about them (The models worked as expected)?

---

---

In addition to these warnings, I am receiving the following error messages when running GPP models with specific knot spacing. I try three variants of knot distances: 25K, 37.5K, and 50K. The model works for the first and the last options but fails for the middle option with the following error. This happens on each chain of this model.
It may not be easy to share a full reproducible script for this model, but I am happy to share the data with you privately when necessary.

....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:141: RuntimeWarning: divide by zero encountered in divide
  W12 = d12 / rLPar["alphapw"][:,0,None,None]
....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:141: RuntimeWarning: invalid value encountered in divide
  W12 = d12 / rLPar["alphapw"][:,0,None,None]
....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:144: RuntimeWarning: divide by zero encountered in divide
  W22 = d22 / rLPar["alphapw"][:,0,None,None]
....\hmsc-venv\Lib\site-packages\hmsc\utils\import_utils.py:144: RuntimeWarning: invalid value encountered in divide
  W22 = d22 / rLPar["alphapw"][:,0,None,None]
sampling
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "....\hmsc-venv\Lib\site-packages\hmsc\run_gibbs_sampler.py", line 266, in <module>
    run_gibbs_sampler(
  File "....\hmsc-venv\Lib\site-packages\hmsc\run_gibbs_sampler.py", line 82, in run_gibbs_sampler
    parSamples = gibbs.sampling_routine(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "....\hmsc-venv\Lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "....\hmsc-venv\Lib\site-packages\tensorflow\python\eager\execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node while/z/truncated_normal_tf/parameterized_truncated_normal/ParameterizedTruncatedNormal defined at (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "....\hmsc-venv\Lib\site-packages\hmsc\run_gibbs_sampler.py", line 266, in <module>

  File "....\hmsc-venv\Lib\site-packages\hmsc\run_gibbs_sampler.py", line 82, in run_gibbs_sampler

  File "....\hmsc-venv\Lib\site-packages\hmsc\gibbs_sampler.py", line 105, in sampling_routine

  File "....\hmsc-venv\Lib\site-packages\hmsc\gibbs_sampler.py", line 146, in sampling_routine

  File "....\hmsc-venv\Lib\site-packages\hmsc\utils\tf_named_func.py", line 7, in decorated_func

  File "....\hmsc-venv\Lib\site-packages\hmsc\updaters\updateZ.py", line 107, in updateZ

  File "....\hmsc-venv\Lib\site-packages\hmsc\updaters\updateZ.py", line 115, in updateZ

  File "....\hmsc-venv\Lib\site-packages\hmsc\updaters\updateZ.py", line 160, in calculate_z_probit

  File "....\hmsc-venv\Lib\site-packages\hmsc\utils\tf_named_func.py", line 7, in decorated_func

  File "....\hmsc-venv\Lib\site-packages\hmsc\updaters\updateZ.py", line 21, in truncated_normal_tf

  File "....\hmsc-venv\Lib\site-packages\hmsc\updaters\updateZ.py", line 24, in truncated_normal_tf

TruncatedNormal exponential distribution rejection sampler failed to accept a sample.
	 [[{{node while/z/truncated_normal_tf/parameterized_truncated_normal/ParameterizedTruncatedNormal}}]] [Op:__inference_sampling_routine_3116]

args=Namespace(samples=1000, transient=2000, thin=5, chains=[0], input='Data/Model_Test_DE/InitMod_GPP37_NoTree.rds', output='Data/Model_Test_DE/GPP37_NoTree_Chain1_post.rds', verbose=2500, hmcleapfrog=10, hmcthin=0, updbe=0, tnlib='tf', fse=1, profile=0, rngseed=0, fp=64)
working directory ****
Initializing TF graph
retracing

Thanks, Ahmed

[elgabbas]

I think the error has resulted as the Hmsc::constructKnots function may return some knots at the exact coordinates of the sampling points. I was able to avoid this error by manually shifting the knot points overlapping with the sampling points using e.g. sf::st_jitter.
I am waiting now for the GPP models to be finished, but at least the error did not appear at starting fitting the models.