Feature/reduce mem pathfinder #3325

SteveBronder · 2024-12-13T19:44:42Z

Submission Checklist

closes #3323
closes #3322

Run unit tests: ./runTests.py src/test/unit
Run cpplint: make cpplint
Declare copyright holder and open-source license: see below

Summary

This removes the large memory footprint for writing to csv for pathfinder. So large models should be nicer to the system memory. This may decrease performance a bit, but the memory savings for large models will be significant so I think it is worth it.

When running single pathfinder from pathfinder we now just return the pathfinder at the best elbo estimates instead of generating all of the constrained parameters within single pathfinder. The constrained parameters are now generated on the fly before we send them to the parameter writer.

For pathfinder that does not do PSIS resampling we write to the parameter writer from within each single pathfinders thread. To avoid contention when writing these we use a mutex before doing a bulk write for each pathfinder instance. I thought about putting each pathfinders results to a seperate file and then combining them at the end. But at the end of the day I think this would still have the same problems as multiple writes to a single file. We only have one system and it can only handle so many writes at the same time. So idt the effect of splitting the writes to multiple files would be noticable for saving time. One possible idea would be to have a thread that is busy spinning while checking a multi producer single consumer queue of values to write to the parameter writer.

Intended Effect

Have pathfinder use less memory when saving parameters to the writer

How to Verify

All previous tests pass

Side Effects

Reduce memory usage

Documentation

Added docs for function that generates parameters and writes them.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Simons Foundation

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

…ager and large chunks of memory

src/stan/callbacks/unique_stream_writer.hpp

stan-buildbot · 2024-12-13T23:52:53Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.4	0.81	-23.9% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.07	6.15% faster
gp_regr/gen_gp_data.stan	0.03	0.03	1.02	1.92% faster
gp_regr/gp_regr.stan	0.09	0.1	0.97	-3.09% slower
sir/sir.stan	70.76	68.67	1.03	2.94% faster
irt_2pl/irt_2pl.stan	4.21	3.85	1.09	8.48% faster
eight_schools/eight_schools.stan	0.06	0.05	1.03	2.46% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.27	0.24	1.12	10.95% faster
pkpd/one_comp_mm_elim_abs.stan	19.37	18.67	1.04	3.61% faster
garch/garch.stan	0.43	0.4	1.07	6.37% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.71	2.6	1.04	4.0% faster
arK/arK.stan	1.72	1.7	1.01	0.9% faster
gp_pois_regr/gp_pois_regr.stan	2.77	2.73	1.02	1.63% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.45	8.33	1.01	1.38% faster
performance.compilation	184.67	179.89	1.03	2.59% faster
Mean result: 1.0231304115474922

Jenkins Console Log
Blue Ocean
Commit hash: 64c5132f901b65575d6999315a9e9cba94b947b3

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

…pathfinder' into feature/reduce-mem-pathfinder

stan-buildbot · 2025-01-06T18:27:03Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.34	0.31	1.1	8.92% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.23	18.39% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.19	15.65% faster
gp_regr/gp_regr.stan	0.09	0.09	1.06	6.03% faster
sir/sir.stan	70.74	70.54	1.0	0.29% faster
irt_2pl/irt_2pl.stan	4.13	3.95	1.05	4.34% faster
eight_schools/eight_schools.stan	0.06	0.05	1.07	6.87% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.26	0.24	1.09	7.84% faster
pkpd/one_comp_mm_elim_abs.stan	19.32	19.43	0.99	-0.55% slower
garch/garch.stan	0.44	0.4	1.1	8.79% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.69	2.6	1.04	3.45% faster
arK/arK.stan	1.77	1.7	1.04	3.92% faster
gp_pois_regr/gp_pois_regr.stan	2.9	2.69	1.08	7.22% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.72	8.38	1.04	3.94% faster
performance.compilation	183.45	180.32	1.02	1.71% faster
Mean result: 1.072236135440247

Jenkins Console Log
Blue Ocean
Commit hash: 64c5132f901b65575d6999315a9e9cba94b947b3

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

stan-buildbot · 2025-01-08T01:19:16Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.34	0.3	1.12	10.64% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.23	18.97% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.19	16.03% faster
gp_regr/gp_regr.stan	0.09	0.09	1.03	3.06% faster
sir/sir.stan	70.5	70.62	1.0	-0.17% slower
irt_2pl/irt_2pl.stan	4.14	3.99	1.04	3.77% faster
eight_schools/eight_schools.stan	0.06	0.05	1.12	10.72% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.24	1.03	2.81% faster
pkpd/one_comp_mm_elim_abs.stan	19.3	18.96	1.02	1.76% faster
garch/garch.stan	0.43	0.4	1.07	6.73% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.74	2.62	1.05	4.36% faster
arK/arK.stan	1.79	1.74	1.03	2.9% faster
gp_pois_regr/gp_pois_regr.stan	2.84	2.69	1.06	5.35% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.76	8.35	1.05	4.7% faster
performance.compilation	180.26	183.45	0.98	-1.77% slower
Mean result: 1.0677421021147349

Jenkins Console Log
Blue Ocean
Commit hash: 318925c5ff1fb92e891ac6e599f194f520a8d465

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

stan-buildbot · 2025-01-08T18:17:34Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.3	1.07	6.23% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.2	16.49% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.09	8.44% faster
gp_regr/gp_regr.stan	0.09	0.09	1.06	5.24% faster
sir/sir.stan	70.53	68.43	1.03	2.98% faster
irt_2pl/irt_2pl.stan	4.13	4.3	0.96	-4.05% slower
eight_schools/eight_schools.stan	0.06	0.06	1.06	5.8% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.25	1.0	0.45% faster
pkpd/one_comp_mm_elim_abs.stan	19.3	18.79	1.03	2.63% faster
garch/garch.stan	0.43	0.42	1.01	0.55% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.73	2.64	1.03	3.28% faster
arK/arK.stan	1.8	2.11	0.85	-17.44% slower
gp_pois_regr/gp_pois_regr.stan	2.88	2.75	1.05	4.5% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.87	8.34	1.06	6.01% faster
performance.compilation	179.87	185.4	0.97	-3.08% slower
Mean result: 1.0312310987573479

Jenkins Console Log
Blue Ocean
Commit hash: 318925c5ff1fb92e891ac6e599f194f520a8d465

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

SteveBronder · 2025-02-12T21:49:23Z

Odd, cmdstan tests are passing locally for me. Going to run this one more time and then will investigate if that same failure pops up again

SteveBronder · 2025-02-12T21:49:44Z

Also odd, this test is running locally for me even with threading on

[ RUN      ] CmdStan.pathfinder_40_draws
src/test/interface/pathfinder_test.cpp:88: Failure
Expected equality of these values:
  1
  count_matches(" seconds (PSIS)", output)
    Which is: 0
[  FAILED  ] CmdStan.pathfinder_40_draws (260 ms)

SteveBronder · 2025-02-13T16:13:38Z

@serban-nicusor-toptal it looks like Jenkins is hanging when trying to spin up the windows unit job. Can you took a look?

Edit: nvm!

stan-buildbot · 2025-02-13T18:13:19Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.32	1.0	-0.34% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.15	13.37% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.06	5.48% faster
gp_regr/gp_regr.stan	0.09	0.09	0.99	-1.23% slower
sir/sir.stan	70.86	68.39	1.04	3.49% faster
irt_2pl/irt_2pl.stan	4.22	4.09	1.03	3.09% faster
eight_schools/eight_schools.stan	0.06	0.06	1.0	0.07% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.26	0.25	1.04	3.89% faster
pkpd/one_comp_mm_elim_abs.stan	19.78	18.9	1.05	4.42% faster
garch/garch.stan	0.48	0.41	1.17	14.5% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.84	2.58	1.1	9.07% faster
arK/arK.stan	1.85	1.71	1.08	7.71% faster
gp_pois_regr/gp_pois_regr.stan	2.89	2.76	1.04	4.3% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	9.15	8.4	1.09	8.18% faster
performance.compilation	197.03	183.15	1.08	7.04% faster
Mean result: 1.060992397358182

Jenkins Console Log
Blue Ocean
Commit hash: 1c418093dfd3248c686e713b8a190cf69f7c63ba

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

stan-buildbot · 2025-02-14T18:42:48Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.33	1.0	-0.13% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.02	1.64% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.34	25.22% faster
gp_regr/gp_regr.stan	0.11	0.09	1.15	13.32% faster
sir/sir.stan	74.66	70.44	1.06	5.65% faster
irt_2pl/irt_2pl.stan	4.34	4.22	1.03	2.79% faster
eight_schools/eight_schools.stan	0.06	0.06	1.1	9.0% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.27	0.25	1.05	4.95% faster
pkpd/one_comp_mm_elim_abs.stan	20.22	19.1	1.06	5.52% faster
garch/garch.stan	0.47	0.48	0.98	-2.07% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.87	2.71	1.06	5.59% faster
arK/arK.stan	1.89	1.72	1.1	8.99% faster
gp_pois_regr/gp_pois_regr.stan	3.07	2.72	1.13	11.48% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	9.16	8.38	1.09	8.54% faster
performance.compilation	189.2	191.42	0.99	-1.17% slower
Mean result: 1.076888101906899

Jenkins Console Log
Blue Ocean
Commit hash: 1c418093dfd3248c686e713b8a190cf69f7c63ba

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

src/stan/callbacks/concurrent_writer.hpp

stan-buildbot · 2025-02-19T00:29:54Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.36	0.32	1.13	11.81% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.3	22.93% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.38	27.38% faster
gp_regr/gp_regr.stan	0.1	0.09	1.07	6.71% faster
sir/sir.stan	71.55	72.61	0.99	-1.48% slower
irt_2pl/irt_2pl.stan	4.19	4.6	0.91	-9.6% slower
eight_schools/eight_schools.stan	0.06	0.06	0.96	-4.04% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.26	0.97	-3.52% slower
pkpd/one_comp_mm_elim_abs.stan	19.42	20.6	0.94	-6.07% slower
garch/garch.stan	0.46	0.42	1.1	8.82% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.82	2.67	1.06	5.47% faster
arK/arK.stan	1.81	1.8	1.0	0.32% faster
gp_pois_regr/gp_pois_regr.stan	2.98	2.93	1.02	1.9% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.8	8.92	0.99	-1.36% slower
performance.compilation	190.15	195.86	0.97	-3.0% slower
Mean result: 1.0521769134568444

Jenkins Console Log
Blue Ocean
Commit hash: 1e79dd5a82df2796732f643eb10420d634ec4200

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 3441.032
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

WardBrian

I am hopeful these will be my final comments before approving. It's looking real good at this point

src/stan/callbacks/concurrent_writer.hpp

src/stan/services/pathfinder/multi.hpp

src/stan/services/pathfinder/single.hpp

WardBrian · 2025-02-19T16:49:11Z

src/test/unit/services/pathfinder/normal_glm_test.cpp

Noting, but optional: Might be nice to add a github actions job which runs some tests with -DSTAN_THREADS and a non-1 number of threads. We can make this and issue rather than do it here, IMO

cc @serban-nicusor-toptal

stan-buildbot · 2025-02-20T00:44:53Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.31	1.04	3.95% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.2	16.83% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.11	10.21% faster
gp_regr/gp_regr.stan	0.1	0.09	1.1	9.35% faster
sir/sir.stan	68.66	66.23	1.04	3.55% faster
irt_2pl/irt_2pl.stan	3.93	4.0	0.98	-1.78% slower
eight_schools/eight_schools.stan	0.06	0.05	1.02	1.92% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.24	0.23	1.03	2.89% faster
pkpd/one_comp_mm_elim_abs.stan	18.95	18.36	1.03	3.12% faster
garch/garch.stan	0.43	0.41	1.05	5.14% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.77	2.6	1.07	6.14% faster
arK/arK.stan	1.79	1.71	1.05	5.0% faster
gp_pois_regr/gp_pois_regr.stan	2.91	2.74	1.06	5.86% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.88	8.38	1.06	5.66% faster
performance.compilation	186.14	179.96	1.03	3.32% faster
Mean result: 1.0593314052446863

Jenkins Console Log
Blue Ocean
Commit hash: 1e79dd5a82df2796732f643eb10420d634ec4200

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

stan-buildbot · 2025-02-20T23:03:41Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.33	0.32	1.03	2.77% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.15	12.72% faster
gp_regr/gen_gp_data.stan	0.03	0.02	1.12	10.44% faster
gp_regr/gp_regr.stan	0.11	0.1	1.16	13.76% faster
sir/sir.stan	71.48	71.13	1.0	0.49% faster
irt_2pl/irt_2pl.stan	4.29	4.03	1.06	5.95% faster
eight_schools/eight_schools.stan	0.06	0.05	1.09	8.16% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.26	0.26	1.01	0.53% faster
pkpd/one_comp_mm_elim_abs.stan	20.16	18.85	1.07	6.52% faster
garch/garch.stan	0.45	0.4	1.11	10.31% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.81	2.6	1.08	7.79% faster
arK/arK.stan	1.8	1.7	1.06	5.69% faster
gp_pois_regr/gp_pois_regr.stan	2.86	2.72	1.05	4.93% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	9.18	8.38	1.1	8.78% faster
performance.compilation	184.03	187.1	0.98	-1.67% slower
Mean result: 1.071597581146215

Jenkins Console Log
Blue Ocean
Commit hash: 1e79dd5a82df2796732f643eb10420d634ec4200

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

SteveBronder added 2 commits December 12, 2024 17:56

Pathfinder lazily evaluates and writes parameters instead of taking e…

87be301

…ager and large chunks of memory

add docs

3c932f0

WardBrian requested changes Dec 13, 2024

View reviewed changes

src/stan/callbacks/unique_stream_writer.hpp Outdated Show resolved Hide resolved

SteveBronder and others added 4 commits December 13, 2024 15:32

remove stringstream from unique stream writer

f1a8e56

update laplace

48bbb6e

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

0b46b4a

remove newline

c644df0

SteveBronder and others added 14 commits December 17, 2024 18:11

fixing pathfinder return results

814f8df

adds concurrent queue so writes are thread safe to one file

e484fa1

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

e88ca19

fix docs for multi_stream_writer

6b24bbf

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

0e48078

fix docs for multi_stream_writer

298761e

fix docs for multi_stream_writer

ae3740c

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

b98f60f

update normal_glm test for better inits

a18d7e9

Merge remote-tracking branch 'refs/remotes/origin/feature/reduce-mem-…

9250e76

…pathfinder' into feature/reduce-mem-pathfinder

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

4a058bc

update headers

09d66e5

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

c25ec46

fix stringstream in unique stream vector writer

2edfd85

SteveBronder and others added 3 commits January 7, 2025 18:00

fix writes for single pathfinder when doing multi pathfinder

6814bd1

Merge commit '108daa93b6dd0d7b351864494bdcd5a255f6a765' into HEAD

e1c59da

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

6326a66

update time string for single path in multi path

79e3a9d

update stream writer is_nonnull and the writes for multi

a012b88

SteveBronder and others added 2 commits February 12, 2025 16:30

make concurrent writer producers block if the queues are almost full

23be4cb

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

456f290

make sure not producer threads are waiting before joining

7bd2826

SteveBronder and others added 5 commits February 13, 2025 14:56

update to fix bug where if PSIS time was 0 then it would not be printed

069d0e5

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

61a85df

fix printout for best elbo during single pathfinder

6e4d184

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

18ff476

fix printout for best elbo during single pathfinder

ce4d346

Update name of pathfinder column to path__

727e359

WardBrian reviewed Feb 18, 2025

View reviewed changes

src/stan/callbacks/concurrent_writer.hpp Outdated Show resolved Hide resolved

src/stan/callbacks/concurrent_writer.hpp Outdated Show resolved Hide resolved

SteveBronder and others added 4 commits February 18, 2025 16:46

slim down the concurrent writer so it just writes samples

3b35ecf

Merge commit '30098c05bf4fcfa3a82cee48ae9751c4dc4aa0e9' into HEAD

a60f816

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

d06fe3b

update tests for path number starting from 1 instead of 0

45712ce

WardBrian reviewed Feb 19, 2025

View reviewed changes

update with review comments

181f965

SteveBronder and others added 2 commits February 20, 2025 15:48

update docs

7dc38a8

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

52ab8a3

WardBrian approved these changes Feb 20, 2025

View reviewed changes

SteveBronder merged commit 86fb358 into develop Feb 21, 2025
3 checks passed

WardBrian deleted the feature/reduce-mem-pathfinder branch February 21, 2025 16:07

jgabry mentioned this pull request Mar 4, 2025

Remove deprecated arguments and functions stan-dev/cmdstanr#1061

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/reduce mem pathfinder #3325

Feature/reduce mem pathfinder #3325

SteveBronder commented Dec 13, 2024

stan-buildbot commented Dec 13, 2024

stan-buildbot commented Jan 6, 2025

stan-buildbot commented Jan 8, 2025

stan-buildbot commented Jan 8, 2025

SteveBronder commented Feb 12, 2025

SteveBronder commented Feb 12, 2025 •

edited

Loading

SteveBronder commented Feb 13, 2025 •

edited

Loading

stan-buildbot commented Feb 13, 2025

stan-buildbot commented Feb 14, 2025

stan-buildbot commented Feb 19, 2025

WardBrian left a comment

WardBrian Feb 19, 2025

stan-buildbot commented Feb 20, 2025

stan-buildbot commented Feb 20, 2025

Feature/reduce mem pathfinder #3325

Feature/reduce mem pathfinder #3325

Conversation

SteveBronder commented Dec 13, 2024

Submission Checklist

Summary

Intended Effect

How to Verify

Side Effects

Documentation

Copyright and Licensing

stan-buildbot commented Dec 13, 2024

stan-buildbot commented Jan 6, 2025

stan-buildbot commented Jan 8, 2025

stan-buildbot commented Jan 8, 2025

SteveBronder commented Feb 12, 2025

SteveBronder commented Feb 12, 2025 • edited Loading

SteveBronder commented Feb 13, 2025 • edited Loading

stan-buildbot commented Feb 13, 2025

stan-buildbot commented Feb 14, 2025

stan-buildbot commented Feb 19, 2025

WardBrian left a comment

Choose a reason for hiding this comment

WardBrian Feb 19, 2025

Choose a reason for hiding this comment

stan-buildbot commented Feb 20, 2025

stan-buildbot commented Feb 20, 2025

SteveBronder commented Feb 12, 2025 •

edited

Loading

SteveBronder commented Feb 13, 2025 •

edited

Loading