-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bigmemory (or similar method) for large matrix storage #28
Labels
Comments
ericbair-sciome
added
enhancement
New feature or request
version 2.0
Features that will be included in future releases
labels
Nov 12, 2023
a def v2.0 |
sciome-bot
pushed a commit
that referenced
this issue
Jan 13, 2024
Merge in STAT/prestogp from master to to-git Squashed commit of the following: commit db6cd4f Merge: 55607e2 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 12:11:56 2024 -0500 Pull request #29: Fixed some bugs caused by the linter Merge in STAT/prestogp from eb-dev to master * commit '7da2588abba4491c1ff490de95e834d00f5df88e': Fixed some bugs caused by the linter commit 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 01:37:58 2024 -0500 Fixed some bugs caused by the linter commit 55607e2 Merge: c7ed481 81b675d Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:37:55 2024 -0500 Pull request #28: Final linting fixes Merge in STAT/prestogp from build-workflow to master * commit '81b675dda9b57f2a362c34876364adf5bcf45560': Fix remaining linting issues Fix all indentation errors Add indentation_linter configuration to lintr config file commit 81b675d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:36:05 2024 -0500 Fix remaining linting issues commit 06673a4 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:23:11 2024 -0500 Fix all indentation errors commit c7ed481 Merge: 723c3eb 7bcfa1a Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:02:59 2024 -0500 Pull request #27: Lintr fixes. Added release and sanitizer actions Merge in STAT/prestogp from build-workflow to master * commit '7bcfa1aca78cc6b994cc0227c083357c58d50ca0': (21 commits) Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter WIP: Fix linter warnings for 1:nrow, 1:ncol and 1:length. Increase line length to 160 chars Don't build vignettes during the release action Move release action file to correct directory Add release action Reorder imports in RcppExports file Add missing dependency in Namespace/Description Remove unneeded exports from NAMESPACE Add missing comma in Imports section of DESCRIPTION Add missing comma in imports section of DESCRIPTION Rerun auto-formatter WIP - linting Add ignore rules for .lintr and .github for R build remove linter options from lint action as we have added .lintr project file. Fix all vector_logic_linter warnings Remove line length linter to see the remaining errors/warnings workaround for lintr bug: REditorSupport/languageserver#89 Run auto-lint on vscode enable linting on build-workflow Add missing imports in DESCRIPTION ... commit 29d82cd Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:40:17 2024 -0500 Add indentation_linter configuration to lintr config file commit 7bcfa1a Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:16:01 2024 -0500 Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file commit b6df473 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 17:54:50 2024 -0500 More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter commit 6f6f91d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 17:32:12 2024 -0500 WIP: Fix linter warnings for 1:nrow, 1:ncol and 1:length. Increase line length to 160 chars commit 2db2e19 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:48:59 2024 -0500 Don't build vignettes during the release action commit c539223 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:39:52 2024 -0500 Move release action file to correct directory commit f83df2b Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:36:03 2024 -0500 Add release action commit c84e159 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:04:36 2024 -0500 Reorder imports in RcppExports file commit 0460202 Merge: 245eaa3 43ae272 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 15:48:30 2024 -0500 Merge branch 'build-workflow' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP into build-workflow commit 245eaa3 Merge: 5355e24 c060815 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 15:47:42 2024 -0500 Merge branch 'main' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP into build-workflow commit c060815 Merge: 731da76 6d69b69 Author: {SET}group <127860447+Spatiotemporal-Exposures-and-Toxicology@users.noreply.github.com> Date: Thu Jan 11 16:02:52 2024 -0500 Merge pull request #44 from Spatiotemporal-Exposures-and-Toxicology/main-sciome Sciome Update 1/10/2024 commit 6d69b69 Author: sciome-bot <software.tools@sciome.com> Date: Thu Jan 11 14:39:39 2024 -0500 Add missing dependency in Namespace/Description commit bdf52c6 Author: sciome-bot <software.tools@sciome.com> Date: Thu Jan 11 14:30:38 2024 -0500 Remove unneeded exports from NAMESPACE commit 06918b0 Author: sciome-bot <software.tools@sciome.com> Date: Thu Jan 11 14:15:45 2024 -0500 Add missing comma in Imports section of DESCRIPTION ... and 13 more commits
sciome-bot
pushed a commit
that referenced
this issue
Jan 14, 2024
Merge in STAT/prestogp from master to to-git Squashed commit of the following: commit 439790c Merge: f4af0d1 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sun Jan 14 10:58:49 2024 -0500 Pull request #32: Fixed some testthat issues Merge in STAT/prestogp from eb-dev to master * commit '56f881f96a0e55e60bca3c8a840ec31291665594': Fixed some testthat issues commit 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 14:05:12 2024 -0500 Fixed some testthat issues commit f4af0d1 Merge: db6cd4f ee193de Author: sciome-bot <software.tools@sciome.com> Date: Sat Jan 13 12:37:28 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit db6cd4f Merge: 55607e2 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 12:11:56 2024 -0500 Pull request #29: Fixed some bugs caused by the linter Merge in STAT/prestogp from eb-dev to master * commit '7da2588abba4491c1ff490de95e834d00f5df88e': Fixed some bugs caused by the linter commit 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 01:37:58 2024 -0500 Fixed some bugs caused by the linter commit 55607e2 Merge: c7ed481 81b675d Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:37:55 2024 -0500 Pull request #28: Final linting fixes Merge in STAT/prestogp from build-workflow to master * commit '81b675dda9b57f2a362c34876364adf5bcf45560': Fix remaining linting issues Fix all indentation errors Add indentation_linter configuration to lintr config file commit 81b675d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:36:05 2024 -0500 Fix remaining linting issues commit 06673a4 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:23:11 2024 -0500 Fix all indentation errors commit c7ed481 Merge: 723c3eb 7bcfa1a Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:02:59 2024 -0500 Pull request #27: Lintr fixes. Added release and sanitizer actions Merge in STAT/prestogp from build-workflow to master * commit '7bcfa1aca78cc6b994cc0227c083357c58d50ca0': (21 commits) Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter WIP: Fix linter warnings for 1:nrow, 1:ncol and 1:length. Increase line length to 160 chars Don't build vignettes during the release action Move release action file to correct directory Add release action Reorder imports in RcppExports file Add missing dependency in Namespace/Description Remove unneeded exports from NAMESPACE Add missing comma in Imports section of DESCRIPTION Add missing comma in imports section of DESCRIPTION Rerun auto-formatter WIP - linting Add ignore rules for .lintr and .github for R build remove linter options from lint action as we have added .lintr project file. Fix all vector_logic_linter warnings Remove line length linter to see the remaining errors/warnings workaround for lintr bug: REditorSupport/languageserver#89 Run auto-lint on vscode enable linting on build-workflow Add missing imports in DESCRIPTION ... commit 29d82cd Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:40:17 2024 -0500 Add indentation_linter configuration to lintr config file commit 7bcfa1a Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:16:01 2024 -0500 Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file commit b6df473 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 17:54:50 2024 -0500 More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter commit 6f6f91d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 17:32:12 2024 -0500 WIP: Fix linter warnings for 1:nrow, 1:ncol and 1:length. Increase line length to 160 chars commit 2db2e19 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:48:59 2024 -0500 Don't build vignettes during the release action commit c539223 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:39:52 2024 -0500 Move release action file to correct directory commit f83df2b Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:36:03 2024 -0500 Add release action commit c84e159 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 16:04:36 2024 -0500 Reorder imports in RcppExports file commit 0460202 Merge: 245eaa3 43ae272 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 15:48:30 2024 -0500 Merge branch 'build-workflow' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP into build-workflow commit 245eaa3 Merge: 5355e24 c060815 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 15:47:42 2024 -0500 Merge branch 'main' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP into build-workflow commit c060815 Merge: 731da76 6d69b69 Author: {SET}group <127860447+Spatiotemporal-Exposures-and-Toxicology@users.noreply.github.com> Date: Thu Jan 11 16:02:52 2024 -0500 Merge pull request #44 from Spatiotemporal-Exposures-and-Toxicology/main-sciome Sciome Update 1/10/2024 ... and 13 more commits
sciome-bot
pushed a commit
that referenced
this issue
Feb 8, 2024
…Fixed R CMD issues Merge in STAT/prestogp from master to to-git Squashed commit of the following: commit 7e330f3 Merge: 13a6f44 f49d076 Author: sciome-bot <software.tools@sciome.com> Date: Wed Feb 7 23:25:05 2024 -0500 Merge branch 'master' of ssh://sciome-bot/stat/prestogp commit f49d076 Merge: 439790c 59baf72 Author: Eric Bair <eric.bair@sciome.com> Date: Wed Feb 7 23:19:54 2024 -0500 Pull request #35: Eb dev Merge in STAT/prestogp from eb-dev to master * commit '59baf720737abe59a04c98705c950ef1c7b63321': Fixed some documentation issues Removed C++11 compiler flag Added RandomFields data sets to PrestoGP Added knitr to Suggests in DESCRIPTION Documentation, R-cmd-check fixes, binomial regression commit 59baf72 Author: Eric Bair <eric.bair@sciome.com> Date: Tue Feb 6 17:49:48 2024 -0500 Fixed some documentation issues commit d857d8b Author: Eric Bair <eric.bair@sciome.com> Date: Thu Feb 1 02:16:44 2024 -0500 Removed C++11 compiler flag commit 662907d Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 15:36:37 2024 -0500 Added RandomFields data sets to PrestoGP commit cf2aadb Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 13:40:24 2024 -0500 Added knitr to Suggests in DESCRIPTION commit ef5f2c9 Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 01:31:53 2024 -0500 Documentation, R-cmd-check fixes, binomial regression commit 13a6f44 Merge: 439790c 8ffb20b Author: sciome-bot <software.tools@sciome.com> Date: Sun Jan 14 11:46:17 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit 439790c Merge: f4af0d1 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sun Jan 14 10:58:49 2024 -0500 Pull request #32: Fixed some testthat issues Merge in STAT/prestogp from eb-dev to master * commit '56f881f96a0e55e60bca3c8a840ec31291665594': Fixed some testthat issues commit 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 14:05:12 2024 -0500 Fixed some testthat issues commit f4af0d1 Merge: db6cd4f ee193de Author: sciome-bot <software.tools@sciome.com> Date: Sat Jan 13 12:37:28 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit db6cd4f Merge: 55607e2 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 12:11:56 2024 -0500 Pull request #29: Fixed some bugs caused by the linter Merge in STAT/prestogp from eb-dev to master * commit '7da2588abba4491c1ff490de95e834d00f5df88e': Fixed some bugs caused by the linter commit 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 01:37:58 2024 -0500 Fixed some bugs caused by the linter commit 55607e2 Merge: c7ed481 81b675d Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:37:55 2024 -0500 Pull request #28: Final linting fixes Merge in STAT/prestogp from build-workflow to master * commit '81b675dda9b57f2a362c34876364adf5bcf45560': Fix remaining linting issues Fix all indentation errors Add indentation_linter configuration to lintr config file commit 81b675d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:36:05 2024 -0500 Fix remaining linting issues commit 06673a4 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:23:11 2024 -0500 Fix all indentation errors commit c7ed481 Merge: 723c3eb 7bcfa1a Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:02:59 2024 -0500 Pull request #27: Lintr fixes. Added release and sanitizer actions Merge in STAT/prestogp from build-workflow to master * commit '7bcfa1aca78cc6b994cc0227c083357c58d50ca0': (21 commits) Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter WIP: Fix linter warnings for 1:nrow, 1:ncol and 1:length. Increase line length to 160 chars Don't build vignettes during the release action Move release action file to correct directory Add release action Reorder imports in RcppExports file Add missing dependency in Namespace/Description Remove unneeded exports from NAMESPACE Add missing comma in Imports section of DESCRIPTION Add missing comma in imports section of DESCRIPTION Rerun auto-formatter WIP - linting Add ignore rules for .lintr and .github for R build remove linter options from lint action as we have added .lintr project file. Fix all vector_logic_linter warnings Remove line length linter to see the remaining errors/warnings workaround for lintr bug: REditorSupport/languageserver#89 Run auto-lint on vscode enable linting on build-workflow Add missing imports in DESCRIPTION ... commit 29d82cd Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:40:17 2024 -0500 Add indentation_linter configuration to lintr config file commit 7bcfa1a Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 18:16:01 2024 -0500 Add R_LINTR_LINTER_FILE to lint action to point to global .lintr file commit b6df473 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 17:54:50 2024 -0500 More lintr fixes for 1:length, 1:nrow and line length. Disabled object_length_linter ... and 21 more commits
sciome-bot
added a commit
that referenced
this issue
Feb 8, 2024
Merge in STAT/prestogp from master to to-git Squashed commit of the following: commit bab8357 Merge: 55c0e8e 5394772 Author: Eric Bair <eric.bair@sciome.com> Date: Thu Feb 8 14:19:24 2024 -0500 Pull request #38: Removed vignette, fixed lint issues Merge in STAT/prestogp from eb-dev to master * commit '53947729624b47fa1920ef39bd23d86f6e5cd7b7': Removed vignette, fixed lint issues commit 55c0e8e Merge: 7e330f3 076c0a1 Author: sciome-bot <software.tools@sciome.com> Date: Thu Feb 8 14:18:48 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit 5394772 Merge: b431df4 7e330f3 Author: Eric Bair <eric.bair@sciome.com> Date: Thu Feb 8 14:07:54 2024 -0500 Resolved merge conflict commit b431df4 Author: Eric Bair <eric.bair@sciome.com> Date: Thu Feb 8 13:56:38 2024 -0500 Removed vignette, fixed lint issues commit 7e330f3 Merge: 13a6f44 f49d076 Author: sciome-bot <software.tools@sciome.com> Date: Wed Feb 7 23:25:05 2024 -0500 Merge branch 'master' of ssh://sciome-bot/stat/prestogp commit f49d076 Merge: 439790c 59baf72 Author: Eric Bair <eric.bair@sciome.com> Date: Wed Feb 7 23:19:54 2024 -0500 Pull request #35: Eb dev Merge in STAT/prestogp from eb-dev to master * commit '59baf720737abe59a04c98705c950ef1c7b63321': Fixed some documentation issues Removed C++11 compiler flag Added RandomFields data sets to PrestoGP Added knitr to Suggests in DESCRIPTION Documentation, R-cmd-check fixes, binomial regression commit 59baf72 Author: Eric Bair <eric.bair@sciome.com> Date: Tue Feb 6 17:49:48 2024 -0500 Fixed some documentation issues commit d857d8b Author: Eric Bair <eric.bair@sciome.com> Date: Thu Feb 1 02:16:44 2024 -0500 Removed C++11 compiler flag commit 662907d Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 15:36:37 2024 -0500 Added RandomFields data sets to PrestoGP commit cf2aadb Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 13:40:24 2024 -0500 Added knitr to Suggests in DESCRIPTION commit ef5f2c9 Author: Eric Bair <eric.bair@sciome.com> Date: Wed Jan 31 01:31:53 2024 -0500 Documentation, R-cmd-check fixes, binomial regression commit 13a6f44 Merge: 439790c 8ffb20b Author: sciome-bot <software.tools@sciome.com> Date: Sun Jan 14 11:46:17 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit 439790c Merge: f4af0d1 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sun Jan 14 10:58:49 2024 -0500 Pull request #32: Fixed some testthat issues Merge in STAT/prestogp from eb-dev to master * commit '56f881f96a0e55e60bca3c8a840ec31291665594': Fixed some testthat issues commit 56f881f Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 14:05:12 2024 -0500 Fixed some testthat issues commit f4af0d1 Merge: db6cd4f ee193de Author: sciome-bot <software.tools@sciome.com> Date: Sat Jan 13 12:37:28 2024 -0500 Merge branch 'main-sciome' of sciome-bot-git:Spatiotemporal-Exposures-and-Toxicology/PrestoGP commit db6cd4f Merge: 55607e2 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 12:11:56 2024 -0500 Pull request #29: Fixed some bugs caused by the linter Merge in STAT/prestogp from eb-dev to master * commit '7da2588abba4491c1ff490de95e834d00f5df88e': Fixed some bugs caused by the linter commit 7da2588 Author: Eric Bair <eric.bair@sciome.com> Date: Sat Jan 13 01:37:58 2024 -0500 Fixed some bugs caused by the linter commit 55607e2 Merge: c7ed481 81b675d Author: Shail Choksi <shail.choksi@sciome.com> Date: Fri Jan 12 19:37:55 2024 -0500 Pull request #28: Final linting fixes Merge in STAT/prestogp from build-workflow to master * commit '81b675dda9b57f2a362c34876364adf5bcf45560': Fix remaining linting issues Fix all indentation errors Add indentation_linter configuration to lintr config file commit 81b675d Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:36:05 2024 -0500 Fix remaining linting issues commit 06673a4 Author: sciome-bot <software.tools@sciome.com> Date: Fri Jan 12 19:23:11 2024 -0500 Fix all indentation errors ... and 25 more commits
@ericbair-sciome Another option to consider for 2.0 is Apache-Arrow. Fast and efficient read/write combined with adapting how the Vecchia neighborhood is stored and updated could help. |
We'll want to analyze areas in PrestoGP where memory is built up, opportunities where off-loading memory in exchange for I/O could be used. |
sciome-bot
pushed a commit
that referenced
this issue
Dec 29, 2024
Merge in STAT/prestogp from build-workflow to master * commit '81b675dda9b57f2a362c34876364adf5bcf45560': Fix remaining linting issues Fix all indentation errors Add indentation_linter configuration to lintr config file
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Utilizing the correlation based distance can require storage of the full covariance matrix. We will explore ways to ensure it is scalable for n > 60K.
https://fbertran.github.io/bigalgebra/
The text was updated successfully, but these errors were encountered: