Various minor updates (duplicate of broken #1149) #1151

rykelly · 2016-10-28T20:23:49Z

Description

get.trait.data:
- forceupdate argument defaults to FALSE
- pft ouput dir is created if it doesn't exist (even though typically check.settings will have done that)
Everywhere db.open is used, I immediately schedule closing it via on.exit. This ensures the connection is closed even if the function errors out.
- Where db.connection was wrapped in try(), check now for a result of class 'try-error' rather than 'character'
Added Western_* pfts to pftmapping.csv
get.parameter.samples moved to utils package, which is the main place it's used
Inadvertent but presumably correct changes to modules/uncertainty by re-roxygenizing
Cleanup, add overwrite args to do.conversions
In some but not all places, remove reliance on NOW() for finding just-added BETY entries.
- For one thing, per Remove / deprecate use of now() from SQL insert statements #1083 we shouldn't be setting the updated/created_at timestamps in BETY manually.
- For another, if multiple workflows are running it's entirely possible to get duplicate timestamps (at least down to the second, which is what we were using.
- Using the psql RETURNING keyword is cleaner and safer.
- I'd doing this everywhere in the code we're currently relying on the set/query "unique" timestamp design, but again, have currently only done it for the ones that were causing me problems.
Updated get.analysis.filenames to put EA/SA data files into their own directories. It gets messy otherwise when doing a MultiSettings run with hundreds of sites...

Motivation and Context

Fixing / cleaning things as they come up while setting up my multisite runs

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

- Default to not forcing trait data update - Create PFT directory if not already (currently done by check.settings too, but no need for get.trait.data to rely on that)

Plus re-roxygenize

Prevents too many open connections. on.exit runs when the function exits, even on error.

This was causing bugs for me when running multiple workflows at once (not too hard to get two with the same "now" timestamp), and it also addresses PecanProject#1083

…inor_updates

mdietze · 2016-10-29T05:44:31Z

utils/R/do.conversions.R

-                                                     browndog = settings$browndog)
-        settings$run$inputs[[i]][["path"]] <- result
-        status.end()
+      if (is.null(input$path)) {


Not clear why status.check was removed from this if statement

Accident. Reverted.

mdietze · 2016-10-29T05:47:20Z

utils/R/get.analysis.filenames.r

@@ -23,7 +23,7 @@ ensemble.filename <- function(settings, prefix = "ensemble.samples", suffix = "R
    ensemble.id <- "NOENSEMBLEID"
  }

-  ensemble.dir <- settings$outdir
+  ensemble.dir <- file.path(settings$outdir, 'ensemble.samples')


This appears to put the ensemble samples in their own directory rather than in the base outdir. This will make them unaccessible via the web interface. Not sure the reason for this change

Hm. Well, the reason for this change is that it's obnoxious to have hundreds of *.samples.Rdata files in the root workflow dir if you're doing a big Multi run. But I reverted anyway, since obviously breaking the web interface is worse.

mdietze · 2016-10-29T05:47:59Z

utils/R/get.analysis.filenames.r

  } else {
    ind <- which(sapply(settings$pfts, function(x) x$name) == pft)
    if (length(ind) == 0) {
      ## no match
      logger.warn("sensitivity.filename: unmatched PFT = ", pft, " not among ", 
                  sapply(settings$pfts, function(x) x$name))
-      sensitivity.dir <- file.path(settings$outdir, "pfts", pft)
+      sensitivity.dir <- file.path(settings$outdir, "pfts", pft, "sensitivity.samples")


same issue as above with ensemble.dir

mdietze · 2016-10-29T05:48:04Z

utils/R/get.analysis.filenames.r

@@ -76,14 +76,14 @@ sensitivity.filename <- function(settings,

  if (is.null(pft)) {
    # Goes in main output directory.
-    sensitivity.dir <- settings$outdir
+    sensitivity.dir <- file.path(settings$outdir, 'sensitivity.samples')


same issue as above with ensemble.dir

mdietze · 2016-10-29T05:48:12Z

utils/R/get.analysis.filenames.r

@@ -95,7 +95,7 @@ sensitivity.filename <- function(settings,
        ## no outdir
        settings$pfts[[ind]]$outdir <- file.path(settings$outdir, "pfts", pft)
      }
-      sensitivity.dir <- settings$pfts[[ind]]$outdir
+      sensitivity.dir <- file.path(settings$pfts[[ind]]$outdir, "sensitivity.samples")


same issue as above with ensemble.dir

Ryan Kelly added 14 commits October 27, 2016 07:13

Add Western PFTs to ed/data/pftmapping

015a716

get.trait.data updates

039a214

- Default to not forcing trait data update - Create PFT directory if not already (currently done by check.settings too, but no need for get.trait.data to rely on that)

Move get.paramneter.samples to PEcAn.utils

a777f6f

Plus re-roxygenize

Close db connections with on.exit

f4ec017

Prevents too many open connections. on.exit runs when the function exits, even on error.

Don't print so much

4019cf3

Add error count to papply warning

1dbe0ec

Fix ED pftmapping

9e917f9

Remove reliance on manually-edited DB timestamps

726cff6

This was causing bugs for me when running multiple workflows at once (not too hard to get two with the same "now" timestamp), and it also addresses PecanProject#1083

Move EA/SA files to own directories by default

9c61e3f

Minor updates do.conversions and met.process

5fee5a6

Better check for try-error

1356e5d

Typo

2adafba

Cleanup db queries to find/fix typos

6629039

Merge branch 'master' of github.com:PecanProject/pecan into various_m…

777c113

…inor_updates

mdietze requested changes Oct 29, 2016

View reviewed changes

Ryan Kelly added 3 commits October 31, 2016 07:04

Create dir before checking it...

ff1c66e

Bugfix

206ebb4

revert get.analysis.filenames.r

a29fef5

mdietze approved these changes Oct 31, 2016

View reviewed changes

mdietze merged commit 0f5451e into PecanProject:master Oct 31, 2016

rykelly deleted the various_minor_updates branch November 3, 2016 14:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various minor updates (duplicate of broken #1149) #1151

Various minor updates (duplicate of broken #1149) #1151

rykelly commented Oct 28, 2016

mdietze Oct 29, 2016

rykelly Oct 31, 2016

mdietze Oct 29, 2016

rykelly Oct 31, 2016

mdietze Oct 29, 2016

mdietze Oct 29, 2016

mdietze Oct 29, 2016

Various minor updates (duplicate of broken #1149) #1151

Various minor updates (duplicate of broken #1149) #1151

Conversation

rykelly commented Oct 28, 2016

Description

Motivation and Context

Types of changes

Checklist:

mdietze Oct 29, 2016

Choose a reason for hiding this comment

rykelly Oct 31, 2016

Choose a reason for hiding this comment

mdietze Oct 29, 2016

Choose a reason for hiding this comment

rykelly Oct 31, 2016

Choose a reason for hiding this comment

mdietze Oct 29, 2016

Choose a reason for hiding this comment

mdietze Oct 29, 2016

Choose a reason for hiding this comment

mdietze Oct 29, 2016

Choose a reason for hiding this comment