[help] I am puzzled by why I'm getting an out of date target on one machine but not another

[kendonB]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

No reprex unfortunately as this project has proprietary data.

The target code is:

  tar_qs(parcels_with_buildings_all, {
    sf::st_as_sf(as_tibble(data.table::rbindlist(parcels_with_buildings |> map(~ {
      .x |>
        select(-building_geometry) |>
        filter(!st_is_empty(geometry)) |>
        mutate(geometry = geometry |> st_cast("POLYGON"))
    })))) |>
      select(parcel_id = id, building_location) |>
      filter(!st_is_empty(building_location))
  }),

• parcels_with_buildings is a 1000-length dynamic target.
• both parcels_with_buildings and parcels_with_buildings_all were built on machine A (the cluster computer).
• code is the same on both machines.
• I wiped out _targets on machine B (my laptop), used rsync -rcvP ... (which compares hashes) to bring the machine A _targets on to machine B. I ran this several times to give it a few chances to sync everything.
• I run tar_make() on machine B and parcels_with_buildings_all is out of date, even though it was built successfully on machine A.
• I checked tar_meta("parcels_with_buildings_all") on both and they were the same, even after remaking on machine A. I am now making parcels_with_buildings_all on machine B to see if tar_meta("parcels_with_buildings_all") changes there.

Any other troubleshooting I should do?

[kendonB]

OK, curious - I remade "parcels_with_buildings_all", got different data and depends hashes, which makes sense, but running digest::digest(targets::tar_read(parcels_with_buildings_all)) in a fresh session on each machine gives the same hash.

[kendonB]

May actually be a bug:
Machine A:

 > digest::digest(targets::tar_read(parcels_with_buildings_all))
[1] "ad46ede5868f4514063bc6d5ff952406"
>  targets:::digest_obj64(targets::tar_read(parcels_with_buildings_all))
[1] "3b6259b8f0c4ea24"

Machine B:

r$> digest::digest(targets::tar_read(parcels_with_buildings_all))
[1] "ad46ede5868f4514063bc6d5ff952406"

r$> targets:::digest_obj64(targets::tar_read(parcels_with_buildings_all))
[1] "a82d6670cb4cd3d4"

[wlandau]

Hard to say what's happening. It could be something strange about sf objects/attributes and how they are serialized. If you can share an object that reproduces #1108 (comment), that would help.

Otherwise, if it's a different problem with the pipeline, it would help to isolate the problem more and peel back layers, i.e. with fewer branches and public data. It's also worth checking the versions of packages across machines, especially sf and digest. Unfortunately there's little I can do without a reprex.

[kendonB]

reprex!
On machine A:

>  targets:::digest_obj64(mtcars)
[1] "04304fa81fcd8794"

On machine B:

r$> targets:::digest_obj64(mtcars)
[1] "4bcc78b833d43362"

[wlandau]

Thanks!

I only get "04304fa81fcd8794", so I'm not sure what might be happening, but now it is possible to dive deeper. targets:::digest_obj64() just calls targets:::vdigest64(), which is is generated from digest::getVDigest():

targets/R/utils_digest.R

Lines 31 to 33 in 3fa768c

	digest_obj64 <- function(object, ...) {
	vdigest64(list(object), serialize = TRUE, serializeVersion = 3L, ...)
	}

targets/R/utils_digest.R

Line 3 in 3fa768c

vdigest64 <- digest::getVDigest(algo = "xxhash64")

Does that help you track this down? Also, are the versions of R and digest the same on both machines?

[kendonB]

So it's something to do with serialising being different across the systems?

[kendonB]

How bizarre - it's due to the use of radian on machine B:

r$> callr::r(function() digest::getVDigest(algo = "xxhash64")(list(mtcars), serialize = TRUE, serializeVersion = 3L))
[1] "04304fa81fcd8794"

r$> digest::getVDigest(algo = "xxhash64")(list(mtcars), serialize = TRUE, serializeVersion = 3L)
[1] "4bcc78b833d43362"

Using Rterm:

> digest::getVDigest(algo = "xxhash64")(list(mtcars), serialize = TRUE, serializeVersion = 3L)
[1] "04304fa81fcd8794"

I'll report it as a bug for radian. I guess if you want to address this in targets you could use callr within your digest functions - I presume that's going to add unacceptable overhead. Perhaps a higher level part of tar_make() could be executed using callr. However, it seems like there's an issue with how serialisation is done in radian.

[kendonB]

FYI randy3k/radian#430

[kendonB]

So revisiting this with my real data, I'm still seeing parcels_with_buildings_all being out of date after after wiping _targets, and re-rsyncing _targets.

On machine A:

> targets::tar_meta("parcels_with_buildings_all")[,c("data", "depend")]
# A tibble: 1 × 2
  data             depend
  <chr>            <chr>
1 a83be1f39d360069 40c57895651e2c66

> digest::getVDigest(algo = "xxhash64")(targets::tar_read(parcels_with_buildings_all), serialize = TRUE, serializeVersion = 3L)
[1] "03a034340d94ec9c" "17adf0783dcac8fc" "460f0dfb5519e956"

On machine B after wiping _targets and remaking parcels_with_buildings_all:

> targets::tar_meta("parcels_with_buildings_all")[,c("data", "depend")]
# A tibble: 1 × 2
  data             depend
  <chr>            <chr>
1 e70b90eaa2b09f04 849062eddfb124ae

> digest::getVDigest(algo = "xxhash64")(targets::tar_read(parcels_with_buildings_all))
[1] "03a034340d94ec9c" "17adf0783dcac8fc" "460f0dfb5519e956"

data being different may well be due to different underlying GEOS versions, that's fine and expected after remaking - but depends should just be calculated from the data field in tar_meta of the dependencies right?

In my case, it's only one target and the data field of tar_meta is the same in both. Any further troubleshooting steps?

On both machines:

> targets::tar_meta("parcels_with_buildings")[,c("data", "depend")]
# A tibble: 1 × 2
  data             depend
  <chr>            <chr>
1 9a82d04eb709a474 NA

[wlandau]

tar_sitrep() and tar_visnetwork() can sometimes give you a better picture of the invalidation pattern and point you to the most upstream part of the invalidated subgraph.

If you still see different hashes on different machines even without radian, then the general approach in #1108 (reply in thread) still applies: see how much you can isolate the problem: fewer branches, public data in a full reprex, align versions of packages like digest, etc.

[kendonB]

I believe this is minimal for a reprex:

Use this _targets.R:

library(targets)
options(tidyverse.quiet = TRUE)
tar_option_set(packages = c(
  "tidyverse"
))

list(
  tar_target(test, {
    data.table::rbindlist(list(mtcars) |> map(~ {
      .x
    }))
  })
)

Make on linux first:
_targets_linux.zip

Remaking on windows puts the target out of date:
_targets_after_remaking_on_windows.zip

> targets::tar_sitrep()
# A tibble: 1 × 11
  name  record always never command depend format repository iteration file
  <chr> <lgl>  <lgl>  <lgl> <lgl>   <lgl>  <lgl>  <lgl>      <lgl>     <lgl>
1 test  FALSE  FALSE  FALSE FALSE   TRUE   FALSE  FALSE      FALSE     FALSE

[wlandau]

Thanks, the reprex helps.

This one is tricky to debug, but I tracked it down to the way dependency hashes are computed:

targets/R/class_meta.R

Lines 70 to 88 in e47d4b8

	hash_dep = function(name, pipeline) {
	exists <- self$exists_record(name) &&
	pipeline_exists_object(pipeline, name)
	if_any(
	exists,
	self$get_record(name)$data,
	""
	)
	},
	hash_deps = function(deps, pipeline) {
	hashes <- vapply(
	X = sort.int(deps),
	FUN = self$hash_dep,
	pipeline = pipeline,
	FUN.VALUE = character(1L)
	)
	string <- paste(c(names(hashes), hashes), collapse = "")
	digest_chr64(string)
	},

In hash_deps(), the deps argument is a character vector of all potential dependencies detected with static code analysis. hash_dep() goes through each one of these, looks up a hash if it exists, and returns the empty string "" if the object was not hashed. Objects not hashed include non-global symbols in package environments from purrr to base. On Windows, sort.int(deps) is:

#> sort.int(deps)
[1] ".x"     "::"     "list"   "map"    "mtcars" "{" 

whereas on Mac and Linux, it is

#> sort.int(deps)
[1] "::"     ".x"     "list"   "map"    "mtcars" "{" 

I plan to implement a two-part patch:

1. Set Sys.setlocale(category = "LC_COLLATE", locale = "C") temporarily for each sort.
2. Remove superfluous symbols like "::" and ".x" from the final hash calculation in order to further decouple the command hash from the depend hash.

We will have to see if (1) has unintended consequences, and (2) will cause most existing targets to rerun, but I still believe this is the right call.

[wlandau]

Fixed as of 5e2df18.

[kendonB]

Speedy work as always @wlandau and much appreciated! I will give it a try now.

[kendonB]

The only caveat is that I presume this will invalidate any pipelines with different locales and including '::' and '.x' - still worth it IMO as targets is still relatively new.

[kendonB]

Can confirm this is a fix for my original problem. thanks heaps!

[kendonB]

So what does it think it "depends" on?

[kendonB]

If it helps, I believe the data.table::rbindlist, the map, and the .x are all required for the reprex.