Approximate memory usage?

[ColCarroll]

Hi! Thanks for the nice library!

I'm attempting to run train and predict on the wind dataset, but running into memory problems. I am wondering if you can provide any intuition (or numbers!) on the memory usage, or tips for being able to successfully run this.

More specifically,

• I'm on a machine with ~64GB of RAM,
• the training dataset has 769,179 rows
• There are 12 locations with readings

The code I am running looks like

train <- read.csv('data/wind.train.csv')
train <- na.omit(train)
train$wind <- pmax(train$wind, 1e-4)
train$datetime <- as.POSIXct(train$datetime)
train <- add_utm_columns(train, utm_names=c("x", "y"))
train$location <- as.factor(train$location)
train$days <- as.integer(as.numeric(train$datetime - min(train$datetime), units='days'))

mesh <- make_mesh(train, c("x", "y"), cutoff=1)

# This fails with OOM
m <- sdmTMB(
    data = train,
    formula = wind ~ location + 1,
    mesh=mesh,
    family = lognormal(),
    spatial = "off",
    time = "days",
    spatiotemporal = "RW",
)

[seananderson]

Just quickly: the random walk random field will result in a random effect matrix that is much less sparse than IID random fields, and hence more memory intensive—especially with >6000 time steps. You could try IID fields although I'm still not sure how that will perform well here. I have plans in the future for a conditional autoregressive model that is likely faster and more memory efficient in cases like this.

[ColCarroll]

Thanks for the input! It looks like using IID fields also OOMs.

I'll probably use a different approach, but do let me know if anything comes up that might work for situations like these, or if there's a way to approximate the memory usage beforehand!