Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up docs #142

Merged
merged 1 commit into from
Dec 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions docs/src/datahandling.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# `DataHandling`
# [`DataHandling`](@id datahandling_module)

The `DataHandling` module is responsible for reading data from files and
resampling it onto the simulation grid.
Expand All @@ -11,7 +11,7 @@ This is no trivial task. Among the challenges:

The `DataHandling` takes the divide-and-conquer approach: the various core tasks
and features and split into other independent modules (chiefly
[`FileReaders`](@ref), and [`Regridders`](@ref)). Such modules can be developed,
[`FileReaders`](@ref file_reader_module), and [`Regridders`](@ref regridder_module) regridder_module). Such modules can be developed,
tested, and extended independently (as long as they maintain a consistent
interface). For instance, if need arises, the `DataHandler` can be used (almost)
directly to process files with a different format from NetCDF.
Expand Down Expand Up @@ -47,7 +47,7 @@ the `NCFileReader`.

> This extension is loaded when loading `ClimaCore` and `NCDatasets` are loaded.
> In addition to this, a `Regridder` is needed (which might require importing
> additional packages) - see [`Regridders`](@ref) for more information.
> additional packages) - see [`Regridders`](@ref regridder_module) for more information.

It is possible to pass down keyword arguments to underlying constructors in
`DataHandler` with the `regridder_kwargs` and `file_reader_kwargs`. These have
Expand All @@ -65,28 +65,28 @@ Note that, if a non-identity pre-processing function is provided as part of
`file_reader_kwargs`, it will be applied to each input variable before they
are composed.
Composing multiple input variables is currently only supported with the
`InterpolationsRegridder`, not with `TempestRegridder`.
[`InterpolationsRegridder`](@ref interp_regridder), not with [`TempestRegridder`](@ref tempest_regridder).

Sometimes, the time development of a variable is split across multiple NetCDF
files. `DataHandler` knows how to combine them and treat multiple files as if
they were a single one. To use this feature, just pass the list of NetCDF files
(while the file don't have to be sorted, it is good practice to pass them sorted
in ascending order by time).
in ascending order by time).

#### Heuristics to do-what-you-mean

`DataHandler` tries to interpret the files provided and identify if they are
split across variables or along the time dimension. The heuristics implement are
the following:
- When just a file is passed, it is assumed that it contains everything
- When just a file is passed, it is assumed that it contains everything
- When multiple files are passed, `DataHandler` will assume that the files are
split along variables if the number of files is the same the number of
variables, otherwise, it will assume that each file contains all the variables
for a portion of the total time.
- When the above assumption is incorrect, you can pass a list of list of files
that fully specifies variables and times.

For example,
For example,
```julia
data_handler = DataHandling.DataHandler(
["era1980.nc", "era1981.nc"],
Expand All @@ -98,9 +98,9 @@ data_handler = DataHandling.DataHandler(

In this case, `DataHandler` will incorrectly assume that `lai_hv` is contained
in `era1980.nc`, and `lai_lv` is in `era1980.nc`. Instead, construct the
`data_handler` by passing a list of lists
`data_handler` by passing a list of lists
```julia
files = ["era1980.nc", "era1981.nc"]
files = ["era1980.nc", "era1981.nc"]
data_handler = DataHandling.DataHandler(
[files, files],
["lai_hv", "lai_lv"],
Expand Down Expand Up @@ -163,7 +163,7 @@ file path), which knows how to combine them.
Suppose that the input NetCDF file `era5_example.nc` contains two variables `u`
and `v`, and we care about their sum `u + v` but not their individual values.
We can provide a pointwise composing function to perform the sum, along with
the `InterpolationsRegridder` to produce the data we want, `u + v`.
the [`InterpolationsRegridder`](@ref interp_regridder) to produce the data we want, `u + v`.
The `preprocess_func` passed in `file_reader_kwargs` will be applied to `u`
and to `v` individually, before the composing function is applied. The regridding
is applied after the composing function. `u` and `v` could also come from separate
Expand Down
6 changes: 3 additions & 3 deletions docs/src/faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

## Is it possible to preprocess the data in `TimeVaryingInput` or `SpaceVaryingInput`, for instance, to remove NaNs or change units?

Yes, [`TimeVaryingInput`](@ref) and [`SpaceVaryingInput`](@ref) support this
feature. [`TimeVaryingInput`](@ref) and [`SpaceVaryingInput`](@ref) that read
NetCDF files use [`NCFileReader`](@ref) under the hood. `NCFileReader`s can be
Yes, [`TimeVaryingInput`](@ref timevaryinginput) and [`SpaceVaryingInput`](@ref spacevaryinginput) support this
feature. [`TimeVaryingInput`](@ref timevaryinginput) and [`SpaceVaryingInput`](@ref spacevaryinginput) that read
NetCDF files use [`NCFileReader`](@ref ncfilereaders) under the hood. `NCFileReader`s can be
constructed with an optional keyword argument `preprocess_func`, a pointwise
function that transforms the data read into something else. `Input`s can be
constructed to pass down this keyword argument. Let us have a look at an
Expand Down
4 changes: 2 additions & 2 deletions docs/src/filereaders.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# `FileReaders`
# [`FileReaders`](@id file_reader_module)

Reading files is a common need for most scientific projects. This can come with
a series of problems that have to be solved, from performance (accessing can be
Expand All @@ -14,7 +14,7 @@ Future extensions might include:
- doing chunked reads;
- async reads.

## `NCFileReaders`
## [`NCFileReaders`](@id ncfilereaders)

> This extension is loaded when loading `NCDatasets`

Expand Down
2 changes: 1 addition & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ is mostly and interface barrier to provide a path for future improvements.
### `Regridders`

`ClimaUtilities` comes with two modules to map rectangular grids two (extruded)
finite spectral elements, `InterpolationsRegridder` and `TempestRegridder`.
finite spectral elements, `InterpolationsRegridder` and [`TempestRegridder`](@ref tempest_regridder).
These modules are primarily used to ingest data and resample it onto the
computational grid.

Expand Down
34 changes: 18 additions & 16 deletions docs/src/inputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ describes the types of plants on the surface of the globe. The
`SpaceVaringInputs` and `TimeVaryingInputs` modules provide a unified
infrastructure to handle all these cases.

## `TimeVaryingInputs`
## [`TimeVaryingInputs`](@id timevaryinginput)

> This extension is loaded when loading `ClimaCore` is loaded. In addition to
> this, if NetCDF files are used, `NCDatasets` has to be loaded too. Finally, a
Expand All @@ -21,9 +21,9 @@ developers to update their `Field`s.
This example shows that `TimeVaryingInput` can take different types of inputs
and be used with a single interface (`evaluate!`). In all of this,
`TimeVaryingInput`s internally handle all the complexity related to reading
files (using [`FileReaders`](@ref)), dealing with parallelism and GPUs,
regridding onto the computational domains (using [`Regridders`](@ref) and
[`DataHandling`](@ref)), and so on.
files (using [`FileReaders`](@ref file_reader_module)), dealing with parallelism and GPUs,
regridding onto the computational domains (using [`Regridders`](@ref regridder_module) and
[`DataHandling`](@ref datahandling_module)), and so on.

`TimeVaryingInputs` support:
- analytic functions of time;
Expand Down Expand Up @@ -53,7 +53,7 @@ All input variables to be composed together must have the same spatial and
temporal dimensions.

Composing multiple input variables is currently only supported with the
`InterpolationsRegridder`, not with `TempestRegridder`. The regridding
[`InterpolationsRegridder`](@ref interp_regridder), not with [`TempestRegridder`](@ref tempest_regridder). The regridding
is applied after the pre-processing and composing.

Composing multiple input variables in one `Input` is also possible with
Expand All @@ -64,7 +64,7 @@ a `SpaceVaryingInput`, and everything mentioned here applies in that case.
Suppose that the input NetCDF file `era5_example.nc` contains two variables `u`
and `v`, and we care about their sum `u + v` but not their individual values.
We can provide a pointwise composing function to perform the sum, along with
the `InterpolationsRegridder` to produce the data we want, `u + v`.
the [`InterpolationsRegridder`](@ref interp_regridder) to produce the data we want, `u + v`.
The `preprocess_func` passed in `file_reader_kwargs` will be applied to `u`
and to `v` individually, before the composing function is applied. The regridding
is applied after the composing function. `u` and `v` could also come from separate
Expand Down Expand Up @@ -109,7 +109,7 @@ timevaryinginput = TimeVaryingInputs.TimeVaryingInput(["era5_1980.nc", "era5_198

This capability is only available for the `InterpolationsRegridder`.

Read more about this feature in the page about [`DataHandler`](@ref).
Read more about this feature in the page about [`DataHandler`](@ref datahandling_module).

### Extrapolation boundary conditions

Expand Down Expand Up @@ -213,23 +213,24 @@ albedo_tv = TimeVaryingInputs.TimeVaryingInput("cesem_albedo.nc", "alb", target_

!!! note

In this example we used the [`TempestRegridder`](@ref). This is not the best
choice in most cases because the [`TempestRegridder`](@ref) is slower, and
In this example we used the [`TempestRegridder`](@ref tempest_regridder).
This is not the best
choice in most cases because the [`TempestRegridder`](@ref tempest_regridder) is slower, and
not well-compatible with MPI and GPUs (`ClimaUtilities` implements
workarounds for this, so the code would still work).
[`InterpolationsRegridder`](@ref) should be preferred, unless there is a
strict requirement of conservation: while [`TempestRegridder`](@ref) is
guaranteed to conserve various properties, [`InterpolationsRegridder`](@ref)
[`InterpolationsRegridder`](@ref interp_regridder) should be preferred, unless there is a
strict requirement of conservation: while [`TempestRegridder`](@ref tempest_regridder) is
guaranteed to conserve various properties, [`InterpolationsRegridder`](@ref interp_regridder)
is not.

## `SpaceVaryingInputs`
## [`SpaceVaryingInputs`](@id spacevaryinginput)

> This extension is loaded when loading `ClimaCore` is loaded. In addition to
> this, if NetCDF files are used, `NCDatasets` has to be loaded too. Finally, a
> `Regridder` is needed (which might require importing additional packages).

`SpaceVaryingInput`s uses the same building blocks as `TimeVaryingInput`
(chiefly the [`DataHandling`](@ref) module) to construct a `Field` from
(chiefly the [`DataHandling`](@ref datahandling_module) datahandling_module) to construct a `Field` from
different sources.

`SpaceVaryingInputs` support:
Expand All @@ -246,7 +247,7 @@ be a named tuple or a dictionary that maps `Symbol`s to values.

`SpaceVaryingInputs` support reading individual input variables from NetCDF files,
as well as composing multiple input variables into one `SpaceVaryingInput`.
See the [`TimeVaryingInput`](@ref) "NetCDF file inputs" section for more
See the [`TimeVaryingInput`](@ref timevaryinginput) "NetCDF file inputs" section for more
information about this feature.

### Example
Expand Down Expand Up @@ -284,5 +285,6 @@ ClimaUtilities.TimeVaryingInputs.Flat
ClimaUtilities.TimeVaryingInputs.evaluate!
ClimaUtilities.TimeVaryingInputs.extrapolation_bc
Base.in
Base.close
Base.close(::ClimaUtilities.TimeVaryingInputs.AbstractTimeVaryingInput)

```
2 changes: 1 addition & 1 deletion docs/src/onlinelogging.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ about current step, simulation time, and average performance. With

`WallTimeInfo` is a struct that holds information about wall time (the time you
see on your watch, not the simulation time) and that can be used to report
timing information with [`report_walltime`](@ref).
timing information with [`report_walltime`](@ref ClimaUtilities.OnlineLogging.report_walltime).

`WallTimeInfo` keeps track and accumulates how much time has elapsed since the
last time it was updated. In this, `WallTimeInfo` tries to automatically remove
Expand Down
6 changes: 3 additions & 3 deletions docs/src/regridders.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Regridders
# [`Regridders`](@id regridder_module)

Simulations often need to import external data directly onto the computational
grid. The `Regridders` module implements different schemes to accomplish this
Expand Down Expand Up @@ -27,7 +27,7 @@ If a regridder type is not specified by the user, and multiple are available,
the `InterpolationsRegridder` will be used by default. At least one regridder
extension must be loaded to be able to use regridding.

## `TempestRegridder`
## [`TempestRegridder`](@id tempest_regridder)

> This extension is loaded when loading `ClimaCoreTempestRemap`

Expand Down Expand Up @@ -57,7 +57,7 @@ reg = Regridders.TempestRegridder(target_space, "regrid_output", "u", "era5_exam
regridded_u = Regridders.regrid(reg, target_date)
```

## `InterpolationsRegridder`
## [`InterpolationsRegridder`](@id interp_regridder)

> This extension is loaded when loading `ClimaCore` and `Interpolations`

Expand Down
4 changes: 4 additions & 0 deletions src/TimeManager.jl
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,11 @@ datetime_to_strdate(datetime::Dates.DateTime) =
string(lpad(Dates.day(datetime), 2, "0"))

abstract type AbstractFrequency end

"Struct used to dispatch callback that is triggered monthly"
struct Monthly <: AbstractFrequency end

"Struct used to dispatch callback that is triggered every timestep"
struct EveryTimestep <: AbstractFrequency end

"""
Expand Down
Loading