Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge cgroups and environment variables for the resource allocation table. #56

Merged
merged 7 commits into from
Jan 13, 2022
Merged

Merge cgroups and environment variables for the resource allocation table. #56

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d19625b
Update seal-workers.md
SgtCoin Jan 10, 2022
d74d71f
Update configuration.md
SgtCoin Jan 10, 2022
c64590d
Update seal-workers.md
SgtCoin Jan 10, 2022
29849ef
Merge branch 'filecoin-project:main' into main
SgtCoin Jan 12, 2022
755c8d0
Apply suggestions from code review
johnnymatthews Jan 12, 2022
20bdbe3
Adds detail shortcode blocks.
johnnymatthews Jan 12, 2022
79fa10b
Adds empty line after paragraph and before title.
johnnymatthews Jan 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
352 changes: 352 additions & 0 deletions content/en/docs/set-up/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,358 @@ Variables specific to the _Lotus daemon_:
- `LOTUS_CHAIN_INDEX_CACHE`: Sets the size for the epoch index cache. Defaults to `32768`. Increase if you perform frequent deep chain lookups for block heights far from the latest height.
- `LOTUS_BSYNC_MSG_WINDOW`: Sets the initial maximum window size for message fetching blocksync request. Set to 10-20 if you have an internet connection with low bandwidth.

Variables for resource allocation tuning (overrides settings in the resource allocation table).

* 32G environment:
```
AP_32G_BASE_MIN_MEMORY=1073741824
AP_32G_GPU_UTILIZATION=0
AP_32G_MAX_MEMORY=4294967296
AP_32G_MAX_PARALLELISM=1
AP_32G_MAX_PARALLELISM_GPU=0
AP_32G_MIN_MEMORY=4294967296
C1_32G_BASE_MIN_MEMORY=1073741824
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These things are super long. I'm gonna try and add them into a <details> tag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasnt sure of the best way to format it, but a drop down is a great idea.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds a <details> shortcode: #61

C1_32G_GPU_UTILIZATION=0
C1_32G_MAX_MEMORY=1073741824
C1_32G_MAX_PARALLELISM=0
C1_32G_MAX_PARALLELISM_GPU=0
C1_32G_MIN_MEMORY=1073741824
C2_32G_BASE_MIN_MEMORY=34359738368
C2_32G_GPU_UTILIZATION=1
C2_32G_MAX_MEMORY=161061273600
C2_32G_MAX_PARALLELISM=-1
C2_32G_MAX_PARALLELISM_GPU=6
C2_32G_MIN_MEMORY=32212254720
GET_32G_BASE_MIN_MEMORY=0
GET_32G_GPU_UTILIZATION=0
GET_32G_MAX_MEMORY=1048576
GET_32G_MAX_PARALLELISM=0
GET_32G_MAX_PARALLELISM_GPU=0
GET_32G_MIN_MEMORY=1048576
PC1_32G_BASE_MIN_MEMORY=10485760
PC1_32G_GPU_UTILIZATION=0
PC1_32G_MAX_MEMORY=68719476736
PC1_32G_MAX_PARALLELISM=1
PC1_32G_MAX_PARALLELISM_GPU=0
PC1_32G_MIN_MEMORY=60129542144
PC2_32G_BASE_MIN_MEMORY=1073741824
PC2_32G_GPU_UTILIZATION=1
PC2_32G_MAX_MEMORY=16106127360
PC2_32G_MAX_PARALLELISM=-1
PC2_32G_MAX_PARALLELISM_GPU=6
PC2_32G_MIN_MEMORY=16106127360
UNS_32G_BASE_MIN_MEMORY=10485760
UNS_32G_GPU_UTILIZATION=0
UNS_32G_MAX_MEMORY=68719476736
UNS_32G_MAX_PARALLELISM=1
UNS_32G_MAX_PARALLELISM_GPU=0
UNS_32G_MIN_MEMORY=60129542144
```
* 512MB Environment:
```
AP_512M_BASE_MIN_MEMORY=1073741824
AP_512M_GPU_UTILIZATION=0
AP_512M_MAX_MEMORY=1073741824
AP_512M_MAX_PARALLELISM=1
AP_512M_MAX_PARALLELISM_GPU=0
AP_512M_MIN_MEMORY=1073741824
C1_512M_BASE_MIN_MEMORY=1073741824
C1_512M_GPU_UTILIZATION=0
C1_512M_MAX_MEMORY=1073741824
C1_512M_MAX_PARALLELISM=0
C1_512M_MAX_PARALLELISM_GPU=0
C1_512M_MIN_MEMORY=1073741824
C2_512M_BASE_MIN_MEMORY=10737418240
C2_512M_GPU_UTILIZATION=1
C2_512M_MAX_MEMORY=1610612736
C2_512M_MAX_PARALLELISM=1
C2_512M_MAX_PARALLELISM_GPU=0
C2_512M_MIN_MEMORY=1073741824
GET_512M_BASE_MIN_MEMORY=0
GET_512M_GPU_UTILIZATION=0
GET_512M_MAX_MEMORY=1048576
GET_512M_MAX_PARALLELISM=0
GET_512M_MAX_PARALLELISM_GPU=0
GET_512M_MIN_MEMORY=1048576
PC1_512M_BASE_MIN_MEMORY=1048576
PC1_512M_GPU_UTILIZATION=0
PC1_512M_MAX_MEMORY=1073741824
PC1_512M_MAX_PARALLELISM=1
PC1_512M_MAX_PARALLELISM_GPU=0
PC1_512M_MIN_MEMORY=805306368
PC2_512M_BASE_MIN_MEMORY=1073741824
PC2_512M_GPU_UTILIZATION=0
PC2_512M_MAX_MEMORY=1610612736
PC2_512M_MAX_PARALLELISM=-1
PC2_512M_MAX_PARALLELISM_GPU=0
PC2_512M_MIN_MEMORY=1073741824
UNS_512M_BASE_MIN_MEMORY=1048576
UNS_512M_GPU_UTILIZATION=0
UNS_512M_MAX_MEMORY=1073741824
UNS_512M_MAX_PARALLELISM=1
UNS_512M_MAX_PARALLELISM_GPU=0
UNS_512M_MIN_MEMORY=805306368
```
* 64GB Environment:
```
AP_64G_BASE_MIN_MEMORY=1073741824
AP_64G_GPU_UTILIZATION=0
AP_64G_MAX_MEMORY=8589934592
AP_64G_MAX_PARALLELISM=1
AP_64G_MAX_PARALLELISM_GPU=0
AP_64G_MIN_MEMORY=8589934592
C1_64G_BASE_MIN_MEMORY=1073741824
C1_64G_GPU_UTILIZATION=0
C1_64G_MAX_MEMORY=1073741824
C1_64G_MAX_PARALLELISM=0
C1_64G_MAX_PARALLELISM_GPU=0
C1_64G_MIN_MEMORY=1073741824
C2_64G_BASE_MIN_MEMORY=68719476736
C2_64G_GPU_UTILIZATION=1
C2_64G_MAX_MEMORY=204010946560
C2_64G_MAX_PARALLELISM=-1
C2_64G_MAX_PARALLELISM_GPU=6
C2_64G_MIN_MEMORY=64424509440
GET_64G_BASE_MIN_MEMORY=0
GET_64G_GPU_UTILIZATION=0
GET_64G_MAX_MEMORY=1048576
GET_64G_MAX_PARALLELISM=0
GET_64G_MAX_PARALLELISM_GPU=0
GET_64G_MIN_MEMORY=1048576
PC1_64G_BASE_MIN_MEMORY=10485760
PC1_64G_GPU_UTILIZATION=0
PC1_64G_MAX_MEMORY=137438953472
PC1_64G_MAX_PARALLELISM=1
PC1_64G_MAX_PARALLELISM_GPU=0
PC1_64G_MIN_MEMORY=120259084288
PC2_64G_BASE_MIN_MEMORY=1073741824
PC2_64G_GPU_UTILIZATION=1
PC2_64G_MAX_MEMORY=32212254720
PC2_64G_MAX_PARALLELISM=-1
PC2_64G_MAX_PARALLELISM_GPU=6
PC2_64G_MIN_MEMORY=32212254720
UNS_64G_BASE_MIN_MEMORY=10485760
UNS_64G_GPU_UTILIZATION=0
UNS_64G_MAX_MEMORY=137438953472
UNS_64G_MAX_PARALLELISM=1
UNS_64G_MAX_PARALLELISM_GPU=0
UNS_64G_MIN_MEMORY=120259084288
```
* All Environment variables:
```
AP_2K_BASE_MIN_MEMORY=2048
AP_2K_GPU_UTILIZATION=0
AP_2K_MAX_MEMORY=2048
AP_2K_MAX_PARALLELISM=1
AP_2K_MAX_PARALLELISM_GPU=0
AP_2K_MIN_MEMORY=2048
AP_32G_BASE_MIN_MEMORY=1073741824
AP_32G_GPU_UTILIZATION=0
AP_32G_MAX_MEMORY=4294967296
AP_32G_MAX_PARALLELISM=1
AP_32G_MAX_PARALLELISM_GPU=0
AP_32G_MIN_MEMORY=4294967296
AP_512M_BASE_MIN_MEMORY=1073741824
AP_512M_GPU_UTILIZATION=0
AP_512M_MAX_MEMORY=1073741824
AP_512M_MAX_PARALLELISM=1
AP_512M_MAX_PARALLELISM_GPU=0
AP_512M_MIN_MEMORY=1073741824
AP_64G_BASE_MIN_MEMORY=1073741824
AP_64G_GPU_UTILIZATION=0
AP_64G_MAX_MEMORY=8589934592
AP_64G_MAX_PARALLELISM=1
AP_64G_MAX_PARALLELISM_GPU=0
AP_64G_MIN_MEMORY=8589934592
AP_8M_BASE_MIN_MEMORY=8388608
AP_8M_GPU_UTILIZATION=0
AP_8M_MAX_MEMORY=8388608
AP_8M_MAX_PARALLELISM=1
AP_8M_MAX_PARALLELISM_GPU=0
AP_8M_MIN_MEMORY=8388608
C1_2K_BASE_MIN_MEMORY=2048
C1_2K_GPU_UTILIZATION=0
C1_2K_MAX_MEMORY=2048
C1_2K_MAX_PARALLELISM=0
C1_2K_MAX_PARALLELISM_GPU=0
C1_2K_MIN_MEMORY=2048
C1_32G_BASE_MIN_MEMORY=1073741824
C1_32G_GPU_UTILIZATION=0
C1_32G_MAX_MEMORY=1073741824
C1_32G_MAX_PARALLELISM=0
C1_32G_MAX_PARALLELISM_GPU=0
C1_32G_MIN_MEMORY=1073741824
C1_512M_BASE_MIN_MEMORY=1073741824
C1_512M_GPU_UTILIZATION=0
C1_512M_MAX_MEMORY=1073741824
C1_512M_MAX_PARALLELISM=0
C1_512M_MAX_PARALLELISM_GPU=0
C1_512M_MIN_MEMORY=1073741824
C1_64G_BASE_MIN_MEMORY=1073741824
C1_64G_GPU_UTILIZATION=0
C1_64G_MAX_MEMORY=1073741824
C1_64G_MAX_PARALLELISM=0
C1_64G_MAX_PARALLELISM_GPU=0
C1_64G_MIN_MEMORY=1073741824
C1_8M_BASE_MIN_MEMORY=8388608
C1_8M_GPU_UTILIZATION=0
C1_8M_MAX_MEMORY=8388608
C1_8M_MAX_PARALLELISM=0
C1_8M_MAX_PARALLELISM_GPU=0
C1_8M_MIN_MEMORY=8388608
C2_2K_BASE_MIN_MEMORY=2048
C2_2K_GPU_UTILIZATION=1
C2_2K_MAX_MEMORY=2048
C2_2K_MAX_PARALLELISM=1
C2_2K_MAX_PARALLELISM_GPU=0
C2_2K_MIN_MEMORY=2048
C2_32G_BASE_MIN_MEMORY=34359738368
C2_32G_GPU_UTILIZATION=1
C2_32G_MAX_MEMORY=161061273600
C2_32G_MAX_PARALLELISM=-1
C2_32G_MAX_PARALLELISM_GPU=6
C2_32G_MIN_MEMORY=32212254720
C2_512M_BASE_MIN_MEMORY=10737418240
C2_512M_GPU_UTILIZATION=1
C2_512M_MAX_MEMORY=1610612736
C2_512M_MAX_PARALLELISM=1
C2_512M_MAX_PARALLELISM_GPU=0
C2_512M_MIN_MEMORY=1073741824
C2_64G_BASE_MIN_MEMORY=68719476736
C2_64G_GPU_UTILIZATION=1
C2_64G_MAX_MEMORY=204010946560
C2_64G_MAX_PARALLELISM=-1
C2_64G_MAX_PARALLELISM_GPU=6
C2_64G_MIN_MEMORY=64424509440
C2_8M_BASE_MIN_MEMORY=8388608
C2_8M_GPU_UTILIZATION=1
C2_8M_MAX_MEMORY=8388608
C2_8M_MAX_PARALLELISM=1
C2_8M_MAX_PARALLELISM_GPU=0
C2_8M_MIN_MEMORY=8388608
GET_2K_BASE_MIN_MEMORY=0
GET_2K_GPU_UTILIZATION=0
GET_2K_MAX_MEMORY=1048576
GET_2K_MAX_PARALLELISM=0
GET_2K_MAX_PARALLELISM_GPU=0
GET_2K_MIN_MEMORY=1048576
GET_32G_BASE_MIN_MEMORY=0
GET_32G_GPU_UTILIZATION=0
GET_32G_MAX_MEMORY=1048576
GET_32G_MAX_PARALLELISM=0
GET_32G_MAX_PARALLELISM_GPU=0
GET_32G_MIN_MEMORY=1048576
GET_512M_BASE_MIN_MEMORY=0
GET_512M_GPU_UTILIZATION=0
GET_512M_MAX_MEMORY=1048576
GET_512M_MAX_PARALLELISM=0
GET_512M_MAX_PARALLELISM_GPU=0
GET_512M_MIN_MEMORY=1048576
GET_64G_BASE_MIN_MEMORY=0
GET_64G_GPU_UTILIZATION=0
GET_64G_MAX_MEMORY=1048576
GET_64G_MAX_PARALLELISM=0
GET_64G_MAX_PARALLELISM_GPU=0
GET_64G_MIN_MEMORY=1048576
GET_8M_BASE_MIN_MEMORY=0
GET_8M_GPU_UTILIZATION=0
GET_8M_MAX_MEMORY=1048576
GET_8M_MAX_PARALLELISM=0
GET_8M_MAX_PARALLELISM_GPU=0
GET_8M_MIN_MEMORY=1048576
PC1_2K_BASE_MIN_MEMORY=2048
PC1_2K_GPU_UTILIZATION=0
PC1_2K_MAX_MEMORY=2048
PC1_2K_MAX_PARALLELISM=1
PC1_2K_MAX_PARALLELISM_GPU=0
PC1_2K_MIN_MEMORY=2048
PC1_32G_BASE_MIN_MEMORY=10485760
PC1_32G_GPU_UTILIZATION=0
PC1_32G_MAX_MEMORY=68719476736
PC1_32G_MAX_PARALLELISM=1
PC1_32G_MAX_PARALLELISM_GPU=0
PC1_32G_MIN_MEMORY=60129542144
PC1_512M_BASE_MIN_MEMORY=1048576
PC1_512M_GPU_UTILIZATION=0
PC1_512M_MAX_MEMORY=1073741824
PC1_512M_MAX_PARALLELISM=1
PC1_512M_MAX_PARALLELISM_GPU=0
PC1_512M_MIN_MEMORY=805306368
PC1_64G_BASE_MIN_MEMORY=10485760
PC1_64G_GPU_UTILIZATION=0
PC1_64G_MAX_MEMORY=137438953472
PC1_64G_MAX_PARALLELISM=1
PC1_64G_MAX_PARALLELISM_GPU=0
PC1_64G_MIN_MEMORY=120259084288
PC1_8M_BASE_MIN_MEMORY=8388608
PC1_8M_GPU_UTILIZATION=0
PC1_8M_MAX_MEMORY=8388608
PC1_8M_MAX_PARALLELISM=1
PC1_8M_MAX_PARALLELISM_GPU=0
PC1_8M_MIN_MEMORY=8388608
PC2_2K_BASE_MIN_MEMORY=2048
PC2_2K_GPU_UTILIZATION=0
PC2_2K_MAX_MEMORY=2048
PC2_2K_MAX_PARALLELISM=-1
PC2_2K_MAX_PARALLELISM_GPU=0
PC2_2K_MIN_MEMORY=2048
PC2_32G_BASE_MIN_MEMORY=1073741824
PC2_32G_GPU_UTILIZATION=1
PC2_32G_MAX_MEMORY=16106127360
PC2_32G_MAX_PARALLELISM=-1
PC2_32G_MAX_PARALLELISM_GPU=6
PC2_32G_MIN_MEMORY=16106127360
PC2_512M_BASE_MIN_MEMORY=1073741824
PC2_512M_GPU_UTILIZATION=0
PC2_512M_MAX_MEMORY=1610612736
PC2_512M_MAX_PARALLELISM=-1
PC2_512M_MAX_PARALLELISM_GPU=0
PC2_512M_MIN_MEMORY=1073741824
PC2_64G_BASE_MIN_MEMORY=1073741824
PC2_64G_GPU_UTILIZATION=1
PC2_64G_MAX_MEMORY=32212254720
PC2_64G_MAX_PARALLELISM=-1
PC2_64G_MAX_PARALLELISM_GPU=6
PC2_64G_MIN_MEMORY=32212254720
PC2_8M_BASE_MIN_MEMORY=8388608
PC2_8M_GPU_UTILIZATION=0
PC2_8M_MAX_MEMORY=8388608
PC2_8M_MAX_PARALLELISM=-1
PC2_8M_MAX_PARALLELISM_GPU=0
PC2_8M_MIN_MEMORY=8388608
UNS_2K_BASE_MIN_MEMORY=2048
UNS_2K_GPU_UTILIZATION=0
UNS_2K_MAX_MEMORY=2048
UNS_2K_MAX_PARALLELISM=1
UNS_2K_MAX_PARALLELISM_GPU=0
UNS_2K_MIN_MEMORY=2048
UNS_32G_BASE_MIN_MEMORY=10485760
UNS_32G_GPU_UTILIZATION=0
UNS_32G_MAX_MEMORY=68719476736
UNS_32G_MAX_PARALLELISM=1
UNS_32G_MAX_PARALLELISM_GPU=0
UNS_32G_MIN_MEMORY=60129542144
UNS_512M_BASE_MIN_MEMORY=1048576
UNS_512M_GPU_UTILIZATION=0
UNS_512M_MAX_MEMORY=1073741824
UNS_512M_MAX_PARALLELISM=1
UNS_512M_MAX_PARALLELISM_GPU=0
UNS_512M_MIN_MEMORY=805306368
UNS_64G_BASE_MIN_MEMORY=10485760
UNS_64G_GPU_UTILIZATION=0
UNS_64G_MAX_MEMORY=137438953472
UNS_64G_MAX_PARALLELISM=1
UNS_64G_MAX_PARALLELISM_GPU=0
UNS_64G_MIN_MEMORY=120259084288
UNS_8M_BASE_MIN_MEMORY=8388608
UNS_8M_GPU_UTILIZATION=0
UNS_8M_MAX_MEMORY=8388608
UNS_8M_MAX_PARALLELISM=1
UNS_8M_MAX_PARALLELISM_GPU=0
UNS_8M_MIN_MEMORY=8388608
```


## Controlling a remote daemon

The `lotus` application, as a client to the lotus daemon, can talk to a Lotus daemon running on any location (not just the local one), by setting the following environment variable:
Expand Down
7 changes: 7 additions & 0 deletions content/en/docs/storage-providers/seal-workers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ Default resource value table. Some of these values are _fairly_ conservative:

The Unseal task has the same resource use as the PreCommit1 task.

{{< alert icon="info" >}}
The default and custom configurations in the task resource table can be overridden using environment variables on a per worker basis. See the [environment variables]({{< relref "config#environment-variables" >}}) section for information. You can also gather these details using the `lotus-worker resources --default` command.
{{< /alert >}}

### Resource windows

The scheduler uses the concept of resource windows to prevent resource starvation of tasks requiring larger amounts of resources by tasks with smaller resource requirements.
Expand Down Expand Up @@ -101,6 +105,9 @@ When comparing task priority:
- If the above is equal, tasks are selected based on priorities in the table
- If the above is equal, sectors with lower sector numbers are selected (this can optimize gas usage slightly when submitting messages to the chain)

### Control groups

Countrol groups (cgroups) is a Linux Kernel feature that limits, accounts for, and isolates the resource usage of a collection of processes. If cgroups are in use on the host, the Lotus Worker will honor the cgroup memory limits configured on the host.
johnnymatthews marked this conversation as resolved.
Show resolved Hide resolved
## Installation

{{< alert icon="callout" >}}
Expand Down