Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify what happens during each Lotus Worker window. #176

Closed
shotcollin opened this issue Aug 27, 2021 · 0 comments
Closed

Clarify what happens during each Lotus Worker window. #176

shotcollin opened this issue Aug 27, 2021 · 0 comments
Labels
dif/trivial Can be confidently tackled by newcomers effort/hours Estimated to take one or several hours kind/maintenance P2 Medium: Good to have, but can wait until someone steps up

Comments

@shotcollin
Copy link
Contributor

shotcollin commented Aug 27, 2021

Did you find a bug, typo or other issue in an item of Filecoin documentation? Please use this template to let us know!

URL of the page in question:
https://docs.filecoin.io/mine/lotus/seal-workers/#resource-allocation-in-lotus-workers

What's wrong with this page?

Each Lotus Worker can run multiple tasks, depending on your hardware resources. Each slot is called a window. The final number is determined by the number of available cores and the requirements of the sealing phase allocated to it. That means that a single worker on a 8-core CPU with a single GPU will run at most:

  • Number of CPU threads the task will use.
  • Minimum amount of RAM required for good performance.
  • Maximum amount of RAM required to run the task, where the system can swap out part of the RAM to disk, and the performance won't be affected too much.
  • Whether the system can use a GPU.

The bulleted list doesn't appear to be completing the sentence that comes before it, i.e. "That means that a single worker on a 8-core CPU with a single GPU will run at most:" should be followed with bullets like "one AddPiece task or one such and such task..." The bullets sound like a list of additional considerations to figure out what exactly you can run on a given worker. Also, the first two sentences could use clarification.

What would you like to have changed/amended/fixed in order to make things right?
I'm not sure of the accuracy of what follows, but something like this would be helpful:

Each Lotus Worker can run multiple tasks. Tasks are grouped and run in windows (see below). Ultimately the number and type of tasks that can be run simultaneously is determined by the CPU core count, RAM, and the requirements of each sealing phase, which are:

  • Number of CPU threads the task will use.
  • Minimum amount of RAM required for good performance.
  • Maximum amount of RAM required to run the task, where the system can swap out part of the RAM to disk, and the performance won't be affected too much.
  • Whether the system can use a GPU.

That means that a single worker on a 8-core CPU with a single GPU will run at most:

  • x AddPiece tasks and y Commit1 tasks
  • x PreCommit1 tasks
  • etc.

Would you be interested in helping fix this page?
I'd love to help but unfortunately don't yet know enough to accurately document this. Will definitely consider submitting PRs for anything I can confidently contribute, though.

Is there any other feedback you'd like to share about Filecoin docs?
Pretty amazing job from what I've seen so far. Much appreciated!

@shotcollin shotcollin added the need/triage Needs initial labeling and prioritization label Aug 27, 2021
@johnnymatthews johnnymatthews changed the title [DOCS ISSUE] Page: Lotus Miner: seal workers Clarify what happens during each Lotus Worker window. Sep 7, 2021
@johnnymatthews johnnymatthews added dif/trivial Can be confidently tackled by newcomers effort/hours Estimated to take one or several hours kind/maintenance P2 Medium: Good to have, but can wait until someone steps up and removed need/triage Needs initial labeling and prioritization labels Sep 7, 2021
shotcollin referenced this issue in shotcollin/lotus-docs Jan 5, 2022
This actually resolves #1021 in the old filecoin-docs (https://github.com/filecoin-project/filecoin-docs/issues/1021). The issue title now refers to clarifying what happens in worker windows, but the original request was just to make this part of the docs make sense as a sentence, and the table below gives enough detail for the reader to calculate what could be done with a given set of resources, which was the other part of the question in original issue #1021.
johnnymatthews referenced this issue Jan 13, 2022
* Remove out of date info on control addresses

Referenced feature is in current release.

* Clarify resources re worker windows

This actually resolves #1021 in the old filecoin-docs (https://github.com/filecoin-project/filecoin-docs/issues/1021). The issue title now refers to clarifying what happens in worker windows, but the original request was just to make this part of the docs make sense as a sentence, and the table below gives enough detail for the reader to calculate what could be done with a given set of resources, which was the other part of the question in original issue #1021.

* Updated Lotus worker co-location section

Modified co-location info for multi GPU setups since support was added to last RC and appears to be moving forward. Also added use case (which I can confirm is feasible) for co-location related to parallel GPU task processing.

* Reorganized storage groups section

I found the new documentation on storage groups a bit confusing so I thought I'd attempt to help clarify it a bit. Also corrected some errors/typos in config file and storage attach command syntax. One thing that's still not clear (to me) from this documentation is what the use case would be for having more than one value in Groups for any given sectorstore file. It also might be helpful to suggest how storage groups could work in an environment where some storage locations are shared and others are local, as well as considerations when workers have multiple (local) sector stores, but I haven't spent enough time configuring my own storage groups to be able to add that at this point.

I moved the storage groups documentation under a new heading at the bottom for multi-worker setups, in hopes that eventually there might be more robust official documentation on best practices for multiple workers since that's what most people seem to doing.

* Adds minor formatting tweaks.

* Addresses broken relref links.

Co-authored-by: Johnny <9611008+johnnymatthews@users.noreply.github.com>
johnnymatthews referenced this issue Jan 24, 2022
* Change table (#44)

* Update multisig.md

Add in description for cancel a multisig transaction and inspect msig params with lotus-shed.

* Update multisig.md

format updates to cancel and inspect processes

* Update mining-architectures.md

* Update mining-architectures.md

* Formatting fixes.

* Adds closing codeblock.

Co-authored-by: Johnny <9611008+johnnymatthews@users.noreply.github.com>

* Moves back to original theme. (#39)

* Moves back to original theme.

* Adds proper homescreen.

* Temp config change for Netlify preview.

* Adds Lotus logo as SVG to topbar nav.

* Sorting layout for homepage.

* Adds placeholder text to homepage.

* Completes homepage copy and layout.

* Attempts to fix logo SVG.

* Adds link to install from homepage.

* Adds getting started section.

* Adds basic progress bar movement.

* Adds progress bar for scrolling.

* Adds plaintext option for code blocks.

* Adds plaintext to code blocks.

* Links to new 'What is Lotus' page.

* Changes identifer to make title easier to read.

* Adds placeholders for getting started content.

* Adds styling for progressbar in dark mode.

* Orders all pages. Again.

* Changes unfinished posts to draft.

* Adds draft warning to content.

* Fixing broken external links.

* Edits external links.

* Hides 'Edit' page for now.

* Further external link fixes.

* Fix/minor style edits (#45)

* Updates GitHub link.

* Fixes external links.

* Adds ZH version dropdown.

* Adds ZH language to dropdown.

* Cleans up ZH menu.

* Corrects base URL for search.

* Fix typo (#47)

* Adds minor error fixes from filecoin PR. (#50)

* Adds minor error fixes from filecoin PR.

* Fixes mutliple formatting errors.

* add partial retrieval CLI and update client CLI (#54)

* Updates Mermaid package. (#55)

* Update/default configs (#58)

* Adds default Lotus config.

* Adds default SP config.

* Adds a details shortcode. (#61)

* typofix & bullet addition (#46)

* Merge cgroups and environment variables for the resource allocation table. (#56)

* Update seal-workers.md

* Update configuration.md

* Update seal-workers.md

* Apply suggestions from code review

* Adds detail shortcode blocks.

* Adds empty line after paragraph and before title.

Co-authored-by: Johnny <9611008+johnnymatthews@users.noreply.github.com>

* Updates markdown-it to latest version dev. (#62)

* Adds warning callout for config updates. (#59)

* Adds warning callout for config updates.

* Adds config upgrade warning for miner and client.

* Storage provider updates and copy edits (#49)

* Remove out of date info on control addresses

Referenced feature is in current release.

* Clarify resources re worker windows

This actually resolves #1021 in the old filecoin-docs (https://github.com/filecoin-project/filecoin-docs/issues/1021). The issue title now refers to clarifying what happens in worker windows, but the original request was just to make this part of the docs make sense as a sentence, and the table below gives enough detail for the reader to calculate what could be done with a given set of resources, which was the other part of the question in original issue #1021.

* Updated Lotus worker co-location section

Modified co-location info for multi GPU setups since support was added to last RC and appears to be moving forward. Also added use case (which I can confirm is feasible) for co-location related to parallel GPU task processing.

* Reorganized storage groups section

I found the new documentation on storage groups a bit confusing so I thought I'd attempt to help clarify it a bit. Also corrected some errors/typos in config file and storage attach command syntax. One thing that's still not clear (to me) from this documentation is what the use case would be for having more than one value in Groups for any given sectorstore file. It also might be helpful to suggest how storage groups could work in an environment where some storage locations are shared and others are local, as well as considerations when workers have multiple (local) sector stores, but I haven't spent enough time configuring my own storage groups to be able to add that at this point.

I moved the storage groups documentation under a new heading at the bottom for multi-worker setups, in hopes that eventually there might be more robust official documentation on best practices for multiple workers since that's what most people seem to doing.

* Adds minor formatting tweaks.

* Addresses broken relref links.

Co-authored-by: Johnny <9611008+johnnymatthews@users.noreply.github.com>

* add chiese version of Developer section

Co-authored-by: sgt_coin <91992934+SgtCoin@users.noreply.github.com>
Co-authored-by: Johnny <9611008+johnnymatthews@users.noreply.github.com>
Co-authored-by: Andy Schwab <59572086+andyschwab@users.noreply.github.com>
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
Co-authored-by: Mosh <1306020+mishmosh@users.noreply.github.com>
Co-authored-by: shotcollin <virtueisitsownreward@protonmail.com>
@johnnymatthews johnnymatthews transferred this issue from filecoin-project/filecoin-docs May 2, 2022
@rjan90 rjan90 closed this as completed Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dif/trivial Can be confidently tackled by newcomers effort/hours Estimated to take one or several hours kind/maintenance P2 Medium: Good to have, but can wait until someone steps up
Projects
None yet
Development

No branches or pull requests

3 participants