translate Cluster Training and Prediction #9356
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
typhoonzero
reviewed
Mar 25, 2018
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| 1. Network connection errors in the log during muliti-node cluster training | ||
| ------------------------------------------------ | ||
| The errors in the log belong to network connection during mulilti-node cluster training, for example, :code:`Connection reset by peer`. |
There was a problem hiding this comment.
This sentence only have subject but no predicate and no object.
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| * Find the first error in the :code:`train.log`, :code:`server.log`, check whether other fault casued the problem, such as FPE, lacking of memory or disk. | ||
|
|
||
| * If network connection gave rise to the first error in the log, this may be caused by the port conflict of the non-exclusive execution. Connect with the operator to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If so, change the port of job. |
There was a problem hiding this comment.
"If network connection gave rise to the first error in the log" => If the first error in server.log says "Address already used"
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| * Find the first error in the :code:`train.log`, :code:`server.log`, check whether other fault casued the problem, such as FPE, lacking of memory or disk. | ||
|
|
||
| * If network connection gave rise to the first error in the log, this may be caused by the port conflict of the non-exclusive execution. Connect with the operator to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If so, change the port of job. |
There was a problem hiding this comment.
Connect with the operator => Contact the sys-admin
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| * Find the first error in the :code:`train.log`, :code:`server.log`, check whether other fault casued the problem, such as FPE, lacking of memory or disk. | ||
|
|
||
| * If network connection gave rise to the first error in the log, this may be caused by the port conflict of the non-exclusive execution. Connect with the operator to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If so, change the port of job. |
There was a problem hiding this comment.
If so, change the port of job. => If the current MPI cluster does not support this parameter, change the server port and try agian.
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| * If network connection gave rise to the first error in the log, this may be caused by the port conflict of the non-exclusive execution. Connect with the operator to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If so, change the port of job. | ||
|
|
||
| * If the currnet MPI cluster does not support exclusive pattern, ask the operator to replace or update the current cluster. |
There was a problem hiding this comment.
people may want to know what the "exclusive pattern" is.
There was a problem hiding this comment.
operator => cluster administrator
There was a problem hiding this comment.
people may want to know what the "exclusive pattern" is.
Which doc should we refer this term to ?
abhinavarora
suggested changes
Mar 26, 2018
doc/v2/faq/cluster/index_en.rst
Outdated
| TBD | ||
| .. contents:: | ||
|
|
||
| 1. Network connection errors in the log during muliti-node cluster training |
There was a problem hiding this comment.
muliti-node -> multi-node
doc/v2/faq/cluster/index_en.rst
Outdated
| .. contents:: | ||
|
|
||
| 1. Network connection errors in the log during muliti-node cluster training | ||
| ------------------------------------------------ |
There was a problem hiding this comment.
mulilti-node -> multi-node
doc/v2/faq/cluster/index_en.rst
Outdated
| 1. Network connection errors in the log during muliti-node cluster training | ||
| ------------------------------------------------ | ||
| The errors in the log belong to network connection during mulilti-node cluster training, for example, :code:`Connection reset by peer`. | ||
| This kind of error is usually caused by the abnormal exit of the training process in some node, and the others cannot connect with this node any longer. Steps to troubleshoot the problem as follows: |
There was a problem hiding this comment.
of the training process -> of a training process
There was a problem hiding this comment.
others -> other nodes
There was a problem hiding this comment.
the problem as follows -> the problem are as follows
doc/v2/faq/cluster/index_en.rst
Outdated
|
|
||
| * If network connection gave rise to the first error in the log, this may be caused by the port conflict of the non-exclusive execution. Connect with the operator to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If so, change the port of job. | ||
|
|
||
| * If the currnet MPI cluster does not support exclusive pattern, ask the operator to replace or update the current cluster. |
|
done |
blacksheep-Aristotle
pushed a commit
to blacksheep-Aristotle/Paddle
that referenced
this pull request
Nov 22, 2024
* support optimizer state offload (PaddlePaddle#8715) * support optimizer offload * update doc * [FleetY]offload optimizer state after load optmizer state (PaddlePaddle#9352) * add offload optimizer * fix memory * [FleetY] Add reload/offload for optimizer (PaddlePaddle#9356) * add reload/offload for optimizer * fix bug --------- Co-authored-by: Guoxia Wang <mingzilaochongtu@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix #8954