Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discourage rebalance, warn against stopping it #1298

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Discourage rebalance, warn against stopping it #1298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a9bfde7
draft: discourage rebalance, warn against stopping it
feorlen Aug 14, 2024
40606ce
additional discussion of mismatched pool sizes
feorlen Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions source/operations/checklists/hardware.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,9 @@ Storage
MinIO limits the maximum usable size per drive to the smallest size in the deployment.
For example, if a deployment has 15 10TB drives and 1 1TB drive, MinIO limits the per-drive capacity to 1TB.

For deployments with multiple server pools, each individual pool may have its own hardware configuration.
However, significant capacity differences between pools may temporarily result in high loads on a new pool's nodes during :ref:`expansion <expand-minio-distributed>`. For more information, see :ref:`How do I manage object distribution across a MinIO deployment? <minio-rebalance>`
Comment on lines +342 to +343
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to say this in the hardware checklist?


Recommended Hardware Tests
--------------------------

Expand Down
17 changes: 11 additions & 6 deletions source/operations/concepts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,21 +140,26 @@ There are several options to manage your MinIO deployments and clusters:
How do I manage object distribution across a MinIO deployment?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

MinIO optimizes storage of objects across available pools by writing new objects (that is, objects with no existing versions) to the server pool with the most free space compared total amount of free space on all available server pools.
MinIO optimizes storage of objects across available pools by writing new objects (that is, objects with no existing versions) to the server pool with the most free space compared to the total amount of free space on all available server pools.
MinIO does not perform the costly action of rebalancing objects from older pools to newer pools.
Instead, new objects typically route to the new pool as it has the most free space.
As that pool fills, new write operations eventually balance out across all pools in the deployment.
For more information on write preference calculation logic, see :ref:`Writing Files <minio-writing-files>` below.

As the new pool fills, write operations eventually balance out across all pools in the deployment.
Until then, the new pool's nodes may experience higher loads and slower writes.

To reduce this temporary performance impact, MinIO recommends expanding a deployment well before its existing pools are near capacity and with new pools of a similar size.
For more information on write preference calculation logic, see :ref:`Writing Files <minio-writing-files>`.
Comment on lines +147 to +151
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accurate? Sufficient?

Other mentions of pool sizing link to this section


Rebalancing data across all pools after an expansion is an expensive operation that requires scanning the entire deployment and moving objects between pools.
This may take a long time to complete depending on the amount of data to move.

Starting with MinIO Client version RELEASE.2022-11-07T23-47-39Z, you can manually initiate a rebalancing operation across all server pools using :mc:`mc admin rebalance`.

If required, you can manually initiate a rebalancing operation across all server pools using :mc:`mc admin rebalance`.
feorlen marked this conversation as resolved.
Show resolved Hide resolved
Rebalancing does not block ongoing operations and runs in parallel to all other I/O.
This can result in reduced performance of regular operations.
Consider scheduling rebalancing operations during non-peak periods to avoid impacting production workloads.
You can start and stop rebalancing at any time

MinIO recommends `SUBNET <https://min.io/pricing?jmp=docs>`__ users `log in <https://subnet.min.io/>`__ and create a new issue to discuss capacity planning or rebalancing considerations for their deployments.


How do I upload objects to MinIO?
---------------------------------
Expand Down
14 changes: 8 additions & 6 deletions source/operations/install-deploy-manage/expand-minio-deployment.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,7 @@ Writing Files
~~~~~~~~~~~~~

MinIO does not automatically rebalance objects across the new server pools.
Instead, MinIO performs new write operations to the pool with the most free
storage weighted by the amount of free space on the pool divided by the free space across all available pools.
Instead, MinIO performs new write operations to the pool with the most free storage weighted by the amount of free space on the pool divided by the free space across all available pools.

The formula to determine the probability of a write operation on a particular pool is

Expand All @@ -162,11 +161,14 @@ MinIO calculates the probability of a write operation to each of the pools as:
- Pool B: 20% chance (:math:`2TiB / 10TiB`)
- Pool C: 50% chance (:math:`5TiB / 10TiB`)

In addition to the free space calculation, if a write option (with parity) would bring a drive
usage above 99% or a known free inode count below 1000, MinIO does not write to the pool.
In addition to the free space calculation, if a write option (with parity) would bring a drive usage above 99% or a known free inode count below 1000, MinIO does not write to the pool.

If desired, you can manually initiate a rebalance procedure with :mc:`mc admin rebalance`.
For more about how rebalancing works, see :ref:`managing objects across a deployment <minio-rebalance>`.
Since a pool with more free space has a higher probability of being written to, the nodes of that pool may experience higher loads as free space equalizes.

If required, you can manually initiate a rebalance procedure with :mc:`mc admin rebalance`.
feorlen marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +166 to +168
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain what happens if pools have very different available free space. Is this text an accurate characterization?

MinIO recommends `SUBNET <https://min.io/pricing?jmp=docs>`__ users `log in <https://subnet.min.io/>`__ and create a new issue to discuss capacity planning or rebalancing considerations for their deployments.

For more about how rebalancing works see :ref:`managing objects across a deployment <minio-rebalance>`.

Likewise, MinIO does not write to pools in a decommissioning process.

Expand Down
11 changes: 9 additions & 2 deletions source/reference/minio-mc-admin/mc-admin-rebalance.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Description

.. start-mc-admin-rebalance-desc

The :mc-cmd:`mc admin rebalance` command allows starts, monitors, or stops a rebalancing operation on a MinIO deployment.
The :mc-cmd:`mc admin rebalance` command starts, stops, or monitors a rebalancing operation on a MinIO deployment.
Rebalancing redistributes objects across all pools in the deployment.

.. end-mc-admin-rebalance-desc
Expand All @@ -35,7 +35,7 @@ This is an expensive and time consuming operation.
Consider only running a rebalance procedure during light or no use of the deployment.
If write operations do occur during a rebalance operation, they process in parallel and write to a pool not actively in rebalancing.

You can stop a rebalance and start it again later as needed.
`MinIO SUBNET <https://min.io/pricing?jmp=docs>`__ users can `log in <https://subnet.min.io/>`__ and create a new issue to discuss appropriate rebalancing strategies for deployments.

Follow the progress of an ongoing rebalance operation using the following command:

Expand Down Expand Up @@ -70,6 +70,7 @@ The :mc-cmd:`mc admin rebalance` command has the following subcommands:
* - :mc-cmd:`mc admin rebalance stop`
- Stops an in-progress rebalance operation.


Syntax
------

Expand Down Expand Up @@ -135,6 +136,12 @@ Syntax

Ends an in-progress rebalance job on the specified deployment.

.. admonition:: Stopping a rebalance job on previous versions of MinIO may cause data loss
:class: warning

A bug in MinIO prior to :minio-release:`RELEASE.2024-08-17T01-24-54Z` can overwrite objects while stopping a in-progress rebalance operation.
Interrupting rebalance on these older versions may result in data loss.

Comment on lines +139 to +144
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a usual way we reference things like this? What should we say about the now fixed bug?

.. tab-set::

.. tab-item:: EXAMPLES
Expand Down