DOCS-987 sharding page and sharding docs

source/administration/sharded-clusters.txt

@@ -0,0 +1,282 @@
+.. index:: sharded clusters
+.. _sharding-sharded-cluster:
+
+===============================
+Components in a Sharded Cluster
+===============================
+
+.. default-domain:: mongodb
+
+Sharding occurs within a :term:`sharded cluster`. A sharded cluster
+consists of the following components:
+
+- :ref:`Shards <sharding-shards>`. Each shard is a separate
+  :program:`mongod` instance or :term:`replica set` that holds a portion
+  of the your database collections.
+
+- :ref:`Config servers <sharding-config-server>`. Each config server is
+  a :program:`mongod` instances that holds metadata about the cluster.
+  The metadata maps :term:`chunks <chunk>` to shards.
+
+- :ref:`mongos instances <sharding-mongos>`. The :program:`mongos`
+  instances route the reads and writes to the shards.
+
+.. seealso::
+
+   - For specific configurations, see :ref:`sharding-architecture`.
+
+   - To set up sharded clusters, see :ref:`sharding-procedure-setup`.
+
+.. index:: sharding; shards
+.. index:: shards
+.. _sharding-shards:
+
+Shards
+------
+
+A shard is a container that holds a subset of a collection’s data. Each
+shard is either a single :program:`mongod` instance or a :term:`replica
+set`. In production, all shards should be replica sets.
+
+Applications do not access the shards directly. Instead, the
+:ref:`mongos instances <sharding-mongos>` routes reads and writes from
+applications to the shards.
+
+.. index:: sharding; config servers
+.. index:: config servers
+.. _sharding-config-server:
+
+Config Servers
+--------------
+
+Config servers maintain the shard metadata in a config database. The
+:term:`config database` stores the relationship between :term:`chunks
+<chunk>` and where they reside within a :term:`sharded cluster`. Without
+a config database, the :program:`mongos` instances would be unable to
+route queries or write operations within the cluster.
+
+Config servers *do not* run as replica sets. Instead, a :term:`cluster
+<sharded cluster>` operates with a group of *three* config servers that use a
+two-phase commit process that ensures immediate consistency and
+reliability.
+
+For testing purposes you may deploy a cluster with a single
+config server, but this is not recommended for production.
+
+.. warning::
+
+   If your cluster has a single config server, this
+   :program:`mongod` is a single point of failure.  If the instance is
+   inaccessible the cluster is not accessible. If you cannot recover
+   the data on a config server, the cluster will be inoperable.
+
+   **Always** use three config servers for production deployments.
+
+The actual load on configuration servers is small because each
+:program:`mongos` instances maintains a cached copy of the configuration
+database. MongoDB only writes data to the config server to:
+
+- create splits in existing chunks, which happens as data in
+  existing chunks exceeds the maximum chunk size.
+
+- migrate a chunk between shards.
+
+Additionally, all config servers must be available on initial setup
+of a sharded cluster, each :program:`mongos` instance must be able
+to write to the ``config.version`` collection.
+
+If one or two configuration instances become unavailable, the
+cluster's metadata becomes *read only*. It is still possible to read
+and write data from the shards, but no chunk migrations or splits will
+occur until all three servers are accessible. At the same time, config
+server data is only read in the following situations:
+
+- A new :program:`mongos` starts for the first time, or an existing
+  :program:`mongos` restarts.
+
+- After a chunk migration, the :program:`mongos` instances update
+  themselves with the new cluster metadata.
+
+If all three config servers are inaccessible, you can continue to use
+the cluster as long as you don't restart the :program:`mongos`
+instances until after config servers are accessible again. If you
+restart the :program:`mongos` instances and there are no accessible
+config servers, the :program:`mongos` would be unable to direct
+queries or write operations to the cluster.
+
+Because the configuration data is small relative to the amount of data
+stored in a cluster, the amount of activity is relatively low, and 100%
+up time is not required for a functioning sharded cluster. As a result,
+backing up the config servers is not difficult. Backups of config
+servers are critical as clusters become totally inoperable when
+you lose all configuration instances and data. Precautions to ensure
+that the config servers remain available and intact are critical.
+
+.. note::
+
+   Configuration servers store metadata for a single sharded cluster.
+   You must have a separate configuration server or servers for each
+   cluster you administer.
+
+.. index:: mongos
+.. _sharding-mongos:
+.. _sharding-read-operations:
+
+Mongos Instances
+----------------
+
+The :program:`mongos` provides a single unified interface to a sharded
+cluster for applications using MongoDB. Except for the selection of a
+:term:`shard key`, application developers and administrators need not
+consider any of the :ref:`internal details of sharding <sharding-internals>`.
+
+:program:`mongos` caches data from the :ref:`config server
+<sharding-config-server>`, and uses this to route operations from
+applications and clients to the :program:`mongod` instances.
+:program:`mongos` have no *persistent* state and consume
+minimal system resources.
+
+The most common practice is to run :program:`mongos` instances on the
+same systems as your application servers, but you can maintain
+:program:`mongos` instances on the shards or on other dedicated
+resources.
+
+.. note::
+
+   .. versionchanged:: 2.1
+
+   Some aggregation operations using the :dbcommand:`aggregate`
+   command (i.e. :method:`db.collection.aggregate()`,) will cause
+   :program:`mongos` instances to require more CPU resources than in
+   previous versions. This modified performance profile may dictate
+   alternate architecture decisions if you use the :term:`aggregation
+   framework` extensively in a sharded environment.
+
+.. _sharding-query-routing:
+
+Mongos Routing
+~~~~~~~~~~~~~~
+
+:program:`mongos` uses information from :ref:`config servers
+<sharding-config-server>` to route operations to the cluster as
+efficiently as possible. In general, operations in a sharded
+environment are either:
+
+1. Targeted at a single shard or a limited group of shards based on
+   the shard key.
+
+2. Broadcast to all shards in the cluster that hold documents in a
+   collection.
+
+When possible you should design your operations to be as targeted as
+possible. Operations have the following targeting characteristics:
+
+- Query operations broadcast to all shards [#namespace-exception]_
+  **unless** the :program:`mongos` can determine which shard or shard
+  stores this data.
+
+  For queries that include the shard key, :program:`mongos` can target
+  the query at a specific shard or set of shards, if the portion
+  of the shard key included in the query is a *prefix* of the shard
+  key. For example, if the shard key is:
+
+  .. code-block:: javascript
+
+     { a: 1, b: 1, c: 1 }
+
+  The :program:`mongos` *can* route queries that include the full
+  shard key or either of the following shard key prefixes at a
+  specific shard or set of shards:
+
+  .. code-block:: javascript
+
+     { a: 1 }
+     { a: 1, b: 1 }
+
+  Depending on the distribution of data in the cluster and the
+  selectivity of the query, :program:`mongos` may still have to
+  contact multiple shards [#possible-all]_ to fulfill these queries.
+
+- All :method:`insert() <db.collection.insert()>` operations target to
+  one shard.
+
+- All single :method:`update() <db.collection.update()>` operations
+  target to one shard. This includes :term:`upsert` operations.
+
+- The :program:`mongos` broadcasts multi-update operations to every
+  shard.
+
+- The :program:`mongos` broadcasts :method:`remove()
+  <db.collection.remove()>` operations to every shard unless the
+  operation specifies the shard key in full.
+
+While some operations must broadcast to all shards, you can improve
+performance by using as many targeted operations as possible by
+ensuring that your operations include the shard key.
+
+.. [#namespace-exception] If a shard does not store chunks from a
+   given collection, queries for documents in that collection are not
+   broadcast to that shard.
+
+.. [#a/c-as-a-case-of-a] In this example, a :program:`mongos` could
+   route a query that included ``{ a: 1, c: 1 }`` fields at a specific
+   subset of shards using the ``{ a: 1 }`` prefix. A :program:`mongos`
+   cannot route any of the following queries to specific shards
+   in the cluster:
+
+   .. code-block:: javascript
+
+      { b: 1 }
+      { c: 1 }
+      { b: 1, c: 1 }
+
+.. [#possible-all] :program:`mongos` will route some queries, even
+   some that include the shard key, to all shards, if needed.
+
+Sharded Query Response Process
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To route a query to a :term:`cluster <sharded cluster>`,
+:program:`mongos` uses the following process:
+
+#. Determine the list of :term:`shards <shard>` that must receive the query.
+
+   In some cases, when the :term:`shard key` or a prefix of the shard
+   key is a part of the query, the :program:`mongos` can route the
+   query to a subset of the shards. Otherwise, the :program:`mongos`
+   must direct the query to *all* shards that hold documents for that
+   collection.
+
+   .. example::
+
+      Given the following shard key:
+
+      .. code-block:: javascript
+
+         { zipcode: 1, u_id: 1, c_date: 1 }
+
+      Depending on the distribution of chunks in the cluster, the
+      :program:`mongos` may be able to target the query at a subset of
+      shards, if the query contains the following fields:
+
+      .. code-block:: javascript
+
+         { zipcode: 1 }
+         { zipcode: 1, u_id: 1 }
+         { zipcode: 1, u_id: 1, c_date: 1 }
+
+#. Establish a cursor on all targeted shards.
+
+   When the first batch of results returns from the cursors:
+
+   a. For query with sorted results (i.e. using
+      :method:`cursor.sort()`) the :program:`mongos` performs a merge
+      sort of all queries.
+
+   b. For a query with unsorted results, the :program:`mongos` returns
+      a result cursor that "round robins" results from all cursors on
+      the shards.
+
+      .. versionchanged:: 2.0.5
+         Before 2.0.5, the :program:`mongos` exhausted each cursor,
+         one by one.

source/administration/sharding-architectures.txt

@@ -9,63 +9,49 @@ Sharded Cluster Architectures
 .. default-domain:: mongodb
 
 This document describes the organization and design of :term:`sharded
-cluster` deployments. For documentation of common administrative tasks
-related to sharded clusters, see :doc:`/administration/sharding`. For
-complete documentation of sharded clusters see the :doc:`/sharding`
-section of this manual.
+cluster` deployments.
 
-.. seealso:: :ref:`sharding-requirements`.
-
-Deploying a Test Cluster
-------------------------
-
-.. warning:: Use this architecture for testing and development only.
+Test Cluster Architecture
+-------------------------
 
 You can deploy a very minimal cluster for testing and
 development. These *non-production* clusters have the following
 components:
 
-- 1 :ref:`config server <sharding-config-server>`.
+- One :ref:`config server <sharding-config-server>`.
 
 - At least one :program:`mongod` instance (either :term:`replica sets <replica set>`
   or as a standalone node.)
 
-- 1 :program:`mongos` instance.
+- One :program:`mongos` instance.
+
+.. warning:: Use the test cluster architecture for testing and development only.
 
 .. _sharding-production-architecture:
 
-Deploying a Production Cluster
-------------------------------
+Production Cluster Architecture
+-------------------------------
 
-When deploying a production cluster, you must ensure that the
-data is redundant and that your systems are highly available. To that
-end, a production-level cluster must have the following
-components:
+In a production cluster, you must ensure that data is redundant and that
+your systems are highly available. To that end, a production-level
+cluster must have the following components:
 
-- 3 :ref:`config servers <sharding-config-server>`, each residing on a
+- Three :ref:`config servers <sharding-config-server>`, each residing on a
   discrete system.
 
-  .. note::
+  A single :term:`sharded cluster` must have exclusive use of its
+  :ref:`config servers <sharding-config-server>`. If you have multiple
+  shards, you will need to have a group of config servers for each
+  cluster.
 
-     A single :term:`sharded cluster` must have exclusive use of its
-     :ref:`config servers <sharding-config-server>`. If you have
-     multiple shards, you will need to have a group of config servers
-     for each cluster.
+- Two or more :term:`replica sets <replica set>` to serve as
+  :term:`shards <shard>`. For information on replica sets, see
+  :doc:`/replication`.
 
-- 2 or more :term:`replica sets <replica set>`, for the :term:`shards
-  <shard>`.
-
-  .. see:: For more information on replica sets see
-     :doc:`/administration/replication-architectures` and
-     :doc:`/replication`.
-
-- :program:`mongos` instances. Typically, you will deploy a single
-  :program:`mongos` instance on each application server. Alternatively,
-  you may deploy several `mongos` nodes and let your application connect
-  to these via a load balancer.
-
-.. seealso:: :ref:`sharding-procedure-add-shard` and
-   :ref:`sharding-procedure-remove-shard`.
+- Two or more :program:`mongos` instances. Typically, you deploy a
+  single :program:`mongos` instance on each application server.
+  Alternatively, you may deploy several :program:`mongos` nodes and let
+  your application connect to these via a load balancer.
 
 Sharded and Non-Sharded Data
 ----------------------------
@@ -77,12 +63,10 @@ deployments some databases and collections will use sharding, while
 other databases and collections will only reside on a single database
 instance or replica set (i.e. a :term:`shard`.)
 
-.. note::
-
-   Regardless of the data architecture of your :term:`sharded cluster`,
-   ensure that all queries and operations use the :term:`mongos`
-   router to access the data cluster. Use the :program:`mongos` even
-   for operations that do not impact the sharded data.
+Regardless of the data architecture of your :term:`sharded cluster`,
+ensure that all queries and operations use the :term:`mongos` router to
+access the data cluster. Use the :program:`mongos` even for operations
+that do not impact the sharded data.
 
 Every database has a "primary" [#overloaded-primary-term]_ shard that
 holds all un-sharded collections in that database. All collections
@@ -119,7 +103,7 @@ High Availability and MongoDB
 
 A :ref:`production <sharding-production-architecture>`
 :term:`cluster` has no single point of failure. This section introduces the
-availability concerns for MongoDB deployments, and highlights
+availability concerns for MongoDB deployments and highlights
 potential failure scenarios and available resolutions:
 
 - Application servers or :program:`mongos` instances become unavailable.