From 037fdf6c7f40fcad7b18e5d584cbffaf068722d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Redzy=C5=84ski?= Date: Wed, 18 Dec 2019 10:51:44 +0100 Subject: [PATCH 01/13] pull: add 'too many open files error' solution --- public/static/docs/command-reference/pull.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/public/static/docs/command-reference/pull.md b/public/static/docs/command-reference/pull.md index 482e82b69e..c0fc6c4e80 100644 --- a/public/static/docs/command-reference/pull.md +++ b/public/static/docs/command-reference/pull.md @@ -100,6 +100,18 @@ reflinks or hardlinks to put it in the workspace without copying. See - `-v`, `--verbose` - displays detailed tracing information. +## Common problems + +- `[Errno 24] Too many open files` - most common for MacOs users who work with +s3 remote. Each download thread needs to open one or more file descriptor(s). +The more `jobs` is specified, the more file descriptors needs to be open on +host filesystem. If there is too many open file descriptors, host filesystem +will not allow opening new ones. Recommended approach to handle this error is +to increase open file descriptors limit. On UNIX systems that can be done using + `ulimit -n`. In case of Windows, please refer to [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). + If changing limits is impossible, the workaround would be to use lower `jobs` + option value. + ## Examples For using the `dvc pull` command, a remote storage must be defined. (See From 9ccc2cd6ffc1499bca1ad0d2ef94fd72e5a5f2bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Redzy=C5=84ski?= Date: Wed, 18 Dec 2019 10:55:39 +0100 Subject: [PATCH 02/13] push: add 'too many open files error' solution --- public/static/docs/command-reference/pull.md | 17 +++++++++-------- public/static/docs/command-reference/push.md | 13 +++++++++++++ 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/public/static/docs/command-reference/pull.md b/public/static/docs/command-reference/pull.md index c0fc6c4e80..8118545895 100644 --- a/public/static/docs/command-reference/pull.md +++ b/public/static/docs/command-reference/pull.md @@ -103,14 +103,15 @@ reflinks or hardlinks to put it in the workspace without copying. See ## Common problems - `[Errno 24] Too many open files` - most common for MacOs users who work with -s3 remote. Each download thread needs to open one or more file descriptor(s). -The more `jobs` is specified, the more file descriptors needs to be open on -host filesystem. If there is too many open file descriptors, host filesystem -will not allow opening new ones. Recommended approach to handle this error is -to increase open file descriptors limit. On UNIX systems that can be done using - `ulimit -n`. In case of Windows, please refer to [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). - If changing limits is impossible, the workaround would be to use lower `jobs` - option value. + s3 remote. Each download thread needs to open one or more file descriptor(s). + The more `jobs` is specified, the more file descriptors needs to be open on + host filesystem. If there is too many open file descriptors, host filesystem + will not allow opening new ones. Recommended approach to handle this error is + to increase open file descriptors limit. On UNIX systems that can be done + using `ulimit -n`. In case of Windows, please refer to + [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). + If changing limits is impossible, the workaround would be to use lower `jobs` + option value. ## Examples diff --git a/public/static/docs/command-reference/push.md b/public/static/docs/command-reference/push.md index 629c9fc2dc..941d0253ae 100644 --- a/public/static/docs/command-reference/push.md +++ b/public/static/docs/command-reference/push.md @@ -104,6 +104,19 @@ to push. - `-v`, `--verbose` - displays detailed tracing information. +## Common problems + +- `[Errno 24] Too many open files` - most common for MacOs users who work with + s3 remote. Each upload thread needs to open one or more file descriptor(s). + The more `jobs` is specified, the more file descriptors needs to be open on + host filesystem. If there is too many open file descriptors, host filesystem + will not allow opening new ones. Recommended approach to handle this error is + to increase open file descriptors limit. On UNIX systems that can be done + using `ulimit -n`. In case of Windows, please refer to + [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). + If changing limits is impossible, the workaround would be to use lower `jobs` + option value. + ## Examples For using the `dvc push` command, a remote storage must be defined. (See From 7aa1b779c94c252b7c509b8ead6b5149880ebf39 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Redzy=C5=84ski?= Date: Thu, 19 Dec 2019 15:35:00 +0100 Subject: [PATCH 03/13] push/pull: roll back common problems, add troubleshooting guide --- public/static/docs/command-reference/pull.md | 13 ------------- public/static/docs/command-reference/push.md | 13 ------------- static/docs/user-guide/troubleshooting.md | 17 +++++++++++++++++ 3 files changed, 17 insertions(+), 26 deletions(-) create mode 100644 static/docs/user-guide/troubleshooting.md diff --git a/public/static/docs/command-reference/pull.md b/public/static/docs/command-reference/pull.md index 8118545895..482e82b69e 100644 --- a/public/static/docs/command-reference/pull.md +++ b/public/static/docs/command-reference/pull.md @@ -100,19 +100,6 @@ reflinks or hardlinks to put it in the workspace without copying. See - `-v`, `--verbose` - displays detailed tracing information. -## Common problems - -- `[Errno 24] Too many open files` - most common for MacOs users who work with - s3 remote. Each download thread needs to open one or more file descriptor(s). - The more `jobs` is specified, the more file descriptors needs to be open on - host filesystem. If there is too many open file descriptors, host filesystem - will not allow opening new ones. Recommended approach to handle this error is - to increase open file descriptors limit. On UNIX systems that can be done - using `ulimit -n`. In case of Windows, please refer to - [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). - If changing limits is impossible, the workaround would be to use lower `jobs` - option value. - ## Examples For using the `dvc pull` command, a remote storage must be defined. (See diff --git a/public/static/docs/command-reference/push.md b/public/static/docs/command-reference/push.md index 941d0253ae..629c9fc2dc 100644 --- a/public/static/docs/command-reference/push.md +++ b/public/static/docs/command-reference/push.md @@ -104,19 +104,6 @@ to push. - `-v`, `--verbose` - displays detailed tracing information. -## Common problems - -- `[Errno 24] Too many open files` - most common for MacOs users who work with - s3 remote. Each upload thread needs to open one or more file descriptor(s). - The more `jobs` is specified, the more file descriptors needs to be open on - host filesystem. If there is too many open file descriptors, host filesystem - will not allow opening new ones. Recommended approach to handle this error is - to increase open file descriptors limit. On UNIX systems that can be done - using `ulimit -n`. In case of Windows, please refer to - [official resources](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/). - If changing limits is impossible, the workaround would be to use lower `jobs` - option value. - ## Examples For using the `dvc push` command, a remote storage must be defined. (See diff --git a/static/docs/user-guide/troubleshooting.md b/static/docs/user-guide/troubleshooting.md new file mode 100644 index 0000000000..b3d3a973cf --- /dev/null +++ b/static/docs/user-guide/troubleshooting.md @@ -0,0 +1,17 @@ +# Troubleshooting + +In this section we cover some of the known issues that DVC user might stumbe +upon. + +## Too many open files error + +A known problem some users run into with `pull`, `fetch` and `push` commands is +`[Errno 24] Too many open files` (most common for S3 remotes on MacOS). The more +`--jobs` specified, the more file descriptors need to be open on the host file +system for each download thread, and the limit may be reached, causing this +error. + +To solve this, it's often possible to increase the open file descriptors limit, +for example with `ulimit -n` on UNIX-like system, or +[increasing Handles limit](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/) +on Windows. Otherwise, please try using a lower `JOBS` value. From a473a5bbc73106c55902a5bdc54126de2b377c4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Redzy=C5=84ski?= Date: Fri, 20 Dec 2019 09:58:51 +0100 Subject: [PATCH 04/13] Update static/docs/user-guide/troubleshooting.md Co-Authored-By: Jorge Orpinel --- static/docs/user-guide/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/docs/user-guide/troubleshooting.md b/static/docs/user-guide/troubleshooting.md index b3d3a973cf..0fb5f2e3fc 100644 --- a/static/docs/user-guide/troubleshooting.md +++ b/static/docs/user-guide/troubleshooting.md @@ -5,7 +5,7 @@ upon. ## Too many open files error -A known problem some users run into with `pull`, `fetch` and `push` commands is +A known problem some users run into with the `dvc pull`, `dvc fetch` and `dvc push` commands is `[Errno 24] Too many open files` (most common for S3 remotes on MacOS). The more `--jobs` specified, the more file descriptors need to be open on the host file system for each download thread, and the limit may be reached, causing this From 775ea17770af4b520c0855278204f00d1c514fc4 Mon Sep 17 00:00:00 2001 From: pawel Date: Fri, 27 Dec 2019 04:50:14 +0100 Subject: [PATCH 05/13] troubleshooting: add sidebar.json entry --- public/static/docs/sidebar.json | 4 ++++ static/docs/user-guide/troubleshooting.md | 16 ++++++++-------- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/public/static/docs/sidebar.json b/public/static/docs/sidebar.json index c92333fbb3..1f5eaefd6e 100644 --- a/public/static/docs/sidebar.json +++ b/public/static/docs/sidebar.json @@ -156,6 +156,10 @@ "label": "Running DVC on Windows", "slug": "running-dvc-on-windows" }, + { + "label": "Troubleshooting guide", + "slug": "troubleshooting" + }, { "label": "Anonymized Usage Analytics", "slug": "analytics" diff --git a/static/docs/user-guide/troubleshooting.md b/static/docs/user-guide/troubleshooting.md index 0fb5f2e3fc..af4ca9c1f3 100644 --- a/static/docs/user-guide/troubleshooting.md +++ b/static/docs/user-guide/troubleshooting.md @@ -1,17 +1,17 @@ # Troubleshooting -In this section we cover some of the known issues that DVC user might stumbe -upon. +In this section we provide help for some of the problems that DVC user might +stumbe upon. ## Too many open files error -A known problem some users run into with the `dvc pull`, `dvc fetch` and `dvc push` commands is -`[Errno 24] Too many open files` (most common for S3 remotes on MacOS). The more -`--jobs` specified, the more file descriptors need to be open on the host file -system for each download thread, and the limit may be reached, causing this -error. +A known problem some users run into with the `dvc pull`, `dvc fetch` and +`dvc push` commands is `[Errno 24] Too many open files` (most common for S3 +remotes on MacOS). The more `--jobs` specified, the more file descriptors need +to be open on the host file system for each download thread, and the limit may +be reached, causing this error. To solve this, it's often possible to increase the open file descriptors limit, -for example with `ulimit -n` on UNIX-like system, or +with `ulimit` on UNIX-like system (for example `ulimit -n 1024`), or [increasing Handles limit](https://blogs.technet.microsoft.com/markrussinovich/2009/09/29/pushing-the-limits-of-windows-handles/) on Windows. Otherwise, please try using a lower `JOBS` value. From d2293b58de266938ad5732a7786a6736a8f5ce39 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 10 Jan 2020 11:56:57 -0600 Subject: [PATCH 06/13] user-guide: add HTTP redirect from err.dvc.org/{header} to troubleshooting guide header per https://github.com/iterative/dvc.org/pull/875#issuecomment-573035112 --- server.js | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/server.js b/server.js index 1c1e079440..dc92181499 100644 --- a/server.js +++ b/server.js @@ -45,6 +45,13 @@ app.prepare().then(() => { : pathname) }) res.end() + } else if (req.headers.host === 'err.dvc.org') { + // err.dvc.org/{hdr} -> dvc.org/doc/user-guide/troubleshooting#{hdr}, + res.writeHead(301, { + 'Cache-Control': 'no-cache', + Location: `https://dvc.org/doc/user-guide/troubleshooting#${pathname}` + }) + res.end() } else if (/^(code|data|remote)\.dvc\.org$/.test(req.headers.host)) { // {code/data/remote}.dvc.org -> corresponding S3 bucket res.writeHead(301, { From 7d4887ea278657329852750669efefeecccf567a Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 10 Jan 2020 12:47:21 -0600 Subject: [PATCH 07/13] user-guide: move troubleshooting guide to correct location --- {static => public/static}/docs/user-guide/troubleshooting.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {static => public/static}/docs/user-guide/troubleshooting.md (100%) diff --git a/static/docs/user-guide/troubleshooting.md b/public/static/docs/user-guide/troubleshooting.md similarity index 100% rename from static/docs/user-guide/troubleshooting.md rename to public/static/docs/user-guide/troubleshooting.md From ad1b786f013605b49073e40442aa4777fe92fa34 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 10 Jan 2020 12:47:56 -0600 Subject: [PATCH 08/13] user-guide: force header id in troubleshooting guide per https://github.com/iterative/dvc.org/pull/875#discussion_r365360659 --- public/static/docs/user-guide/troubleshooting.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/public/static/docs/user-guide/troubleshooting.md b/public/static/docs/user-guide/troubleshooting.md index af4ca9c1f3..c693434609 100644 --- a/public/static/docs/user-guide/troubleshooting.md +++ b/public/static/docs/user-guide/troubleshooting.md @@ -1,9 +1,9 @@ # Troubleshooting In this section we provide help for some of the problems that DVC user might -stumbe upon. +stumble upon. -## Too many open files error +

Too many open files error

A known problem some users run into with the `dvc pull`, `dvc fetch` and `dvc push` commands is `[Errno 24] Too many open files` (most common for S3 From c2dedde9f8234190e3c1cce6d1040633e70adcbb Mon Sep 17 00:00:00 2001 From: pawel Date: Mon, 13 Jan 2020 16:29:36 +0100 Subject: [PATCH 09/13] server: rename err to error --- server.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/server.js b/server.js index dc92181499..1341f7373a 100644 --- a/server.js +++ b/server.js @@ -45,8 +45,8 @@ app.prepare().then(() => { : pathname) }) res.end() - } else if (req.headers.host === 'err.dvc.org') { - // err.dvc.org/{hdr} -> dvc.org/doc/user-guide/troubleshooting#{hdr}, + } else if (req.headers.host === 'error.dvc.org') { + // error.dvc.org/{hdr} -> dvc.org/doc/user-guide/troubleshooting#{hdr}, res.writeHead(301, { 'Cache-Control': 'no-cache', Location: `https://dvc.org/doc/user-guide/troubleshooting#${pathname}` From 6630b4641a2d197d95e17c5cf01ae6003e20ea3b Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 14 Jan 2020 17:21:36 -0600 Subject: [PATCH 10/13] server: use HTTP 303 for man. redirect and refactor redirect code for consistency with existing ones per https://github.com/iterative/dvc.org/pull/875#pullrequestreview-341328648 --- server.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/server.js b/server.js index 1341f7373a..2481eaf603 100644 --- a/server.js +++ b/server.js @@ -47,9 +47,9 @@ app.prepare().then(() => { res.end() } else if (req.headers.host === 'error.dvc.org') { // error.dvc.org/{hdr} -> dvc.org/doc/user-guide/troubleshooting#{hdr}, - res.writeHead(301, { + res.writeHead(303, { 'Cache-Control': 'no-cache', - Location: `https://dvc.org/doc/user-guide/troubleshooting#${pathname}` + Location: 'https://dvc.org/doc/user-guide/troubleshooting#' + pathname }) res.end() } else if (/^(code|data|remote)\.dvc\.org$/.test(req.headers.host)) { From d5340e6b52434a8e7488e2bdf2fc5dbf2cce5e86 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 14 Jan 2020 19:10:24 -0600 Subject: [PATCH 11/13] user-guide: hardcode short id/href in troubleshooting header per https://github.com/iterative/dvc.org/pull/875#discussion_r366648445 --- public/static/docs/user-guide/troubleshooting.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/public/static/docs/user-guide/troubleshooting.md b/public/static/docs/user-guide/troubleshooting.md index c693434609..d3ade26caf 100644 --- a/public/static/docs/user-guide/troubleshooting.md +++ b/public/static/docs/user-guide/troubleshooting.md @@ -3,7 +3,8 @@ In this section we provide help for some of the problems that DVC user might stumble upon. -

Too many open files error

+ +

Too many open files error

A known problem some users run into with the `dvc pull`, `dvc fetch` and `dvc push` commands is `[Errno 24] Too many open files` (most common for S3 From 5199aa015c83504def72ea075a985f9e322bf171 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 14 Jan 2020 19:32:46 -0600 Subject: [PATCH 12/13] user-guide: remove long hardcoded HTML code, leave simple h2 with id for troubleshooting header per https://github.com/iterative/dvc.org/commit/d5340e6b52434a8e7488e2bdf2fc5dbf2cce5e86#r36802142 --- public/static/docs/user-guide/troubleshooting.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/public/static/docs/user-guide/troubleshooting.md b/public/static/docs/user-guide/troubleshooting.md index d3ade26caf..2d7ed118da 100644 --- a/public/static/docs/user-guide/troubleshooting.md +++ b/public/static/docs/user-guide/troubleshooting.md @@ -3,8 +3,8 @@ In this section we provide help for some of the problems that DVC user might stumble upon. - -

Too many open files error

+ +

Too many open files error

A known problem some users run into with the `dvc pull`, `dvc fetch` and `dvc push` commands is `[Errno 24] Too many open files` (most common for S3 From 3472e5b7225e45d385eb9d1056de127496325a57 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 14 Jan 2020 19:39:23 -0600 Subject: [PATCH 13/13] user-guide: remove "guide" from Troubleshooting nav item --- public/static/docs/sidebar.json | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/public/static/docs/sidebar.json b/public/static/docs/sidebar.json index 1f5eaefd6e..e83d1c776a 100644 --- a/public/static/docs/sidebar.json +++ b/public/static/docs/sidebar.json @@ -156,10 +156,7 @@ "label": "Running DVC on Windows", "slug": "running-dvc-on-windows" }, - { - "label": "Troubleshooting guide", - "slug": "troubleshooting" - }, + "troubleshooting", { "label": "Anonymized Usage Analytics", "slug": "analytics"