A urllib3 connection is not put back in the pool after HEAD #1248
Labels
dependencies
This issue is a problem in a dependency.
enhancement
This issue requests an improvement to a current feature.
Comments
|
Here's an example to reproduce the issue: I can reproduce it after HEAD returns 404, but not after 200. |
joguSD
added
dependency
enhancement
|
I can confirm that using the above snippet in python 2.7.13 will fail to return the connection back into the pool. Thanks for bringing this to our attention. |
joguSD
added
the
investigating
|
Spent more time debugging this. It's not actually fixed in urllib3 and requests as I original thought. I will open an issue with urllib3 and then with requests about this. |
|
Filed bugs in urllib3 (urllib3/urllib3#1234) and requests (psf/requests#4199), along with a PR to fix this in urllib3 (urllib3/urllib3#1235). There is a packaging issue for botocore. New versions of requests do not vendor urllib3 anymore. Does botocore want to continue vendoring urllib3 (previously indirectly through requests)? That would require adding a hack to register it so that requests can find it. Or if boto does not vendor urllib3, the dependency can be set in requirements. |
|
Upgrading |
|
Seems that this was fixed upstream in: urllib3/urllib3#1235 |
|
@joguSD do you know what the plan for botocore regarding urllib3 is? Currently, botocore vendors an old Happy to submit a PR if there is some clarity on what the right path forward is. |
|
We currently vendor requests as we make some modifications to it, so upgrading it isn't straight forward. I'll bring this up and see what path we want to take moving forward. At least personally I think moving away from vendoring requests would be ideal so we can get upstream bug fixes and avoid issues when some some distributions (ubuntu for example) re-package our library. I can't make any promises though, and I wasn't around when the decision to vendor requests was made so once I hear the reasons behind that my opinion may change. |
genben
added a commit
to miarec/botocore
that referenced
this issue
Oct 19, 2017
Issue "A urllib3 connection is not put back in the pool after HEAD" boto#1248
|
@joguSD do you think maybe the |
joguSD
removed
the
investigating
Closed
|
Another reason to update urllib3: boto/boto3#1053 |
|
Another reason: ipv6 proxy supprt. |
ezyang
added a commit
to ezyang/botocore
that referenced
this issue
Feb 13, 2018
…ixes boto#1248 Signed-off-by: Edward Z. Yang <ezyang@fb.com>
|
Any update on this fix? |
|
We are also experiencing this issue now. We desperately need a new urllib to fix multiple bugs. Boto3 is currently not usable for our company. |
This was referenced Aug 23, 2018
|
We have un-vendored requests/urllib3 and are now using urllib3 directly as of |
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jun 26, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jun 27, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jun 27, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jul 12, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jul 29, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Jul 29, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Aug 6, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Aug 6, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
DinoBektesevic
added a commit
to DinoBektesevic/daf_butler
that referenced
this issue
Aug 6, 2019
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
timj
pushed a commit
to lsst/resources
that referenced
this issue
Nov 29, 2021
Hardcoded Butler.get from S3 storage for simplest of Datasets with no
- rebased to include newest changes to daf_butler (namely, ButlerURI)
- removed parsePathToUriElements, S3Location and S3LocationFactory,
and replaced all path handling with ButlerURI and its test cases.
- added transaction checks back into S3Datastore. Unsure on proper
usage.
- added more path manipulation methods/properties to LocationFactory
and Location.
* bucketName returns the name of the bucket in the Location/
LocationFactory
* Location has a pathInBucket property that will convert posix
style paths to S3 protocol style paths. The main difference is
with respect to leading and trailing separators. S3 interprets
`/path/`, `/path`, `path/` and `path` keys differently, even
though some of them are equivalent on a POSIX compliant system.
So what `/path/to/file.ext` would be on a POSIX system, on S3
it would read `path/to/file.ext` and the bucket is referenced
separately with boto3.
- For saving Config as file, moved all the URI handling logic into
Config and out of Butler makeRepo. The only logic there is root
directory creation.
* The call to save config as a file at the butler root directory
is now done through dumpToUri which then resolves the
appropriate backend method to call.
- Improved(?) on the proposed scheme for checking if we are
dealing with a file or a directory in absence of the trailing
path separator.
* Noted some differences between the requested generality of code
in the review for writing to files and inits of config classes.
For inits it seems as if there's an 2+ year old commit limiting
the Config files to `yaml` type files only. However, the review
implied that `dumpToFile` on Config class should be file format
independent. Then, for `dumpTo*` methods, to check whether we
have a dir or a file I only inspect whether the path ends on a
'something.something' style. However, since I can count on
files having to be `yaml` type and having `yaml` extensions in
inits I use a simplified logic to determine if we have a dir or
file. It is possible to generalize inits to other filetypes, as
long as they have an extension added to them.
* I assume we do not want to force users to be mindfull of
trailing separators.
- Now raising errors on unrecognized schemes on all `if scheme ==`
patterns.
- Closed the StreamIO in Config.dumpToFile
- fixed up the Formatters to return bytes instead of strings. The
fromBytes methods now expect bytes as well. JSON and YAML were
main culprints. Fixed the docs for them. At this point I am
confident I just overwrite the fixes when rewinding changes on
rebase by accident because I have done that before, twice.
- Added a different way to check if files exist, cheaper but can
be slower. From boto/botocore#1248 it
is my understanding that this should not be an issue anymore.
But the newer boto3 versions are slow to hit package managers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issuing a HEAD request results in the connection not being put back into the urllib3 pool.
This is tied to this issue in urllib3: urllib3/urllib3#673. The problem is resolved in urllib3 >= 1.11. However, botocore vendors requests 2.7.0, which vendors urllib3 1.10.4. Updating the vendored requests library to >= 2.8.0 would resolve this issue.Relevant issue in urllib3: urllib3/urllib3#1234
The text was updated successfully, but these errors were encountered: