Merge branch 'master' of https://github.com/python/cpython

* 'master' of https://github.com/python/cpython: (550 commits)
  bpo-31867: Remove duplicates in default mimetypes. (python#4388)
  tokenizer: Remove unused tabs options (python#4422)
  bpo-31691: Specify where to find build instructions for the Windows installer (python#4426)
  Fix typo in atexit documentation. (pythonGH-4419)
  bpo-31702: Allow to specify rounds for SHA-2 hashing in crypt.mksalt(). (python#4110)
  bpo-32043: New "developer mode": "-X dev" option (python#4413)
  bpo-30349: Raise FutureWarning for nested sets and set operations (python#1553)
  bpo-32037: Use the INT opcode for 32-bit integers in protocol 0 pickles. (python#4407)
  bpo-30143: 2to3 now generates a code that uses abstract collection classes (python#1262)
  bpo-32030: Enhance Py_Main() (python#4412)
  bpo-32030: Split Py_Main() into subfunctions (python#4399)
  bpo-32034: Make IncompleteReadError & LimitOverrunError pickleable python#4409
  bpo-32025: Add time.thread_time() (python#4410)
  bpo-32018: Fix inspect.signature repr to follow PEP 8 (python#4408)
  bpo-30399: Get rid of trailing comma in the repr of BaseException. (python#1650)
  bpo-30950: Convert round() to Argument Clinic. (python#2740)
  bpo-32011: Revert "Issue python#15480: Remove the deprecated and unused TYPE_INT64 code from marshal." (python#4381)
  bpo-32023: Disallow genexprs without parenthesis in class definitions. (python#4400)
  bpo-31949: Fixed several issues in printing tracebacks (PyTraceBack_Print()). (python#4289)
  bpo-32032: Test both implementations of module-level pickle API. (python#4401)
  ...

Author: jimmylai

.gitattributes

@@ -27,9 +27,6 @@ Lib/test/test_email/data/*.txt -text
 Lib/test/xmltestdata/* -text
 Lib/test/coding20731.py -text
 
-# Special files in third party code
-Modules/zlib/zlib.map -text
-
 # CRLF files
 *.bat text eol=crlf
 *.ps1 text eol=crlf

.github/CODEOWNERS

@@ -24,19 +24,30 @@
 Python/bootstrap_hash.c       @python/crypto-team
 
 # Email and related
-**/*mail*                     @bitdancer
-**/*smtp*                     @bitdancer
-**/*mime*                     @bitdancer
-**/*imap*                     @bitdancer
-**/*poplib*                   @bitdancer
+**/*mail*                     @python/email-team
+**/*smtp*                     @python/email-team
+**/*mime*                     @python/email-team
+**/*imap*                     @python/email-team
+**/*poplib*                   @python/email-team
 
 # subprocess
 **/*subprocess*               @gpshead
 
 # Windows
 /PC/                          @python/windows-team
-/PCBuild/                     @python/windows-team
+/PCbuild/                     @python/windows-team
 
 # Windows installer packages
 /Tools/msi/                   @python/windows-team
 /Tools/nuget/                 @python/windows-team
+
+**/*itertools*                @rhettinger
+**/*collections*              @rhettinger
+**/*random*                   @rhettinger
+**/*queue*                    @rhettinger
+**/*bisect*                   @rhettinger
+**/*heapq*                    @rhettinger
+**/*functools*                @ncoghlan @rhettinger
+**/*decimal*                  @rhettinger @skrah
+
+**/*idlelib*                  @terryjreedy

.github/CODE_OF_CONDUCT.rst

@@ -0,0 +1,14 @@
+Code of Conduct
+===============
+
+Please note that all interactions on
+`Python Software Foundation <https://www.python.org/psf-landing/>`__-supported
+infrastructure is `covered
+<https://www.python.org/psf/records/board/minutes/2014-01-06/#management-of-the-psfs-web-properties>`__
+by the `PSF Code of Conduct <https://www.python.org/psf/codeofconduct/>`__,
+which includes all infrastructure used in the development of Python itself
+(e.g. mailing lists, issue trackers, GitHub, etc.).
+
+In general this means everyone is expected to be open, considerate, and
+respectful of others no matter what their position is within the project.
+

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,10 @@
+!!! If this is a backport PR (PR made against branches other than `master`),
+please ensure that the PR title is in the following format:
+```
+[X.Y] <title from the original PR> (GH-NNNN)
+```
+Where: [X.Y] is the branch name, e.g. [3.6].
+
+GH-NNNN refers to the PR number from `master`.
+
+PLEASE: Remove this headline!!!

.github/appveyor.yml

@@ -5,13 +5,17 @@ branches:
     - master
     - /\d\.\d/
     - buildbot-custom
+cache:
+  - externals -> PCbuild\*
 build_script:
 - cmd: PCbuild\build.bat -e
 - cmd: PCbuild\win32\python.exe -m test.pythoninfo
 test_script:
 - cmd: PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 --fail-env-changed -j0
 environment:
   HOST_PYTHON: C:\Python36\python.exe
+image:
+- Visual Studio 2017
 
 # Only trigger AppVeyor if actual code or its configuration changes
 only_commits:
@@ -24,7 +28,7 @@ only_commits:
     - Modules/
     - Objects/
     - PC/
-    - PCBuild/
+    - PCbuild/
     - Parser/
     - Programs/
     - Python/

.gitignore

@@ -19,6 +19,9 @@
 .gdb_history
 Doc/build/
 Doc/venv/
+Doc/.venv/
+Doc/env/
+Doc/.env/
 Include/pydtrace_probes.h
 Lib/distutils/command/*.pdb
 Lib/lib2to3/*.pickle

.travis.yml

@@ -3,8 +3,10 @@ dist: trusty
 sudo: false
 group: beta
 
-# To cache doc-building dependencies.
-cache: pip
+# To cache doc-building dependencies and C compiler output.
+cache:
+    - pip
+    - ccache
 
 branches:
   only:
@@ -23,15 +25,6 @@ matrix:
       # compiler here and the other to run the coverage build. Clang is preferred
       # in this instance for its better error messages.
       env: TESTING=cpython
-    - os: osx
-      language: c
-      compiler: clang
-      # Testing under macOS is optional until testing stability has been demonstrated.
-      env: OPTIONAL=true
-      before_install:
-        - brew install openssl xz
-        - export CPPFLAGS="-I$(brew --prefix openssl)/include"
-        - export LDFLAGS="-L$(brew --prefix openssl)/lib"
     - os: linux
       language: python
       # Build the docs against a stable version of Python so code bugs don't hold up doc-related PRs.
@@ -41,7 +34,7 @@ matrix:
         - cd Doc
         # Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
         # (Updating the version is fine as long as no warnings are raised by doing so.)
-        - python -m pip install sphinx~=1.6.1
+        - python -m pip install sphinx~=1.6.1 blurb
       script:
         - make check suspicious html SPHINXOPTS="-q -W -j4"
     - os: linux
@@ -63,7 +56,7 @@ matrix:
             ./venv/bin/python -m test.pythoninfo
       script:
         # Skip tests that re-run the entire test suite.
-        - ./venv/bin/python -m coverage run --pylib -m test -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn
+        - ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures
       after_script:  # Probably should be after_success once test suite updated to run under coverage.py.
         # Make the `coverage` command available to Codecov w/ a version of Python that can parse all source files.
         - source ./venv/bin/activate
@@ -98,6 +91,8 @@ script:
   - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then ./python Tools/scripts/patchcheck.py --travis $TRAVIS_PULL_REQUEST; fi
   # `-r -w` implicitly provided through `make buildbottest`.
   - make buildbottest TESTOPTS="-j4 -uall,-cpu"
+  # Check that all symbols exported by libpython start with "Py" or "_Py"
+  - make smelly
 
 notifications:
   email: false

Doc/Makefile

@@ -5,7 +5,9 @@
 
 # You can set these variables from the command line.
 PYTHON       = python3
-SPHINXBUILD  = sphinx-build
+VENVDIR      = ./venv
+SPHINXBUILD  = PATH=$(VENVDIR)/bin:$$PATH sphinx-build
+BLURB        = PATH=$(VENVDIR)/bin:$$PATH blurb
 PAPER        =
 SOURCES      =
 DISTVERSION  = $(shell $(PYTHON) tools/extensions/patchlevel.py)
@@ -38,6 +40,20 @@ help:
 	@echo "  serve      to serve the documentation on the localhost (8000)"
 
 build:
+	-mkdir -p build
+# Look first for a Misc/NEWS file (building from a source release tarball
+# or old repo) and use that, otherwise look for a Misc/NEWS.d directory
+# (building from a newer repo) and use blurb to generate the NEWS file.
+	@if [ -f  ../Misc/NEWS ] ; then \
+		echo "Using existing Misc/NEWS file"; \
+		cp ../Misc/NEWS build/NEWS; \
+	elif [ -d ../Misc/NEWS.d ]; then \
+		echo "Building NEWS from Misc/NEWS.d with blurb"; \
+		$(BLURB) merge -f build/NEWS; \
+	else \
+		echo "Neither Misc/NEWS.d nor Misc/NEWS found; cannot build docs"; \
+		exit 1; \
+	fi
 	$(SPHINXBUILD) $(ALLSPHINXOPTS)
 	@echo
 
@@ -103,11 +119,12 @@ htmlview: html
 	 $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')"
 
 clean:
-	-rm -rf build/* venv/*
+	-rm -rf build/* $(VENVDIR)/*
 
 venv:
-	$(PYTHON) -m venv venv
-	./venv/bin/python3 -m pip install -U Sphinx
+	$(PYTHON) -m venv $(VENVDIR)
+	$(VENVDIR)/bin/python3 -m pip install -U Sphinx blurb
+	@echo "The venv has been created in the $(VENVDIR) directory"
 
 dist:
 	rm -rf dist
@@ -153,7 +170,7 @@ dist:
 	cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub
 
 check:
-	$(PYTHON) tools/rstlint.py -i tools -i venv -i README.rst
+	$(PYTHON) tools/rstlint.py -i tools -i $(VENVDIR) -i README.rst
 
 serve:
 	../Tools/scripts/serve.py build/html

Doc/README.rst

@@ -14,38 +14,46 @@ developers guide.
 Building the docs
 =================
 
-You need to have `Sphinx <http://sphinx-doc.org/>`_ installed; it is the toolset
-used to build the docs.  It is not included in this tree, but maintained
-separately and `available from PyPI <https://pypi.python.org/pypi/Sphinx>`_.
+The documentation is built with several tools which are not included in this
+tree but are maintained separately and are available from
+`PyPI <https://pypi.org/>`_.
+
+* `Sphinx <https://pypi.org/project/Sphinx/>`_
+* `blurb <https://pypi.org/project/blurb/>`_
+
+The easiest way to install these tools is to create a virtual environment and
+install the tools into there.
 
 
 Using make
 ----------
 
-A Makefile has been prepared so that on Unix, provided you have installed
-Sphinx, you can just run ::
+To get started on UNIX, you can create a virtual environment with the command ::
 
-   make html
+  make venv
 
-to build the HTML output files.
-
-On Windows, we try to emulate the Makefile as closely as possible with a
-``make.bat`` file.
+That will install all the tools necessary to build the documentation. Assuming
+the virtual environment was created in the ``env`` directory (the default;
+configurable with the VENVDIR variable), you can run the following command to
+build the HTML output files::
 
-To use a Python interpreter that's not called ``python``, use the standard
-way to set Makefile variables, using e.g. ::
+  make html
 
-   make html PYTHON=python3
+By default, if the virtual environment is not created, the Makefile will
+look for instances of sphinxbuild and blurb installed on your process PATH
+(configurable with the SPHINXBUILD and BLURB variables).
 
-On Windows, set the PYTHON environment variable instead.
-
-To use a specific sphinx-build (something other than ``sphinx-build``), set
-the SPHINXBUILD variable.
+On Windows, we try to emulate the Makefile as closely as possible with a
+``make.bat`` file. If you need to specify the Python interpreter to use,
+set the PYTHON environment variable instead.
 
 Available make targets are:
 
 * "clean", which removes all build files.
 
+* "venv", which creates a virtual environment with all necessary tools
+  installed.
+
 * "html", which builds standalone HTML files for offline viewing.
 
 * "htmlview", which re-uses the "html" builder, but then opens the main page
@@ -96,7 +104,7 @@ Available make targets are:
 Without make
 ------------
 
-Install the Sphinx package and its dependencies from PyPI.
+First, install the tool dependencies from PyPI.
 
 Then, from the ``Doc`` directory, run ::
 
@@ -112,8 +120,7 @@ Contributing
 Bugs in the content should be reported to the
 `Python bug tracker <https://bugs.python.org>`_.
 
-Bugs in the toolset should be reported in the
-`Sphinx bug tracker <https://github.com/sphinx-doc/sphinx/issues>`_.
+Bugs in the toolset should be reported to the tools themselves.
 
 You can also send a mail to the Python Documentation Team at docs@python.org,
 and we will process your request as soon as possible.

Doc/c-api/buffer.rst

@@ -473,7 +473,7 @@ Buffer-related functions
    (*order* is ``'A'``).  Return ``0`` otherwise.
 
 
-.. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char order)
+.. c:function:: void PyBuffer_FillContiguousStrides(int ndims, Py_ssize_t *shape, Py_ssize_t *strides, int itemsize, char order)
 
    Fill the *strides* array with byte-strides of a :term:`contiguous` (C-style if
    *order* is ``'C'`` or Fortran-style if *order* is ``'F'``) array of the