Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Fixing possible bugs in the CI #23727

Merged
merged 12 commits into from
Nov 24, 2018
Merged

CI: Fixing possible bugs in the CI #23727

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
57aca9a
Fixing possible bugs in the CI
datapythonista Nov 15, 2018
08ccb80
Moving Python 3.7 test to 3.6, as moto is failing on 3.7 at the moment
datapythonista Nov 20, 2018
cfc89a5
Testing with Italian locale (was supposed to be tested, but Chinese w…
datapythonista Nov 20, 2018
c8d20b1
Restoring -m 'not single and slow' to see if removing it caused the v…
datapythonista Nov 20, 2018
9718fd5
callng multi/slow tests in a consistent way with the rest
datapythonista Nov 21, 2018
b3f5744
Reducing verbosity of slow tests, to avoid reaching the maximum size …
datapythonista Nov 21, 2018
22b8fc2
restoring -m slow so the build passes, and added comment explaining why
datapythonista Nov 21, 2018
5afef71
Moving azure 37 build back to 3.7, and installing moto from pip, as c…
datapythonista Nov 21, 2018
6f17fe5
Filtering xlrd warnings in Python 3.7
datapythonista Nov 21, 2018
23d0c17
Merging from master
datapythonista Nov 21, 2018
d08c0a8
Merge remote-tracking branch 'upstream/master' into ci_bugs
datapythonista Nov 23, 2018
e1a6f89
Using a context manager to ignore xlrd warnings
datapythonista Nov 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ci/azure/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
CONDA_ENV: pandas
TEST_ARGS: "--skip-slow --skip-network"

py36_locale:
py37_locale:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In CONDA_PY we have 37, as well as in the yaml file, so I assume the name is wrong.

ENV_FILE: ci/deps/azure-37-locale.yaml
CONDA_PY: "37"
CONDA_ENV: pandas
Expand All @@ -27,6 +27,7 @@ jobs:
CONDA_PY: "36"
CONDA_ENV: pandas
TEST_ARGS: "--only-slow --skip-network"
LOCALE_OVERRIDE: "it_IT.UTF-8"

steps:
- script: |
Expand Down
3 changes: 3 additions & 0 deletions ci/deps/azure-37-locale.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ dependencies:
- pymysql
- pytables
- python-dateutil
# XXX We should be testing ``python=3.7*`` here, but `moto` is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can just fix this, add moto with a min version to the pip section

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what I understood, the moto version compatible with Python 3.7 is not yet released. Did I misunderstood something?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datapythonista @jreback
Did you see my comment above #23727 (comment) ?

@datapythonista @TomAugspurger
Moto is supporting python 3.7. starting with 1.3.7 (the latest version), but that isn't reflected in the requirements yet: getmoto/moto#1886
It's possible to install it through pip - I did that in #23731, but then the problem is that 1.3.7 requires a boto version that will run into #23754, probably because of a moto issue: getmoto/moto#1941

The credentials issue would be resolved (or rather deferred) by #23731, which makes all boto tests use the s3_resource fixture, which would turn those NoCredentialsErrors into skips.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still if you can just pip install it i am not sure of the problem.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to pip, let's see if tests pass. For @h-vetinari comment I understood that even pip version would make the tests fail.

# failing with py3.7 at the moment.
# See: https://github.com/pandas-dev/pandas/pull/23727/files#r234240004
- python=3.6*
- pytz
- s3fs
Expand Down
9 changes: 7 additions & 2 deletions ci/script_multi.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ source activate pandas

if [ -n "$LOCALE_OVERRIDE" ]; then
export LC_ALL="$LOCALE_OVERRIDE";
export LANG="$LOCALE_OVERRIDE";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this in ci/script_single.sh but not here. I don't see why in one case should be needed and not in the other. I guess it's missing here (or could be removed there).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could add this would prob be ok
but all of the locale tests are done in the single i think

echo "Setting LC_ALL to $LOCALE_OVERRIDE"

pycmd='import pandas; print("pandas detected console encoding: %s" % pandas.get_option("display.encoding"))'
Expand All @@ -32,8 +33,12 @@ elif [ "$COVERAGE" ]; then

elif [ "$SLOW" ]; then
TEST_ARGS="--only-slow --skip-network"
echo pytest -m "not single and slow" -v --durations=10 --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
pytest -m "not single and slow" -v --durations=10 --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
# The `-m " and slow"` is redundant here, as `--only-slow` is already used (via $TEST_ARGS). But is needed, because with
# `--only-slow` fast tests are skipped, but each of them is printed in the log (which can be avoided with `-q`),
# and also added to `test-data-multiple.xml`, and then printed in the log in the call to `ci/print_skipped.py`.
# Printing them to the log makes the log exceed the maximum size allowed by Travis and makes the build fail.
echo pytest -n 2 -m "not single and slow" --durations=10 --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
pytest -n 2 -m "not single and slow" --durations=10 --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not easy to see in the diff, but note that I added the -n 2 here, so the slow tests are being called with two processes in script_multi.sh now.


else
echo pytest -n 2 -m "not single" --durations=10 --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
Expand Down