Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support gokart 1.4.0 with some fixed tests #41

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Support gokart 1.4.0 with some fixed tests #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4dc2bba
update gokart
Jun 14, 2024
0ffa812
remove default env test cases
Jun 14, 2024
290c0f7
Fix python version along with new gokart dependency
Jun 14, 2024
214df08
Update numpy to 1.26.4 to prevent build errors
Jun 14, 2024
27a27f5
update tox
Jun 14, 2024
0664f8c
Update mypy regulations along with gokart
Jun 14, 2024
a19a828
Update pyarrow
Jun 14, 2024
eff6b3b
relock
Jun 14, 2024
4a3d686
update pyyaml
Jun 14, 2024
653eea4
add source for coverage
Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
max-parallel: 4
matrix:
platform: [ubuntu-latest]
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v3
Expand Down
1,673 changes: 702 additions & 971 deletions poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

11 changes: 5 additions & 6 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ style = "pep440"
pattern = "^(?P<base>\\d+\\.\\d+\\.\\d+)"

[tool.poetry.dependencies]
python = "^3.8,<3.12"
gokart = "^1.2.2"
python = "<4,>=3.9"
gokart = ">=1.4.0"
kubernetes = "^26.1.0"

[tool.poetry.group.dev.dependencies]
Expand All @@ -27,7 +27,9 @@ isort = "^5.7"
yapf = ">=0.32"
toml = "*"
mypy = "^1.2.0"
pandas-stubs = "^1.5.3.230321"
pandas-stubs = "*"
pyyaml = "6.0.1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question]

new dependency...?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

 name = "pyyaml"
-version = "6.0"
+version = "6.0.1"

seems not new.

6.0.0 has any issue? I expect we don't need to pin but just require newer than 6.0.1

Suggested change
pyyaml = "6.0.1"
pyyaml = ">=6.0.1"



[tool.flake8]
per-file-ignores = """
Expand All @@ -50,9 +52,6 @@ ignore_patterns = [
line_length = 160

[tool.mypy]
strict = true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mamo3gr

need to remove strict ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yokomotod
Should it be redirected to @maronuu ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kusa

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😅 😅 😅

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting that aside, I srtongly disaggree with removing the strict setting.

disallow_subclassing_any = false # revert strict
implicit_reexport = true # revert strict
ignore_missing_imports = true

[build-system]
Expand Down
1 change: 1 addition & 0 deletions test/integration_test/test_master_build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ def requires(self) -> list[Child]:
self.assertEqual(
cm.output,
[
'INFO:kannon.master:No dynamic config files are given.',
'INFO:kannon.master:Creating task queue...',
f'INFO:kannon.master:Task {c1_task_info} is pushed to task queue',
f'INFO:kannon.master:Task {c2_task_info} is pushed to task queue',
Expand Down
25 changes: 0 additions & 25 deletions test/unit_test/test_master.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,6 @@ class Example(gokart.TaskOnKart):
# following should be overwritten
self.assertEqual(child_job.spec.template.spec.containers[0].command, ["python", __file__, "--task-pkl-path", f"'{path_to_pkl}'"])
self.assertEqual(child_job.metadata.name, child_job_name)
# envvar TASK_WORKSPACE_DIRECTORY should be inherited
child_env = child_job.spec.template.spec.containers[0].env
self.assertEqual(len(child_env), 1)
self.assertEqual(child_env[0], {"name": "TASK_WORKSPACE_DIRECTORY", "value": "/cache"})

def test_success_custom_env(self) -> None:

Expand Down Expand Up @@ -151,27 +147,6 @@ class Example(gokart.TaskOnKart):
with self.assertRaises(AssertionError):
master._create_child_job_object("test-job", path_to_pkl)

def test_fail_default_env_not_exist(self) -> None:

class Example(gokart.TaskOnKart):
pass

path_to_pkl = "path/to/obj"
template_job = self._get_template_job()

cases = [None, ["TASK_WORKSPACE_DIRECTORY", "MY_ENV0", "MY_ENV1"]]
for case in cases:
with self.subTest(case=case):
master = Kannon(
api_instance=None,
template_job=template_job,
job_prefix="",
path_child_script=__file__, # just pass any existing file as dummy
env_to_inherit=case,
)
with self.assertRaises(ValueError):
master._create_child_job_object("test-job", path_to_pkl)

def test_owner_reference_set(self) -> None:

class Example(gokart.TaskOnKart):
Expand Down
6 changes: 3 additions & 3 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
[tox]
envlist = py{38,39,310,311},yapf,isort,flake8,mypy
envlist = py{39,310,311,312},yapf,isort,flake8,mypy
isolated_build = true

[testenv]
allowlist_externals = coverage
skip_install = true
commands = coverage run -m unittest discover -v
commands = coverage run -m unittest discover -v -s test/unit_test

[testenv:yapf]
allowlist_externals = yapf
Expand All @@ -29,7 +29,7 @@ commands = mypy ./kannon ./test ./example {posargs}

[gh-actions]
python =
3.8: py38
3.9: py39
3.10: py310
3.11: py311
3.12: py312
Loading