Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job start and end time #30

Merged
merged 10 commits into from
Nov 8, 2024

Conversation

leewesleyv
Copy link
Contributor

@leewesleyv leewesleyv commented Aug 30, 2024

Resolves #11

  • Add start time to job data for the Docker launcher
  • Add start time to job data for the K8s launcher
  • Add end time to job data for the Docker launcher
  • Add end time to job data for the K8s launcher
  • Add unit tests
  • Add documentation

@leewesleyv leewesleyv marked this pull request as draft August 30, 2024 06:55
@leewesleyv leewesleyv force-pushed the feature/11-job-start-and-end-time branch from 1df3848 to 0090c20 Compare August 30, 2024 07:03
@leewesleyv leewesleyv added the enhancement New feature or request label Aug 30, 2024
Copy link
Member

@wvengen wvengen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
Great to see some cleanup too (though some comments).
I still need to get used to the idea of introducing typing here. Maybe discuss this first at a standup, and then perhaps introduce later?

scrapyd_k8s/launcher/docker.py Outdated Show resolved Hide resolved
scrapyd_k8s/launcher/docker.py Show resolved Hide resolved
test_api.py Outdated Show resolved Hide resolved
scrapyd_k8s/settings.py Outdated Show resolved Hide resolved
scrapyd_k8s/launcher/docker.py Outdated Show resolved Hide resolved
scrapyd_k8s/launcher/k8s.py Outdated Show resolved Hide resolved
Copy link
Member

@wvengen wvengen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, thanks!
I see you've added the first unit test. Great!
The API test is not a regular test, as it is meant to run against a running instance. I'm not really sure if it fits well into the general pytest infrastructure like this. What do you think?

I'd split the API tests (which can be run completely separate from scrapyd-k8s, and could even be run against a different implementation, iirc), and the unit tests, into separate CI jobs.

scrapyd_k8s/launcher/docker.py Outdated Show resolved Hide resolved
Copy link
Member

@wvengen wvengen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR!
In general, fine 👍 Some notes that may help!

I understand the impulse to change some things as you go through the code :) When the PR would be ready, I would expect that you'd either explain the reason for your changes that are not strictly related to this PR (preferably having them in separate commits, so you can easily filter them out later if needed), or move them to a different PR.

Keep it up!

scrapyd_k8s/launcher/k8s.py Outdated Show resolved Hide resolved
scrapyd_k8s/tests/integration/test_api.py Outdated Show resolved Hide resolved
scrapyd_k8s/tests/integration/test_api.py Outdated Show resolved Hide resolved
scrapyd_k8s/launcher/k8s.py Outdated Show resolved Hide resolved
@leewesleyv leewesleyv force-pushed the feature/11-job-start-and-end-time branch from d6a1d58 to 0409c5e Compare September 20, 2024 06:50
@leewesleyv leewesleyv force-pushed the feature/11-job-start-and-end-time branch from 0409c5e to 0b7cd4a Compare November 5, 2024 09:29
@leewesleyv leewesleyv marked this pull request as ready for review November 5, 2024 10:45
README.md Outdated Show resolved Hide resolved
@wvengen
Copy link
Member

wvengen commented Nov 7, 2024

Can we make it a bit terser? (a separate heading breaks the flow of reading through the endpoints; this is just a detail you're only interested in if you want to know more about this endpoint)

--- a/README.md
+++ b/README.md
@@ -228,10 +228,7 @@ Lists spiders from the spider image's `org.scrapy.spiders` label.
 ### `listjobs.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listjobs-json))
 
 Lists current jobs by looking at Docker containers or Kubernetes jobs.
-
-#### Limitations
-
-* **End time**; The job's end time will be populated only for the Kubernetes (k8s) launcher, provided the job finishes successfully and is not canceled. For Docker, this value will always be null.
+Note that `end_time` is not yet supported for Docker.
 
 ### ~~`delversion.json`~~ ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#delversion-json))
 

@wvengen
Copy link
Member

wvengen commented Nov 7, 2024

Otherwise, good to go!

@leewesleyv
Copy link
Contributor Author

@wvengen Sorry, I did not see your suggestion. Do you think the change I've made is OK? Or should I change it to the suggestion?

@wvengen
Copy link
Member

wvengen commented Nov 8, 2024

Yes, please use the diff I posted. Thanks!

README.md Outdated Show resolved Hide resolved
@wvengen
Copy link
Member

wvengen commented Nov 8, 2024

Super, thanks!

@wvengen wvengen merged commit 33bae3e into q-m:main Nov 8, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add start_time and end_time in listjobs.json
2 participants