Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log file sizes and free mem #305

Merged
merged 8 commits into from
Jun 14, 2022
Merged

Log file sizes and free mem #305

merged 8 commits into from
Jun 14, 2022

Conversation

EnricoMi
Copy link
Owner

@EnricoMi EnricoMi commented Jun 14, 2022

Logs the number of files to read and their size, the available memory, and logs progress every 10s.

Fixes #304.

@github-actions

This comment has been minimized.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (macOS python installed)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Test Files)

  26 files  ±0      4 errors  23 suites  ±0   39m 21s ⏱️ ±0s
279 tests ±0  228 ✔️ ±0  20 💤 ±0  25 ±0  6 🔥 ±0 
450 runs  ±0  358 ✔️ ±0  57 💤 ±0  28 ±0  7 🔥 ±0 

For more details on these parsing errors, failures and errors, see this check.

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Test File)

97 tests  ±0   80 ✔️ ±0   3m 25s ⏱️ ±0s
  1 suites ±0   17 💤 ±0 
  1 files   ±0     0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Docker Image)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (macOS 12 python installed)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Dockerfile)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Linux python installed)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Linux 22.04 python installed)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Windows python installed)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (reference)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@EnricoMi EnricoMi force-pushed the branch-log-file-sizes branch from 2f697c0 to 9399470 Compare June 14, 2022 10:27
@github-actions

This comment has been minimized.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (macOS python 3.6)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 14, 2022

Test Results (Linux python 3.6)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

♻️ This comment has been updated with latest results.

@EnricoMi EnricoMi force-pushed the branch-log-file-sizes branch from 9399470 to fef0c81 Compare June 14, 2022 12:10
@@ -114,7 +120,7 @@ def parse(path: str) -> Union[str, Any]:
except BaseException as e:
return e

parsed_files = [(result_file, parse(result_file))
parsed_files = [progress((result_file, parse(result_file)))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incompatible parameter type: Expected Variable[T] for 1st positional only parameter to anonymous call but got typing.Tuple[str, typing.Any].

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

progress = Progress(items)
plogger = ProgressLogger(progress, interval_seconds, progress_template, logger).start()
try:
yield progress.observe
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incompatible return type: Expected typing.Callable[[Variable[T]], Variable[T]] but got typing.Generator[BoundMethod[typing.Callable(Progress.observe)[[Named(self, Progress[typing.Any]), Named(observation, typing.Any)], typing.Any], Progress[typing.Any]], None, None].

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

@@ -114,7 +120,7 @@ def parse(path: str) -> Union[str, Any]:
except BaseException as e:
return e

parsed_files = [(result_file, parse(result_file))
parsed_files = [progress((result_file, parse(result_file)))
for result_file in files]
junits = [(result_file, junit)
for result_file, junit in parsed_files
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unable to unpack: Unable to unpack Variable[python.publish.junit.T] into 2 values.

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

interval_seconds: int,
progress_template: str,
finish_template: Optional[str],
logger: Logger) -> Callable[[T], T]:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Invalid type variable: The type variable Variable[T] isn't present in the function's parameters.

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

@github-actions

This comment has been minimized.

@EnricoMi EnricoMi force-pushed the branch-log-file-sizes branch from fef0c81 to 4621b00 Compare June 14, 2022 12:23
@github-actions

This comment has been minimized.

progress_template: str,
finish_template: Optional[str],
logger: Logger,
progress_item_type: Type[T] = Any) -> Callable[[Any], Any]:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incompatible variable type: progress_item_type is declared to have type Type[Variable[T]] but is used as type object.

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

progress = Progress[progress_item_type](items)
plogger = ProgressLogger(progress, interval_seconds, progress_template, logger).start()
try:
yield progress.observe
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incompatible return type: Expected typing.Callable[[typing.Any], typing.Any] but got typing.Generator[BoundMethod[typing.Callable(Progress.observe)[[Named(self, Progress[Variable[T]]), Named(observation, Variable[T])], Variable[T]], Progress[Variable[T]]], None, None].

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

@github-actions

This comment has been minimized.

@EnricoMi EnricoMi force-pushed the branch-log-file-sizes branch from 8b23c9f to f28d9a8 Compare June 14, 2022 14:32
progress = Progress[progress_item_type](items)
plogger = ProgressLogger(progress, interval_seconds, progress_template, logger).start()
try:
yield progress.observe
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incompatible return type: Expected typing.Callable[[Variable[T]], Variable[T]] but got typing.Generator[BoundMethod[typing.Callable(Progress.observe)[[Named(self, Progress[Variable[T]]), Named(observation, Variable[T])], Variable[T]], Progress[Variable[T]]], None, None].

Reply with "@sonatype-lift help" for more info.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

@github-actions
Copy link

Test Results (setup-python)

       69 files  ±    0         69 suites  ±0   3m 54s ⏱️ +16s
     290 tests +    2       290 ✔️ +    2      0 💤 ±0  0 ±0 
20 010 runs  +138  19 470 ✔️ +138  540 💤 ±0  0 ±0 

Results for commit f28d9a8. ± Comparison against base commit 3281b65.

@EnricoMi EnricoMi merged commit ca6e0bd into master Jun 14, 2022
@EnricoMi EnricoMi deleted the branch-log-file-sizes branch June 14, 2022 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve logging reading the files
1 participant