Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hog upload fix #897

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Hog upload fix #897

wants to merge 5 commits into from

Conversation

ArneTR
Copy link
Member

@ArneTR ArneTR commented Sep 15, 2024

This is currently a draft PR to make the PowerHog data upload usable again.

It was deactivated as we were seeing regular errors.

It has now been deactivated with some learnings. More are to come in this PR.

Learnings

  • PowerHog is posting often frames larger than 1 MB
    • NGINX POST body size was increased to 3 MB. Not too large to block the server, but a bit of headroom
  • For reasons unknown many requests where issued to GET /v1/hog/add. Since we monitor 404s this was clogging up our logs. I have deactivated the route for now ... but why is this happening?

@github-actions GitHub Actions
Copy link

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 24.1873 1620.98 3.89 417
Measurement #1 24.1177 1620.98 3.91 415

🌳 CO2 Data:
City: Chicago, Lat: 41.8874, Lon: -87.6318
IP: 172.183.76.129
CO₂ from energy is: 0.585173780 g
CO₂ from manufacturing (embodied carbon) is: 0.118975858 g
Carbon Intensity for this location: 361 gCO₂eq/kWh
SCI: 0.704150 gCO₂eq / pipeline run emitted

@ArneTR ArneTR linked an issue Sep 16, 2024 that may be closed by this pull request
Coalition DB ID not found in HOG #903
Open
@ArneTR ArneTR linked an issue Sep 18, 2024 that may be closed by this pull request
PowerHog Data can contain NaN values #906
Closed
@ArneTR ArneTR linked an issue Sep 18, 2024 that may be closed by this pull request
PowerHog data often sends JSON with missing disk type #907
Open
@ribalba
Copy link
Member

ribalba commented Sep 24, 2024

  • Large uploads:
    We limit the upload to 10 rows [0] which I found to be under 1MB. This could have increased over time the more apps are running. Setting the upload to 3MB seems like a good solution

  • Access to GET /v1/hog/add.
    I can not explain why this is happening. The upload is with POST [1]. Do you see some correlation in the log files? What IPs are the GET requests coming from?

[0] https://github.com/green-coding-solutions/hog/blob/ea2f5970b8281650ac21687613b5ce098ddf8de4/power_logger.py#L180
[1] https://github.com/green-coding-solutions/hog/blob/ea2f5970b8281650ac21687613b5ce098ddf8de4/power_logger.py#L207

@ArneTR
Copy link
Member Author

ArneTR commented Nov 27, 2024

@ribalba Can you please close this and move code to the EE repository

@ArneTR
Copy link
Member Author

ArneTR commented Nov 27, 2024

Also please include any lingering HOG fixes we have around. I remember you posted somewhere refactorings to the object_specification.py by moving all checks for HOG data to pydantic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ribalba ribalba

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@ArneTR @ribalba