[Bug]: webapi query on bylocation shows 24 hours worth of data even when smaller range requested

[trent-s]

Contact Details

trent@jp.ibm.com

What happened?

Thank you for your ongoing efforts for this very beneficial project.

After installing the Carbon Aware SDK from the Helm chart on an OpenShift cluster, I ran into behavior that seems different from what is described in the documentation. I have a workaround for this, so it is not an urgent problem for me, but thought that I would report it the event that it could be useful.

Following the web API descriptions here: https://github.com/Green-Software-Foundation/carbon-aware-sdk/blob/dev/casdk-docs/docs/tutorial-basics/carbon-aware-webapi.md

Expecting to get 2 hours worth of entries, I ran the following command on 10/01 at about 4:30 UTC, I get about 24 hours hours worth of data.

$ curl -s  "http://webapi-green-software-foundation.hostname.com/emissions/bylocation?location=japanwest&time=2024-10-01T01%3A00Z&toTime=2024-10-01T02%3A00Z" | jq
[
  {
    "location": "japanwest",
    "time": "2024-09-30T05:00:00+00:00",
    "rating": 337,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T06:00:00+00:00",
    "rating": 355,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T07:00:00+00:00",
    "rating": 384,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T08:00:00+00:00",
    "rating": 396,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T09:00:00+00:00",
    "rating": 396,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T10:00:00+00:00",
    "rating": 391,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T11:00:00+00:00",
    "rating": 385,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T12:00:00+00:00",
    "rating": 374,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T13:00:00+00:00",
    "rating": 363,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T14:00:00+00:00",
    "rating": 356,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T15:00:00+00:00",
    "rating": 347,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T16:00:00+00:00",
    "rating": 334,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T17:00:00+00:00",
    "rating": 328,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T18:00:00+00:00",
    "rating": 333,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T19:00:00+00:00",
    "rating": 338,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T20:00:00+00:00",
    "rating": 334,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T21:00:00+00:00",
    "rating": 330,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T22:00:00+00:00",
    "rating": 305,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-09-30T23:00:00+00:00",
    "rating": 290,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T00:00:00+00:00",
    "rating": 281,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T01:00:00+00:00",
    "rating": 271,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T02:00:00+00:00",
    "rating": 269,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T03:00:00+00:00",
    "rating": 312,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T04:00:00+00:00",
    "rating": 312,
    "duration": "01:00:00"
  }
]

I was expecting to see a more limited time range, such as perhaps:

[
  {
    "location": "japanwest",
    "time": "2024-10-01T00:00:00+00:00",
    "rating": 281,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T01:00:00+00:00",
    "rating": 271,
    "duration": "01:00:00"
  },
  {
    "location": "japanwest",
    "time": "2024-10-01T02:00:00+00:00",
    "rating": 269,
    "duration": "01:00:00"
  },
]

client

WebAPI (Default)

Relevant log output

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://webapi-green-software-foundation.apps.hostname.com/emissions/bylocation?location=japanwest&time=2024-10-01T01%3A00Z&toTime=2024-10-01T02%3A00Z - - -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'CarbonAware.WebApi.Controllers.CarbonAwareController.GetEmissionsDataForLocationByTime (CarbonAware.WebApi)'
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[102]
      Route matched with {action = "GetEmissionsDataForLocationByTime", controller = "CarbonAware"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] GetEmissionsDataForLocationByTime(System.String, System.Nullable`1[System.DateTimeOffset], System.Nullable`1[System.DateTimeOffset]) on controller CarbonAware.WebApi.Controllers.CarbonAwareController (CarbonAware.WebApi).
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[101]
      Executing action method CarbonAware.WebApi.Controllers.CarbonAwareController.GetEmissionsDataForLocationByTime (CarbonAware.WebApi) - Validation state: Valid
info: CarbonAware.LocationSources.LocationSource[0]
      1 files discovered
info: CarbonAware.LocationSources.LocationSource[0]
      Reading Location data source from /app/location-sources/json/azure-regions.json
info: System.Net.Http.HttpClient.ElectricityMapsClient.LogicalHandler[100]
      Start processing HTTP request GET https://api.electricitymap.org/v3/carbon-intensity/history?lat=34.6939&lon=135.5022
info: System.Net.Http.HttpClient.ElectricityMapsClient.ClientHandler[100]
      Sending HTTP request GET https://api.electricitymap.org/v3/carbon-intensity/history?lat=34.6939&lon=135.5022
Activity.TraceId:            292a7fdddde524ebd40e3da26aefaebc
Activity.SpanId:             5a1f3d0a0bfa8e95
Activity.TraceFlags:         Recorded
Activity.ParentSpanId:       faa0ef7cf762fbb1
Activity.ActivitySourceName: System.Net.Http
Activity.DisplayName:        HTTP GET
Activity.Kind:               Client
Activity.StartTime:          2024-10-01T04:49:12.5073855Z
Activity.Duration:           00:00:00.5997288
Activity.Tags:
    http.scheme: https
    http.method: GET
    net.peer.name: api.electricitymap.org
    http.url: https://api.electricitymap.org/v3/carbon-intensity/history?lat=34.6939&lon=135.5022
    http.flavor: 1.1
    http.status_code: 200
Resource associated with Activity:
    service.name: CarbonAware.WebAPI
    service.version: 1.0.0
    service.instance.id: ca9e91e7-dede-4d73-9f10-b19f455a2c8f
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.6.0

info: System.Net.Http.HttpClient.ElectricityMapsClient.ClientHandler[101]
      Received HTTP response headers after 605.309ms - 200
info: System.Net.Http.HttpClient.ElectricityMapsClient.LogicalHandler[101]
      End processing HTTP request after 614.2769ms - 200
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[103]
      Executed action method CarbonAware.WebApi.Controllers.CarbonAwareController.GetEmissionsDataForLocationByTime (CarbonAware.WebApi), returned result Microsoft.AspNetCore.Mvc.OkObjectResult in 683.3091ms.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
      Executing OkObjectResult, writing value of type 'System.Linq.Enumerable+SelectListIterator`2[[CarbonAware.Model.EmissionsData, CarbonAware, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null],[GSF.CarbonAware.Models.EmissionsData, GSF.CarbonAware, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]'.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[105]
      Executed action CarbonAware.WebApi.Controllers.CarbonAwareController.GetEmissionsDataForLocationByTime (CarbonAware.WebApi) in 742.3564ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'CarbonAware.WebApi.Controllers.CarbonAwareController.GetEmissionsDataForLocationByTime (CarbonAware.WebApi)'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://webapi-green-software-foundation.apps.hostname.com/emissions/bylocation?location=japanwest&time=2024-10-01T01%3A00Z&toTime=2024-10-01T02%3A00Z - 200 - application/json;+charset=utf-8 779.0769ms
Activity.TraceId:            292a7fdddde524ebd40e3da26aefaebc
Activity.SpanId:             faa0ef7cf762fbb1
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName:        emissions/bylocation
Activity.Kind:               Server
Activity.StartTime:          2024-10-01T04:49:12.3782214Z
Activity.Duration:           00:00:00.7792018
Activity.Tags:
    net.host.name: webapi-green-software-foundation.apps.hostname.com
    http.method: GET
    http.scheme: http
    http.target: /emissions/bylocation
    http.url: http://webapi-green-software-foundation.apps.hostname.com/emissions/bylocation?location=japanwest&time=2024-10-01T01%3A00Z&toTime=2024-10-01T02%3A00Z
    http.flavor: 1.1
    http.user_agent: curl/7.61.1
    http.route: emissions/bylocation
    http.status_code: 200
Resource associated with Activity:
    service.name: CarbonAware.WebAPI
    service.version: 1.0.0
    service.instance.id: ca9e91e7-dede-4d73-9f10-b19f455a2c8f
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.6.0

Code of Conduct

• I agree to follow this project's Code of Conduct

[YaSuenag]

@trent-s
Thanks for your report! Did you configure Electricity Maps as a data source, right?

@vaughanknight @danuw @tiwatsuka
I think this is similar with my past report for EMFree data source in #389 . We need to filter with time range in ElectricityMapsDataSouce::HistoryCarbonIntensityToEmissionsData.

[trent-s]

Thank you @YaSuenag for your kind consideration. Yes, I am using "ElectricityMaps" as the "EmissionsDataSource" for this query.

[trent-s]

Thank you very much for your excellent work on this!