[Cost Report] Add some UI fixes #1788

sumanthgenz · 2023-03-16T21:06:50Z

Limit lines in each table to 5 lines (this number is up for debate).
Display total cost across all reports in cost-report (even costs that are not displayed due to the above limit).

(sky) sumanth@MacBook-Pro-5 skypilot % sky cost-report   
Clusters
NAME     LAUNCHED      DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
cluster  1 month ago   6m 19s    1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $0.311       
cluster  1 month ago   22m 31s   1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $1.108       
cluster  2 months ago  3m 41s    1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.029       
cluster  2 months ago  10m 13s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.081       

Managed spot controller (will be autostopped if idle for 10min)
NAME                          LAUNCHED     DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
sky-spot-controller-d16122f1  1 month ago  10m 25s   1x AWS(m6i.2xlarge, disk_size=50)  TERMINATED  $ 0.384       $0.067       

TOTAL COST: $189.105
NOTE: This feature is experimental. Costs for clusters with auto{stop,down} scheduled may not be accurate.

Michaelvll

Thanks for submitting the PR @sumanthgenz! Left several comments.

sky/cli.py

Michaelvll · 2023-03-17T05:33:54Z

sky/utils/cli_utils/status_utils.py

+    num_lines_to_display = _NUM_COST_REPORT_LINES
+    if show_all:
+        num_lines_to_display = len(cluster_records)
+
+    for record in cluster_records[:num_lines_to_display]:


The cost records seems to be in the order of launch time, but it might be better to place the clusters that have not been terminated yet at the beginning.

sky/utils/cli_utils/status_utils.py

sky/cli.py

sumanthgenz · 2023-03-20T03:50:51Z

(sky) sumanth@MacBook-Pro-5 skypilot % sky cost-report                    
Clusters
NAME     LAUNCHED      DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
cluster  1 month ago   6m 19s    1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $0.311       
cluster  1 month ago   22m 31s   1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $1.108       
cluster  3 months ago  3m 41s    1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.029       
cluster  3 months ago  10m 13s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.081       
min      3 months ago  6m 25s    1x AWS(m6i.2xlarge)                TERMINATED  $ 0.384       $0.041       
min      3 months ago  24m 40s   1x AWS(m6i.2xlarge)                TERMINATED  $ 0.384       $0.158       
mount    3 months ago  7m 26s    1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.059       
mount2   3 months ago  12m 28s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.098       
mount    3 months ago  16m 26s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.130       

Managed spot controller (will be autostopped if idle for 10min)
NAME                          LAUNCHED     DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
sky-spot-controller-d16122f1  1 month ago  10m 25s   1x AWS(m6i.2xlarge, disk_size=50)  TERMINATED  $ 0.384       $0.067       

Total Cost: $219.710
NOTE: Since --all is not set, not all cost report records may be displayed above.
NOTE: This feature is experimental. Costs for clusters with auto{stop,down} scheduled may not be accurate.

concretevitamin

Thanks @sumanthgenz. This is great.

UX comments:

Clusters
NAME             LAUNCHED    DURATION           RESOURCES                             STATUS      HOURLY_PRICE  COST (est.)
...

Managed spot controller (will be autostopped if idle for 10min)
NAME                          LAUNCHED      DURATION            RESOURCES                           STATUS      HOURLY_PRICE  COST (est.)
sky-spot-controller-8a3968f2  2 months ago  4 days 12h 41m 44s  1x GCP(n1-highmem-8, disk_size=50)  TERMINATED  $ 0.473       $51.436

Total Cost: $16524.966
NOTE: Since --all is not set, not all cost report records may be displayed above.
NOTE: This feature is experimental. Costs for clusters with auto{stop,down} scheduled may not be accurate.

Can we make Total Cost sum up what's being displayed, rather than everything in history? In the table above my COST (est.) column adds up to <$100, so it took me a while to realize why Total Cost displays a 5-figure number (which I'm not sure is accurate).
Consider removing NOTE: since we've yellow'd the font.
Consider Since --all is not set, not all cost report records may be displayed above. -> Showing the N most recent clusters. To see all clusters in history, pass the --all flag.

sky/cli.py

sky/utils/cli_utils/status_utils.py

concretevitamin · 2023-03-23T22:53:20Z

sky/utils/cli_utils/status_utils.py

+    if status is None:
+        return -1
+    return 1


Why are we sorting a bunch of -1's and 1's? From the output, seems like we are sorting by launch-time?

We wanted to bubble up clusters that were active and un-terminated to the top since they are actively incurring cost. In addition, we also sort by launch time.

Can we put this comment as comment before L172?

sumanthgenz · 2023-04-05T20:20:43Z

(sky) sumanth@MacBook-Pro-5 skypilot % sky cost-report
Clusters
NAME     LAUNCHED      DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
cluster  2 months ago  6m 19s    1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $0.311       
cluster  2 months ago  22m 31s   1x GCP(n1-highmem-8, {'V100': 1})  TERMINATED  $ 2.953       $1.108       
cluster  3 months ago  3m 41s    1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.029       
cluster  3 months ago  10m 13s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.081       
min      4 months ago  6m 25s    1x AWS(m6i.2xlarge)                TERMINATED  $ 0.384       $0.041       
min      4 months ago  24m 40s   1x AWS(m6i.2xlarge)                TERMINATED  $ 0.384       $0.158       
mount    4 months ago  7m 26s    1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.059       
mount2   4 months ago  12m 28s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.098       
mount    4 months ago  16m 26s   1x GCP(n1-highmem-8)               TERMINATED  $ 0.473       $0.130       

Managed spot controller (will be autostopped if idle for 10min)
NAME                          LAUNCHED      DURATION  RESOURCES                          STATUS      HOURLY_PRICE  COST (est.)  
sky-spot-controller-d16122f1  2 months ago  10m 25s   1x AWS(m6i.2xlarge, disk_size=50)  TERMINATED  $ 0.384       $0.067       

Total Cost: $2.08
Showing the N most recent clusters. To see all clusters in history, pass the --all flag.
This feature is experimental. Costs for clusters with auto{stop,down} scheduled may not be accurate.

concretevitamin · 2023-04-06T02:55:10Z

After switching to this branch, now seeing

» sky cost-report
...
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/core.py", line 143, in cost_report
    cluster_reports = global_user_state.get_clusters_from_history()
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/global_user_state.py", line 608, in get_clusters_from_history
    'duration': _get_cluster_duration(cluster_hash),
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/global_user_state.py", line 449, in _get_cluster_duration
    start_time, end_time = int(start_time), int(end_time)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Expected?

sumanthgenz · 2023-04-06T02:57:44Z

Hi @concretevitamin, thanks for showing this. @Michaelvll was seeing this issue too, but I myself have not been encountering it. In another branch I've added a fix where it skips entries where start_time is None (which it should never be given the way the durations are tracked, so I am not sure where the bug originates from). I can add that fix here too.

concretevitamin

Thanks @sumanthgenz. It works now. A few comments.

Also:

» sky cost-report
Clusters
NAME                  LAUNCHED     DURATION                   RESOURCES                                 STATUS      HOURLY_PRICE  COST (est.)
llama-ckpts           3 weeks ago  2 weeks 2 days 12h 54m...  1x GCP(n2-standard-4, disk_size=512)      STOPPED     $ 0.194       $77.094
dbg                   1 day ago    19m 25s                    1x Lambda(gpu_1x_a100_sxm4, {'A100': 1})  TERMINATED  $ 1.100       $0.356
...

Managed spot controller (will be autostopped if idle for 10min)
NAME                          LAUNCHED      DURATION            RESOURCES                           STATUS      HOURLY_PRICE  COST (est.)
sky-spot-controller-8a3968f2  2 months ago  4 days 12h 41m 44s  1x GCP(n1-highmem-8, disk_size=50)  TERMINATED  $ 0.473       $51.436

Total Cost: $53.12
...

Two questions:

Why does total cost show $53.12, while the COST (est.) add up to something bigger?
For the first cluster, llama-ckpts, the duration 2 weeks 2 days 12h 54m... seems too long (I don't remember its uptime being this long). Is it a known issue for clusters that might have utilized autostop?

sky/utils/cli_utils/status_utils.py

sky/cli.py

concretevitamin · 2023-04-06T05:07:30Z

sky/utils/cli_utils/status_utils.py

+    if status is None:
+        return -1
+    return 1


Can we put this comment as comment before L172?

sky/cli.py

sumanthgenz · 2023-04-06T05:33:50Z

@concretevitamin

I tried sorting by cluster status as well in get_total_cost_of_displayed_records, perhaps this should fix the costs to align with the displayed records.
Yes, in another PR I nullify cost for autostop clusters since their durations and associated costs.

concretevitamin · 2023-04-06T16:25:55Z

Thanks @sumanthgenz.

Now sky cost-report's total is right. Passing -a flag however gives the same total, but it should be higher since there are many more terminated clusters?
My status table has a stopped controller

sky-spot-controller-8a3968f2  10 hrs ago  1x GCP(n2-standard-4, disk_size=50)   STOPPED  10m       sky spot launch -dy --cloud...

but cost-report doesn't show it and it shows a terminated one. Is this a known issue?

sky-spot-controller-8a3968f2  2 months ago  4 days 12h 41m 44s  1x GCP(n1-highmem-8, disk_size=50)  TERMINATED  $ 0.473       $51.436

sumanthgenz · 2023-04-11T19:52:38Z

@concretevitamin

Fixed
In [Spot] [Cost Report] Add CLI support for getting VM cost for spot jobs #1669, this will be fully fixed since there is logic to aggregate record entries there. In this current PR, for reserved_clusters, we know display the most recent entry for each reserved_cluster.

concretevitamin

LGTM, thanks @sumanthgenz.

sky/cli.py

sumanthgenz requested a review from Michaelvll March 16, 2023 21:06

Michaelvll reviewed Mar 17, 2023

View reviewed changes

Michaelvll requested a review from concretevitamin March 21, 2023 22:28

concretevitamin added this to the 0.3 milestone Mar 22, 2023

concretevitamin reviewed Mar 23, 2023

View reviewed changes

sumanthgenz force-pushed the sgurram_cost_report_ui_fixes branch from 22e28b3 to 70b07c9 Compare April 5, 2023 20:19

sumanthgenz requested a review from concretevitamin April 5, 2023 20:20

concretevitamin reviewed Apr 6, 2023

View reviewed changes

Sumanth and others added 7 commits April 11, 2023 12:24

add some ui fixes to cost report

d3c09a8

address PR comments

4b010e0

address PR comments by Zongheng

ebc1613

lint

b33c0fe

temp fix for start_time null issue

c2ee042

fix PR comments

4178533

address PR comments

70f1d1e

sumanthgenz force-pushed the sgurram_cost_report_ui_fixes branch from e6958fd to 70f1d1e Compare April 11, 2023 19:50

format

eb4f3e6

concretevitamin approved these changes Apr 12, 2023

View reviewed changes

sky/cli.py Outdated Show resolved Hide resolved

sky/cli.py Show resolved Hide resolved

add final PR fixes

801fb15

sumanthgenz merged commit 2380776 into skypilot-org:master Apr 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cost Report] Add some UI fixes #1788

[Cost Report] Add some UI fixes #1788

sumanthgenz commented Mar 16, 2023

Michaelvll left a comment

Michaelvll Mar 17, 2023

sumanthgenz commented Mar 20, 2023

concretevitamin left a comment

concretevitamin Mar 23, 2023

sumanthgenz Apr 5, 2023

concretevitamin Apr 6, 2023

sumanthgenz commented Apr 5, 2023

concretevitamin commented Apr 6, 2023

sumanthgenz commented Apr 6, 2023 •

edited

Loading

concretevitamin left a comment

concretevitamin Apr 6, 2023

sumanthgenz commented Apr 6, 2023

concretevitamin commented Apr 6, 2023

sumanthgenz commented Apr 11, 2023

concretevitamin left a comment

[Cost Report] Add some UI fixes #1788

[Cost Report] Add some UI fixes #1788

Conversation

sumanthgenz commented Mar 16, 2023

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll Mar 17, 2023

Choose a reason for hiding this comment

sumanthgenz commented Mar 20, 2023

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Mar 23, 2023

Choose a reason for hiding this comment

sumanthgenz Apr 5, 2023

Choose a reason for hiding this comment

concretevitamin Apr 6, 2023

Choose a reason for hiding this comment

sumanthgenz commented Apr 5, 2023

concretevitamin commented Apr 6, 2023

sumanthgenz commented Apr 6, 2023 • edited Loading

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Apr 6, 2023

Choose a reason for hiding this comment

sumanthgenz commented Apr 6, 2023

concretevitamin commented Apr 6, 2023

sumanthgenz commented Apr 11, 2023

concretevitamin left a comment

Choose a reason for hiding this comment

sumanthgenz commented Apr 6, 2023 •

edited

Loading