[batch] Update IP Fee pricing for February 2024 price increase (hail-…

…is#14190)

Fixes hail-is#13784 

Here's the GCP documentation:
https://cloud.google.com/vpc/pricing-announce-external-ips

We were previously billing the same IP-Fee for both spot and regular
instances. I changed it so we're billing for each instance type
accordingly.

Following hail-is#13542, I hard coded the new resource rates.

batch/batch/cloud/gcp/instance_config.py

@@ -57,7 +57,7 @@ def create(
             GCPStaticSizedDiskResource.create(product_versions, 'pd-ssd', boot_disk_size_gb, region),
             data_disk_resource,
             GCPDynamicSizedDiskResource.create(product_versions, 'pd-ssd', region),
-            GCPIPFeeResource.create(product_versions, 1024),
+            GCPIPFeeResource.create(product_versions, 1024, preemptible),
             GCPServiceFeeResource.create(product_versions),
             GCPSupportLogsSpecsAndFirewallFees.create(product_versions),
         ]

batch/batch/cloud/gcp/resources.py

@@ -270,17 +270,18 @@ class GCPIPFeeResource(IPFeeResourceMixin, GCPResource):
     TYPE = 'gcp_ip_fee'
 
     @staticmethod
-    def product_name(base: int) -> str:
-        return f'ip-fee/{base}'
+    def product_name(base: int, preemptible: bool) -> str:
+        preemptible_str = 'preemptible' if preemptible else 'nonpreemptible'
+        return f'ip-fee/{preemptible_str}/{base}'
 
     @staticmethod
     def from_dict(data: Dict[str, Any]) -> 'GCPIPFeeResource':
         assert data['type'] == GCPIPFeeResource.TYPE
         return GCPIPFeeResource(data['name'])
 
     @staticmethod
-    def create(product_versions: ProductVersions, base: int) -> 'GCPIPFeeResource':
-        product = GCPIPFeeResource.product_name(base)
+    def create(product_versions: ProductVersions, base: int, preemptible: bool) -> 'GCPIPFeeResource':
+        product = GCPIPFeeResource.product_name(base, preemptible)
         name = product_versions.resource_name(product)
         assert name, product
         return GCPIPFeeResource(name)

batch/sql/update-ip-fee-resource.py

@@ -0,0 +1,52 @@
+import os
+import asyncio
+from gear import Database, transaction, Transaction
+
+
+async def main():
+    cloud = os.environ['HAIL_CLOUD']
+    if cloud != 'gcp':
+        return
+
+    db = Database()
+    await db.async_init()
+    try:
+        @transaction(db)
+        async def insert(tx: Transaction):
+            await tx.execute_many(
+                '''
+INSERT INTO latest_product_versions (product, version)
+VALUES (%s, %s);
+''',
+                [('ip-fee/preemptible/1024', '1'),
+                 ('ip-fee/nonpreemptible/1024', '1')]
+            )
+
+            # https://cloud.google.com/vpc/pricing-announce-external-ips
+            # from hailtop.utils import rate_instance_hour_to_fraction_msec
+            # spot_ip_fee = rate_instance_hour_to_fraction_msec(0.0025, 1024)
+            spot_ip_fee = 6.781684027777778e-13
+            # standard_ip_fee = rate_instance_hour_to_fraction_msec(0.005, 1024)
+            standard_ip_fee = 1.3563368055555557e-12
+
+            await tx.execute_many(
+                '''
+INSERT INTO resources (resource, rate)
+VALUES (%s, %s);
+''',
+                [('ip-fee/preemptible/1024/1', spot_ip_fee),
+                 ('ip-fee/nonpreemptible/1024/1', standard_ip_fee)]
+            )
+
+            await tx.execute_update('''
+UPDATE resources
+SET deduped_resource_id = resource_id
+WHERE resource = 'ip-fee/preemptible/1024/1' OR resource = 'ip-fee/nonpreemptible/1024/1';
+''')
+
+        await insert()
+    finally:
+        await db.async_close()
+
+
+asyncio.run(main())

build.yaml

@@ -2361,6 +2361,9 @@ steps:
       - name: remove-v2-billing-writes
         script: /io/sql/remove-v2-billing-writes.sql
         online: true
+      - name: update-ip-fee-resource
+        script: /io/sql/update-ip-fee-resource.py
+        online: true
     inputs:
       - from: /repo/batch/sql
         to: /io/sql

hail/python/hailtop/batch/docs/service.rst

@@ -105,7 +105,7 @@ Billing
 -------
 
 The cost for executing a job depends on the underlying machine type, the region in which the VM is running in,
-and how much CPU and memory is being requested. Currently, Batch runs most jobs on 16 core, preemptible, n1
+and how much CPU and memory is being requested. Currently, Batch runs most jobs on 16 core, spot, n1
 machines with 10 GB of persistent SSD boot disk and 375 GB of local SSD. The costs are as follows:
 
 - Compute cost
@@ -116,11 +116,11 @@ machines with 10 GB of persistent SSD boot disk and 375 GB of local SSD. The cos
         based on the current spot prices for a given worker type and the region in which the worker is running in.
         You can use :meth:`.Job.regions` to specify which regions to run a job in.
 
-   = $0.01 per core per hour for **preemptible standard** worker types
+   = $0.01 per core per hour for **spot standard** worker types
 
-   = $0.012453 per core per hour for **preemptible highmem** worker types
+   = $0.012453 per core per hour for **spot highmem** worker types
 
-   = $0.0074578 per core per hour for **preemptible highcpu** worker types
+   = $0.0074578 per core per hour for **spot highcpu** worker types
 
    = $0.04749975 per core per hour for **nonpreemptible standard** worker types
 
@@ -163,22 +163,27 @@ machines with 10 GB of persistent SSD boot disk and 375 GB of local SSD. The cos
 
 
 - IP network cost
-   = $0.00025 per core per hour
+   = $0.0003125 per core per hour for **nonpreemptible** worker types
+
+   = $0.00015625 per core per hour for **spot** worker types
 
 - Service cost
    = $0.01 per core per hour
 
+- Logs, Specs, and Firewall Fee
+   = $0.005 per core per hour
+
 
-The sum of these costs is **$0.021935** per core/hour for standard workers, **$0.024388** per core/hour
-for highmem workers, and **$0.019393** per core/hour for highcpu workers. There is also an additional
+The sum of these costs is **$0.02684125** per core/hour for standard spot workers, **$0.02929425** per core/hour
+for highmem spot workers, and **$0.02429905** per core/hour for highcpu spot workers. There is also an additional
 cost of **$0.00023** per GB per hour of extra storage requested.
 
 At any given moment as many as four cores of the cluster may come from a 4 core machine if the worker type
 is standard. If a job is scheduled on this machine, then the cost per core hour is **$0.02774** plus
 **$0.00023** per GB per hour storage of extra storage requested.
 
-For jobs that run on non-preemptible machines, the costs are **$0.060462** per core/hour for standard workers, **$0.072114** per core/hour
-for highmem workers, and **$0.048365** per core/hour for highcpu workers.
+For jobs that run on non-preemptible machines, the costs are **$0.06449725** per core/hour for standard workers, **$0.076149** per core/hour
+for highmem workers, and **$0.0524218** per core/hour for highcpu workers.
 
 .. note::