Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Enable autoscaling based on worksets #1056

Closed
wants to merge 112 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
4f1cc9e
refactor pool autoscaling
demoray Feb 5, 2021
2d82caf
regen
demoray Feb 5, 2021
b8bb75d
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 5, 2021
66d684f
continued dev
demoray Feb 5, 2021
ba2fbf7
add autoscaling to CLI
demoray Feb 5, 2021
bcbddca
DRY for default image
demoray Feb 5, 2021
1314aea
lint
demoray Feb 5, 2021
3b2f7d6
use proper arg name
demoray Feb 5, 2021
ac00f22
use worksets & busy nodes
demoray Feb 5, 2021
c84f3a3
Merge branch 'main' into refactor-autoscale
demoray Feb 5, 2021
e5ef6f7
lint
demoray Feb 5, 2021
3972dd8
clarify states used for autoscale counting
demoray Feb 5, 2021
81021df
add an event for updating scaleset size
demoray Feb 5, 2021
15b25a4
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 5, 2021
e0fbe86
use set_new_size helper, which triggers event callbacks
demoray Feb 5, 2021
e3b081e
ensure scaleset size is within max size boundaries
demoray Feb 5, 2021
8f49be1
log pool.name rather than full pool info
demoray Feb 5, 2021
e850b27
reset siaze as we go based on ground truth for autoscaling scalesets
demoray Feb 5, 2021
f3b66a8
use synthetic worksets to trigger nodes to recycle
demoray Feb 5, 2021
7596cee
simplify logging
demoray Feb 5, 2021
669110e
fix min/max checks
demoray Feb 5, 2021
f8281e7
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 5, 2021
ab523c0
rename based on reality
demoray Feb 5, 2021
6f5c1e8
continued simplifying autoscale
demoray Feb 8, 2021
8096ff9
Merge branch 'main' into refactor-autoscale
demoray Feb 8, 2021
e36af31
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 8, 2021
12377bd
continued dev
demoray Feb 8, 2021
3d8edb1
unify log messages
demoray Feb 8, 2021
a84ab78
only count worksets with work
demoray Feb 9, 2021
097bc57
skip worksets without work units in work queue population as well
demoray Feb 9, 2021
c53eccc
clear synthetic worksets from the queue
demoray Feb 9, 2021
e43bf23
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 9, 2021
591a162
cleanup synthetic worksets directly
demoray Feb 9, 2021
7284420
added log count to workset cleanup
demoray Feb 9, 2021
bf017df
drastically reduce logging for cleanup
demoray Feb 9, 2021
a660979
set SignalR transport
demoray Feb 9, 2021
6f7c8cb
use the cached Azure Identity instance
demoray Feb 9, 2021
348fe2d
lint
demoray Feb 9, 2021
b6e33d1
Merge branch 'use-cached-identity' into refactor-autoscale
demoray Feb 9, 2021
480a760
Merge branch 'set-signalr-transport' into refactor-autoscale
demoray Feb 9, 2021
fc5f9d0
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 9, 2021
4e26c88
don't readd within receive
demoray Feb 9, 2021
71e8371
Fix wait_for_running for templates
demoray Feb 9, 2021
000bfbb
Merge branch 'fix-wait_for_running' into refactor-autoscale
demoray Feb 9, 2021
419bb5a
Merge branch 'main' into refactor-autoscale
demoray Feb 10, 2021
c5f93dc
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 16, 2021
3096b83
Merge branch 'main' into refactor-autoscale
bmc-msft Feb 16, 2021
488cc17
Merge branch 'main' into refactor-autoscale
demoray Mar 24, 2021
a0f1845
lint
demoray Mar 24, 2021
1363330
Merge remote-tracking branch 'main' into refactor-autoscale
chkeita Apr 19, 2021
32ee664
Merge branch 'main' into refactor-autoscale
chkeita Apr 23, 2021
5ab5696
small cleanup
chkeita Apr 26, 2021
943765c
Merge branch 'main' into refactor-autoscale
chkeita Apr 26, 2021
bfabcce
formatting
chkeita Apr 26, 2021
b15143d
undo could_shrink_scaleset changes
chkeita Apr 27, 2021
3359ff5
Merge branch 'main' into refactor-autoscale
chkeita Apr 27, 2021
ee30880
formatting
chkeita Apr 27, 2021
a9b736e
Merge branch 'main' into refactor-autoscale
chkeita May 5, 2021
a2f496f
Merge branch 'main' into refactor-autoscale
bmc-msft May 6, 2021
c8ab30d
Merge branch 'main' into refactor-autoscale
chkeita May 10, 2021
51c82a6
fix build, remove unused reference to yaque
chkeita May 10, 2021
2f7fbc0
formatting
chkeita May 10, 2021
37e714f
build fix
chkeita May 10, 2021
f63f1ce
Merge branch 'main' into refactor-autoscale
bmc-msft May 11, 2021
b2ddd63
Merge remote-tracking branch 'upstream/main' into refactor-autoscale
chkeita May 17, 2021
9056252
update integration tests autoscale
chkeita May 17, 2021
4306753
Tests fix
chkeita May 17, 2021
4669e51
fix import
chkeita May 17, 2021
53011a4
Merge branch 'main' into refactor-autoscale
bmc-msft May 20, 2021
bdc8a54
Merge branch 'main' into refactor-autoscale
bmc-msft May 27, 2021
bc9b0b0
Merge branch 'main' into refactor-autoscale
bmc-msft Jun 7, 2021
b7da587
Merge branch 'main' into refactor-autoscale
demoray Jul 7, 2021
5206ee8
merge changes
demoray Jul 7, 2021
4a39638
regen
demoray Jul 7, 2021
9d6b897
move to can_update
demoray Jul 7, 2021
aa1600c
remove redundent entry
demoray Jul 7, 2021
b61c471
use the built-in size validators
demoray Jul 7, 2021
fa40b76
lint
demoray Jul 7, 2021
05c4854
Merge branch 'use-pydantic-builtin-validators' into refactor-autoscale
demoray Jul 7, 2021
144475e
remove extra docs
demoray Jul 7, 2021
aecd130
make backwards compatable from the API perspective
demoray Jul 7, 2021
9ad1781
use set_size
demoray Jul 7, 2021
ac1d3bc
use current resize functionality
demoray Jul 7, 2021
37e943b
set pool_id on node creation
demoray Jul 7, 2021
74280d7
Merge branch 'set-pool-id-on-node-create' into refactor-autoscale
demoray Jul 7, 2021
a6ed5fd
remove unused change
demoray Jul 7, 2021
83435d7
remove changes from upstream
demoray Jul 7, 2021
dd11a4c
add a pool-based shrink-queue
demoray Jul 7, 2021
0b75f2b
Merge branch 'pool-shrink-queue' into refactor-autoscale
demoray Jul 7, 2021
6b61a69
more removing changes from usptream
demoray Jul 7, 2021
f40f630
remove from upstream
demoray Jul 7, 2021
7b96deb
Merge branch 'main' into refactor-autoscale
demoray Jul 8, 2021
27307f7
decode doesn't return an optional
demoray Jul 8, 2021
5010047
remove unused code, and only use set_state for changing scaleset state
demoray Jul 8, 2021
fb85932
Merge branch 'main' into refactor-autoscale
demoray Jul 9, 2021
bf6aa93
add unit tests for calculating change size
demoray Jul 9, 2021
e44a681
regen docs
demoray Jul 9, 2021
cb54d30
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 9, 2021
8e75be3
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 9, 2021
cccb433
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 9, 2021
8527eea
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 13, 2021
d34df42
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 14, 2021
96b9234
add extra available node size
demoray Jul 14, 2021
9976de9
use available size
demoray Jul 14, 2021
5f38f48
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 14, 2021
1b265dd
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 14, 2021
9ef9d7a
Merge branch 'main' into refactor-autoscale
bmc-msft Jul 19, 2021
728eeee
Merge branch 'main' into refactor-autoscale
demoray Aug 24, 2021
bed7ec1
Merge branch 'main' into refactor-autoscale
bmc-msft Sep 2, 2021
4036bdf
Merge branch 'main' into refactor-autoscale
bmc-msft Sep 20, 2021
85f400c
Merge branch 'main' into refactor-autoscale
bmc-msft Sep 23, 2021
a6ab9b4
Merge branch 'main' into refactor-autoscale
bmc-msft Sep 24, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions docs/webhook_events.md
Original file line number Diff line number Diff line change
Expand Up @@ -1245,20 +1245,26 @@ Each event will be submitted via HTTP POST to the user provided URL.
"title": "Ephemeral Os Disks",
"type": "boolean"
},
"extra_available_size": {
"description": "number of unused but available nodes",
"minimum": 0,
"title": "Extra Available Size",
"type": "integer"
},
"image": {
"title": "Image",
"type": "string"
},
"max_size": {
"default": 1000,
"maximum": 1000,
"description": "maximum size of the pool",
"minimum": 0,
"title": "Max Size",
"type": "integer"
},
"min_size": {
"default": 0,
"maximum": 1000,
"description": "minimum size of the pool",
"minimum": 0,
"title": "Min Size",
"type": "integer"
Expand All @@ -1268,6 +1274,8 @@ Each event will be submitted via HTTP POST to the user provided URL.
"type": "string"
},
"scaleset_size": {
"default": 1,
"description": "unused",
"title": "Scaleset Size",
"type": "integer"
},
Expand All @@ -1283,7 +1291,6 @@ Each event will be submitted via HTTP POST to the user provided URL.
},
"required": [
"image",
"scaleset_size",
"vm_sku"
],
"title": "AutoScaleConfig",
Expand Down Expand Up @@ -4791,20 +4798,26 @@ Each event will be submitted via HTTP POST to the user provided URL.
"title": "Ephemeral Os Disks",
"type": "boolean"
},
"extra_available_size": {
"description": "number of unused but available nodes",
"minimum": 0,
"title": "Extra Available Size",
"type": "integer"
},
"image": {
"title": "Image",
"type": "string"
},
"max_size": {
"default": 1000,
"maximum": 1000,
"description": "maximum size of the pool",
"minimum": 0,
"title": "Max Size",
"type": "integer"
},
"min_size": {
"default": 0,
"maximum": 1000,
"description": "minimum size of the pool",
"minimum": 0,
"title": "Min Size",
"type": "integer"
Expand All @@ -4814,6 +4827,8 @@ Each event will be submitted via HTTP POST to the user provided URL.
"type": "string"
},
"scaleset_size": {
"default": 1,
"description": "unused",
"title": "Scaleset Size",
"type": "integer"
},
Expand All @@ -4829,7 +4844,6 @@ Each event will be submitted via HTTP POST to the user provided URL.
},
"required": [
"image",
"scaleset_size",
"vm_sku"
],
"title": "AutoScaleConfig",
Expand Down
171 changes: 0 additions & 171 deletions src/api-service/__app__/onefuzzlib/autoscale.py

This file was deleted.

8 changes: 8 additions & 0 deletions src/api-service/__app__/onefuzzlib/azure/queue.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
from azure.storage.queue import (
QueueMessage,
QueueSasPermissions,
QueueServiceClient,
generate_queue_sas,
Expand Down Expand Up @@ -172,9 +173,16 @@ def peek_queue(
result.append(object_type.parse_obj(raw))
except ResourceNotFoundError:
return result

return result


def decode_message(message: QueueMessage, object_type: Type[A]) -> A:
decoded = base64.b64decode(message.content)
raw = json.loads(decoded)
return object_type.parse_obj(raw)


def queue_object(
name: QueueNameType,
message: BaseModel,
Expand Down
18 changes: 0 additions & 18 deletions src/api-service/__app__/onefuzzlib/tasks/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
from onefuzztypes.models import Error
from onefuzztypes.models import Task as BASE_TASK
from onefuzztypes.models import TaskConfig, TaskVm, UserInfo
from onefuzztypes.primitives import PoolName

from ..azure.image import get_os
from ..azure.queue import create_queue, delete_queue
Expand Down Expand Up @@ -171,23 +170,6 @@ def get_by_task_id(cls, task_id: UUID) -> Union[Error, "Task"]:
task = tasks[0]
return task

@classmethod
def get_tasks_by_pool_name(cls, pool_name: PoolName) -> List["Task"]:
tasks = cls.search_states(states=TaskState.available())
if not tasks:
return []

pool_tasks = []

for task in tasks:
task_pool = task.get_pool()
if not task_pool:
continue
if pool_name == task_pool.name:
pool_tasks.append(task)

return pool_tasks

def mark_stopping(self) -> None:
if self.state in TaskState.shutting_down():
logging.debug(
Expand Down
Loading