Placement scriptlet not called when moving instances between projects

[victoitor]

When moving an instance between projects and a placement scriptlet is defined through instances.placement.scriptlet, it is not called to place the instance in the new project. This can lead to inconsistent configurations similar to #589 if the consistency of a configuration is checked through the placement scriptlet and user keys instead of through incus defined keys.

[victoitor]

Btw, I know that consistency cannot be enforced everywhere. But it seems projects are designed to have different rules for consistency and moving between them should at least recheck those rules at least once. So it would make sense to call the placement scriptlet as if enforces general rules.

[stgraber]

root@server01:~# incus move u1 --project foo --target-project bar
root@server01:~# incus move u1 --project bar --target-project foo

Now gives:

ERROR  [2024-09-25T14:42:44Z] Instance placement scriptlet: stgraber: name=u1, reason=relocation, candidates=["server04", "server03"] 
ERROR  [2024-09-25T14:42:45Z] Instance placement scriptlet: stgraber: name=u1, reason=new, candidates=["server04"] 
ERROR  [2024-09-25T14:42:50Z] Instance placement scriptlet: stgraber: name=u1, reason=relocation, candidates=["server01", "server02"] 
ERROR  [2024-09-25T14:42:50Z] Instance placement scriptlet: stgraber: name=u1, reason=new, candidates=["server02"] 

The relocation event is what sets the final location, then the actual copy re-enters the placement logic (as it's effectively a new instance from our point of view) but this time with the fixed location.

[victoitor]

I just tested the move on incus 6.6 and on a move between projects, request.project is set as the current project, and not the target project. So a lot of issues are happening when using the scriptlet for a move operation between projects.

Btw, I decided not to create a new issue since it's related to this one. Should I create a new one or just keep the discussion here?

[victoitor]

Something is odd and I still can't quite put my finger on it.

In the start of my scriptlet I have the following to print the request, the candidate members and the project.

def instance_placement(request, candidate_members):
    project = get_project( request.project )
    log_error("SCRIPTLET DEBUG Request: ", request, "\nSCRIPTLET DEBUG Candidade members: ", candidate_members, "\nSCRIPTLET DEBUG Project: ", project)

When I copy between projects with the following

incus copy victor-teste victor-teste --target-project intel-12700

I get logged

ired: true\ntimezone: America/Fortaleza\nusers:\n- gecos: Default pargo user\n  groups: sudo, video, render\n  name: pargo\n  lock_passwd: true\n  sudo: ALL=(ALL) NOPASSWD:ALL\n  shell: /bin/bash\n", "image.architecture": "amd64", "image.description": "Debian bookworm amd64 (20241004_05:24)", "image.os": "Debian", "image.release": "bookworm", "image.serial": "20241004_05:24", "image.type": "squashfs", "image.variant": "default", "limits.cpu": "0-15", "limits.memory": "24GB", "security.nesting": "true", "user.responsavel": "Victor Campos", "volatile.base_image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320"}, "devices": {"eth0": {"name": "eth0", "nictype": "bridged", "parent": "br0", "type": "nic"}, "root": {"path": "/", "pool": "local", "type": "disk"}}, "ephemeral": False, "profiles": ["default"], "restore": "", "stateful": False, "description": "", "name": "victor-teste", "source": {"type": "copy", "certificate": "", "alias": "", "fingerprint": "", "properties": {}, "server": "", "secret": "", "protocol": "", "base-image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320", "mode": "", "operation": "", "secrets": {}, "source": "victor-teste", "live": False, "instance_only": False, "refresh": False, "project": "amd-5700g", "allow_inconsistent": False}, "instance_type": "", "type": "container", "start": False, "reason": "new", "project": "intel-12700"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-15", "user.experimentos.limits.memory": "24GB"}, "groups": ["intel-12700"], "server_name": "intel02", "url": "https://10.11.16.32:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database-standby"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-15", "user.experimentos.limits.memory": "24GB"}, "groups": ["intel-12700"], "server_name": "intel03", "url": "https://10.11.16.33:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-15", "user.experimentos.limits.memory": "24GB"}, "groups": ["intel-12700"], "server_name": "intel01", "url": "https://10.11.16.31:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.cluster.groups": "intel-12700", "restricted.cluster.target": "allow", "restricted.containers.nesting": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "0-15", "user.node.limits.cpu.unique": "true", "user.node.limits.memory": "24GB", "user.node.represented": "true", "user.node.represented.unique": "true"}, "description": "Experimentos - máquinas intel-12700", "name": "intel-12700", "used_by": []} 

So the project is correct and the candidate members as well.

When I move between projects with the following

incus move victor-teste --target-project intel-12700

I get logged

ERROR  [2024-10-04T09:24:46-03:00] Instance placement scriptlet: SCRIPTLET DEBUG Request: {"architecture": "x86_64", "config": {"cloud-init.vendor-data": "#cloud-config\npackage_update: true\npackage_upgrade: true\npackage_reboot_if_required: true\ntimezone: America/Fortaleza\nusers:\n- gecos: Default pargo user\n  groups: sudo, video, render\n  name: pargo\n  lock_passwd: true\n  sudo: ALL=(ALL) NOPASSWD:ALL\n  shell: /bin/bash\n", "image.architecture": "amd64", "image.description": "Debian bookworm amd64 (20241004_05:24)", "image.os": "Debian", "image.release": "bookworm", "image.serial": "20241004_05:24", "image.type": "squashfs", "image.variant": "default", "limits.cpu": "0-15", "limits.memory": "24GB", "security.nesting": "true", "user.responsavel": "Victor Campos", "volatile.apply_template": "create", "volatile.base_image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320", "volatile.eth0.hwaddr": "00:16:3e:36:f3:18"}, "devices": {"eth0": {"name": "eth0", "nictype": "bridged", "parent": "br0", "type": "nic"}, "root": {"path": "/", "pool": "local", "type": "disk"}}, "ephemeral": False, "profiles": ["default"], "restore": "", "stateful": False, "description": "", "name": "victor-teste", "source": {"type": "copy", "certificate": "", "alias": "", "fingerprint": "", "properties": {}, "server": "", "secret": "", "protocol": "", "base-image": "", "mode": "", "operation": "", "secrets": {}, "source": "victor-teste", "live": False, "instance_only": False, "refresh": False, "project": "amd-5700g", "allow_inconsistent": False}, "instance_type": "", "type": "container", "start": False, "reason": "new", "project": "intel-12700"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-15", "user.experimentos.limits.memory": "24GB"}, "groups": ["intel-12700"], "server_name": "intel02", "url": "https://10.11.16.32:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.cluster.groups": "intel-12700", "restricted.cluster.target": "allow", "restricted.containers.nesting": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "0-15", "user.node.limits.cpu.unique": "true", "user.node.limits.memory": "24GB", "user.node.represented": "true", "user.node.represented.unique": "true"}, "description": "Experimentos - máquinas intel-12700", "name": "intel-12700", "used_by": []} 

So the target project is correct but the candidate members is not.

When I move or copy to another project things are different.

Using incus copy victor-teste victor-teste --target-project auxiliar I get

ERROR  [2024-10-04T09:26:26-03:00] Instance placement scriptlet: SCRIPTLET DEBUG Request: {"architecture": "x86_64", "config": {"cloud-init.vendor-data": "#cloud-config\npackage_update: true\npackage_upgrade: true\npackage_reboot_if_required: true\ntimezone: America/Fortaleza\nusers:\n- gecos: Default pargo user\n  groups: sudo, video, render\n  name: pargo\n  lock_passwd: true\n  sudo: ALL=(ALL) NOPASSWD:ALL\n  shell: /bin/bash\n", "image.architecture": "amd64", "image.description": "Debian bookworm amd64 (20241004_05:24)", "image.os": "Debian", "image.release": "bookworm", "image.serial": "20241004_05:24", "image.type": "squashfs", "image.variant": "default", "limits.cpu": "6-7,14-15", "limits.cpu.allowance": "100%", "limits.memory": "1GiB", "user.responsavel": "Victor Campos", "volatile.base_image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320"}, "devices": {"eth0": {"name": "eth0", "nictype": "bridged", "parent": "br0", "type": "nic"}, "root": {"path": "/", "pool": "local", "type": "disk"}}, "ephemeral": False, "profiles": ["default"], "restore": "", "stateful": False, "description": "", "name": "victor-teste", "source": {"type": "copy", "certificate": "", "alias": "", "fingerprint": "", "properties": {}, "server": "", "secret": "", "protocol": "", "base-image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320", "mode": "", "operation": "", "secrets": {}, "source": "victor-teste", "live": False, "instance_only": False, "refresh": False, "project": "amd-5700g", "allow_inconsistent": False}, "instance_type": "", "type": "container", "start": False, "reason": "new", "project": "auxiliar"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database-leader", "database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd01", "url": "https://10.11.16.11:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": [], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd02", "url": "https://10.11.16.12:8443", "database": False, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": [], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd03", "url": "https://10.11.16.13:8443", "database": False, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database-standby"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd04", "url": "https://10.11.16.14:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.backups": "allow", "restricted.cluster.groups": "amd-5700g", "restricted.cluster.target": "allow", "restricted.containers.lowlevel": "allow", "restricted.containers.nesting": "allow", "restricted.devices.disk": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "6-7,14-15", "user.node.represented": "true"}, "description": "Montagem e estacionamento", "name": "auxiliar", "used_by": []} 

in which everything is correct.

Using incus move victor-teste --target-project auxiliar I get

ERROR  [2024-10-04T09:28:13-03:00] Instance placement scriptlet: SCRIPTLET DEBUG Request: {"architecture": "", "config": {"cloud-init.vendor-data": "#cloud-config\npackage_update: true\npackage_upgrade: true\npackage_reboot_if_required: true\ntimezone: America/Fortaleza\nusers:\n- gecos: Default pargo user\n  groups: sudo, video, render\n  name: pargo\n  lock_passwd: true\n  sudo: ALL=(ALL) NOPASSWD:ALL\n  shell: /bin/bash\n", "image.architecture": "amd64", "image.description": "Debian bookworm amd64 (20241004_05:24)", "image.os": "Debian", "image.release": "bookworm", "image.serial": "20241004_05:24", "image.type": "squashfs", "image.variant": "default", "limits.cpu": "0-5,8-13", "limits.memory": "24GB", "security.nesting": "true", "user.responsavel": "Victor Campos", "volatile.apply_template": "create", "volatile.base_image": "65ef3a44145b9617a64a0c82357233f2363f207789070bbc6e144e9073d1a320", "volatile.cloud-init.instance-id": "ba35f52d-343e-41a2-8557-88ae0c7e49f7", "volatile.eth0.hwaddr": "00:16:3e:36:f3:18", "volatile.idmap.base": "0", "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]", "volatile.last_state.idmap": "[]", "volatile.uuid": "598838d8-6555-4221-99fb-3b101f3286de", "volatile.uuid.generation": "598838d8-6555-4221-99fb-3b101f3286de"}, "devices": {"eth0": {"name": "eth0", "nictype": "bridged", "parent": "br0", "type": "nic"}, "root": {"path": "/", "pool": "local", "type": "disk"}}, "ephemeral": False, "profiles": [], "restore": "", "stateful": False, "description": "", "name": "victor-teste", "source": {"type": "", "certificate": "", "alias": "", "fingerprint": "", "properties": {}, "server": "", "secret": "", "protocol": "", "base-image": "", "mode": "", "operation": "", "secrets": {}, "source": "", "live": False, "instance_only": False, "refresh": False, "project": "", "allow_inconsistent": False}, "instance_type": "", "type": "", "start": False, "reason": "relocation", "project": "amd-5700g"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database-leader", "database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd01", "url": "https://10.11.16.11:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.cluster.groups": "amd-5700g", "restricted.cluster.target": "allow", "restricted.containers.nesting": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "0-5,8-13", "user.node.limits.cpu.unique": "true", "user.node.limits.memory": "24GB", "user.node.represented": "true", "user.node.represented.unique": "true"}, "description": "Experimentos - máquinas amd-5700g", "name": "amd-5700g", "used_by": []} 

In which I get an incorrect list of candidate members and an incorrect request.project (which is the current project).