Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

snapshot of Instance(VM) ceph-rbd based return -1 size #14017

Closed
lccnblack opened this issue Aug 30, 2024 · 9 comments
Closed

snapshot of Instance(VM) ceph-rbd based return -1 size #14017

lccnblack opened this issue Aug 30, 2024 · 9 comments
Assignees
@kadinsayani
Labels
Incomplete Waiting on more information from reporter
Milestone
lxd-6.2

Comments

@lccnblack
Copy link

Required information

  • Distribution: 22.04.4 LTS

  • Distribution version:
    lxd 6.1-c14927a 29551 6.1/stable canonical** in-cohort
    lxd 6.1-0d4d89b 29469 6.1/stable canonical** disabled,in-cohort
    microceph 0+git.4a608fc 793 quincy/stable canonical** in-cohort
    microcloud 1.1-04a1c49 734 latest/stable canonical** in-cohort

  • The output of "snap list --all lxd core20 core22 core24 snapd":

  • The output of "lxc info" or if that fails:

    • Kernel version: 5.15.0-116-generic
    • LXC version:
    • LXD version:
    • Storage backend in use: microceph

Issue description

The customer has a vm that is ceph rbd based.
lxc config show / shows that the size is -1.
However, the rbd du /.block returns normally.
What's interesting is that only the first snapshot returns the normal size, and all the other snapshots return size -1.
It seems that the other snapshots are taken via the schedule.

lxc info ConvertVM

Name: ConvertVM
Status: RUNNING
Type: virtual-machine
Architecture: x86_64
Location: node2
PID: 71917
Created: 2024/06/14 02:28 UTC
Last Used: 2024/08/28 09:08 UTC
 
Resources:
  Processes: 23
  CPU usage:
    CPU usage (in seconds): 3041
  Memory usage:
    Memory (current): 1.85GiB
  Network usage:
    enp5s0:
      Type: broadcast
      State: UP
      Host interface: tap716b3cff
      MAC address: 00:16:3e:4d:24:d0
      MTU: 1450
      Bytes received: 12.42MB
      Bytes sent: 907.53kB
      Packets received: 12904
      Packets sent: 9896
      IP addresses:
        inet:  240.242.0.55/8 (global)
        inet6: fe80::216:3eff:fe4d:24d0/64 (link)
    enp6s0:
      Type: broadcast
      State: DOWN
      Host interface: mac4d62bf00
      MAC address: 00:16:3e:d2:2a:22
      MTU: 1500
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
      IP addresses:
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 89.40MB
      Bytes sent: 89.40MB
      Packets received: 1164630
      Packets sent: 1164630
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)
 
Snapshots:
+--------+----------------------+----------------------+----------+
|  NAME  |       TAKEN AT       |      EXPIRES AT      | STATEFUL |
+--------+----------------------+----------------------+----------+
| snap1  | 2024/07/31 03:49 UTC |                      | NO       |
+--------+----------------------+----------------------+----------+
| snap2  | 2024/07/31 03:56 UTC |                      | NO       |
+--------+----------------------+----------------------+----------+
| snap23 | 2024/08/23 11:59 UTC | 2024/08/30 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap24 | 2024/08/24 11:59 UTC | 2024/08/31 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap25 | 2024/08/25 11:59 UTC | 2024/09/01 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap26 | 2024/08/26 12:00 UTC | 2024/09/02 12:00 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap27 | 2024/08/28 11:59 UTC | 2024/09/04 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap28 | 2024/08/29 11:59 UTC | 2024/09/05 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+

lxc config show ConvertVM/snap22

architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tapf70fb09c
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac32d52912
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-08-22T11:59:56.735294543Z
expires_at: 2024-08-29T11:59:56.733439655Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tapf70fb09c
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac32d52912
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap22
profiles:
- default
- macvlan
stateful: false
size: -1

lxc config show ConvertVM/snap1

architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tap019c0ee9
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac626955d9
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-07-31T03:49:50.041802922Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tap019c0ee9
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac626955d9
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
- macvlan
stateful: false
size: 94795464704

rbd du remote/virtual-machine_ConvertVM.block

NAME                                             PROVISIONED  USED
virtual-machine_ConvertVM.block@snapshot_snap1       210 GiB   88 GiB
virtual-machine_ConvertVM.block@snapshot_snap2       210 GiB   92 MiB
virtual-machine_ConvertVM.block@snapshot_snap20      210 GiB   12 GiB
virtual-machine_ConvertVM.block@snapshot_snap21      210 GiB  117 GiB
virtual-machine_ConvertVM.block@snapshot_snap22      210 GiB  1.6 GiB
virtual-machine_ConvertVM.block@snapshot_snap23      210 GiB   58 GiB
virtual-machine_ConvertVM.block@snapshot_snap24      210 GiB  1.1 GiB
virtual-machine_ConvertVM.block@snapshot_snap25      210 GiB  1.1 GiB
virtual-machine_ConvertVM.block@snapshot_snap26      210 GiB   15 GiB
virtual-machine_ConvertVM.block                      210 GiB  1.3 GiB
<TOTAL>                                              210 GiB  296 GiB
@tomponline
Copy link
Member

Did you have lxc storage set <pool> ceph.rbd.du=true enabled?

@tomponline tomponline added this to the lxd-6.2 milestone Aug 30, 2024
@lccnblack
Copy link
Author

Thank you for the reply!
I will ask customer to try it.
However, in my lab, even though I didn't set this parameter, I still can't see the problem of -1.
It seems that this parameter is for stopped instance.
https://documentation.ubuntu.com/lxd/en/latest/reference/storage_ceph/#storage-pool-configuration
This option specifies whether to use RBD du to obtain disk usage data for stopped instances.

@lccnblack
Copy link
Author 

root@node4:~# lxc storage set remote ceph.rbd.du=true
root@node4:~# lxc config show ConvertVM/snap40
architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tape7b3ae59
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: macc762736b
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-09-12T11:59:55.432874933Z
expires_at: 2024-09-19T11:59:55.430551817Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tape7b3ae59
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: macc762736b
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap40
profiles:
- default
- macvlan
stateful: false
size: -1

Do we need to take a new snapshot after setting ceph.rbd.du=true?

@kadinsayani
Copy link
Contributor

kadinsayani commented Oct 10, 2024 •
edited
Loading

Do we need to take a new snapshot after setting ceph.rbd.du=true?

Can you please try running lxc storage set <pool> ceph.rbd.du=true and then take another snapshot?

@tomponline
Copy link
Member

Can you please try running lxc storage set <pool> ceph.rbd.du=true and then take another snapshot?

Does this affect existing snapshots?

@tomponline tomponline assigned kadinsayani and unassigned boltmark Oct 15, 2024
@kadinsayani
Copy link
Contributor

Does this affect existing snapshots?

Yes - after changing the setting and running lxc config show <vm>/<snap>, the expected size is returned.

@kadinsayani
Copy link
Contributor

kadinsayani commented Oct 18, 2024 •
edited
Loading

Here are my test results using a VM with a ceph backend (single node cluster) 

root@v1:~# lxc storage set pool1 ceph.rbd.du=false
root@v1:~# lxc snapshot v1 snap1
root@v1:~# lxc config show v1/snap1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
created_at: 2024-10-18T20:55:57.744013777Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: pool1
    type: disk
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
expanded_devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: pool1
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
stateful: false
size: -1
root@v1:~# lxc storage set pool1 ceph.rbd.du=true
root@v1:~# lxc config show v1/snap1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
created_at: 2024-10-18T20:55:57.744013777Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: pool1
    type: disk
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
expanded_devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: pool1
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
stateful: false
size: 138412032

@tomponline
Copy link
Member

@lccnblack happy to close this now?

@tomponline tomponline added the Incomplete Waiting on more information from reporter label Oct 21, 2024
@kadinsayani
Copy link
Contributor

Closing this issue but if you have any concerns or run into any more issues please feel free to reopen this or reach out! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kadinsayani kadinsayani

Labels
Incomplete Waiting on more information from reporter
Projects
None yet
Milestone
lxd-6.2
Development

No branches or pull requests
4 participants
@tomponline @kadinsayani @lccnblack @boltmark