Failed to upload 151MB or 3GB iso file

[rgl]

Overview of the Issue

The plugin cannot upload an iso file with a 151MB or 3GB when using a additional_iso_files block.

Please note that using iso_url works fine. This problem seems to apply only to the additional_iso_files block.

Reproduction Steps

• create a packer file that references a not so big 151MB or a big 3G+ iso file.
• execute packer build.

Plugin and Packer version

Plugin v1.1.3

Packer v1.9.1

Simplified Packer Buildfile

source "proxmox-iso" "example" {
  ...
  additional_iso_files {
    device       = "scsi5"
    iso_url      = "tmp/example.iso"
    unmount      = true
    iso_checksum = "none"
  }
}

Operating system and Environment details

Ubuntu 22-04 amd64 host.

Log Fragments and crash.log files

Abbreviated:

==> proxmox-iso.example: tmp/example.iso => /home/vagrant/Projects/macos-proxmox/tmp/example.iso
==> proxmox-iso.example: 501 for data too large
==> proxmox-iso.example: 501 for data too large
==> proxmox-iso.example: Step "stepUploadAdditionalISO" failed, aborting...
==> proxmox-iso.example: aborted: skipping cleanup of step "StepDownload"
==> proxmox-iso.example: aborted: skipping cleanup of step "StepCreateCD"
==> proxmox-iso.example: aborted: skipping cleanup of step "stepUploadISO"
==> proxmox-iso.example: aborted: skipping cleanup of step "StepDownload"
Build 'proxmox-iso.example' errored after 3 seconds 560 milliseconds: 501 for data too large

[nick-holmquist]

Seeing this issue with 1.1.3 (Specifying) or 1.1.5

The only place in code I can find the 501 for data too large is in the pve http server: https://git.proxmox.com/?p=pve-http-server.git;a=blob;f=PVE/APIServer/AnyEvent.pm;h=c1dade89da47813ca61601d629fe5e565d197221;hb=refs/heads/master#l1327

if ($len > $limit_max_post) {
$self->error($reqstate, 501, "for data too large");
return;
}

The variable for max post is set as such: $limit_max_post = 64*1024;
Ultimately, when looking at the content length sent in the packer build process it definitely appears to be over that limit. That being said, uploading the same iso file with curl works without an issue and the content length is the same (Minus some additional bytes from the packer process).

My guess is the error has to be in either the proxmox plugin or the client but so far I can't debug where.
https://github.com/Telmate/proxmox-api-go/blob/03f4e428f6c62f9496d388d1735777b6c832cba2/proxmox/client.go#L1283
https://github.com/hashicorp/packer-plugin-proxmox/blob/main/builder/proxmox/common/step_upload_additional_iso.go

[lbajolet-hashicorp]

With @nick-holmquist's contribution on PR #238, this might fix this issue.

I triggered a release for v1.1.6, which should be available within the hour if things go according to plan. Could you confirm if you're still hitting the issue with that version @rgl?

Thanks

[rgl]

I've tried with v1.1.7, and it still has the same problem :-(

PS I've created a packer template that reproduces the problem at https://github.com/rgl/packer-plugin-proxmox-issue-197.

[mpywell]

I also had this issue in v1.1.6 and v1.1.7 until I worked out that you need to provide a iso_storage_pool value in the additional_iso_files block. Packer errors if iso_storage_pool isn't defined for the main ISO attached to a VM, but doesn't error for additional_iso_files blocks.

Running the packer build with the -debug flag showed the POST URI during ISO upload as /nodes/proxmox_node/storage//upload, the underlying API module is performing a substitution for the storage pool between storage and upload (https://github.com/Telmate/proxmox-api-go/blob/master/proxmox/client.go#L1369C2-L1369C86).

[rgl]

@mpywell oh nice! thank you for catching that! it's working here too :-)

by any change, will you be able to create a PR to make additional_iso_files validate iso_storage_pool? or even default its value to the parent iso_storage_pool?

[mpywell]

@rgl PR #252 submitted, given there could be scenarios where a build requires ISO attachments from different storage pools, have kept it simple and added validation only.

[lbajolet-hashicorp]

With #252 merged, I'm pre-emptively marking this issue as needs-verification; I'll release the plugin next week, and we can close this after a few weeks/months, unless we see more reports.
Thanks all for following up on this one.

[Torqu3Wr3nch]

Specified iso_storage_pool in the additional_iso_files block which resolved this issue for me. Thank you @mpywell and @lbajolet-hashicorp for your investigation and work.