This allow to use a file path towards a folder instead of a VHD

Signed-off-by: BenjiReis <benjamin.reis@vates.fr>
Co-authored-by: Ronan Abhamon <ronan.abhamon@vates.fr>

- `Pool.set_uefi_certificates` is implemented and writes the certificates
on all its hosts' disks

- `Host.set_uefi_certificates` is now deprecated and transmit the call to the pool method
- `Host.uefi_certificates` is deprecated as well as it's getter, the value is not updated.

- On XAPI startup certificates stored in XAPI's `Pool.uefi_certificates`
are written on disks

- When a host joins the pool's certificates are written on its disk.

This means:
	- At every XAPI startup the certificates in host disks are synced with
XAPI's `Pool.uefi_certificates`

	- When `Pool.set_uefi_certificates` is called all hosts are synced on their disks
with XAPI's `Pool.uefi_certificates`.

Also: `Host.set_uefi_certificates` calls should be replaced by `Pool.set_uefi_certificates`, this requires changes
in external libs (varstored, uefistored, etc) to set the pool's certificates: call `Pool.set_uefi_certificates`.

See: xapi-project#4647

Signed-off-by: BenjiReis <benjamin.reis@vates.fr>

This restricts the /repository endpoint to local-root only. This
endpoint is exposed only on the coordinator, and used by other pool
members to access the pool's yum repo mirror.

Yum calls to this endpoint now require a pool secret to be used in a
cookie, which is implemented through a yum plugin.

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>

CP-35846: Restrict access to internal yum repo server (members only)

Close both FDs.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

Signed-off-by: Edwin Török <edvin.torok@citrix.com>

emu-manager needs '-dm qemu' flag for migration to work correctly with
upstream QEMU.
Otherwise emu-manager doesn't send xen-set-global-dirty-log and some
guest drivers may crash upon resume (e.g. the mouse).

Treat Qemu_upstream_uefi as upstream too and pass the flag.

Commit 0e8624a introduced '-dm qemu'
but the UEFI version of the qemu profile didn't exist yet at the time.

Later commit 971d788 introduced the
UEFI qemu profile, and the build still worked due to the wildcard,
so the missing flag went unnoticed until now.

Replace the wildcard with an explicit list of when '-dm qemu' shouldn't
be added, so the next time we introduce a new qemu profile we get a
build failure and must decide which behaviour would be correct.

Signed-off-by: Edwin Török <edvin.torok@citrix.com>

It is unused and incomplete (it lacks UEFI).
Drop it to avoid confusion.

Signed-off-by: Edwin Török <edvin.torok@citrix.com>

After starting a new server (for a varstored), report the number of open
file descriptors in the log.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

There were a lot of warnings here, mostly missing named parameters,
quite a few of these were able to be automated using sed to add tildes,
others couldn't because of how irregular the name of the parameters
are, sometimes using self, sometimes the type of object; or they were
too particular.

There we also unused bindings which allowed for removal of parameters.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

…/CA-364138

 CA-364138 XSI-1217: fix FD leak, Unix.EMFILE

These were mostly about removing unused code and removing usages of
xenstore and xenctrl handles which allows avoiding opening them in quite
a few places.

I would not have expected the code around PCI devices to be unused, I
wonder if it should actually be used.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

…-reformats

maintenance: ignore reformatting commits in git blame

CP-39551: avoid warnings in xapi-cli-server and xenopsd/xc

…een-hosts

Sync varstore certificates in XAPI with those on disks

Add `9pfs` backend to vbds

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

CP-39551: avoid warnings in xenopsd/xc and libs

Issues:
- DNS is not set when configuring an IPv6
- When a DNS is set all dns entries are overriden, this is problematic when using both IPv4 and IPv6 in the same PIF.

Fixes:
- set DNS when an IPv6 is configured
- only overrides DNS entries of the same family when an IP/IPv6 is configured
- 'determine_gateway_and_dns_ifs' also look for 'ipv6_configuration_mode' to find possible gateway and dns pifs

Signed-off-by: BenjiReis <benjamin.reis@vates.fr>

fixes regarding DNS management

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

The compression library Xapi_compression spawns processes to compress
and uncompress streams. To facilitate easier testing, add a small
stand-alone filter that compresses or uncompresses its standard input.
This filter requires the presence of forkexecd to work - so typically a
Citrix Hypervisor installation. We don't install this binary but it is
available for developers.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

In commit d951478, a new function was added to restrict the repository
access to pool members only. With this funciton, the pool members need
to use pool_secret in access requests to the mirrorred repositories on
the pool coordinator for authentication.

But in updating, this would cause the pool members can't access the
mirrored repositories as at that time the pool coordinator has been
updated to check the pool_secret while the pool members have not been
udpated to use pool_secret.

Per suggestion from Rob Hoes, this commit adds a temporary feature flag
to make the pool coordinator be able to distinguish the case in the
mixed mode in updating.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

xs-opam's CI is the only environment where /tmp is not writable and
XDG_RUNTIME_DIR is not set. Using TMPDIR with /tmp as a backup will mean
the socket is writeable in all environments

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

…ernal_repo_access

Add temporary feature 'Internal_repo_access' to allow update in mix mode

Add binary xapi_gzip for testing Xapi_compression

These files are owned by other RPM packages.
Also upgrade would need to be taken into consideration where
uefi-certificates would already be set on a pool, but just with KEK+db
without PK+dbx.
In that case if we order uploading PK/dbx and restart xapi the wrong way
around we may get these files wiped completely from the system.
Wiping just KEK+db should be enough to ensure we don't have stale files
around.

Signed-off-by: Edwin Török <edvin.torok@citrix.com>

…/pkauth

CA-366430: do not wipe PK.auth/dbx.auth

Normally gzip stops reading input when it has reached its own end-of
input marker even when that is before the end of input. This is
overridden by --force for decompression and it is creating a problem for
us because suspend images on disk are represented as devices that are
larger than the actual content - so --force kept reading beyond the
actual input and this in turn was not accepted by the code taking the
uncompressed output.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

CP-39551: Remove all compiler warnings (finally!)

Also adds a pretty_printer and an equal function

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Change signature functions that generate UUIDs from inputs to return a
UUID as an option. They now transform the input to make sure the
resulting string is a valid UUID.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Also reorganized the functions and signatures to group similar
functionality together.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

It's useful to extract uuid from hashed bytes

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Compared to /dev/random uses the same CSPRNG (since Linux 4.8) and it
doesn't block

OCaml's PRNG is not a CSPRNG as it is predictable

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Now the only modules where Uuidm is used is inside Uuid, and in
varstoredguard's RPC. This is to avoid using GADTs for little benefit

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Now all alcotest binaries are executed in compact mode when running make
test. This allows the developers to run the tests with dune runtest
--profile=debug to see all the output of test if they wish to.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

This commit doesn't change any code. It only move some codes to a few
new files so that they could be used more easily later.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

…atches

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

This commit fixes the bug introduced in c081442:
prior to c081442, the bool value is for [up_to_date], while after the
commit, it is for [pkg_updates_available].
If there is(are) RPM package updates, it means a host is NOT up to date.

This commit is just for this bug.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

…32574

Support livepatch - part 1

…efi-qemu

CA-366014: avoid qemu device state issues during migration in UEFI mode

…/gzip

Don't use --force in gzip decompress

CP-34028: implement Uuid with Uuidm

In practice means single values don't block others values from being
tested. Using alcotest failures make test failures easier to read and
reason about.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

This makes it easier to locate which parts of it are tested and to what
extent.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Previously the task was reported as cancelled, which was nonsensical and
made triaging issues harder than necessary. Instead report an internal
error as this is not expected. The task is still cancelled to make it
disappear.

The internal wait method has been changed to be able to return relevant
information about the event captured, allowing to not just return a
boolean if desired. This is useful to retrieve information about tasks
just once instead of losing the information and making callers retrieve
again the information which makes the code flow difficult to navigate
and understand.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

The conversions from and to nanoseconds in floats are going to be
deprecated in mtime 1.4, avoid using them.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>

maintenance: run uuid tests as part of that package

xsh was closing the file descriptor both in the 'proxy' function and in
the stunnel wrapper code.
Push down the Unixfd.(!) dereference into the proxy code to make it
obvious that we have a Unix.close there, and replace that with
Unixfd.safe_close which avoids double close.

This was causing blob sync to fail in XAPI with an rsync protocol error
(due to the extra EBADF printed by xsh).

Fixes: c6511f0 "CA-337546: update to new Stunnel_cache API with Safe_resources support"

Signed-off-by: Edwin Török <edvin.torok@citrix.com>

xsh: fix XAPI blob sync in pools

Datamodel: replace some recent rel_next entries

The RPC overhead when destroying many messages is high. To avoid it,
introcuce a message to destroy several messages in one call. This
introduces a new API call Message.destroy_many. The corresponding CLI
command message-destroy is extended to accept a parameter "uuids" to
accept multiple UUIDs in additio to the existing "uuid" parameter.

I notice that destroyig a message by itself is not cheap because it
parses the XML content of the message being destroyed. In an unlucky
case it also involves scanning a directory on the file system to remove
an alias.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

This is needed for the case that the type of the backend's internal
state is extended in a xenopsd update, such that any running VMs have
the current runtime metadata. Case in point is the recent addition of
the platformdata key, where the default value (an empty map) is not
sufficient.

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>

Prior to this fix, "yum updateinfo list updates" was used to get the
accumulative update RPM package list. The update IDs of these packages
would be used to pick up associated update metadata from updateinfo.
And similarly, another command "repoquery -a --pkgnarrow=updates" is
used to get the RPM packages to be installed in the update.

The outputs of both command lines above, however, don't include the net
new RPM packages. This causes a bug that the metadata of updates
associated with these net new RPM packages are missed in constructing
overall update infomation, especially the update guidance.

The fix is to use "yum updateinfo list available" and
"repoquery -a --pkgnarrow=available" to include the net new RPM packages
as well.

Meanwhile, some obsoleted packages might be reported as net new RPMs.
Therefore, an extra step is added to filter out these obsoleted RPMs.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

This commit fixes a bug in following case:
One new RPM package is to be installed due to being required by an
existing package. The new package has not been installed. There are
accumulative updates for this new package. Example:
foo-1-1.rpm introduced in update-01, but not installed
foo-2-1.rpm introduced in update-02, but not installed
foo-3-1.rpm introduced in update-03, and to be installed in this update

In this case, the accumulative updates (update-01 and update-02) should
be included in consolidating update information.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

The YUM/RPM command lines may return long ouptput strings which might be
truncated by logging.

This commit adds extra debug logs for the outputs of these command lines
as they are useful in debugging.

Signed-off-by: Ming Lu <ming.lu@citrix.com>

…367120

Missing net new RPMs in picking up metadata from updateinfo

Update the definition of Message.destroy_many to use the default flags,
which inlcudes the Async flag that ensures an async version of this API
call is provided. This should further improve user experiences since we
expect this call to be slow when many messages are removed.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

A CH host is meant to re-apply it's license once a day, in order to
correctly handle expiry and to pick up any new licenses. This is some by
setting a timer in a thread in v6d, which triggers a host.apply_edition
call to xapi after 24 hours. The problem is that this call is rejected
if HA is enabled, which breaks the mechanism.

We'll fix this by allowing a host.apply_edition call if the edition is
not being changed. This is still consistent with the original intention
of the restriction (as evidenced by the code comment).

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>

Upgrade VM runtime state when xenopsd restarts

…/CP-38688

CP-38688 make Message.destroy_many() async, too

Instead of throwing an exception when a DNS doesn't have
the correct ip familly: just ignore it

It means when reconfiguring:
- an ipv4: only the ipv4 entries on the input dns will kept and
only the ipv6 dns already set to the pif will be kept
- an ipv6: only the ipv6 entries on the input dns will kept and
only the ipv4 dns already set to the pif will be kept

This is usefull in the case both family are configured on a pif

Right now the installer and xsconsole can fail when reconfiguring an
IP or an IPv6 and both family DNS are present, which can often happen
in case of dual stack

Signed-off-by: BenjiReis <benjamin.reis@vates.fr>

XSI-1246/CA-367232: Daily license re-apply fails is HA is enabled

…nfiguring-pif

Filter input dns when reconfiguring a pif IP(v6)

In particular, the evacuation plan is computed in a different way when
HA is enabled. We want the update readiness to check things as if HA is
disabled, because before updates are applied, HA will be disabled anyway.

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>

CA-366309: ignore HA when checking update readiness

This fixes xapi-project#4716

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

XenAPI.py: define how to build package in pyproject.toml

maintenance: make xapi-xenops tests more granular

Add a new field that reflects the last time Host.apply_software_updates
was run (or the time of the first installation).

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

The check against an empty string needs to be kept as some of the
variables use an empty string as an alternative to None, meaning not to
create the file at all.

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

use forkexecd instead

Signed-off-by: Pau Ruiz Safont <pau.safont@citrix.com>

…/CP-38583

CP-38583 add Host.last_software_update field with data/time

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>

Signed-off-by: Ming Lu <ming.lu@citrix.com>