One platform — standardized, built and operated by many.
One platform — standardized, built and operated by many.
OpenStack supports users to upload custom VM images in a number of formats to be used for creating Virtual Machines. The simplest is a raw copy of a harddisk, while other formats such as QCOW2 and VMDK are more advanced in being compressed and allowing for embedding additional information. They also support referencing other files. The latter feature needs to be controlled by OpenStack to avoid exposing sensitive files. OpenStack thus disallows such external references for customer images. The protections were found to be incomplete before which lead to CVE-2022-47951 and CVE-2024-32498.
Unfortunately, the last fixes with CVE-2024-32498 still were incomplete: Arnaud Morin (OVH) reported a vulnerability in Nova. By supplying a raw format image which is actually a specially crafted QCOW2 image with a backing file path or VMDK flat image with a descriptor file path, an authenticated user may convince systems to return a copy of the referenced file’s contents from the server resulting in unauthorized access to potentially sensitive data. All Nova deployments are affected.
The vulnerability has been assigned CVE-2024-40767.
The Sovereign Cloud Stack reference implementation (by OSISM) runs these services in containers and thus most files from the hypervisor’s file system are not directly accessible from inside the container via this vulnerability. However, this depends on the service inside the container. Some services have access to data from the hypervisor or they contain sensitive data themselves (e.g. credentials to connect to other services), and thus they are accessible by a potential attacker. They have also access to log files from the services which may expose sensitive data.
To abuse this vulnerability, the attacker must be an authenticated user of the platform.
We were not able to exploit this vulnerability in systems that included
the hotfix patches for
CVE-2024-32498
in our testing.
Nova did treat the VMDK and QCOW images declared as raw as raw (as did
Cinder, but Cinder is not suspected of mistreating them).
Boot disks created by Nova from the crafted VMDK and QCOW2 images contain
the raw contents of the images, without their external references being
resolved and thus without exposing any external content.
This is probably caused by SCS not changing the default to
use_cow_images=False. We currently are not aware of working exploits
without that (non-default) setting.
We would argue that mislabeled images should better be rejected early, as the investigation chain may otherwise be confused which has the potential to cause exploitable vulnerabilities. So we still highly recommend to patch the weakness soon.
Operators that are using use_cow_images=False in their nova config are
advised to apply the fixes urgently.
The issue has been reported to the OpenStack Vulnerability Management Team in private. The reporters and upstream developers have worked together to address the issue with fixes and an embargo date has been set to Tuesday, 2024-07-23, 15:00 UTC (17:00 CEST). At this point in time, the patches will get merged and the OpenStack Security Advisory (OSSA-2024-002) will be published. The issue is tracked in OpenStack issue #2071734, which should become publically accessible after the lift of the embargo and the publication of this advisory.
Under the used responsible disclosure approach, the information was shared with a select group of trustable users of OpenStack, so they can prepare updates and protect their user data in time for the publication.
Temporary workarounds that would disable image uploads for QCOW2
and VMDK formats are not desirable and incomplete, as images can also
be created from volumes, which would also have to be blocked.
The complexity of doing all the changes exceeds the complexity of
deploying the fixes.
A better workaround would be to undo the setting of use_cow_images=False
in the nova configuration, as this stops all known exploitable code paths
from being used.
There are patches from the upstream OpenStack developers available. They work by rejecting images in Nova where contents and the claimed format do not match and require that a format inspector is available prior to further processing them. This provides effective protection against confusing Nova’s image handling.
Note that the upstream changes also include improvements to the handling of ISO images from Jens Harbott (OSISM), which have further tighten the inspection and which have been merged and backported upstream already without having received a separate advisory (due to not having fixed known exploitable situations).
The reference implementation for the IaaS layer of Sovereign Cloud Stack is provided by the OSISM project – the OSISM team normally builds the artifacts in a completely public process. To avoid waiting for a few hours to have all image builds (nova-compute containers) completed and tested after the disclosure date, the OSISM project has built private images beforehand which we have tested to protect against all known scenarios in which these image files are used. These images were handed to our SCS operators a few hours ahead of time (without a detailed description) to allow them to protect their infrastructure.
Instructions to install the fixed containers (hotfix on top of OSISM 7.0.x/OSISM 7.1.0):
# Override nova-compute image in your configuration repository:
cat >environments/kolla/images.yml <<EOT
---
my_registry_kolla: osism.harbor.regio.digital
my_namespace2: osism-20240723
nova_compute_tag: build-20240717-stable-2023.2
nova_compute_image: "{{ my_registry_kolla }}/{{ my_namespace2 }}/nova-compute"
EOT
Now, the nova compute image will be downloaded and rolled out on all nova compute nodes:
osism apply -a pull nova -l compute # optional
osism apply -a refresh-containers nova -l compute
If you are not yet on R6 (OSISM 7.0.x), replace stable-2023.2 with the version you are on
(e.g. stable-2023.1 if you are on R5 (OSISM 6.x).
Note that the above instructions do NOT contain cinder and glance images; they should have been fixed already with the installation of fixes for CVE-2024-32498 or with the rollout of OSISM 7.1.0.
The deployment will have the following user-visible effects:
Most users won’t notice this. Please note that running workloads will not be affected at all.
After the upstream patches have been merged in the hours after the disclosure date, OSISM will build, test and publish an official 7.1.1 release containing the updated containers (nova-compute, glance-api, cinder-volume) as well as a fix for octavia-api. Please refer to the OSISM release notes. You can upgrade from this hotfix toiOSISM 7.1.1 using the regular upgrade mechanism.
If your exposure is low (e.g. private cloud or not using use_cow_images=False)
you may decide to wait for the regular 7.1.1 and go there directly from 7.x.
Warning: If you are still on OSISM 7.0.x (not 7.1.x yet) and create the images.yml as
described here, osism apply -a pull and osism apply -a refresh-containers
will use the base 7.0.x containers with just the single nova_compute
override which would make you vulnerable to the glance and cinder based
attacks from CVE-2024-32498 again.
We recommend to avoid this by using the latest 7.1.0 release.
If you can’t, you can either ensure you avoid pulling and refreshing containers
other than the nova_compute one or you construct a combined images.yml from the
description in CVE-2024-32498
and the above section. The latter would ensure you have all needed image overrides for
both CVEs.
Providers that are unable to patch this issue on the day of the release
and use use_cow_images=False in their nova configuration
need to consider the possibility of having been attacked. In this case,
passwords/keys for internal communication should be considered to be
leaked and will need to be rotated to reestablish the defense-in-depth
lines. We recommend providers check for suspicious image operations
(QCOW2 and VMDK image registrations, snapshots with QCOW2 signature) in
their logs for the time after the embargo lift.
In particular, every QCOW2 image that has been uploaded should be investigated
for the datafile property; it should be assumed that images that have been
registered and removed again contain the exploit.
VMDK images with external Flatfile references (createType="monolithicFlat")
are suspicious as well. Evil images should also be pruned from the image cache.
Our publically known providers of SCS IaaS platforms have all been
notified and have started to ensure that their use_cow_images setting
does not expose them and have started the rollout of updated images which
we were able to provide them privately a few hours ahead of time (without
sharing all the details on the issue). We expect that none of them has
any exploitable vulnerability left open by the time this gets public.
Please refer to their communication for precise information on the
patch status.
The authors would like to thank Arnaud Morin, Jens Harbott, the OpenStack developers and reviewers, testers and the OpenStack Vulnerability Management Team for the responsible reporting, careful analysis, fixing, testing and professional handling of the issue and the OSISM team for providing the fixed containers.
SCS security contact is security@scs.community, as published on https://scs.community/.well-known/security.txt.
use_cow_images=False, v0.2, 2024-07-20, 11:30 CEST.