Sovereign Cloud Stack

Eine Plattform — standardisiert, entwickelt und betrieben von Vielen.

SCS Security Advisory on incomplete QCOW2 and VMDK image handling protections (CVE-2024-40767)

Kurt Garloff, Christian Berendt 23. Juli 2024

The vulnerability

OpenStack supports users to upload custom VM images in a number of formats to be used for creating Virtual Machines. The simplest is a raw copy of a harddisk, while other formats such as QCOW2 and VMDK are more advanced in being compressed and allowing for embedding additional information. They also support referencing other files. The latter feature needs to be controlled by OpenStack to avoid exposing sensitive files. OpenStack thus disallows such external references for customer images. The protections were found to be incomplete before which lead to CVE-2022-47951 and CVE-2024-32498.

Unfortunately, the last fixes with CVE-2024-32498 still were incomplete: Arnaud Morin (OVH) reported a vulnerability in Nova. By supplying a raw format image which is actually a specially crafted QCOW2 image with a backing file path or VMDK flat image with a descriptor file path, an authenticated user may convince systems to return a copy of the referenced file’s contents from the server resulting in unauthorized access to potentially sensitive data. All Nova deployments are affected.

The vulnerability has been assigned CVE-2024-40767.

Impact on the SCS reference implementation

The Sovereign Cloud Stack reference implementation (by OSISM) runs these services in containers and thus most files from the hypervisor’s file system are not directly accessible from inside the container via this vulnerability. However, this depends on the service inside the container. Some services have access to data from the hypervisor or they contain sensitive data themselves (e.g. credentials to connect to other services), and thus they are accessible by a potential attacker. They have also access to log files from the services which may expose sensitive data.

To abuse this vulnerability, the attacker must be an authenticated user of the platform.

We were not able to exploit this vulnerability in systems that included the hotfix patches for CVE-2024-32498 in our testing. Nova did treat the VMDK and QCOW images declared as raw as raw (as did Cinder, but Cinder is not suspected of mistreating them). Boot disks created by Nova from the crafted VMDK and QCOW2 images contain the raw contents of the images, without their external references being resolved and thus without exposing any external content. This is probably caused by SCS not changing the default to use_cow_images=False. We currently are not aware of working exploits without that (non-default) setting.

We would argue that mislabeled images should better be rejected early, as the investigation chain may otherwise be confused which has the potential to cause exploitable vulnerabilities. So we still highly recommend to patch the weakness soon.

Operators that are using use_cow_images=False in their nova config are advised to apply the fixes urgently.

Embargo

The issue has been reported to the OpenStack Vulnerability Management Team in private. The reporters and upstream developers have worked together to address the issue with fixes and an embargo date has been set to Tuesday, 2024-07-23, 15:00 UTC (17:00 CEST). At this point in time, the patches will get merged and the OpenStack Security Advisory (OSSA-2024-002) will be published. The issue is tracked in OpenStack issue #2071734, which should become publically accessible after the lift of the embargo and the publication of this advisory.

Under the used responsible disclosure approach, the information was shared with a select group of trustable users of OpenStack, so they can prepare updates and protect their user data in time for the publication.

Mitigation and Fixes

Temporary workarounds that would disable image uploads for QCOW2 and VMDK formats are not desirable and incomplete, as images can also be created from volumes, which would also have to be blocked. The complexity of doing all the changes exceeds the complexity of deploying the fixes. A better workaround would be to undo the setting of use_cow_images=False in the nova configuration, as this stops all known exploitable code paths from being used.

There are patches from the upstream OpenStack developers available. They work by rejecting images in Nova where contents and the claimed format do not match and require that a format inspector is available prior to further processing them. This provides effective protection against confusing Nova’s image handling.

Note that the upstream changes also include improvements to the handling of ISO images from Jens Harbott (OSISM), which have further tighten the inspection and which have been merged and backported upstream already without having received a separate advisory (due to not having fixed known exploitable situations).

The reference implementation for the IaaS layer of Sovereign Cloud Stack is provided by the OSISM project – the OSISM team normally builds the artifacts in a completely public process. To avoid waiting for a few hours to have all image builds (nova-compute containers) completed and tested after the disclosure date, the OSISM project has built private images beforehand which we have tested to protect against all known scenarios in which these image files are used. These images were handed to our SCS operators a few hours ahead of time (without a detailed description) to allow them to protect their infrastructure.

Instructions to install the fixed containers (hotfix on top of OSISM 7.0.x/OSISM 7.1.0):


# Override nova-compute image in your configuration repository:
cat >environments/kolla/images.yml <<EOT
---
my_registry_kolla: osism.harbor.regio.digital
my_namespace2: osism-20240723
nova_compute_tag: build-20240717-stable-2023.2
nova_compute_image: "{{ my_registry_kolla }}/{{ my_namespace2 }}/nova-compute"
EOT

Now, the nova compute image will be downloaded and rolled out on all nova compute nodes:

osism apply -a pull nova -l compute     # optional
osism apply -a refresh-containers nova -l compute

If you are not yet on R6 (OSISM 7.0.x), replace stable-2023.2 with the version you are on (e.g. stable-2023.1 if you are on R5 (OSISM 6.x).

Note that the above instructions do NOT contain cinder and glance images; they should have been fixed already with the installation of fixes for CVE-2024-32498 or with the rollout of OSISM 7.1.0.

The deployment will have the following user-visible effects:

Most users won’t notice this. Please note that running workloads will not be affected at all.

After the upstream patches have been merged in the hours after the disclosure date, OSISM will build, test and publish an official 7.1.1 release containing the updated containers (nova-compute, glance-api, cinder-volume) as well as a fix for octavia-api. Please refer to the OSISM release notes. You can upgrade from this hotfix toiOSISM 7.1.1 using the regular upgrade mechanism.

If your exposure is low (e.g. private cloud or not using use_cow_images=False) you may decide to wait for the regular 7.1.1 and go there directly from 7.x.

Warning: If you are still on OSISM 7.0.x (not 7.1.x yet) and create the images.yml as described here, osism apply -a pull and osism apply -a refresh-containers will use the base 7.0.x containers with just the single nova_compute override which would make you vulnerable to the glance and cinder based attacks from CVE-2024-32498 again. We recommend to avoid this by using the latest 7.1.0 release. If you can’t, you can either ensure you avoid pulling and refreshing containers other than the nova_compute one or you construct a combined images.yml from the description in CVE-2024-32498 and the above section. The latter would ensure you have all needed image overrides for both CVEs.

Have my systems been compromised?

Providers that are unable to patch this issue on the day of the release and use use_cow_images=False in their nova configuration need to consider the possibility of having been attacked. In this case, passwords/keys for internal communication should be considered to be leaked and will need to be rotated to reestablish the defense-in-depth lines. We recommend providers check for suspicious image operations (QCOW2 and VMDK image registrations, snapshots with QCOW2 signature) in their logs for the time after the embargo lift. In particular, every QCOW2 image that has been uploaded should be investigated for the datafile property; it should be assumed that images that have been registered and removed again contain the exploit. VMDK images with external Flatfile references (createType="monolithicFlat") are suspicious as well. Evil images should also be pruned from the image cache.

Patch status of SCS clouds

Our publically known providers of SCS IaaS platforms have all been notified and have started to ensure that their use_cow_images setting does not expose them and have started the rollout of updated images which we were able to provide them privately a few hours ahead of time (without sharing all the details on the issue). We expect that none of them has any exploitable vulnerability left open by the time this gets public. Please refer to their communication for precise information on the patch status.

Thanks

The authors would like to thank Arnaud Morin, Jens Harbott, the OpenStack developers and reviewers, testers and the OpenStack Vulnerability Management Team for the responsible reporting, careful analysis, fixing, testing and professional handling of the issue and the OSISM team for providing the fixed containers.

Sovereign Cloud Stack Security Contact

SCS security contact is security@scs.community, as published on https://scs.community/.well-known/security.txt.

Version history

Über den Autor

Kurt Garloff
CTO Sovereign Cloud Stack @ Open Source Business Alliance
While working on Physics as student and researcher in Dortmund, Wuppertal and Eindhoven, Kurt started to work with and on Linux, with first patches to the SCSI layer in the mid 90s. He has spent his post-university life in Open Source, as kernel engineer, leader of SUSE Labs (kernel, compiler, X11, security), and engineering and business leadership at SUSE. Since 2011 he has been working on Open Source cloud software, at Deutsche Telekom, as Freelancer, at T-Systems (as chief architect for the OTC) and also has been serving on the Open Infra Foundation's board. Since 2019 he has been pushing the Sovereign Cloud Stack idea which resulted in a publically funded project that he now technically leads. He still loves to occasionally write code (mostly python these days) or at least test out code from the colleagues and project. He spends his free time with his family or with running and playing table tennis.