Container Plumbing Days 2025

Filesystem monitoring was added to fanotify in kernel 5.1
and the first HSM feature was added to fanotify in kernel 6.12.
See this Linux Plumbers talk by fanotify maintainer Jan Kara for a
good overview:
https://lpc.events/event/18/contributions/1717/attachments/1648/3404/fanotify.pdf

This talk will present our work towards making those features available inside containers using two different strategies:

1. For filesystems that were mounted inside userns or idmapped into userns, userns admin would be able to use fanotify to monitor those filesystems.

2. For filesystems that were mounted by the host, container users would be able to subscribe to a service on the host to receive filesystem monitoring events contained to the scope of the container.

Traditional container runtimes rely on OS-level isolation using namespaces and cgroups. While efficient, this approach can fall short in multi-tenant environments where stronger workload isolation is essential. To mitigate these risks, containers execute within sandboxes, often in the form of VMs. However, this approach typically involves extra components to manage the container lifecycle within the VM, adding complexity and increasing resource usage.

What if we could have the best of both worlds, strong isolation and low overhead? The key lies in specialization! Unikernels and
stripped-down Linux VMs, tailored for a single application, offer VM-grade isolation with small resource usage and fast boot times.

This talk introduces urunc, a novel container runtime that makes this approach practical. Urunc reverses the traditional model: instead of running containers inside VMs, it runs lightweight VMs, as containers. CRI-compatible, urunc integrates seamlessly with Kubernetes, enabling the orchestration of such VMs just like regular containers. The talk covers the design and architecture of urunc, its key differences with other sandboxing technologies and includes a live demo.

Container Images are foundational to modern infrastructure, yet their internal structure can often be opaque and difficult to debug. Large image sizes, hidden redundant files, and inefficient layering can lead to slower deployments and wasted resources. Manually dissecting layers or understanding disk usage across an image stack is often cumbersome and time-consuming.

This lightning talk introduces skiff, a powerful command-line utility designed to simplify OCI image introspection. Built using podman's container libraries, skiff provides developers and operators with tooling to introspect image layers. We will demonstrate how skiff can be used for the following tasks:
- Identify large layers and files
- Explore layer contents directly
- Visualize disk usage
- Locate whiteout files

Join this session to learn how skiff can help you analyze, debug, and optimize container images for better performance and resource utilization.

When using tools like RPM or Zypper for updating packages, there is a risk of incomplete updates or breaking the running system. To overcome these challenges, we have developed container-snap, a prototype plugin designed to deliver atomic OS updates that are fully applied or rolled back without compromising the system's state.

container-snap leverages OCI images as the source for updates and integrates seamlessly with openSUSE’s tukit for transactional OS updates. By utilizing Podman’s btrfs storage driver, it creates bootable btrfs subvolumes directly from OCI images, effectively turning them into atomic OS snapshots. This allows you to build OS images using familiar tools like Docker or Buildah and deploy the container image on your host.

This lightning talk covers the following topics:
- The container-snap architecture and implementation details
- Main development challenges and solutions
- Lessons learned in bridging container tech and OS updates
- A live demo showcasing atomic updates in action

Join this session to learn more about how to boot from an OCI image without bricking your system!

In the field of AI and machine learning, model training has become an increasingly complex and resource-intensive task. Training jobs often run for days or weeks, distributed across multiple nodes with expensive GPU accelerators. Container checkpointing is a crucial technique for implementing fault tolerance, mitigating the impact of hardware and software failures by periodically saving the state of computations and resuming from the last checkpoint in the event of failures. While support for checkpointing has been recently integrated into Kubernetes, enabling checkpoint/restore coordination across multiple containers and nodes remains a challenge. In this talk, we are going to discuss how we have extended container runtimes and CRIU to synchronize checkpointing operations among multiple container instances in Kubernetes clusters. The talk will cover how we enable efficient end-to-end encryption for sensitive data in checkpoints and the integration with existing container platforms.

This session is about freeing your workloads from the shackles of traditional maintenance windows. By dynamically relocating running Virtual Machines (VMs) between Kubernetes nodes, KubeVirt keeps apps online during upgrades, scaling, or node failures.
Beneath the surface, KubeVirt orchestrates memory transfers and status checks, making VMs appear gravity-defying. “Pre-copy” sends most pages while VMs stay active, followed by a quick “stop-and-copy.” A “domain notify pipe” coordinates source and destination. For high dirty rates, “auto-converge” throttles CPU or “post-copy” starts the VM on the target, fetching pages on demand. Dedicated “migration0” and configurable parameters (e.g. completionTimeoutPerGiB) prevent stalls and manage bandwidth.
Learn best practices for migration timeouts, TLS toggling, and traffic isolation. We’ll also explore real-world scenarios like draining nodes for maintenance or using auto-converge on heavy VMs.

As ARM adoption grows for cloud-native workloads, thanks to its efficiency and cost advantages, multi-architecture container support is becoming a must-have, not a nice-to-have. But cross-arch builds come with their quirks: emulation headaches, manifest missteps, and pipeline slowdowns.

In this talk, Aditya and Anshika walk you through a CNCF-aligned, open source workflow to automate multi-arch builds using BuildKit, Kaniko, and Skopeo, producing reproducible, OCI-compliant images for both ARM and x86_64. They'll cover everything from base image strategy to manifest creation, registry publishing, and even image signing with cosign.

What you’ll learn:

1. How to configure and run multi-arch builds using open source tools like Kaniko and BuildKit.
2. Use Skopeo to inspect, sync, and validate multi-arch manifests.
3. Sign and verify images with cosign for trusted distribution.

By the end, you’ll have a solid, cloud-native blueprint to build once, run anywhere—from cloud to edge, x86 to ARM.