Container Plumbing Days 2025

Filesystem monitoring was added to fanotify in kernel 5.1
and the first HSM feature was added to fanotify in kernel 6.12.
See this Linux Plumbers talk by fanotify maintainer Jan Kara for a
good overview:
https://lpc.events/event/18/contributions/1717/attachments/1648/3404/fanotify.pdf

This talk will present our work towards making those features available inside containers using two different strategies:

1. For filesystems that were mounted inside userns or idmapped into userns, userns admin would be able to use fanotify to monitor those filesystems.

2. For filesystems that were mounted by the host, container users would be able to subscribe to a service on the host to receive filesystem monitoring events contained to the scope of the container.

Traditional container runtimes rely on OS-level isolation using namespaces and cgroups. While efficient, this approach can fall short in multi-tenant environments where stronger workload isolation is essential. To mitigate these risks, containers execute within sandboxes, often in the form of VMs. However, this approach typically involves extra components to manage the container lifecycle within the VM, adding complexity and increasing resource usage.

What if we could have the best of both worlds, strong isolation and low overhead? The key lies in specialization! Unikernels and
stripped-down Linux VMs, tailored for a single application, offer VM-grade isolation with small resource usage and fast boot times.

This talk introduces urunc, a novel container runtime that makes this approach practical. Urunc reverses the traditional model: instead of running containers inside VMs, it runs lightweight VMs, as containers. CRI-compatible, urunc integrates seamlessly with Kubernetes, enabling the orchestration of such VMs just like regular containers. The talk covers the design and architecture of urunc, its key differences with other sandboxing technologies and includes a live demo.

Container Images are foundational to modern infrastructure, yet their internal structure can often be opaque and difficult to debug. Large image sizes, hidden redundant files, and inefficient layering can lead to slower deployments and wasted resources. Manually dissecting layers or understanding disk usage across an image stack is often cumbersome and time-consuming.

This lightning talk introduces skiff, a powerful command-line utility designed to simplify OCI image introspection. Built using podman's container libraries, skiff provides developers and operators with tooling to introspect image layers. We will demonstrate how skiff can be used for the following tasks:
- Identify large layers and files
- Explore layer contents directly
- Visualize disk usage
- Locate whiteout files

Join this session to learn how skiff can help you analyze, debug, and optimize container images for better performance and resource utilization.

When using tools like RPM or Zypper for updating packages, there is a risk of incomplete updates or breaking the running system. To overcome these challenges, we have developed container-snap, a prototype plugin designed to deliver atomic OS updates that are fully applied or rolled back without compromising the system's state.

container-snap leverages OCI images as the source for updates and integrates seamlessly with openSUSE’s tukit for transactional OS updates. By utilizing Podman’s btrfs storage driver, it creates bootable btrfs subvolumes directly from OCI images, effectively turning them into atomic OS snapshots. This allows you to build OS images using familiar tools like Docker or Buildah and deploy the container image on your host.

This lightning talk covers the following topics:
- The container-snap architecture and implementation details
- Main development challenges and solutions
- Lessons learned in bridging container tech and OS updates
- A live demo showcasing atomic updates in action

Join this session to learn more about how to boot from an OCI image without bricking your system!