Module store

Expand description

§Storing layered container images as OSTree commits

This module implements the core storage and import logic for container images in ostree. It handles fetching images from registries, caching layers as ostree commits, and creating merge commits that represent the complete image filesystem.

§Overview

The primary entry point is ImageImporter, which orchestrates the import of a container image. The import process efficiently handles incremental updates by caching each layer as a separate ostree commit, only fetching layers that aren’t already present.

§Reference Namespace Constants

Layers and images are stored using these ref prefixes (defined as constants in this module):

ostree/container/blob: Individual OCI layers stored as commits
ostree/container/image: Merge commits for complete images
BASE_IMAGE_PREFIX (ostree/container/baseimage): Protected base images (public)

Layer refs use escaped digests (e.g., sha256:abc... becomes sha256_3A_abc...) via crate::refescape to conform to ostree ref naming requirements.

§Import Process

A typical import flow:

Create importer: ImageImporter::new initializes the proxy connection to the container registry (via containers-image-proxy/skopeo).
Prepare import: ImageImporter::prepare fetches the manifest and analyzes which layers need to be downloaded:
- Returns PrepareResult::AlreadyPresent if the image is unchanged
- Returns PrepareResult::Ready with a PreparedImport containing the download plan
Execute import: ImageImporter::import downloads missing layers and creates the merge commit:
- Each layer is fetched, decompressed, and imported as an ostree commit
- The merge commit overlays all layers, processing whiteouts
- Image metadata (manifest, config) is stored in commit metadata

§Layer Types

The manifest layout is parsed to identify different layer types:

Commit layer: The base ostree commit layer (identified by ostree.final-diffid)
Component layers: Additional ostree “chunk” layers containing split objects
Derived layers: Non-ostree layers from Containerfile RUN commands

Each layer type is handled differently during import:

Ostree layers use object-set import mode for efficient reconstruction
Derived layers are imported as regular filesystem trees with SELinux labeling

§Merge Commit Metadata

The merge commit stores essential metadata for image management:

ostree.manifest-digest: The canonical manifest digest (e.g., sha256:...)
ostree.manifest: Complete OCI manifest as canonical JSON
ostree.container.image-config: OCI image configuration as canonical JSON

This metadata enables:

Detecting when updates are available
Re-exporting images with preserved structure
Querying image state via query_image and query_image_commit

§Layer Caching and Deduplication

Layers are cached by their content digest, enabling:

Incremental updates: Only changed layers are downloaded
Cross-image sharing: Images sharing layers reuse cached commits
Efficient storage: Ostree’s content-addressed storage deduplicates files

The query_layer function checks if a layer is already cached by looking up its ref. During import, cached layers are skipped entirely.

§Garbage Collection

Unreferenced layers are automatically pruned after imports via gc_image_layers:

Collect all layer digests referenced by stored images and deployments
List all layer refs under ostree/container/blob/
Remove refs for layers not in the referenced set

Note: This only removes refs; actual object pruning requires a separate call to ostree::Repo::prune.

§Key Types

ImageImporter: Main import orchestrator with progress tracking
PrepareResult: Result of preparing an import (already present vs. ready)
PreparedImport: Detailed import plan with layer analysis
ManifestLayerState: Per-layer state (descriptor, ref, cached commit)
LayeredImageState: Complete state of a pulled image
CachedImageUpdate: Cached metadata for pending updates
ImportProgress: Progress events for layer fetches
LayerProgress: Byte-level progress for a single layer

§Example Usage

use ostree_ext::container::{OstreeImageReference, store::ImageImporter};

let imgref: OstreeImageReference = "ostree-unverified-registry:quay.io/fedora/fedora-bootc:latest".parse()?;
let mut importer = ImageImporter::new(&repo, &imgref, Default::default()).await?;

match importer.prepare().await? {
    PrepareResult::AlreadyPresent(state) => {
        println!("Image already at {}", state.manifest_digest);
    }
    PrepareResult::Ready(prep) => {
        println!("Fetching {} layers", prep.layers_to_fetch().count());
        let state = importer.import(prep).await?;
        println!("Imported {}", state.merge_commit);
    }
}

§See Also

[super::encapsulate]: Export ostree commits to container images
crate::tar: Tar stream format for layer content

Structs§

CachedImageUpdate: Locally cached metadata for an update to an existing image.
CompareState 🔒
ExportToOCIOpts: Options controlling commit export into OCI
ImageImporter: Context for importing a container image.
ImageProxyConfig: Configuration for the proxy.
LayerProgress: Sent across a channel to track the byte-level progress of a layer fetch.
LayeredImageState: State of an already pulled layered image.
ManifestLayerState: A container image layer with associated downloaded-or-not state.
PreparedImport: Information about which layers need to be downloaded.

Enums§

ImportProgress: Sent across a channel to track start and end of a container fetch.
PrepareResult: Result of invoking ImageImporter::prepare.

Constants§

BASE_IMAGE_PREFIX: The ostree ref prefix for “base” image references that are used by derived images. If you maintain tooling which is locally building derived commits, write a ref with this prefix that is owned by your code. It’s a best practice to prefix the ref with the project name, so the final ref may be of the form e.g. ostree/container/baseimage/bootc/foo.
IMAGE_PREFIX 🔒: The ostree ref prefix for image references.
LAYER_PREFIX 🔒: The ostree ref prefix for blobs.
META_CONFIG 🔒: The key injected into the merge commit with the image configuration serialized as JSON.
META_MANIFEST 🔒: The key injected into the merge commit with the manifest serialized as JSON.
META_MANIFEST_DIGEST 🔒: The key injected into the merge commit for the manifest digest.
OSTREE_BASE_DEPLOYMENT_REFS 🔒: The ref prefixes which point to ostree deployments. (TODO: Add an official API for this)
RPMOSTREE_BASE_REFS 🔒: A layering violation we’ll carry for a bit to band-aid over https://github.com/coreos/rpm-ostree/issues/4185

Functions§

chunking_from_layer_committed 🔒: The way we store “chunk” layers in ostree is by writing a commit whose filenames are their own object identifier. This function parses what is written by the ImporterMode::ObjectSet logic, turning it back into a “chunked” structure that is used by the export code.
cleanup_root 🔒: Automatically clean up files that may have been injected by container builds. xref https://github.com/containers/buildah/issues/4242
clear_cached_update: Remove any cached
compare_commit_trees 🔒
compare_file_info 🔒
copy: Copy a downloaded image from one repository to another, while also optionally changing the image reference type.
export: Given a container image reference which is stored in repo, export it to the target image location.
export_to_oci 🔒: Export an imported container image to a target OCI directory.
gc_image_layers: Garbage collect unused image layer references.
gc_image_layers_impl 🔒
image_config_from_commitmeta 🔒
image_filtered_content_warning: Generate a suitable warning message from given list of filtered files, if any.
inode_of_object 🔒
layer_from_diffid 🔒: Given a target diffid, return its corresponding layer. In our current model, we require a 1-to-1 mapping between the two up until the ostree level. For a bit more information on this, see https://github.com/opencontainers/image-spec/blob/main/config.md
list_container_deployment_manifests 🔒: Iterate over deployment commits, returning the manifests from commits which point to a container image.
list_images: List all images stored
manifest_data_from_commitmeta 🔒
manifest_digest_from_commit: Return the original digest of the manifest stored in the commit metadata. This will be a string of the form e.g. sha256:<digest>.
manifest_for_image 🔒
parse_cached_update 🔒: Given detached commit metadata, parse the data that we serialized for a pending update (if any).
parse_manifest_layout 🔒
parse_ostree_manifest_layout 🔒: Like parse_manifest_layout but requires the image has an ostree base.
query_image: Query metadata for a pulled image.
query_image_commit: Query metadata for a pulled image via an OSTree commit digest. The digest must refer to a pulled container image’s merge commit.
query_layer 🔒
ref_for_blob_digest 🔒: Convert e.g. sha256:12345… into /ostree/container/blob/sha256_2B12345....
ref_for_image 🔒: Convert e.g. sha256:12345… into /ostree/container/blob/sha256_2B12345....
ref_for_layer 🔒: Convert e.g. sha256:12345… into /ostree/container/blob/sha256_2B12345....
remove_image: Remove the specified image reference. If the image is already not present, this function will successfully perform no operation.
remove_images: Remove the specified image references. If an image is not found, further images will be removed, but an error will be returned.
timestamp_of_manifest_or_config 🔒: Find the timestamp of the manifest (or config), ignoring errors.
try_query_image 🔒: Attempt to query metadata for a pulled image; if it is corrupted, the error is printed to stderr and None is returned.
verify_container_image 🔒

Type Aliases§

MetaFilteredData: The type used to store content filtering information.

Module store

Module store Copy item path

§Storing layered container images as OSTree commits

§Overview

§Reference Namespace Constants

§Import Process

§Layer Types

§Merge Commit Metadata

§Layer Caching and Deduplication

§Garbage Collection

§Key Types

§Example Usage

§See Also

Structs§

Enums§

Constants§

Functions§

Type Aliases§

Module store