https://video.osgeo.org/videos/watch/597e9776-c3bc-4319-99f5-cc09d7b0952c The Open Container Initiative (OCI) 1.1 specification has expanded container registries beyond traditional software images, enabling them to store and distribute a wide variety of digital artifacts, from software build artifacts to machine learning (ML) models and arbitrarily large data blobs. As the volume of Earth Observation (EO) data generated by satellites and remote sensing applications continues to increase, scalable and efficient distribution methods are becoming essential. OCI registries are well-suited for end-to-end supply chains due to their built-in capabilities for integrity verification and attestations, such as quality assurance, allowing for the application of common tooling and best practices across various steps in the supply chain. Their layered design allows for selective retrieval of specific parts, and optimizations like compression and deduplication can be applied individually, making them ideal for managing EO data of arbitrary size. Challenges in Optimizing OCI Registries for EO DataHowever, despite these advantages, significant challenges remain in optimizing both client-side parallelization and addressing server-side limitations of existing OCI registries. A critical research question arises: How should EO data be structured within OCI registries to maximize performance? While OCI registries support multiple storage layers and optimizations, the practical implications of storing EO data in this format have not been thoroughly explored. Key concerns include whether OCI registries can effectively support arbitrarily sized EO data and how different storage layouts affect retrieval speed and storage efficiency. Investigating Best Practices for EO Data Storage in OCI RegistriesThis research paper investigates how to structure EO data within OCI registries to optimize performance. By examining various physical data layouts—such as chunking data into blocks or organizing data into multiple layers—the goal is to identify best practices for storing and accessing large EO datasets. Benchmarking common OCI client tooling against a variety of OCI-compliant registries, including public offerings like DockerHub and Quay.io, managed services like AWS ECR, and bespoke cloud-based implementations, will help evaluate retrieval latency, throughput, and parallelization techniques to enhance the efficiency of EO data distribution at scale. The research paper will also examine the impact of different compression, deduplication, and data layout strategies on storage efficiency and retrieval performance. Advantages of OCI Registries for EO Data Storage and DistributionAn OCI image, as specified through the Linux Foundation's Open Container Interface, is actually a collection of multiple components. At the top level is an index of all the other included components. It references, in a JSON format, all the other layers with their digest or the cryptographic hash of the content itself. The OCI distribution spec describes how clients pull images from a registry, which is done layer by layer. OCI registries offer several inherent benefits that make them attractive for EO data storage and distribution. These include: Layered Storage Model: OCI artifacts utilize a layered approach, allowing incremental and block-wise storage and retrieval, enabling efficient updates and minimizing redundant data transfers., Efficient Distribution: Content-addressable storage allows fetching only changed layers, which minimizes bandwidth and storage costs and supports incremental updates., Versioning and Tagging: Version control is inherent in OCI registries, enabling precise tracking of updates. This is crucial as data moves through various stages of processing, validation, and final distribution., Attestation and Integrity: Data integrity is ensured using cryptographic hashes, verifying the authenticity and trustworthiness of the supply chain, from raw input to final products., Addressing Practical Limitations in OCI RegistriesDespite these benefits, the practical limitations of OCI registries for handling large-scale EO data are not fully understood. Specifically, the impact of physical data layout on retrieval speed and storage efficiency requires further investigation. This research paper will explore several strategies for structuring EO data within OCI registries: Chunking and Layering Strategies: Investigating whether data should be stored in large monolithic layers or smaller, granular chunks, and evaluating the effects of compression and deduplication on retrieval performance., Client-Side Parallelization: Analyzing the impact of parallelized downloads on pull speeds and comparing performance improvements with different concurrent retrieval configurations., Server-Side Constraints: Assessing registry performance ..., Stefan Achtsnit https://talks.osgeo.org/foss4g-europe-2025/talk/HNZK37/ Room: PA01 (Quarticle) @ 17.07.2025 12:00:00 #foss4ge2025 #AcademicTrack Thu, 30 Jul 2026 02:18:10 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://video.osgeo.org https://video.osgeo.org/lazy-static/avatars/de841c88-3a57-45ac-9eff-a00237356bfc.png https://video.osgeo.org/videos/watch/597e9776-c3bc-4319-99f5-cc09d7b0952c All rights reserved, unless otherwise specified in the terms specified at https://video.osgeo.org/about and potential licenses granted by each content's rightholder.