Iceberg Architecture Examples: How Iceberg powers data and ML applications

In recent years, Apache Iceberg has seen considerable advancements that highlights its growing importance. Major tech companies like Google, Snowflake, and Databricks have increasingly embraced this table format.

• Google integrated Iceberg into BigLake and BigQuery, thus making it a leading choice for open-format lakehouses.
• Snowflake has significantly boosted the performance of Iceberg tables and is set to introduce REST catalog support, further enhancing its compatibility across various platforms.
• Databricks has also announced its plans to support Iceberg, reflecting its escalating acceptance within the industry.

This trend, driven by major tech companies, highlights a transformative shift in the data warehousing landscape as Iceberg gains traction.

In this blog, we will explore how data management and machine learning applications are powered by the Iceberg architecture. We will delve into its core features and demonstrate how Iceberg enhances data and ML workflows through practical examples and industry use cases.

Exploring the Iceberg architecture layer-by-layer

Apache Iceberg is a revolutionary table format designed to bring high performance and reliability to big data workloads. Its architecture is divided into several layers, each playing a crucial role in managing and processing large datasets efficiently. Let us explore the Iceberg architecture layers in depth as shown in the figure below:

1. Data layer: The core of storage

The core of Iceberg’s architecture is its data layer, the fundamental component responsible for storage. This layer holds the actual data files, which can be formatted as Parquet, ORC, or Avro, and are generally stored in distributed storage systems such as HDFS, S3, or ADLS. The data layer which is positioned at the base of the architecture, provides the crucial foundation for the subsequent layers. Its key functions are efficient storage management, which ensures that the organization is handling large datasets with high efficiency, and support for diverse file formats, guaranteeing compatibility and performance across various data formats. It also helps reduce storage needs and enhances read/write operations.

2. Manifest files

Manifest files function as the metadata catalog for the data layer, detailing all data files and their associated metadata. They monitor the manifest lists, which contain vital information about the table’s schema, partitions, snapshots, and current state. These files store the metadata in JSON file format.

3. Manifest lists

Manifest lists are nothing but the metadata aggregator. This means that they keep track of the manifest files, and record their locations, partition details, and the range of partition columns they cover. Stored in Avro file format, each manifest list file acts as a snapshot of the Iceberg table, including snapshot details identified by a unique ‘snapshot_id.’

4. Metadata layer: The central coordinator

The metadata layer, which is the central coordinator, acts as the central repository for all the table-related information, managing a snapshot of the table’s schema, partitioning, and version history. It is at the top of the architecture and acts as the primary coordinator for all metadata activities. It helps in schema management, which tracks schema evolution to ensure both backward and forward compatibility; snapshot management, which oversees table snapshots to facilitate time travel and consistent query results; and partition specification, which maintains details about table partitioning to optimize data access.

5. Catalog

The catalog layer is the entry point for users and applications to interact with Iceberg tables. It is basically the user interface. It provides an interface for creating, deleting, and managing tables. As shown in the figure, it is at the topmost level of the architecture, hence, signifying its role as the interface between users/applications and the Iceberg architecture.

This layer helps in creating, deleting, and managing the Iceberg tables. It ensures that the tasks are performed efficiently. Additionally, it provides APIs that enable seamless interaction between users and applications with the tables, facilitating smooth and effective data operations.

Integrating all the layers

• Apache Iceberg’s layered structure and comprehensive architecture promote efficient data management and superior performance.
• Each layer serves a distinct purpose, from raw data storage to user interaction, forming a cohesive system.
• The architecture provides a robust framework that manages complex data workflows effectively.
• Understanding this architecture reveals Iceberg’s design principles and its approach to modern data lake challenges.
•  By leveraging its layered approach, Iceberg offers reliable, scalable, and high-performance data management solutions.

Enhancing Data Lakes with Iceberg by bridging the gap between flexibility and reliability

The integration of Apache Iceberg with traditional data warehouses introduces sophisticated features that enhance the capabilities of traditional data warehouses. This significantly improved the reliability and performance of contemporary data analytics. Traditional data warehouses are known for their robust infrastructure and optimized performance capabilities. They excel at SQL query optimization and stringent query controls, which ensures high levels of data security and consistency across various operations.

Traditional data warehouses offer several advantages, including well-established table structures that provide a reliable base for efficient data storage and querying. They are known for optimized performance, enabling rapid data retrieval for time-sensitive analytical tasks. Strong access controls protect data integrity by restricting access to authorized personnel. Additionally, query isolation prevents conflicts and maintains data accuracy.

The Advantages of Integrating Iceberg with Data Warehouses

Combining Iceberg with traditional data warehouses gives the user numerous benefits that cater to modern data management needs. Iceberg helps enhance the warehouses by supporting a range of contemporary data workflows, such as parallel data loading and real-time processing through streaming frameworks. This aligns with the evolving demands of data processing influenced by various tools and user preferences.

Iceberg meets the growing need for adaptable data processing solutions by integrating different data environments. Additionally, it boosts user productivity by providing tools that streamline data management, thereby addressing a wide array of operational requirements. The integration also helps increase the flexibility of data warehouses, enabling them to handle a broader spectrum of data processing tasks effectively.

Moreover, Iceberg’s robust data management features improve data integrity and reliability within the traditional warehouse framework. It also allows for consistent querying across multiple tools, from Business Intelligence platforms for reporting to Spark for intensive data processing, thus offering greater flexibility in data analysis. Finally, by merging the capabilities of data lakes and warehouses, Iceberg delivers a unified solution for managing and processing data across various operational domains.

Building a Data and ML Application with Iceberg 

In this section, we will take an example table ‘customer_data’, to build a data and ML application using Iceberg to perform ‘Customer Segmentation’ at Amazon, aimed at targeted marketing and personalized recommendations.

Environment Setup 

Set up a Spark session with Iceberg configurations.

Ingesting the Data into Iceberg

For ingesting data into Iceberg, we will create a table by partitioning based on ‘transaction_date’ for better query performance.

spark.sql("""
CREATE TABLE spark_catalog.defaul.customer_data (
customer_id BIGINT,
transaction_data STRING,
transaction_amount DOUBLE,
category STRING
)
USING iceberg
PARTITIONED BY (transaction_date)
""")

Querying the Data

Aggregate the customer data by counting transactions for each customer:

customer_df = spark.sql("""
SELECT customer_id, COUNT(*) as transaction_count
FROM spark_catalog.default.customer_data
GROUP BY customer_id
""")
customer_df.show()

The output will be something like this:

+------------------+------------------+
| customer_id      | transaction_count|
+------------------+------------------+
|          12345678|                10|
|          87654321|                15|
|          11223344|                22|
+------------------+------------------+

Optimize the query performance by filtering the data by date:

customer_df = spark.sql("""
SELECT * FROM spark_catalog.default.customer_data
WHERE tranasction_date >= '2023-01-01'
""")
customer_df.show()

The output will be:

+------------------+------------------+------------------+--------+
| customer_id      | transaction_date |transaction_amount|category|
+------------------+------------------+------------------+--------+
|          12345678|        2023-01-05|              50.0|   Books|
|          87654321|        2023-02-10|             100.0|    Toys|
|          11223344|        2023-03-15|              75.0|   Games|
+------------------+------------------+------------------+--------+

Integrating Iceberg with ML Pipelines

Feature Engineering: Prepare data by combining relevant columns into feature vectors.

from pyapark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=["transaction_count", "transaction_amount"],
                            outputCol="features")
customer_features = assembler.transform(customer_df)

Model training using K-Means clustering will help identify distinct customer groups based on the transaction count and the transaction amount. We are segmenting customers into 5 groups:

from pysprak.ml.clustering import KMeans

kmeans = Kmeans().setK(5).setSeed(1)
model = kmeans.fit(customer_features)
predictions = model.transform(customer_features)
predictions.show()

Here we are segmenting customers into 5 groups and each customer will be assigned a label corresponding to one of these 5 clusters (0, 1, 2, 3, or 4).

The output:

+------------+------------------+------------------+--------+------------+---------+
|customer_id| transaction_count|transaction_amount|category |   features|prediction|
+------------+------------------+------------------+--------+------------+---------+
|    12345678|                10|              50.0|   Books| [10.0,50.0]|        0|
|    87654321|                15|             100.0|    Toys|[15.0,100.0]|        1|
|    11223344|                22|              75.0|   Games| [22.0,75.0]|        2|
+------------+------------------+------------------+--------+------------+---------+

For the above output: 

• Cluster 0 might represent customers with moderate transaction counts (10 transactions) and moderate transaction amounts ($50.0), possibly interested in books. 
• Cluster 1 might represent customers with higher transaction counts (15 transactions) and higher transaction amounts ($100.0), likely interested in toys. 
• Cluster 2 might represent customers with high transaction counts (22 transactions) and moderate transaction amounts ($75.0), possibly interested in games.

For feature store management, save and load feature data for consistent inputs in ML workflows.

Case Studies of Companies using Iceberg to Power Data and ML Applications

1. Apache Iceberg-powered data lake architecture at Amazon on Amazon Web Services (AWS)

Amazon has embraced Apache Iceberg as a pivotal component within its AWS ecosystem, leveraging its advanced capabilities to manage expansive analytic datasets stored on Amazon S3. The distinctive open table format of Iceberg captures detailed metadata and facilitates seamless integration with leading compute engines such as Spark, Trino, and Hive. This integration empowers Amazon EMR to provision clusters efficiently without additional setup, enhancing operational agility and scalability. 

The recent enhancement of Athena with ACID transactions powered by Iceberg underscores Amazon’s commitment to bolstering transactional capabilities within their analytics offerings. This strategic adoption enables Amazon to effectively handle schema evolution, time travel queries, and complex data operations while ensuring data integrity and query performance at scale. By utilizing Iceberg, Amazon meets the evolving demands of modern data lakes and optimizes cost efficiency and operational effectiveness across their analytics workflows.

Read more about AWS Big Data

2. The data cloud at Snowflake leverages Apache Iceberg for scalable and secure analytics

Snowflake has integrated Apache Iceberg into its Data Cloud platform, empowering organizations to handle expansive and complex data workloads using various file formats, including Parquet, Avro, ORC, JSON, and XML. Snowflake’s internal, fully managed table format simplifies storage maintenance with features like encryption, transactional consistency, versioning, fail-safe, and time travel. However, some organizations prefer or are required to store data externally in open formats due to regulatory or other constraints.

With the introduction of Iceberg tables, Snowflake combines the high performance and familiar query semantics of its native tables with the flexibility of customer-managed cloud storage. This integration, currently in public preview, allows Snowflake users to leverage the open table format of Iceberg, which captures detailed metadata and supports advanced data management capabilities.

Snowflake users can now overcome common data barriers, such as siloed data and inefficient data sharing. The platform enables secure sharing of complex data sets both internally and with external partners, and facilitates large-scale analytics tasks on massive data sets quickly and efficiently. By adopting Iceberg Tables, Snowflake ensures that organizations can achieve optimal performance, scalability, and data governance while maintaining the flexibility of open storage formats.

What innovative solutions will you create with Iceberg?

Imagine developing a customer segmentation model that dynamically adapts to new data inputs without extensive reconfiguration, thanks to the support for schema evolution provided by Iceberg. Picture optimizing your machine learning workflows with real-time data access and historical querying capabilities, enabled by the time travel feature. By adopting this powerful tool, you’re not just optimizing your current operations but setting the stage for future innovations.

The Iceberg advantage: What’s next for you?

Apache Iceberg is revolutionizing modern data architectures with its robust design and advanced features. Its scalable metadata layer, sophisticated data layout, and comprehensive API support deliver unmatched efficiency and reliability for data and ML applications.

We’ve seen a clear shift from legacy data lake and warehousing solutions to Iceberg. With improvements in schema evolution, partitioning, time travel, scalability, and interoperability, companies are now building large-scale ingestion architectures around Iceberg and open-source tools like Spark. This shift underscores the need for flexible and performant data management solutions.

Organizations leveraging Iceberg streamline data processing workflows, enhance data integrity, and drive insightful analytics. Its compatibility with engines like Apache Spark and Flink, ACID transactions, and schema evolution capabilities make Iceberg a cornerstone for scalable, high-performing data lakes.

Future-proof your data operations with Hevo’s comprehensive data integration platform. Even if you’re not using Iceberg, our solutions ensure smooth data handling and integration across your entire ecosystem. Register for a personalized demo, no credit card required.

Frequently Asked Questions