Director (Data/AI) @Cloudera Contributor: Apache Iceberg | Apache Hudi Distributed Computing | Technical Author (O’Reilly, Packt) Prev📍: DevRel @Onehouse.ai, Dremio, Engineering @Qlik, @OTIS Elevators Book 📕: https://a.co/d/fUDs7G6https://bsky.app/profile/dipankartnt.bsky.socialhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lknyugni322r[NEW BLOG]: What is Apache Arrow Flight / Flight SQL / ADBC? 🎉 We need to ask, Why don’t ODBC & JDBC fit in today’s analytical world? These protocols were designed particularly for row-based workloads. What about columnar “Arrow” based data? https://dipankar-tnt.medium.com/what-is-apache-arrow-flight-flight-sql-adbc-a076511122ac18 Mar 2025 15:39 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lknyugni322rhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lirts37p2k2bBlogged: ACID in Lakehouse - How Apache Hudi, Iceberg & Delta Lake implements it? https://www.onehouse.ai/blog/acid-transactions-in-an-open-data-lakehouse22 Feb 2025 17:28 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lirts37p2k2bhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lgxylu5ils26Blogged: Concurrency control methods in a Lakehouse with Apache Hudi, Iceberg & Delta Lake. In this blog I go into the fundamentals of concurrency control, explore why it is essential for lakehouses with OCC, MVCC & Non blocking control. https://hudi.apache.org/blog/2025/01/28/concurrency-control30 Jan 2025 17:20 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lgxylu5ils26https://bsky.app/profile/dipankartnt.bsky.social/post/3lgirdefs6k2zBlogged: Clustering Algorithms in Open Lakehouse formats such as Apache Hudi, Apache Iceberg & Delta Lake. Querying huge volumes of data from storage demands optimized query speed Your queries are fast today, but they might not be over time! Read: https://www.onehouse.ai/blog/what-is-clustering-in-an-open-data-lakehouse24 Jan 2025 16:00 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lgirdefs6k2zhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lfnnjjgxzs2b“Bringing the database kernel to data lakes” - this is what Apache Hudi started with before the world heard of something called “Lakehouse”. Lakehouse means only one thing- data lakes needed the “transactional layer” on top of Parquet for running db-style workloads (both transactional & analytical)13 Jan 2025 21:10 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lfnnjjgxzs2bhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lbinki56nc2hApache XTable at Scale in Production! This is a solid example to show the metadata translation capability for open table formats like Iceberg, Hudi & Delta. Fabric users can work with Iceberg tables written by Snowflake without any rewrites/stuff. Link: https://blog.fabric.microsoft.com/en-us/blog/store-and-use-your-snowflake-iceberg-data-in-onelake?ft=All21 Nov 2024 23:46 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lbinki56nc2hhttps://bsky.app/profile/dipankartnt.bsky.social/post/3lbinki56nc2hApache XTable at Scale in Production! This is a solid example to show the metadata translation capability for open table formats like Iceberg, Hudi & Delta. Fabric users can work with Iceberg tables written by Snowflake without any rewrites/stuff. Link: https://blog.fabric.microsoft.com/en-us/blog/store-and-use-your-snowflake-iceberg-data-in-onelake?ft=All21 Nov 2024 23:46 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3lbinki56nc2hhttps://bsky.app/profile/dipankartnt.bsky.social/post/3laiijz62w22x5 techniques to optimize performance for your Open Lakehouse. No matter how performant your compute engine is, your storage needs data to be optimally organized. Here’s a new blog I published! https://www.onehouse.ai/blog/how-to-optimize-performance-for-your-open-data-lakehouse09 Nov 2024 04:51 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3laiijz62w22xhttps://bsky.app/profile/dipankartnt.bsky.social/post/3la5l7zy3hk23Fast "Copy-On-Write" on Apache Parquet. Upserts are crucial for use cases like CDC. There are two ways for record-level updates in data lakes: Copy-On-Write (CoW) Merge-On-Read (MoR) A 🧵on Uber's new CoW optimization for Parquet.04 Nov 2024 20:40 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3la5l7zy3hk23https://bsky.app/profile/dipankartnt.bsky.social/post/3l7vx6gu5262uAnnouncing my new book - "Engineering Lakehouses with Open Table Formats" 🎉 TBH, I have been thinking about this for quite some time. Most of the times, in conversations with engineers exploring these formats, so many questions have come up.01 Nov 2024 19:53 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3l7vx6gu5262uhttps://bsky.app/profile/dipankartnt.bsky.social/post/3l7mpryyxky2vI started this new repo to serve as a central hub for content that I create & insights that I share from different sources about the Lakehouse architecture. Here’s what you will find in the repo: ⭐️ Blogs ⭐️ Research Papers summary ⭐️ Code ⭐️ Crisp Social posts Link: https://github.com/dipankarmazumdar/awesome-lakehouse-guide29 Oct 2024 03:46 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3l7mpryyxky2vhttps://bsky.app/profile/dipankartnt.bsky.social/post/3l7mpryyxky2vI started this new repo to serve as a central hub for content that I create & insights that I share from different sources about the Lakehouse architecture. Here’s what you will find in the repo: ⭐️ Blogs ⭐️ Research Papers summary ⭐️ Code ⭐️ Crisp Social posts Link: https://github.com/dipankarmazumdar/awesome-lakehouse-guide29 Oct 2024 03:46 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3l7mpryyxky2vhttps://bsky.app/profile/dipankartnt.bsky.social/post/3l7ly4c6i2b2hHey 👋 folks! It’s very nice to see a lot of data community people here. I am Dipankar & I have been involved in open source space for some time. Currently I focus on Data Infra with projects such as Apache Hudi, Iceberg, Arrow & XTable. In my prev gigs, I worked on different spectrum of Data.28 Oct 2024 20:43 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3l7ly4c6i2b2hhttps://bsky.app/profile/dipankartnt.bsky.social/post/3l7kax32wma2zWhat is Incremental Processing & how orgs like Uber benefit from it in a Lakehouse? Incremental processing is a technique that processes data in small increments rather than in one large batch.28 Oct 2024 04:16 +0000at://did:plc:jccfu22khman7hppgwdevmml/app.bsky.feed.post/3l7kax32wma2z