<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Chasing Insights by Corey Satnick]]></title><description><![CDATA[Chasing Insights by Corey Satnick]]></description><link>https://chasinginsights.com</link><generator>RSS for Node</generator><lastBuildDate>Tue, 08 Sep 2026 01:00:13 GMT</lastBuildDate><atom:link href="https://chasinginsights.com/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How to Design a Medallion Architecture for Real-Time Intelligence]]></title><description><![CDATA[In my last post on medallion architecture, we talked about Bronze, Silver, and Gold layers — how to land your data, clean it, and serve it up in a way that doesn't make your analysts cry. That post wa]]></description><link>https://chasinginsights.com/how-to-design-a-medallion-architecture-for-real-time-intelligence</link><guid isPermaLink="true">https://chasinginsights.com/how-to-design-a-medallion-architecture-for-real-time-intelligence</guid><category><![CDATA[microsoft fabric]]></category><category><![CDATA[Real-time-intelligence]]></category><dc:creator><![CDATA[Corey Satnick]]></dc:creator><pubDate>Sun, 26 Apr 2026 17:06:21 GMT</pubDate><enclosure url="https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/688cdb4af23a6b911e3c8352/4ab4b9cc-f11b-41ce-97eb-1b8d4b5494e0.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In my last post on medallion architecture, we talked about Bronze, Silver, and Gold layers — how to land your data, clean it, and serve it up in a way that doesn't make your analysts cry. That post was focused on batch: scheduled pipelines, predictable loads, data that waits politely for its turn.</p>
<p>But what happens when your data <em>doesn't</em> wait?</p>
<p>What happens when a customer abandons a cart, a machine on the factory floor throws an error, or a fraudulent transaction hits your payment system — and you need to know about it in seconds, not hours?</p>
<p>That's where streaming comes in. And no, streaming medallion isn't just "batch but faster." It's a fundamentally different way of thinking about your pipeline — one where the tools change, the tradeoffs change, and the failure modes change. The good news is the layers themselves — Bronze, Silver, Gold — still hold up. You're just building them with different materials.</p>
<h2>Batch vs. Streaming: The Core Difference</h2>
<p>Before we get into the architecture, let's be honest about what we're actually changing.</p>
<p>In batch, you tolerate staleness. You run a pipeline every hour, every night, or every week — and that's fine, because the business question doesn't demand anything fresher. "What were yesterday's sales by region?" works just fine on a midnight load.</p>
<p>In streaming, staleness <em>is</em> the problem. Fraud detection that runs every hour isn't fraud detection — it's an audit. A real-time inventory dashboard that's 45 minutes behind might as well be wrong. When the business question has a latency requirement, batch doesn't cut it.</p>
<p>The other thing that changes is <em>tolerance for imperfection</em>. In batch, your data arrives in a tidy package: here's everything that happened between 2 AM and 3 AM, in order, go process it. In streaming, data arrives out of order, at irregular intervals, sometimes duplicated, and occasionally very late. A mobile app might buffer events when the device is offline and flush them 20 minutes later. An IoT sensor might retry a failed send. Your pipeline needs to handle all of that gracefully — or you end up with numbers that quietly don't add up.</p>
<h2>The Fabric Real-Time Intelligence Stack</h2>
<p>Microsoft Fabric has a purpose-built stack for streaming workloads under its Real-Time Intelligence (RTI) umbrella. The goal, in Microsoft's own framing, is to go from "data-in-motion to action-in-motion." Here's what that actually looks like in practice:</p>
<ul>
<li><p><strong>Event Hubs / Kafka</strong> — the source. This is where your event producers (apps, devices, services) push data. Event Hubs is Azure's managed service; Kafka is the open-source equivalent that Fabric also supports natively.</p>
</li>
<li><p><strong>Eventstream</strong> — Fabric's no-code/low-code ingestion layer. It connects to your Event Hubs or Kafka topic, fans data out to multiple destinations simultaneously (Eventhouse, Lakehouse, Activator), and can apply lightweight transformations and schema validation before data arrives downstream.</p>
</li>
<li><p><strong>Eventhouse / KQL Database</strong> — the real-time analytics engine. Built for high-frequency, time-series, append-heavy workloads. Uses KQL (Kusto Query Language). This is where Bronze, Silver, and Gold live for streaming workloads — not as physically separated storage locations, but as logical layers implemented through ingestion policies and query constructs.</p>
</li>
<li><p><strong>Activator</strong> — the action engine. This is the part most batch-centric data engineers haven't heard of, and it's worth paying attention to. Activator watches your data for conditions and thresholds, then fires automated actions: Teams messages, Power Automate flows, email alerts, pipeline triggers, notebook runs. It's what turns a streaming architecture from a "reporting faster" play into a genuine "respond to events" platform.</p>
</li>
</ul>
<h2>Bronze: Capture Everything, Immediately</h2>
<p>The Bronze mindset in streaming is the same as in batch: land it raw, land it fast, don't drop anything.</p>
<p>In batch, Bronze usually means a Lakehouse table — append-only, full fidelity, whatever format the source sent. In streaming, the same principle applies, but the Eventhouse is a better home for raw high-frequency events. KQL Databases are built for this exact scenario — they can ingest millions of events per second, handle out-of-order arrival natively, and don't suffer from the small-file problem that plagues Delta tables under heavy write loads. (If you've read my post on Silver layer optimization, you know exactly what I mean.)</p>
<p>Your Bronze KQL table should be append-only, preserve the full original payload, and include both an event timestamp (when the thing happened) and an ingestion timestamp (when it arrived). These are not the same thing. In a perfect world, they're milliseconds apart. In the real world, mobile apps go offline, networks hiccup, sensors retry — and debugging a downstream discrepancy almost always requires going back to both.</p>
<p>One thing worth flagging early: how Eventstream gets data into Eventhouse matters. There are two modes:</p>
<ul>
<li><p><strong>Pull mode</strong> is faster and cheaper. The Eventhouse itself reaches out to the underlying Event Hub and pulls events. No extra transformation layer. This is the default for direct Bronze landing.</p>
</li>
<li><p><strong>Push mode</strong> kicks in when you apply transformations inside Eventstream. Under the covers, Fabric spins up an Azure Stream Analytics job to process your data before pushing it to the destination. This is powerful, but it adds cost and latency — so apply transformations in Eventstream only when you genuinely need them at that layer.</p>
</li>
</ul>
<p>The other thing Bronze gets right by design is <strong>schema drift</strong> — or rather, the absence of it. Store your raw payload as a <code>dynamic</code> column in KQL, which is Kusto's semi-structured type for JSON. No columns to enforce, no fields to map, no pipeline to break when the source team renames something without telling anyone. You're just landing the event exactly as it arrived. Schema drift becomes Silver's problem — which is exactly where it should be, since that's where you're actually parsing <code>dynamic</code> fields into typed columns. If a malformed payload is going to cause issues, let it surface there through your update policy, where you can handle it with a dead-letter routing pattern, rather than at Bronze where you're trying to touch as little as possible.</p>
<h2>Silver: Transform on the Way In</h2>
<p>This is where streaming medallion gets interesting — and where most people either get it right or create a mess they'll regret later.</p>
<p>In the Eventhouse, Silver transformations are handled primarily through <strong>KQL Update Policies</strong>. An update policy is a rule attached to a target table that says: "whenever data lands in Bronze, run this KQL function and write the result here." It fires automatically at ingest time — which means your Silver table stays in sync with Bronze without any separate pipeline job.</p>
<p>A few things to understand about how update policies actually work under the hood, because they have real operational implications:</p>
<ul>
<li><p>They execute <strong>synchronously at ingestion time</strong>. That means a poorly designed update policy doesn't just slow down your Silver table — it slows down Bronze ingestion itself. Keep them lean and deterministic.</p>
</li>
<li><p>If an update policy query fails, Bronze ingestion still succeeds. Your source table gets the data, but the target table doesn't. Use <code>.show ingestion failures</code> to catch these — they won't announce themselves otherwise.</p>
</li>
<li><p>You <strong>cannot chain update policies</strong>. If your Silver table has its own update policy, Fabric will not recursively fire it. Bronze → Silver is fine. Bronze → Silver → Gold via update policies is not supported.</p>
</li>
</ul>
<p>Update policies are ideal for: parsing raw JSON or string payloads into typed columns, renaming or reshaping fields to match your enterprise schema, enriching events with reference data (device IDs → device types, store codes → store names), and forking a raw stream into multiple shaped tables for different consumer teams.</p>
<p>What update policies are <em>not</em> ideal for: anything stateful. And "stateful" is worth unpacking, because it's where a lot of people get tripped up.</p>
<p>Update policies are <strong>stateless</strong> — they only know about the events in the current ingestion batch. They have no memory of what came before. If your logic can be answered by looking at a single row in isolation, update policies work great. If your logic requires "look at this row <em>and</em> remember what happened before," you need something with memory.</p>
<p>Examples:</p>
<ul>
<li><p><strong>Cart abandonment detection</strong>. A customer adds items to their cart and goes quiet. You want to fire an alert after 15 minutes of inactivity. An update policy can't do this — when the next event arrives, it can see that event, but it has no way of knowing when that customer last did anything, or whether 15 minutes have elapsed since their last cart action. That requires tracking state across events, specifically last-activity-time per customer.</p>
</li>
<li><p><strong>Card velocity fraud detection</strong> — "flag if the same payment card is used three or more times in five minutes." The update policy sees one transaction. It can't count how many times that card has appeared in the last five minutes because those events already landed before this ingestion batch.</p>
</li>
</ul>
<p>For these patterns, <strong>Activator</strong> is the right tool. It maintains in-memory state per tracked entity (per customer, per card, per device) and evaluates rules across that history — exactly what update policies can't do. For more complex stateful aggregations, a scheduled KQL query against your Bronze or Silver table can cover the gap.</p>
<p>The other thing Silver has to deal with is <strong>late-arriving data</strong>. Your pipeline needs to decide: how long do I wait for out-of-order events before closing a time window? There's no universal answer, but not having one is worse than having a slightly wrong one. Pick a watermark, document it, and make sure your downstream materialized views account for it.</p>
<h2>Gold: Real-Time Insights, Not Real-Time Regrets</h2>
<p>Gold in streaming is where you ask: what does the business actually need to see, and how fresh does it need to be?</p>
<p>In the Eventhouse, Gold is typically served through <strong>Materialized Views</strong>. A materialized view pre-computes an aggregation over a source table and maintains it incrementally as new data arrives. The internal architecture is worth understanding: the engine maintains a hidden physical table (the materialized part) plus a "delta" of recently ingested records that haven't been materialized yet. When you query a materialized view, you get both parts combined — so you're never looking at stale data, even between materialization cycles.</p>
<p>A few constraints worth knowing: materialized views are defined over a single source table with a single <code>summarize</code> statement. No joins. No multi-stage logic. No chaining (except in the narrow case where the upstream view uses <code>take_any()</code> for deduplication). If you need those things, materialize into an intermediate table first and then build the view on top of it.</p>
<p>For performance-critical dashboards where you'd rather trade a few minutes of freshness for guaranteed sub-second query times, you can reference only the materialized portion using <code>materialized_view("ViewName", max_age)</code>. If the view was materialized more recently than <code>max_age</code>, you get the fast pre-computed result. If it's stale, the engine falls back to querying the full table. Useful lever to have when your ops team is staring at a live dashboard during an incident.</p>
<p>Common Gold materialized view patterns:</p>
<ul>
<li><p><strong>Downsampling:</strong> <code>summarize count() by bin(Timestamp, 1h)</code> — great for time-series dashboards and trend lines</p>
</li>
<li><p><strong>Last known state:</strong> <code>summarize arg_max(Timestamp, *) by DeviceId</code> — current status per entity</p>
</li>
<li><p><strong>Deduplication:</strong> <code>summarize take_any(*) by EventId, SessionId</code> — clean up noisy or retried event streams</p>
</li>
</ul>
<p>For your Power BI real-time dashboards, the modeling guidance from a streaming context mirrors what you'd do in batch: fact tables on DirectQuery, small dimension tables on Dual mode. The key difference is that your fact table is now a KQL materialized view or table queried directly — which means datetime handling matters a lot. Always work in UTC inside Eventhouse. Convert to local time at the visualization layer, not in the query, or you'll pay for it in performance.</p>
<p>The reason is subtle but important: Eventhouse indexes on UTC datetime. If you run <code>datetime_utc_to_local()</code> inside your KQL query, the conversion happens <em>before</em> the filter gets applied, which means the engine can't use the index to push your time range filter down. You've just turned a fast indexed scan into a full table scan. Filter first in UTC, then convert for display.</p>
<p>The cleanest place to do the conversion is a DAX measure in your Power BI semantic model. That way the KQL query runs against UTC (fast), and only the values that survive the filter get converted for display. A simple example for Eastern Time:</p>
<pre><code class="language-dax">Transaction Time (Eastern) =
[TransactionTimestamp] - TIME(5, 0, 0)
</code></pre>
<p>A couple of caveats: this approach uses a fixed offset, which means you'll need to update it for daylight saving time (or build DST logic into the measure). For most operational dashboards where users are looking at the last few hours of data, a static offset is fine. For anything that spans DST transitions, you'll want a more robust time zone table or a calculated column on your date dimension that accounts for DST by date range. Either way, keep the conversion in DAX — not in KQL.</p>
<h2>Beyond Gold: Activator and the Action Layer</h2>
<p>Here's where streaming architecture does something batch simply can't: it closes the loop.</p>
<p>In a batch world, Gold is the end of the line. You produce reports, people look at them, and — if something's wrong — someone notices, opens a ticket, and eventually someone acts. In streaming, you can skip the human in the middle.</p>
<p>Activator connects to your Eventhouse data and watches for conditions: a temperature sensor crossing a threshold, a fulfillment SLA at risk, a fraud pattern matching a known signature. When the condition fires, Activator executes: a Teams alert to the operations team, a Power Automate flow to open a work order, a Fabric pipeline to trigger a remediation process.</p>
<p>Activator supports both stateless rules (value &gt; threshold, evaluated on every event) and stateful rules that track transitions ("value exceeded threshold for more than 5 consecutive minutes" or "device went offline"). It maintains in-memory state per tracked entity — per device, per order, per session — and includes suppression logic so you're not bombarding your team with 10,000 alerts for the same event.</p>
<p>The reason this matters architecturally is that it changes the design conversation. You're not just asking "how do we store and query this data faster?" You're asking "what should happen when this data means something?" That's a more interesting question — and one that's worth having with your business stakeholders before you finalize your Gold layer design.</p>
<h2>OneLake Availability: One Copy, Two Worlds</h2>
<p>Here's where things get genuinely elegant — and where the streaming and batch medallion architectures reunite.</p>
<p>One of the most common pain points in streaming architectures is data fragmentation: your real-time data lives in one system, your historical data lives in another, and getting them to talk to each other involves either copying data or building custom connectors.</p>
<p>Fabric solves this through <strong>OneLake availability</strong> in Eventhouse. When enabled on a KQL table, Fabric automatically converts Eventhouse data to Parquet format and makes it available in OneLake. And the critical detail here: it's not a copy. Eventhouse uses <strong>shared storage pointers</strong> — the same data is referenced in OneLake as Delta/Parquet without duplication, which means zero additional storage cost for the exposure.</p>
<p>The internal flow looks like this: when events land in Eventhouse, they're initially stored in <strong>rowstore format</strong> — optimized for the lowest possible ingest latency and immediately queryable. In the background, Eventhouse asynchronously converts that rowstore data into <strong>columnar Parquet</strong> (columnstore) for long-term analytics performance and compression. It's the OneLake-exposed columnstore version that other compute engines — Spark notebooks, Lakehouse shortcuts, Power BI Direct Lake — can see and query.</p>
<p>A few things worth knowing before you flip the switch:</p>
<ul>
<li><p><strong>Trickling ingestion introduces lag.</strong> If your event rate is low (a handful of events per second), it can take up to <strong>3 hours</strong> for data to appear in OneLake. Heavier ingestion rates trigger faster columnstore materialization. If you're building integrations that depend on OneLake availability, design your ingest rate accordingly.</p>
</li>
<li><p><strong>Schema and deletion become restricted.</strong> Once OneLake availability is enabled on a table, you cannot alter the table's schema or delete data from it without first disabling the feature. Design for append-only workloads from the start.</p>
</li>
<li><p><strong>Expose strategically.</strong> Not every table needs OneLake visibility. Over-exposing can lead to operational overhead and unnecessary cost. Enable it on the tables where cross-engine access genuinely adds value — your Gold materialized views being the most obvious candidates.</p>
</li>
</ul>
<p>The payoff: your data science team can run Spark notebooks against real-time event data. Your Warehouse can join streaming metrics against historical batch dimensions. The ops dashboard and the quarterly business review are reading from the same truth. That's a genuinely hard problem, and the one-copy design solves it cleanly.</p>
<h2>A Quick Word on Complexity (the Reality Check)</h2>
<p>Streaming adds real engineering overhead. Update policies need to be lean or they'll drag down ingestion. Schema drift will bite you the first time a source team renames a field without telling anyone. Late-arriving data needs a handling strategy. Activator rules need suppression logic or your on-call team will mute the alerts within a week. and OneLake availability has enough gotchas — especially around trickling ingestion and schema locking — that it deserves its own design conversation before you enable it in production.</p>
<p>When the use case genuinely demands it — fraud detection, IoT monitoring, live inventory, operational alerting, clickstream personalization — the Fabric RTI stack gives you a clean, scalable path from event source to business insight and automated action, with the same Bronze/Silver/Gold discipline you'd apply in batch. Same architecture. Different speed. And if you set it up right, something batch can never be: self-responding.</p>
<hr />
<p><em>Further reading:</em></p>
<ul>
<li><p>Chris Schmidt's Blogs:</p>
<ul>
<li><p><a href="https://www.linkedin.com/pulse/data-transformation-real-time-intelligence-christopher-schmidt-i9tse?trackingId=FcikO9%2FjSu26nVbvJwuiUA%3D%3D&amp;lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_recent_activity_content_view%3BkQSNjuDURGK9M535zDW4zw%3D%3D">Data Transformation RTI</a></p>
</li>
<li><p><a href="https://learn.microsoft.com/en-us/fabric/real-time-intelligence/eventhouse">Eventhouse overview</a></p>
</li>
</ul>
</li>
<li><p>Microsoft Learn</p>
<ul>
<li><p><a href="https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/update-policy">KQL Update Policies</a></p>
</li>
<li><p><a href="https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/materialized-views/materialized-view-overview">Materialized views overview</a></p>
</li>
<li><p><a href="https://learn.microsoft.com/en-us/fabric/real-time-intelligence/onelake-availability">OneLake availability in Eventhouse</a></p>
</li>
<li><p><a href="https://learn.microsoft.com/en-us/fabric/real-time-intelligence/data-activator/activator-introduction">Activator overview</a></p>
</li>
<li><p><a href="https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/overview">Eventstream overview</a></p>
</li>
</ul>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Optimizing Delta Tables in the Silver Layer]]></title><description><![CDATA[Congrats you’ve built your first medallion architecture. Good news is you’ve finally gotten through all the business lines of questions, validation, and intense engineering workloads. The issue is now your processes are slowly getting slower and slow...]]></description><link>https://chasinginsights.com/optimizing-delta-tables-in-the-silver-layer</link><guid isPermaLink="true">https://chasinginsights.com/optimizing-delta-tables-in-the-silver-layer</guid><dc:creator><![CDATA[Corey Satnick]]></dc:creator><pubDate>Mon, 29 Dec 2025 16:24:13 GMT</pubDate><content:encoded><![CDATA[<p>Congrats you’ve built your first medallion architecture. Good news is you’ve finally gotten through all the business lines of questions, validation, and intense engineering workloads. The issue is now your processes are slowly getting slower and slower. Why? How do you fix that? Why is your capacity spiking?</p>
<p>In this blog post I’ll talk about how to optimize your silver layer. And no I’m not talking about the easy stuff like only bringing in the columns you need, or proper data types. We’re going to dive deep into spark properties and setting up your environment.</p>
<p>To keep performance fast as data grows and changes, Fabric relies on Delta Lake optimizations—not Parquet optimizations. Because Delta Tables support updates, deletes, merges, and streaming workloads, the transaction log can grow, files can fragment, and query performance can degrade over time. The good news: Fabric includes several intelligent features that mitigate this. Below is an overview of the major Delta optimization mechanisms available in Fabric today, why they matter, and when to use them.</p>
<p>But first lets break down what Parquet files really are. If you already know this (or simply don’t care) you can skip this part. But I know some people like to know how the sausage gets made.</p>
<h2 id="heading-parquet-what-is-it-and-why-you-should-care">Parquet- What is it and why you should care?</h2>
<p>Parquet gets talked about like it’s some magical file format that automatically makes your analytics fast just by existing. And to be fair… it kind of is. But only if you treat it right.</p>
<p>Parquet is an open-source columnar storage format designed specifically for analytics.</p>
<p>At a high level, Parquet stores data by column instead of by row, which is what unlocks most of its benefits:</p>
<ul>
<li><p>Better compression (similar values compress really well together)</p>
</li>
<li><p>Faster queries because engines only read the columns they need</p>
</li>
<li><p>Predicate pushdown meaning filters get applied before all the data is read</p>
</li>
<li><p>Schema evolution so your data doesn’t explode the first time someone adds a column</p>
</li>
</ul>
<p>That’s why Parquet is the default for basically every modern analytics engine. It’s efficient, flexible, and plays nicely with distributed systems.</p>
<p><code>quick note on faster queries – I recently had a customer try to convince me that they need a table with 100 columns in it for analytics. It doesn’t matter what file format you use – a select * against that many columns will be slow. No amount of optimization can save poor data modeling. Ok first rant over.</code></p>
<p><strong>Okay, But What Does “Columnar” Actually Mean?</strong></p>
<p>Well to oversimplify it- columnar means all of your data is stored column by column, not row by row. Look at this visual below.</p>
<p><img src="https://res.cloudinary.com/dgzgmhbah/image/upload/v1607008073/Hittly/BigQuery_Columnar_Storage_ybqm5s.png" alt="BigQuery Columnar Storage" /></p>
<p>When you run a report, you’re usually asking questions like:</p>
<ul>
<li><p>“What was total revenue last month?”</p>
</li>
<li><p>“How many orders were flagged as delayed?”</p>
</li>
<li><p>“Show me counts by status”</p>
</li>
</ul>
<p>You don’t need every column. You need two or three per query which means less data scanned resulting in:</p>
<ul>
<li><p>Faster queries</p>
</li>
<li><p>Lower compute usage</p>
</li>
<li><p>Happier capacity admins and business owners who have to spend less money.</p>
</li>
</ul>
<h3 id="heading-what-about-the-negatives-of-parquet">What About the Negatives of Parquet?</h3>
<p>Up to this point, Parquet sounds perfect. And honestly, for analytics reads, it mostly is.<br />But here’s the part that usually gets glossed over:</p>
<p><strong>Parquet is immutable.</strong></p>
<p>Once a Parquet file is written, it can’t be updated in place. There’s no “go change row 37” operation. That design choice is what makes it fast for reads—but it also introduces some very real tradeoffs once you start changing data.</p>
<p>And in the Silver layer, you change data a lot.</p>
<p>So when you “update” or “delete” data, what’s really happening is:</p>
<ul>
<li><p>New Parquet files get written</p>
</li>
<li><p>Old data gets logically invalidated</p>
</li>
<li><p>The old files still exist but fabric is smart enough to point to the proper file(s).</p>
</li>
</ul>
<p>Over time, this creates:</p>
<ul>
<li><p>Lots of small files</p>
</li>
<li><p>Files full of data you no longer care about</p>
</li>
<li><p>Extra work for the engine every time you query</p>
</li>
</ul>
<p>Don’t believe me? Open a spark notebook and run a describe detail on a table that has a lot of transactions on it.</p>
<p><code>%sql Describe Detail [table]</code></p>
<p>Let me know what the stats show on one of your non optimized tables.</p>
<p>I bet you’ll see a bunch of files and more so a lot of them aren’t needed. If you’re from the Stone Age you probably are thinking to yourself “That sounds like dead tuples” and you’d be right.  It’s data that is technically gone, but still hanging around until someone cleans it up.</p>
<p>Now that you understand Parquet, it’s important to recognize that Delta Tables are a transactional layer built on top of Parquet. Fabric stores Delta Tables as Parquet files plus a transaction log. Because of this extra metadata and transactional behavior, optimizing Delta Tables requires different techniques than simply optimizing Parquet.</p>
<h2 id="heading-delta-features-that-actually-matter-in-the-silver-layer"><strong>Delta Features That Actually Matter in the Silver Layer</strong></h2>
<h3 id="heading-deletion-vectors">Deletion Vectors</h3>
<p>Deletion vectors are a Delta Lake optimization that work around Parquet immutability, avoiding full file rewrites when only a handful of rows change.</p>
<p>By default, if a single row in a Parquet file needs to be deleted or updated, Delta Lake performs a Copy on Write approach - which rewrites the whole file. For large Parquet files, that’s expensive, especially in the Silver layer where you’re continuously applying row‑level CDC, merges, or backfills.</p>
<p>With deletion vectors enabled, Delta takes a smarter approach. Instead of rewriting the file immediately, it records which rows are no longer valid and stores that information separately. This is often referred to as Merge‑on‑Read. When the table is queried, Delta applies those deletion markers to exclude invalid rows at read time.</p>
<p>The result:</p>
<ul>
<li><p>Far fewer file rewrites</p>
</li>
<li><p>Less write amplification</p>
</li>
<li><p>Faster DELETE and MERGE operations</p>
</li>
</ul>
<p>In other words: you can invalidate one row out of 100,000 without paying the cost of rewriting the entire file.</p>
<h3 id="heading-auto-compaction"><strong>Auto Compaction</strong></h3>
<p>Auto compaction combines small Parquet files within a Delta table’s partitions to reduce the classic small file problem. It’s triggered after a successful write and runs synchronously on the cluster that performed the write.</p>
<p>Rather than rewriting the entire table, auto compaction groups small files together and writes fewer, larger files. Files may be compacted multiple times as new data arrives, and only stop being considered once they reach the effective size threshold (for example, at least half of the target file size). This keeps ingest and merge pipelines fast while gradually improving file layout over time.</p>
<p>Why it matters</p>
<ul>
<li><p>Reduce file sprawl</p>
</li>
<li><p>Improve read performance (less files = fewer metadata lookups)</p>
</li>
<li><p>Keep your table layout healthy—without manual OPTIMIZE jobs. Auto compaction quietly maintains good file hygiene, this way engineers don’t have to routinely schedule OPTIMIZE jobs or manually manage file layout.</p>
</li>
</ul>
<h3 id="heading-adaptive-target-file-size"><strong>Adaptive Target File Size</strong></h3>
<p>Hardcoding Parquet file sizes is a guessing game. At some point, everyone hears a rule like “128 MB or 256 MB files are ideal” and locks it into a Spark config. It works - until your data, usage patterns, or table size change.</p>
<p>Silver workloads are not static. Ingest volumes grow, merges become more frequent, backfills happen, and CDC patterns evolve. A file size that was perfect at 5 million rows quietly becomes a bottleneck at 500 million.</p>
<p>Adaptive target file sizing removes that guesswork. Instead of enforcing a fixed size forever, Fabric dynamically adjusts file sizes based on table-level metrics (such as overall table size) at write time. The engine determines an appropriate target size per write, rather than relying on a single hardcoded value in the table definition.</p>
<h3 id="heading-fast-optimize">Fast OPTIMIZE</h3>
<p>Traditional <code>OPTIMIZE</code> rewrites every bin of small files it finds - even ones that are already “good enough.” That wastes compute, drags out jobs, and doesn’t always move the performance needle. Fast OPTIMIZE thinks before it rewrites. It scans each bin and only compacts it when the result is expected to meet the minimum file size threshold or when there are enough small files to justify compaction.</p>
<p>The outcome is a tighter file layout, fewer tiny files, and more efficient reads. Query planning becomes cheaper, scans become faster, and metadata pressure drops. Capacity usage becomes far more predictable, because the work happens through Fabric’s native engine instead of large Spark shuffles.</p>
<p>What Fast OPTIMIZE Actually Does</p>
<ul>
<li><p>Looks at each group of files (bins) before rewriting anything.</p>
</li>
<li><p>Skips bins where compaction won’t improve things.</p>
</li>
<li><p>Focuses compaction work only where it materially improves the resulting file layout.</p>
</li>
</ul>
<h3 id="heading-file-level-compaction-targets">File Level Compaction Targets</h3>
<p>File-level compaction targets define the file size thresholds used to guide compaction decisions. They determine when combining small files is worthwhile and when a file is considered large enough to stop being compacted. By providing clear size boundaries, they prevent unnecessary rewrites and keep compaction work focused on changes that meaningfully improve file layout.</p>
<h3 id="heading-optimize-write-and-v-order">Optimize Write and V-Order:</h3>
<p>Optimize Write acts as pre-write compaction. It introduces additional shuffle and coordination during writes to avoid producing small files in the first place. This can be extremely valuable for trickle or micro-batch workloads (for example, structured streaming jobs or frequent small inserts) where each write would otherwise generate undersized files. In those cases, Optimize Write can reduce downstream compaction work and improve overall efficiency.</p>
<p>However, for many workloads Optimize Write is often a poor trade. Paying extra write-time cost to produce “perfect” files is inefficient when those files are likely to be invalidated, merged, or compacted again shortly after. In these scenarios, pre-write compaction simply shifts work earlier without reducing total work.</p>
<p>V-Order has a similar trade-off. It optimizes file layout for analytical read patterns, which is highly effective when data is relatively stable. Optimizing read layout in a layer where data is constantly changing often results in wasted write effort with little lasting benefit.</p>
<h2 id="heading-closing-thoughts">Closing Thoughts:</h2>
<p>The Silver mindset should be cheap, resilient writes first - smart cleanup later. Enable Optimize Write only when the write pattern would otherwise generate excessive small files. Let deletion vectors absorb row-level churn, rely on auto compaction and adaptive target file sizing to manage file sprawl over time, and use Fast OPTIMIZE to focus heavy rewrite work where it actually matters. Save aggressive read-path optimizations like V-Order for Gold when using direct lake.</p>
<p><a target="_blank" href="https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0">Additional Sources:</a></p>
<ul>
<li><p><a target="_blank" href="https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0">What are deletion vectors? | Delta</a> <a target="_blank" href="https://docs.delta.io/delta-deletion-vectors/">Lake</a></p>
</li>
<li><p><a target="_blank" href="https://docs.delta.io/delta-deletion-vectors/">Unlock Faster Writes in Delta Lake w</a><a target="_blank" href="https://milescole.dev/data-engineering/2024/11/04/Deletion-Vectors.html">ith Deletion Vectors | Miles Cole</a></p>
</li>
<li><p><a target="_blank" href="https://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors">What are deletion vectors? - Azure Databricks | Microsoft Learn</a></p>
</li>
<li><p><a target="_blank" href="https://milescole.dev/data-engineering/2025/02/26/The-Art-and-Science-of-Table-Compaction.html">Mastering Spark: The Art and Science of Table Compaction | Miles Cole</a></p>
</li>
<li><p><a target="_blank" href="https://milescole.dev/data-engineering/2025/02/26/The-Art-and-Science-of-Table-Compaction.html">Configure Delta Lake to control data file size - Azure Databricks</a> <a target="_blank" href="https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size">| Microsoft Learn</a></p>
</li>
</ul>
<p>Additional Notes:</p>
<p>With the new release of Runtime 2.0 there are some exciting things to look for (Spark 4.0 and Delta Lake 4.0 to name a few). There are some limitations though since this is experimental preview. One of the most notable to me is:</p>
<ul>
<li>You can read and write to the Lakehouse with Delta Lake 4.0, but some advanced features like V-order, native Parquet writing, autocompaction, optimize write, low-shuffle merge, merge, schema evolution, and time travel aren't included in this early release.</li>
</ul>
<p>For more information check out: <a target="_blank" href="https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0">Runtime 2.0 in Fabric - Microsoft Fabric | Microsoft Learn</a></p>
]]></content:encoded></item><item><title><![CDATA[Dimensional Modeling 101]]></title><description><![CDATA[In my last post, we talked about the medallion architecture - how to layer your data into Bronze, Silver, and Gold so things are cleaner, faster, and easier to manage.
But here’s the catch: you can build the optimal pipelines, a performant Lakehouse,...]]></description><link>https://chasinginsights.com/dimensional-modeling-101</link><guid isPermaLink="true">https://chasinginsights.com/dimensional-modeling-101</guid><category><![CDATA[data-modeling]]></category><category><![CDATA[dimensional data modeling]]></category><dc:creator><![CDATA[Corey Satnick]]></dc:creator><pubDate>Sun, 07 Sep 2025 16:16:55 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1757262938230/bc442f87-55cc-499e-9a89-86bdc05fed43.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In my last post, we talked about the medallion architecture - how to layer your data into Bronze, Silver, and Gold so things are cleaner, faster, and easier to manage.</p>
<p>But here’s the catch: you can build the optimal pipelines, a performant Lakehouse, and governance rules tighter than Fort Knox… and your reports can still suck if the data model is wrong.</p>
<p>Slow reports. High capacity usage. Dashboards timing out. Most people point fingers at Power BI - but nine times out of ten, the real culprit is a poorly designed data model.</p>
<p>So how do we fix it? How do we build something that performs, scales, and doesn’t drive users back to their old friend Excel?</p>
<h2 id="heading-start-with-kimball"><strong>Start with Kimball</strong></h2>
<p><strong>Ralph Kimball</strong> is widely regarded as the <strong>godfather of modern data modeling for analytics</strong> - not because he invented databases, but because he understood something many people still overlook:</p>
<p><em>A good data model isn’t just technically correct. It has to make sense to the people using it.</em></p>
<p>Before Kimball, most models followed the Inmon/Third Normal Form approach. Great for transactional systems (where you care about writes), not so great for analytics (where you care about reads). Kimball flipped the script and said: keep it simple, keep it intuitive, and the whole system runs better.</p>
<h2 id="heading-the-heart-of-the-dimensional-model-the-star-schema">The Heart of the Dimensional Model: The Star Schema</h2>
<h3 id="heading-fact-tables-the-core-of-measurement">Fact Tables: The Core of Measurement</h3>
<p>Fact Tables are where your aggregations live. <strong>Sales, revenue, units sold, discounts, returns</strong> - if your business measures it, it belongs here.</p>
<p>Each row = a real-world event at its lowest grain. That could be a single transaction at the register, a return at the service desk, or an online order being shipped.</p>
<p>A fact table always includes foreign keys to the dimensions around it - <strong>customer, product, date, store, promotion</strong> - so you can answer questions like:</p>
<ul>
<li><p>Which products sell the most by store?</p>
</li>
<li><p>Which customers are most profitable?</p>
</li>
<li><p>Which promotions actually drove traffic instead of just margin loss?</p>
</li>
</ul>
<p>And because fact tables are massive, they must be centralized. If every region or department builds their own version of “sales,” you’ll end up with different numbers for the same metric - and good luck explaining that to the CFO when the dashboards don’t match.</p>
<p>Not all facts are created equal, though. Here’s the quick rundown:</p>
<ul>
<li><p><strong>Additive</strong>: Can be summed across any dimension (sales amount, quantity sold).</p>
</li>
<li><p><strong>Semi-additive</strong>: Can be summed across <em>some</em> dimensions but not all (inventory balances add up across products, not across time).</p>
</li>
<li><p><strong>Non-additive</strong>: Ratios and percentages (gross margin %, conversion rate) that need to be calculated, often in the BI layer, from their additive building blocks.</p>
</li>
</ul>
<p>Keeping these distinctions straight will save you from half the reporting headaches that developers run into when the “same metric” looks different in finance vs. operations.</p>
<h2 id="heading-the-three-flavors-of-fact-tables">The Three Flavors of Fact Tables</h2>
<h3 id="heading-1-transaction-fact-tables">1. Transaction Fact Tables</h3>
<ul>
<li><p>Grain = the individual sale or return.</p>
</li>
<li><p>One row = one receipt line.</p>
</li>
<li><p>Perfect for <strong>basket analysis, customer journeys, or SKU-level profitability.</strong></p>
</li>
</ul>
<h3 id="heading-2-periodic-snapshot-fact-tables">2. Periodic Snapshot Fact Tables</h3>
<ul>
<li><p>Grain = time period (day, week, month).</p>
</li>
<li><p>One row = “what did sales or inventory look like on this day?”</p>
</li>
<li><p>Perfect for <strong>trending KPIs, tracking store comps, or monitoring daily inventory.</strong></p>
</li>
<li><p>Even if no sales happen in a store, you still log a row- otherwise your dashboards show gaps.</p>
</li>
</ul>
<h3 id="heading-3-accumulating-snapshot-fact-tables">3. Accumulating Snapshot Fact Tables</h3>
<ul>
<li><p>Grain = process with a defined start and finish (e.g., order fulfillment, supply chain, loyalty enrollment).</p>
</li>
<li><p>One row, updated as milestones are hit - order placed, shipped, delivered, returned.</p>
</li>
<li><p>Unique because it’s <em>updated</em> instead of just appended.</p>
</li>
<li><p>Perfect for <strong>tracking online orders, click-to-delivery times, or end-to-end supply chain visibility.</strong></p>
</li>
</ul>
<p>Together, these three types cover almost every analytic need - from <strong>SKU-level margin analysis</strong> to <strong>enterprise-wide sales trends</strong>.</p>
<h3 id="heading-quick-note-on-nulls-and-a-personal-pet-peeve">Quick Note on Nulls (and a personal pet peeve)</h3>
<p>Fact table measures can handle nulls just fine - SUM, COUNT, AVG all behave. But foreign keys? That’s a hard no. Nulls there break referential integrity.</p>
<p>The fix: create an “Unknown” row in your dimension with a surrogate key (often <code>-1</code>). Use <code>COALESCE(key, -1)</code> on load. That way, if something breaks upstream, your report shows “Unknown” instead of silently dropping rows. Plus, it’s a giant neon sign to the data engineer: something went sideways.</p>
<h2 id="heading-dimensions-giving-numbers-their-meaning">Dimensions: Giving Numbers Their Meaning</h2>
<p>If fact tables are where the aggregations live, <strong>dimension tables are what make those numbers make sense.</strong> They turn “$10,392.57” into “Total Net Sales on Sept 1st, 2025 was $10,392.57.”</p>
<p>Every dimension has a single <strong>primary key</strong>, which shows up as a foreign key in your fact table. That’s how you join a row of sales data to its <strong>customer, product, store, date, or marketing campaign</strong>.</p>
<p>Unlike fact tables (which are tall and skinny), dimension tables are usually <strong>wide and flat</strong> - packed with descriptive attributes that people actually filter and group by. Things like:</p>
<ul>
<li><p>Product name, brand, category</p>
</li>
<li><p>Store region, format (mall kiosk vs. superstore)</p>
</li>
<li><p>Promotion type (BOGO, clearance, loyalty points, Labor Day Sale that extended are still happening)</p>
</li>
<li><p>Customer demographics or segments</p>
</li>
</ul>
<h3 id="heading-dealing-with-codes-and-flags">Dealing with Codes and Flags</h3>
<p>Notice what’s not helpful? <strong>Cryptic codes and flags.</strong> “PromoType = C3” column won’t mean much to your business user. Instead, dimensions should spell it out: “PromoType = Clearance.” If your source system insists on giving you codes, expand them into human-friendly descriptions in your dimension table. Keep those keys in your fact table to keep it nice and tight.</p>
<h3 id="heading-drilling-down-and-hierarchies">Drilling Down and Hierarchies</h3>
<p>One of the main reasons dimensions exist is to make analysis intuitive. <a target="_blank" href="https://www.sqlbi.com/articles/introducing-the-3-30-300-rule-for-better-reports/">Kurt Buhler wrote a fantastic blog about the 3, 30, 300 rule I highly recommend reading.</a> Business users should be able to drill down to their desired dataset in under 30 seconds, and get to their detailed information in under 300 seconds.</p>
<ul>
<li><p>Sales by month → week → day</p>
</li>
<li><p>Revenue by region → store → aisle</p>
</li>
<li><p>Inventory by category → brand → SKU</p>
</li>
</ul>
<p>Good dimension design makes this seamless. You don’t need to hardcode every path; as long as the attributes are there, users can explore naturally.</p>
<p>Most dimensions also support <strong>more than one hierarchy</strong>. Examples:</p>
<ul>
<li><p><strong>Date</strong>: Day → Week → Fiscal Period, or Day → Month → Year</p>
</li>
<li><p><strong>Product</strong>: SKU → Category → Department, or SKU → Brand → Line → Group</p>
</li>
</ul>
<p>The takeaway? <strong>Don’t over-engineer the drill path.</strong> Put the attributes in the dimension table, and let users choose the path that matches their business question</p>
<h3 id="heading-the-calendar-date-dimension">The Calendar Date Dimension</h3>
<p>Almost every fact table connects to a <strong>date dimension</strong> - it’s how you move through time in your analysis. Without it, you’re stuck writing messy SQL to figure out things like fiscal periods or holidays (and trust me, you don’t want to compute Easter yourself).</p>
<p>A solid date dimension comes preloaded with all the attributes people care about:</p>
<ul>
<li><p>Day, week, month, quarter, year</p>
</li>
<li><p>Fiscal periods (because finance always has its own calendar)</p>
</li>
<li><p>Holidays and special events (Black Friday, Cyber Monday, Easter, etc.)</p>
</li>
<li><p>Week numbers and month names for easy grouping</p>
</li>
</ul>
<p>To make partitioning easier, the primary key is often a <strong>smart integer</strong> like <code>20250905</code> (YYYYMMDD). But here’s the important part: business users shouldn’t rely on that key. They should slice and filter using the attributes - month name, fiscal week, holiday flag - because that’s what actually makes sense to them.</p>
<p>And don’t forget the edge cases:</p>
<ul>
<li><p>You’ll need a special row for <strong>“Unknown”</strong> or <strong>“TBD”</strong> dates.</p>
</li>
<li><p>If you need more precision, like <em>time of day</em>, you can add a separate <strong>time-of-day dimension</strong> (shifts, day parts, hours).</p>
</li>
<li><p>For detailed timestamps (like order created at 10:42:17 AM), just keep the raw datetime column in the fact table - no need to overcomplicate it.</p>
</li>
</ul>
<hr />
<p>In retail, one of the most common calendar setups is the <strong>4-5-4 model</strong>. Instead of neat calendar months, the year is broken into quarters where the first month has 4 weeks, the second has 5 weeks, and the third has 4 weeks. This design keeps weeks aligned to the same day of the week year over year—so a Saturday in week 32 this year is still a Saturday in week 32 next year—making it much easier to compare sales, traffic, and promotions across periods. It also ensures holidays and seasonal events line up consistently, which is critical for planning, reporting, and year-over-year comp analysis in retail. Here is an example below:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1757095286222/d5ecc478-42a2-47ef-8bb5-73c2f854cbde.png" alt class="image--center mx-auto" /></p>
<p><a target="_blank" href="https://nrf.com/resources/4-5-4-calendar">4-5-4 Calendar | NRF</a></p>
<hr />
<h2 id="heading-role-playing-dimensions-same-table-different-hats">Role-Playing Dimensions: Same Table, Different Hats</h2>
<p>Sometimes the same dimension needs to show up more than once in a fact table - just playing a different role. That’s where <strong>role-playing dimensions</strong> come in.</p>
<p>The best example is the <strong>date dimension</strong>. A single transaction might reference multiple important dates:</p>
<ul>
<li><p><strong>Order Date</strong> → when the customer placed it</p>
</li>
<li><p><strong>Ship Date</strong> → when it left the warehouse</p>
</li>
<li><p><strong>Delivery Date</strong> → when it arrived at their doorstep</p>
</li>
<li><p><strong>Return Date</strong> → when it came back to the store</p>
</li>
</ul>
<p>All of those link back to the <em>same</em> date dimension table - but each plays a different role. Instead of building four separate date tables, you just reuse the one dimension and give each instance an alias (OrderDateKey, ShipDateKey, etc.).</p>
<p>Role-playing isn’t limited to dates, either. You might use the same <strong>employee dimension</strong> to represent both the cashier who rang up a sale and the manager who approved a discount. Or the same <strong>store dimension</strong> to represent both the selling location and the return location.</p>
<h2 id="heading-wrapping-it-up">Wrapping It Up</h2>
<p>Dimensional modeling is more than just a way to organize tables - it’s the <strong>foundation of an enterprise data model</strong>. Get this part right, and everything else - pipelines, governance, dashboards, even AI - sits on top of a structure that’s consistent, scalable, and built for the business.</p>
<p>A strong dimensional model doesn’t just deliver faster queries and cleaner reports. It also sets you up for <strong>optimized AI insights</strong> - because models trained on clean, business-ready data actually produce results you can trust.</p>
<p>And most importantly, it keeps your end users where they belong: <strong>in your BI environment, not back in Excel</strong>. When reports run fast, metrics stay consistent, and the data model “just makes sense,” people stop exporting and start exploring. That’s when your data platform shifts from being a cost center to becoming a competitive advantage.</p>
]]></content:encoded></item><item><title><![CDATA[How to Design a Medallion Architecture for Batch Processing]]></title><description><![CDATA[Let’s be honest— “medallion architecture” sounds like something cooked up to make a slide deck sound cooler than it actually is. But I promise you this one’s actually worth taking the time to learn an]]></description><link>https://chasinginsights.com/the-medallion-architecture-batch</link><guid isPermaLink="true">https://chasinginsights.com/the-medallion-architecture-batch</guid><category><![CDATA[medallion architecture]]></category><category><![CDATA[microsoft fabric]]></category><dc:creator><![CDATA[Corey Satnick]]></dc:creator><pubDate>Tue, 05 Aug 2025 22:06:32 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1754928241412/1aaf6059-3e90-4652-8b32-68a492e3c32d.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Let’s be honest— “medallion architecture” sounds like something cooked up to make a slide deck sound cooler than it actually is. But I promise you this one’s actually worth taking the time to learn and implement.</p>
<p>At its core the medallion architecture is a simple, scalable way to organize your data into three layers: <strong>Bronze, Silver, and Gold</strong>. Yes, like Olympic medals and no you don’t get a podium, but you will get recognition if you implement it correctly and here’s why.</p>
<p>It’s not just about being neat. This structure helps you:</p>
<ul>
<li><p>Cut down on redundant processing</p>
</li>
<li><p>Improve data quality and traceability</p>
</li>
<li><p>Apply smarter governance and access controls</p>
</li>
<li><p>Use your capacity more efficiently (which saves money and will get you that recognition I was talking about above)</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1754070067689/3a693b60-32cd-4ddb-a6ad-152988bda8e5.png" alt="" style="display:block;margin:0 auto" />

<p>Before we talk about how it makes your life easier, let’s walk through what each layer is actually for—and why they matter.</p>
<p><strong>Bronze: The Raw Landing Zone</strong></p>
<p>Welcome to the starting line (well for analytics at least).</p>
<p>The Bronze layer is where <strong>all your data lands first</strong>, untouched and unfiltered. Think of it as your inbox before any rules are applied—where everything piles up before you sort through it. It’s raw, messy, and that’s exactly the point. It’s a replica of your transactional system, excel files of every version of next fiscals budget, and that random json file someone needs for a data science project.</p>
<p>In Fabric, this usually lives in a <strong>Lakehouse</strong>, because it handles both structured (delta tables) and unstructured data (csv, parquet, excel).</p>
<p>So why keep things raw? Because Bronze is your <strong>system of record</strong>—a snapshot of your data exactly as it arrived. No filters. No transformations. Just full fidelity, ready for anything you might need later (like debugging that error that shows up once a month, or proving to your coworker that you were right and it’s the source system’s fault).</p>
<p><strong>Here’s what makes Bronze Bronze:</strong></p>
<ul>
<li><p><strong>Raw by design</strong> – Stores whatever shows up: JSON, CSV, Parquet, you name it. No changes.</p>
</li>
<li><p><strong>Append-only</strong> – New records are added over time. Think of it as a historical log that you can always replay if something goes sideways.</p>
</li>
<li><p><strong>Not for analysis</strong> – This is not the layer your analysts should be querying. You <em>should use it for validation though</em>.</p>
</li>
<li><p><strong>Great for traceability</strong> – You’re keeping the original structure, which helps when you need to trace an issue back to the source.</p>
</li>
<li><p><strong>Flexible ingestion</strong> – Works with both batch and streaming sources—ADLS, S3, Kafka, Event Hubs, you get the idea.</p>
</li>
</ul>
<p>In short, Bronze is the foundation. It’s the layer that lets you confidently say, “Yes, we have the original data—no, we didn’t accidentally overwrite it three months ago.”</p>
<p>Next up? We take that messy inbox and start cleaning it up.</p>
<p><strong>Silver: Cleaned and Modeled</strong></p>
<p>This is where things start to get interesting—and where, in my experience, most of the work actually happens.</p>
<p>Silver is the layer where you take all that raw, messy data from Bronze and start making sense of it. You clean it up, apply structure, and turn it into something the business can actually use. Think of it as the <strong>translation layer</strong>—you're taking “data” and turning it into “information.”</p>
<p>In Fabric, that usually means using <strong>Notebooks</strong>, <strong>Data Pipelines</strong>, or a mix of both to apply your business logic. Maybe you’re flattening nested JSON. Maybe you’re fixing timestamp formats. Maybe you're adding logic that finance swears is <em>critical</em>—kind of like how WeWork swore their “community-adjusted EBITDA” was a thing.</p>
<p>Here’s what Silver is all about:</p>
<ul>
<li><p><strong>Transformation starts here</strong> – Filtering, joining, standardizing, deduplicating—this is where raw data starts becoming analysis-ready.</p>
</li>
<li><p><strong>Business logic lives here</strong> – Whether it’s calculating revenue, flagging status fields, or prepping a clean dimension table, this is your playground.</p>
</li>
<li><p><strong>Not quite final</strong> – It’s not ready for dashboards yet, but it’s miles ahead of where it started. Think: clean ingredients, not the final dish.</p>
</li>
<li><p><strong>Reduces pain later</strong> – Validating and standardizing early keeps things clean downstream—especially when people start building reports on top of it.</p>
</li>
</ul>
<p>If Bronze is your raw transactional system duplicate, <strong>Silver is where the analytic layer starts to take form</strong>. It’s where you bring structure and logic into place.</p>
<p>But here’s the thing—your exec team probably doesn’t care about Silver. They want polished dashboards, KPIs, and numbers that “just make sense.” They don’t want to know how the hotdog is made – they just want the finished product. That’s where <strong>Gold</strong> comes in.</p>
<p><strong>Gold: Business-Ready Insights</strong></p>
<p>The <strong>Gold layer</strong> is your polished, analytics-ready data—this is the stuff your executives, analysts, and self-service heroes actually see. It’s where you take everything you’ve cleaned and modeled in Silver, and serve it up in a way that’s fast, clear, and business-friendly.</p>
<p>In Fabric, this usually means:</p>
<ul>
<li><p>Loading into <strong>Warehouse tables</strong></p>
</li>
<li><p>Building <strong>views with clean, readable column names</strong></p>
</li>
<li><p>Designing <strong>semantic models</strong> in Power BI</p>
</li>
<li><p>Applying <strong>row-level security</strong> to make sure the right people see the right numbers (and only those numbers)</p>
</li>
</ul>
<p>Unlike the granular detail in Bronze or Silver, <strong>Gold is typically aggregated</strong>—daily, weekly, monthly—whatever fits the business question.</p>
<p>This data is optimized for reporting and querying. Think that you’re serving it on a platter for your end users to consume. Theres no additional work for them to be done. You can attach AI models to it, create suites of reporting, even give access to power users for reporting.</p>
<p>Because the gold layer models a business domain, some customers create multiple gold layers to meet different business needs, such as HR, finance, and operations.</p>
<p>Use cases:</p>
<ul>
<li><p>Semantic Models</p>
</li>
<li><p>Reports &amp; Dashboards</p>
</li>
<li><p>LLM Built on Clean Datasets</p>
</li>
</ul>
<p><strong>Final Thoughts (and a Quick Reality Check)</strong></p>
<p>No architecture is a silver bullet. You still need good data practices, solid governance, and a team that understands the business. But Medallion gives you a head start. It gives your data a home, a purpose, and a path forward.</p>
]]></content:encoded></item></channel></rss>