3. Big Data Concepts

This is the archive of concepts in Big Data. It consists of the notes of personal learning and taking Udemy courses: The Complete dbt (Data Build Tool) Bootcamp: Zero to Hero by Zoltan C. Toth and Miklos (Mike) Petridisz and Big Data Engineering Bootcamp with GCP, and Azure Cloud by Krish Naik and Mayank Aggarwal.

Big Data 6. Slowly Changing Dimensions (SCD)

Slowly Changing Dimensions (SCD) Slowly Changing Dimensions (SCDs) refer to attributes in a data warehouse dimension table that change infrequently—yet unpredictably—over time. When a source system updates a dimension attribute, we need a strategy to decide how that change will be reflected (or not) in the data warehouse. In some cases, storing historical values isn’t necessary; in others, retaining a full history is critical for reporting, auditing, or trend analysis. ...

Big Data 3. Data Warehouses, Data Lakes, and Lake Houses

1. Data Warehouses A Data Warehouse (DW) is a centralized repository that aggregates data from multiple sources into a unified, consistent store optimized for analytics and business intelligence (BI) activities, such as reporting, forecasting, and data mining. Unlike transactional databases (OLTP), data warehouses are built for analytical workloads (OLAP): they support complex queries over large volumes of data, often using denormalized schemas, dimensions, and facts for performance. Data warehouses typically use: ...

Big Data 4. Denormalized Schema in Data Warehouses

1. Denormalized Schema A denormalized schema is a database design that reduces the number of joins by combining related data into fewer tables, often at the expense of some redundancy. Optimized for read-heavy analytical workloads Not ideal for OLTP systems (which benefit from normalization) In a typical data warehouse: Data is modeled to maximize query performance and simplify analytics. This contrasts with normalized schemas used in transactional databases, where the goal is data integrity and avoiding redundancy. 2. Star and Snowflake Schemas Two common denormalized designs are: ...

Big Data 5. Modern Data Stack

As the cost of cloud storage and compute continues to fall and internet speeds rise, data engineers are rethinking how data is stored, processed, and analyzed. This shift has led to what we now call the modern data stack: a flexible, scalable, cloud-native approach that replaces traditional monolithic systems. In this post, we’ll explore key architectural changes that enabled the modern data stack, from the transition from SMP to MPP, to the decoupling of storage and compute, and the rise of columnar databases, ELT workflows, and the tooling ecosystem that glues it all together. ...

Big Data 1. Data Maturity Model

1. Intro The Maslow’s hierarchy of needs models the basic needs of humans as a hierarchy of 5 levels from the bottom to higher-level needs at the top. In this model, one cannot proceed to the next level unless the basic needs of the current level is met. Silmiarly, one can think of data hierarchy of needs. The 5 levels are Data Collection, Data Wrangling, Data Integration, BI and Analytics, and Artificail Intellgence. ...

Big Data 2. OLTP vs OLAP

When working with databases, it’s essential to distinguish between two fundamental system types: OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). These systems serve very different purposes and are optimized accordingly. OLTP — Online Transaction Processing OLTP systems are designed to handle a large number of short, atomic transactions such as inserts, updates, and deletes. They are optimized for speed and reliability in transactional environments. Example Use Cases Banking systems E-commerce checkout Inventory management Booking/reservation systems Characteristics Feature OLTP Main Use Day-to-day operations (CRUD) Queries Simple and frequent Data Volume Current data (live, real-time) Users Many concurrent users Schema Highly normalized Transactions Short and fast Performance Goal High throughput and low latency Storage Type Primarily Row-based ACID Compliance Fully ACID-compliant What is ACID? ...