6.1 Columnar Layout, Parquet, and Encoding

Row and column layouts determine which bytes a query reads and decodes. An analytical query may need two columns while row pages carry many unrelated fields. A columnar layout permits separate column reads and encodes similar values together. This lesson follows the file structure to identify those savings without counting unread columns as compressed storage.

Learning Objectives

• Compare row-oriented and column-oriented data layouts;
• Explain the components of Parquet files: row groups, column chunks, and pages;
• Distinguish between encoding and compression;
• Understand the appropriate use cases for dictionary encoding, RLE/bit-packing, and delta encoding;
• Explain why nested data structures require definition and repetition levels.

Advantages and Costs of Row Layout

A row-based storage layout groups attributes of a single row together in close proximity:

[id=1, type=device, value=120000, name=...]
[id=2, type=spare_part,  value=95000,  name=...]

This format is well-suited for:

• Primary-key lookups that return most of a row;
• Inserting or updating an entire row;
• Transactional or MVCC systems that use row versions as their core unit;
• Narrow, operational queries.

Analyzing a query:

SELECT item_type_id, SUM(value_cents)
FROM catalog_items
WHERE value_cents >= 50000
GROUP BY item_type_id;

When scanning row stores, the engine typically must read entire pages containing full rows, even if the query only projects two columns. However, it's inaccurate to claim that "all 20 additional columns are necessarily decoded into objects": the engine can selectively parse only the required fields, and out-of-line values may be skipped entirely. The primary performance overhead lies in page bytes and cache density, not in the full materialization of every column.

Index-only scans, vertical partitioning, and columnar extensions can alter this trade-off.

Column layout

Columnar organization stores columns separately within a batch of rows:

item_id:      [1, 2, 3, ...]
item_type_id: [1, 2, 1, ...]
value_cents:  [120000, 95000, 180000, ...]
name:         [...]

Advantages:

• Projections only read the necessary column chunks;
• Values of the same type or semantics are adjacent, enabling specialized encoding and compression;
• Tight arrays are well-suited for SIMD and vectorized operators;
• Metadata such as min/max, dictionaries, and Bloom filters can support skipping.

Disadvantages:

• Reconstructing wide rows requires aligning multiple column vectors;
• Small, random row updates and version management become more complex;
• Frequent small writes often require delta storage, buffering, or immutable segments;
• Selective lookups require index or primary-key structures to locate granules.

Modern systems often combine layouts: row-oriented write buffers with columnar main storage, columnar segments with delete bitmaps, OLTP primary stores with analytical replicas. Don't judge the actual architecture solely based on product labels.

Parquet file layout

Parquet is a self-describing columnar file format, not a database engine. Conceptual structure:

PAR1
Row Group 0:
  Column Chunk A -> data/dictionary pages
  Column Chunk B -> data/dictionary pages
Row Group 1:
  Column Chunk A -> ...
  Column Chunk B -> ...
File Metadata
metadata length
PAR1

Row group

A horizontal partition of a batch of rows. Each row group contains column chunks for each column in those rows. The size of a row group affects:

• parallel scheduling;
• granularity of metadata pruning;
• scope of compression and dictionary encoding;
• memory required for writing and reading;
• pattern of object store requests.

There is no universally correct "128 MB to 1 GB" value across workloads. The optimal size must be determined by considering filesystem or object store size, query selectivity, column width, and writer memory.

Column chunk

The data for a specific leaf column within a row group, composed of contiguous pages. The file metadata records the locations and statistics of column chunks, allowing readers to first access the footer and metadata before selecting the required chunks.

Page

The encoding and compression unit within a column chunk. If present, both dictionary pages and data pages carry their own headers. An optional page index can assist readers in skipping pages; without a page index or when the reader does not use it, page-level pruning cannot be guaranteed.

The claim that "a page is the smallest physical I/O unit" is overly absolute. Readers or object stores can merge reads of multiple pages over byte ranges, and actual I/O behavior is also influenced by compression blocks and filesystem blocks.

Official format: Apache Parquet File Format.

Encoding and Compression Are Different

• Encoding uses value structure to convert logical values into more compact or manageable representations;
• Compression codecs further reduce the size of already-encoded bytes, for example using Snappy or ZSTD;
• Encryption and checksums are separate concerns with distinct responsibilities.

Within the same column chunk, a portion may be dictionary-encoded while another portion uses a different encoding scheme, depending on the writer and format rules.

Dictionary encoding

dictionary: 0=device, 1=spare_part, 2=consumable
ids:        0, 0, 1, 0, 2, 1, ...

Effective for columns with low to medium cardinality or high repetition. The benefit depends on dictionary size, ID bit width, cache efficiency, and data distribution. If cardinality becomes too high, dictionary encoding may bloat and the writer may fall back to plain encoding.

It's incorrect to say "Parquet always defaults to dictionary encoding for strings" or to provide a fixed threshold across implementations. The choice depends on writer options, version, and page or chunk distribution.

RLE and bit packing

Run-length encoding encodes consecutive identical values as (value, run_length):

1,1,1,1,2,2,3 -> (1×4),(2×2),(3×1)

If values do not form runs, simple RLE can actually increase size. Parquet's RLE/bit-packed hybrid represents small integers or dictionary IDs as runs and packed groups. It therefore differs from textbook RLE that only compresses repeated values.

Sorting or clustering can increase runs and reduce min/max ranges, but it also impacts ingestion, other query dimensions, and distributed skew.

Delta encoding

Monotonic or locally correlated integers:

100, 103, 108, 115
base=100, deltas=3,5,7

Small deltas can be encoded with fewer bits or varints. Timestamps and sorted IDs often benefit significantly; random values or sudden jumps see limited gains. Strings can also be delta-encoded using byte-array or prefix encoding.

NULL and Nested Data

Columnar files must preserve row alignment. Formats differ: Arrow memory arrays commonly use validity bitmaps, while Parquet uses definition levels. Both preserve logical row positions without storing a full value for each NULL.

Lists, structs, and maps complicate the relationship between "the i-th column value and the i-th row." Dremel-style repetition and definition levels address:

• Which level of the nested path is defined;
• Whether the value of a repeated field belongs to the current record or to a new parent record.

Schema evolution must also handle field identity, nullability and defaults, type widening, and compatibility between readers and writers. Simply merging nested schemas by field name can lead to errors.

ORC's Similarities and Differences

ORC also employs columnar stripes, streams, indexes/statistics, and footer/postscript components. It shares similar goals with Parquet, but differs in physical encoding, metadata structure, and reader ecosystem. ORC stripe indexes do not imply that Parquet lacks skipping metadata: Parquet also has row-group statistics, optional column and page indexes, and Bloom filters.

When selecting a format, evaluate engine integration, schema evolution, predicate pushdown support, codecs, nested type handling, metadata size, and object store behavior, rather than relying solely on theoretical compression ratios.

How to Report Compression Ratio

encoded_compressed_file_bytes / logical_uncompressed_input_bytes

Under this definition, an assumed 100 MB input written as a 25 MB file gives 0.25; smaller means less storage. Some tools report the inverse, 4:1, so label the direction.

Specify the input representation: baseline compression efficiency differs significantly between CSV text, Arrow arrays, and database rows. Reducing the number of columns read during a query decreases the amount of data fetched, but it does not reduce the total size of a pre-written Parquet file. Therefore, the fact that "18 columns were not read" cannot be counted as a source of storage compression.

A complete report must include the following metrics:

• File size in bytes
• Bytes of selected columns read
• Decompressed data size
• CPU time for encoding and decoding
• Writer memory usage
• Query wall time
• Row-group or page pruning effectiveness

Lesson Summary

The key benefits of columnar layout are reduced column reads, efficient encoding of like-values, and memory access patterns that are friendly to vectorized operations. Parquet packages these capabilities into a file format using row groups, column chunks, pages, and footer metadata. However, skipping and performance characteristics still depend heavily on the writer's layout choices and the reader's implementation. In the next lesson, we move into scan execution, where we distinguish between projection, pruning, predicate evaluation, and late materialization.