Data warehouse / Doris / Flink / Paimon

E-commerce Batch & Streaming
Lakehouse Warehouse

The same e-commerce metrics are produced through a Doris batch warehouse and a Flink SQL streaming chain. Streaming outputs are materialized to both Paimon layers and Doris ADS, then reconciled field by field in Doris.

View Architecture View Reconciliation View GitHub

Why Build Both Batch and Streaming Paths?

Batch processing prioritizes stable recomputation and historical definitions; streaming prioritizes continuous updates and low-latency dashboards. The difficult part is not running two systems—it is keeping business definitions consistent and making the results explainable and reproducibly verifiable.

Metric definitions drift

When batch reports and streaming dashboards are implemented independently, GMV, order counts, and refunds can diverge because of filters or date boundaries.

System responsibilities must be explicit

Doris provides OLAP serving and the unified ADS endpoint; Paimon stores ODS/DWD/DWS/ADS results materialized from the same streaming DAG.

Keep the validation loop reproducible

The implementation stays focused on MySQL, Doris, Kafka, Flink, and Paimon so the local dual-path run and field-level reconciliation remain the primary evidence.

Architecture: Two Implementations, One Metric Contract

The batch path loads a MySQL source into Doris and transforms it through physical layers. The streaming path consumes Kafka events in Flink SQL, expresses DWD → DWS → ADS as temporary views, and uses one Statement Set to write Paimon layers and Doris ADS in parallel. Offline and real-time ADS are then reconciled in Doris.

01

Doris Batch Warehouse

MySQL data is loaded into Doris ODS/DIM and transformed through DWD, DWS, and ADS with Doris SQL.

02

Flink + Paimon Streaming Lakehouse

Kafka events enter a Flink SQL temporary-view DAG; a Statement Set materializes ODS/DWD/DWS/ADS and Doris ADS as parallel sinks.

03

Unified Queries in Doris

Outputs align at dt + recent_days granularity and compare GMV, distinct orders/users, refunds, and average order value field by field.

Architecture for parallel materialization across the batch warehouse and streaming view chain — The diagram separates three boundaries: MySQL-to-Doris batch layers, Kafka-to-Flink logical views with parallel Paimon/Doris sinks, and offline/realtime ADS reconciliation in Doris.

Doris Batch Warehouse Path

MySQL represents the commerce source system. Scripts load ODS and DIM tables into Doris, where SQL standardizes DWD details, aggregates fixed reporting windows in DWS, and produces ADS metrics.

MySQL ecommerce_oltp -> Doris ODS / DIM -> Doris DWD -> Doris DWS -> ads.ads_trade_stats_offline

Doris row counts across ODS, DIM, DWD, DWS, and ADS layers — A single Doris query lists row counts across every warehouse layer, proving that the batch path produces inspectable intermediate datasets rather than only a final ADS table.

Flink + Paimon Streaming Lakehouse Path

Deterministic Kafka events feed the streaming path. Flink SQL temporary views form the DWD, DWS, and ADS logic chain; multiple inserts in one Statement Set materialize Paimon ODS/DWD/DWS/ADS and the final Doris ADS output in parallel. Downstream logic does not read Paimon physical tables layer by layer in this job.

Flink Dashboard showing a running job, nine tasks, the DAG, and operator status — The running Flink job contains nine tasks. The DAG and operator table show the Kafka source, DWD/DWS aggregations, and multiple Paimon/Doris writers; checkpoint and persisted-output evidence is recorded in the local acceptance section.

Metric Contract: Reconciliation Requires Shared Definitions

Both paths output at dt + recent_days granularity. Core metrics include GMV, distinct orders, distinct buyers, refund orders, refund users, and average order value. The business date is fixed at 2026-07-01; recent_days represents deterministic reporting windows, not a Flink Window TVF.

recent_days	GMV	Orders	Buyers	Refunds	Refund Users	Avg. Order Value
1	505.50	5	4	2	2	101.10
7	1078.50	9	6	3	3	119.83
30	1578.50	10	7	4	4	157.85

Unified Doris Query and Field-level Reconciliation

Validation goes beyond checking that both tables contain rows. A single Doris query aligns offline ADS and realtime ADS by dt + recent_days, then compares every metric field. All three fixed windows pass under the same sample data and contract.

Doris Playground reconciliation results showing PASS for 1-, 7-, and 30-day windows — Doris Playground aligns offline and realtime ADS directly; all 1-, 7-, and 30-day windows return PASS. The full contract and 3/3 acceptance summary are shown above and in the local acceptance section.

Engineering Decisions: Fewer Components, Stronger Evidence

Doris as the unified serving layer

Both batch and streaming ADS outputs land in Doris, so BI queries and demonstrations use one OLAP layer and can directly reconcile the two paths.

Logical computation separated from materialization

Temporary views express DWD → DWS → ADS dependencies. Paimon layers and Doris ADS are parallel materialization targets used for inspection, retention, and serving.

Doris Connector instead of a generic JDBC sink

A generic JDBC sink encountered compatibility issues with the Doris MySQL protocol and ON DUPLICATE KEY, so the project uses the Doris Flink Connector and Stream Load.

Deterministic fixtures protect reproducibility

The Kafka topic is reset before sample events are written, preventing duplicated events from inflating streaming GMV and keeping reconciliation stable.

Local Reproduction and Acceptance

The first version runs on Docker Desktop + WSL2. After MySQL, Doris, Kafka, and Flink start, the scripted sequence reproduces the complete flow.

docker compose ps output showing MySQL, Doris, Kafka, and Flink services running — After startup, Doris, MySQL, Kafka, Flink JobManager, and Flink TaskManager are all running.

Fresh clone PASS · 5 services · Flink RUNNING · 8 RUNNING + 1 FINISHED task · checkpoint 1 → 2 with no new failure · Paimon 5 / 5 current-job files · 3 / 3 reconciliation PASS

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-repo.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\run-demo.ps1 -Reset

Current Scope and Extension Path

This version proves the dual-path metric contract and the local reproduction loop without presenting planned enhancements as delivered capabilities.

Current scope

Streaming input uses deterministic Kafka fixtures; MySQL CDC is not yet connected.
PowerShell scripts orchestrate the flow; DolphinScheduler is not yet included.
Paimon uses a local filesystem catalog rather than Hive Metastore or object storage.
Streaming layers are logical temporary views with parallel sinks, not physical Paimon table-to-table reads.
The 1-, 7-, and 30-day metrics use fixed date predicates; Window TVF, out-of-order events, and late data are not implemented.
The dataset is intentionally small and validates definitions and flow integrity rather than performance at scale.

Next extensions

Orchestrate batch initialization, synchronization, and ADS aggregation as a DolphinScheduler DAG.
Add MySQL CDC or Debezium/Flink CDC input.
Add Window TVF plus out-of-order and late-data cases for event-time semantics.
Move end-to-end container acceptance into a repeatable dedicated CI environment.

What this project demonstrates

I can implement one e-commerce metric contract through a Doris batch warehouse and a Flink SQL streaming chain, materialize Paimon layers and Doris ADS in parallel with a Statement Set, reconcile results field by field in Doris, and state the limits of a local MVP precisely.

E-commerce Batch & StreamingLakehouse Warehouse