incremental-processing

Incremental processing and maintaining data freshness with CocoIndex and LanceDB

A modern banking data pipeline built with Dagster and DBT!

Monitor AI agent memory, skills, and behavior in a live terminal HUD for Hermes.

Reusable data matching system with incremental processing for efficient reuse of historical results.

Production-style lakehouse pipeline demonstrating incremental ingestion, deterministic keys, idempotent merges, auditing, and restart-safe processing.

Production-style Enterprise Sales Lakehouse using PySpark, Delta Lake and Medallion Architecture with incremental processing, data quality, monitoring and business analytics.

A production-grade cryptocurrency data pipeline built on GCP that ingests real-time market data from the CoinGecko API, implements Medallion architecture (Raw → Staging → Curated), and supports idempotent backfill and metadata-driven incremental processing for reliable, scalable analytics.

Obsidian 知识工作台：增量整理新增资料与对话续聊，支持主题治理、证据检索、每周归档和换机迁移。

Incremental DNS telemetry with transactional SQLite rollups, atomic checkpoints, fair cached summaries, and reproducible benchmarks.

Reproducible lakehouse benchmark lab exploring Parquet file layouts, incremental processing, and Apache Iceberg snapshots.

End-to-end data pipeline that ingests job market data from the Adzuna API and processes it using a Medallion Architecture (Bronze→Silver→Gold). Includes data quality validation, incremental loading into DuckDB, and Airflow orchestration. Generates analytics-ready datasets for role demand, salary trends, and skill demand across multiple countries.

Databricks Lakehouse + Apache Spark data engineering: PySpark, Spark SQL, Delta Lake, Bronze→Silver→Gold, CDC, SCD2, DQ, monitoring, reliability and performance tuning.

End- to-End Performance-optimized sales data pipeline using Medallion Architecture with broadcast joins, fact/dimension modeling, Autoloader & incremental processing

Versioned document ingestion, retrieval and an incremental event mart on SQLite, with optional Azure Cosmos DB vector search

基于 PySpark 与 Parquet 的共享单车离线湖仓，覆盖 Bronze/Silver/Gold 分层、SCD2、增量回填、数据质量、小文件治理和 Airflow 编排。