☄️ Python's nested data operator (and CLI), for all your declarative restructuring needs. Got data? Glom it! ☄️
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Deterministic computation ledger for agentic data work.
O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian
🤖 An automated machine learning framework for audio, text, image, video, or .CSV files (50+ featurizers and 15+ model trainers). Python 3.6 required.
Python package implementing ML feature engineering and pre-processing for polars or pandas dataframes.
Agentic & No-Code Data Transformations
SQL transformation framework with custom compile-time checks, end-to-end testing and safe renames that keep your table history.
Open-source, local-first visual ETL and data pipeline builder. Drag-and-drop data prep and ML workflows, preview every step, schedule runs, and export clean pandas or Polars Python code. Self-hosted, no lock-in.
A tool to read CSV files with CSVW metadata and transform them into other formats.
In this project, I loaded the FSI 2020 and 2021 Excel datasets using Pandas. I selected the required columns, merged important data, renamed the columns, and saved the processed files as pickle files. Finally, I combined both datasets, created a line chart with Matplotlib, customized the plot, and automatically exported the figure as a PNG image.
Easy ETL
DEPRECATED: YAML-based data transformations
An open-source framework for automated sensitive data classification and adaptive privacy-preserving transformations in data pipelines.
This repository is a working ETL framework which utilizes user data from Spotify API using ➲Python for Extraction and Transformation ➲SQL for Data Loading and Staging ➲Airflow for Data Orchestration and Monitoring ➲PowerBI for Reporting
Reusable Python classes that extend open source PySpark capabilities. Examples of implementation is available under notebooks of repo https://github.com/bennyaustin/synapse-dataplatform
[VLDB'25] Official repo for Paper "Weak-to-Strong Prompts with Lightweight-to-Powerful LLMs for High-Accuracy, Low-Cost, and Explainable Data Transformation"
GUI and library made to flatten HUGE JSON files. A library and utility for exploring, analyzing, and flattening JSON files of any size (LARGE - GBs) into CSVs, along with CSV transformations, dynamic CSV filtering, and all with low memory utilization.
fast-resource is a data transformation layer that sits between the database and the application's users, enabling quick data retrieval. It further enhances performance by caching data using Redis and Memcached.
A Holistic Platform for Automating Data Preparation
To associate your repository with the data-transformation topic, visit your repo's landing page and select "manage topics."