data-engineering-master
From the author
数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schem
Install
npx skills add swaylq/master-skill --skill data-engineering-master -a claude-code
Installs to .claude/skills/data-engineering-master/ · Each command fetches the skill straight from GitHub with the open skills CLI.
Byte-identical to the scanned commit, with the repository’s licence and a SOURCE.txt.
Want your agent to suggest skills while you type? See how to use skills from your IDE.
Safety checks on SKILL.md
- Ships scripts we have not reviewedLow
Why 76
aiqbench computes the trust score from public signals. It is not a security audit, so read a skill before you install it.
Similar skills
Closest to this one by what they do and when they are used.
- data-engineeringcbrock84/headcountBuilds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.Data★ 2.5K
- fluxtonone-ai/tononeData engineer — databases, migrations, pipelines, schema design, and query optimization. Use when asked to "design a schema", "write a migration", "build a data pipeline", or "optimize this query".Data★ 77
- engineering-data-pipelinestelagod/code-abyssData engineering knowledge reference covering Airflow, Dagster, Kafka Streams, Flink, dbt, and data quality patterns. Use when building data pipelines, ETL workflows, stream processing, or data quality checks.Data★ 244
- data-engineersaeed-vayghan/gemini-agent-skillsExpert data engineer specializing in building scalable data pipelines, ETL/ELT processes, and data infrastructure. Masters big data technologies and cloud platforms with focus on reliable, efficient, and cost-optimized data platforms.DevOps★ 51
- tool.data_engineering.134100a5a52f2e93AI45Lab/OpenARTData engineering practice patterns for data pipelines, ETL/ELT, orchestration, data quality, and infrastructure. Covers Airflow, Dagster, Prefect, Spark, dbt, data lakes, lakehouses, Iceberg, Delta Lake, warehouses (Snowflake, BigQuery, Databricks), streaming (Kafka, Kinesis, Flink), CDC, data contracts, data governance, data mesh, and batch processing. Use when reviewing or building data pipelines, ingestion systems, stream processing, data infrastructure, or data platform architecture. Do not use for SQL modeling or dbt projects (use analytics-engineering), business dashboards (use analytics), or ML model training (use data-science).Data★ 263
- tool.data_engineering.55c8f35d7b3cac72AI45Lab/OpenARTData engineering practice patterns for data pipelines, ETL/ELT, orchestration, data quality, and infrastructure. Covers Airflow, Dagster, Prefect, Spark, dbt, data lakes, lakehouses, Iceberg, Delta Lake, warehouses (Snowflake, BigQuery, Databricks), streaming (Kafka, Kinesis, Flink), CDC, data contracts, data governance, data mesh, and batch processing. Use when reviewing or building data pipelines, ingestion systems, stream processing, data infrastructure, or data platform architecture. Do not use for SQL modeling or dbt projects (use analytics-engineering), business dashboards (use analytics), or ML model training (use data-science).Data★ 263
Ratings & reviews
—
no reviews
Tried this skill? Rate it to help others. Sign in
Discussion
Join the discussion. Sign in