senior-data-engineer

Topics in the source

• Senior Data Engineer
• Table of Contents
• Trigger Phrases
• Quick Start
• Core Tools
• Workflows
• Architecture Decision Framework
• Batch vs Streaming

Production-grade data engineering skill for building scalable, reliable data systems. 1. Trigger Phrases 2. Quick Start 3. Workflows 4. Architecture Decision Framework 5. Tech Stack 6. Reference Documentation 7. Troubleshooting -- Activate this skill when you see: Pipeline Design: "Design a data pipeline for..." "Build an ETL/ELT process..." "How should I ingest data from..." "Set up data extraction from..." Architecture: "Should I use batch or streaming?" "Lambda vs Kappa architecture" "How to handle late-arriving data" "Design a data lakehouse" Data Modeling: "Create a dimensional model..." "Star schema vs snowflake" "Implement slowly changing dimensions" "Design a data vault" Data Quality: "Add data validation to..." "Set up data quality checks" "Monitor data freshness" "Implement data contracts" Performance: "Optimize this Spark job" "Query is running slow" "Reduce pipeline execution time" "Tune Airflow DAG" -- -- → See references/workflows.md for details Use this framework to choose the right approach for your data pipeline. | Criteria | Batch | Streaming | |----------|-------|-----------| | Latency requirement | Hours to days | Seconds to minutes | | Data volume | Large histo

This is an excerpt. Refer to the complete source and its current requirements.