Required stack
필요 기술
PythonPySparkSQLScalaSparkHadoopBigQuerySnowflakeAirflowdbt데이터베이스ETL/ELTJava
Project brief
프로젝트 내용
I’m sitting on a large database that stores semi-structured records, and I need a robust transformation layer that turns this raw content into analysis-ready tables. The data is already captured and stored; the task begins once the records land in the database and ends when the transformed results are written back to a target schema (or files, if that proves more efficient).
Key points you should know
• Source: relational database containing nested JSON / key-value blobs.
• Goal: parse, normalize, and flatten these blobs into well-defined columns while preserving relationships and lineage.
• Scale: millions of rows, so solutions that leverage Spark, Hadoop, BigQuery, Snowflake, or well-tuned SQL/Python pipelines are welcome—as long as they remain maintainable.
Deliverables
1. Transformation code (Python, PySpark, SQL, or Scala) with clear comments.
2. A runnable job definition or workflow file (Airflow DAG, Spark submit script, dbt model, etc.) that shows how to execute the pipeline end-to-end.
3. Simple README explaining prerequisites, run steps, and how new fields should be added in future.
Acceptance criteria
• Pipeline processes at least 10 GB of source data without errors.
• Output tables/files match the target schema I’ll provide and contain no missing or malformed records.
• Execution can be parameterized for date ranges or incremental loads.
If you’ve built similar ETL or ELT jobs against semi-structured data and can demonstrate performance at scale, I’d love to see your approach.
Key points you should know
• Source: relational database containing nested JSON / key-value blobs.
• Goal: parse, normalize, and flatten these blobs into well-defined columns while preserving relationships and lineage.
• Scale: millions of rows, so solutions that leverage Spark, Hadoop, BigQuery, Snowflake, or well-tuned SQL/Python pipelines are welcome—as long as they remain maintainable.
Deliverables
1. Transformation code (Python, PySpark, SQL, or Scala) with clear comments.
2. A runnable job definition or workflow file (Airflow DAG, Spark submit script, dbt model, etc.) that shows how to execute the pipeline end-to-end.
3. Simple README explaining prerequisites, run steps, and how new fields should be added in future.
Acceptance criteria
• Pipeline processes at least 10 GB of source data without errors.
• Output tables/files match the target schema I’ll provide and contain no missing or malformed records.
• Execution can be parameterized for date ranges or incremental loads.
If you’ve built similar ETL or ELT jobs against semi-structured data and can demonstrate performance at scale, I’d love to see your approach.