Senior Data Engineer - Databricks

Data Pipeline Development & Operations
. Design, build, and operate scalable and reliable data pipelines on the Databricks platform
. Develop end-to-end data workflows from ingestion through transformation to consumption
. Implement robust error handling, monitoring, and alerting mechanisms
. Ensure data pipeline reliability, performance, and maintainability
. Optimize pipeline performance through efficient Spark job design and cluster configuration
. Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization
. Refactor legacy code and data pipelines to PySpark for improved performance and scalability
. Migrate traditional ETL processes to modern ELT patterns on Databricks
. Assess existing codebases and identify opportunities for optimization and modernization
. Ensure backward compatibility and data integrity during migration processes
. Document refactoring approaches and crea...