SUMMARY:
Our client in the Telecommunications sector is looking for an experienced Apache Airflow. This is a contract role for Three months.
POSITION INFO:
- Description: This is a 3-month remote engagement working for a large telco. They will be working with a team supporting a Big Data Platform consisting of an ever-growing list of Airflow DAGs and associated HQL jobs. Apache Airflow is part of a much larger Cloudera implementation.
- They will be expected to help stabilise existing Apache Airflow DAGs, to ensure predictable and stable execution of the existing jobs.
- They will be joining the core platform team as a lead Airflow administrator, and developer and assisting Operations to effectively monitor Apache Airflow.
- Additionally they will be expected to be ensure that act as a mentor to at least two more junior consultants in order to advance their understanding and competence with Apache Airflow.
- Requirements:
- Design, develop, and maintain data pipelines using Apache Airflow to ensure data reliability and availability
- Proficiency in Python, and PySpark to support optimal DAG job design and development
- Experience with Apache Airflow setup and configuration, including Installation and Configuration, Database Management,
- Experience with monitoring, resource management and integration with third-party APM tools.
- In-depth understanding of troubleshooting, stabilising and fine-tuning Apache Airflow
- Experience with integrating Apache Airflow with the Application Performance Monitoring (APM) software.
- Experience with Apache Hive databases and SQL (Oracle and MySQL)
- Nice-to-haves
- In-depth experience and understanding of the Cloudera Data Platform.
NB! This job is now closed. You can apply for other jobs by uploading your CV.