SUMMARY:
Our client is looking for a Data Engineer/Scientist (Advanced) with Bachelor’s or Master’s degree in Computer Science, Software Engineering. minimum of 3 years’ experience as a Data Engineer, 2 years of experience working with AWS services must be based in Gauteng, contract December 2027
POSITION INFO:
Essential skills requirements:
- Proficiency in Python 3.x for data processing and automation.
- Experience with AWS Glue for ETL processes.
- Strong knowledge of AWS Athena for querying large datasets.
- Hands-on experience with AWS Lambda for serverless computing.
- Familiarity with AWS EC2 for scalable computing resources.
- Expertise in AWS CloudWatch for monitoring and logging.
- Proficiency in working with PostgreSQL RDS for database management.
- Experience with AWS QuickSight for data visualization and reporting.
- Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).
- Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines.
- Any additional responsibilities assigned in the Agile Working Model (AWM) Charter
Advantageous skills requirements:
- Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimizing AWS resources.
- Experience with other AWS services such as S3, Kinesis, and DynamoDB.
- Knowledge of data formats such as Parquet, AVRO, JSON, and CSV.
- Experience with Docker for containerization.
- Understanding of Big Data technologies and frameworks.
- Familiarity with Agile working models and tools like JIRA and Confluence.
- Experience with data quality tools such as Great Expectations.
- Knowledge of REST API development and integration.
- Strong analytical skills for troubleshooting and optimizing data pipelines.
- Experience in developing technical documentation and artefacts.
What will be your role and responsibilities?
- Design, develop, and maintain data ingestion pipelines for BMW’s connected fleet, focusing on Call Detail Records (CDRs).
- Utilize AWS Glue for ETL processes to transform and load data into the analytics platform.
- Implement efficient querying solutions using AWS Athena.
- Develop serverless applications and workflows using AWS Lambda.
- Monitor and optimize data pipelines using AWS CloudWatch.
- Manage and maintain PostgreSQL RDS databases for data storage and retrieval.
- Create interactive dashboards and reports using AWS QuickSight.
- Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.
- Use Git and GitHub for version control, collaboration, and maintaining code repositories.
- Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes.
- Ensure data security and compliance with BMW’s information classification requirements.
- Stay updated with the latest data engineering tools, technologies, and industry trends.
- Identify opportunities for process improvements and automation to enhance pipeline efficiency.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Develop and maintain technical documentation for data engineering processes and solutions.
Apply now!!! send CV to
