Data Engineer | Cape Town

Data Engineer

Contract or Permanent

Cape Town

Purpose of the Job:

As a Data Engineer, you will build and support data pipelines and datamarts built off those pipelines. Both must be scalable, repeatable and secure. The Data Engineer helps to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously.

This role functions as a core member of an agile team.  You will also be responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly.

You will enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.

Additional responsibilities include developing proof of concepts and implements complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets.  We need someone who knows how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions.

The Data Engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields.

You will need to have knowledge of data analysis, end user requirements and business requirements analysis to develop a clear understanding of the business need and to incorporate these needs into a technical solution.

You will need to have a solid understanding of physical database design and the systems development lifecycle.

Job Objectives:

• Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment.
• Design and develop programmatic transformations of the solution, by correctly partitioning, formatting and validating the data quality.
• Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the data lake.
• Provide operational support to DataMart data feeds and DataMarts Design infrastructure required to develop and operate data lake data feeds.
• Design infrastructure required to develop and operate DataMart, their user interfaces and the feeds required to populate the data lake.

Requirements

• AWS is essential
• 3 - 5 years Business Intelligence
• 3 - 5 years in ETL processes
• Talend (1 year)
• Python (1 year)
• Business Intelligence Data modelling (3 years)
• SQL (3 years)

You will need Knowledge of:

• Creating data feeds from on-premise to AWS Cloud (1 year)
• Support data feeds in production on break fix basis (1 year)
• Creating data marts using Talend or similar ETL development tool (2 years)
• Manipulating data using python and pyspark (1 year)
• Processing data using the Hadoop paradigm particularly using EMR, AWSs distribution of Hadoop (1 year)
• Devop for Big Data and Business Intelligence including automated testing and deployment (1 year)

Desirable:

• AWS: EMR, EC2, S3 (1 year)
• PySpark or Spark (1 year)
• Big Data (1+ years) Cloud AWS (1+ years)