Cloud AWS Data Engineer

SUMMARY:
Data Engineers build and support data pipelines and datamarts built off those pipelines

POSITION INFO:

Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment.

Design and develop programmatic transformations of the solution, by correctly partitioning,  formatting and validating the data quality.

Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake.

Provide operational support to datamart datafeeds and datamarts.

Design infrastructure required to develop and operate datalake data feeds.

Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.

Qualifications

IT Degree/Diploma (3 years)

AWS Certification at least to associate level

Experience

Business Intelligence (5+ years)

Extract Transform and Load (ETL) processes (5+ years)

Cloud AWS (2+ years)

Agile exposure, Kanban or Scrum (2+ years)

Desirable:

Retail Operations (5+ years)

Big Data (2+ years)

Experience working as a Technical Lead in the space

Knowledge and Skills

Knowledge:

Creating data feeds from on-premise to AWS Cloud (2 years)

Support data feeds in production on break fix basis (2 years)

Creating data marts using Talend or similar ETL development tool (4 years)

Manipulating data using python and pyspark (2 years)

Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop (2 years)

Devop for Big Data and Business Intelligence including automated testing and deployment (2 years)

Skills:

Talend (1 year)

AWS: EMR, EC2, S3 (1 year)

Python (1 year)

Business Intelligence Data modelling (3 years)

SQL (3 years)

Desirable

PySpark or Spark (1 year)

Additional Information for Agencies

NB! This job is now closed. You can apply for other jobs by uploading your CV.