As a Data Engineer you are expected to contribute towards
Create and maintain optimal data pipeline architecture,
Assemble large data sets that meet functional / non-functional business requirements.
Design and implement internal process improvements: automating manual processes, and optimizing data delivery etc.
Help build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python, SQL and GCP ‘big data’ technologies.
Work with stakeholders including Product, Data Integration and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Create data tools for BI reporting, digital analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
Work with data and analytics experts to strive for greater functionality in our data systems.
The below Key Performance Areas include but are not limited to:
Takes a breadth first approach to problem solving and encourages systematic thinking.
Good communicator and collaborator, able to simplify concepts for wider non-tech audiences.
Able to demonstrate a passion for data engineering with the ability to work cross functional.
Culturally sensitive and able to work across cultures and time zones.
Experience
Experience of designing, building and maintaining scalable ETL data processing systems using Python, Java, Scala or similar programming language – this is a hands-on coding role.
Understanding and experience in data processing techniques and analytics with Strong SQL experience.
Understanding and experience of GCP data tools (GBQ, Dataflow, DataProc, GC Data Transfer etc.)
Understanding and experience of using and creating CICD pipelines using Azure Devops, Jenkins, GoCD or any similar technologies
Understanding and experience of using orchestration tools such as Apache Airflow, Azure Datafactory, AWS Data pipelines or similar
Understanding and experience of working with Cloud Deployment Environments (GCP or AWS preferred)
Proponent for software craftmanship, TDD and automated testing
DevOps background with understanding of cloud-based IAC technologies
Terraform or serverless
Github as version control
Familiar with CI/CD
Uunderstanding of Data Modelling and/or Data Warehousing techniques such as star schema or raw data vault
Experience of working with large-scale data sets, this includes both structured and unstructured data such as parquet, avro, json etc.
Advertised: India Standard Time Application close: India Standard Time