Data Engineer 4

Join our dynamic team at our energy partner as a Data Engineer 4, where you will play a pivotal role in harnessing the power of data to optimize operations, enhance decision-making processes, and drive innovation in the rapidly evolving energy sector. As a seasoned professional, you will leverage your expertise to design, implement, and maintain robust data pipelines, ensuring the seamless flow of information critical to powering the future of sustainable energy solutions

 

Core Technical Skills

·        3+ years of AWS experience

·        Hands-on Experience with AWS services - S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, Step Functions, Opensearch

·        Extensive Experience with EMR databases such as Glue Catalog, Lake Formation, Redshift, DynamoDB and Aurora

·        Required Tools and Languages – Python, Spark, PySpark and Pandas

·        Infrastructure as Code technology- specifically Terraform

·        Experience with Secrets Management Platform like Vault and AWS Secrets manager

·        Experience with DevOps pipeline (CI/CD) - Bitbucket; Concourse, Github, Jenkins

·        Experience with RDBMS platforms and Strong proficiency with SQL

·        Deep knowledge of IAM roles and Policies

·        Experience using AWS monitoring services like CloudWatch, CloudTrail ad CloudWatch events

·        Experience with AWS workflow orchestration tools like Airflow (MWAA), EventBridge, or Step Functions

·        Experience with Kafka/Messaging preferably Confluent Kafka

·        Experience with Event Driven Architecture

·        AWS Cloud Engineer Additional Technical Skills (nice to have, but not required for the role)

·        Experience with native AWS technologies for data and analytics such as Kinesis

·        Databases - Document DB, Mongo DB

·        Hadoop platform (Hive; HBase; Druid)

·        Java, Scala, Node JS

·        Workflow Automation

·        Experience transitioning on premise big data platforms into cloud-based platforms such as AWS

·        Strong Background in Kubernetes, Distributed Systems, Microservice architecture and containers

·        Experience with Rest APIs and API gateway

·        Deep understanding of networking DNS, TCP/IP and VPN

 

Core Responsibilities

·        Provides technical direction, guides the team on key technical aspects and responsible for product tech delivery

·        Lead the Design, Build, Test and Deployment of components

·        Where applicable in collaboration with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead)

·        Understand requirements / use case to outline technical scope and lead delivery of technical solution

·        Confirm required developers and skillsets specific to product

·        Provides leadership, direction, peer review and accountability to developers on the product (key responsibility)

·        Works closely with the Product Owner to align on delivery goals and timing

·        Assists Product Owner with prioritizing and managing team backlog

·        Collaborates with Data and Solution architects on key technical decisions

·        The architecture and design to deliver the requirements and functionality

 

Core Experience and Abilities

·        Ability to perform hands on development and peer review for certain components / tech stack on the product

·        Standing up of development instances and migration path (with required security, access/roles)

·        Develop components and related processes (e.g. data pipelines and associated ETL processes, workflows)

·        Lead implementation of integrated data quality framework

·        Ensures optimal framework design and load testing scope to optimize performance (specifically for Big Data)

·        Supports data scientist with test and validation of models

·        Performs impact analysis and identifies risk to design changes

·        Ability to build new data pipelines, identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications

·        Ensures Test Driven development

·        3+ years of Experience leading teams to deliver complex products

·        Strong technical skills and communication skills

·        Strong skills with business stakeholder interactions

·        Strong solutioning and architecture skills

·        5+ years of Experience building real time data ingestion streams (event driven)

·        Ensure data security and permissions solutions, including data encryption, user access controls and logging