Position Description

Data Engineer 4
Location Nationwide (Remote)
NetSuite Employee Status Contractor
Apply Now

Embark on a transformative career at a leading energy producer as a Data Engineer, where innovation and energy converge to shape the future. Join a dynamic team at the forefront of revolutionizing the energy sector, leveraging cutting-edge data technologies to drive sustainable solutions and power a brighter, more connected tomorrow!

Feel free to reach out to michaelbenson@kcillc.com with any questions or apply directly. No C2C. W2 Only.

Core Technical Skills

  • 3+ years of AWS experience
  • Hands-on Experience with AWS services - S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, Step Functions, Opensearch
  • Extensive Experience with EMR databases such as Glue Catalog, Lake Formation, Redshift, DynamoDB and Aurora
  • Required Tools and Languages – Python, Spark, PySpark and Pandas
  • Infrastructure as Code technology- specifically Terraform
  • Experience with Secrets Management Platform like Vault and AWS Secrets manager
  • Experience with DevOps pipeline (CI/CD) - Bitbucket; Concourse
  • Experience with RDBMS platforms and Strong proficiency with SQL
  • Deep knowledge of IAM roles and Policies
  • Experience using AWS monitoring services like CloudWatch, CloudTrail ad CloudWatch events
  • Experience with AWS workflow orchestration tools like Airflow (MWAA), EventBridge, or Step Functions
  • Experience with Kafka/Messaging preferably Confluent Kafka
  • Experience with Event Driven Architecture

 

  • AWS Cloud Engineer Additional Technical Skills (nice to have, but not required for the role)
  • Experience with native AWS technologies for data and analytics such as Kinesis
  • Databases - Document DB, Mongo DB
  • Hadoop platform (Hive; HBase; Druid)
  • Java, Scala, Node JS
  • Workflow Automation
  • Experience transitioning on premise big data platforms into cloud-based platforms such as AWS
  • Strong Background in Kubernetes, Distributed Systems, Microservice architecture and containers
  • Experience with Rest APIs and API gateway
  • Deep understanding of networking DNS, TCP/IP and VPN

 

Core Responsibilities

  • Provides technical direction, guides the team on key technical aspects and responsible for product tech delivery
  • Lead the Design, Build, Test and Deployment of components
  • Where applicable in collaboration with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead)
  • Understand requirements / use case to outline technical scope and lead delivery of technical solution
  • Confirm required developers and skillsets specific to product
  • Provides leadership, direction, peer review and accountability to developers on the product (key responsibility)
  • Works closely with the Product Owner to align on delivery goals and timing
  • Assists Product Owner with prioritizing and managing team backlog
  • Collaborates with Data and Solution architects on key technical decisions
  • The architecture and design to deliver the requirements and functionality

 

Core Experience and Abilities

  • Ability to perform hands on development and peer review for certain components / tech stack on the product
  • Standing up of development instances and migration path (with required security, access/roles)
  • Develop components and related processes (e.g. data pipelines and associated ETL processes, workflows)
  • Lead implementation of integrated data quality framework
  • Ensures optimal framework design and load testing scope to optimize performance (specifically for Big Data)
  • Supports data scientist with test and validation of models
  • Performs impact analysis and identifies risk to design changes
  • Ability to build new data pipelines, identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications
  • Ensures Test Driven development
  • 3+ years of Experience leading teams to deliver complex products
  • Strong technical skills and communication skills
  • Strong skills with business stakeholder interactions
  • Strong solutioning and architecture skills
  • 5+ years of Experience building real time data ingestion streams (event driven)
  • Ensure data security and permissions solutions, including data encryption, user access controls and logging
Back Apply Now