Senior Data Engineer
Ivy Partners • Porto
Tempo inteiro
Outros
Salário não
especificado
Descrição da Vaga
**Our mission**
Ivy Partners is a Swiss consulting firm that supports businesses in their strategic, technological, and organizational challenges.
We are committed to providing our team members with a career that fosters both personal and professional growth. We support their skill development and offer real opportunities for advancement.
**As a Senior Data Engineer at Ivy Partners, you will:**
* Advise on data strategy, architecture, and implementation approaches within a cloud\-first environment.
* Design and develop data pipelines on public cloud platforms, primarily AWS, with exposure to Azure/GCP where relevant.
* Implement and optimize Lakehouse architectures, focusing primarily on Apache Iceberg (experience with Hudi and Delta is a plus).
* Build, automate, and maintain ETL processes, data workflows, and orchestration pipelines using tools like Airflow, Step Functions, and AWS Workflow.
* Ensure high standards of reliability, automation, maintainability, and security across all delivered solutions.
* Support incident resolution, root cause analysis, and preventive measures as required.
* Share knowledge and best practices with internal teams to enhance engineering capabilities.
* Produce comprehensive documentation covering designs, implementations, and operational guidelines.
* Participate in Agile ceremonies, technical reviews, and knowledge exchange sessions.
**About You**
**You are our ideal candidate if you:**
* Have 3 to 5 years of experience building data pipelines on public cloud services, preferably AWS (Azure/GCP also considered).
* Possess at least 1 year of experience in data solution architecture, design, development, and support.
* Are experienced in database architecture, ETL, BI, Big Data, ML, and Advanced Analytics.
* Have hands\-on experience with Spark ecosystems (experience with Hadoop is a plus).
* Are proficient with pipeline orchestration tools such as Airflow, Step Functions, and AWS Workflow.
* Have demonstrated experience with Lakehouse architectures (Apache Iceberg) and table optimizations like compaction and Z\-Order.
* Have software development skills in Python, SQL, and SQL Server; familiarity with SSIS packages and Docker is beneficial.
* Are experienced with AWS services including Lambda, Glue, Crawlers, Step Functions, AWS Workflow, DynamoDB, and AWS Batch.
* Have experience with DevOps and Agile practices, infrastructure as code (CloudFormation, Terraform), and artifact/vulnerability management.
* Understand cloud foundation services: networking, security, storage, and IAM.
* Preferably have experience with container\-based applications (Docker) and Linux\-based infrastructure.
* Are fluent in English (written and spoken); French is a plus.
**Why Join Ivy Partners?**
* **Care** \| We care about our employees by creating a supportive environment where everyone is valued and empowered with training and growth prospects.
* **Trust** \| We believe in building strong and long\-lasting relationships with our clients. Working with us is about establishing a partnership based on trust, professionalism, and transparency at all times.
* **Innovation** \| We empower effective digital transformation through a unique blend of innovative technology and creative thinking.
* **Responsibility** \| Community is at the heart of everything we do, and we take that responsibility seriously by working hard to create a positive impact.