performance and responsiveness. Stay Up to Date with Technology: Keep yourself and the team updated on the latest Python technologies, frameworks, and tools like Apache Spark , Databricks , Apache Pulsar , ApacheAirflow , Temporal , and Apache Flink , sharing knowledge and suggesting improvements. Documentation: Contribute to clear and … or Azure . DevOps Tools: Familiarity with containerization ( Docker ) and infrastructure automation tools like Terraform or Ansible . Real-time Data Streaming: Experience with Apache Pulsar or similar systems for real-time messaging and stream processing is a plus. Data Engineering: Experience with Apache Spark , Databricks , or similar … big data platforms for processing large datasets, building data pipelines, and machine learning workflows. Workflow Orchestration: Familiarity with tools like ApacheAirflow or Temporal for managing workflows and scheduling jobs in distributed systems. Stream Processing: Experience with Apache Flink or other stream processing frameworks is a plus. More ❯
performance and responsiveness. Stay Up to Date with Technology : Keep yourself and the team updated on the latest Python technologies, frameworks, and tools like Apache Spark , Databricks , Apache Pulsar , ApacheAirflow , Temporal , and Apache Flink , sharing knowledge and suggesting improvements. Documentation : Contribute to clear and … or Azure . DevOps Tools : Familiarity with containerization ( Docker ) and infrastructure automation tools like Terraform or Ansible . Real-time Data Streaming : Experience with Apache Pulsar or similar systems for real-time messaging and stream processing is a plus. Data Engineering : Experience with Apache Spark , Databricks , or similar … big data platforms for processing large datasets, building data pipelines, and machine learning workflows. Workflow Orchestration : Familiarity with tools like ApacheAirflow or Temporal for managing workflows and scheduling jobs in distributed systems. Stream Processing : Experience with Apache Flink or other stream processing frameworks is a plus. More ❯
quickly and apply new skills Desirable Solid understanding of microservices development SQL and NoSQL databases working set Familiar with or able to quickly learn Apache NiFi, ApacheAirflow, Apache Kafka, KeyCloak, Serverless Computing, GraphQL, APIs, APIM Good skills working with JSON, XML, YAML files Knowledge in More ❯
Manager - Life Sciences at RED Global UK based On-site: 2+ days a week Key Responsibilities: Orchestrate data workflows: Utilize workflow management tools like ApacheAirflow (or similar) to schedule, monitor, and manage the execution of data pipelines. Manage Cloud Infrastructure: Leverage cloud platforms like AWS and Azure … Python programming skills: Experience writing and debugging complex Python code, including experience with libraries like Pandas, PySpark, and related data science libraries. Experience with Apache Spark and Databricks: Deep understanding of Apache Spark principles and experience with Databricks notebooks, clusters, and workspace management. Orchestration tools expertise: Strong experience … with workflow management tools like ApacheAirflow, Prefect, or similar. This includes designing and implementing complex DAGs (Directed Acyclic Graphs) for pipeline orchestration. Cloud platform experience: Hands-on experience with AWS or Azure , including services related to data processing, storage, and compute. Infrastructure as Code (IaC): Familiarity with More ❯
innovative projects, transforming raw data into powerful insights Design and implement efficient ETL and ELT pipelines using modern tools such as Python, SQL, and ApacheAirflow Build scalable data solutions leveraging cloud platforms and technologies Develop and maintain sophisticated data models, employing dimensional modelling techniques to support comprehensive … Demonstrate: Strong proficiency in SQL and Python for handling complex data problems Experience building and optimising ETL/ELT pipelines Hands-on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like ApacheAirflow Experience with containerisation … translation into technical specifications You may also have some of the desirable skills and experience: Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Familiarity More ❯
Synchronous and Asynchronous integrations into a Salesforce environment. Advanced Database skills in SQL, Stored Procedures, Triggers, Architecture. Practical commercial experience across Azure Data Factory, ApacheAirflow, Databricks AutoLoader. Success in IAC lead solutions (Infrastructure as Code). Strong understanding of data flow and message services such as Event … Hub, Apache Kafka. Please note: This role requires you to be onsite 3 days a week; therefore, a reasonable commutable domicile is expected. Where Leeds is the preferred location, other offices such as Middlesbrough, Bristol, or Bournemouth can be considered. This role does not currently provide sponsorship support. Reward … enhanced pension, and an extensive employee benefits programme. Data Integration Engineer, Integration Engineer, Salesforce, .Net, Azure, Azure Data Factory, RDBMS, SQL, Stored Procedures, Triggers, ApacheAirflow, Databricks, Autoloader, CI, CD, IAC, Infrastructure as Code, DevOps, Agile, Event Hub, Apache Kafka. We are Disability Confident and neurodiverse aware. More ❯
automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … in languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like ApacheAirflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud More ❯
extension) - Rate: £500 - 600 per day (Outside IR35) - Location: Remote (must be UK-based) with occasional travel to London, UK - Tech Stack: Snowflake, AWS, ApacheAirflow, SQL, ETL, Python, DBT, Data Pipelines The Role: Join a dynamic team as our client expands their utilisation of Snowflake for Data … the client scales out their Data Warehousing capabilities. You'll play a key role in designing, building, and optimising data pipelines, leveraging AWS and ApacheAirflow for automation and scalability. Could this role be of interest? If so, please get in touch with Alex at iO Associates. For More ❯
Saffron Walden, Essex, South East, United Kingdom Hybrid / WFH Options
EMBL-EBI
services, including microservice deployment Developing and maintaining data pipelines to process and manage large-scale structural biology data Working with workflow orchestration tools like ApacheAirflow and Nextflow To liaise with the EMBL-EBI Technical Services Cluster To support the development and deployment of other scientific software The … Proficiency in containerisation Proficiency in relational databases (Oracle, PostgreSQL) Experience in developing and maintaining data pipelines Hands-on experience with workflow orchestration tools (e.g., ApacheAirflow, Nextflow) Strong interpersonal and communication skills Proficiency in oral and written English You may also have PhD in computer science, IT or … a related field, or in bioinformatics with a demonstrated IT expertise Experience in using Kubernetes Experience with web servers (Apache/Nginx) Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with networking Familiarity with Java Knowledge of, or affinity with, structural biology and bioinformatics More ❯
Saffron Walden, England, United Kingdom Hybrid / WFH Options
EMBL-EBI
services, including microservice deployment Developing and maintaining data pipelines to process and manage large-scale structural biology data Working with workflow orchestration tools like ApacheAirflow and Nextflow To liaise with the EMBL-EBI Technical Services Cluster To support the development and deployment of other scientific software The … Proficiency in containerisation Proficiency in relational databases (Oracle, PostgreSQL) Experience in developing and maintaining data pipelines Hands-on experience with workflow orchestration tools (e.g., ApacheAirflow, Nextflow) Strong interpersonal and communication skills Proficiency in oral and written English You may also have PhD in computer science, IT or … a related field, or in bioinformatics with a demonstrated IT expertise Experience in using Kubernetes Experience with web servers (Apache/Nginx) Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with networking Familiarity with Java Knowledge of, or affinity with, structural biology and bioinformatics More ❯
with hands-on experience in ETL/ELT pipelines and data governance best practices. Proficiency in modern big data frameworks and tools such as Apache Spark, ApacheAirflow, dbt, as well as familiarity with cloud-based data services (AWS, Azure, or GCP). Strong understanding of distributed … business analysts, application developers) to translate business requirements into scalable, high-performance data solutions. Design and implement robust data pipelines using tools like Spark, Airflow, and dbt, ensuring data quality, reliability, and availability for analytics and reporting. Oversee data architecture standards and governance practices, including data security, compliance, lineage More ❯
with hands-on experience in ETL/ELT pipelines and data governance best practices. Proficiency in modern big data frameworks and tools such as Apache Spark, ApacheAirflow, dbt, as well as familiarity with cloud-based data services (AWS, Azure, or GCP). Strong understanding of distributed … business analysts, application developers) to translate business requirements into scalable, high-performance data solutions. Design and implement robust data pipelines using tools like Spark, Airflow, and dbt, ensuring data quality, reliability, and availability for analytics and reporting. Oversee data architecture standards and governance practices, including data security, compliance, lineage More ❯
Science, Math, or Financial Engineering degree Strong knowledge in other programming language(s) - e.g., JavaScript, Typescript, Kotlin Strong knowledge of data orchestration technologies - e.g., ApacheAirflow, Dagster, AWS Step Functions Understanding of ETL/ELT workflows, data modeling, and performance optimization for both batch and real-time processing. More ❯
understanding of tradable financial instruments (securities, derivatives) and capital markets. Computer Science, Math, or Financial Engineering degree. Strong knowledge of data orchestration technologies – e.g., ApacheAirflow, Dagster, AWS Step Functions. Understanding of ETL/ELT workflows, data modeling, and performance optimization for both batch and real-time processing. More ❯
or Angular good but not necessary) Agile The following is DESIRABLE, not essential: AWS or GCP Buy-side Data tools such as Glue, Athena, Airflow, Ignite, DBT, Arrow, Iceberg, Dremio Fixed Income performance, risk or attribution TypeScript and Node Role: Python Developer (Software Engineer Programmer Developer Python Fixed Income … the office 1-2 times a week. The tech environment is very new and will soon likely include exposure to the following: Glue, Athena, Airflow, Ignite, DBT, Arrow, Iceberg, Dremio This is an environment that has been described as the only corporate environment with a start-up/fintech More ❯
experience using Scala, Python, or Java Experience in most data and cloud technologies such as Hadoop, HIVE, Spark, Pig, SQOOP, Flume, PySpark, Databricks, Cloudera, Airflow, Oozie, S3, Glue, Athena, Terraform, etc. Experience with schema design using semi-structured and structured data structures Experience on messaging technologies (e.g. Kafka, Spark More ❯
of the open-source libraries we use extensively. We implement the systems that require the highest data throughput in Java and C++. We use Airflow for workflow management, Kafka for data pipelines, Bitbucket for source control, Jenkins for continuous integration, Grafana + Prometheus for metrics collection, ELK for log More ❯
and similar) queries and optimize performance. Proficiency in Python or Java/Scala . Experience building and maintaining complex ETL pipelines with tools like ApacheAirflow, dbt, or custom scripts. Strong understanding of dimensional modeling, star/snowflake schemas, normalization/denormalization principles. Proven experience with platforms like … Snowflake , Redshift , BigQuery , Synapse . Expert knowledge of Apache Spark , Kafka , Flink , or similar. Strong understanding of data security and privacy standards. Good to Have A degree in Computer Science, Engineering, Mathematics, or a related field. Familiarity with one of the major cloud platforms ( AWS , GCP , Azure ) and their … code tools (e.g., Terraform, CloudFormation). Exposure to containerization/orchestration (Docker, Kubernetes). Familiarity with data governance , data lineage , and catalog tools (e.g., Apache Atlas, Amundsen). Hands-on with observability and monitoring tools for data pipelines (e.g., Monte Carlo, Datadog). Knowledge of machine learning pipelines . More ❯
the big 3 cloud ML stacks (AWS, Azure, GCP). Hands-on experience with open-source ETL, and data pipeline orchestration tools such as ApacheAirflow and Nifi. Experience with large scale/Big Data technologies, such as Hadoop, Spark, Hive, Impala, PrestoDb, Kafka. Experience with workflow orchestration … tools like Apache Airflow. Experience with containerisation using Docker and deployment on Kubernetes. Experience with NoSQL and graph databases. Unix server administration and shell scripting experience. Experience in building scalable data pipelines for highly unstructured data. Experience in building DWH and data lakes architectures. Experience in working in cross More ❯
processing. Hands-on experience building and operation for data lake using one or more of the following big data frameworks or services: Spark, Kafka, Airflow, DBT, Debezium, AWS Athena, AWS Glue, Delta lake/Iceberg etc. Experience with Kubernetes, Docker, Terraform or other cluster management solutions on AWS or More ❯
diagram of proposed tables to enable discussion. Good communicator and comfortable with presenting ideas and outputs to technical and non-technical users. Worked on ApacheAirflow before to create DAGs. Ability to work within Agile, considering minimum viable products, story pointing, and sprints. More information: Enjoy fantastic perks More ❯
Proficiency in version control tools like Git ensures effective collaboration and management of code and data models. Experience with workflow automation tools, such as ApacheAirflow, is crucial for streamlining and orchestrating complex data processes. Skilled at integrating data from diverse sources, including APIs, databases, and third-party More ❯
Science or equivalent Experience in developing Finance or HR related applications Working experience with Tableau Working experience with Terraform Experience in creating workflows for ApacheAirflow and Jenkins Benefits Roku is committed to offering a diverse range of benefits as part of our compensation package to support our More ❯
with Terraform and Terragrunt. Engage in all stages of the software development lifecycle, from design to support. Utilize scheduling systems, with a preference for Airflow, to manage workflows. Automate deployment, releases, and testing in CI/CD pipelines. Write and automate unit, component, integration, and end-to-end tests. More ❯
