4 of 4 Apache Spark Jobs in Yorkshire

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

from ingestion and streaming through to warehousing, serving, and cost optimisation You have deep experience with streaming architectures – Kinesis, Kafka/MSK, Flink, or Spark Streaming – and know when each is the right choice You can model the cost implications of architecture decisions and advise clients on trade‐offs … between throughput, latency, durability, and spend AWS certification at professional or specialty level is strongly preferred Mastery of Python, SQL, and Spark – and strong opinions on how they should be used in production data pipelines Deep experience with dbt, Airflow, or equivalent orchestration – including how to build pipelines that ...

must have demonstrable experience designing, deploying, and operating big data platforms at scale. You will have hands-on experience with OSS data ecosystems (e.g., Spark, Trino, Airflow, Apache Superset, and similar technologies). You will have knowledge of cloud platforms and traditional on-premise hosting; hands-on experience ...