8 of 8 Apache Spark Jobs in Yorkshire

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

architecture and end-to-end system design Proven Experience with big data platforms at scale Deep knowledge of open-source data ecosystems, such as: Apache Spark Trino Airflow Apache Superset Cloud & Infrastructure Experience across cloud and on-premise environments Familiarity with AWS, GCP, and/or Azure ...

strong Python/PySpark engineers who can learn Foundry quickly are encouraged to apply. Essential qualifications • Extensive experience with Python and PySpark/Apache Spark. • Proven track record designing and delivering scalable data pipelines and data products. • Strong SQL and data modelling skills. • Experience with modern data architectures ...

Education Required Approximately 10 years of industry experience implementing data integration tools such as Microsoft SSIS, Azure Data Factory, Databricks, Glue, Step Functions, Airflow, Apache Flume/Sqoop/Pig, and similar technologies. 3–5 years of team management experience. 3–5 years of consulting experience (preferred). Minimum … knowledge of data warehousing, OLTP systems, data integration, and the software development lifecycle (SDLC). Hands-on experience with big data frameworks such as Spark, Hadoop, or Hive, including derivatives like PySpark (preferred), Spark Scala, or Spark SQL, along with familiarity with development-accelerating libraries and frameworks. ...

teams Ensure solutions meet security, compliance, and regulatory standards Key Skills Required: Experience in solution design/architecture Strong background in big data tools (Spark, Airflow, Trino, etc.) Knowledge of cloud platforms (AWS/Azure/GCP) Understanding of cybersecurity principles (IAM, encryption, secure design) Ability to work hands ...

Join DWP Digital and lead the future of data engineering in one of the UK's largest and most impactful digital environments As a Senior Data Engineer, you'll join our Data Platform and Operations ...

Job Description Join DWP Digital and lead the future of data engineering in one of the UK's largest and most impactful digital environmentsAs a Senior Data Engineer, you'll join our Data Platform and ...

needed). Skills and knowledge Demonstrable experience designing, deploying, and operating big data platforms at scale. Hands-on experience with OSS data ecosystems (e.g., Spark, Trino, Airflow, Apache Superset, and similar technologies). Knowledge of cloud platforms and traditional on-premise hosting; hands-on experience with Azure ...