4 of 4 Apache Spark Jobs in Yorkshire

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

Education Required Approximately 10 years of industry experience implementing data integration tools such as Microsoft SSIS, Azure Data Factory, Databricks, Glue, Step Functions, Airflow, Apache Flume/Sqoop/Pig, and similar technologies. 3–5 years of team management experience. 3–5 years of consulting experience (preferred). Minimum … knowledge of data warehousing, OLTP systems, data integration, and the software development lifecycle (SDLC). Hands-on experience with big data frameworks such as Spark, Hadoop, or Hive, including derivatives like PySpark (preferred), Spark Scala, or Spark SQL, along with familiarity with development-accelerating libraries and frameworks. ...

teams Ensure solutions meet security, compliance, and regulatory standards Key Skills Required: Experience in solution design/architecture Strong background in big data tools (Spark, Airflow, Trino, etc.) Knowledge of cloud platforms (AWS/Azure/GCP) Understanding of cybersecurity principles (IAM, encryption, secure design) Ability to work hands ...

logic into data pipelines and services. Conduct bespoke, high-complexity analysis in support of customer-facing or operational needs. Guide team best practices in Spark SQL usage, data documentation, and exploratory reproducibility. Desired Qualifications: 5+ years of experience in data science, applied analytics, machine learning, or analytical R&D. … Advanced expertise in Python and distributed compute frameworks (e.g., Spark, Databricks), including strong proficiency in Spark SQL . Strong background in statistical inference, anomaly detection, clustering, interaction modeling , or other analytical methods suited to large and heterogeneous datasets. Experience working with multi-source, semi-structured, geospatial, or entity ...