star/snowflake schemas, normalization/denormalization principles. Proven experience with platforms like Snowflake , Redshift , BigQuery , Synapse . Expert knowledge of Apache Spark , Kafka , Flink , or similar. Strong understanding of data security and privacy standards. Good to Have A degree in Computer Science, Engineering, Mathematics, or a related field. More ❯
/CD pipelines. Big Data & Data Engineering: Strong background in processing large datasets and building data pipelines using platforms like Apache Spark , Databricks , ApacheFlink , or similar big data tools. Experience with batch and stream processing. Security: In-depth knowledge of security practices in cloud environments, including identity management More ❯
Manchester, Lancashire, United Kingdom Hybrid / WFH Options
Smart DCC
real-time data processing pipelines using platforms like Apache Kafka or cloud-native tools. Optimize batch processing workflows with tools like Apache Spark and Flink for scalable performance. Infrastructure Automation: Implement Infrastructure as Code (IaC) using tools like Terraform and Ansible. Leverage cloud-native services (AWS, Azure) to streamline More ❯
skills: Experience in retail and/or e-commerce. Understanding of Big Data and Distributed Computing. Understanding of streaming technologies (Spark Structured streaming, ApacheFlink, etc). Other programming languages: PowerShell, Bash. Understanding of Databricks Ecosystem (Unity Catalog, Workflows, Delta Live Tables). Understanding of any Data Observability or More ❯
skills: Experience in retail and/or e-commerce. Understanding of Big Data and Distributed Computing. Understanding of streaming technologies (Spark Structured streaming, ApacheFlink, etc). Other programming languages: PowerShell, Bash. Understanding of Databricks Ecosystem (Unity Catalog, Workflows, Delta Live Tables). Understanding of any Data Observability or More ❯
Have: Experience in retail and/or e-commerce. Understanding of Big Data and Distributed Computing. Understanding of streaming technologies (Spark Structured streaming, ApacheFlink, etc). Other programming languages: PowerShell, Bash. Understanding of Databricks Ecosystem (Unity Catalog, Workflows, Delta Live Tables). Understanding of any Data Observability or More ❯
Code (IaC) using Terraform and Terragrunt. Experience with scheduling systems, with Airflow. Basic understanding of data streaming and messaging frameworks (Kafka, Spark Structured Streaming, Flink, etc.). Good knowledge of the Spark framework and its deployment with cloud services. Preferred Qualifications, Capabilities and Skills Academic qualification in computer science More ❯
advanced analytics infrastructure. Familiarity with infrastructure-as-code (IaC) tools such as Terraform or CloudFormation. Experience with modern data engineering technologies (e.g., Kafka, Spark, Flink, etc.). Why join YouLend? Award-Winning Workplace: YouLend has been recognised as one of the "Best Places to Work 2024" by the Sunday More ❯
managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. Programming skills in Python More ❯
managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. Ability to optimise and More ❯
databases to build data solutions, such as SQL Server/Oracle , experience with relational and dimensional data structures. Experience in using distributed frameworks ( Spark, Flink, Beam, Hadoop ). Proficiency in infrastructure as code (IaC) using Terraform . Experience with CI/CD pipelines and related tools/frameworks. Containerisation More ❯
engineering experience - Experience with data modeling, warehousing, and building ETL pipelines - Bachelor's degree - Knowledge of batch and streaming data architectures like Kafka, Kinesis, Flink, Storm, Beam - Knowledge of distributed systems as it pertains to data storage and computing - Experience programming with at least one modern language such as More ❯
of the field will drive your journey.Requirements: - Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Databricks is a plus.- Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. More ❯
of the field will drive your journey.Requirements: - Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Databricks is a plus.- Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. More ❯
IaC) using Terraform and Terragrunt. Experience with scheduling systems, with Airflow preferred. Basic understanding of data streaming and messaging frameworks (Kafka, Spark Structured Streaming, Flink, etc.). Good knowledge of the Spark framework and its deployment with cloud services. Preferred Qualifications: Academic qualification in computer science or a STEM More ❯
/Nice to have Experience with a messaging middleware platform like Solace, Kafka or RabbitMQ. Experience with Snowflake and distributed processing technologies (e.g., Hadoop, Flink, Spark More ❯
and experience with DevOps practices (CI/CD). Familiarity with containerization (Docker, Kubernetes), RESTful APIs, microservices architecture, and big data technologies (Hadoop, Spark, Flink). Knowledge of NoSQL databases (MongoDB, Cassandra, DynamoDB), message queueing systems (Kafka, RabbitMQ), and version control systems (Git). Preferred Skills: Experience with natural More ❯
efficient integration into Feast feature store Requirements Good knowledge of programming languages such as Python or Java Strong experience with streaming technologies (Spark, PySpark, Flink, KSQL or similar) for developing data transformation pipelines Solid understanding and practical experience with SQL and relational databases (PostgreSQL preferred) Proficiency with AWS EMR More ❯
efficient integration into Feast feature store. Requirements Good knowledge of programming languages such as Python or Java. Strong experience with streaming technologies (Spark, PySpark, Flink, KSQL or similar) for developing data transformation pipelines. Solid understanding and practical experience with SQL and relational databases (PostgreSQL preferred). Proficiency with AWS More ❯
oriented programming (OOP) principles & concepts Familiarity with advanced SQL techniques Familiarity with data visualization tools such as Tableau or Power BI Familiarity with ApacheFlink or Apache Storm Understanding of DevOps practices and tools for (CI/CD) pipelines. Awareness of data security best practices and compliance requirements (e.g. More ❯
security Cloud performance: experience optimising performance and cost efficiency of data platforms Exposure to tools like S3, BigQuery, Snowflake, Databricks Exposure to tools like Flink, Spark, Kafka Exposure to containerised environments: Docker/Kubernetes These roles are looking to start in May and are outside IR35. #J-18808-Ljbffr More ❯
development experience using SQL. Hands-on experience with MPP databases such as Redshift, BigQuery, or Snowflake, and modern transformation/query engines like Spark, Flink, Trino. Familiarity with workflow management tools (e.g., Airflow) and/or dbt for transformations. Comprehensive understanding of modern data platforms, including data governance and More ❯
solutions. Exposure to some of the following technologies: Python, AWS Redshift, AWS Athena/Apache Presto, Big Data technologies (e.g S3, Hadoop, Hive, Spark, Flink, Kafka etc), NoSQL systems like Cassandra, DBT is nice to have. What you'll get: Full responsibility for projects from day one, a collaborative More ❯
solutions. Exposure to some of the following technologies: Python, AWS Redshift, AWS Athena/Apache Presto, Big Data technologies (e.g., S3, Hadoop, Hive, Spark, Flink, Kafka, etc.), NoSQL systems like Cassandra, DBT is nice to have. What you'll get: Full responsibility for projects from day one, a collaborative More ❯
