flows & pipelines in a cloud environment using a progressive tech stack e.g. Databricks, Spark, Python, PySpark, Delta Lake, SQL, Logic Apps, Azure Functions, ADLS, Parquet, Neo4J, Flask. Ingest and integrate data from a large number of disparate data sources. Design and build complex data models for analytical and marketing more »
flows & pipelines in a cloud environment using a progressive tech stack e.g. Databricks, Spark, Python, PySpark, Delta Lake, SQL, Logic Apps, Azure Functions, ADLS, Parquet, Neo4J, Flask. Ingest and integrate data from a large number of disparate data sources. Design and build complex data models for analytical and marketing more »
with experience in Kafka Real-time messaging or Azure Stream Analytics/Event Hub. Spark processing and performance tuning. File formats partitioning for e.g. Parquet, JSON, XML, CSV. Azure DevOps, GitHub actions. Hands-on experience in at least one of Python with knowledge of the others. Experience in Data more »
Azure service bus, Function Apps, ADFs Possesses knowledge on data related technologies like - Data Warehouse, snowflake, ETL, Data pipelines, pyspark, delta tables, file formats - parquet, columnar Have a good understanding of SQL, stored procedures Be able to lead development and execution of performance and automation testing for large-scale more »
Experience in data modelling and design patterns; in-depth knowledge of relational databases (PostgreSQL) and familiarity with data lakehouse formats (storage formats, e.g. ApacheParquet, Delta tables). Experience with Spark, Databricks, data lakes/lakehouses. Experience working with external data suppliers (defining requirements for suppliers, defining Service Level more »
Experience in data modelling and design patterns; in-depth knowledge of relational databases (PostgreSQL) and familiarity with data lakehouse formats (storage formats, e.g. ApacheParquet, Delta tables). Experience with Spark, Databricks, data lakes/lakehouses. Experience working with external data suppliers (defining requirements for suppliers, defining Service Level more »
Proficient in Java or Scala or Python Advanced SQL skills Experience with DBT and modern data pipeline tools Strong knowledge of file formats (ORC, Parquet, AVRO) and optimization techniques Experience with DevOps practices and containerization technologies Proven track record of stakeholder management Experience working in regulated environments Desirable: Experience more »
Employment Type: Permanent
Salary: £60000 - £80000/annum 10% bonus, 10% pension, medical
experience with Matillion. Familiarity with a variety of Databases, incl. structured RDBMS. Experience in working with a variety of data formats, JSON, XML, CSV, Parquet, etc. Experience with building and maintaining data dictionaries/meta-data. Experience with Linux and cloud environments. Data Visualisation Technologies (e.g. Amazon QuickSight, Tableau more »
Starburst and Athena Kafka and Kinesis DataHub ML Flow and Airflow Docker and Terraform Kafka, Spark, Kafka Streams and KSQL DBT AWS, S3, Iceberg, Parquet, Glue and EMR for our Data Lake Elasticsearch and DynamoDB More information: Enjoy fantastic perks like private healthcare & dental insurance, a generous work from more »
platform Kubernetes for data services and task orchestration Streamlit for data applications Airflow purely for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing dbt for data modelling SparkSQL for analytics Why else you more »
Services (S3, Lambda, Glue, API Gateway, Kinesis, IAM) Integrations (Email, SFTP, API, Webhooks, Streaming) Data Formats and Structures (XML, Excel, CSV, TSV, JSON, AVRO, Parquet) Qualifications Basic Requirements: Self-Starter: Ability to take initiative and work independently. Confident Speaker: Strong communication skills, comfortable presenting and discussing ideas. Technically Inclined more »
and Lambda IAM - Experience handling IAM resource permissions Networking - fundamental understanding of VPC, subnet routing and gateways Storage - strong understanding of S3, EBS and Parquet Databases - RDS, DynamoDB Experience doing cost estimation in Cost Explorer and planning efficiency changes Terraform and containerisation experience Understanding of a broad range of more »
Computer Science or meaningful relevant work experience Preferred Qualifications Experience with large scale data platform infrastructure such as Spark, Flink, HDFS, AWS/S3, Parquet, Kubernetes is a plus more »
We are looking for a Data Engineer to join our growing data engineering team at Our Future Health. The Data Engineer will bring an in-depth knowledge of NHS data and data solutions to help solve some of the key more »
