Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 3 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
3 May 2024
Same period 2023 Same period 2022
Rank 227 275 223
Rank change year-on-year +48 -52 +85
Permanent jobs citing Data Pipeline 1,281 1,329 2,895
As % of all permanent jobs advertised in the UK 1.30% 1.28% 1.85%
As % of the Processes & Methodologies category 1.51% 1.34% 1.93%
Number of salaries quoted 701 796 1,116
10th Percentile £37,000 £46,250 £47,500
25th Percentile £50,000 £56,770 £57,375
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +15.38%
75th Percentile £83,750 £87,500 £87,500
90th Percentile £105,000 £100,000 £97,500
UK excluding London median annual salary £60,000 £60,000 £65,000
% change year-on-year - -7.69% +11.12%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 84,746 99,225 149,638
As % of all permanent jobs advertised in the UK 85.94% 95.61% 95.61%
Number of salaries quoted 59,834 58,281 83,297
10th Percentile £29,000 £34,000 £33,500
25th Percentile £40,000 £45,000 £43,750
Median annual salary (50th Percentile) £55,000 £61,500 £60,000
Median % change year-on-year -10.57% +2.50% +9.09%
75th Percentile £72,500 £81,750 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +9.38%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 3 May 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 3 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +36 1,038 £67,500 -2.80% 302
UK excluding London +54 781 £60,000 - 130
Work from Home -12 497 £69,235 +2.57% 166
London +46 442 £75,000 -6.25% 183
North of England +22 244 £52,000 -15.00% 29
Scotland +45 188 £67,000 +7.20% 10
North West +76 150 £50,000 - 12
South East +40 117 £67,500 +12.50% 20
Midlands +35 104 £52,000 -13.33% 26
South West -53 77 £67,500 +10.66% 17
West Midlands +7 58 £60,000 - 18
Yorkshire -48 49 £62,000 -7.56% 14
East Midlands +28 46 £43,750 -20.45% 8
North East +36 45 £51,250 -17.34% 3
East of England +36 33 £60,000 -5.88% 17
Wales +19 19 £46,131 -29.03% 8
Northern Ireland +26 3 £60,000 -7.69% 4

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 150 (11.71%) Apache Spark
2 105 (8.20%) Apache
3 77 (6.01%) Apache Airflow
4 6 (0.47%) SharePoint
5 5 (0.39%) SAS
6 3 (0.23%) Microsoft Exchange
7 2 (0.16%) Confluence
7 2 (0.16%) Luigi
8 1 (0.078%) Apache Synapse
Applications
1 23 (1.80%) Microsoft Excel
2 8 (0.62%) Microsoft Office
3 4 (0.31%) Spreadsheet
4 3 (0.23%) Microsoft PowerPoint
4 3 (0.23%) MS Visio
Business Applications
1 4 (0.31%) Primavera
2 1 (0.078%) Salesforce CRM
2 1 (0.078%) SAP S/4HANA
2 1 (0.078%) Shopify
Cloud Services
1 476 (37.16%) AWS
2 460 (35.91%) Azure
3 359 (28.02%) GCP
4 209 (16.32%) Power Platform
5 163 (12.72%) Azure Data Factory
6 153 (11.94%) Snowflake
7 85 (6.64%) Serverless
8 66 (5.15%) dbt
9 65 (5.07%) Azure Synapse Analytics
10 63 (4.92%) Amazon S3
11 60 (4.68%) Cloud Computing
12 57 (4.45%) Matillion
13 44 (3.43%) Azure DevOps
13 44 (3.43%) GitHub
14 37 (2.89%) AWS Lambda
15 36 (2.81%) Azure Cosmos DB
16 23 (1.80%) SaaS
17 22 (1.72%) Amazon EMR
17 22 (1.72%) PaaS
18 19 (1.48%) Fivetran
Communications & Networking
1 65 (5.07%) Kerberos
2 18 (1.41%) DSL
3 8 (0.62%) Firewall
4 2 (0.16%) Intrusion Detection
4 2 (0.16%) Network Security
4 2 (0.16%) VPN
5 1 (0.078%) 3GPP
5 1 (0.078%) 5G
5 1 (0.078%) PPP
5 1 (0.078%) SMS
Database & Business Intelligence
1 367 (28.65%) Data Warehouse
2 253 (19.75%) Data Lake
3 236 (18.42%) BigQuery
4 205 (16.00%) Power BI
5 191 (14.91%) Looker
6 186 (14.52%) Tableau
7 144 (11.24%) Big Data
8 143 (11.16%) SQL Server
9 128 (9.99%) Metadata
10 126 (9.84%) NoSQL
11 75 (5.85%) SQL Server Integration Services
12 67 (5.23%) Amazon Redshift
13 56 (4.37%) PostgreSQL
14 53 (4.14%) Hadoop
15 48 (3.75%) Elasticsearch
16 44 (3.43%) Azure SQL Data Warehouse
17 43 (3.36%) Azure SQL Database
18 42 (3.28%) Star Schema
19 37 (2.89%) Relational Database
20 32 (2.50%) Delta Lake
Development Applications
1 244 (19.05%) Git
2 119 (9.29%) Jenkins
3 71 (5.54%) Bitbucket
4 65 (5.07%) Artifactory
5 51 (3.98%) GitLab
6 15 (1.17%) Spinnaker
7 10 (0.78%) CircleCI
8 8 (0.62%) JIRA
8 8 (0.62%) Oracle Forms
9 6 (0.47%) RStudio
10 5 (0.39%) GDB
11 4 (0.31%) Octopus Deploy
11 4 (0.31%) TeamCity
12 3 (0.23%) Visual Studio
12 3 (0.23%) Visual Studio Team System
13 2 (0.16%) Cucumber
13 2 (0.16%) Selenium
13 2 (0.16%) SQL Server Management Studio (SSMS)
13 2 (0.16%) webpack
14 1 (0.078%) SpecFlow
General
1 414 (32.32%) Analytical Skills
2 317 (24.75%) Social Skills
3 310 (24.20%) Finance
4 199 (15.53%) Inclusion and Diversity
5 164 (12.80%) Law
6 137 (10.69%) Presentation Skills
7 131 (10.23%) Retail
8 127 (9.91%) Marketing
9 89 (6.95%) Banking
10 55 (4.29%) Legal
11 37 (2.89%) Public Sector
12 22 (1.72%) Games
13 21 (1.64%) Investment Banking
14 20 (1.56%) Financial Institution
15 16 (1.25%) Billing
16 12 (0.94%) Back Office
16 12 (0.94%) Influencing Skills
17 11 (0.86%) Retail Banking
17 11 (0.86%) Telecoms
18 10 (0.78%) Front Office
Job Titles
1 689 (53.79%) Data Engineer
2 331 (25.84%) Senior
3 180 (14.05%) Platform Engineer
4 178 (13.90%) Lead
5 177 (13.82%) Principal Engineer
6 174 (13.58%) Principal Data Engineer
7 141 (11.01%) Senior Data Engineer
8 131 (10.23%) Software Engineer
9 130 (10.15%) Analyst
10 119 (9.29%) Data Analyst
11 82 (6.40%) Developer
12 67 (5.23%) Lead Software Engineer
13 66 (5.15%) Senior Analyst
14 63 (4.92%) Senior Data Analyst
15 61 (4.76%) Architect
16 56 (4.37%) Data Architect
17 55 (4.29%) AWS Engineer
18 54 (4.22%) Senior Software Engineer
19 49 (3.83%) Lead Data Engineer
20 46 (3.59%) Azure Engineer
Libraries, Frameworks & Software Standards
1 101 (7.88%) Kafka
2 74 (5.78%) JSON
2 74 (5.78%) PySpark
2 74 (5.78%) RESTful
2 74 (5.78%) XML
3 70 (5.46%) Web Services
4 65 (5.07%) JSON-LD
4 65 (5.07%) LDAP
4 65 (5.07%) OpenID
4 65 (5.07%) SAML
5 53 (4.14%) Pandas
6 38 (2.97%) Apache Beam
7 23 (1.80%) Apache NiFi
7 23 (1.80%) XGBoost
8 20 (1.56%) PyTorch
8 20 (1.56%) TensorFlow
9 19 (1.48%) React
10 17 (1.33%) .NET
11 16 (1.25%) REST
12 12 (0.94%) Flask
Miscellaneous
1 213 (16.63%) Self-Motivation
2 103 (8.04%) Data Structures
3 102 (7.96%) Management Information System
4 86 (6.71%) Algorithms
5 75 (5.85%) Cloud Native
6 65 (5.07%) Enterprise Cloud
6 65 (5.07%) Reference Data
7 64 (5.00%) Onboarding
8 54 (4.22%) Product Ownership
9 33 (2.58%) Public Cloud
10 24 (1.87%) Taxonomies
11 23 (1.80%) IoT
12 18 (1.41%) Greenfield Project
13 15 (1.17%) Enterprise Software
14 14 (1.09%) Distributed Systems
14 14 (1.09%) Hedge funds
15 9 (0.70%) Client/Server
15 9 (0.70%) Distributed Applications
16 8 (0.62%) Analytical Mindset
16 8 (0.62%) Data Centre
Operating Systems
1 37 (2.89%) Linux
2 19 (1.48%) Unix
3 2 (0.16%) Windows
4 1 (0.078%) Android
4 1 (0.078%) Apple iOS
4 1 (0.078%) Windows Server
Processes & Methodologies
1 555 (43.33%) Agile
2 519 (40.52%) Data Engineering
3 481 (37.55%) ETL
4 476 (37.16%) Analytics
5 448 (34.97%) Problem-Solving
6 432 (33.72%) Data Science
7 417 (32.55%) Data Modelling
8 338 (26.39%) CI/CD
9 334 (26.07%) Visualisation
10 315 (24.59%) DevOps
11 284 (22.17%) Machine Learning
12 281 (21.94%) Data Analysis
13 268 (20.92%) Business Intelligence
14 259 (20.22%) Mentoring
15 232 (18.11%) Software Engineering
16 215 (16.78%) Platform Engineering
17 187 (14.60%) Artificial Intelligence
17 187 (14.60%) DataOps
18 165 (12.88%) Change Management
19 161 (12.57%) Data Transformation
Programming Languages
1 775 (60.50%) SQL
2 712 (55.58%) Python
3 160 (12.49%) Java
4 96 (7.49%) Scala
5 82 (6.40%) Go
6 65 (5.07%) Prolog
6 65 (5.07%) SPARQL
7 61 (4.76%) T-SQL
8 36 (2.81%) C#
9 29 (2.26%) R
10 23 (1.80%) JavaScript
11 22 (1.72%) PL/SQL
12 18 (1.41%) Data Analysis Expressions
13 17 (1.33%) TypeScript
14 15 (1.17%) Kotlin
15 9 (0.70%) C
15 9 (0.70%) PowerShell
16 5 (0.39%) MATLAB
17 4 (0.31%) C++
17 4 (0.31%) Clojure
Qualifications
1 244 (19.05%) Degree
2 128 (9.99%) DBS Check
3 65 (5.07%) Computer Science Degree
4 46 (3.59%) Master's Degree
5 30 (2.34%) Security Cleared
6 20 (1.56%) SC Cleared
7 16 (1.25%) Microsoft Certification
8 15 (1.17%) Azure Certification
9 14 (1.09%) DV Cleared
10 12 (0.94%) PhD
11 5 (0.39%) Numerate Degree
12 4 (0.31%) BPSS Clearance
12 4 (0.31%) CIMA
12 4 (0.31%) PMI Certification
13 3 (0.23%) AWS Certification
13 3 (0.23%) AWS Certified Solutions Architect
14 1 (0.078%) HND
Quality Assurance & Compliance
1 190 (14.83%) GDPR
2 167 (13.04%) Data Quality
3 21 (1.64%) QA
4 19 (1.48%) SLA
5 12 (0.94%) Accessibility
6 4 (0.31%) Solvency II
7 3 (0.23%) ISO/IEC 27001
8 2 (0.16%) HIPAA
9 1 (0.078%) California Consumer Privacy Act
9 1 (0.078%) GMP
9 1 (0.078%) NCSC
System Software
1 67 (5.23%) Active Directory
2 55 (4.29%) Docker
3 9 (0.70%) HDFS
4 2 (0.16%) Virtual Machines
Systems Management
1 102 (7.96%) Kubernetes
2 62 (4.84%) Terraform
3 11 (0.86%) Ansible
4 8 (0.62%) Argo
5 5 (0.39%) Kubeflow
6 1 (0.078%) Grafana
6 1 (0.078%) Prometheus
Vendors
1 355 (27.71%) Databricks
2 236 (18.42%) Google
3 152 (11.87%) Microsoft
4 51 (3.98%) Oracle
5 29 (2.26%) Alteryx
6 20 (1.56%) Informatica
7 18 (1.41%) Cognos
7 18 (1.41%) Sybase
8 13 (1.01%) SAP
9 12 (0.94%) OpenAI
10 11 (0.86%) Cloudera
11 10 (0.78%) Teradata
11 10 (0.78%) YouTube
12 9 (0.70%) Confluent
13 7 (0.55%) Talend
14 6 (0.47%) Dataiku
14 6 (0.47%) F5
15 3 (0.23%) MuleSoft
15 3 (0.23%) SnapLogic
15 3 (0.23%) Splunk