Data Pipeline Contracts

Data Pipelines
UK

The following table provides summary statistics for contract job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 20 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
20 May 2024
Same period 2023 Same period 2022
Rank 188 288 268
Rank change year-on-year +100 -20 -15
Contract jobs citing Data Pipeline 525 520 930
As % of all contract jobs advertised in the UK 1.23% 0.91% 1.05%
As % of the Processes & Methodologies category 1.43% 1.01% 1.16%
Number of daily rates quoted 389 356 612
10th Percentile £363 £417 £450
25th Percentile £450 £492 £506
Median daily rate (50th Percentile) £525 £550 £600
Median % change year-on-year -4.55% -8.33% +9.09%
75th Percentile £638 £675 £714
90th Percentile £738 £738 £813
UK excluding London median daily rate £510 £525 £550
% change year-on-year -2.86% -4.55% +10.00%
Number of hourly rates quoted 7 6 6
10th Percentile £25.34 £50.63 £27.00
25th Percentile £37.25 £65.00 £28.56
Median hourly rate £62.50 £70.25 £33.75
Median % change year-on-year -11.03% +108.15% -17.68%
75th Percentile £66.25 £80.81 £68.25
90th Percentile £67.25 £86.25 £84.83
UK excluding London median hourly rate £62.50 £55.88 -
% change year-on-year +11.86% - -

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for process or methodology skills.

Contract vacancies with a requirement for process or methodology skills 36,622 51,397 80,021
As % of all contract IT jobs advertised in the UK 86.09% 89.76% 90.61%
Number of daily rates quoted 23,613 35,611 56,174
10th Percentile £300 £325 £344
25th Percentile £413 £439 £430
Median daily rate (50th Percentile) £525 £550 £530
Median % change year-on-year -4.55% +3.77% +7.07%
75th Percentile £638 £650 £638
90th Percentile £750 £750 £738
UK excluding London median daily rate £500 £500 £480
% change year-on-year - +4.17% +9.09%
Number of hourly rates quoted 2,439 1,690 1,913
10th Percentile £12.75 £10.63 £12.50
25th Percentile £16.00 £15.77 £15.50
Median hourly rate £36.50 £36.00 £25.68
Median % change year-on-year +1.39% +40.19% +7.00%
75th Percentile £60.00 £65.00 £49.50
90th Percentile £72.50 £75.00 £65.00
UK excluding London median hourly rate £38.00 £35.00 £20.00
% change year-on-year +8.57% +75.00% -6.54%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Data Pipeline.

Daily rate trend for Data Pipeline in the UK

Data Pipeline
Daily Rate Histogram

Daily rate distribution for jobs citing Data Pipeline over the 6 months to 20 May 2024.

Daily rate histogram for Data Pipeline in the UK

Data Pipeline
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Data Pipeline.

Hourly rate trend for Data Pipeline in the UK

Data Pipeline
Hourly Rate Histogram

Hourly rate distribution of jobs citing Data Pipeline over the 6 months to 20 May 2024.

Hourly rate histogram for Data Pipeline in the UK

Data Pipeline
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 20 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +102 469 £550 - 394
London +81 315 £575 +1.77% 266
Work from Home +47 168 £525 -4.55% 186
UK excluding London +50 146 £510 -2.86% 199
North of England +45 51 £500 -6.98% 41
Midlands +36 42 £550 +0.46% 25
West Midlands +34 35 £550 +0.92% 21
North West +11 27 £500 - 19
Yorkshire +39 24 £563 -13.46% 18
South East -10 19 £475 -9.52% 68
Scotland +37 12 £453 +6.59% 19
South West +3 12 £550 -18.52% 22
East of England +17 8 £538 +2.38% 11
East Midlands +22 7 £413 -38.89% 4
Wales +18 1 £500 - 6
Northern Ireland - 1 £425 - 8

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 80 (15.24%) Apache Spark
2 65 (12.38%) Apache
3 63 (12.00%) Apache Airflow
4 5 (0.95%) Confluence
5 4 (0.76%) Luigi
6 3 (0.57%) SharePoint
7 1 (0.19%) ArcGIS
7 1 (0.19%) Jupyter
Applications
1 13 (2.48%) Microsoft Excel
2 6 (1.14%) Microsoft Office
3 4 (0.76%) Microsoft PowerPoint
3 4 (0.76%) Microsoft Project
3 4 (0.76%) MS Visio
4 2 (0.38%) Sketch
Business Applications
1 2 (0.38%) Dynamics CRM
Cloud Services
1 224 (42.67%) AWS
2 214 (40.76%) Azure
3 78 (14.86%) Serverless
4 67 (12.76%) AWS Lambda
5 66 (12.57%) GCP
6 65 (12.38%) Azure Data Factory
7 62 (11.81%) Amazon S3
8 53 (10.10%) Snowflake
9 39 (7.43%) Power Platform
10 36 (6.86%) Azure Synapse Analytics
11 33 (6.29%) Amazon EMR
12 32 (6.10%) dbt
13 30 (5.71%) Amazon Kinesis
13 30 (5.71%) GitHub
14 26 (4.95%) AWS Step Functions
14 26 (4.95%) Azure DevOps
15 23 (4.38%) Amazon EC2
16 13 (2.48%) Amazon MSK
16 13 (2.48%) Azure Logic Apps
17 12 (2.29%) Cloud Computing
Communications & Networking
1 14 (2.67%) Intrusion Detection
1 14 (2.67%) Network Security
2 5 (0.95%) FTP
3 2 (0.38%) SAN
4 1 (0.19%) Firewall
4 1 (0.19%) HTTP
4 1 (0.19%) SSL
Database & Business Intelligence
1 139 (26.48%) Data Warehouse
2 108 (20.57%) Big Data
3 90 (17.14%) Data Lake
4 60 (11.43%) Amazon Redshift
5 42 (8.00%) NoSQL
6 39 (7.43%) Power BI
7 35 (6.67%) Relational Database
8 33 (6.29%) SQL Server
9 32 (6.10%) BigQuery
10 26 (4.95%) Flink
11 24 (4.57%) Amazon Athena
11 24 (4.57%) Parquet
12 21 (4.00%) Tableau
13 19 (3.62%) Azure SQL Database
14 17 (3.24%) DynamoDB
15 16 (3.05%) Hadoop
16 15 (2.86%) Amazon RDS
17 13 (2.48%) Delta Lake
18 12 (2.29%) Metadata
18 12 (2.29%) PostgreSQL
Development Applications
1 57 (10.86%) Git
2 25 (4.76%) JIRA
3 10 (1.90%) Jenkins
4 7 (1.33%) GitLab
4 7 (1.33%) MLflow
5 5 (0.95%) JMeter
5 5 (0.95%) TestRail
6 4 (0.76%) Deequ
6 4 (0.76%) ERwin
6 4 (0.76%) Mercurial
7 3 (0.57%) Bitbucket
7 3 (0.57%) Postman
8 2 (0.38%) Bugzilla
8 2 (0.38%) ER Studio
8 2 (0.38%) JUnit
8 2 (0.38%) Selenium
8 2 (0.38%) TestNG
8 2 (0.38%) Travis CI
9 1 (0.19%) GoCD
9 1 (0.19%) kappa
General
1 120 (22.86%) Social Skills
2 84 (16.00%) Analytical Skills
3 76 (14.48%) Finance
4 35 (6.67%) Public Sector
5 27 (5.14%) Banking
6 15 (2.86%) Front Office
6 15 (2.86%) Presentation Skills
7 12 (2.29%) Pharmaceutical
8 8 (1.52%) Investment Banking
9 7 (1.33%) Manufacturing
10 6 (1.14%) Marketing
10 6 (1.14%) Retail
11 4 (0.76%) Inclusion and Diversity
11 4 (0.76%) Multimedia
12 3 (0.57%) Back Office
12 3 (0.57%) Games
12 3 (0.57%) Law
13 2 (0.38%) Czech Language
13 2 (0.38%) Financial Institution
13 2 (0.38%) Influencing Skills
Job Titles
1 270 (51.43%) Data Engineer
2 73 (13.90%) Senior
3 60 (11.43%) Azure Engineer
4 59 (11.24%) Developer
5 54 (10.29%) Azure Data Engineer
6 51 (9.71%) AWS Engineer
7 43 (8.19%) Architect
8 42 (8.00%) Senior Data Engineer
9 37 (7.05%) AWS Data Engineer
10 33 (6.29%) Lead
11 27 (5.14%) Analyst
12 23 (4.38%) Python Developer
13 20 (3.81%) Data Scientist
14 19 (3.62%) Data Analyst
14 19 (3.62%) Lead Data Engineer
14 19 (3.62%) Snowflake Engineer
15 16 (3.05%) Data Architect
15 16 (3.05%) Platform Engineer
16 15 (2.86%) Python Engineer
17 14 (2.67%) Snowflake Data Engineer
Libraries, Frameworks & Software Standards
1 71 (13.52%) PySpark
2 61 (11.62%) Kafka
3 37 (7.05%) Pandas
4 26 (4.95%) JSON
5 24 (4.57%) NumPy
6 20 (3.81%) Apache Avro
7 17 (3.24%) SciPy
8 15 (2.86%) Apache NiFi
9 11 (2.10%) REST
10 10 (1.90%) scikit-learn
10 10 (1.90%) TensorFlow
11 8 (1.52%) Elastic Stack
12 7 (1.33%) Flask
12 7 (1.33%) React
13 6 (1.14%) ARM Templates
13 6 (1.14%) FastAPI
13 6 (1.14%) Node.js
13 6 (1.14%) XML
14 5 (0.95%) .NET
14 5 (0.95%) Great Expectations
Miscellaneous
1 28 (5.33%) Algorithms
2 25 (4.76%) Distributed Systems
3 22 (4.19%) Management Information System
4 19 (3.62%) Data Structures
4 19 (3.62%) Derivative
5 15 (2.86%) Cyber Defence
5 15 (2.86%) Distributed Applications
6 12 (2.29%) Linux Command Line
7 11 (2.10%) Greenfield Project
8 9 (1.71%) Distributed Computing
8 9 (1.71%) Onboarding
8 9 (1.71%) Self-Motivation
9 7 (1.33%) Cloud Native
9 7 (1.33%) Social Media
10 6 (1.14%) Hedge funds
10 6 (1.14%) Renewable Energy
11 5 (0.95%) Analytical Mindset
11 5 (0.95%) NHS
12 4 (0.76%) Life Science
12 4 (0.76%) Virtual Team
Operating Systems
1 36 (6.86%) Linux
2 27 (5.14%) Windows
3 15 (2.86%) Unix
4 12 (2.29%) Android
4 12 (2.29%) Apple iOS
5 8 (1.52%) Ubuntu
6 1 (0.19%) Mac OS
6 1 (0.19%) Windows Server
Processes & Methodologies
1 211 (40.19%) ETL
2 176 (33.52%) Analytics
2 176 (33.52%) Data Modelling
3 165 (31.43%) Data Engineering
4 115 (21.90%) CI/CD
5 106 (20.19%) DevOps
6 96 (18.29%) Problem-Solving
7 94 (17.90%) Agile
8 86 (16.38%) Data Science
9 82 (15.62%) Business Intelligence
10 74 (14.10%) Data Integration
11 60 (11.43%) Machine Learning
12 59 (11.24%) Computer Science
13 57 (10.86%) Data Ingestion
14 53 (10.10%) Data Management
15 49 (9.33%) Data Governance
15 49 (9.33%) Visualisation
16 48 (9.14%) Data Transformation
17 47 (8.95%) Test Automation
18 42 (8.00%) Software Engineering
Programming Languages
1 330 (62.86%) Python
2 276 (52.57%) SQL
3 76 (14.48%) Java
4 65 (12.38%) Scala
5 21 (4.00%) C#
5 21 (4.00%) C++
6 19 (3.62%) JavaScript
7 17 (3.24%) T-SQL
8 14 (2.67%) R
9 12 (2.29%) Rust
10 10 (1.90%) Bash
11 7 (1.33%) TypeScript
12 6 (1.14%) Bicep
13 5 (0.95%) VBA
14 4 (0.76%) Go
14 4 (0.76%) Perl
14 4 (0.76%) PowerShell
14 4 (0.76%) Shell Script
15 3 (0.57%) Data Analysis Expressions
15 3 (0.57%) MATLAB
Qualifications
1 96 (18.29%) Security Cleared
2 74 (14.10%) SC Cleared
3 51 (9.71%) Degree
4 37 (7.05%) Computer Science Degree
5 17 (3.24%) DV Cleared
6 10 (1.90%) AWS Certification
7 9 (1.71%) Master's Degree
8 8 (1.52%) Azure Certification
8 8 (1.52%) BPSS Clearance
9 6 (1.14%) CISSP
10 4 (0.76%) AWS Certified Solutions Architect
11 3 (0.57%) CISA
11 3 (0.57%) CRISC
11 3 (0.57%) Microsoft Certification
12 2 (0.38%) AWS Certified Developer
12 2 (0.38%) AWS Certified DevOps Engineer
12 2 (0.38%) PhD
13 1 (0.19%) (ISC)2 CCSP
13 1 (0.19%) Cisco Certification
13 1 (0.19%) CTC Cleared
Quality Assurance & Compliance
1 91 (17.33%) Data Quality
2 14 (2.67%) Actionable Recommendations
2 14 (2.67%) QA
3 6 (1.14%) GDPR
4 4 (0.76%) Accessibility
4 4 (0.76%) NCSC
5 2 (0.38%) PCI DSS
6 1 (0.19%) California Consumer Privacy Act
6 1 (0.19%) JSP 440
6 1 (0.19%) NIST
System Software
1 44 (8.38%) Docker
2 8 (1.52%) Active Directory
2 8 (1.52%) iptables
3 7 (1.33%) HDFS
4 2 (0.38%) VMware Infrastructure
5 1 (0.19%) Virtual Machines
Systems Management
1 55 (10.48%) Terraform
2 42 (8.00%) Kubernetes
3 8 (1.52%) Kibana
3 8 (1.52%) logstash
4 7 (1.33%) Ansible
5 5 (0.95%) Argo
6 3 (0.57%) Progress Chef
7 2 (0.38%) Puppet
8 1 (0.19%) Control-M
8 1 (0.19%) Hadoop YARN
8 1 (0.19%) Kubeflow
Vendors
1 117 (22.29%) Databricks
2 36 (6.86%) Microsoft
3 28 (5.33%) Google
4 18 (3.43%) Informatica
5 13 (2.48%) Oracle
6 10 (1.90%) Red Hat
6 10 (1.90%) SAP
7 8 (1.52%) Palantir
8 7 (1.33%) Ab Initio
9 6 (1.14%) Cloudera
9 6 (1.14%) Foundry
9 6 (1.14%) Talend
10 4 (0.76%) Facebook
10 4 (0.76%) IFS
10 4 (0.76%) Twitter
11 3 (0.57%) Alteryx
11 3 (0.57%) Instagram
11 3 (0.57%) Splunk
11 3 (0.57%) TIBCO
11 3 (0.57%) YouTube