Data Pipeline Contracts

Data Pipelines
UK

The following table provides summary statistics for contract job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 1 June 2024 with a comparison to the same period in the previous 2 years.

6 months to
1 Jun 2024
Same period 2023 Same period 2022
Rank 187 291 266
Rank change year-on-year +104 -25 -5
Contract jobs citing Data Pipeline 534 500 909
As % of all contract jobs advertised in the UK 1.24% 0.89% 1.04%
As % of the Processes & Methodologies category 1.45% 0.99% 1.15%
Number of daily rates quoted 395 340 608
10th Percentile £363 £413 £450
25th Percentile £449 £488 £513
Median daily rate (50th Percentile) £525 £550 £600
Median % change year-on-year -4.55% -8.33% +9.09%
75th Percentile £650 £650 £714
90th Percentile £750 £738 £825
UK excluding London median daily rate £500 £525 £550
% change year-on-year -4.76% -4.55% +10.00%
Number of hourly rates quoted 6 5 6
10th Percentile £34.24 £47.75 £27.00
25th Percentile £49.75 £65.00 £28.56
Median hourly rate £62.50 £75.50 £33.75
Median % change year-on-year -17.22% +123.70% -17.68%
75th Percentile £66.25 £82.50 £68.25
90th Percentile £67.50 £87.00 £84.83
UK excluding London median hourly rate £62.50 £55.88 -
% change year-on-year +11.86% - -

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for process or methodology skills.

Contract vacancies with a requirement for process or methodology skills 36,802 50,277 79,372
As % of all contract IT jobs advertised in the UK 85.47% 89.69% 90.61%
Number of daily rates quoted 23,632 34,811 55,790
10th Percentile £300 £325 £345
25th Percentile £413 £438 £431
Median daily rate (50th Percentile) £525 £550 £538
Median % change year-on-year -4.55% +2.33% +7.50%
75th Percentile £638 £650 £644
90th Percentile £750 £750 £738
UK excluding London median daily rate £500 £500 £488
% change year-on-year - +2.56% +9.18%
Number of hourly rates quoted 2,530 1,673 1,868
10th Percentile £12.75 £10.63 £12.50
25th Percentile £16.00 £15.72 £15.63
Median hourly rate £36.50 £36.25 £25.68
Median % change year-on-year +0.69% +41.16% +9.74%
75th Percentile £60.75 £65.00 £50.00
90th Percentile £72.50 £75.00 £65.00
UK excluding London median hourly rate £38.12 £35.00 £20.00
% change year-on-year +8.91% +75.00% -4.76%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Data Pipeline.

Daily rate trend for Data Pipeline in the UK

Data Pipeline
Daily Rate Histogram

Daily rate distribution for jobs citing Data Pipeline over the 6 months to 1 June 2024.

Daily rate histogram for Data Pipeline in the UK

Data Pipeline
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Data Pipeline.

Hourly rate trend for Data Pipeline in the UK

Data Pipeline
Hourly Rate Histogram

Hourly rate distribution of jobs citing Data Pipeline over the 6 months to 1 June 2024.

Hourly rate histogram for Data Pipeline in the UK

Data Pipeline
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 1 June 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +95 479 £550 - 318
London +89 320 £575 +2.22% 184
Work from Home +49 163 £525 -4.55% 148
UK excluding London +48 150 £500 -4.76% 156
North of England +37 54 £500 -8.05% 49
Midlands +45 45 £550 +0.92% 26
West Midlands +32 39 £550 +0.92% 20
North West +18 30 £500 - 32
Yorkshire +35 24 £563 -11.76% 12
South East -18 19 £375 -28.57% 31
Scotland +35 11 £453 +5.96% 19
South West +1 11 £550 -15.38% 16
East of England +11 8 £538 +2.38% 8
East Midlands +16 6 £400 -40.74% 6
Wales +17 1 £500 - 3
Northern Ireland - 1 £425 - 5

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 85 (15.92%) Apache Spark
2 71 (13.30%) Apache Airflow
3 68 (12.73%) Apache
4 4 (0.75%) Confluence
4 4 (0.75%) Luigi
5 3 (0.56%) SharePoint
6 1 (0.19%) ArcGIS
6 1 (0.19%) Jupyter
Applications
1 13 (2.43%) Microsoft Excel
2 6 (1.12%) Microsoft Office
3 4 (0.75%) Microsoft PowerPoint
3 4 (0.75%) Microsoft Project
3 4 (0.75%) MS Visio
4 3 (0.56%) Sketch
Business Applications
1 2 (0.37%) Dynamics CRM
Cloud Services
1 218 (40.82%) AWS
2 210 (39.33%) Azure
3 76 (14.23%) Serverless
4 70 (13.11%) GCP
5 64 (11.99%) Amazon S3
5 64 (11.99%) AWS Lambda
5 64 (11.99%) Azure Data Factory
6 56 (10.49%) Snowflake
7 36 (6.74%) Azure Synapse Analytics
8 35 (6.55%) Power Platform
9 34 (6.37%) Amazon EMR
10 33 (6.18%) dbt
11 32 (5.99%) GitHub
12 30 (5.62%) Azure DevOps
13 27 (5.06%) AWS Step Functions
14 22 (4.12%) Amazon EC2
14 22 (4.12%) Amazon Kinesis
15 13 (2.43%) Amazon MSK
15 13 (2.43%) Azure Logic Apps
16 11 (2.06%) Cloud Computing
Communications & Networking
1 18 (3.37%) Intrusion Detection
1 18 (3.37%) Network Security
2 5 (0.94%) FTP
3 2 (0.37%) SAN
4 1 (0.19%) Firewall
Database & Business Intelligence
1 144 (26.97%) Data Warehouse
2 109 (20.41%) Big Data
3 90 (16.85%) Data Lake
4 58 (10.86%) Amazon Redshift
5 42 (7.87%) NoSQL
6 35 (6.55%) BigQuery
6 35 (6.55%) Power BI
6 35 (6.55%) Relational Database
7 34 (6.37%) SQL Server
8 25 (4.68%) Amazon Athena
9 23 (4.31%) Parquet
10 22 (4.12%) Flink
11 20 (3.75%) Hadoop
11 20 (3.75%) Tableau
12 19 (3.56%) Azure SQL Database
13 15 (2.81%) DynamoDB
14 14 (2.62%) Amazon RDS
14 14 (2.62%) Spark SQL
15 13 (2.43%) Metadata
15 13 (2.43%) PostgreSQL
Development Applications
1 55 (10.30%) Git
2 27 (5.06%) JIRA
3 12 (2.25%) Jenkins
4 8 (1.50%) GitLab
5 7 (1.31%) MLflow
6 5 (0.94%) JMeter
6 5 (0.94%) TestRail
7 4 (0.75%) Deequ
7 4 (0.75%) ERwin
7 4 (0.75%) Mercurial
8 3 (0.56%) Bitbucket
8 3 (0.56%) Postman
9 2 (0.37%) Bugzilla
9 2 (0.37%) ER Studio
9 2 (0.37%) JUnit
9 2 (0.37%) Selenium
9 2 (0.37%) TestNG
9 2 (0.37%) Travis CI
10 1 (0.19%) GoCD
10 1 (0.19%) IBM UrbanCode
General
1 122 (22.85%) Social Skills
2 97 (18.16%) Analytical Skills
3 79 (14.79%) Finance
4 37 (6.93%) Public Sector
5 29 (5.43%) Banking
6 15 (2.81%) Front Office
6 15 (2.81%) Presentation Skills
7 14 (2.62%) Pharmaceutical
8 9 (1.69%) Marketing
9 8 (1.50%) Investment Banking
10 6 (1.12%) Manufacturing
10 6 (1.12%) Retail
11 4 (0.75%) Inclusion and Diversity
11 4 (0.75%) Multimedia
12 3 (0.56%) Back Office
12 3 (0.56%) Financial Institution
12 3 (0.56%) Games
12 3 (0.56%) Law
13 2 (0.37%) Czech Language
13 2 (0.37%) Wearables
Job Titles
1 272 (50.94%) Data Engineer
2 78 (14.61%) Senior
3 61 (11.42%) Developer
4 57 (10.67%) Azure Engineer
5 51 (9.55%) Azure Data Engineer
6 50 (9.36%) AWS Engineer
7 45 (8.43%) Senior Data Engineer
8 43 (8.05%) Architect
9 37 (6.93%) AWS Data Engineer
10 32 (5.99%) Lead
11 27 (5.06%) Analyst
12 22 (4.12%) Python Developer
13 21 (3.93%) Data Scientist
14 19 (3.56%) Data Analyst
14 19 (3.56%) Snowflake Engineer
15 17 (3.18%) Lead Data Engineer
16 16 (3.00%) Python Engineer
17 15 (2.81%) Platform Engineer
18 14 (2.62%) Snowflake Data Engineer
19 13 (2.43%) Cloud Engineer
Libraries, Frameworks & Software Standards
1 74 (13.86%) PySpark
2 60 (11.24%) Kafka
3 35 (6.55%) Pandas
4 26 (4.87%) JSON
5 24 (4.49%) NumPy
6 20 (3.75%) Apache Avro
7 19 (3.56%) Apache NiFi
8 16 (3.00%) SciPy
9 11 (2.06%) TensorFlow
10 10 (1.87%) REST
10 10 (1.87%) scikit-learn
11 8 (1.50%) Elastic Stack
11 8 (1.50%) React
12 7 (1.31%) Node.js
13 6 (1.12%) ARM Templates
13 6 (1.12%) FastAPI
13 6 (1.12%) Flask
13 6 (1.12%) XML
14 5 (0.94%) PyTorch
14 5 (0.94%) Spark Streaming
Miscellaneous
1 28 (5.24%) Algorithms
2 26 (4.87%) Management Information System
3 25 (4.68%) Distributed Systems
4 20 (3.75%) Cyber Defence
4 20 (3.75%) Data Structures
5 19 (3.56%) Derivative
5 19 (3.56%) Distributed Applications
6 11 (2.06%) Greenfield Project
6 11 (2.06%) Linux Command Line
7 10 (1.87%) Self-Motivation
8 9 (1.69%) Distributed Computing
8 9 (1.69%) Onboarding
9 7 (1.31%) Cloud Native
9 7 (1.31%) Hedge funds
9 7 (1.31%) Social Media
10 6 (1.12%) Renewable Energy
11 5 (0.94%) Analytical Mindset
11 5 (0.94%) Life Science
11 5 (0.94%) NHS
11 5 (0.94%) Security Posture
Operating Systems
1 40 (7.49%) Linux
2 31 (5.81%) Windows
3 19 (3.56%) Unix
4 16 (3.00%) Android
4 16 (3.00%) Apple iOS
5 8 (1.50%) Ubuntu
6 1 (0.19%) VMS
6 1 (0.19%) Windows Server
Processes & Methodologies
1 211 (39.51%) ETL
2 176 (32.96%) Analytics
3 174 (32.58%) Data Modelling
4 172 (32.21%) Data Engineering
5 114 (21.35%) CI/CD
6 110 (20.60%) DevOps
7 100 (18.73%) Problem-Solving
8 95 (17.79%) Agile
9 91 (17.04%) Data Science
10 77 (14.42%) Business Intelligence
11 74 (13.86%) Data Integration
12 64 (11.99%) Computer Science
13 62 (11.61%) Machine Learning
14 59 (11.05%) Data Ingestion
15 57 (10.67%) Data Management
16 52 (9.74%) Data Governance
17 49 (9.18%) Data Transformation
18 47 (8.80%) Test Automation
19 46 (8.61%) Visualisation
20 44 (8.24%) Data Analysis
Programming Languages
1 326 (61.05%) Python
2 282 (52.81%) SQL
3 75 (14.04%) Java
4 66 (12.36%) Scala
5 23 (4.31%) C++
6 21 (3.93%) C#
6 21 (3.93%) JavaScript
7 16 (3.00%) T-SQL
8 14 (2.62%) R
9 12 (2.25%) Rust
10 10 (1.87%) Bash
11 7 (1.31%) Go
11 7 (1.31%) TypeScript
12 6 (1.12%) Bicep
13 5 (0.94%) VBA
14 4 (0.75%) Perl
14 4 (0.75%) PowerShell
15 3 (0.56%) Data Analysis Expressions
15 3 (0.56%) MATLAB
15 3 (0.56%) Shell Script
Qualifications
1 94 (17.60%) Security Cleared
2 69 (12.92%) SC Cleared
3 56 (10.49%) Degree
4 40 (7.49%) Computer Science Degree
5 22 (4.12%) DV Cleared
6 14 (2.62%) Master's Degree
7 8 (1.50%) Azure Certification
8 7 (1.31%) AWS Certification
8 7 (1.31%) BPSS Clearance
9 6 (1.12%) CISSP
10 3 (0.56%) AWS Certified Solutions Architect
10 3 (0.56%) CISA
10 3 (0.56%) CRISC
10 3 (0.56%) Microsoft Certification
10 3 (0.56%) PhD
11 2 (0.37%) AWS Certified Developer
11 2 (0.37%) AWS Certified DevOps Engineer
12 1 (0.19%) (ISC)2 CCSP
12 1 (0.19%) CSSLP
12 1 (0.19%) Numerate Degree
Quality Assurance & Compliance
1 95 (17.79%) Data Quality
2 18 (3.37%) Actionable Recommendations
3 14 (2.62%) QA
4 7 (1.31%) GDPR
5 6 (1.12%) Accessibility
6 2 (0.37%) NCSC
6 2 (0.37%) NIST
6 2 (0.37%) PCI DSS
7 1 (0.19%) California Consumer Privacy Act
7 1 (0.19%) ISO/IEC 27001
System Software
1 49 (9.18%) Docker
2 8 (1.50%) Active Directory
2 8 (1.50%) iptables
3 7 (1.31%) HDFS
4 2 (0.37%) VMware Infrastructure
5 1 (0.19%) Virtual Machines
Systems Management
1 55 (10.30%) Terraform
2 48 (8.99%) Kubernetes
3 8 (1.50%) Ansible
3 8 (1.50%) Kibana
3 8 (1.50%) logstash
4 5 (0.94%) Argo
5 3 (0.56%) Progress Chef
6 2 (0.37%) Puppet
7 1 (0.19%) Control-M
7 1 (0.19%) Hadoop YARN
7 1 (0.19%) Kubeflow
Vendors
1 117 (21.91%) Databricks
2 35 (6.55%) Google
2 35 (6.55%) Microsoft
3 18 (3.37%) Informatica
4 14 (2.62%) Red Hat
5 12 (2.25%) Oracle
6 10 (1.87%) SAP
7 8 (1.50%) Ab Initio
8 7 (1.31%) Palantir
9 6 (1.12%) Foundry
10 5 (0.94%) Cloudera
10 5 (0.94%) Talend
11 4 (0.75%) Facebook
11 4 (0.75%) IFS
11 4 (0.75%) Twitter
12 3 (0.56%) Alteryx
12 3 (0.56%) Instagram
12 3 (0.56%) Splunk
12 3 (0.56%) TIBCO
12 3 (0.56%) YouTube