Senior Data Engineer

Job description

Job responsibilities

Data extraction and integration:

In this role, you will design, build, and maintain code to extract data adapting your methods based on the environment. You'll take ownership of the methodologies used for data extraction, regularly reviewing and refining them to ensure their effectiveness. Your expertise will enable you to rapidly understand and evaluate various databases, structured files and APIs, considering their structure, documentation, and contents. You'll efficiently handle complex datasets, extracting data from large-scale NHS datasets, publicly available health data, and commercial or research environments. At the HEU we regularly work with a wide range of data, examples include SUS, Fingertips, CPRD, and OpenSafely.

Data workflow management:

Managing the flow of data through pipelines involves ensuring the development and maintenance of reproducible analytical pipelines (RAPs). Your tasks will include scheduling jobs, monitoring performance, handling errors, testing, and automating complex processes. When querying large datasets, even in environments where control may be limited, you will employ techniques that continue to ensure efficiency.

Data processing and transformation:

Planning, developing, and evaluating methods and processes for transforming and cleaning data is vital to producing valuable insights. Collaborating closely with colleagues, youll appreciate their data requirements, allowing you to direct the processing and transformation of raw data into formats ready for analysis. As part of this process, youll produce flexible and replicable code to ensure efficiency, effectiveness, and security.

Data storage and management:

Your responsibilities also extend to efficiently storing and managing large volumes of data. This involves selecting appropriate storage solutions for performance and developing comprehensive documentation for assets and analyses. You will also support with ensuring datasets are held in an appropriately secure environment and destroyed when appropriate.

Quality assurance and governance:

You will always maintain security and the confidentiality of personal and commercially sensitive data in line with the relevant information governance (IG), standards and policies. Additionally, you will investigate data quality issues, understanding how inconsistencies could affect ability to analyse the data you extract. You will provide work arounds when possible and articulate inconsistencies and solutions to the team and clients. You will support the team to implement techniques to ensure efficient queries; advising on queries across the entire team.

Data analysis:

You'll also engage in data analysis tasks from time to time. Collaborating closely with the Data Scientists in our team, you will use creative approaches to extract valuable information to provide clear insights to decision-makers.

Collaboration and development:

Collaborative coding approaches through version control and peer review will enable our work to be of the highest quality. You will ensure client requirements are met by supporting project managers with required documentation and delivering on time and within budget. You will instill a culture of learning and knowledge sharing; you will develop and maintain your specialist knowledge whilst providing educational opportunities to enhance colleagues skills in your areas of expertise.

Person Specification

Experience

• Considerable experience of extracting data, manipulating, understanding, transforming, wrangling, cleaning, and storing health data.
• Ability to write well-designed code (e.g. SQL or Python) which follows good coding standards.

Skills

• Demonstrates leadership skills in approach to data engineering tasks and delivery of projects on time and within budget.
• Fosters a culture of learning and improvement within the team.
• Master's degree in a related STEM subject or equivalent level of experience of working at a similar level in a similar setting for at least 3 years.

Certificate of Sponsorship

Applications from job seekers who require current Skilled worker sponsorship to work in the UK are welcome and will be considered alongside all other applications. For further information visit the UK Visas and Immigration website.

From 6 April 2017, skilled worker applicants, applying for entry clearance into the UK, have had to present a criminal record certificate from each country they have resided continuously or cumulatively for 12 months or more in the past 10 years. Adult dependants (over 18 years old) are also subject to this requirement. Guidance can be found here Criminal records checks for overseas applicants.

Employer details

Employer name

Midlands and Lancashire Commissioning Support Unit

Address

10 South Colonnade

10 South Colonnade Canary Wharf

London

E14 5EA

https://www.midlandsandlancashirecsu.nhs.uk/