Data Quality Engineer
We are looking for a Data Quality Engineer with 4 to 5 years of experience to build and run data quality controls for an enterprise finance data platform on Azure, Databricks and Snowflake. You will define and automate quality rules, reconciliations and matching checks so that data used for financial control, tax and management reporting can be trusted. Candidates at the upper end of the range may lead a small team of data quality engineers.
KEY RESPONSIBILITIES
Work with business analysts and data owners to define data quality rules for finance data
Build automated quality checks in Databricks pipelines for completeness, validity, consistency and timeliness
Build reconciliations between source systems, the platform and reports, including general ledger to sub-ledger checks
Build matching rules for bank, call account and intra-group reconciliations
Set up exception handling and route failed records to data owners for correction
Build data quality dashboards in Power BI
Record rule definitions and results in Microsoft Purview
Investigate recurring data issues and agree fixes with source system owners
Support testing and production monitoring of quality controls
TECHNICAL SKILLS
Must-have
Databricks, including PySpark and Spark SQL
Advanced SQL
Python
Data quality rule design
Reconciliation and matching logic
Good-to-have
Delta Live Tables expectations
Great Expectations or similar tools
Snowflake
Power BI
Microsoft Purview
DOMAIN KNOWLEDGE
Finance data, including general ledger, sub-ledgers and intra-group balances
Balance sheet substantiation and reconciliation controls
Data quality needs for month-end close, tax and management reporting
CERTIFICATIONS (GOOD TO HAVE)
Databricks Certified Data Engineer Associate
DAMA Certified Data Management Professional (CDMP)
EXPERIENCE AND QUALIFICATIONS
4 to 5 years in data engineering or data quality roles
Built data quality or reconciliation controls on a cloud data platform
Experience with finance or banking data
Degree in engineering or computer applications (B.E., B.Tech or MCA)
Degree in engineering or computer applications (B.E., B.Tech or MCA)