Karan · Azure Databricks Data Engineer · 4+ yrs
Mid-LevelAbout Karan
Azure data engineer (SQL optimisation focus) with delivery on sheet-metal manufacturing data integration, enterprise data lake + analytics platform on ADF + Databricks, and financial transaction reporting + risk analysis.
4+ years of commercial experience in
Skills(19)
Why hire Karan?
Built data platform ingesting manufacturing/production/quality data from shop-floor systems.
Delivered enterprise data lake + analytics on Azure Data Factory + Databricks across multiple departments.
Shipped financial transaction reporting + risk analysis system.
Azure ADF + Databricks data engineering
Manufacturing + finance domain spread
Project highlights(3)
Sheet Metal Manufacturing Data Integration & Optimization Platform – Data Engineer (SQL Optimization & ADF Pipeline Monitoring)
- Data platform ingesting manufacturing, production and quality data from multiple shop-floor systems into a centralised analytics system.
- Designed and monitored ADF pipelines ingesting production, machine and quality data from on-premise systems into ADLS.
- Optimised complex SQL queries and stored procedures for production-efficiency, defect-rate and material-usage reporting.
- Implemented incremental and scheduled data loads using watermarking and triggers in ADF for near-real-time data availability.
- Monitored pipeline executions, handled failures/retries/alerts and performed root-cause analysis.
- Validated data accuracy with manufacturing and QA teams.
Key outcomes:
Built incremental ADF pipelines with watermarking + triggers for near-real-time manufacturing data availability.
Enterprise Data Lake & Analytics Platform – Azure Data Engineer
- Centralised data platform for multiple departments using Azure Data Factory + Databricks.
- Designed end-to-end ETL pipelines in ADF ingesting from multiple enterprise sources.
- Implemented transformations in Databricks PySpark with Bronze/Silver/Gold layers.
- Optimised SQL queries and Spark jobs for performance and cost efficiency.
- Implemented data validation, schema enforcement and error handling.
- Monitored pipelines via Azure Monitor + Log Analytics.
Key outcomes:
Implemented Bronze/Silver/Gold medallion architecture in Databricks PySpark for enterprise multi-source data lake.
Financial Transaction Reporting & Risk Analytics System – Azure Data Engineer
- System processing financial transactions, generating reports and supporting fraud/risk analysis.
- Built secure ETL pipelines ingesting transaction data from banking and payment systems.
- Implemented data masking, encryption and role-based access for sensitive financial data.
- Developed aggregation logic for daily/monthly/yearly financial reporting.
- Implemented reconciliation and data-quality checks.
- Supported auditors and compliance teams with traceable, auditable data outputs.
Key outcomes:
Built secure financial-transaction ETL with data masking, encryption and audit-traceable outputs.
Industry experience
FinTech
1 project- •Financial Transaction Reporting & Risk Analytics System— Azure Data EngineerAzure Data Factory · Azure Databricks · PySpark · SQL +2
Manufacturing & Industrial
3 projects- •Sheet Metal Manufacturing Data Integration & Optimization Platform— Data Engineer (SQL Optimization & ADF Pipeline Monitoring)Azure Data Factory · ADLS · SQL · SQL Server +1
- •Enterprise Data Lake & Analytics Platform— Azure Data EngineerAzure Data Factory · Azure Databricks · PySpark · Delta Lake +3
- •Financial Transaction Reporting & Risk Analytics System— Azure Data EngineerAzure Data Factory · Azure Databricks · PySpark · SQL +2
Cybersecurity
1 project- •Financial Transaction Reporting & Risk Analytics System— Azure Data EngineerAzure Data Factory · Azure Databricks · PySpark · SQL +2
At a Glance
Single contract. No agency markup confusion.
Typically responds within 4 business hours.
Vetted by WitaristNot sure if this is the right fit?
Tell us your requirements and we'll match you with the best candidates.
Karan
Azure Data Engineer