Deepak Patel  ·  AWS Data Engineer  ·  3+ yrs

Overview: Architected and implemented a scalable, secure, and automated data analytics platform on Microsoft Azure for a leading financial institution, enabling real-time insights into key financial metrics, credit risk, and regulatory reporting. Responsibilities: Ingested data from multiple sources (PostgreSQL, APIs, flat files) into the Azure Data Lake Storage (ADLS) Landing zone, handling 50+ GB of data daily. Developed scalable ETL/ELT pipelines in Azure Databricks (PySpark/SQL) to transform data across Landing → Staging → Sanitized → Conformed → Curated layers. Orchestrated workflows and scheduled pipelines using Azure Data Factory (ADF) and Apache Airflow for advanced dependency management.

Overview: Developed and optimized scalable data pipelines on Snowflake to support healthcare business intelligence and reporting needs. Responsibilities: Ingested structured and semi-structured data from PostgreSQL, Workday Prism Analytics, and healthcare systems into Snowflake Landing layer, processing millions of patient records. Built modular dbt models to transform healthcare data across Landing → Staging → Sanitized → Conformed → Curated layers.

Overview: Led data analysis efforts by leveraging Snowflake, Power BI, and AWS services to provide actionable insights on user engagement. Responsibilities: Analyzed and processed large-scale user data (500,000+ records) using SQL and PySpark. Developed and maintained Power BI dashboards to visualize key user engagement metrics.

Overview: This project focused on developing an Enterprise Data Lakehouse to centralize and optimize data management across financial, operational, and HR functions. Responsibilities: Extract, transform, and analyze financial, operational, and HR data from Snowflake, NetSuite, and Workday. Develop interactive dashboards and reports using Power BI to provide actionable insights.

Overview: This project focused on leveraging real-time data to analyze customer behavior and transaction trends for an e-commerce company. Responsibilities: Analyzed real-time customer transaction and behavioral data to generate actionable insights. Designed and maintained interactive dashboards and reports using Power BI.