Deepali · Senior Spark / Hadoop Data Engineer · 5+ yrs
Mid-LevelProof of scale
About Deepali
Deepali is a Data Engineer with 5. 5+ years of proven experience in optimizing data pipelines and managing cloud services.
5+ years of commercial experience in
Skills(14)
Why hire Deepali?
Proven ownership in migrating and optimizing data pipelines to AWS Step Functions.
Demonstrated expertise in Redshift cluster management, including encryption and WLM settings.
Consistent recognition for contributions and customer obsession with multiple awards.
Managed encryption across five critical Redshift clusters, minimizing business disruptions.
Successfully migrated all pipelines to Step Functions seamlessly without errors, ensuring uninterrupted functionality.
Spearheaded the migration of S3 files to Glacier for enhanced cost efficiency and S3 management.
Project highlights(4)
Data Pipeline Optimization – Data Engineer
Overview: Currently focused on creating and optimizing data pipelines for efficient data processing. Responsibilities: Created and optimized data pipelines, contributing to efficient data processing. Developed and tuned SQL queries to enhance performance in data processing. Managed and maintained Redshift databases to ensure optimal performance and reliability.
Key outcomes:
Created and optimized data pipelines for efficient data processing.
Developed and tuned SQL queries for performance enhancement.
Managed and maintained Redshift databases to ensure optimal performance.
Redshift Cluster Management – Data Engineer
Overview: Managed encryption across five critical Redshift clusters and orchestrated synchronization for production clusters. Responsibilities: Managed encryption across five critical Redshift clusters, collaborating to minimize business disruptions. Orchestrated synchronization for two production clusters, ensuring seamless encryption and smooth transition of WBR jobs. Proposed and implemented migration of all pipelines to Step Functions seamlessly, ensuring uninterrupted functionality.
Key outcomes:
Managed encryption across five critical Redshift clusters minimizing business disruptions.
Successfully migrated all pipelines to Step Functions seamlessly without errors, ensuring uninterrupted functionality.
Salesforce Data Migration – Big Data Developer
Overview: Responsible for data extraction, transformation, and production job management within a Big Data environment. Responsibilities: Extracted and Imported data from Salesforce to DataLake using Teradata Hadoop Connector, ensuring seamless integration. Transformed data according to CDC logic, maintaining historical data using DataFrame in PySpark. Automated the Salesforce Migration component using UNIX shell scripting for efficiency.
Key outcomes:
Automated Salesforce Migration component using UNIX shell scripting for efficiency.
Optimized HiveQL queries for ORC transactional tables, enhancing query performance and efficiency.
Data Lake Ingestion Pipelines – Hadoop Developer
Overview: Implemented data ingestion pipelines from various sources into a DataLake using CDC logic. Responsibilities: Implemented data ingestion pipelines to extract data from various sources (SFDC, SAP, VISTAAR, EMPOWER, BRAZIL) and loaded it into the A3 DataLake using CDC logic. Utilized Sqoop for importing and exporting data between relational database systems (SQL Server, Oracle) and HDFS.
Key outcomes:
Implemented data ingestion pipelines from various sources using CDC logic.
Optimized Spark code performance by implementing best practices.
Industry experience
SaaS / B2B
Reported in resumeAt a Glance
Single contract. No agency markup confusion.
Typically responds within 4 business hours.
Vetted by WitaristNot sure if this is the right fit?
Tell us your requirements and we'll match you with the best candidates.
Deepali
Big Data Engineer