Shivani is a Data Engineer with 4. 2 years of experience specializing in data pipeline development and management.
Designed and developed data pipelines for scalability, reliability, and security.
Expertise in query optimization for Redshift and PostgreSQL.
Developed and deployed robust ETL pipelines for data processing across multiple projects.
Successfully migrated data systems to AWS cloud infrastructure and next-gen platforms.
Implemented data enrichment and data lineage tracking for improved data quality and governance.
Automated data extraction and loading processes into data warehouses like Redshift.
Optimized data queries and managed database maintenance for improved performance.
Successfully transitioned systems to next-gen products using modern data tools.
Overview: Developed a tool to automate email address extraction using first name, last name, and company domain. Responsibilities: Extracted valuable information, performed data enrichment, and maintained data lineage.
Key outcomes:
Successfully automated the process of finding and extracting email addresses.
Automated data loading into Redshift.
Overview: Focused on transitioning to next-gen products leveraging Talend, Golden Gate, Python, and AWS services. Responsibilities: Developed robust ETL pipelines for comprehensive data processing.
Key outcomes:
Successfully transitioned systems to next-gen products using modern data tools.
Developed and deployed comprehensive ETL pipelines for data processing.
Overview: Managed data ingestion from multiple channels (email, FTP, Amazon S3) into ETL pipelines. Responsibilities: Processed collected data to feed into forecasting models.
Key outcomes:
Successfully managed data from diverse sources for forecasting models.
Automated database maintenance and job monitoring.
Overview: Extracted valuable information from PDF files and stored it in JSON format in AWS S3. Responsibilities: Led data extraction, enrichment, and loading processes.
Key outcomes:
Successfully extracted and structured data from unstructured PDF files.
Ensured data quality through enrichment and maintained data lineage.
Shivani
Data Engineer