Shivani  ·  Senior Python / AWS Data Engineer  ·  5+ yrs

Overview: Developed a tool to find and generate email addresses using first name, last name, and company domain. Responsibilities: Engineered Python scripts for extracting, converting (CSV/JSON), and loading data into Redshift. Designed and implemented ETL scripts using AWS Lambda, Glue, and crawlers for processing millions of social media records. Developed a framework for cleaning source data and conducted data transformations based on specific scenarios. Deployed AWS services using CloudFormation, Ansible, and Terraform for infrastructure as code. Migrated the system to Python, RDS PostgreSQL, AWS Redshift, AWS Lambda, Glue, and implemented job scheduling with Step Functions. Implemented parallel file processing and optimized long-running queries for enhanced database performance.

Overview: Implemented a Strategic ERP system leveraging SQL, Python, and Jaspersoft for advanced reporting and analytics. Responsibilities: Designed scalable and secure database schema and optimized SQL queries, conducting performance tuning. Developed backend logic, integrated ERP modules, and created ETL processes for data integration. Designed and developed interactive reports and dashboards using Tableau and PowerBI. Analyzed data for actionable insights, validated reports, and conducted testing to ensure system quality. Gathered requirements and facilitated stakeholder communication. Managed system deployment, security, and performance.

Overview: Expertise in data transition from IBM Infosphere DataStage to next-gen products like Talend, Golden Gate, Python, and AWS. Responsibilities: Engineered Python scripts to extract, convert, and load data. Designed and implemented an ETL pipeline for processing data from diverse sources to AWS RDS and CRM systems, storing it in a data lake. Extracted data from multiple sources, applied transformations, and loaded it into the database. Developed and maintained Python scripts for full and incremental data processing. Implemented the entire back-end business logic and requisite database operations. Executed a strategic migration plan, transitioning the Data Warehouse from an Oracle platform to AWS Redshift. Implemented security best practices, safeguarding data and controlling access based on IAM policies.