Nitesh Madan  ·  Senior AWS Spark Data Engineer  ·  5+ yrs

Overview: This project supports Mastercard Inc., the second-largest payment-processing corporation worldwide, by providing services for payment transaction processing. Responsibilities: Understood upstream source nature and business cases to ensure data quality and relevance. Monitored multiple batches and executed various scripts across several servers. Performed initial data loads using Initial_Dataload.sql, checked table structures, and filtered data from files. Created Standard Operating Procedures (SOP) for automation, enhancing operational efficiency.

Overview: This project supported UniPol, a global manufacturing company specializing in investment casting technology for automotive and aerospace industries. Responsibilities: Understood upstream source data and business requirements to facilitate data processing. Connected to a PostgreSQL instance using workbench for database operations. Performed initial data loads using Initial_Dataload.sql, checking table structures and sample data. Copied all dependencies to EMR and created HBase tables using Phoenix scripts. Developed Spark jobs to process data, supporting data transformation workflows.

• This role involved developing and executing test plans for quality assurance, monitoring test scripts, and identifying/remedying production defects.

• Developed and executed test plans to ensure all project objectives were met.
• Implemented and monitored test scripts to assess functionality, reliability, performance, and quality.
• Identified and remedied defects within the production process, contributing to system stability.
• Imported structured data from RDBMS to HDFS using Sqoop and performed necessary transformations.
• Processed data into HDFS by developing solutions, analyzing data using Spark and Hive, and producing summary results.