Kopal Ghosh

Currently leading data engineering development for critical banking applications, focusing on robust data pipelines and real-time streaming solutions to enhance operational efficiency and reporting.

• Engineered and optimized Hive queries and Unix shell scripts for comprehensive data analysis and automated report generation, ensuring timely delivery to stakeholders.
• Developed high-performance Spark solutions using Scala, Dataframes, and Spark SQL API to accelerate data processing for interactive queries and ingesting terabytes of data into MongoDB.
• Designed and implemented advanced ETL processes and data models, improving reporting efficiency and data availability through Kafka-based real-time streaming solutions.
• Analyzed structured, semi-structured, and unstructured data to ensure quality and completeness, extracting critical business value for downstream applications.

Developed and optimized Apache Spark-based data pipelines and REST APIs for Mastercard's card processing and settlement domain, enhancing data access and integration.

• Built and maintained robust data pipelines using Apache Spark for structured and unstructured data, ensuring high availability and integrity for critical transaction data.
• Developed REST APIs for seamless data access and integration with third-party platforms, improving system interoperability and data exchange capabilities.
• Optimized database indexing and query performance, reducing response times by 20% for critical card transaction data analysis.
• Processed and transformed large datasets (AVRO, PARQUET, ORC, CSV, JSON, XML) using Spark, providing real-time serviceability insights during crisis periods.
• Managed CI/CD processes by monitoring Spark jobs via Spark UI, pushing code to GIT, conducting rigorous code reviews, and automating deployments with Jenkins.

Contributed to the ESA Settlement project, developing Spark-based solutions for end-to-end data integration and analysis, transforming raw data into actionable insights for banking operations.

• Led end-to-end data integration for the Autowrite project, pulling data from Windows-based applications via API and establishing connectivity with Synapse, JTux, and JNI.
• Automated data verification using Postgres, Toad, and Unix, streamlining the data processing pipeline for settlement information files.
• Implemented and managed CI/CD processes, including Jenkins pipelines, Git code management, peer reviews, and Chef changes for efficient deployment.
• Designed and deployed NIFI pipelines for automated, scheduled data ingestion, processing large volumes of structured and unstructured datasets.
• Developed Spark code to cleanse, merge, and process diverse datasets, ensuring data quality and readiness for storage in Hive, HBase, or S3 for upstream consumption.

Designed and implemented APIs for E-Wallets and Core Billing applications, enabling mobile payments and credit card billing while ensuring high-quality deliverables.

• Automated end-to-end API testing for E-Wallet platforms (Samsung Pay, Apple Pay, Amazon Pay, Android Pay) using Cucumber and Java, ensuring robust functionality for digital payment services.
• Developed and automated data validation processes for Oracle and PostgreSQL databases, enhancing data integrity for financial transactions.
• Managed and monitored Jenkins jobs for continuous integration and deployment, improving delivery efficiency for E-Wallets and Core Billing applications.
• Utilized the CRAFT framework to design and execute batch jobs for over 600 test cases on credit card operations via TS2, ensuring comprehensive backend application testing.
• Supported core billing operations by handling complex calculations for minimum payments, finance charges, late fees, and membership fees, contributing to accurate billing cycles.

Recognized for outstanding contributions and performance.

Awarded for dedicated focus and swift resolution in critical tomcat migration efforts.

Recognized for significant contributions to the 'Priceless Planet Tree' initiative.

Awarded for demonstrating exceptional focus and urgency on the Autowrite Project.