Smartbridge

• Worked on AWS Data pipeline to configure data loads from S3 to into Redshift.• Using AWS Redshift, I Extracted, transformed, and loaded data from various heterogeneous data sources and destinations.• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.• Involving converting SQL queries into Spark transformations using Spark RDDs, Python.• Working with application teams to install an operating system, Hadoop updates, patches, version upgrades as require.• Monitor performance and optimize SQL queries for reducing engineering time and better efficiency.• Working in an Agile environment, with an ability to accommodate and test the newly proposed changes at any point of time during the release.• Used Apache Kafka to store streaming data to HDFS and implemented PySpark for faster processing of data.• Creating various capacity planning reports using Python packages such as NumPy, Pandas, Matplotlib, SciPy, Scikit-learn, Seaborn and TensorFlow.