Hiring - Data Engineer - Remote
Job Tittle: Data Engineer
Job Type: W2 Contract
Location: 100% Remote must be in the East coast.
Duration: 24-month contract
Lead/Senior Big Data Developer - This person will not be leading the team or have any direct reports but will be a senior developer on the team who can provide technical mentorship to other engineers. Preference is to start with candidates local to DC/MD/VA and NYC. This is a 100% remote role and candidates will not be required onsite regularly.
- Prior to an interview with the client, candidates must complete a Glider assessment, which I will send to candidates directly.
1. Exp programming in Scala.
2. Strong experience in big data technologies like AWS EMR and Apache Spark
3. Strong experience with serverless technologies like AWS DynamoDB and AWS Lambda
- Chosen database is AWS Aurora
4. Technical experience in all the areas listed below:
- Experience working with JSON files as data will be coming in as JSON files
- Ability to write complex SQL queries
- Strong experience in performance tuning and optimization
- Strong unit testing using JUnit or ScalaTest is minimum expectation, data testing experience would be great
- Git/Maven/Gradle
Tech Stack:
Scala is main programming language for the team.
Aurora is database.
SQL is backend database
ETL process is Scala/Spark on EMR clusters.
Code reviews are large aspect of team culture.
Agile environment with 2-week sprints
Job Responsibilities
• Understand complex business requirements
• Design and develop ETL pipeline for collecting, validating and transforming data according to the specification
• Develop automated unit tests, functional tests and performance tests.
• Maintain optimal data pipeline architecture
• Design ETL jobs for optimal execution in AWS cloud environment
• Reduce processing time and cost of ETL workloads
• Lead peer reviews and design/code review meetings
• Provide support for production support operations team
• Implement data quality checks.
• Identify areas where machine learning can be used to identify data anomalies
Experience & Qualifications
• 7+ years of experience in programming language Java or Scala
• 7+ years of experience in ETL projects
• 5+ years of experience in big data projects
• 3+ years of experience with API development (REST API’s)
• Believes in Scrum/Agile, and has deep experience delivering software when working on teams that use Scrum/Agile methodology
• Strong and creative analytical and problem-solving skills
Required Technical Skills & Knowledge
• Strong experience in Java or Scala
• Strong experience in big data technologies like AWS EMR, AWS EKS, Apache Spark
• Strong experience with serverless technologies like AWS Dynamo DB, AWS Lambda
• Strong experience in processing with JSON and csv files
• Must be able to write complex SQL queries
• Experience in performance tuning and optimization
• Familiar with columnar storage formats (ORC, Parquet) and various compression techniques
• Experience in writing Unix shell scripts
• Unit testing using JUnit or ScalaTest
• Git/Maven/Gradle
• Code Reviews
• Experience with CI/CD pipelines
• Agile
The following skills a plus:
• AWS Cloud
• BPM/ AWS Step Functions
• Python scripting
• Performance testing tools like Gatling or JMeter
Nice to have skills:
- AWS Aurora
- Data testing
Thanks & Regards
Javid Ahmad
Sr. Technical Recruiter
Tigerbells LLC
Suite 52, 1405 Chews Landing Rd.
Laurel Springs, NJ 08021
Phone: +1 609 759 1987
LinkedIn: linkedin.com/in/javid-ahmad-598111158
Email: javid@tigerbells.com