Job description

Job Title: Data Engineer

Job Location: Morris Plains, NJ (remote to start)

Key Technology: Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop

Job Responsibilities:

• Work with business and technical leadership to understand requirements.
• Design to the requirements and document the designs.
• Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark
• Do data modeling as needed for the requirements.
• Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive
• Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or Hadoop Edge Nodes
• Implement Hadoop job orchestration using Shell scripting, Apache Oozie, CA7 Enterprise Scheduler and Airflow
• Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment
• Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.

Skills and Experience Required:

• Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
• Experience of Ab Initio is a bonus.
• Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures
• Excellent work experience on Hadoop as data warehouse/Data Lake implementations
• Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory)
• Unix/Linux Shell scripting (KSH) and basic administration of Unix servers
• CA7 Enterprise Scheduler
• Experience with AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch)
• Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration)
• Experience in Jira and Confluence
• Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
• Health care domain knowledge is a plus.

Job Type: Full-time

Salary: $70,000.00 - $75,000.00 per year

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Employee assistance program
• Employee discount
• Flexible schedule
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Parental leave
• Professional development assistance
• Referral program
• Relocation assistance
• Retirement plan
• Tuition reimbursement
• Vision insurance

Schedule:

• Monday to Friday

Work Location: One location

Speak with the employer
+91 7036439638

colinoncars.com is the go-to platform for job seekers looking for the best job postings from around the web. With a focus on quality, the platform guarantees that all job postings are from reliable sources and are up-to-date. It also offers a variety of tools to help users find the perfect job for them, such as searching by location and filtering by industry. Furthermore, colinoncars.com provides helpful resources like resume tips and career advice to give job seekers an edge in their search. With its commitment to quality and user-friendliness, colinoncars.com is the ideal place to find your next job.