Job description
Senior Data Engineer 14924 (H)
A leading tech company is seeking a Senior Data Engineer. The successful candidate will establish a fully functional object store compatible with S3 access APIs. The ideal candidate is an expert Senior Data Engineer with Spark, S3, and Rapid prototyping. The company offers a great work environment!
Senior Data Engineer Pay and Benefits:
- Hourly pay: $80/hr
- Worksite: Leading technology and data solutions company (Albany, NY 12201, Open to remote)
- W2 Employment, Group Medical, Dental, Vision, Life, Retirement Savings Program
- 40 hours/week, 3 Month Assignment
Senior Data Engineer Responsibilities:
- Establish a fully functional object store compatible with S3 access APIs which will:
- Provide the ability to define fine-grain access control lists on our content platform objects.
- Support multiple storage tiers in the view of cost optimization and allows moving objects between storage tiers.
- Support event notifications over objects, allowing a subscription model to build data pipelines
- Record all the object-level operations, and expose access logs for audit
- Support multi-part uploads to move handle large objects into the store
- Support replication of objects between different object stores
- Provide comprehensive monitoring and observability metrics, and generate alerts on configured thresholds.
- Create appropriate documentation for the activities above Integrate Iceberg with our content platform, Spark, and Trino which will:
- Provide a central metastore or catalog pluggable with multiple compute engines.
- Support time travel, and automated maintenance like purging, and compression.
- Support native reads on Iceberg, which can be leveraged by compute engines.
- Create appropriate documentation for the activities above.
Senior Data Engineer Qualifications:
- +10 years of relevant experience.
- Senior Data Engineer / Spark / S3 / Rapid prototyping.
- Spark expertise.
- S3 storage and data management.
- Agile.
- Apache Iceberg (Nice to have) or similar.
- Can search and investigate on tools.
- PoC approach implementation experience.
- Delivery Ownership.
- Autonomous.
This is a remote position.
colinoncars.com is the go-to platform for job seekers looking for the best job postings from around the web. With a focus on quality, the platform guarantees that all job postings are from reliable sources and are up-to-date. It also offers a variety of tools to help users find the perfect job for them, such as searching by location and filtering by industry. Furthermore, colinoncars.com provides helpful resources like resume tips and career advice to give job seekers an edge in their search. With its commitment to quality and user-friendliness, colinoncars.com is the ideal place to find your next job.