Data Engineer

Glencore

, headquartered in Baar, Switzerland is one of the world's leading integrated producers and marketers of commodities that industries around the world need. Glencore has worldwide activities in the production, sourcing, processing, refining, transporting, storage, financing and supply of metals and minerals, energy products, and agricultural products. We strive to be a reliable and competitive partner in the markets in which we serve and to support our customers and suppliers at each stage of their expansion and development.

, a US-based branch of Glencore AG, is a wholly owned subsidiary of Glencore International AG, where it employees approximately 300 people.

We are seeking a highly skilled and motivated Data Engineer to join our business; you will be part of a global Data Engineering team working on data engineering activities for our trading desks with a focus on the desks in New York.

The role reports to the Head of Data Architecture and Engineering.

- Design, implement, and maintain scalable data engineering pipelines using Python, Spark, and SQL, applying clean coding principles, and optimizing for performance in Azure and Databricks.

- Use good software engineering principles to enable modularity, reliability, and reusability of code.

- Orchestrate complex data workflows using frameworks like Airflow, Databricks workflows (or) equivalent, adopting a code-first approach.

- Provide data engineering support for trading analysts in NY.

- Use CI/CD tools like GitHub actions or equivalent for deploying data pipeline to higher environments efficiently.

- Implement data quality checks, validation frameworks, and monitoring systems to ensure data integrity and reliability.

- Design effective data models, optimize storage and retrieval processes, and efficiently handle semi-structured and unstructured data.

- Create and manage APIs for data access and integration and develop web scraping solutions using leading frameworks when necessary.

- Build scalable data ingestion pipelines for various external data feeds with different formats, velocity, and patterns.

- Collaborate closely with trading analysts in an agile environment, iterating data products in an efficient manner demonstrating strong understanding of the domain context.

- Apply strong analytical and logical problem-solving skills to address complex data engineering challenges and develop innovative solutions.

- Ensure data security and compliance with relevant regulations, implementing best practices for data protection and access control.

- Continuously optimize pipeline performance, evaluate new technologies, and drive innovation in the data engineering ecosystem.

- Maintain comprehensive documentation and contribute to the team's knowledge base, fostering a culture of continuous learning and improvement.

- Collaborate closely with data engineers in London to implement as per standardized guidelines and guardrails.

Qualifications and Requirements:

- Degree educated in either Computer Science, Mathematics, Engineering, or a related field.

- Strong hands-on experience with Python, Spark, SQL and strong knowledge of clean coding principles.

- Strong understanding of Data Governance best practices (DQ handling, Lineage..etc)

- Experience in handling various data ingestion patterns.

- Strong accountability and Entrepreneurialism in day-to-day work.

- Storing working experience with Data Orchestration (Airflow), Containerization (Docker, K8) and CI/CD tools.

- Desirable to have working knowledge in Microsoft Azure and Databricks.

- Advantageous to have experience working with Commodity/Energy trading data.

- Demonstrated ability to work effectively both independently and as part of a collaborative global team following internal Data engineering standards.

- Excellent problem-solving skills and a strong attention to detail.

- Strong stakeholder communication skills.

What We Offer:

- The expected base salary ranges from $120K -$180K. Salary offers are based on a wide range of factors including relevant skills, training, experience, education, and, where applicable, certifications and licenses obtained. Market and organizational factors are also considered. In addition to salary and a generous employee benefits package, successful candidates are eligible to receive a discretionary bonus.