Software Engineer - Python

Role Overview

We are hiring for one of our clients, seeking a Senior Software Engineer – Python (LLM Evaluation & Repository Validation) to work on a part-time basis. The role involves building verifiable software engineering tasks based on public repository histories using synthetic approaches with human-in-the-loop validation. The projects focus on creating LLM evaluation and training datasets for realistic software engineering problems.

Key Responsibilities:

• Analyze and triage GitHub issues across trending open-source libraries to assess LLM performance in bug-fixing scenarios.
• Set up and configure code repositories, including Dockerization and environment setup, for evaluation purposes.
• Evaluate unit test coverage and quality in public repositories to ensure robust dataset validation.
• Modify and run codebases locally to assess how LLMs interact with real-world software projects.
• Collaborate with researchers to identify repositories and issues that present challenges for LLMs, expanding dataset coverage.

Required Skills & Qualifications:

• Minimum 3+ years of overall software engineering experience with strong Python proficiency.
• Hands-on experience with Git, Docker, and basic software pipeline setup for repository evaluation.
• Ability to navigate and understand complex codebases, including open-source projects.
• Experience contributing to or evaluating open-source projects is a plus.
• Comfort with running, modifying, and testing real-world projects locally in development environments.

More About the Opportunity:

This role offers a unique opportunity to work on cutting-edge AI projects with leading LLMs, contributing to advancements in AI-assisted software development. The work involves blending practical software engineering with AI research to shape the future of developer tools.