Remote | Senior Software Engineer - AI Evaluation / Coding Agents — $100–$150/hour

24-MAG LLC makes this freelance opportunity available through its platform; the hiring client is not identified. The role evaluates AI coding systems by reviewing generated code in real-world repositories, developing evaluation rubrics, and supporting scalable assessment workflows. It also involves collaborating with research and engineering teams and reporting findings and recommendations.

Responsibilities:

• Review code produced by AI coding agents
• Assess implementations for correctness, robustness, and maintainability
• Determine whether agents selected appropriate technical approaches
• Identify subtle implementation errors and engineering weaknesses
• Explain clearly why generated solutions succeed or fail
• Design and refine technical evaluation rubrics
• Define criteria for assessing coding-agent performance
• Review and label preference and evaluation data
• Apply consistent quality standards across repeated assessments
• Capture nuanced differences between competing implementations
• Work within substantial real-world software repositories
• Analyse code changes in their broader architectural context
• Evaluate repository-level behaviour rather than isolated snippets
• Review implementation quality using senior engineering judgement
• Identify recurring failure patterns across coding tasks
• Build and maintain pipelines supporting data generation and evaluation
• Improve infrastructure for collecting and reviewing model outputs
• Support scalable evaluation and feedback workflows
• Contribute to reliable processes for repeated technical assessment
• Help translate qualitative engineering judgement into structured systems
• Collaborate with research and engineering teams
• Summarise evaluation findings in clear written reports
• Provide actionable recommendations based on observed model behaviour
• Help improve evaluation methodologies and coding-agent benchmarks
• Communicate complex technical issues in concise, structured language

Requirements:

• 5+ years of hands-on software engineering experience
• Strong proficiency in Python, TypeScript, JavaScript, Go, Java, or another major production language
• Experience working in substantial real-world codebases
• Strong code-review and technical-analysis skills
• Ability to assess implementation correctness and maintainability
• Strong understanding of modern software-engineering practices
• Experience with GitHub-based development workflows
• Familiarity with CI/CD and production engineering processes
• Comfortable analysing unfamiliar repositories and code changes
• Ability to identify subtle technical issues others may overlook
• Strong written communication and structured technical reasoning
• Experience using modern LLMs or AI-assisted coding tools
• Candidates should provide a specific hourly rate expectation
• Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, repositories, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party
• Familiarity with open-source development is valuable
• LLM evaluation or coding-agent experience is advantageous
• Experience with RLHF, preference data, rubric design, or post-training is useful but not required