Job Details

Evaluate AI-generated Python code for a cutting-edge LLM project. Remote, flexible, impactful, and highly technical work.

This Jobot Consulting Job is hosted by: Eric Kee
Are you a fit? Easy Apply now by clicking the "Apply Now" buttonand sending us your resume.
Salary: $35 - $55 per hour

A bit about us:

Founded nearly a decade ago and based in San Francisco, with teams across the U.S., we are leading the way in AI-powered evaluation and tooling for large language models. We partner with top-tier clients to ensure their AI systems are accurate, functional, and safe, using scalable human-in-the-loop processes and expert technical validation.

Why join us?

• Competitive Compensation: Contract rate DOE, with project bonuses for high performance
• Fully Remote Flexibility: Work from anywhere, on your own schedule
• Challenging Work: Support one of the most innovative LLM projects in the industry
• Technical Growth: Apply and expand your real-world coding, testing, and evaluation skills
• Impactful Work: Help ensure safety, performance, and factual accuracy of AI systems used globally

Job Details

• Evaluate two AI-generated code responses to a given programming prompt
• Verify adherence to prompt instructions and evaluate response structure, completeness, and accuracy
• Test and validate all code snippets using internal or external tools (e.g. Colab, JSFiddle)
• Flag responses with major logic, syntax, or functionality issues and provide concise written feedback
• Write a brief analysis of instruction-following and factual correctness
• Rank responses based on quality and provide clear justification
• Submit "proof of work" code blocks or screenshots for each evaluated response
• Escalate items that exceed time thresholds or require out-of-scope expertise
• Strong background in Python; bonus if experienced with JavaScript, C++, or Java
• Experience in writing or validating unit tests, code debugging, or LLM evaluation
• Proven ability to identify code functionality issues and edge case handling
• Comfort evaluating unfamiliar prompts or flagging complexity above scope
• Strong writing and documentation skills to explain technical decisions clearly

Interested in hearing more? Easy Apply now by clicking the "Apply Now" button.

Jobot is an Equal Opportunity Employer. We provide an inclusive work environment that celebrates diversity and all qualified candidates receive consideration for employment without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

Sometimes Jobot is required to perform background checks with your authorization. Jobot will consider qualified candidates with criminal histories in a manner consistent with any applicable federal, state, or local law regarding criminal backgrounds, including but not limited to the Los Angeles Fair Chance Initiative for Hiring and the San Francisco Fair Chance Ordinance.