Job Details

Legal Subject Matter Expert (SME) – Quality Review & AI Evaluation

Location

Remote (must be able to support PST hours)

Duration: 3 months with possibility to extend

Hours: ranging from 40-70 hours/week depending on business needs

Our Client is looking for their next Legal SME

Responsibilities

• Incoming Data Quality Control (QC)
• Review tasks and rubrics to confirm they meet campaign standards (complexity,
• feasibility, completeness, clarity) and legal-domain standards (e.g., correct
• terminology, workflow realism, appropriate constraints, reasonable deliverable
• expectations).
• Ensure tasks do not require inaccessible systems or privileged information
• (unless explicitly permitted).
• Validate legal realism: proper framing (jurisdiction/context), reasonable
• assumptions, appropriate deliverable types (memos, issue spotting, drafting,
• research outlines, etc.).
• Identify and correct common legal-domain failure patterns (missing jurisdiction,
• overbroad prompts, impossible deadlines, unclear client goals).
• Provide actionable feedback and track resolution.

Data Collection Process Monitoring

• Monitor campaign health and throughput (submission volume, rework rates,
• bottlenecks, common failure modes).
• Proactively flag blocked contributors/reviewers, recurring quality issues, or
• emerging drift from guidelines.
• Surface systematic issues to Frontier Evals leads and propose concrete
• mitigations (rubric tweaks, examples, quick clarifications).
• Maintain lightweight QC logs/audit trails to support retrospectives and
• downstream analysis.

Contributor Support and Coordination

• Answer contributor questions to improve first-pass quality and reduce iteration
• cycles.
• Coordinate with reviewers on edge cases and escalate ambiguous decisions
• quickly.
• Produce weekly (or ad hoc) summaries: quality themes, top issue categories, and
• recommended guideline updates.

Technical Work (light)

• Run simple Python scripts and SQL queries, spot anomalies, and support
• workflow optimization or automation (you can use ChatGPT!)
• Additional tasks (as needed)
• Draft/update legal-specific examples, checklists, and "good vs. bad" task patterns.
• Participate in QA sweeps for high-risk segments and new task templates.
• Assist tagging/categorization for analysis.

Qualifications / profile

• 3+ years in legal field, including legal ops
• Strong attention to detail and comfort applying rubrics consistently.
• Able to write clear feedback that improves quality quickly.
• Comfortable with spreadsheets; basic SQL/Python helpful but not required