AI Evaluation & Data Engineer for LLM Metrics

We are looking for AI Evaluation & Data Engineering Specialists to design, curate, and operationalize datasets and evaluation frameworks for AI product performance assessment.

This role involves working with large language models (LLMs), human raters, and automation tools to measure model accuracy, correctness, and usability.

Key Responsibilities

Develop and apply data labeling and scoring guidelines based on Google’s evaluation framework.

Implement LLM-judge calibration workflows to align automated and human evaluations.

Perform error analysis, drift detection , and regression testing of AI model outputs.

Collaborate with automation engineers to integrate datasets into evaluation pipelines.

Support rater training , inter-rater reliability checks, and dataset validation reviews.

Manage data quality assurance and documentation for contributions to Google-maintained repos...