AI Safety Data Scientist

Aquent is partnering with a global leader in audio streaming and innovative technology, a company dedicated to enriching lives through sound. This organization is at the forefront of developing cutting-edge AI experiences, from conversational interfaces to personalized recommendations. Join a team where your expertise will directly shape the future of responsible AI, ensuring user trust and product integrity on a massive scale. This is an unparalleled opportunity to make a profound impact on how millions interact with advanced AI systems, contributing to a safer and more reliable digital experience.

About the Role

We are seeking an experienced Data Scientist to join a pioneering team focused on AI safety. In this critical role, you will be instrumental in identifying, measuring, and mitigating risks within advanced conversational and agentic AI products. You will transform emerging safety challenges into robust, scalable measurement and monitoring systems, directly influencing policy, model development, and product enhancements. Your work will not only safeguard user experience but also drive the ethical evolution of AI, making a tangible difference in the deployment of responsible technology.

What You'll Do

• Develop and scale comprehensive risk monitoring and safety measurement systems across diverse AI features, including conversational agents, recommender systems, and tool-using AI.
• Design sophisticated safety metrics and evaluation frameworks for production systems, encompassing false positive/negative rates, safety risk prevalence, and precise decision rubrics.
• Inspect, debug, and optimize Python data analysis scripts and SQL pipelines, leveraging advanced LLM coding tools such as Claude or Codex to accelerate analytical workflows.
• Calibrate LLM-as-a-judge evaluation workflows, meticulously label evaluation datasets, and construct impactful reporting dashboards and visualizations for leadership and key product stakeholders.
• Collaborate cross-functionally with Product, Engineering, and Trust & Safety teams, translating real-world production insights into actionable safety mitigations and refined safety policies.
• Clearly and concisely communicate complex analytical findings and measurement metrics to both technical and non-technical audiences, fostering informed decision-making.

What You'll Bring

• Proven experience delivering safety evaluations, risk metrics, or mitigations for a live AI or machine-learning product.
• Strong proficiency in Python and SQL for data analysis, with the technical acumen to independently inspect, critique, and debug pipeline code and queries.
• Hands-on experience designing robust evaluation datasets, clear rubrics, and effective measurement metrics.
• A knack for structuring ambiguous problems and adeptly defining success criteria, methodologies, and tradeoffs with diverse stakeholders.
• Experience working cross-functionally across multiple domains, such as research, engineering, product, policy, or Trust & Safety.
• Exceptional written communication skills and a demonstrated ability to translate analytical research findings into clear, impactful product or policy actions.
• Minimum 2 years of direct AI safety data science or safety evaluation experience, OR 3 to 5+ years of broader Data Science experience with foundational or project-based safety involvement.

Even Better If You Have

• Experience calibrating LLM judges or developing human-in-the-loop evaluation processes.
• Experience evaluating complex multi-turn or tool-using agentic systems.
• Experience with multilingual evaluation methodologies.

About Skill

Skill connects the best professional, IT, engineering, financial and administrative talent with the world's biggest brands. Our eligible talent get access to benefits such as health benefit contributions, retirement plans with match and flexible spending accounts.

Skill is an equal-opportunity employer. We evaluate qualified applicants without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics. We're about creating an inclusive environment-one where different backgrounds, experiences, and perspectives are valued, and everyone can contribute, grow their careers, and thrive.

#LI-CF1