ML/LLM Operations Engineer job opportunity at Evolent Health, Inc..

Home Jobs In United States Of America Tag: Job ML/LLM Operations Engineer position at Evolent Health, Inc.


DatePosted 26 Days Ago bot
Evolent Health, Inc. ML/LLM Operations Engineer
Experience: 2-years
Pattern: full-time
Country:
apply Apply Now
Salary:
Status:

Job

Copy Link Report
degreeOND
loacation Work at Home, United States Of America
loacation Work at Home....United States Of America

Your Future Evolves Here Evolent partners with health plans and providers to achieve better outcomes for people with most complex and costly health conditions. Working across specialties and primary care, we seek to connect the pieces of fragmented health care system and ensure people get the same level of care and compassion we would want for our loved ones. Evolent employees enjoy work/life balance, the flexibility to suit their work to their lives, and autonomy they need to get things done. We believe that people do their best work when they're supported to live their best lives, and when they feel welcome to bring their whole selves to work. That's one reason why diversity and inclusion are core to our business. Join Evolent for the mission. Stay for the culture. What You’ll Be Doing: We are seeking a skilled ML/LLM Operations Engineer to join our Data Science team at Evolent Health to ensure our AI systems deliver consistent, reliable, and compliant results in healthcare settings. This role is perfect for someone who thrives at the intersection of machine learning, operations, and healthcare compliance. The role combines deep understanding of LLM behavior and evaluation with a meticulous approach to monitoring, quality assurance, and regulatory compliance in healthcare applications.   Collaboration Opportunities: This position will play a critical role partnering with our Data Science and Engineering teams while also interacting with cross-functional organizations including DevOps, Compliance, Quality Assurance, Clinical Support, and Product Management to ensure our AI systems operate reliably and meet all healthcare industry requirements.   What You Will Be Doing: Develop and maintain standardized evaluation frameworks to consistently measure LLM performance across relevant healthcare metrics Build monitoring systems using Logfire to track AI model performance, detect drift, and alert the team to anomalies Create testing infrastructure for prompt versions, model selection, and quality assurance processes Design and implement audit sampling processes for continuous quality monitoring and clinical review workflows Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare Optimize LLM operations through intelligent model selection, prompt engineering, and cost management strategies Support the transition from successful POCs to production-ready services with appropriate testing and validation Partner with DevOps on infrastructure requirements while focusing on AI-specific monitoring and optimization Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems Collaborate with Clinical Support Liaison to incorporate clinical feedback into system improvements Prepare regular reports on AI system quality, performance metrics, and compliance status   Required Qualifications: Bachelor's or master's degree in computer science, data science, or related field 2+ years of experience with Python development and at least one production LLM implementation Strong proficiency in SQL for complex log analysis and metrics generation Demonstrated experience with LLM APIs and frameworks (experience with PydanticAI, LangChain, or similar) Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting Understanding of LLM behavior, prompt engineering, and common failure modes in production Experience building evaluation or testing frameworks for AI/ML systems Strong communication skills for cross-functional collaboration Experience with healthcare AI applications and compliance requirements is preferred Familiarity with multiple LLM providers (OpenAI, Anthropic, Google, Azure) is preferred Knowledge of Pydantic ecosystem including PydanticAI and Logfire is preferred Understanding of LLM evaluation metrics and methodologies is preferred Experience building tools for non-technical users is preferred Basic knowledge of containerization (Docker) for local testing and development is preferred Experience with cloud environments (AWS, Azure) as a user is preferred Understanding of API rate limiting, quota management, and cost optimization strategies is preferred Knowledge of CI/CD concepts for ML model deployments is preferred Experience with regulatory compliance and audit processes is preferred Excellent documentation skills and attention to detail is preferred To ensure a secure hiring process we have implemented several identity verification steps, including submission of a government issued photo ID. We conduct identity verification during interviews, and final interviews may require onsite attendance. All candidates must complete a comprehensive background check, in-person I-9 verification, and may be subject to drug screening prior to employment. The use of artificial intelligence tools during interviews is prohibited and monitored. Misrepresentation will result in immediate disqualification from consideration. Technical Requirements: We require that all employees have the following technical capability at their home: High speed internet over 10 Mbps and, specifically for all call center employees, the ability to plug in directly to the home internet router. These at-home technical requirements are subject to change with any scheduled re-opening of our office locations.  Evolent is an equal opportunity employer and considers all qualified applicants equally without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, or disability status. If you need reasonable accommodation to access the information provided on this website, please contact recruiting@evolent.com for further assistance. The expected base salary/wage range for this position is $120,000 - 140,000. This position is also eligible for a bonus component that would be dependent on pre-defined performance factors. As part of our total compensation package, Evolent is proud to offer comprehensive benefits (including health insurance benefits) to qualifying employees. All compensation determinations are based on the skills and experience required for the position and commensurate with experience of selected individuals, which may vary above and below the stated amounts.

Other Ai Matches

Field Medical Director, Radiation Oncologist Applicants are expected to have a solid experience in handling Radiation Oncologist related tasks
Sr Auditor, Risk Advisory and Assurance (IT Audit) Applicants are expected to have a solid experience in handling Risk Advisory and Assurance (IT Audit) related tasks
Sr Tech Analyst, HR Information Systems Applicants are expected to have a solid experience in handling HR Information Systems related tasks
VP, Cardiovascular Medicine Applicants are expected to have a solid experience in handling Cardiovascular Medicine related tasks
Account Manager Applicants are expected to have a solid experience in handling Job related tasks
Software Engineer Applicants are expected to have a solid experience in handling Job related tasks
Managing Director, Strategic Accounts (California) Applicants are expected to have a solid experience in handling Strategic Accounts (California) related tasks
Sr. Director, Software Engineering Applicants are expected to have a solid experience in handling Software Engineering related tasks
ML/LLM Operations Engineer Applicants are expected to have a solid experience in handling Job related tasks
Correspondence Policy Specialist Applicants are expected to have a solid experience in handling Job related tasks
Sr Advisor, Practice Performance (Hawaii Based) Applicants are expected to have a solid experience in handling Practice Performance (Hawaii Based) related tasks
Sr Director, Strategic Accounts (New York) Applicants are expected to have a solid experience in handling Strategic Accounts (New York) related tasks
SaaS Administrator, Enterprise Applications Applicants are expected to have a solid experience in handling Enterprise Applications related tasks
Software Engineer - Backend Applicants are expected to have a solid experience in handling Job related tasks
Field Medical Director, Oncology Applicants are expected to have a solid experience in handling Oncology related tasks
Field Medical Director, Cardiology Applicants are expected to have a solid experience in handling Cardiology related tasks
Field Medical Director, Radiology - (MD/DO) Applicants are expected to have a solid experience in handling Radiology - (MD/DO) related tasks
Associate Director, Provider Engagement - Oncology Applicants are expected to have a solid experience in handling Provider Engagement - Oncology related tasks
Associate Director, Strategic Accounts Applicants are expected to have a solid experience in handling Strategic Accounts related tasks
Clinical Reviewer, Nurse Practitioner/Physician Assistant (Oncology) Applicants are expected to have a solid experience in handling Nurse Practitioner/Physician Assistant (Oncology) related tasks
Field Medical Director, Cardiology Applicants are expected to have a solid experience in handling Cardiology related tasks
Sr Quality Specialist (Accreditation) Applicants are expected to have a solid experience in handling Job related tasks
Lead Software Engineer Applicants are expected to have a solid experience in handling Job related tasks