Loại hình
Toàn thời gian
Hình thức
Tại văn phòng
Cấp bậc
Nhân viên
Ngành nghề
Ngành nghề khác
Phòng ban
VinDynamics
Mức lương
Thương lượng
Địa điểm
Thành phố Hà Nội, Hà Nội
Tổng quan
- Testing strategy & methodology
- Own the conversation testing strategy for VinDynamics' humanoid robots across English and Vietnamese, including multi-turn dialogues, code-switching, and domain-specific scenarios (home, assistance, concierge, safety-critical interactions).
- Define and continuously evolve the test case taxonomy: happy path, edge cases, adversarial inputs, ambiguity, interruptions, repairs, long-horizon context, and emotional/social interactions.
- Establish release criteria, go/no-go gates, and quality bars for every conversational AI update shipped to robot fleet.
- End-to-end log tracing & root cause analysis
- Chase failing conversations end-to-end across the full voice pipeline: audio capture → ASR → NLU / intent → LLM / agent (including tool calls, RAG, memory) → TTS → robot action.
- Build and maintain tracing infrastructure (OpenTelemetry, LangSmith/Langfuse, custom span schemas) so every user turn can be reconstructed with inputs, intermediate states, latencies, and outputs.
- Localize defects to the correct component (ASR mis-recognition vs. LLM hallucination vs. TTS prosody vs. tool-call failure) and drive fixes with the owning team.
- Automation framework for conversation testing
- Design and build an automation framework that runs large-scale conversation tests against both the cloud stack and on-device robot builds — programmatic text-in, synthesized or recorded speech-in, and live microphone runs.
- Integrate automated conversation tests into CI/CD so every model, prompt, or pipeline change is gated on regression suites before deployment.
- Scale test execution to thousands of scenarios across languages, accents, and acoustic conditions; build dashboards that make regressions obvious at a glance.
- Evaluation metrics & benchmarks for LLMs and agents
- Define the metric stack for conversational quality: task success, turn-level correctness, groundedness/faithfulness, refusal correctness, safety, hallucination rate, latency budgets, ASR WER/CER, and TTS naturalness.
- Build internal benchmarks for English and Vietnamese conversations that reflect real VinDynamics use cases, and maintain golden datasets that evolve with the product.
- Design LLM-as-a-judge evaluation pipelines with calibration against human raters; run structured human evaluation when automated judges are insufficient.
- Publish weekly/release quality reports with trend analysis, regression attribution, and clear recommendations for model retraining, prompt refinement, or rollback.
- Leadership & cross-functional impact
- Set the hiring bar, mentor, and grow the conversation QA team (QC AI Engineers, Robot Interaction Testers); define the career ladder for conversation testing at VinDynamics.
- Partner with AI research, NLP/Speech, agentic AI, robotics integration, and product deployment teams to shape models and products from the quality side.
- Represent conversation quality in executive release reviews; translate technical failure modes into business-level risk.
Yêu cầu
- 5+ years in QA / test engineering, with at least 3 years dedicated to conversational AI, voice assistants, chatbots, or LLM-based systems.
- Deep expertise in designing test strategies for multi-turn dialogue, covering intent coverage, slot filling, context retention, interruption handling, barge-in, disambiguation, and graceful failure.
- Proven track record building automation frameworks for conversation testing at scale (e.g., pytest / Robot Framework / custom harnesses driving ASR, LLM, and TTS under simulated and real audio conditions).
- Strong hands-on experience tracing end-to-end voice pipelines: ASR (WER/CER, latency, noise robustness) → NLU/Intent → LLM/Agent (tool calls, planning, memory) → TTS (MOS, prosody, latency). Able to chase a failing conversation across all hops through logs, traces, and audio artifacts.
- Mastery of evaluation metrics and benchmarking for LLMs and conversational agents: task success rate, turn-level correctness, faithfulness, groundedness, safety, refusal correctness, hallucination rate, LLM-as-a-judge setups, and human evaluation protocols.
- Fluent working English; able to design and review test sets for both languages including code-switching scenarios.
- Strong Python skills for scripting test harnesses, parsing logs, computing metrics, and building internal QA tooling.
- Experience with observability stacks (OpenTelemetry, LangSmith, Langfuse, Arize, Datadog, ELK) and distributed tracing for multi-service AI pipelines.
- Ability to read AI/ML research papers and translate findings into concrete test protocols.
Quyền lợi
- Attractive income, competitive and commensurate with individual capabilities.
- 13th-month salary, gifts on public holidays and special occasions, and performance-based bonuses.
- Meal allowance, annual company trips, health insurance, and exclusive benefits within the Group’s ecosystem
- Clear career development opportunities aligned with the company’s growth, with access to training programs based on capability and job role.
- A dynamic and open working environment with diverse cultural and sports activities.
Chế độ thưởngĐào tạoPhụ cấp ăn trưa
Tóm tắt thông tin từ tin tuyển dụng chính thức. Xem bản gốc ↗
Quan tâm đến vị trí này?
Bạn sẽ được chuyển đến trang ứng tuyển chính thức của nhà tuyển dụng.
Ứng tuyển tại trang chính thức ↗
Đây là doanh nghiệp của bạn?
Nhận quản lý trang, yêu cầu chỉnh sửa hoặc gỡ bỏ
→