# InterviewCoach Job Eval Progress Readout

- Generated: `2026-05-28T01:26:27Z`
- Purpose: use the real InterviewCoach mobile/backend path to prepare against recent real jobs using the private local CV candidate profile.
- Privacy: raw CV text stayed in ignored local files and is not in public ops artifacts.

## What Was Done

- Built a reproducible job corpus from recent remote agentic/AI engineering opportunities.
- Ranked jobs against the CV fit signals: AI systems architecture, agentic AI, RAG/vector systems, LangGraph/LangChain, automation, Node-RED/n8n, DevOps, SaaS infrastructure, CRM/API automation, and education infrastructure.
- Converted the top 50 jobs into InterviewCoach mobile evaluation scenarios.
- Ran each scenario through the mobile session API path: create candidate, create job description, create mobile interview session, submit answers, complete session, fetch report, fetch history.
- Published a public-safe summary to the ops surface.

## Data Sources And Basis For Decisions

- Job source: `Agentic Engineering Jobs`
- Source URL: https://agentic-engineering-jobs.com/jobs/remote
- API reference: https://agentic-engineering-jobs.com/api-reference
- Window: `2026-05-07` through `2026-05-28`
- Fetched remote jobs: `249`
- Recent jobs inside the window: `91`
- Selected jobs: `50`
- Ranking basis: weighted keyword/metadata match against the CV strengths plus seniority, role architecture depth, remote status, salary presence, and interview usefulness.
- Job details retained: job URL, apply URL, company, title, posted date, location, remote policy, salary if available, fit score, matched CV signals, fit rationale, questions, sample answers.

## Test Volume And Realism

- Scenarios attempted: `50`
- Sessions completed: `50`
- Reports loaded: `50`
- Failures: `0`
- Questions generated per job: `5` role-targeted interview prompts.
- Answers submitted per job in this run: `2` concise candidate answers.
- Total candidate answers submitted: `100`.
- Total transcript turns recorded by backend: `250`.
- How close to a real interview: medium. It used real job descriptions, real CV-derived candidate profile, real backend/mobile session/report/history APIs, real retrieval indexing, and role-specific questions. It was not a full live human interview because answers were short canned samples and the full 50-job pass used deterministic local runtime rather than live model reasoning.

## Numbers

- Average report score: `2.25`
- Score range: `2.12` to `2.41`
- p50 scenario time: `0.453s`
- p95 scenario time: `0.565s`
- Candidate profiles in eval backend: `1`
- Job descriptions in eval backend: `50`
- Product sessions in eval backend: `50`
- Product reports in eval backend: `50`
- Document ingestions: `51`
- Retrieval-ready chunks: `278`
- Qdrant retrieval queries: `300`
- Qdrant retrieval hits: `891`

## Main Findings

- Mobile/backend journey is stable across 50 recent real job scenarios: zero report failures.
- Report scoring is consistently low because the canned candidate answers are intentionally short and do not carry quantified business impact.
- The most repeated preparation gap is missing hard metrics, ranges, and before/after outcomes.
- The second recurring gap is insufficient role-specific evidence for agentic, RAG, LangGraph/LangChain, and multi-agent requirements.
- The app produces actionable next steps, but future UX should surface a job-specific evidence checklist before the interview starts.

## Top Repeated Gaps

- `50`x Answers did not quantify impact with metrics, ranges, or explicit before/after outcomes.
- `31`x Answer depth was thin, which limits confidence in repeatability under real interview pressure.
- `19`x The recruiter-facing story did not clearly show stakeholder alignment, influence, or how the work landed across teams.
- `6`x Important role skills were not clearly evidenced: agentic, agent, multi-agent, rag.
- `5`x Important role skills were not clearly evidenced: agentic, agent, multi-agent, langchain.
- `5`x Important role skills were not clearly evidenced: agentic, agent, rag, eval.
- `4`x Important role skills were not clearly evidenced: agentic, agent, langchain, rag.
- `4`x Important role skills were not clearly evidenced: agentic, agent, langgraph, langchain.

## Recommended Next Product Moves

- Add a pre-interview evidence checklist after JD paste: metrics, architecture story, retrieval/eval proof, security/ops proof, business outcome.
- Show matched job skills as chips before Start chat, with missing-proof warnings when the CV/JD match needs stronger evidence.
- Add a report aggregate view for repeated gaps across multiple target jobs.
- Run a capped live-model/LangSmith pass on the top 3-5 jobs to compare deterministic report quality against live reasoning traces.

## Where To See Reports And Statistics

- Public ops page: https://opsmbapp1.sellsystems.agency/
- Public-safe eval JSON: https://opsmbapp1.sellsystems.agency/job-evals.json
- Full local corpus report: `ops/job-evals/job-corpus-2026-05-28.md`
- Full local mobile run report: `ops/job-evals/job-eval-mobile-2026-05-28.md`
- Local findings report: `ops/job-evals/job-eval-findings-2026-05-28.md`
- Local machine-readable corpus: `ops/job-evals/job-corpus-2026-05-28.json`
- Local machine-readable run: `ops/job-evals/job-eval-mobile-2026-05-28.json`
- Local public-safe ops payload: `ops/www/ops/job-evals.json`

## Metabase / Grafana / LangSmith Status

- Ops page: wired and live for this run.
- `/job-evals.json`: wired and live for this run.
- Grafana: not yet wired to this isolated job-eval run. Existing Grafana remains infrastructure/runtime telemetry; it will not show these 50 eval scenarios unless we add a job-eval metrics exporter or dashboard panel.
- Metabase: not yet wired to this isolated `interviewcoach_jobeval` database/run. Existing Metabase is the business/demo analytics surface for the main backend database; it will not show this run until we either ingest summarized eval rows into the main analytics DB or add a dedicated eval dashboard.
- LangSmith: not used for the full 50-job pass. I deliberately kept `IC_JOB_EVAL_LANGSMITH_ENABLED=false` because sending the private CV and job-eval traces to LangSmith is external data transfer. The live runtime is LangSmith-capable, but this specific full run was deterministic/local. A next pass can run top 3-5 jobs with LangSmith enabled after explicit approval for private CV trace upload, or with a public-safe candidate profile.

## Issues / Limitations Noticed

- The corpus source has duplicate/near-duplicate roles, especially multiple Grafana Staff AI Engineer variants. They are still useful as real job scenarios, but deduplication would improve the final shortlist for personal decision-making.
- The scoring is intentionally low because the test submitted two short answers per job. That is good for exposing preparation gaps, but not a realistic final interview performance.
- Full 50-job run was deterministic and fast; it proves product flow stability and report mechanics, not live model interview depth.
- Job eval metrics are visible in the ops page/JSON, but not yet first-class in Grafana or Metabase.
- Browser-level verification of the public ops page was blocked by this runtime SSRF policy for hostname navigation; HTTPS curl verification succeeded.

## Jobs Tested

### 1. Elastic AI Engineer — Elastic

- Fit score: `187`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.25`
- Session ID: `4c803589-ab5e-40ef-8664-71c3fc7c54a5`
- Job URL: https://agentic-engineering-jobs.com/jobs/elastic-elastic-ai-engineer-wBN5pn
- Apply URL(s): https://jobs.elastic.co/jobs?gh_jid=7858138
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langgraph, langchain.
- Matched signals: agentic, agent, multi-agent, langgraph, langchain, rag, retrieval, langsmith

### 2. Staff AI Engineer | US | Remote (Marketing Ops) — Grafana Labs

- Fit score: `181`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `recruiter`
- Report score: `2.13`
- Session ID: `572e9cab-c7fb-4fdc-8130-bc3670331b9c`
- Job URL: https://agentic-engineering-jobs.com/jobs/grafana-labs-staff-ai-engineer-or-us-or-remote-marketing-ops-Hg9k3P
- Apply URL(s): https://job-boards.greenhouse.io/grafanalabs/jobs/5806328004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langchain, rag.
- Matched signals: agentic, agent, multi-agent, langchain, rag, retrieval, vector, langsmith

### 3. Staff AI Engineer United States | Remote — Grafana Labs

- Fit score: `181`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `system_design`
- Report score: `2.23`
- Session ID: `133c9406-60bb-4e71-8d28-44d099ba93d2`
- Job URL: https://agentic-engineering-jobs.com/jobs/grafana-labs-staff-ai-engineer-united-states-or-remote-bLFHt0
- Apply URL(s): https://job-boards.greenhouse.io/grafanalabs/jobs/5735539004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langchain, rag.
- Matched signals: agentic, agent, multi-agent, langchain, rag, retrieval, vector, langsmith

### 4. Staff AI Engineer | US | Remote — Grafana Labs

- Fit score: `181`
- Posted: `2026-05-19T09:35:50.564Z`
- Mode used: `recruiter`
- Report score: `2.16`
- Session ID: `3843a80b-a4a1-4adb-8201-51da95a7748a`
- Job URL: https://agentic-engineering-jobs.com/jobs/grafana-labs-staff-ai-engineer-or-us-or-remote-vGzk__
- Apply URL(s): https://job-boards.greenhouse.io/grafanalabs/jobs/5839189004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langchain, rag.
- Matched signals: agentic, agent, multi-agent, langchain, rag, retrieval, vector, langsmith

### 5. Staff AI Engineer | Canada | Remote — Grafana Labs

- Fit score: `178`
- Posted: `2026-05-19T08:10:16.857Z`
- Mode used: `recruiter`
- Report score: `2.15`
- Session ID: `acba56e7-cd37-43c4-b56f-a80b693c51ee`
- Job URL: https://agentic-engineering-jobs.com/jobs/grafana-labs-staff-ai-engineer-or-canada-or-remote-h1Yxkf
- Apply URL(s): https://job-boards.greenhouse.io/grafanalabs/jobs/5806327004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langchain, rag.
- Matched signals: agentic, agent, multi-agent, langchain, rag, retrieval, vector, langsmith

### 6. Lead Agentic AI Engineer — OLX

- Fit score: `154`
- Posted: `2026-05-08T15:12:15.656Z`
- Mode used: `system_design`
- Report score: `2.28`
- Session ID: `98ed147e-41a3-49b4-8990-78d420fcccaa`
- Job URL: https://agentic-engineering-jobs.com/jobs/olx-lead-agentic-ai-engineer-q0aC8q
- Apply URL(s): https://jobs.eu.lever.co/olx/118bd6a4-a5af-4f99-98bb-7f51c08736d1
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langgraph, langchain.
- Matched signals: agentic, agent, multi-agent, langgraph, langchain, evaluation, eval, observability

### 7. Senior AI Engineer — Finom

- Fit score: `142`
- Posted: `2026-05-08T15:12:16.416Z`
- Mode used: `system_design`
- Report score: `2.29`
- Session ID: `51fd0ea1-51ae-484c-8c7d-ccb03c3d50c7`
- Job URL: https://agentic-engineering-jobs.com/jobs/finom-senior-ai-engineer-FfBKkQ
- Apply URL(s): https://jobs.eu.lever.co/pnlfin/733b12b7-c794-42d0-89f5-fcc24061ef0a
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, rag, retrieval.
- Matched signals: agentic, agent, langgraph, rag, retrieval, vector, evaluation, eval

### 8. Principal Engineer, Agentic Engineering — Pinterest

- Fit score: `141`
- Posted: `2026-05-12T07:38:57.821Z`
- Mode used: `recruiter`
- Report score: `2.22`
- Session ID: `2d6e71a1-57c8-4b2d-ba0f-7a59454b1f8a`
- Job URL: https://agentic-engineering-jobs.com/jobs/pinterest-principal-engineer-agentic-engineering-BYrR1P
- Apply URL(s): https://www.pinterestcareers.com/jobs/?gh_jid=7775690
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langgraph, langchain.
- Matched signals: agentic, agent, multi-agent, langgraph, langchain, evaluation, eval, observability

### 9. Lead AI Engineer & Technical Architect (Remote) — Bold Business

- Fit score: `136`
- Posted: `2026-05-19T08:10:16.857Z`
- Mode used: `system_design`
- Report score: `2.33`
- Session ID: `fc775b90-fca9-4d41-94ae-c0964fa4cc2a`
- Job URL: https://agentic-engineering-jobs.com/jobs/bold-business-lead-ai-engineer-and-technical-architect-remote-np_4-_
- Apply URL(s): https://job-boards.greenhouse.io/boldbusiness/jobs/4239340009
- Fit rationale: Strong fit because the role overlaps with agent, multi-agent, langgraph, langchain, rag.
- Matched signals: agent, multi-agent, langgraph, langchain, rag, retrieval, vector, evaluation

### 10. Senior AI ML Engineer — 3Pillar

- Fit score: `132`
- Posted: `2026-05-19T09:35:50.564Z`
- Mode used: `recruiter`
- Report score: `2.13`
- Session ID: `1ffa8310-13cd-4ab7-b714-0cf631f0c541`
- Job URL: https://agentic-engineering-jobs.com/jobs/3pillar-senior-ai-ml-engineer-tez9AX
- Apply URL(s): https://jobs.lever.co/3pillarglobal/1a3ee3b8-5215-47dc-9628-f8200066e5f8
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, langchain, rag.
- Matched signals: agentic, agent, multi-agent, langchain, rag, eval, observability, automation

### 11. Forward Deployed Engineer — Warp

- Fit score: `126`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `recruiter`
- Report score: `2.12`
- Session ID: `79b398c2-8294-451c-b099-fa79402d5324`
- Job URL: https://agentic-engineering-jobs.com/jobs/warp-forward-deployed-engineer-Ya4g2f
- Apply URL(s): https://job-boards.greenhouse.io/warp/jobs/5749183004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langchain, rag, eval.
- Matched signals: agentic, agent, langchain, rag, eval, observability, mcp, automation

### 12. Staff Software Engineer, Agentic Patient Outreach — Aledade

- Fit score: `126`
- Posted: `2026-05-12T07:38:07.405Z`
- Mode used: `recruiter`
- Report score: `2.2`
- Session ID: `3f64863d-0e36-4852-9e10-a3a7b4ab8f8e`
- Job URL: https://agentic-engineering-jobs.com/jobs/aledade-staff-software-engineer-agentic-patient-outreach-2hrztd
- Apply URL(s): https://jobs.lever.co/aledade/37625a71-4c88-4f2c-8ea4-8a8fa630415b
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, evaluation, eval.
- Matched signals: agentic, agent, rag, evaluation, eval, observability, workflow, platform

### 13. Sr. / Lead Forward Deployed Engineer (AI) — Alpha Financial Markets Consulting

- Fit score: `125`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `system_design`
- Report score: `2.26`
- Session ID: `dd81bd71-14ef-4520-82ef-eef66ce2985c`
- Job URL: https://agentic-engineering-jobs.com/jobs/alpha-financial-markets-consulting-sr-lead-forward-deployed-engineer-ai-KmWwcp
- Apply URL(s): https://job-boards.greenhouse.io/alphafmcroles/jobs/8521640002
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, langchain, rag.
- Matched signals: agentic, agent, langgraph, langchain, rag, vector, evaluation, eval

### 14. Technical Team Lead — EggAI

- Fit score: `122`
- Posted: `2026-05-08T15:12:16.416Z`
- Mode used: `system_design`
- Report score: `2.25`
- Session ID: `c298dd01-31ab-46ba-b00c-0d1024940752`
- Job URL: https://agentic-engineering-jobs.com/jobs/eggai-technical-team-lead-YZOA5A
- Apply URL(s): https://job-boards.eu.greenhouse.io/eggai/jobs/4772514101
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, retrieval.
- Matched signals: agentic, agent, multi-agent, rag, retrieval, eval, automation, platform

### 15. Software Architect – AI / Agentic Systems — Ubiminds

- Fit score: `122`
- Posted: `2026-05-08T15:12:15.656Z`
- Mode used: `recruiter`
- Report score: `2.2`
- Session ID: `b3354482-8bcf-4177-9792-ace018181c02`
- Job URL: https://agentic-engineering-jobs.com/jobs/ubiminds-software-architect-ai-agentic-systems-4Ag0N-
- Apply URL(s): https://jobs.lever.co/Ubiminds/04709843-ed35-4f75-8b9b-38fe0315ce52
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, eval.
- Matched signals: agentic, agent, multi-agent, rag, eval, automation, workflow, platform

### 16. Senior Software Engineer II - Applied AI (Remote Eligible) — Smartsheet

- Fit score: `121`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.3`
- Session ID: `c86b1bc7-4d6a-4bd4-bc31-dc7ef04d652a`
- Job URL: https://agentic-engineering-jobs.com/jobs/smartsheet-senior-software-engineer-ii-applied-ai-remote-eligible-0mvC80
- Apply URL(s): https://job-boards.greenhouse.io/smartsheet/jobs/7849785
- Fit rationale: Strong fit because the role overlaps with rag, retrieval, vector, langsmith, evaluation.
- Matched signals: rag, retrieval, vector, langsmith, evaluation, eval, observability, platform

### 17. Senior Forward Deployed AI Engineer (Remote Eligible in the UK) — Smartsheet

- Fit score: `119`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `recruiter`
- Report score: `2.25`
- Session ID: `c68800e4-9e35-4579-8b46-e87c11f5b90d`
- Job URL: https://agentic-engineering-jobs.com/jobs/smartsheet-senior-forward-deployed-ai-engineer-remote-eligible-in-the-uk-n956Hc
- Apply URL(s): https://job-boards.greenhouse.io/smartsheet/jobs/7873872
- Fit rationale: Strong fit because the role overlaps with agent, multi-agent, langchain, rag, evaluation.
- Matched signals: agent, multi-agent, langchain, rag, evaluation, eval, mcp, workflow

### 18. Senior Forward Deployed AI Engineer (Remote Eligible in Germany) — Smartsheet

- Fit score: `119`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `recruiter`
- Report score: `2.25`
- Session ID: `f6901636-aec6-4365-a935-0695554aebbf`
- Job URL: https://agentic-engineering-jobs.com/jobs/smartsheet-senior-forward-deployed-ai-engineer-remote-eligible-in-germany-YCU4kJ
- Apply URL(s): https://job-boards.greenhouse.io/smartsheet/jobs/7874013
- Fit rationale: Strong fit because the role overlaps with agent, multi-agent, langchain, rag, evaluation.
- Matched signals: agent, multi-agent, langchain, rag, evaluation, eval, mcp, workflow

### 19. AI Engineer - Everest — Everest (Infinity Constellation)

- Fit score: `119`
- Posted: `2026-05-08T15:12:16.416Z`
- Mode used: `system_design`
- Report score: `2.27`
- Session ID: `e4acbdeb-36a5-474d-8614-3f3016ed307e`
- Job URL: https://agentic-engineering-jobs.com/jobs/everest-infinity-constellation-ai-engineer-everest-7t-j06
- Apply URL(s): https://jobs.ashbyhq.com/infinity-constellation/e84aea13-1d51-4a4e-b179-0d9a3cb890ae
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, retrieval.
- Matched signals: agentic, agent, multi-agent, rag, retrieval, vector, eval, observability

### 20. AI Engineer — New Relic

- Fit score: `118`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.28`
- Session ID: `af0e519f-7086-4796-9719-0ce620b64824`
- Job URL: https://agentic-engineering-jobs.com/jobs/new-relic-ai-engineer-0aqhGD
- Apply URL(s): https://job-boards.greenhouse.io/newrelic/jobs/4987225008
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langchain, rag, retrieval.
- Matched signals: agentic, agent, langchain, rag, retrieval, vector, eval, observability

### 21. Director, Forward Deployed AI Engineering — Natera

- Fit score: `118`
- Posted: `2026-05-19T08:10:16.857Z`
- Mode used: `recruiter`
- Report score: `2.13`
- Session ID: `9d630ffc-cf25-4a9b-b679-bdfcfb138b93`
- Job URL: https://agentic-engineering-jobs.com/jobs/natera-director-forward-deployed-ai-engineering-Btljx8
- Apply URL(s): https://job-boards.greenhouse.io/natera/jobs/5991157004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langchain, rag, eval.
- Matched signals: agentic, agent, langchain, rag, eval, observability, mcp, automation

### 22. Senior Platform Engineer — Toptal

- Fit score: `117`
- Posted: `2026-05-08T15:12:15.656Z`
- Mode used: `system_design`
- Report score: `2.28`
- Session ID: `3a473dd6-3ac4-46e7-9295-2b5a63c9c228`
- Job URL: https://agentic-engineering-jobs.com/jobs/toptal-senior-platform-engineer-xSkHj2
- Apply URL(s): https://jobs.lever.co/toptal/ee2331e7-6aff-4d71-980f-9e1f55d29d21
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, eval, observability.
- Matched signals: agentic, agent, rag, eval, observability, automation, workflow, platform

### 23. Staff AI Engineer - Grafana Ops, AI/ML | USA | Remote — Grafana Labs

- Fit score: `116`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `system_design`
- Report score: `2.26`
- Session ID: `a5fb6ca6-60a0-49f0-8727-b820c2a0f3d1`
- Job URL: https://agentic-engineering-jobs.com/jobs/grafana-labs-staff-ai-engineer-grafana-ops-aiml-or-usa-or-remote-2OgDda
- Apply URL(s): https://job-boards.greenhouse.io/grafanalabs/jobs/5689218004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, observability, automation.
- Matched signals: agentic, agent, multi-agent, observability, automation, workflow, infrastructure, devops

### 24. Principal Applied ML Researcher (Agentic Systems & Applied AI Platform) — Trase Systems

- Fit score: `115`
- Posted: `2026-05-19T08:10:19.091Z`
- Mode used: `system_design`
- Report score: `2.32`
- Session ID: `1b9c6036-6807-4431-8a23-3122f01f4b2d`
- Job URL: https://agentic-engineering-jobs.com/jobs/trase-systems-principal-applied-ml-researcher-agentic-systems-and-applied-ai-platform-OvCaPc
- Apply URL(s): https://job-boards.greenhouse.io/redcellpartners/jobs/5093368007
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, evaluation.
- Matched signals: agentic, agent, multi-agent, rag, evaluation, eval, workflow, platform

### 25. Enterprise Agent Solutions Developer — StackAdapt

- Fit score: `115`
- Posted: `2026-05-12T07:38:48.333Z`
- Mode used: `recruiter`
- Report score: `2.18`
- Session ID: `9544a373-5b19-41e2-903e-a88191114d36`
- Job URL: https://agentic-engineering-jobs.com/jobs/stackadapt-enterprise-agent-solutions-developer-hj3KU2
- Apply URL(s): https://job-boards.greenhouse.io/stackadapt/jobs/4195722009
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, retrieval, vector.
- Matched signals: agentic, agent, rag, retrieval, vector, eval, observability, automation

### 26. AI Engineering I - Marketing — Sezzle

- Fit score: `115`
- Posted: `2026-05-12T07:38:38.212Z`
- Mode used: `system_design`
- Report score: `2.25`
- Session ID: `98eab27c-df24-49d9-8337-f9091229e4ff`
- Job URL: https://agentic-engineering-jobs.com/jobs/sezzle-ai-engineering-i-marketing-H2y1Gp
- Apply URL(s): https://job-boards.greenhouse.io/sezzle/jobs/7709978003
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, langchain, vector.
- Matched signals: agentic, agent, langgraph, langchain, vector, automation, workflow, n8n

### 27. AI Software Engineer — EggAI

- Fit score: `113`
- Posted: `2026-05-08T15:12:16.416Z`
- Mode used: `system_design`
- Report score: `2.25`
- Session ID: `5d1d2005-26f1-472b-ad66-fb72f9076f22`
- Job URL: https://agentic-engineering-jobs.com/jobs/eggai-ai-software-engineer-aHC0oY
- Apply URL(s): https://job-boards.eu.greenhouse.io/eggai/jobs/4772834101
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, evaluation.
- Matched signals: agentic, agent, multi-agent, rag, evaluation, eval, observability, automation

### 28. Senior AI Engineer (m/w/d) — GovRadar

- Fit score: `112`
- Posted: `2026-05-12T07:38:38.212Z`
- Mode used: `system_design`
- Report score: `2.3`
- Session ID: `2533008b-2107-41dd-a68e-24eca111ab11`
- Job URL: https://agentic-engineering-jobs.com/jobs/govradar-senior-ai-engineer-mwd-aFQIO2
- Apply URL(s): https://govradar.jobs.personio.com/job/2588587
- Fit rationale: Strong fit because the role overlaps with agent, langgraph, langchain, rag, vector.
- Matched signals: agent, langgraph, langchain, rag, vector, evaluation, eval, workflow

### 29. Staff Backend Engineer (AI Native), Family AI Lab — Life360

- Fit score: `110`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `system_design`
- Report score: `2.41`
- Session ID: `c400c724-be50-437e-87d8-3710c5bfb6d8`
- Job URL: https://agentic-engineering-jobs.com/jobs/life360-staff-backend-engineer-ai-native-family-ai-lab-wtedFu
- Apply URL(s): https://job-boards.greenhouse.io/life360/jobs/8537012002
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, retrieval, evaluation.
- Matched signals: agentic, agent, rag, retrieval, evaluation, eval, workflow, infrastructure

### 30. Senior Software Engineer, AI  (Remote UK) — Justworks

- Fit score: `106`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.31`
- Session ID: `ffeb4383-b009-416a-83f7-41295e4f735b`
- Job URL: https://agentic-engineering-jobs.com/jobs/justworks-senior-software-engineer-ai-remote-uk-U_8NOY
- Apply URL(s): https://boards.greenhouse.io/justworks/jobs/7797380?gh_jid=7797380
- Fit rationale: Strong fit because the role overlaps with agent, rag, evaluation, eval, automation.
- Matched signals: agent, rag, evaluation, eval, automation, workflow, platform, infrastructure

### 31. Node.js Developer (LangChain, LangGraph, RAG) — Viseven

- Fit score: `106`
- Posted: `2026-05-12T07:38:38.212Z`
- Mode used: `system_design`
- Report score: `2.3`
- Session ID: `170e53ed-39f0-4039-929c-fe6dfbba54c3`
- Job URL: https://agentic-engineering-jobs.com/jobs/viseven-nodejs-developer-langchain-langgraph-rag-tfgm4g
- Apply URL(s): https://jobs.lever.co/viseven/01d833c9-2b93-4828-bab8-161cbfe2b6ea
- Fit rationale: Strong fit because the role overlaps with agent, langgraph, langchain, rag, retrieval.
- Matched signals: agent, langgraph, langchain, rag, retrieval, vector, langsmith, eval

### 32. Principal Machine Learning Engineer-Gen AI, Machine Learning, Graph ML (10189) — Extreme Networks

- Fit score: `104`
- Posted: `2026-05-22T18:01:18.485Z`
- Mode used: `system_design`
- Report score: `2.3`
- Session ID: `321a86b4-d222-451d-acba-0d29b55918f2`
- Job URL: https://agentic-engineering-jobs.com/jobs/extreme-networks-principal-machine-learning-engineer-gen-ai-machine-learning-graph-ml-10189-IQ7Ahs
- Apply URL(s): https://jobs.lever.co/extremenetworks/f99032c4-6048-484d-b93e-f8dfbb62aa19
- Fit rationale: Strong fit because the role overlaps with agent, multi-agent, rag, automation, workflow.
- Matched signals: agent, multi-agent, rag, automation, workflow, platform, aws, docker

### 33. AI Solutions Engineer — Pinterest

- Fit score: `103`
- Posted: `2026-05-12T07:38:38.212Z`
- Mode used: `recruiter`
- Report score: `2.17`
- Session ID: `1713e4b4-1dfb-4751-ae59-26bb2735520f`
- Job URL: https://agentic-engineering-jobs.com/jobs/pinterest-ai-solutions-engineer-qKAOO_
- Apply URL(s): https://www.pinterestcareers.com/jobs/?gh_jid=7714127
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, rag, vector.
- Matched signals: agentic, agent, langgraph, rag, vector, eval, mcp, automation

### 34. Applied AI Engineer - Federal (TS Required) — Snorkel AI

- Fit score: `103`
- Posted: `2026-05-12T07:38:38.212Z`
- Mode used: `recruiter`
- Report score: `2.19`
- Session ID: `229fa31e-2982-4ac3-89c6-112657af4fae`
- Job URL: https://agentic-engineering-jobs.com/jobs/snorkel-ai-applied-ai-engineer-federal-ts-required-kl_9zz
- Apply URL(s): https://job-boards.greenhouse.io/snorkelai/jobs/5721276004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, rag, retrieval.
- Matched signals: agentic, agent, langgraph, rag, retrieval, vector, evaluation, eval

### 35. Senior AI Engineer — Jeeves

- Fit score: `103`
- Posted: `2026-05-08T15:12:15.656Z`
- Mode used: `system_design`
- Report score: `2.3`
- Session ID: `9f5243b1-ccf2-44c3-9c33-3387e137d02e`
- Job URL: https://agentic-engineering-jobs.com/jobs/jeeves-senior-ai-engineer-POlXl-
- Apply URL(s): https://jobs.lever.co/tryjeeves/03f901fc-7a43-4fae-9916-3b287a4bdff6
- Fit rationale: Strong fit because the role overlaps with langchain, rag, retrieval, vector, evaluation.
- Matched signals: langchain, rag, retrieval, vector, evaluation, eval, observability, workflow

### 36. Senior Backend Engineer, AI Agents — Toptal

- Fit score: `103`
- Posted: `2026-05-08T15:12:15.656Z`
- Mode used: `system_design`
- Report score: `2.27`
- Session ID: `9b66f278-aba9-4a19-8dae-e864e952b277`
- Job URL: https://agentic-engineering-jobs.com/jobs/toptal-senior-backend-engineer-ai-agents-_r3y2z
- Apply URL(s): https://jobs.lever.co/toptal/5e236599-9746-4e95-94c5-f405138dcbd7
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, eval, automation.
- Matched signals: agentic, agent, rag, eval, automation, workflow, platform, infrastructure

### 37. Senior AI Software Engineer, Internal Enablement — Extend

- Fit score: `102`
- Posted: `2026-05-22T18:01:18.484Z`
- Mode used: `recruiter`
- Report score: `2.12`
- Session ID: `c533ca03-8629-488f-8e00-21bdfb58878e`
- Job URL: https://agentic-engineering-jobs.com/jobs/extend-senior-ai-software-engineer-internal-enablement-n7B7kO
- Apply URL(s): https://job-boards.greenhouse.io/extend/jobs/5989772004
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langchain, observability, mcp.
- Matched signals: agentic, agent, langchain, observability, mcp, crm, platform, infrastructure

### 38. Senior Software Engineer II - Applied AI and Evaluations (Remote Eligible) — Smartsheet

- Fit score: `102`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.38`
- Session ID: `508bf855-a4df-4902-984c-f52bfb122b4b`
- Job URL: https://agentic-engineering-jobs.com/jobs/smartsheet-senior-software-engineer-ii-applied-ai-and-evaluations-remote-eligible-T_EV7j
- Apply URL(s): https://job-boards.greenhouse.io/smartsheet/jobs/7782945
- Fit rationale: Strong fit because the role overlaps with agent, multi-agent, rag, retrieval, evaluation.
- Matched signals: agent, multi-agent, rag, retrieval, evaluation, eval, platform, infrastructure

### 39. Lead Software Engineer — New Relic

- Fit score: `98`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.27`
- Session ID: `77073f29-8425-4a27-87ba-98308846714f`
- Job URL: https://agentic-engineering-jobs.com/jobs/new-relic-lead-software-engineer-J44JQ_
- Apply URL(s): https://job-boards.greenhouse.io/newrelic/jobs/5189062008
- Fit rationale: Strong fit because the role overlaps with agent, eval, observability, mcp, workflow.
- Matched signals: agent, eval, observability, mcp, workflow, platform, infrastructure, docker

### 40. Staff Software Engineer, Agent Foundations — Pinterest

- Fit score: `97`
- Posted: `2026-05-12T07:38:48.332Z`
- Mode used: `system_design`
- Report score: `2.33`
- Session ID: `0452a655-a893-4700-9308-085cde2234f5`
- Job URL: https://agentic-engineering-jobs.com/jobs/pinterest-staff-software-engineer-agent-foundations-AzeQP9
- Apply URL(s): https://www.pinterestcareers.com/jobs/?gh_jid=7494612
- Fit rationale: Strong fit because the role overlaps with agent, rag, evaluation, eval, platform.
- Matched signals: agent, rag, evaluation, eval, platform, infrastructure, aws, docker

### 41. Forward Deployed AI Engineer — Defense Unicorns

- Fit score: `96`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `recruiter`
- Report score: `2.25`
- Session ID: `25a42de6-be14-4dd8-96b2-b3ec185a8fd8`
- Job URL: https://agentic-engineering-jobs.com/jobs/defense-unicorns-forward-deployed-ai-engineer-QL5wLV
- Apply URL(s): https://job-boards.greenhouse.io/defenseunicorns/jobs/5132886007
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, retrieval, vector.
- Matched signals: agentic, agent, rag, retrieval, vector, evaluation, eval, kubernetes

### 42. Senior Machine Learning Engineer (Agentic AI) — Insider

- Fit score: `96`
- Posted: `2026-05-12T07:38:48.333Z`
- Mode used: `system_design`
- Report score: `2.27`
- Session ID: `54687c71-a172-40a1-bf85-3540dc43f773`
- Job URL: https://agentic-engineering-jobs.com/jobs/insider-senior-machine-learning-engineer-agentic-ai-jPHIGd
- Apply URL(s): https://jobs.lever.co/insiderone/63f94b52-1c50-41c8-a510-b0f1fcb11bf5
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, langchain, rag.
- Matched signals: agentic, agent, langgraph, langchain, rag, evaluation, eval, observability

### 43. Senior Agentic Engineer — Webflow

- Fit score: `94`
- Posted: `2026-05-19T08:10:16.857Z`
- Mode used: `recruiter`
- Report score: `2.21`
- Session ID: `5bc12b66-dd88-41c5-8c16-37fabba33285`
- Job URL: https://agentic-engineering-jobs.com/jobs/webflow-senior-agentic-engineer-UvoRHX
- Apply URL(s): https://job-boards.greenhouse.io/webflow/jobs/7914493
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, vector, evaluation.
- Matched signals: agentic, agent, rag, vector, evaluation, eval, automation, workflow

### 44. Staff Software Engineer, Communication Products — Airbnb

- Fit score: `93`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.38`
- Session ID: `d9521844-4e21-4fd2-95bf-8bd60f63e42f`
- Job URL: https://agentic-engineering-jobs.com/jobs/airbnb-staff-software-engineer-communication-products-LwqEPw
- Apply URL(s): https://careers.airbnb.com/positions/7655958?gh_jid=7655958
- Fit rationale: Strong fit because the role overlaps with rag, retrieval, evaluation, eval, observability.
- Matched signals: rag, retrieval, evaluation, eval, observability, platform, infrastructure, architect

### 45. Senior Machine Learning Engineer, Zeitgeist, Personalization — Spotify

- Fit score: `93`
- Posted: `2026-05-19T09:35:50.565Z`
- Mode used: `system_design`
- Report score: `2.33`
- Session ID: `f3557f11-7bd8-4da8-a0b8-cbc25849640a`
- Job URL: https://agentic-engineering-jobs.com/jobs/spotify-senior-machine-learning-engineer-zeitgeist-personalization-I-aM1x
- Apply URL(s): https://jobs.lever.co/spotify/351ad979-231f-4bee-ae49-8ff55b64f605
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langchain, rag, vector.
- Matched signals: agentic, agent, langchain, rag, vector, evaluation, eval, workflow

### 46. Senior Software Engineer, Partnerships & Integrations (Open-Source) — CopilotKit

- Fit score: `92`
- Posted: `2026-05-12T07:39:06.640Z`
- Mode used: `recruiter`
- Report score: `2.14`
- Session ID: `db68ae56-3af2-432d-9aa9-c1d5adde7877`
- Job URL: https://agentic-engineering-jobs.com/jobs/copilotkit-senior-software-engineer-partnerships-and-integrations-open-source-S8w2ib
- Apply URL(s): https://jobs.lever.co/copilotkit/2d8b533b-1432-4ac0-9319-10f7c99b38fc
- Fit rationale: Strong fit because the role overlaps with agentic, agent, langgraph, langchain, rag.
- Matched signals: agentic, agent, langgraph, langchain, rag, workflow, platform, infrastructure

### 47. Senior Forward Deployed Engineer (AI Agent) — Cresta

- Fit score: `92`
- Posted: `2026-05-12T07:38:57.821Z`
- Mode used: `recruiter`
- Report score: `2.24`
- Session ID: `19d1f29f-754e-4ea6-a483-2b88cf1ffd64`
- Job URL: https://agentic-engineering-jobs.com/jobs/cresta-senior-forward-deployed-engineer-ai-agent-uo9fU4
- Apply URL(s): https://job-boards.greenhouse.io/cresta/jobs/5205398008
- Fit rationale: Strong fit because the role overlaps with agent, rag, retrieval, eval, workflow.
- Matched signals: agent, rag, retrieval, eval, workflow, crm, platform, devops

### 48. Senior Engineer - Artificial Intelligence — Tucows Domains

- Fit score: `91`
- Posted: `2026-05-08T15:12:16.416Z`
- Mode used: `system_design`
- Report score: `2.23`
- Session ID: `d67aaef1-8d72-47ed-abf5-3c54b0809e8e`
- Job URL: https://agentic-engineering-jobs.com/jobs/tucows-domains-senior-engineer-artificial-intelligence-1JkP8k
- Apply URL(s): https://job-boards.greenhouse.io/domains/jobs/7712091003
- Fit rationale: Strong fit because the role overlaps with agentic, agent, multi-agent, rag, platform.
- Matched signals: agentic, agent, multi-agent, rag, platform, infrastructure, aws, docker

### 49. Deployed Engineer (Germany) — LangChain

- Fit score: `90`
- Posted: `2026-05-21T20:23:31.110Z`
- Mode used: `system_design`
- Report score: `2.37`
- Session ID: `63ebe372-09ab-45ca-8656-6c42c3f45da3`
- Job URL: https://agentic-engineering-jobs.com/jobs/langchain-deployed-engineer-germany-2LP_XU
- Apply URL(s): https://jobs.ashbyhq.com/langchain/31ff2b5b-d5c5-443e-bf11-ef02481df579
- Fit rationale: Strong fit because the role overlaps with agent, langgraph, langchain, langsmith, evaluation.
- Matched signals: agent, langgraph, langchain, langsmith, evaluation, eval, workflow, platform

### 50. Senior AI Systems Engineer (Contract) — Orium

- Fit score: `90`
- Posted: `2026-05-19T08:10:19.091Z`
- Mode used: `system_design`
- Report score: `2.35`
- Session ID: `6a58fcd3-d484-4b9d-964a-2e143186ef3b`
- Job URL: https://agentic-engineering-jobs.com/jobs/orium-senior-ai-systems-engineer-contract-kJz1dR
- Apply URL(s): https://job-boards.greenhouse.io/orium/jobs/7746800
- Fit rationale: Strong fit because the role overlaps with agentic, agent, rag, retrieval, eval.
- Matched signals: agentic, agent, rag, retrieval, eval, automation, workflow, n8n

## Question Template Used

Each job received five generated questions. Example from the top Elastic role:

- Walk me through a production AI system you owned end to end. What changed because of it?
- How would you evaluate retrieval quality for this role's knowledge base?
- How would you design MCP connectors so business users can safely automate internal tools?
- Design an n8n or Node-RED style workflow for this role and explain where AI agents belong.
- How would you design the retrieval and evaluation loop for this role's most important workflow?

## Answer Template Used

Each job received the first two concise sample answers from the scenario answer bank. The bank was:

- I would start from the production workflow, not the model. I would map the manual loop, define the deterministic checks, add retrieval only where context changes the decision, and put every non-deterministic step behind traceable evaluation and rollback controls.
- For retrieval I would keep source domains separate: CV evidence, job requirements, product docs, runbooks, and telemetry. The system should return citations, confidence, and missing-evidence flags so the interviewer can see whether an answer is grounded or generic.
- For automation ROI I would measure cycle time, manual touches removed, failure rate, cost per successful run, and adoption. LangSmith-style traces help with model behavior, but product totals and ops metrics are still needed for the business view.