# Spark Range Cohort Report - 2026-05-28 - Generated: `2026-05-28T13:22:48Z` - Scenarios attempted: `13` - Reports loaded: `10` - Failures: `3` - Answers submitted: `50` - Score range: `3.98`-`4.53` - Average score: `4.33` ## Completed Reports ### Rank 20 - New Relic - AI Engineer - Situation: `recruiter-screen-motivation-constraints` - Score: `4.42` - Turns: `5` - Seconds: `62.592` - Job URL: https://agentic-engineering-jobs.com/jobs/new-relic-ai-engineer-0aqhGD - Summary: Strong systems-oriented design performance with clear stage boundaries, deterministic replay semantics, and explicit operational guardrails. The candidate reasons in control-loop terms and repeatedly ties correctness to isolation, observability, and controlled rollout. Main weakness is limited concrete operational specification for production sizing, failure-domain ownership, and external dependency runbook behavior (LLM/serving stack), which leaves some ambiguity at production handoff. ### Rank 26 - Sezzle - AI Engineering I - Marketing - Situation: `recruiter-screen-motivation-constraints` - Score: `3.98` - Turns: `5` - Seconds: `63.099` - Job URL: https://agentic-engineering-jobs.com/jobs/sezzle-ai-engineering-i-marketing-H2y1Gp - Summary: The candidate gives a strong production-minded systems design for a marketing campaign decision pipeline under load, with clear separation of Retrieval, Agent, Dispatcher, and Checkout domains, explicit latency/timeouts, and rollout safety gates. The design shows mature operational thinking (bounded retries, idempotency, outbox/state machine, degraded-mode behavior, canary rollback). Main weakness is incomplete scale/ownership rigor for real-world operation: no concrete capacity model, insufficient blast-radius and incident ownership/playbook detail, and limited discussion of data governance/security and cache invalidation behavior. ### Rank 33 - Pinterest - AI Solutions Engineer - Situation: `recruiter-screen-motivation-constraints` - Score: `4.25` - Turns: `5` - Seconds: `318.035` - Job URL: https://agentic-engineering-jobs.com/jobs/pinterest-ai-solutions-engineer-qKAOO_ - Summary: Candidate shows credible production AI engineering maturity and a strong safety-first delivery mindset, with repeated evidence of owning rollout and control mechanisms. Hireability is good, but recruiter confidence is reduced by limited concrete business outcomes and limited role-specific technical depth on some platform ecosystems that appear important for this role. ### Rank 26 - Sezzle - AI Engineering I - Marketing - Situation: `project-deep-dive-business-context` - Score: `4.33` - Turns: `5` - Seconds: `372.346` - Job URL: https://agentic-engineering-jobs.com/jobs/sezzle-ai-engineering-i-marketing-H2y1Gp - Summary: Strong architectural reasoning for an AI-driven lead qualification workflow under production load. The candidate repeatedly applies a clear boundary model (orchestrator/retrieval/CRM sink), explicit failure gating, and conservative-mode controls. The main weakness is operational incompleteness: they avoid concrete delivery and governance details (data contracts, ownership/incident model, security/compliance, and measurable roll-out policy) and over-rely on high-level mechanisms. ### Rank 33 - Pinterest - AI Solutions Engineer - Situation: `project-deep-dive-business-context` - Score: `4.27` - Turns: `5` - Seconds: `241.988` - Job URL: https://agentic-engineering-jobs.com/jobs/pinterest-ai-solutions-engineer-qKAOO_ - Summary: The candidate presents as a credible hire on paper for production-oriented AI delivery. They consistently anchor decisions to rollout safety, observability, and recovery behavior, and can articulate a realistic migration path to Android with explicit control points. Primary risk is that proof of business impact and concrete cross-functional execution depth is still more declarative than evidence-backed. ### Rank 20 - New Relic - AI Engineer - Situation: `project-deep-dive-business-context` - Score: `4.2` - Turns: `5` - Seconds: `83.404` - Job URL: https://agentic-engineering-jobs.com/jobs/new-relic-ai-engineer-0aqhGD - Summary: The candidate gave a robust architectural design with explicit boundary decomposition, failure-domain isolation, and rollback/replay control under spike pressure, but left several production-control gaps around contract specificity, failure taxonomy, and operational execution playbooks that would matter in a New Relic design review. ### Rank 24 - Trase Systems - Principal Applied ML Researcher (Agentic Systems & Applied AI Platform) - Situation: `project-deep-dive-business-context` - Score: `4.53` - Turns: `5` - Seconds: `76.23` - Job URL: https://agentic-engineering-jobs.com/jobs/trase-systems-principal-applied-ml-researcher-agentic-systems-and-applied-ai-platform-OvCaPc - Summary: Strong systems design response with explicit decomposition, failure isolation, and control loops, aligned to production reliability concerns. Primary weakness is that the answer is operationally mature but still under-specifies global scale limits and ownership/HA details for critical control-plane components. ### Rank 32 - Extreme Networks - Principal Machine Learning Engineer-Gen AI, Machine Learning, Graph ML (10189) - Situation: `project-deep-dive-business-context` - Score: `4.38` - Turns: `5` - Seconds: `93.396` - Job URL: https://agentic-engineering-jobs.com/jobs/extreme-networks-principal-machine-learning-engineer-gen-ai-machine-learning-graph-ml-10189-IQ7Ahs - Summary: Evaluation of this 5-turn systems-design response shows strong architectural discipline for a high-scale multi-tenant control-plane design. The candidate demonstrated clear service partitioning, explicit failure-state control loops, and practical rollout/rollback behavior; however, it remains design-focused without enough operational implementation detail for principal-level production execution under real incidents. ### Rank 24 - Trase Systems - Principal Applied ML Researcher (Agentic Systems & Applied AI Platform) - Situation: `multi-agent-tool-safety` - Score: `4.5` - Turns: `5` - Seconds: `60.737` - Job URL: https://agentic-engineering-jobs.com/jobs/trase-systems-principal-applied-ml-researcher-agentic-systems-and-applied-ai-platform-OvCaPc - Summary: Strong production-oriented systems design under safety and scale pressure. Candidate consistently reasons in terms of bounded services, strict failure boundaries, and automatic control loops, and ties partial-retrieval degradation to safe execution modes. Biggest weaknesses are the missing quantitative operational targets and explicit interface/ownership/runbook details that would make this review-ready for real platform implementation. ### Rank 32 - Extreme Networks - Principal Machine Learning Engineer-Gen AI, Machine Learning, Graph ML (10189) - Situation: `multi-agent-tool-safety` - Score: `4.45` - Turns: `5` - Seconds: `418.162` - Job URL: https://agentic-engineering-jobs.com/jobs/extreme-networks-principal-machine-learning-engineer-gen-ai-machine-learning-graph-ml-10189-IQ7Ahs - Summary: Strong systems-design performance under pressure. The candidate gives a coherent, safety-first architecture with explicit boundaries, control-plane authority, tenant-aware degradation, and observable failure control loops. For a principal ML platform role, the design quality is close to hireable, but some production details remain underspecified (scale economics, HA/DR, migration/runbook mechanics). ## Failures / Runtime Issues - Rank 20 New Relic / `project-deep-dive-business-context`: Spark stream disconnected before completion. request_id=db516430-8b56-4bc6-ab31-8ef8f561bdaf - Rank 21 Natera / `project-deep-dive-business-context`: Spark dependency unavailable. request_id=545938d91663415a860a63250e960d63 - Rank 21 Natera / `multi-agent-tool-safety`: Spark dependency unavailable. request_id=60ebc15f9a6140359c8cc6610323504f