USAproduct engineer, agent
2 нед. назад
USAMiddleOnsite
aiproduct management
Build product experiences and AI agents that improve through analyzing agent interactions and outcomes in production.
Обязанности
- Judgment is the learning infrastructure for AI agents. Agents in production don't improve from prompts alone. They improve from experience: the tasks they attempt, the mistakes they make, the edge cases they hit. Here's how it works:
- We ingest everything your agents do in production: traces, tool calls, decisions, outcomes
- Judgment turns that raw experience into structured signals: failure modes, behaviors, rubrics, evals
- Teams close the loop, shipping agent improvements validated against real production evidence
- You'll build the product experiences that make this loop legible, and you'll build the agents that run it. This is not a role where you implement specs handed down. You'll own problems end-to-end: talking to customers, defining what to build, building it, and iterating until it's great.
Другое
- Judgment Agent: Shape how the Judgment Agent runs large-scale investigations: parallel investigators working across thousands of production traces, each covering a different dimension (failure modes, tool errors, regressions, drift), merging results into one answer.
- Verification: Build the platform for verifying agent changes: hosted simulated environments for stateful agent evals, trajectory replay against changed agents, and monitors for unintended behavior changes.
- Agent investigation interfaces: Design how engineers understand what their agents did and why. Long traces, tool calls, decisions, failures. What does debugging look like when the "program" is a reasoning loop? How do you make a thousand-step trajectory legible in minutes?
- Swarm UX: A hundred parallel investigations is useless if engineers can't follow them. Design how humans watch a swarm work, redirect investigators that go down the wrong path, and consume findings without reading a hundred reports.
- The improvement loop: Build the workflows that turn production trajectories into datasets, judges, and regression checks, so the path from "found a problem" to "verified a fix" feels like one motion.
- The platform underneath: Workspaces, roles, permissions, billing, usage, and limits for teams running many agents across many environments.
- Judgment everywhere agents are built: An SDK and terminal-first experience so Claude Code, Codex, and OpenCode sessions can summon Judgment as a subagent mid-development.
- Experience building and scaling end-to-end production systems, from data layer to UI
- Strong technical problem-solving skills, especially in fast-changing, ambiguous environments
- A builder and tinkerer's mindset with high agency - you find creative ways to overcome obstacles and ship
- Hands-on experience building with LLMs or agents, or the drive to get there fast
- Comfort working directly with customers to understand their needs and solve real-world problems
- Excellent communication skills - clear, direct, and persuasive across technical and non-technical audiences