СШАAI Engineer II – Trust & Explainability (AI Platform)
2 нед. назад
104.1k–177.6k USD / годСШАMiddleRemote
llmobservabilityexplainabilitytracing
Build the trust and explainability layer for AI agent workflows on 's platform, enabling tracing, evaluation, and explanation to foster trust and security.
Обязанности
- This is a deeply technical, hands-on engineering role and an integral part of the platform itself. You will write the code that traces agent actions across a multi-agent workflow, adopt and extend leading open-source explainability and observability frameworks, and build the primitives product teams use to surface explanations to lenders, credit unions, and their borrowers. You should have real hands-on experience with LLM-based systems and a strong pull toward the problems of trust, tracing, and evaluation. You do not need to be an expert in all of it. This is a place to build that expertise while the platform is being built, working alongside a Staff engineer team lead and architects who will invest in your growth.
- In the first few months, a successful hire has shipped tracing that follows a full agent workflow end to end, readable by an engineer who did not write it, and has contributed working components to the platform's evaluation harness. Over the first year, success looks like this: a first version of the customer-facing explanation primitive is live in at least one product, evaluation and isolation checks run automatically on every agent release, and when an engineer asks why an agent did something, or a customer asks whether to trust it, the platform answers. Engineers who thrive here like the intersection of AI and rigor, want to become the team's go-to on agent tracing and explainability, and are motivated by making AI something people can see into and trust.
Будет плюсом
- Experience with OpenTelemetry, including the GenAI semantic conventions, or OpenLLMetry
- Experience with LLM observability and evaluation tools (Langfuse, Arize Phoenix, LangSmith, Braintrust, promptfoo, DeepEval, or equivalent)
- Contributions to open-source AI observability, evaluation, or agent framework projects
- Experience with cloud-managed model services such as AWS Bedrock or Azure OpenAI
- Experience building or operating multi-tenant SaaS systems where tenant isolation was a hard requirement
- Prior experience in financial services, fintech, or another regulated industry where explainability shaped technical decisions
- Experience building developer-facing debugging or visualization tools
Другое
- The AI Platform team is building the shared runtime that every AI agent at will run on as part of the One platform: model gateway, orchestration, memory, and observability, delivered as reusable services so product teams build agents once and run them safely for hundreds of credit union and lender customers. This role builds the trust and explainability layer of that runtime: the tracing, evaluation, and explanation capabilities that let engineers understand exactly what a multi-agent workflow did and why, and that let product teams show end customers the right level of explanation to trust an agent's output.
- This role has a strong dotted-line relationship to 's Security Operations team, so the platform's trust guarantees and Security's threat models are built from the same picture.
- L2 AI Engineers handle a broad range of work independently, from bug fixes to feature development. They have developed a specialization and are deepening it. They write code that other engineers trust, ask good questions, and are starting to help the engineers around them. L2 AI Engineers at are active, daily users of AI-assisted development tools.
- Completes assigned features and bug fixes independently, with limited need for day-to-day guidance
- Designs components within a well-defined scope; escalates questions on complex system design rather than guessing
- Participates in code review and provides constructive feedback
- Surfaces blockers proactively rather than waiting for check-ins
- Writes tests that cover the functionality they ship
- Monitors and responds to issues with their own work
- Documents decisions and implementation details that others will need later
- Understands how to instrument an agent so a single conversation can be followed from the first user message through every model call, tool call, retrieval, agent handoff, and decision to the final response
- Builds the correlation that ties actions across multiple agents in one workflow into a single readable trace
- Turns raw trace data into an explanation a human can follow: what the agent did, what it relied on, and why it chose that path
- Understands that the explanation an engineer needs and the explanation a borrower or loan officer needs are different, and builds for both
- Works with product engineers on what an agent should disclose at the product surface: what it did, what data it used, how confident it was, and what a human should verify
- Applies judgment about the right level of explanation for a regulated lending and account-opening context
- Familiar with the open-source landscape for LLM observability, tracing, and evaluation, and can evaluate a framework against the platform's needs
- Integrates and extends existing frameworks rather than rebuilding them, and builds what does not exist yet
- Keeps the platform's instrumentation aligned to emerging standards so traces stay portable across tools
- Understands how to measure the quality of non-deterministic output: golden datasets, rubric and LLM-as-judge scoring, and regression baselines
- Builds evaluation checks that run in CI so a model swap, prompt change, or tool change is tested before it reaches customers
- Treats evaluation data as versioned, reviewed code
- Knows the common attack patterns against LLM applications (prompt injection, jailbreaks, tool misuse, data exfiltration) and how to write tests for them
- Understands tenant isolation as a platform guarantee and builds tests that prove one customer's agent can never see another customer's data
- Contributes to the shared guardrail layer so every agent inherits protection without re-implementing it
- Build tracing across the platform's gateway, orchestration, memory, and tool layers, following defined designs from the team lead and architects
- Build the correlation that links actions across agents in a single multi-agent workflow, including handoffs, parallel branches, and retries
- Build the developer-facing trace view that makes a full agent conversation readable to someone who did not write the agent
- Build the explanation layer that turns trace data into a human-readable account of why an agent did what it did
- Build the platform primitives product teams surface to end users: explanation records, confidence and provenance metadata, and "what the agent relied on" summaries
- Partner with product engineers on the Document Request Agent and the MLM agents to land those primitives in real products
- Iterate on the explanation format based on feedback from product teams and customer-facing staff
- Evaluate and integrate open-source observability, tracing, and evaluation frameworks into the platform runtime
- Extend those frameworks where our agent workloads need more than they offer, and contribute fixes and extensions back upstream where it makes sense
- Build the gaps: the pieces of trust and explainability tooling the ecosystem does not provide yet
- Build and maintain components of the platform's evaluation framework: golden dataset management, test runners, scoring pipelines, and regression reporting
- Build the tooling that helps teams create and version golden datasets from de-identified real traffic and synthetic cases
- Run model and prompt comparisons for the platform's shared components and report what changed
- Build red-team and adversarial test suites covering prompt injection, jailbreak attempts, tool misuse, and data exfiltration, and run them in the platform's release process
- Build the automated test suite that proves agents on the shared runtime cannot cross tenant boundaries through memory, retrieval, tool calls, or model context
- Share red-team findings and new attack patterns with Security Operations and pull their threat models into the platform's tests
- Participate in design discussions and code reviews; give and receive feedback constructively
- Support onboarding of L1 AI Engineer teammates; share context and help them get unblocked
- Contribute to documentation that reduces tribal knowledge on the team
- 3+ years of professional software engineering experience, delivering features independently in a production environment
- Solid understanding of algorithms, data structures, and software design fundamentals
- Proficiency with standard development tooling: Git, Docker, automated testing, and modern scripting languages
- Active daily use of AI-assisted development tools
- Bachelor's degree in Computer Science, Software Engineering, or equivalent experience
- Hands-on experience building software that integrates large language models (LLM APIs, agent frameworks, RAG pipelines, or similar), in production or in substantial personal or open-source projects
- Proficiency in Python with production experience; TypeScript a plus
- Hands-on experience in at least one of the following, with real interest in growing into the others: LLM observability and tracing, LLM evaluation and testing, agent frameworks and multi-agent orchestration, or application security testing
- Experience with distributed tracing or observability tooling in a production system
- Strong automated testing instincts, including an interest in how to test systems that do not return the same answer twice
- Experience running workloads on Azure or AWS, including the basics of identity and access management, networking, and secrets management