Skip to main content
The Agent Feedback dashboard is where you set up your integrations, review the outcome data your agents produce, and manage your team. Everything in the dashboard is organized under a four-level hierarchy — workspace, product, environment, and integration — so data from different products never overlaps and each team member sees only what their role permits.

​
Hierarchy

Your workspace is the top-level container tied to your OS Account organization. Inside a workspace you create one or more products — each representing a distinct API, website, or MCP surface you want to instrument. Within each product, environments (for example, Production and Staging) separate your deployment stages. Within each environment, integrations hold the specific middleware configurations and API keys.
New workspaces guide you through creating your first product and a Production environment before entering integration setup. Existing workspaces already have a default product and Production environment created during migration.

​
Creating a product and getting an API key

1

Open your workspace

Sign in to the Agent Feedback dashboard and select your workspace from the top navigation. If you are creating a workspace for the first time, the onboarding flow takes you directly to product creation.
2

Create a product

Click New product and give it a name that identifies the surface you are instrumenting — for example, Search API or Docs Site. The product name appears in dashboard filters and in team notifications.
3

Select an environment

After the product is created, select or create the environment you want to instrument — for example, Production or Staging. The dashboard then opens the integration setup view for that environment.
4

Enter integration setup

Select the runtime adapter that matches your stack (Express, Fastify, MCP, ASGI, net/http, Axum, or generic HTTP).
5

Generate an API key

In the API keys section of the integration view, click Create key. The dashboard generates a key in the format af_live_<32 hex>_<secret> and displays it once. Copy it immediately and store it in your secrets manager or environment configuration.
6

Add the key to your middleware

Set the key as the AGENT_FEEDBACK_KEY environment variable (or pass it as apiKey in your middleware options). Deploy your application and the SDK begins creating interactions on the next eligible response.

​
What you see per product

Each product has its own dedicated dashboard view. The tabs and sections available are:

Interactions

Total instrumented responses over the selected time window, with a classification breakdown showing the ratio of confirmed to unclassified interactions, surface type, and per-operation drill-down.

Outcomes

Aggregated success, partial, and failure tallies. Includes note samples and trends over time so you can spot regressions in product reliability from the agent’s perspective.

Reviews

Individual outcome records with their full note text, submission timestamp, capability expiry, and the interaction they belong to. Useful for qualitative analysis.

Sessions

Continuity groups of related interactions. Sessions are only created when company-provided, MCP, or signed-continuation proof is present — never inferred from time windows.

Insights

Derived signals computed from your outcome data, such as the operations with the highest failure rates and trends across customer references.

Collection policy

Controls what the middleware collects, which endpoints are included or excluded, and the feedback mode (auto or ask). See below for details.

​
API key management

You can create and revoke API keys at any time from the API keys section of any integration.
  • Key format: af_live_<32 lowercase hex key ID>_<secret with at least 20 characters>
  • Creating a key: keys are generated server-side and displayed exactly once. Store them immediately in your secrets manager.
  • Revoking a key: click Revoke next to any key. Revocation is immediate — all capability tokens signed with that key are rejected with 401 from that moment forward. Existing outcome records are preserved.
Revoking a key invalidates all tokens that were signed with it, including any still-valid capabilities already embedded in responses that have been served but not yet acted on by agents. Rotate keys during low-traffic windows when possible.

​
Collection policy

The collection policy for a product controls which requests are eligible for instrumentation and how aggressively the SDK requests feedback. You find the collection policy settings under the Collection policy tab inside each product. From there you can configure:
  • Include paths — glob patterns for the routes you want to instrument (for example, /search, /docs/*). If you leave this empty, the SDK instruments all eligible routes.
  • Exclude paths — glob patterns for routes you want to suppress. The SDK always excludes health endpoints, asset files, and its own endpoints regardless of this setting.
  • Feedback mode — auto instructs the agent to submit autonomously; ask makes submission conditional on the agent already knowing the outcome.
Changes to the collection policy take effect when you redeploy your application with the updated SDK configuration.

​
Team management

Your workspace supports multiple OS Account users organized into three roles.

Owner

Full control over the workspace. Owners can invite and remove teammates, change any team member’s role (including promoting to Owner), manage all products, and delete the workspace.

Admin

Can create and manage products, generate and revoke API keys, and invite new Members. Admins cannot change roles or remove other Admins and Owners.

Member

Read-only access to feedback data, outcome reviews, sessions, and observability information. Members cannot create products, manage keys, or change team settings.

​
Inviting teammates

1

Open team settings

From the workspace settings menu, select Team.
2

Send an invitation

Click Invite teammate and enter the recipient’s OS Account email address or handle. Select the role you want to assign.
3

Share the link

The dashboard generates a seven-day invitation link. You can copy it and send it directly. The recipient must sign in with the OS Account that matches the email or handle you specified.
4

Acceptance

Once the recipient accepts, they appear in your team list with the assigned role. You can change their role or remove them at any time from the Team settings page.
Invitation links are bound to the recipient’s OS Account identity. Forwarding a link to a different person will not work — they must sign in with the account that was invited.