feat: Add failure diagnostics to tool-call telemetry (#631)

* feat: add failure diagnostics to tool-call telemetry

Tool failures in Mixpanel were blind spots — every failure was FAILED/SOFT_FAIL
with no way to tell a user typo from a server crash, making it impossible to
prioritize fixes or understand error patterns.

New telemetry fields (non-success only): failure_category (INVALID_INPUT |
AUTH | INTERNAL_ERROR), failure_http_status, failure_detail (truncated error
message), actor_name, and AJV validation diagnostics (keyword, path,
missing/additional property).

Key decisions:
- Three-value enum, not ten — INVALID_INPUT covers both validation and
  not-found (user-caused); validation_* fields disambiguate when needed
- Wire protocol: buildMCPResponse carries transient internal* fields from
  tool helpers to server dispatch, stripped before reaching MCP clients
- actor-mcp is a known gap (opaque isError, defaults to INTERNAL_ERROR) —
  documented as TODO, deferred to future iteration
- Kept server.ts handler structure to minimize structural diff

Other changes:
- preparePayment → prepareToolCallContext (name was misleading — handles
  all tool calls, not just paid ones); cleanArgs → toolArgs, logArgs →
  logSafeArgs, client → apifyClient
- dedent for multi-line error messages in server.ts
- actor_name extracted via extractActorName() — especially needed for
  call-actor where the actor slug is only in arguments
- INVALID_INPUT propagated across tool helpers (one-line additions per file)
- 402 Payment Required now tracked with failure_category instead of empty
  diagnostics
- New helpers: classifyFailureCategory(), extractValidationDiagnostics()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: Review comments

* fix: improve failure category classification in telemetry

* refactor: introduce `ValidationDiagnostics` type for cleaner validation handling

Replaced inline Pick definitions for validation diagnostics across multiple files with a shared `ValidationDiagnostics` type in `types.ts`. Simplifies code and ensures consistency while extracting or handling validation-related diagnostics.

* refactor: simplify failure diagnostics extraction in server handlers

Replaced inline diagnostic extraction logic with `extractToolResponseDiagnostics` utility in `tool_status.ts`. Centralizes diagnostic handling, reduces repetition, and ensures consistent behavior across server responses.

* refactor: use constants for HTTP status codes in failure category classification

Replaced hardcoded HTTP status codes with constants (`HTTP_UNAUTHORIZED`, `HTTP_FORBIDDEN`, `HTTP_NOT_FOUND`) in `tool_status.ts`. Ensures clarity and reduces potential for errors. Added constant definitions to `const.ts`.

* feat: add `actor-mcp` handling and enhance validation diagnostics

Added support for extracting `actorId` for `actor-mcp` tools in `extractActorName`. Enhanced validation diagnostics by including error count (`validation_error_count`) and updated related types (`ValidationDiagnostics`, `ToolCallTelemetryProperties`) to ensure consistency. Improved error handling in `extractValidationDiagnostics` to account for multiple validation errors.

* refactor: centralize `getToolFullName` and `extractActorName` logic in utilities

Removed redundant implementations of actor name and tool full name extraction. Introduced `getToolFullName` and relocated `extractActorName` to `tools.ts` for consistency and reuse. Updated server handlers to use the new utility, simplifying telemetry and tool resolution logic.

* test: add unit tests for `getToolFullName` and `extractActorName` functions

* fix: reformat failure diagnostics assertion in `extractToolResponseDiagnostics` test

* refactor: replace inline failure diagnostics with centralized `telemetry` object

Standardized telemetry reporting across tools. Introduced `telemetry` object to replace inline `toolStatus` and failure diagnostics. Updated utility functions to extract AJV error details and telemetry consistently. Refactored server handlers to align with these changes, improving maintainability and consistency in error handling.

* fix: include actor details in failure diagnostics for enhanced telemetry logging

* refactor: replace `FailureDetails` with `CallDiagnostics` for unified telemetry management

Standardized telemetry tracking across the server by replacing `FailureDetails` with the more comprehensive `CallDiagnostics` type. Updated server handlers, utility functions, and tests to consistently use `CallDiagnostics` for handling failure diagnostics and actor-related fields. Improved maintainability and enhanced consistency in telemetry data logging.

* fix: enhance failure diagnostics with detailed error information and categories

* refactor: clean up CallActorResolvedContext type

* fix: simplify isActorMcpServer assignment in call_actor_common.ts

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: jirispilka

eslint.config.mjs

@@ -24,6 +24,7 @@ export default [
         ignores: [
             '**/dist', // Build output directory
             '**/.venv', // Python virtual environment (if present)
+            '.claude/worktrees/**', // Local Codex/Claude worktrees are outside this repo's TS project
             'evals/*.ts', // Top-level evaluation scripts
             'evals/*.md', // Documentation files
             'evals/*.json', // Test case data files

src/const.ts

@@ -190,7 +190,7 @@ export const SEGMENT_FLUSH_INTERVAL_MS = 5_000;
 
 // Tool status
 /**
- * Unified status constants for tool execution lifecycle.
+ * Unified status constants for the tool execution lifecycle.
  * Single source of truth for all tool status values.
  */
 export const TOOL_STATUS = {
@@ -200,8 +200,17 @@ export const TOOL_STATUS = {
     SOFT_FAIL: 'SOFT_FAIL',
 } as const;
 
+export const FAILURE_CATEGORY = {
+    INVALID_INPUT: 'INVALID_INPUT',
+    AUTH: 'AUTH',
+    INTERNAL_ERROR: 'INTERNAL_ERROR',
+} as const;
+
 // HTTP status codes
+export const HTTP_UNAUTHORIZED = 401;
 export const HTTP_PAYMENT_REQUIRED = 402;
+export const HTTP_FORBIDDEN = 403;
+export const HTTP_NOT_FOUND = 404;
 
 // Modes that allow long running task tool executions
 export const ALLOWED_TASK_TOOL_EXECUTION_MODES = ['optional', 'required'] as const;

src/mcp/server.ts

@@ -4,67 +4,72 @@ import { buildPaymentRequiredResponse, registerPaymentRequiredInterceptor } from
 import type { PaymentMeta, PaymentProvider, RequestHeaders } from './types.js';
 
 /**
- * Result of preparing payment context for a tool call.
- * Centralizes all payment-related processing into a single step.
+ * Result of preparing tool-call context.
+ * Centralizes payment-aware argument sanitization, logging data, and client creation.
  */
-export type PreparePaymentResult = {
+export type PrepareToolCallContextResult = {
     /** Structured error result for a 402 PaymentRequired response. Undefined if no error. */
-    errorResult?: ReturnType<typeof buildPaymentRequiredResponse>;
-    /** Args with payment-specific fields removed — safe for ajv validation and Actor input. */
-    cleanArgs: Record<string, unknown>;
-    /** Args with sensitive payment fields redacted — safe for logging. */
-    logArgs: unknown;
+    paymentRequiredResult?: ReturnType<typeof buildPaymentRequiredResponse>;
+    /** Tool args with payment-specific fields removed — safe for ajv validation and Actor input. */
+    toolArgs: Record<string, unknown>;
+    /** Tool args with sensitive payment fields redacted — safe for logging. */
+    logSafeArgs: unknown;
     /** ApifyClient configured with payment headers (if applicable) or standard token. */
-    client: ApifyClient;
+    apifyClient: ApifyClient;
 };
 
 /**
- * Prepares payment context for a tool call.
+ * Prepares tool-call context before validation and execution.
  *
  * This helper centralizes all payment processing:
  * 1. Validates payment credentials (for tools with `paymentRequired: true`)
  * 2. Strips payment fields from args (for clean ajv validation and Actor input)
  * 3. Redacts sensitive fields for logging
  * 4. Creates an ApifyClient with payment headers or standard token
  *
- * Call this BEFORE ajv validation so `cleanArgs` can be validated without
+ * Call this BEFORE ajv validation so `toolArgs` can be validated without
  * the `additionalProperties: true` hack.
+ *
+ * TODO: This function has mixed responsibilities. It should be split into separate concerns:
+ * - validatePaymentCredentials (returns error or void)
+ * - sanitizeToolArgs (returns toolArgs and logSafeArgs)
+ * - createApifyClient (returns apifyClient)
  */
-export function preparePayment(input: {
+export function prepareToolCallContext(input: {
     provider: PaymentProvider | undefined;
     tool: ToolEntry;
     args: Record<string, unknown>;
     apifyToken: ApifyToken;
     meta?: PaymentMeta;
     requestHeaders?: RequestHeaders;
-}): PreparePaymentResult {
+}): PrepareToolCallContextResult {
     const { provider, tool, args, apifyToken, meta, requestHeaders } = input;
 
     if (!provider) {
-        const client = new ApifyClient({ token: apifyToken });
-        registerPaymentRequiredInterceptor(client);
+        const apifyClient = new ApifyClient({ token: apifyToken });
+        registerPaymentRequiredInterceptor(apifyClient);
         return {
-            cleanArgs: args,
-            logArgs: args,
-            client,
+            toolArgs: args,
+            logSafeArgs: args,
+            apifyClient,
         };
     }
 
     const error = tool.paymentRequired ? provider.validatePayment(args, meta, requestHeaders) : null;
     const errorData = error && provider.getPaymentRequiredData ? provider.getPaymentRequiredData() : undefined;
-    const cleanArgs = provider.removePaymentFields(args);
-    const logArgs = provider.redactForLogging(args);
+    const toolArgs = provider.removePaymentFields(args);
+    const logSafeArgs = provider.redactForLogging(args);
 
     const paymentHeaders = provider.getPaymentHeaders(args, meta, requestHeaders);
-    const client = Object.keys(paymentHeaders).length > 0
+    const apifyClient = Object.keys(paymentHeaders).length > 0
         ? new ApifyClient({ paymentHeaders })
         : new ApifyClient({ token: apifyToken });
-    registerPaymentRequiredInterceptor(client);
+    registerPaymentRequiredInterceptor(apifyClient);
 
     return {
-        errorResult: error ? buildPaymentRequiredResponse(error, errorData) : undefined,
-        cleanArgs,
-        logArgs,
-        client,
+        paymentRequiredResult: error ? buildPaymentRequiredResponse(error, errorData) : undefined,
+        toolArgs,
+        logSafeArgs,
+        apifyClient,
     };
 }

src/payments/helpers.ts

@@ -2,5 +2,5 @@ export type { PaymentHeaders, PaymentMeta, PaymentProvider, PaymentProviderId, R
 export { SkyfirePaymentProvider } from './skyfire.js';
 export { X402PaymentProvider } from './x402.js';
 export { resolvePaymentProvider } from './resolve.js';
-export { preparePayment } from './helpers.js';
-export type { PreparePaymentResult } from './helpers.js';
+export { prepareToolCallContext } from './helpers.js';
+export type { PrepareToolCallContextResult } from './helpers.js';

src/payments/index.ts

@@ -2,12 +2,13 @@ import { z } from 'zod';
 
 import log from '@apify/log';
 
-import { ALLOWED_DOC_DOMAINS, HelperTools, TOOL_STATUS } from '../../const.js';
+import { ALLOWED_DOC_DOMAINS, FAILURE_CATEGORY, HelperTools, TOOL_STATUS } from '../../const.js';
 import { fetchApifyDocsCache } from '../../state.js';
 import type { InternalToolArgs, ToolEntry, ToolInputSchema } from '../../types.js';
 import { compileSchema } from '../../utils/ajv.js';
 import { logHttpError } from '../../utils/logging.js';
 import { buildMCPResponse } from '../../utils/mcp.js';
+import { classifyFailureCategory } from '../../utils/tool_status.js';
 import { fetchApifyDocsToolOutputSchema } from '../structured_output_schemas.js';
 
 const fetchApifyDocsToolArgsSchema = z.object({
@@ -65,7 +66,7 @@ Only documentation URLs from Apify and Crawlee are allowed \
 Please provide a valid documentation URL. \
 You can find documentation URLs using the ${HelperTools.DOCS_SEARCH} tool.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.SOFT_FAIL,
+                telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL, failureCategory: FAILURE_CATEGORY.INVALID_INPUT },
             });
         }
 
@@ -87,7 +88,11 @@ You can find documentation URLs using the ${HelperTools.DOCS_SEARCH} tool.`],
                     return buildMCPResponse({
                         texts: [buildFetchErrorMessage(url, `HTTP Status: ${response.status} ${response.statusText}.`)],
                         isError: true,
-                        toolStatus: isUserError ? TOOL_STATUS.SOFT_FAIL : TOOL_STATUS.FAILED,
+                        telemetry: {
+                            toolStatus: isUserError ? TOOL_STATUS.SOFT_FAIL : TOOL_STATUS.FAILED,
+                            failureCategory: classifyFailureCategory(error),
+                            failureHttpStatus: response.status,
+                        },
                     });
                 }
                 markdown = await response.text();
@@ -97,7 +102,7 @@ You can find documentation URLs using the ${HelperTools.DOCS_SEARCH} tool.`],
                 return buildMCPResponse({
                     texts: [buildFetchErrorMessage(url, `Error: ${error instanceof Error ? error.message : String(error)}.`)],
                     isError: true,
-                    toolStatus: TOOL_STATUS.SOFT_FAIL,
+                    telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL, failureCategory: FAILURE_CATEGORY.INTERNAL_ERROR },
                 });
             }
         }

src/tools/common/fetch_apify_docs.ts

@@ -1,7 +1,7 @@
 import dedent from 'dedent';
 import { z } from 'zod';
 
-import { HelperTools, TOOL_MAX_OUTPUT_CHARS, TOOL_STATUS } from '../../const.js';
+import { FAILURE_CATEGORY, HelperTools, TOOL_MAX_OUTPUT_CHARS, TOOL_STATUS } from '../../const.js';
 import type { InternalToolArgs, ToolEntry, ToolInputSchema } from '../../types.js';
 import { compileSchema } from '../../utils/ajv.js';
 import { getValuesByDotKeys, parseCommaSeparatedList } from '../../utils/generic.js';
@@ -120,7 +120,7 @@ export const getActorOutput: ToolEntry = Object.freeze({
             return buildMCPResponse({
                 texts: [`Dataset '${parsed.datasetId}' not found.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.SOFT_FAIL,
+                telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL, failureCategory: FAILURE_CATEGORY.INVALID_INPUT },
             });
         }
 

src/tools/common/get_actor_output.ts

@@ -1,6 +1,6 @@
 import { z } from 'zod';
 
-import { HelperTools, TOOL_STATUS } from '../../const.js';
+import { FAILURE_CATEGORY, HelperTools, TOOL_STATUS } from '../../const.js';
 import type { InternalToolArgs, ToolEntry, ToolInputSchema } from '../../types.js';
 import { compileSchema } from '../../utils/ajv.js';
 import { buildMCPResponse } from '../../utils/mcp.js';
@@ -46,7 +46,7 @@ USAGE EXAMPLES:
             return buildMCPResponse({
                 texts: [`Dataset '${parsed.datasetId}' not found.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.SOFT_FAIL,
+                telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL, failureCategory: FAILURE_CATEGORY.INVALID_INPUT },
             });
         }
         return { content: [{ type: 'text', text: `\`\`\`json\n${JSON.stringify(v)}\n\`\`\`` }] };

src/tools/common/get_dataset.ts

@@ -84,7 +84,7 @@ export const getDatasetItems: ToolEntry = Object.freeze({
             return buildMCPResponse({
                 texts: [`Dataset '${parsed.datasetId}' not found.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.SOFT_FAIL,
+                telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL },
             });
         }
 

src/tools/common/get_dataset_items.ts

@@ -61,7 +61,7 @@ USAGE EXAMPLES:
             return buildMCPResponse({
                 texts: [`Dataset '${parsed.datasetId}' not found.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.SOFT_FAIL,
+                telemetry: { toolStatus: TOOL_STATUS.SOFT_FAIL },
             });
         }
 
@@ -83,7 +83,7 @@ USAGE EXAMPLES:
             return buildMCPResponse({
                 texts: [`Failed to generate schema for dataset '${parsed.datasetId}'.`],
                 isError: true,
-                toolStatus: TOOL_STATUS.FAILED,
+                telemetry: { toolStatus: TOOL_STATUS.FAILED },
             });
         }