feat: add giskard-llm as lean litellm replacement

[Hartorn]

Summary

• Introduces a new giskard-llm library that replaces litellm as the default routing layer for giskard-agents, wrapping native provider SDKs (OpenAI, Google GenAI, Anthropic, Azure OpenAI, Azure AI Foundry) and eliminating 200+ transitive dependencies.
• Updates giskard-agents to import from giskard.llm via GiskardLLMGenerator, with pass-through extras so users install only the provider SDKs they need (pip install giskard-agents[openai]).
• Keeps LiteLLMGenerator available as a first-class alternative backend behind an optional litellm extra, so both backends can coexist in the same process.
• Adds provider-specific pytest marks (@pytest.mark.google, @pytest.mark.litellm, etc.) with auto-skip logic and a CI matrix that installs/tests each backend + provider independently.

What changed

New: libs/giskard-llm/

• Protocol-based provider system (providers/base.py) — CompletionProvider, EmbeddingProvider, ResponseProvider protocols. Providers implement only the protocols they support (e.g. Anthropic implements only CompletionProvider).
• Provider adapters (providers/openai.py, providers/google.py, providers/anthropic.py, providers/azure_openai.py, providers/azure_ai.py) — thin wrappers over native SDKs with unified error mapping via _map_error(), structured output via native SDK features, and class-level _PROVIDER attribute for correct error attribution in Azure subclasses.
• Response/Interactions API — ResponseProvider protocol with respond() method, supporting OpenAI Responses API and Gemini Interactions API with unified ResponseResult, ResponseOutputText, ResponseOutputFunctionCall types. Tool definitions are converted from nested Chat Completions format to the flat format expected by each API.
• Canonical tool result format — FunctionCallOutput TypedDict as the unified input format for feeding back tool results. GoogleProvider normalizes this to function_result format internally.
• Unified error hierarchy (errors.py) — LLMError base with AuthenticationError, RateLimitError, ServerError, LLMTimeoutError, BadRequestError, UnsupportedOperationError, ProviderNotAvailableError, each carrying a status_code for retry decisions. ProviderNotAvailableError accepts an optional extra hint (e.g. extra="azure" → pip install giskard-llm[azure]).
• Routing (routing.py) — parses "provider/model" strings and dispatches to the correct provider adapter. LLMClient supports named configurations via configure(). Module-level configure(), reset(), acompletion(), aembedding(), aresponse() convenience functions.
• Response types (types.py) — Pydantic v2 models inheriting from _BaseModel (which defaults model_dump to exclude_none=True). CompletionResponse, EmbeddingResponse, ResponseResult. ToolCallFunction.arguments is str (JSON) for wire-compatible round-trips. TypedDicts (ToolDef, ChatMessage, FunctionCallOutput) for user-constructed inputs.
• Retry helper (retry.py) — class-based RETRYABLE_ERRORS = frozenset({LLMTimeoutError, RateLimitError, ServerError}) and a should_retry(exc) helper — no HTTP-status branching at the retry layer.
• Design documentation (docs/design.md) — documents type conventions (TypedDict vs Pydantic), tool format differences, canonical tool result format, and key design decisions.
• Unit tests — cover routing, retry, types, errors, error mapping completeness, message conversion, input normalization, and mocked provider integration including protocol conformance checks.
• Functional tests — per-provider tests for completion, embedding, response API, tool roundtrips (hardcoded + parallel), multi-system messages, and behavior config, auto-skipped if the SDK extra isn't installed.

Changed: libs/giskard-agents/

• GiskardLLMGenerator (renamed from LiteLLMGenerator) and litellm_embedding_model.py now import acompletion, aembedding, should_retry from giskard.llm.
• pyproject.toml replaces the litellm hard dep with giskard-llm>=1.0.0a1 and re-exports its provider extras (openai, google, anthropic, all). A new litellm extra re-enables the optional LiteLLMGenerator.
• LiteLLMGenerator restored as a first-class alternative backend using the litellm SDK. Importing it without the extra raises ImportError with a clear install hint; it is lazily re-exported from generators/__init__.py via __getattr__ so plain imports of giskard.agents never require litellm. GiskardLLMGenerator no longer claims the "litellm" discriminator kind — each class owns its own kind.
• Test fixtures updated to use CompletionResponse / EmbeddingResponse from giskard.llm.
• conftest.py adds a pytest_collection_modifyitems hook to auto-skip provider-marked tests when the SDK isn't installed, plus an autouse fixture to clear the provider cache between tests.
• New test_litellm_generator.py — unit tests for the LiteLLM adapter (mocked acompletion, retry middleware status-code mapping, discriminator round-trip).
• New test_generator_backends.py — functional tests exercising both backends through direct complete() and ChatWorkflow, plus a test_backends_coexist_in_same_process test that only runs when both SDKs are installed.

Changed: CI

• ci.yml adds giskard-llm to the unit test matrix (Python 3.12–3.14) and a test-no-providers job for SDK-less error handling.
• integration-tests.yml:
  • test-agents-functional uses a 3-layout × 3-python matrix (giskard-llm / litellm / both) installed via top-level giskard-agents[...] extras so the pass-through declarations are exercised.
  • Adds test-llm-functional: matrix over 5 providers (openai, google, anthropic, azure, azure_ai), secrets scoped to ci environment.
  • Adds test-checks-functional: matrix over google provider, secrets scoped to ci environment.
  • Adds workflow_dispatch trigger for manual runs with head.sha || github.ref fallback.
  • All checkout steps use persist-credentials: false.
• labeler.yml adds a Scope: LLM label for libs/giskard-llm/**.
• Makefile gains test-no-providers and install-no-providers targets.
• Default models: gpt-4.1-nano (OpenAI/Azure), gemini-2.5-flash (Google), claude-haiku-4-5-20251001 (Anthropic). Azure OpenAI default api_version updated to 2024-10-21.

Bug fixes (from review follow-up)

• Google Interactions API input normalization — rewrote _normalize_input_items to convert all items to Google's preferred flat ContentParam format (Pattern A). function_call uses id (not call_id) with arguments as dict; function_call_output → function_result with resolved name; role-tagged turns → TextContentParam. Docstring links to SDK type definitions, official example, and known issue #1906.
• Google _to_response_result — prioritize output.id over output.call_id for function call IDs (matching what Google actually returns).
• Google _convert_messages — resolve tool_call_id to actual function name from preceding assistant tool_calls; set role="user" for role="tool" messages.
• Google _map_error — handle API_KEY_INVALID string check for AuthenticationError; fall back to status 500 on SDK APIError.
• Anthropic _validate_messages — skip alternation check when either role is "tool" (consecutive tool messages are valid since they merge).
• Anthropic _convert_messages — merge consecutive role="tool" messages into a single user message with multiple tool_result blocks.
• OpenAI _normalize_input_item — serialize function_call.arguments from dict to JSON string (OpenAI Responses API rejects dicts).
• OpenAI _map_error — tolerate APIConnectionError missing a status_code attribute.
• Moved pyright ignores to the from line so they apply to optional SDK imports without leaking to the rest of the module.

Misc

• pyrightconfig.json adds libs/giskard-llm/src to execution environments.
• Workspace pyproject.toml adds the new package.
• uv.lock regenerated after the litellm removal and follow-up dependency upgrades (anthropic, cryptography, google-genai, logfire-api, numpy, and other transitive bumps).
• .secrets.baseline refreshed to track new test-file line numbers already flagged as false positives.

Test plan

• All unit tests pass across giskard-llm, giskard-agents, giskard-checks, giskard-core (103 giskard-llm tests)
• All pre-commit hooks pass (ruff check, ruff format, trailing-whitespace, end-of-file-fixer, detect-secrets)
• basedpyright passes with 0 errors
• CI green on this branch
• Integration tests (last dispatch — squashed commit): https://github.com/Giskard-AI/giskard-oss/actions/runs/24555428273
• Integration tests (previous run, all 18 jobs green): https://github.com/Giskard-AI/giskard-oss/actions/runs/24555298358

To re-run integration tests manually: Dispatch workflow with ref feat/giskard-llm.

Made with Cursor

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the LLM integration within the project by replacing the litellm dependency with a custom-built giskard-llm library. This change aims to streamline LLM interactions, reduce the overall dependency footprint, and provide more granular control over provider-specific functionalities. The update also includes improvements to the testing framework, enabling more efficient and targeted testing of different LLM providers.

Highlights

• New giskard-llm Library: Introduced a new giskard-llm library to replace litellm, providing a lightweight routing layer over native provider SDKs (OpenAI, Google GenAI, Anthropic) and significantly reducing transitive dependencies.
• Updated giskard-agents Integration: giskard-agents now imports directly from giskard.llm and supports pass-through extras, allowing users to install only the necessary provider SDKs (e.g., pip install giskard-agents[openai]).
• Enhanced Testing Infrastructure: Added provider-specific pytest marks (@pytest.mark.google, etc.) with automatic skip logic and a CI matrix to install and test each provider independently.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

• Ignored by pattern: .github/workflows/** (2)
  • .github/workflows/ci.yml
  • .github/workflows/integration-tests.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new giskard-llm library, serving as a lightweight routing layer for various LLM providers (OpenAI, Google Gemini, Anthropic). The giskard-agents library has been refactored to depend on and utilize this new giskard-llm library, replacing its direct dependency on litellm. This change involves updating imports, adapting to the new unified response types for completions and embeddings, and modifying the retry middleware to use giskard-llm's error handling. Additionally, the build system and test suite have been updated to include the new library and support provider-specific functional testing, allowing tests to be skipped if a provider's SDK is not installed. No feedback to provide.