You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[Install our Python library and testing server](#installation)
38
-
-[Scan your model to detect vulnerabilities](#scan-your-model-to-detect-vulnerabilities)
39
-
-[Automatically generate a test suite](#automatically-generate-a-test-suite-based-on-the-scan-results)
40
-
-[Upload your test suite to the Giskard server](#upload-your-test-suite-to-the-giskard-server)
41
-
* 👋 **[How to contribute](#how-to-contribute)**
42
-
* 💖 **[Like what we're doing?](#like-what-were-doing)**
43
-
44
-
45
-
## Why Giskard?
46
-
**Giskard is an open-source testing framework dedicated to ML models, from tabular models to LLMs.**
47
-
48
-
Testing Machine Learning applications can be tedious. Since ML models depend on data, testing scenarios depend on the domain specificities and are often infinite.
34
+
## Install Giskard 🐢
35
+
You can install the latest version of Giskard from PyPi using pip :
At Giskard, we believe that Machine Learning needs its own testing framework. Created by ML engineers for ML engineers, Giskard enables you to:
49
+
Giskard is a Python library that uses a variety of techniques to **detect vulnerabilities**, including:
59
50
60
-
-**Scan your model to find dozens of hidden vulnerabilities**: The Giskard scan automatically detects vulnerability issues such as performance bias, data leakage, unrobustness, spurious correlation, overconfidence, underconfidence, unethical issue, etc.
51
+
-**Data slicing and transformation**: Giskard can automatically generate different data slices and transformations to test the robustness of your model.
52
+
-**Statistical analysis**: Giskard can use statistical analysis to identify patterns and relationships in your data that could indicate a vulnerability.
61
53
54
+
It's a powerful tool that helps data scientists **save time and effort** drilling down on model issues, and produce more **reliable and trustworthy models**.
-**Instantaneously generate domain-specific tests**: Giskard automatically generates relevant tests based on the vulnerabilities detected by the scan. You can easily customize the tests depending on your use case by defining domain-specific data slicers and transformers as fixtures of your test suites.
60
+
Instantaneously generate test suites for your models ⤵️
<imgsrc="readme/suite_example.png"alt="Test Suite Example"width="800">
70
64
</p>
71
65
72
-
-**Leverage the Quality Assurance best practices of the open-source community**: The Giskard catalog enables you to easily contribute and load data slicing & transformation functions such as AI-based detectors (toxicity, hate, etc.), generators (typos, paraphraser, etc.), or evaluators. Inspired by the Hugging Face philosophy, the aim of Giskard is to become the open-source hub of ML Quality Assurance.
After having wrapped your [model](https://docs.giskard.ai/en/latest/guides/wrap_model/index.html) & [dataset](https://docs.giskard.ai/en/latest/guides/wrap_dataset/index.html), you can scan your model for vulnerabilities using:
85
+
## 1. 🔎 Scan your model
86
+
Here's an example of Giskard scan on the famous titanic survival prediction dataset:
# Wrap your Pandas DataFrame with Giskard.Dataset (test set, a golden dataset, etc.). Check the dedicated doc page: https://docs.giskard.ai/en/latest/guides/wrap_dataset/index.html
95
+
# Wrap your Pandas DataFrame with Giskard.Dataset (test set, a golden dataset, etc.).
109
96
giskard_dataset = giskard.Dataset(
110
97
df=df, # A pandas.DataFrame that contains the raw data (before all the pre-processing steps) and the actual ground truth variable (target).
model=prediction_function, # A prediction function that encapsulates all the data pre-processing steps and that could be executed with the dataset used by the scan.
112
+
model=demo_model, # A prediction function that encapsulates all the data pre-processing steps and that could be executed with the dataset used by the scan.
126
113
model_type="classification", # Either regression, classification or text_generation.
127
114
name="Titanic model", # Optional
128
115
classification_labels=demo_sklearn_model.classes_, # Their order MUST be identical to the prediction_function's output order
129
116
feature_names=['PassengerId', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked'], # Default: all columns of your dataset
Once the scan completes, you can display the results directly in your notebook:
138
126
139
127
```python
140
-
display(scan_results)# in your notebook
128
+
display(scan_results)
141
129
```
130
+
*If you're facing issues, check out our wrapping [model](https://docs.giskard.ai/en/latest/guides/wrap_model/index.html) & [dataset](https://docs.giskard.ai/en/latest/guides/wrap_dataset/index.html) docs for more information.*
131
+
## 2. 🪄 Automatically generate a test suite
142
132
143
-
### Automatically generate a test suite based on the scan results
144
-
145
-
If the scan found potential issues in your model, you can automatically generate a test suite.
146
-
147
-
Generating a test suite from your scan results will enable you to:
148
-
- Turn the issues you found into actionable tests that you can directly integrate in your CI/CD pipeline
149
-
- Diagnose your vulnerabilities and debug the issues you found in the scan
133
+
If the scan found potential issues in your model, you can automatically generate a **test suite** based on the vulnerabilities found:
150
134
151
135
```python
152
136
test_suite = scan_results.generate_test_suite("My first test suite")
153
-
154
-
# You can run the test suite locally to verify that it reproduces the issues
137
+
```
138
+
You can then run the test suite locally to verify that it reproduces the issues:
139
+
```python
155
140
test_suite.run()
156
141
```
157
142
158
-
### Upload your test suite to the Giskard server
143
+
Test suites are reusable objects that provide a way to apply consistent checks on your models. To drill down on failing tests and get even more out of the giskard library, we recommend heading over to the Giskard server ⤵️
159
144
160
-
You can then upload the test suite to the local Giskard server. This will enable you to:
161
-
- Compare the quality of different models to decide which one to promote
162
-
- Debug your tests to diagnose the identified issues
163
-
- Create more domain-specific tests relevant to your use case
164
-
- Share results, and collaborate with your team to integrate business feedback
145
+
# ⭐️ Premium Features
165
146
166
-
First, install the Giskard server by following [this documentation](https://docs.giskard.ai/en/latest/guides/installation_app/index.html)
147
+
The Giskard server is Giskard's premium offering. It provides a number of additional capabilities that are not available in the open-source version of Giskard, including:
167
148
168
-
```python
169
-
# Create a Giskard client after having installed the Giskard server (see documentation)
170
-
token ="API_TOKEN"# Find it in Settings in the Giskard server
171
-
client = GiskardClient(
172
-
url="http://localhost:19000", token=token # URL of your Giskard instance
173
-
)
149
+
-**Advanced test generation**: This includes the ability to to diagnose failing tests, debug your models and create more domain-specific tests.
150
+
-**Model comparison**: This includes the ability to compare models in order to decide which one to promote,
151
+
-**Test hub**: This includes a place to gather all of your team's tests in one place to collaborate more efficiently.
152
+
-**Business feedback**: This includes the ability to share your results and collect business feedback from your team.
To start the **Giskard server**, run the following command:
163
+
```sh
164
+
giskard server start
180
165
```
166
+
167
+
🚀 That's it! Access it at http://localhost:19000
168
+
169
+
## 2. Upload your test suite to the Giskard server
170
+
171
+
You can then **upload the test suite** created using the `giskard` Python library to the Giskard server. This will enable you to:
172
+
- Compare the quality of different models to decide which one to promote
173
+
- Debug your tests to diagnose identified vulnerabilities
174
+
- Create more domain-specific tests relevant to your use-case
175
+
- Share results, and collaborate with your team to integrate business feedback
176
+
177
+
1. First, make sure Giskard server is installed
178
+
<details>
179
+
<summary>How to check if the Giskard server is running</summary>
180
+
181
+
- check if http://localhost:19000 is running
182
+
- or use `giskard server status`
183
+
</details>
184
+
185
+
2. Then execute the ML worker in your notebook:
186
+
```python
187
+
!giskard worker start -d -k YOUR_TOKEN
188
+
```
189
+
190
+
3. Finally upload your test suite to the giskard server using the following code:
191
+
```python
192
+
token ="API_TOKEN"# Find it in Settings in the Giskard server
193
+
client = GiskardClient(
194
+
url="http://localhost:19000", token=token # URL of your Giskard instance
195
+
)
181
196
182
-
For more information on uploading to your local Giskard server, go to the [Upload an object to the Giskard server](https://docs.giskard.ai/en/latest/guides/upload/index.html) page.
We welcome contributions from the Machine Learning community!
203
+
# ❓ Where can I get more help?
186
204
187
-
Read this [guide](CONTRIBUTING.md) to get started.
188
205
189
-
<br />
206
+
<details>
207
+
<summary>What is a ML worker?</summary>
208
+
209
+
Giskard executes your model using a worker that runs the model directly in your Python environment containing all the dependencies required by your model. You can either execute the ML worker from a local notebook, a Colab notebook or a terminal.
210
+
</details>
211
+
212
+
<details>
213
+
<summary>How to get the API key</summary>
214
+
215
+
Access the API key in the Settings tab of the Giskard server.
<summary>For more information on uploading to your local Giskard server</summary>
234
+
235
+
Go to the [Upload an object to the Giskard server](https://docs.giskard.ai/en/latest/guides/upload/index.html) page.
236
+
</details>
237
+
238
+
For any other questions and doubts, head over to our [Discord](https://gisk.ar/discord).
239
+
240
+
# 👋 Community
241
+
We welcome contributions from the Machine Learning community! Read this [guide](CONTRIBUTING.md) to get started.
242
+
243
+
Join our thriving community on our Discord server : [join Discord server](https://gisk.ar/discord)
192
244
193
245
🌟 [Leave us a star](https://github.com/Giskard-AI/giskard), it helps the project to get discovered by others and keeps us motivated to build awesome open-source tools! 🌟
194
246
195
247
❤️ You can also [sponsor us](https://github.com/sponsors/Giskard-AI) on GitHub. With a monthly sponsor subscription, you can get a sponsor badge and get your bug reports prioritized. We also offer one-time sponsoring if you want us to get involved in a consulting project, run a workshop, or give a talk at your company.
0 commit comments