[GSK-1235] Fix segmentation fault problems when running pytest

[mattbit]

Should solve the problems with fixtures tripadvisor and fraud_detection when running pytest locally on certain environments. @AbSsEnT could you check that this also solves the problem on your machine?

[AbSsEnT]

@mattbit will check it in 20 minutes

[AbSsEnT]

@mattbit I am still getting segfault for the fraud_detection fixture.

[linear]

GSK-1235 Segfault during unit-testing.

• Environment: Python 3.10, MacOS
• Branch: ai-test-v2-merged
• Error: Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
• Description: If I run unit-testing, I am getting above error.
  1. First, I investigated, that this problem appears, during initialisation of the tripadvisor_text_classification_torchfixture, specifically, under the hood of the DistillBert PyTorch model initialisation. Exact operation, when this error happens is copy_of one tensor values to another. I fixed this problem by adding import torch in the __init__.py of the tests module.
  2. After above fix, this error again appears during execution of the test with fraud_detection__binary_classification fixture. Specifically execution fails, when LGBM classifier model is fit. Exact operation, where an error appears is _LIB.LGBM_DatasetCreateFromMat, where _LIBis a cdll and is loaded using ctypes.cdll.LoadLibrary. As in previous step I tried to fix this problem by importing ctypes module inside tests/__init__.py, but It did not help. For this moment it is not fixed.
  3. Segfault does not appear if run code outside of the unit-testing regime.

[andreybavt]

@AbSsEnT , can you test it again to see if you still get a segfault?

[AbSsEnT]

@AbSsEnT , can you test it again to see if you still get a segfault?

@andreybavt still getting segfault for the fraud_detection fixture.

[mattbit]

@andreybavt can we merge this?

[andreybavt]

@mattbit , do we still need to make changes to the lightgbm version to solve Mykyta's issue?

[mattbit]

@mattbit , do we still need to make changes to the lightgbm version to solve Mykyta's issue?

It would be useful because it removes a few unnecessary steps (download of stopwords which are not used), avoids downloading and initializing DistilBERT at import time, and frees memory since model is now loaded in the test scope.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

100.0% Coverage
0.0% Duplication