Add deploy-unittest-venv.sh

[todor-ivanov]

Fixes #12266

Status

testing

Description

With the current PR we add a script that'd allow to set a minimal set of additional environment variables such that we can use an already deployed and initialized virtual environment to develop and run unit tests as well.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

Documentation update PR

External dependencies / deployment changes

None

[todor-ivanov]

Here is the usage of this:

Setting up:

(WMAgent.venv3) cmst1@vocms0260:WMCore $ source $WMA_DEPLOY_DIR/srv/WMCore/deploy/deploy-unittest-venv.sh
Activating: . /data/WMAgent.venv3/bin/activate
Setting up WMCore related environment variables:


Collecting nose
  Using cached nose-1.3.7-py3-none-any.whl.metadata (1.7 kB)
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Installing collected packages: nose
Successfully installed nose-1.3.7
Setting up WMCore related environment variables:

Running unittests:

(WMAgent.venv3) cmst1@vocms0260:WMCore $ ipython -i $WMA_DEPLOY_DIR/srv/WMCore/test/python/WMCore_t/Services_t/WorkQueue_t/WorkQueue_t.py
Python 3.9.21 (main, Dec  5 2024, 00:00:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.18.1 -- An enhanced Interactive Python. Type '?' for help.
%reload_ext autoreload
%autoreload 2
Reading mocked CRIC data from the file /data/WMAgent.venv3/srv/WMCore/test/python/../data/Mock/CRICMockData.json
Running setUp with emulated services.



/data/WMAgent.venv3/srv/WMCore/src/python/WMCore/Database/DBFactory.py:83: SADeprecationWarning: The create_engine.convert_unicode parameter and corresponding dialect-level parameters are deprecated, and will be removed in a future release.  Modern DBAPIs support Python Unicode natively and this parameter is unnecessary.
  create_engine(self.dburl,
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
.Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
Using MockDBSApi
Using MockDBSApi
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
Using MockDBSApi
/data/WMAgent.venv3/srv/WMCore/src/python/Utils/IteratorTools.py:46: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  if isinstance(v, collections.Mapping):
.Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
.Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
Using MockDBSApi
Using MockDBSApi
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
.Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
Using MockDBSApi
Using MockDBSApi
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
Using MockCRICApi
Using MockCRICApi
Using MockCRICApi
.Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox
Using MockDBSApi
Using MockDBSApi
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
.
----------------------------------------------------------------------
Ran 6 tests in 97.256s

OK

In [1]: 

Work interactively with the unittests:

• In parallel to the already created interactive session, open your editor and start coding your new tests or change existing ones:

(WMAgent.venv3) cmst1@vocms0260:WMAgent.venv3 $ emacs $WMA_DEPLOY_DIR/srv/WMCore/test/python/WMCore_t/Services_t/WorkQueue_t/WorkQueue_t.py

• Execute the test. Your new changes must be already in place because %autoreload is enabled by default for the current environment:

In [7]: workQueueTest = WorkQueueTest()

In [8]: workQueueTest.setUp()
Running setUp with emulated services.
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/workqueue_t_inbox
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t
Installing WorkQueue app into database: http://127.0.0.1:5984/local_workqueue_t_inbox

In [9]: workQueueTest.testUpdatePriorityService()
Using MockDBSApi
Using MockDBSApi
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
Using MockRucioApi: acct=wma_test, url=https://cms-rucio-auth-int.cern.ch, authUrl=http://cms-rucio-int.cern.ch
Using MockCRICApi
Using MockCRICApi
Using MockCRICApi
Using MockCRICApi

In [10]: 

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
• Python3 Pylint check: succeeded
• Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/501/artifact/artifacts/PullRequestReport.html

[mapellidario]

niiiice thanks todor! :)

[amaltaro]

@todor-ivanov can you please make the relevant information also available in our documentation and link it in the PR description?

[amaltaro]

see message above

[todor-ivanov]

Hi @amaltaro, thanks! Sorry, for the delayed reply. I am adding the information to the relevant WMCore doc in a minute.

[todor-ivanov]

Here is the PR for the documentation update: https://gitlab.cern.ch/dmwm/wmcore-docs/-/merge_requests/84

[amaltaro]

Thanks Todor.