Releases: dmwm/WMCore
WMCore 2.4.3 production central services release
This release brings in several updates to third-party packages, such as rucio-clients, CherryPy, psutil, httplib2, etc.
We highlight major changes to MSUnmerged as well, as it now shares the same Debian+Python 3.12 base image as the other WM services, with up-to-date Gfal2 and python-gfal2 libraries and concurrent file/directory cleanup.
Lastly, several bug fixes and optimizations have been included in this release.
Release date: Aug 19, 2025.
Changes since release: 2.4.0.
WMAgent
Software stack
- Update third party python packages (Kenyi Hurtado) #12424
- Added mongomock to requirements (Dennis Lee) #12413
- Update GFAL and MSUnmerged images to Debian Bookworm + Miniconda + Py3.12 dmwm/CMSKubernetes#1627
- update CMSMonitoring version to upgrade jsonschema dependency to align with rucio-clients (Valentin Kuznetsov) #12412
- update rucio client version to 36.5.0 (Valentin Kuznetsov) #12201
Features and/or feature changes
- Replace deleteRule with updateRule with lifetime=0 (Kenyi Hurtado) #12384
- Rename GPU related parameters and split CUDACapability, following (Kenyi Hurtado) #12416
- MSUnmerged refactoring for decreased memory footprint and concurrent deletions through thread pool (Alan Malta Rodrigues) #12059
Bug Fixes
- Fix ReqMgr2 data type validation (Alan Malta Rodrigues) #12423
- Update Reqmgr workload cache for dynamic workflow parameter changes. (Todor Ivanov) #12405
- Install setuptools in the CD pipeline (Alan Malta Rodrigues) #12411
- Update CD python version to 3.12; Skip PyPi publication if same package+version has already been uploaded (Alan Malta Rodrigues) #12410
Enhancements
- Remove unused DQMCat module (Alan Malta Rodrigues) #12341
WMCore 2.4.2 production WMAgent release
This release contains a backward incompatible model for WMAgent SQL schema management, now depending on a new wmcoredb python package. In addition, it brings in several updates to third-party packages, such as rucio-clients, CherryPy, psutil, httplib2, etc.
MSUnmerged deserves to be highlighted as well, as it now shares the same Debian+Python 3.12 base image as the other WM services, with up-to-date Gfal2 and python-gfal2 libraries. Lastly, several bug fixes and optimizations have been included in this release.
Release date: Aug 15, 2025.
Changes since release: 2.4.1.
WMAgent
Software stack
- Update third party python packages (Kenyi Hurtado) #12424
- Add SQL schema wmcoredb package as a dependency (Alan Malta Rodrigues) #12408
- Added mongomock to requirements (Dennis Lee) #12413
- Update GFAL and MSUnmerged images to Debian Bookworm + Miniconda + Py3.12 dmwm/CMSKubernetes#1627
- update CMSMonitoring version to upgrade jsonschema dependency to align with rucio-clients (Valentin Kuznetsov) #12412
- update rucio client version to 36.5.0 (Valentin Kuznetsov) #12201
Features and/or feature changes
- New module and mechanism to monitor CouchDB process and replications (Valentin Kuznetsov) #12353 #12425 #12426
- Replace deleteRule with updateRule with lifetime=0 (Kenyi Hurtado) #12384
- Rename GPU related parameters and split CUDACapability, following (Kenyi Hurtado) #12416
- New model for deploying SQL schema; adopted wmcoredb in WMInit; update Oracle DAOs to use Oracle's auto-increment; etc (Alan Malta Rodrigues) #12341
- Remove Python-based CreateWMBSBase/Create/Destroy modules (Alan Malta Rodrigues) #12341
- MSUnmerged refactoring for decreased memory footprint and concurrent deletions through thread pool (Alan Malta Rodrigues) #12059
Bug Fixes
- Fix ReqMgr2 data type validation (Alan Malta Rodrigues) #12423
- Fix WMAGENT_CONFIG exports (Todor Ivanov) #12400
- Skip None checksum insertion for wmbs_file_checksums; many other bug fixes (Alan Malta Rodrigues) #12341
- Update Reqmgr workload cache for dynamic workflow parameter changes. (Todor Ivanov) #12405
- Install setuptools in the CD pipeline (Alan Malta Rodrigues) #12411
- Update CD python version to 3.12; Skip PyPi publication if same package+version has already been uploaded (Alan Malta Rodrigues) #12410
- fix bug checkProxy incompatible (Viphava280444) #12403
Enhancements
- Remove unused DQMCat module (Alan Malta Rodrigues) #12341
WMCore 2.4.1 production WMAgent release
This is the first release of WMAgent running python 3.12.
Moreover, this release includes a new way to monitor status of components and worker threads, based on psutils library and standard Linux mechanisms.
Lastly, Local WorkQueue is expected to be smaller, since the multiplies factor has been reduced from .75 to .50
Release date: Jun 18, 2025.
Changes since release: 2.3.11.1.
WMAgent
Software Stack
- python 3.12 - necessary requirements.txt upgrades (Dario Mapelli) #12336
- python 3.12 - do not use codecs (Dario Mapelli) #12338
this release also required the following changes to the dmwm/CMSKubernetes repository:
- wmcore - switch to py3.12 for pypi docker images dmwm/CMSKubernetes#1611
- cmsweb/exporters - build a new version dmwm/CMSKubernetes#1613
- dmwm-base - remove pycurl from OS && add procps: dmwm/CMSKubernetes#1622, dmwm/CMSKubernetes#1621
Features and/or feature changes
- [monitoring] add thread info status to WMAgent info pushed to CouchDB (Valentin Kuznetsov) #12302
- [monitoring] fix issues with WMagent start; and couple of major warnings reported by pylint (Valentin Kuznetsov) #12362
- [monitoring] Change look-up states for process status (Valentin Kuznetsov) #12372
- [monitoring] setup default value for cycle_time (Valentin Kuznetsov) #12372
- [monitoring] provide necessary checks to look-up valid process ID status (Valentin Kuznetsov) #12387
- Decrease site slot multiplier factor from .75 to .50 (Alan Malta Rodrigues) #12365
- Add getCodes method to fetch all dbs2go codes from its error JSON response (Valentin Kuznetsov) #12312
bug fixes
- Force cherrypy to set content-length=0 before request body if FE does not provide it (Valentin Kuznetsov) #12371
- fix issue with restartComponent.sh script (Valentin Kuznetsov) #12352
- [stageout] Default first stage out to X509 in case bearer token is not set (Alan Malta Rodrigues) #12380
- [stageout] Do not hang on unset BEARER_TOKEN_FILE env var (Alan Malta Rodrigues) #12380
- [stageout] Avoid token-based file deletion in GFAL2, unless environ is defined (Alan Malta Rodrigues) #12377
Enhancements
- Update checkProxy script to use valid domain name (Alan Malta Rodrigues) #12392
- Add Oracle DatabaseError DPI-1080 as a soft error (Alan Malta Rodrigues) #12379
- Make JobAccountant output module evaluation more verbose (Alan Malta Rodrigues) #12351
- Avoid crashing ErrorHandler on failed DNS resolution (Alan Malta Rodrigues) #12369
- [venv] Split patch files by source files edited && Apply them separately (Todor Ivanov) #12346
- [venv] Add option to chose custom built python (Todor Ivanov) #12381
WMCore 2.4.0 production central services release
This new Central Services tag mainly brings in a new version of the python runtime, python 3.12. Moreover, we increased the target job length from 8h to 12h.
Release date: Jun 17, 2025.
Changes since release: 2.3.10.3.
Central services
Software stack
- python 3.12 - necessary requirements.txt upgrades (Dario Mapelli) #12336
- python 3.12 - do not use codecs (Dario Mapelli) #12338
- switch from distribution.data_files to distribution.package_data (Valentin Kuznetsov) #12359
- Replaces nose with pynose, remove nose2 in requirements (Dennis Lee) #12390-
deployment
this release also required the following changes to the dmwm/CMSKubernetes repository:
- wmcore - switch to py3.12 for pypi docker images dmwm/CMSKubernetes#1611
- cmsweb/exporters - build a new version dmwm/CMSKubernetes#1613
- wmcore - load all CAs from k8s host dmwm/CMSKubernetes#1620
- dmwm-base - remove pycurl from OS && add procps: dmwm/CMSKubernetes#1622, dmwm/CMSKubernetes#1621
Features and/or feature changes
- Switch default job length from 8 to 12h (Alan Malta Rodrigues) #12360
Bug Fixes
- Force cherrypy to set content-length=0 before request body if FE does not provide it (Valentin Kuznetsov) #12371
Enhancements
- Further updates to the JSON templates, fix some comments and resource requirements (Alan Malta Rodrigues) #12354
- Fix bash typoed variable (Fredrik) #12364
- Yuiserver was removed in 2013 (#4495) (Fredrik) #12356
WMCore 2.3.11.1 production WMAgent release
This is the first WMAgent stable release built with the setuptools library, replacing distutils (required for the overall project compliance with PEP-625). Further on the software stack, htcondor, pycurl and PyOpenSSL libraries have been updated.
This release brings in the CPU microarchitecture job feature, some important token and stage out developments and required changes to move towards Python 3.12. Please check out the full release notes for further information on other developments, bug-fixes and optimizations.
Release date: May 2, 2025.
Changes since release: 2.3.9.2.
WMAgent
Software stack
- Upgrade htcondor python bindings to v24.0.7 (Todor Ivanov) #12344
- Update pycurl and PyOpenSSL versions (Kenyi Hurtado) #12286
- Switch WM packaging from distutils to setuptools (Valentin Kuznetsov) #12259 #12273 #12274 #12275 #12276 #12277 #12281
Features and/or feature changes
- Get rid of transactionId from the job configuration, and GlobalTagTransaction from Express specs (Kenyi Hurtado) #12331
- Performance enhancements for primary, parent and pileup data location in global workqueue (Alan Malta Rodrigues) #12330
- Addition of token support for gfal and xrootd, with refactoring of createCopyCommand and createRemoveCommand (Andrea Piccinelli) #12218
- Support nano flavours (Antonio Linares) #12235
- Add support for microarchitecture. When default microarch is not defined in XML file, use "0" as default. (Kenyi Hurtado) #12270
- python 3.11 - inspect.getargspec moved to inspect.getfullargspec (Dario Mapelli) #12289
- Replace schedd.Transaction and queue_with_itemdata() with Schedd.submit() (Kenyi Hurtado) #12287
- Introducing createDebuggingCommand in all the stageout submodules (Andrea Piccinelli) #12284
- python 3.10 - collections.(Mapping|Iterable) moved to collections.abc.(Mapping|Iterable) (Dario Mapelli) #12288
- Replaced ddm_quota with dm_weight, added deprecation warning (Dennis Lee) #12243
- Add token authentication support to job submission, but disable cms token authentication by default. (Kenyi Hurtado) #12196
Bug Fixes
- Properly HandleDidNotfound exception (Todor Ivanov) #12317
- Fix location of rel_microarchs variable in SimpleCondorPlugin (Alan Malta Rodrigues) #12305
Enhancements
- deploy wmange-venv - small changes (Dario Mapelli) #12337
- Code organization for global workqueue data location update (Alan Malta Rodrigues) #12330
- Few typos in the help message (Fredrik) #12322
- Add rucio client instances to wmagent-component-standalone (Todor Ivanov) #12318
- Refactor all JSON workflow templates (Alan Malta Rodrigues) #11949
- Add deploy-unittest-venv.sh (Todor Ivanov) #12310
- Remove unused functions (since 2015 and #5515) (Fredrik) #12300
- Remove unused double stack functions, see #12285 (Fredrik) #12292
- Remove slack from link text (Fredrik) #12268
- Fix logging typo (Fredrik) #12258
WMCore 2.3.10.3 production central services release
This release has backwards incompatible changes, with new PyPi package names compliant with PEP-625. It is also the first stable release that is built with the setuptools library, replacing distutils. Further on the software stack, we upgraded the pycurl and PyOpenSSL libraries.
Concerning the Python codebase, there are a few modifications to start making the code compatible with Python 3.12.
There are other interesting changes in this release and we recommend you to read the full notes below.
Release date: May 2, 2025.
Changes since release: 2.3.8.4.
Central services
Software stack
- Change WM package and images name from reqmgr2ms-xxx to msxxx (Valentin Kuznetsov) dmwm/CMSKubernetes#1586
- Change global-workqueue to wmglobalqueue in its Dockerfile (Valentin Kuznetsov) dmwm/CMSKubernetes#1597
- Move t0_reqmon package to t0reqmon under CMSKubernetes/docker/pypi area (Valentin Kuznetsov) dmwm/CMSKubernetes#1596
- Add PEP-625 compliant changes - package renaming (Valentin Kuznetsov) #12233
- Update pycurl and PyOpenSSL versions (Kenyi Hurtado) #12286
- Switch WM packaging from distutils to setuptools (Valentin Kuznetsov) #12259 #12273 #12274 #12275 #12276 #12277 #12281
Features and/or feature changes
- support raw skim datasets for T0 repack workflows (Antonio) #12298
- Get rid of transactionId from the job configuration, and GlobalTagTransaction from Express specs (Kenyi Hurtado) #12331
- Performance enhancements for primary, parent and pileup data location in global workqueue (Alan Malta Rodrigues) #12330
- use mstrlines to cleanup MSTransferor input rules for announced wfs (Muhammad Hassan Ahmed) #12329
- Support nano flavours (Antonio Linares) #12235
- python 3.13 - deprecate cgi module (Dario Mapelli) #12296
- Adjust AlertManagerAPI to avoid using multiple spaces in various attributes of alert and add UUID label for better traceability of WM alerts (Valentin Kuznetsov) #12237
- python 3.11 - inspect.getargspec moved to inspect.getfullargspec (Dario Mapelli) #12289
- Add appropriate tags and destination for alert routing (Valentin Kuznetsov) #12279
- Replaced ddm_quota with dm_weight, added deprecation warning (Dennis Lee) #12243
Bug Fixes
- Avoid executing archival pipeline for workflows not ready yet (Alan Malta Rodrigues) #12334
- Properly HandleDidNotfound exception (Todor Ivanov) #12317
Enhancements
- Code organization for global workqueue data location update (Alan Malta Rodrigues) #12330
WMCore 2.3.8.4 production central services patch release
In addition to the full release notes described in https://github.com/dmwm/WMCore/releases/tag/2.3.8, it has the following two bug-fixes:
WMCore 2.3.9.2 production WMAgent patch release
A patch release on top of WMAgent https://github.com/dmwm/WMCore/releases/tag/2.3.9.1, resolving the sitelist update feature in the agent, with:
- Make SiteListPoller more verbose; expand update to work not yet in WMBS (Alan Malta Rodrigues) #12254
WMCore 2.3.9.1 production WMAgent release
While several of the bug-fixes provided in this release have also been backported to the previous stable release (2.3.7.x), this release brings in the new functionality for updating workflow site lists for active workflow.
It is also important to highlight that HTCondor Python bindings underwent a major upgrade, from 10.x to 24.3.x. In addition, there are many other bug-fixes and improvements. Note that this WMAgent release cycle is planned to be the last one using the already obsolete Python 3.8.x version.
Release date: 6 February 2025.
Changes since release: 2.3.7.1.
WMAgent
Software stack
- wmagent - htcondor bindings bumped to v24.3 (Dario Mapelli) #12226
- Use s-nail host mail server (Kenyi) dmwm/CMSKubernetes#1573 dmwm/CMSKubernetes#1566
Features and/or feature changes
- Update workqueue selector filter to no longer use ParentQueueUrl (Alan Malta Rodrigues) #12150
- Replace xquery with query, due to deprecation of the former since HTCondor 10.7.0 (Kenyi Hurtado) #12158
- New module to manage site list updates (Valentin Kuznetsov) #12123
- Adopt MSPileup data into PileupFetcher (Alan Malta Rodrigues) #12197
- Switch updateSiteList to updateElementsByWorkflow API (Valentin Kuznetsov) #12219
Bug Fixes
- Fix str to bytes comparisons in checkProxy.py (Todor Ivanov) #12174
- Fix delayed $WMA_DEPLOY_DIR env variable (Todor Ivanov) #12167
- Properly run restartComponent.sh avoiding hardwired paths (Todor Ivanov) #12185
- Few minor environment fixes at deploy time. (Todor Ivanov) #12183
- Fix T1 site cores ResourceControl logic (Alan Malta Rodrigues) #12176
- Change the way to estimate running components (Todor Ivanov) #12189 #12193
- Fix WorkQueue broken reference to rejectedWork and badWork (Alan Malta Rodrigues) #12204
- Fix comparison of task thresholds in ResourceControlUpdater (Alan Malta Rodrigues) #12214
- Populate WMCMSSWSubprocess section for failed jobs too (Alan Malta Rodrigues) #12217
- Adding wmstatsUrl to WorkflowUpdater configuration (Todor Ivanov) #12232 #12236
- Fix Sitelists changes propagation to local workqueue (Todor Ivanov) #12251
- Renaming RequestPriority during WorQueue.updateElements call (Todor Trendafilov Ivanov) #12252
Enhancements
- Changed truster publisher configuration in pypi (Alan Malta Rodrigues)
- Convert reusable to non-reusable GitHub Action (Alan Malta Rodrigues) #12180
- Remove unused JSON encoding choice keywords in RequestDBWriter module (Fredrik) #12141
- Improve patchComponent && Add patchCluster scripts (Todor Ivanov) #12207
WMCore 2.3.8 production central services release
This release offers the capability of changing site lists for active workflows. However, note that it requires WMAgent >=2.3.9 to be fully functional
Additionally, workflow parameter update will now have a tighter validation, in case of unsupported parameter update. It also comes with a couple of improvements for global workqueue and MSOutput.
Release date: 29 January 2025.
Changes since release: 2.3.6.
Central services
Software stack
- Changed truster publisher configuration in pypi (Alan Malta Rodrigues)
- Convert reusable to non-reusable GitHub Action (Alan Malta Rodrigues) #12180
Features and/or feature changes
- Add SiteList actions to transient statuses in ReqMgr2 (Todor Ivanov) #12120
- Add new POST API for transferor data-service (Valentin Kuznetsov) #12155 #12221 #12241
- Add setter methods map to WMWorkload and call it for all reqArg parameters (Todor Ivanov) #12099
- Ignore list order in SiteLists comparison (Todor Ivanov) #12211
- Switch updateSiteList to updateElementsByWorkflow API (Valentin Kuznetsov) #12219
- Add HTTP request from ReqMgr2 to MSTransferor (Valentin Kuznetsov) #12220
Bug Fixes
- Fix WorkQueue broken reference to rejectedWork and badWork (Alan Malta Rodrigues) #12204
- ms-output - do not break out producer and consumer loops in case of generic error (Dario Mapelli) #12194