Change how SQL database schema is initialized - adopting SQL raw files #12341
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
amaltaro
force-pushed
the
fix-12339
branch
2 times, most recently
from
b2fa410 to
c3da3ec
Compare
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
Streamline DB initialization in WMInit and TestInit Backwards compatibility from setSchema() to setSchemaFromModules() Correct map of old to new SQL module names Update DBS3Buffer module map; add tier0 map files Fix agent SQL map; remove call to self.init.setSchemaFromComponent Add WMQuality.TestDB schema to WMInit Avoid primary key violation for WMBS/Job - wmbs_job_assoc Now Job object can be associated multiple times some pylint fixes for WMInit Reorganize SQL files structure to keep db flavor at top level Use WMCORE_ROOT env var to find the SQL files in production Fix exception args in WMInit Avoid double sql directory Fix tier0 module dependency REMOVE: temporary SQL files to test unit tests REMOVE: add tier0 SQL files Add testdb SQL schema Indexes and primary key fixes for wmbs SQL files Remove primary keys for TestDB tables Add SQL files to new structure REMOVE-ME: removed tablespace from wmbs indexes REMOVE-ME: fix initial data for wmbs REMOVE-ME: more consistent database schema for Oracle REMOVE-ME: make wmbs_job.retry_count larger to support 99999 REMOVE-ME: sync last sql schema changes REMOVE-ME: update tier0 create tables file Oracle schema requires splitting of statements Refactor all Oracle SQL files to use slash as statement terminator update create_wmbs_tables with slash terminator Increase id column length for t0_deployment_id Update WMInit to separate statements using slask instead of semicolon Remove hard-coded SQL files from this PR Update requirements to new wmcoredb version; adopt wmcoredb in WMInit Update wmcoredb to 0.9.2 Add debugging to WMInit Extra debugging Remove debugging from WMInit Update wmcoredb to 0.9.3
Remove TestDB Create.py modules Remove ResourceControl Create.py modules Remove Agent Create.py modules Remove BossAir Create.py modules Remove DBS3Buffer Create.py modules
…oincrement now Fix SEQ.nextval changes for wmbs_subscription Ignore duplicate bl_status values in Oracle Fix NewState Oracle DAO Fix again Oracle NewState DAO Stop using SEQ.nextval in the WMComponent DAOs, as it uses autoincrement now
Skip None checksum insertion in JobAccountant
|
Jenkins results:
|
WMInit unit tests for _getSQLStatements Fix unit tests adopting wmcoredb to locate SQL files more fixes to unit test Move ResourceControl_t.py test to SQL based schema Adapt ResourceControl_t to use a friendly DB initialization Fix modules for setSchema in DBSBufferFile_t.py Fix many unit tests Fix ChangeState_t unit test Fix unit tests for wma_init
|
Jenkins results:
|
|
I would say I have completed all this development now and the description should be summarizing all of the changes and pointing to related/dependent external pull requests. Please let me know if you want me to further squash these commits. I tried to separate them by concerns (new sql model, removal of old python schema, bug fix in the CRUD DAOs, unit tests, etc) and I would suggest to keep the current granularity, but I am happy to squeeze them further. Thanks! |
todor-ivanov
approved these changes
Jul 21, 2025
|
Thanks Todor. I am getting this merged and its dependencies merged then. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #12339
Status
ready
Description
This pull request changes how the database schema is defined and initialized in WMCore/WMAgent, using the SQL plain files defined in this repository https://github.com/dmwm/wmcoredb instead. For the conversion from Python-based to plain SQL files, files have been identified and extract in WMCore and T0 repositories.
Summary of changes/commits so far are:
wmcoredb(which now contains the SQL files and look-up functions)WMCore.WMBS) to the new schema name (e.g.wmbs)WMInit.setSchema()with the old module name, but it maps it to the new module and callsWMInit.setSchemaFromModules().WMCORE_ROOTenv var; or fallback to the base path used forwmcoredblibrary;processData()(if we execute the full content of an SQL file through SQLAlchemy, it will fail)wm_inittable creation with the new SQL schema (hence, to be removed from CMSKubernetes)SEQ.nextvalfor Oracle, as we no longer use sequences for autoincrement, but we rely on theIDENTITYfeature available as of Oracle12c.Deprecate old Python-based database schema data:
WMCore actual bug-fixes:
WMBS.Job.AddFilesMySQL DAO was missing anIGNOREclause;nonejob status frombl_statustable (batch system status)wmbs_job_mask.inclusivemaskcolumnIs it backward compatible (if not, which system it affects?)
YES (it should be)
Related PRs
None
External dependencies / deployment changes
wma_inittable creation from CMSKubernetes scripts: Move wma_init table creation to the actual SQL schema CMSKubernetes#1633