feat!: rm user foreign key in db#443
Conversation
When this model is used with huge traffic, students completing multiple blocks in courses. The User table is also locked with other models. This slow all the inserts and updates using User table. (cherry picked from commit 2db3e3a)
openedx-webhooks
added
the
open-source-contribution
|
Thanks for the pull request, @johanseto! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Submit a signed contributor agreement (CLA)
If you've signed an agreement in the past, you may need to re-sign. Once you've signed the CLA, please allow 1 business day for it to be processed. 🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
|
Hi @johanseto! Welcome, and thank you for this contribution! In order for your CLA check to turn green, you'll need to submit a CLA form. If you are contributing as an individual, please fill out the individual CLA form here. If you are contributing on behalf of an organization, please have your manager reach out to oscm@axim.org so you may be added to your org's existing entity agreement. Please let me know if you have any questions. Thanks! |
mphilbrick211
added
the
needs test run
mphilbrick211
moved this from Needs Triage
to Needs Tests Run or CLA Signed
in Contributions
Done, I already filled out the form |
3 participants
Description
Inspired by the courseware student module's good performance.
https://github.com/openedx/openedx-platform/blob/b1b59ddc5a7d8af31091c74d1f3b75ff38acde3e/lms/djangoapps/courseware/models.py#L95
Context & Architecture
The Completion tracking service is responsible for logging student progress as they navigate and finish multiple blocks within courses. Historically, the database schema for the completion model included a strict, database-level Foreign Key (FK) constraint tied directly to the core User table.
The Problem: High-Traffic Lock Contention
Under standard traffic loads, this schema functions as expected. However, during high-traffic spikes—such as thousands of students concurrently engaging with course material—this strict constraint becomes a severe architectural bottleneck.
When a relational database (e.g., MySQL/InnoDB or PostgreSQL) executes an INSERT or UPDATE on a table with a Foreign Key constraint, it must guarantee referential integrity. To do this safely in a highly concurrent environment, the database engine acquires a shared lock (e.g., SELECT ... FOR SHARE) on the referenced row in the parent table—in this case, the User table.
Because the User table is a highly central entity, it is frequently accessed, updated, and locked by various other domains and models across the application. When a massive influx of completion events occurs simultaneously:
The database queues these concurrent transactions, waiting for locks on the User table to be released.
This lock contention causes a dramatic spike in transaction latency, slowing down all INSERT and UPDATE operations linked to the User table.
Prolonged transactions tie up database connection pools and increase the risk of cascading timeouts and deadlocks.
This PR resolves the bottleneck by removing the explicit database-level foreign key constraint between the Completion model and the User table (e.g., utilizing db_constraint=False in the Django ORM).
While the application code will still conceptually treat this field as a relationship (storing the user_id), the underlying database will no longer enforce the strict referential integrity check on every write operation.
Massively Increased Write Throughput: Inserts and updates to the completion logs are now isolated from the User table's lock state. They can execute independently and immediately.
Eliminated Contention Bottleneck: By bypassing the shared locks on the User table, we free up database resources, dramatically reducing transaction times and connection queueing.
Improved System Resilience: The platform can now handle massive traffic spikes and concurrent block completions without degrading the performance of the core User table or the application as a whole.
Inspired in nelc#2
test
clone
run
./manage.py lms migrate completionResults in production instance
Before
After
Merge checklist:
Post merge: