Use tomllib, if available, else fall back to tomli #5043

deepyaman · 2025-08-06T19:13:26Z

Description

tomli is more standards-compliant than toml, and it's including in the standard library as tomllib since Python 3.11.

toml has not had a release since October 31, 2020.

Development notes

Built the compatibility layer, as documented.

Unfortunately, kedro-telemetry relies on kedro to provide the toml dependency, so that needs to be updated as well.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Deepyaman Datta <[email protected]>

Copilot

Pull Request Overview

This PR modernizes TOML handling by migrating from the toml library to use the standard library tomllib (Python 3.11+) with a fallback to the more standards-compliant tomli library for older Python versions.

Updated dependency from toml to tomli with Python version constraint
Implemented compatibility layer using conditional imports based on Python version
Updated TOML file reading across multiple modules to use the new import structure

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
pyproject.toml	Updated dependency from `toml` to `tomli` with version constraint and fixed mkdocstrings version
kedro/templates/project/hooks/utils.py	Added conditional import for tomllib/tomli and updated TOML operations
kedro/framework/startup.py	Added conditional import and updated exception handling for TOML parsing

Copilot · 2025-08-06T19:14:28Z

kedro/framework/startup.py

@@ -7,7 +7,10 @@
 from pathlib import Path
 from typing import NamedTuple

-import toml
+if sys.version_info >= (3, 11):


Missing import for sys module. The code uses sys.version_info but sys is not imported.

kedro/templates/project/hooks/utils.py

Copilot · 2025-08-06T19:14:29Z

kedro/framework/startup.py

@@ -66,8 +69,8 @@ def _get_project_metadata(project_path: Path) -> ProjectMetadata:
        )

    try:
-        metadata_dict = toml.load(pyproject_toml)
-    except Exception as exc:
+        metadata_dict = tomllib.load(pyproject_toml)


The tomllib.load() function expects a binary file opened in 'rb' mode, but pyproject_toml appears to be opened in text mode based on typical usage patterns.

Suggested change

metadata_dict = tomllib.load(pyproject_toml)

with pyproject_toml.open("rb") as f:

metadata_dict = tomllib.load(f)

Copilot · 2025-08-06T19:14:29Z

kedro/templates/project/hooks/utils.py


    # Remove the specified sections
    for section in sections_to_remove:
        _remove_nested_section(data, section)

    with open(file_path, "w") as file:
-        toml.dump(data, file)
+        tomllib.dump(data, file)


The tomllib module (standard library) does not have a dump() function - it's read-only. This will cause an AttributeError. You need to use a different library like tomli_w for writing TOML files.

pyproject.toml

Signed-off-by: Deepyaman Datta <[email protected]>

Co-authored-by: Copilot <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]>

Signed-off-by: Deepyaman Datta <[email protected]>

merelcht

The changes make sense, just had a question on why two dependencies are needed.

merelcht · 2025-08-13T13:42:35Z

pyproject.toml

+    "tomli>=1.1.0;python_version<'3.11'",  # tomllib is part of the standard library in Python 3.11+
+    "tomli-w",


Why do you need the separate tomli-w dependency?

tomli (and the tomllib included in the Python standard library from 3.11) are read-only, and tomli-w is the recommended writing library (same author).

Got it! Shame they didn't put it in the same library 🥲

merelcht

Thanks @deepyaman !

ankatiyar · 2025-08-14T15:19:17Z

features/environment.py

+                "install",
+                "-e",
+                ".",
+                "toml>=0.10.0",  # TODO(deepyaman): Migrate `kedro-telemetry` off `toml`


This still needed after kedro-org/kedro-plugins#1155 is merged?

It shouldn't be necessary once kedro-telemetry is released?

I do think kedro-telemetry should be released first regardless. I'm not sure if there's a specific consideration around using a newer version of Kedro (that uses tomli or tomllib instead of toml) with an older version of kedro-telemetry (that depended on the implicit toml dependency coming from Kedro), but I think this could just be documented/it would be pretty obvious to install toml (or upgrade kedro-telemetry).

Use tomllib, if available, else fall back to tomli

24f255b

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman requested a review from merelcht as a code owner August 6, 2025 19:13

deepyaman requested review from Copilot and removed request for merelcht August 6, 2025 19:13

Copilot AI reviewed Aug 6, 2025

View reviewed changes

deepyaman and others added 3 commits August 6, 2025 13:28

Update references to toml in unit/end-to-end tests

264d8bf

Signed-off-by: Deepyaman Datta <[email protected]>

Update pyproject.toml

6b5bdf3

Co-authored-by: Copilot <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]>

Update kedro/templates/project/hooks/utils.py

c6b9e48

Co-authored-by: Copilot <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman marked this pull request as draft August 6, 2025 19:31

deepyaman added 11 commits August 6, 2025 13:50

Use tomli-w for writing (tomllib is read-only)

ebaa837

Signed-off-by: Deepyaman Datta <[email protected]>

Fix syntax and add explicit toml test dependency

2e77130

Signed-off-by: Deepyaman Datta <[email protected]>

Remove unused imports, add noqas, and fix syntax

5548b07

Signed-off-by: Deepyaman Datta <[email protected]>

Ignore the correct linting and coverage violations

f8047c0

Signed-off-by: Deepyaman Datta <[email protected]>

Install toml in e2e test env, not with test reqs

7129a25

Signed-off-by: Deepyaman Datta <[email protected]>

Add toml back to requirements for e2e unit tests

2e30839

Signed-off-by: Deepyaman Datta <[email protected]>

Patch imported tomllib or aliased tomli object

eb55e84

Signed-off-by: Deepyaman Datta <[email protected]>

Fix linting and formatting errors by running hooks

26e8859

Signed-off-by: Deepyaman Datta <[email protected]>

Add # pragma: no cover to both sides of the if

6108371

Signed-off-by: Deepyaman Datta <[email protected]>

Install toml as part of minimal (and fresh) envs

3ef591a

Signed-off-by: Deepyaman Datta <[email protected]>

Remove the now-unnecessary types-toml dependency

6ef200d

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman marked this pull request as ready for review August 7, 2025 02:00

This was referenced Aug 7, 2025

chore(telemetry): use tomllib (or tomli) over toml kedro-org/kedro-plugins#1155

Merged

Use tomllib, if available, else fall back to tomli #5051

Open

merelcht reviewed Aug 13, 2025

View reviewed changes

merelcht approved these changes Aug 14, 2025

View reviewed changes

merelcht requested a review from ankatiyar August 14, 2025 15:16

ankatiyar reviewed Aug 14, 2025

View reviewed changes

ankatiyar approved these changes Aug 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use tomllib, if available, else fall back to tomli #5043

Use tomllib, if available, else fall back to tomli #5043

Uh oh!

deepyaman commented Aug 6, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Aug 6, 2025

Uh oh!

Uh oh!

Copilot AI Aug 6, 2025

Uh oh!

Copilot AI Aug 6, 2025

Uh oh!

Uh oh!

merelcht left a comment

Uh oh!

merelcht Aug 13, 2025

Uh oh!

deepyaman Aug 13, 2025

Uh oh!

merelcht Aug 14, 2025

Uh oh!

merelcht left a comment

Uh oh!

ankatiyar Aug 14, 2025

Uh oh!

deepyaman Aug 14, 2025

Uh oh!

Uh oh!

	metadata_dict = tomllib.load(pyproject_toml)
	with pyproject_toml.open("rb") as f:
	metadata_dict = tomllib.load(f)

		"tomli>=1.1.0;python_version<'3.11'", # tomllib is part of the standard library in Python 3.11+
		"tomli-w",

Use tomllib, if available, else fall back to tomli #5043

Are you sure you want to change the base?

Use tomllib, if available, else fall back to tomli #5043

Uh oh!

Conversation

deepyaman commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Development notes

Developer Certificate of Origin

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

merelcht left a comment

Choose a reason for hiding this comment

Uh oh!

merelcht Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

deepyaman Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

merelcht Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

merelcht left a comment

Choose a reason for hiding this comment

Uh oh!

ankatiyar Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

deepyaman Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deepyaman commented Aug 6, 2025 •

edited

Loading