Skip to content

Optimize object conversion#11475

Open
pixelrahulnotfound wants to merge 2 commits into
pydata:mainfrom
pixelrahulnotfound:optimize-object-conversion
Open

Optimize object conversion#11475
pixelrahulnotfound wants to merge 2 commits into
pydata:mainfrom
pixelrahulnotfound:optimize-object-conversion

Conversation

@pixelrahulnotfound

@pixelrahulnotfound pixelrahulnotfound commented Jul 25, 2026

Copy link
Copy Markdown

Description

optimise _possibly_convert_objects in xarray/core/variable.py by adding a scan for object arrays.
Here is the logic:
• We scan up the first 10,000(in 99% cases it will usually stop at 1 element and if an array is very large we do it to cap the limit so it doesnt lag in the scan) elements of the flattened object array.
• If a datetime-like type (such as datetime, np.datetime64, pd.Timestamp, pd.NaT, etc.) is found, we flag it as datetime-like and stop the scan.
• If a non-null, non-datetime element is found, we immediately determine it is not datetime-like and stop the scan.
• If no conclusive type is found in the first 10,000 elements (e.g., all elements are null) and the array is larger, we fall back to pandas' highly optimized C-compiled pd.api.types.infer_dtype on the entire array.

Checklist

  • [✓] Tests added
  • User visible changes (including notable bug fixes) are documented in whats-new.rst
  • New functions/methods are listed in api.rst

Tests

• python -m pytest -o addopts="" xarray/tests/test_possibly_convert_objects.py — 11
passed
• ruff check xarray/core/variable.py xarray/tests/test_possibly_convert_objects.py —
passed
• git diff --check — passed

AI Disclosure

[✓] This PR contains AI-generated content.
[✓] I have tested any AI-generated content in my PR.
[✓] I take responsibility for any AI-generated content in my PR.
Tools: Google Antigravity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant