feat(genai): support multimodal file inputs and display_name in function responses

[Saurav Gupta (Saurav-Gupta-9741)]

Description

This PR resolves the issue where multimodal files were disconnected from the FunctionResponse and stripped of their metadata. It updates _convert_tool_message_to_parts to accurately parse file, media, and image_url blocks from a ToolMessage and maps them into FunctionResponsePart objects.

By natively integrating FunctionResponseFileData and FunctionResponseBlob from the google-genai SDK, this strictly maintains data associations and preserves the display_name, ensuring the Gemini API can successfully distinguish between multiple files generated by a single tool call.

Testing

• Updated test_convert_tool_message_to_parts_list_content_with_media to assert the correct bundled Part structure.
• Added test_convert_tool_message_to_parts_with_display_name to explicitly verify display_name metadata preservation.
• Verified all 230 PyTest unit tests pass successfully.

---

Sir, if anything misses out, please let me know and I will fix it according to your expectation. Thank you so much for looking into it!

[Saurav Gupta (Saurav-Gupta-9741)]

Respected Sir Bagatur (@baskaryan) / Sir Harrison Chase (@hwchase17),

I have fully resolved the issue regarding the missing display_name metadata in multimodal function responses.

The architecture has been cleanly upgraded to natively parse interleaved media, file, and image_url blocks directly into the FunctionResponsePart and FunctionResponseFileData schemas as required by the new google-genai SDK standards.

I have also included a comprehensive unit test to verify this behavior, and all strict typing (mypy) and formatting (ruff) checks have successfully passed without any workarounds.

Thank you very much, Sir, for this wonderful opportunity to contribute to the LangChain ecosystem. It is an honor to help improve this library. Please let me know if there are any further adjustments you would like me to make, and I will be happy to implement them immediately!

looking forward for your response

[Saurav Gupta (Saurav-Gupta-9741)]

"Hi maintainers, it looks like the langchain-google-genai-us integration test hit the 60-minute timeout limit on Google Cloud Build. Could someone with write access please re-run the failed jobs when you get a chance? All other 12 checks and unit tests have passed successfully. Thank you!"

[Saurav Gupta (Saurav-Gupta-9741)]

Respected Mason Daugherty (@mdrxy), Bagatur (@baskaryan), and Eugene Yurtsev (@eyurtsev),

I hope you are having a great week! I wanted to gently bump this PR regarding support for multimodal file inputs and display_name within function responses for the Google GenAI integration.

The code is fully complete and all 12 core unit checks passed successfully. However, it looks like the langchain-google-genai-us (llm-integration-tests) job hit the 60-minute Google Cloud Build timeout limit (likely an infrastructure flake). Could someone with write access please re-run the failed jobs when you have a chance?

I know you all manage an incredible volume of work, so I have the utmost respect for your time. Whenever you have the bandwidth to review, I am highly available to make any structural adjustments you recommend to ensure this aligns perfectly with your vision for the package.

Looking forward to your guidance, and thank you for all your hard work!

[Saurav Gupta (Saurav-Gupta-9741)]

Hi Bagatur (@baskaryan) Sir, Eugene Yurtsev (@eyurtsev), and Mason Daugherty (@mdrxy) Sir! I hope you all are having a great week.

I know I have gently bumped this previously, and I completely understand that the core team is incredibly busy managing a massive backlog of PRs! If anyone could spare just a brief moment to leave a quick comment or let me know if this is on your radar, I would be incredibly grateful.

To quickly recap: The architecture has been cleanly upgraded to natively support FunctionResponseFileData and FunctionResponseBlob, which fully resolves the missing display_name metadata issue for multimodal files.

I also noticed that the langchain-google-genai-us integration test keeps timing out after exactly 60 minutes on Cloud Build. It seems to be a hanging test in the infrastructure rather than an issue with the PR code itself.

Thank you so much for all your hard work on LangChain, and I would love any feedback you have whenever you get a chance!

[Mason Daugherty (mdrxy)]

Thanks for this — bundling the media into FunctionResponse.parts is the right direction, and preserving display_name is a nice touch. A few things to address before this can merge:

1. The new media filter is too broad and misclassifies ordinary structured tool output — chat_models.py:678

or block.get("type") in ("media", "file", "image_url")

This classifies any block with a type of media/file/image_url as media, but plenty of valid JSON tool results share those keys without matching a supported media schema. A tool result like {"type": "file", "path": "..."}, or a domain object with an image_url/type field, is legitimate structured output — yet this branch moves it into media_blocks, and _convert_to_parts then fails because it doesn't match a convertible media block. Concretely, {"type": "file", "file_uri": "gs://..."} now falls through to raise ValueError("Unrecognized message part type: file.") at :646, whereas on main it passed through untouched into the response payload. (For the cases that do convert — file_id/url/base64/OpenAI-nested — is_data_content_block/is_openai_data_block already match them, so the type literal is largely redundant anyway.)

Can we narrow this condition so only blocks _convert_to_parts actually knows how to convert are extracted as media? Everything else should stay in the function response payload unchanged. A regression test with a structured tool result containing {"type": "file", "path": "..."} (asserting it lands in response["output"] and doesn't raise) would help lock this down.

2. Part-level metadata is silently dropped — chat_models.py:688-705

FunctionResponsePart only has inline_data/file_data, so reconstructing the blob/file-data from just data/mime_type/file_uri drops any video_metadata, media_resolution, or thought_signature that _convert_to_parts set on the original Part. Because the block was moved out of other_blocks, it isn't in response["output"] either — so it vanishes with no signal. If these fields genuinely can't live inside a FunctionResponse, could we warnings.warn when a routed block carries them (mirroring the existing media_resolution warnings at :426/:516) rather than swallowing them?

3. Test gap: inline + display_name — test_chat_models.py

The two tests cover disjoint halves — inline_data without display_name, and file_data with display_name. That leaves blob.display_name = display_name (:694) unasserted; deleting that line keeps the suite green. Mind adding a base64/inline media block with a display_name and asserting parts[0].inline_data.display_name? A second case with two media blocks (distinct display_names) would also lock in ordering, since the media_blocks[i] mapping (:685) assumes a 1:1 correspondence with _convert_to_parts output.

Nice work overall — the behavioral assertion change in test_..._list_content_with_media (len(result) == 1) is exactly right.

[Mason Daugherty (mdrxy)]

Also, is there an issue this PR should close?

[Andrew Chen (chuenchen309)]

Not a maintainer, just a couple of observations from reading through — Mason Daugherty (@mdrxy)'s review already covers the substantive issues, so this is a small supplement to point 1 (happy to be wrong).

When narrowing that overbroad filter, it's worth keeping the "media" literal specifically. I checked the two predicates against the LangChain.js media block shape and neither matches it:

from langchain_core.messages import is_data_content_block
from langchain_core.language_models import is_openai_data_block

b = {"type": "media", "mime_type": "application/pdf", "file_uri": "gs://b/f.pdf"}
is_data_content_block(b)   # False
is_openai_data_block(b)    # False

That's the exact block shape the new test_..._with_display_name feeds in, and it's handled by the dedicated elif part["type"] == "media" branch in _convert_to_parts. So if the filter is narrowed to "only what is_data_content_block / is_openai_data_block already match," the media path (this PR's main use case) would quietly fall back into response["output"] and lose display_name.

The redundancy actually splits cleanly:

• image_url → already matched by is_openai_data_block (safe to drop)
• file + file_id → already matched by is_data_content_block; only file + file_uri is the harmful case that raises
• media → matched by neither, so it's load-bearing

So ... or block.get("type") == "media" alone might be the minimal safe form, and a regression test with a media + file_uri + display_name block would lock in that the media path survives the narrowing. Thanks for the work on this!