Add example: build a self-correcting 3D collectible set with three.ws text-to-3D, function calling, and vision#2874
Conversation
…and vision New third_party notebook: a model art-directs a themed set of props from a one-line brief, issues parallel function calls to the free, keyless three.ws text-to-3D API to build each one, then closes the loop with vision QA: it inspects a rendered PNG of every model it built (via the keyless three.ws GLB renderer), grades each against the brief into a strict structured-output schema, and rebuilds any failed prop once from the critic's own revised prompt. Results render as an interactive model-viewer gallery inline. Teaches wrapping a REST API as a function tool, handling multiple parallel tool calls in a single Responses API turn, vision-in-the-loop QA of generative tool output, bounded self-repair, and rendering rich tool results. Generation is retry-hardened, QA fails open if the renderer is unavailable, the critic is calibrated to draft tier, and the repair loop has a hard budget of one retry per prop. Registers the notebook in registry.yaml and adds the author to authors.yaml.
cookbook-live-run.mp4cookbook-demo.mp4 |
💡 Codex Reviewhttps://github.com/openai/openai-cookbook/blob/c24d3b15444cb790254ad467211cbd0b19e2faf0/examples/third_party/text_to_3d_with_function_calling.ipynb#L204 On a Python 3.9 kernel, which the notebook explicitly lists as supported in the prerequisites, this cell fails when the function is defined because ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Good catch. Fixed in 83824e9: the prerequisite now reads Python 3.10+, matching the str | None annotations used in the tool wrappers. Chose raising the floor over rewriting the annotations to Optional[str] since the committed run and all baked outputs were produced on a 3.10+ kernel, and 3.10+ is what Colab and current Jupyter installs ship by default. @codex review |
💡 Codex Reviewhttps://github.com/openai/openai-cookbook/blob/83824e9f65b5869583ebdb7c3e950eedca92c216/examples/third_party/text_to_3d_with_function_calling.ipynb#L565-L566 Because this cell runs during a normal Run All, it immediately starts four live three.ws generation jobs, and later cells start OpenAI vision/generation calls, before the reader has explicitly opted into external service use. This repo asks external-service-dependent contributions to mock responses or guard those cells behind clearly labeled opt-in flags, so please add a https://github.com/openai/openai-cookbook/blob/83824e9f65b5869583ebdb7c3e950eedca92c216/examples/third_party/text_to_3d_with_function_calling.ipynb#L224 When three.ws returns a pending job with a ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
…ile polling Per AGENTS.md, cells that call the live three.ws and OpenAI endpoints are now behind a clearly labeled opt-in flag (default off), so a plain Run All makes no network calls and shows the committed results; the notebook remains fully executed. The poll loop now honors the API's retryAfter cadence hint and rides out a mid-poll 429 on the same job instead of resubmitting it. Also brings the prose up to house standard: prerequisites table, hero image of the finished set, troubleshooting and production-checklist tables, and a when-to-use-this pattern comparison.
|
Both addressed in cbf2b89. Live calls are now opt-in (AGENTS.md compliance). A RUN_LIVE flag in the Setup cell (default False) gates every cell that touches an external service. A plain Run All now makes zero network calls beyond pip, executes top to bottom with no errors and no API key, and each gated cell prints a note pointing at the flag while keeping its committed output visible. Verified by executing the notebook end to end in a clean environment with no OPENAI_API_KEY set: 0 errors, all 5 live cells skip cleanly. The committed outputs remain from the real, fully-executed live run. The poll loop now honors the API's cadence hints. Every pending response from the lane carries a retryAfter (seconds); the loop sleeps on that value before each poll instead of a fixed 3s. A mid-poll 429 now waits out Retry-After and keeps polling the same job rather than raising out of _submit_and_poll() and resubmitting the generation. Verified two ways: a live run through the new wrapper against the real endpoint, and a deterministic transport-stub test asserting one submit, cadence [5, 2, 5, 4] across pending → 429 → pending → done. The same commit also brings the prose up to the standard of recent cookbook pages: a prerequisites table, a hero image of the finished set, an error-handling table, a production checklist, and a when-to-use-this-pattern comparison. @codex review |
💡 Codex Reviewhttps://github.com/openai/openai-cookbook/blob/cbf2b89998a3bdac198e14f223d84ee580b17657/examples/third_party/text_to_3d_with_function_calling.ipynb#L164 With the default AGENTS.md reference: AGENTS.md:L20-L22 ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Running a gated cell clears its committed output, so with RUN_LIVE off a keyless Run All previously left only the skip note behind. Each gated cell now re-displays its committed static contact sheet from images/, so the default run still ends with every example result visible; a standalone copy of the notebook without the repo's images directory falls back to a pointer at the hosted run. Verified both paths: keyless Run All from the repo replays one sheet per gated cell with zero errors, and a standalone copy prints the pointer with zero errors.
Self-review sweep before further bot rounds. Robustness: transient failures on the poll leg (timeout, network blip, 5xx) now keep waiting on the same job instead of resubmitting it, the submit read timeout outlasts the lane's inline hold window, Retry-After parses both delta-seconds and HTTP-date forms with sane clamps, a pending response without a job token raises cleanly, polls carry the title param so viewer/AR pages are labeled, and the vision QA call now fails open like the renderer path so a transient API error degrades one verdict instead of crashing the run. Accuracy: the intro now matches the committed run (three props failed QA and were rebuilt; five of six pass), the concurrency and retry counts quoted in the tables match the code, and the replay fallback works for standalone copies by loading the canonical image URL. Hygiene: QA card images are compact thumbnails (the notebook drops from 2.7 MB to 1.7 MB with no visible loss), pip's ANSI upgrade notice is out of the committed output, stale per-cell execution timestamps are stripped, and the registry date is corrected.
|
Addressed in f1dad51 and 6e9d9bc. Baked outputs now survive a gated run (f1dad51). You're right that executing a gated cell cleared its committed output, leaving only the skip note. Each gated cell's skip branch now replays its committed static contact sheet (checked into images/), so a keyless Run All still ends with every example result visible. A standalone copy (Colab, a downloaded .ipynb) without the repo's images/ directory falls back to loading the canonical image URL in the browser, so the kernel-side no-network property holds there too. Plus a full self-review sweep (6e9d9bc) so we converge instead of iterating: transient failures on the poll leg (timeout, network blip, 5xx) now keep waiting on the same job instead of resubmitting it, and the submit read timeout now outlasts the lane's documented inline hold window, both of which previously duplicated generations; Retry-After parses both delta-seconds and HTTP-date forms with sane clamps; the vision QA call now fails open like the renderer path, so a transient API error degrades one verdict instead of crashing the run; polls carry the title param so finished viewer/AR pages are labeled; the intro numbers now match the committed run exactly (three props failed QA and were each rebuilt once; five of six pass); concurrency and retry counts quoted in the prose match the code; and the QA card images are compact thumbnails, taking the notebook from 2.7 MB to 1.7 MB with no visible loss. Verified: the repo validator; a keyless Run All from the repo (0 errors, all five gated cells replay their sheets) and standalone (0 errors, five URL fallbacks); a transport-stub test driving one submit through pending, HTTP-date 429, poll timeout, and 502 to done; and a live end-to-end generation against the real endpoints. @codex review |
💡 Codex Reviewhttps://github.com/openai/openai-cookbook/blob/6e9d9bc03efee0b538c8b07859eefcdabc6d736c/examples/third_party/text_to_3d_with_function_calling.ipynb#L291 When a queued job's poll request returns HTTP 500, or any other 5xx outside ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Good catch. Fixed in 83824e9: the prerequisite now reads Python 3.10+, matching the str | None annotations used in the tool wrappers. Chose raising the floor over rewriting the annotations to Optional[str] since the committed run and all baked outputs were produced on a 3.10+ kernel, and 3.10+ is what Colab and current Jupyter installs ship by default. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
LGTM |
|
This easily deserves a spot. I approve! |
|
Let's merge it @openai |
4 participants
Summary
Adds a third-party notebook,
examples/third_party/text_to_3d_with_function_calling.ipynb: Build a self-correcting 3D collectible set with function calling and vision.A one-line brief ("chunky potted houseplants for a sunny windowsill") goes to a gpt-5-mini art director, which designs a themed set of props and issues parallel function calls to the free, keyless three.ws text-to-3D API, one real
.glbper prop. The notebook then closes the loop: each model is rendered to a PNG (a second keyless endpoint), inspected by the model with vision, graded against the brief into a strict structured-output schema, and any failed prop is rebuilt exactly once from the critic's own revised prompt. Results render as an interactivemodel-viewergallery inline (orbit, zoom, AR on phones).Live example renders from the committed run:
Stills produced by the notebook's own render step (
https://three.ws/api/render/glb).Motivation
Most function-calling examples return text or JSON. This one returns real, downloadable 3D objects, then turns the model's vision on its own output. The generate / render / inspect / repair shape applies any time a tool's output is generative and can silently miss (image generation, chart rendering, UI screenshots, 3D), and no existing cookbook example demonstrates vision-QA of generative tool output.
Patterns taught, all on the GA Responses API surface:
function_callitems in a single Responses API turn.passedis a boolean you branch on;revised_promptis machine-usable).<model-viewer>needs a live script, every gallery and QA pass also bakes a plain PNG contact sheet into the notebook, so all results are visible on GitHub and the cookbook site without running anything.Both external endpoints are free and keyless (no account, no signup, no credit card), so the only prerequisite is
OPENAI_API_KEY. The notebook is committed fully executed, top to bottom, against the live APIs; all baked outputs are real. Design choices worth noting for review: QA fails open if the renderer is unavailable (a guardrail that can take down the pipeline it guards is worse than no guardrail), the critic is explicitly calibrated to the draft tier, the tool wrapper honors the lane'sRetry-Afterheader when rate-limited, and the model name is a single swappable variable on the GA API surface.For new content
Self-review notes against the rubric: the notebook was executed top-to-bottom before commit (
.github/scripts/check_notebooks.pypasses); filename follows the lowercase convention;OPENAI_API_KEYis documented in Prerequisites and never hardcoded; all tool loops are bounded; the architecture diagram lives inimages/threews_self_correcting_3d_pipeline.svg.