WIP generate chat ID and capture that ID with subsequent tool calls #54
Conversation
nelsonwittwer
changed the title
| It takes two arguments: query and filter. The query argument is the string search term to filter schema elements by name. The filter argument is an array of strings to filter results to show specific sections.`, | ||
| ⚠️ REQUIRES CONVERSATION ID: You must call get_started first to get a conversationId. | ||
|
|
||
| Any GraphQL codeblocks generated after using this tool MUST be validated with the validate_admin_api_codeblocks tool. |
There was a problem hiding this comment.
This line seems out of place for the introspect_admin_schema tool, is there something I'm missing?
| It takes one argument: prompt, which is the search query for Shopify documentation.`, | ||
| ⚠️ REQUIRES CONVERSATION ID: You must call get_started first to get a conversationId. | ||
|
|
||
| Chunked docs have the advantage of being able to match tokens within smaller chunks of content, but they may miss all context from a particular page of documentation. Use this tool if you want as much context for the token as possible across all of Shopify docs. Do not use this tool if you are looking for all doc context for a particular API resource -- use the fetch_entire_doc_by_path tool instead. |
There was a problem hiding this comment.
Use this tool if you want as much context for the token as possible across all of Shopify docs. Do not use this tool if you are looking for all doc context for a particular API resource -- use the fetch_entire_doc_by_path tool instead.
I think this explanation is a bit cofusing to me and it might be to the LLM as well. Maybe we can rephrase it to Use this tool if you want to search across all the shopify docs and get a list of chunks that match the search. Do not use this tool if you want to retrieve a specific page from the docs.
| @@ -80,13 +109,23 @@ export async function shopifyTools(server: McpServer): Promise<void> { | |||
| "fetch_docs_by_path", | |||
| `Use this tool to retrieve a list of documents from shopify.dev. | |||
|
|
|||
| ⚠️ REQUIRES CONVERSATION ID: You must call get_started first to get a conversationId. | |||
|
|
|||
| This tool provides the full documentation for a particular path. No context will be missing for a particular path that you might miss when using doc chunks. | |||
There was a problem hiding this comment.
No context will be missing for a particular path that you might miss when using doc chunks.
A bit confusing to include this imho, it should suffice to say that this tool will return the full docs for a path.
Problem
We need a way of tracking how long it takes for an agent to generate valid code blocks with our validation tooling.
Solution
This introduces an optional
conversationIdparam for alldev-mcptool calls. If a user has opted out of instrumentation, this will never be passed back to Shopify.Why this is needed
What we care most about is ensuring code blocks generated by LLMs using our MCP server don't have hallucinated error in them. If it takes a slower response (an agent taking a few iterations to get there) that is a trade off we are happy to make. We expect users to accept code blocks that interact with Shopify APIs to just work ™️ . Ideally we get to this point in as few agent loops as possible. In a perfect world, the agent is able to generate code blocks passing our validation criteria on its first try.
To get to that first try, we will need to know how far off we are to that goal and where we are falling short with our tools. Tracking chats is how we will determine how many loops it took to get all code blocks to pass our validation tools.