Eval bug: Qwen3-Coder template parsing error when tools are enabled

[alexdelorenzo]

Name and Version

❯ ./bin/llama-cli --version
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon 780M Graphics, gfx1100 (0x1100), VMM: no, Wave Size: 32
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
version: 74 (94242a6)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

HIP

Hardware

AMD Ryzen 8840HS + 780M (as gfx1100 & gfx1103)

Models

Unsloth's Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf

Problem description & steps to reproduce

When tools are supplied to the Qwen3-Coder jinja template, it results in this error as reported below in the log output.

When no tools are supplied, template parsing works. When tools are supplied to the API, it results in a FilterExpression error.

First Bad Commit

Haven't done a bisect, but I suspect a recent jinja parser problem

Relevant log output

Logs

[41847] srv  log_server_r: response: {"error":{"code":500,"message":"\n------------\nWhile executing FilterExpression at line 57, column 53 in source:\n...{%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}↵        ...\n                                           ^\nError: selectattr: unknown test 'in'","type":"server_error"}}
srv    operator(): http: streamed chunk: {"error":{"code":500,"message":"\n------------\nWhile executing FilterExpression at line 57, column 53 in source:\n...{%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}↵        ...\n                                           ^\nError: selectattr: unknown test 'in'","type":"server_error"}}
srv    operator(): http: stream ended

Example error response

{
  "error": {
    "name": "AI_APICallError",
    "url": "http://127.0.0.1:8081/v1/chat/completions",
    "requestBodyValues": {
      "model": "Qwen3-Coder-30B-A3B-Instruct-Q4_K_M",
      "max_tokens": 32000,
      "top_p": 1,
      "messages": [
        {
          "role": "system",
          "content": "<snip>"
        },
        {
          "role": "user",
          "content": "example"
        }
      ],
      "tools": [
        {
            "type": "function",
            "function": {
              "name": "todoread",
              "description": "Use this tool to read your todo list",
              "parameters": {
                "$schema": "https://json-schema.org/draft/2020-12/schema",
                "type": "object",
                "properties": {},
                "additionalProperties": false
              }
            }
          }
      ],
      "tool_choice": "auto",
      "stream": true,
      "stream_options": {
        "include_usage": true
      }
    },
    "statusCode": 500,
    "responseHeaders": {
      "connection": "close, close",
      "content-type": "application/json; charset=utf-8",
      "server": "llama.cpp",
      "transfer-encoding": "chunked"
    },
    "responseBody": "{\"error\":{\"code\":500,\"message\":\"\\n------------\\nWhile executing FilterExpression at line 57, column 53 in source:\\n...{%- for json_key in param_fields.keys() | reject(\\\"in\\\", handled_keys) %}↵        ...\\n                                           ^\\nError: selectattr: unknown test 'in'\",\"type\":\"server_error\"}}",
    "isRetryable": true,
    "data": {
      "error": {
        "message": "\n------------\nWhile executing FilterExpression at line 57, column 53 in source:\n...{%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}↵        ...\n                                           ^\nError: selectattr: unknown test 'in'",
        "type": "server_error",
        "code": 500
      }
    }
  }
}

[Smalls1652]

Ran into the same issue with Unsloth's qwen3-4b-instruct and qwen3-30b-a3b (Both instruct and thinking) GGUF models. Haven't tried using a different template yet, but I'd suspect it may have to do with this commit? c15395f

The build, that I reverted back to the other day, was just before that commit was merged in.

[CISC]

The new jinja implementation is apparently missing the in test:

llama.cpp/common/jinja/value.cpp

Lines 318 to 397 in 0c21677

	// tests
	{"test_is_boolean", test_type_fn<value_bool>},
	{"test_is_callable", test_type_fn<value_func>},
	{"test_is_odd", [](const func_args & args) -> value {
	args.ensure_vals<value_int>();
	int64_t val = args.get_pos(0)->as_int();
	return mk_val<value_bool>(val % 2 != 0);
	}},
	{"test_is_even", [](const func_args & args) -> value {
	args.ensure_vals<value_int>();
	int64_t val = args.get_pos(0)->as_int();
	return mk_val<value_bool>(val % 2 == 0);
	}},
	{"test_is_false", [](const func_args & args) -> value {
	args.ensure_count(1);
	bool val = is_val<value_bool>(args.get_pos(0)) && !args.get_pos(0)->as_bool();
	return mk_val<value_bool>(val);
	}},
	{"test_is_true", [](const func_args & args) -> value {
	args.ensure_count(1);
	bool val = is_val<value_bool>(args.get_pos(0)) && args.get_pos(0)->as_bool();
	return mk_val<value_bool>(val);
	}},
	{"test_is_divisibleby", [](const func_args & args) -> value {
	args.ensure_vals<value_int, value_int>();
	bool res = args.get_pos(0)->val_int % args.get_pos(1)->val_int == 0;
	return mk_val<value_bool>(res);
	}},
	{"test_is_string", test_type_fn<value_string>},
	{"test_is_integer", test_type_fn<value_int>},
	{"test_is_float", test_type_fn<value_float>},
	{"test_is_number", test_type_fn<value_int, value_float>},
	{"test_is_iterable", test_type_fn<value_array, value_string>},
	{"test_is_sequence", test_type_fn<value_array, value_string>},
	{"test_is_mapping", test_type_fn<value_object>},
	{"test_is_lower", [](const func_args & args) -> value {
	args.ensure_vals<value_string>();
	return mk_val<value_bool>(args.get_pos(0)->val_str.is_lowercase());
	}},
	{"test_is_upper", [](const func_args & args) -> value {
	args.ensure_vals<value_string>();
	return mk_val<value_bool>(args.get_pos(0)->val_str.is_uppercase());
	}},
	{"test_is_none", test_type_fn<value_none>},
	{"test_is_defined", [](const func_args & args) -> value {
	args.ensure_count(1);
	bool res = !args.get_pos(0)->is_undefined();
	JJ_DEBUG("test_is_defined: result=%d", res ? 1 : 0);
	return mk_val<value_bool>(res);
	}},
	{"test_is_undefined", test_type_fn<value_undefined>},
	{"test_is_eq", test_compare_fn<value_compare_op::eq>},
	{"test_is_equalto", test_compare_fn<value_compare_op::eq>},
	{"test_is_ge", test_compare_fn<value_compare_op::ge>},
	{"test_is_gt", test_compare_fn<value_compare_op::gt>},
	{"test_is_greaterthan", test_compare_fn<value_compare_op::gt>},
	{"test_is_lt", test_compare_fn<value_compare_op::lt>},
	{"test_is_lessthan", test_compare_fn<value_compare_op::lt>},
	{"test_is_ne", test_compare_fn<value_compare_op::ne>},
	{"test_is_test", [](const func_args & args) -> value {
	args.ensure_vals<value_string>();
	auto & builtins = global_builtins();
	std::string test_name = args.get_pos(0)->val_str.str();
	auto it = builtins.find("test_is_" + test_name);
	bool res = it != builtins.end();
	return mk_val<value_bool>(res);
	}},
	{"test_is_sameas", [](const func_args & args) -> value {
	// Check if an object points to the same memory address as another object
	(void)args;
	throw not_implemented_exception("sameas test not implemented");
	}},
	{"test_is_escaped", [](const func_args & args) -> value {
	(void)args;
	throw not_implemented_exception("escaped test not implemented");
	}},
	{"test_is_filter", [](const func_args & args) -> value {
	(void)args;
	throw not_implemented_exception("filter test not implemented");
	}},

[sidmohan0]

hey all. I traced through this: the selectattr/reject filter path in value.cpp looks up test_is_in in global_builtins() and it's not there. The in operator already works in runtime.cpp for arrays, strings, and objects, so the test should just needs to mirror that logic.

I can work on a PR