Do hosted open models support tool calling reliably? How to test one in five minutes

Updated 13 November 2026

A coding agent is a loop of tool calls. The model asks to read a file, run a command or edit a line, the agent does it, and the result goes back. If the model gets a tool name wrong or sends arguments that aren't valid JSON, that turn fails, and an agent makes dozens of turns a task. So tool calling matters more for agents than almost any other model feature.

One loop turn. A coding agent runs this dozens of times per task, so a small failure rate in step 2 shows up often.

What we know and what we haven't measured

NeverMetered's API is OpenAI- and Anthropic-compatible, and coding tools that rely on tool calling work through it: Claude Code, Codex CLI, OpenCode and others have setup guides. Qwen's model card reports agentic coding scores that depend on the model calling tools correctly, with the Claude Code harness. The card doesn't document a tool-call format or a success rate, and we haven't published one of our own.

What failure looks like

  • The model answers in prose when it should have called a tool, so the agent has nothing to run.
  • It calls a tool that doesn't exist, or the wrong one for the job.
  • The arguments aren't valid JSON, or a required field is missing.
  • It repeats the same call after the result came back, which is the start of a loop. See why agent bills get so high.

A five-minute test

This sends the same tool-calling request 10 times and counts how many replies call the right tool with valid JSON arguments. It needs curl and jq, and it uses 10 requests, so check that your plan's allowance covers it. The request has one tool, read_file, and asks the model to open a file.

  1. 1

    Save the request

    Put this in a file called request.json.

    request.json
    {
      "model": "qwen3.8-27b",
      "max_tokens": 512,
      "messages": [
        {"role": "user", "content": "Open the file src/index.ts so I can see it."}
      ],
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "read_file",
            "description": "Read a file from the project and return its contents.",
            "parameters": {
              "type": "object",
              "properties": {"path": {"type": "string", "description": "Path relative to the project root"}},
              "required": ["path"]
            }
          }
        }
      ]
    }
  2. 2

    Run it 10 times

    Set your key in NEVERMETERED_API_KEY first. The test passes when the first tool call is read_file and its arguments parse as JSON with a path.

    for i in $(seq 1 10); do
      curl -s https://api.nevermetered.com/v1/chat/completions \
        -H "Authorization: Bearer $NEVERMETERED_API_KEY" \
        -H "Content-Type: application/json" \
        -d @request.json \
      | jq -e '.choices[0].message.tool_calls[0].function
          | .name == "read_file" and (.arguments | fromjson | has("path"))' >/dev/null \
        && echo pass || echo fail
    done | sort | uniq -c
  3. 3

    Read the result

    A line like 10 pass means every reply was usable. Any fail lines are worth looking into: rerun one request without the jq filter and read the raw reply to see whether the model answered in prose, picked another tool or sent broken arguments.

We ran exactly this script against the live endpoint on 13 November 2026 and got 10 passes out of 10. That's a small sample of the easiest case, so it shows the basics work and nothing more. Make the test harder before you trust it. Add three or four tools, give a task that needs two of them in sequence, and use the tool definitions your own agent sends. A real agent session is the best test of all, and the free plan is enough to run one.

If calls are failing

  • Check the output cap first. A reply that's cut off mid-arguments is invalid JSON. See the truncated file writes guide.
  • Keep tool descriptions short and specific, and keep the number of tools small. Models choose better from five clear tools than from fifty vague ones.
  • Look at the Requests page for the failing call's finish reason. See reading your usage and requests.
  • If your tool lets you set a temperature, use the vendor's recommended value for the model rather than a low one you chose for a different model.
Does Qwen3.8-27B support function calling?
Coding agents that depend on tool calling run on it through NeverMetered, and Qwen's card reports agentic coding scores. Neither Qwen nor we publish a tool-call success rate, which is why this page includes a test you can run yourself.
Is tool calling worse on open models than on Claude or GPT?
It varies by model and by setup, and we don't have a measurement that would let us say. Frontier models are generally more forgiving of vague tool definitions. Test with your own tools.
Will my agent stop if one tool call fails?
Most coding agents report the error back to the model and carry on, which costs a turn. Repeated failures are what to watch for, because they can turn into a loop.
Sources