Batch Inference

Batch inference takes many chat-completion requests in one call and answers them later. You submit the whole job, check on it from time to time, and page through the results once it has finished. FreeInference runs the requests in the background, a few at a time per model, and gives way to interactive traffic.

The API has the shape of OpenAI’s Batch API: the same batch objects, statuses and result lines. The difference is that the requests travel inline in the create call, and the results are read a page at a time instead of downloaded as a file.

Internal accounts only: Batch inference is currently available to internal accounts. Creating a batch with any other key returns 403. It may be opened to other account tiers later.

When to use it

Use a batch for a large job that does not need its answers right away: an evaluation run, labelling or summarizing a dataset, generating synthetic data. Compared with sending the requests yourself:

• One call submits the whole job, up to 50,000 requests by default. You do not hold thousands of connections open, and you can disconnect once it is submitted.

• FreeInference paces and retries the work. Requests wait for a free slot instead of failing on your concurrency limit, and transient errors are retried for you, so you write no rate-limit or retry logic.

• The results wait for you. A finished batch and its results are kept for 7 days by default.

Send the requests yourself when you need answers within seconds, need streaming, or call an endpoint other than chat completions. A batch has no fixed turnaround: its requests start when their models have room, and any that have not started when its completion window closes (24 hours by default) expire. Batch requests are not discounted either: each one counts against your daily quota exactly as the same request sent directly would.

How batches run

• Nothing runs during the create call. FreeInference checks every request, stores the batch and answers at once, with the status validating. The requests start in the background as soon as their models have room.

• A few at a time, behind interactive traffic. Each model runs only a few batch requests at once, and fewer while it is busy with interactive requests. Batches take turns, so a large batch does not hold up a smaller one submitted after it. Depending on how busy its models are, a batch can finish in minutes or take hours. Requests run in no particular order; their results are listed in the order you submitted them.

• Like your own requests. Each request is served exactly as if you had sent it to /v1/chat/completions yourself, under your account, and counts toward your daily quota. While it runs it holds one of your concurrent-request slots, so a running batch leaves fewer for your interactive work. When every slot is in use, the batch request waits.

• Retried when the error may pass. A request that fails with a timeout, a 429 or a 5xx is retried with backoff, up to eight attempts in all. A request rejected as invalid (any other 4xx) fails at once.

• Checked against your key every time. Before each request runs, FreeInference checks the API key that created the batch again:

  • if the key has been revoked or has expired, or the account was suspended or deleted, the batch fails;

  • if your daily quota is used up, your batches pause until it renews at 00:00 UTC;

  • if your account no longer has access to batch inference, your batches wait until it has again.

Keep the key you created a batch with until the batch finishes. The completion window keeps running while a batch is paused or waiting. And while your daily quota is used up, every call made with your key returns 429 until the quota renews, checking on and reading your batches included.

Endpoints

Every endpoint is under the base URL https://freeinference.org/v1 and takes your FreeInference API key, as Authorization: Bearer hyi-your-api-key (or X-API-Key; see the API Headers Reference).

Request	What it does
POST /v1/batches	Create a batch
GET /v1/batches/{batch_id}	Check on a batch
GET /v1/batches/{batch_id}/results	Read its results, a page at a time
POST /v1/batches/{batch_id}/cancel	Cancel it
DELETE /v1/batches/{batch_id}	Delete it, with its results
GET /v1/batches	List your batches

The examples read your key from an environment variable:

export FREEINFERENCE_API_KEY="hyi-your-api-key"

Create a batch

POST /v1/batches with the requests inline:

curl https://freeinference.org/v1/batches \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {
        "custom_id": "q-1",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "glm-5.3-flash",
          "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]
        }
      },
      {
        "custom_id": "q-2",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "qwen3.6-35b",
          "messages": [{"role": "user", "content": "Translate \"good morning\" into French."}],
          "max_tokens": 50
        }
      }
    ],
    "metadata": {"project": "eval-42"}
  }'

Field	Required	Meaning
requests	Yes	The requests, at least one
requests[].custom_id	Yes	Your name for the request: 1–512 characters, unique within the batch. Its result carries it back
requests[].method	No	POST, the default and the only method accepted
requests[].url	No	/v1/chat/completions, the default and the only endpoint a batch can call
requests[].body	Yes	A chat-completion request body, exactly as you would send it to /v1/chat/completions. stream must not be true
metadata	No	Up to 16 string key-value pairs for your own use; keys up to 64 characters, values up to 512

One batch can use several models. Name each one by a model ID from Available Models: auto models (auto, auto-<tier>) cannot be used in a batch.

FreeInference checks every request before it stores anything, and refuses the whole batch with 400 if any request is invalid. The error message names the request at fault, for example requests[1] (custom_id 'q-2'): Model 'no-such-model' not found. As elsewhere in the API, a model your key may not use is reported exactly like one that does not exist.

The answer is the new batch:

{
  "id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": null,
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1792929600,
  "in_progress_at": null,
  "expires_at": 1793016000,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {"total": 2, "completed": 0, "failed": 0},
  "metadata": {"project": "eval-42"},
  "cost_usd": 0.0,
  "error": null
}

The other endpoints take the batch’s id; the examples below read it from $BATCH_ID. Timestamps are Unix seconds, and expires_at is when the completion window closes. The file fields and errors are always null. Two fields are not in OpenAI’s batch object: cost_usd, what the batch’s finished requests have counted against your daily quota so far (an accounting figure, not a bill), and error, which says why the whole batch failed.

Check on a batch

GET /v1/batches/{batch_id} returns the batch object again:

curl https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Poll it until status is completed, failed, expired or cancelled, the four final statuses:

Status	Meaning
validating	Stored; no request has started yet
in_progress	Requests are running
completed	Every request has finished, successfully or not. Read each one’s outcome in the results
failed	The batch stopped because the key or account that created it is no longer valid; its error says so (owner_unauthorized). Requests that had not started fail with batch_failed
expired	The completion window closed before every request had started. Those requests expire with batch_expired; the others keep their results
cancelling	You cancelled the batch while requests were running; they are finishing
cancelled	Cancelled. Requests that had not started never will

request_counts.total is the number of requests in the batch. request_counts.completed counts the requests that succeeded, and request_counts.failed those that failed, including the ones a failed or expired batch never ran. Cancelled requests are in neither.

Read the results

GET /v1/batches/{batch_id}/results pages through the batch’s requests in the order you submitted them, each with its outcome:

curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

{
  "object": "list",
  "data": [
    {
      "id": "batch_req_1041",
      "custom_id": "q-1",
      "status": "completed",
      "response": {
        "status_code": 200,
        "request_id": "req_2c9f6a1d8e7b4c03a5f1e2d3c4b5a697",
        "body": {
          "id": "chatcmpl-9b1e",
          "object": "chat.completion",
          "model": "glm-5.3-flash",
          "choices": [
            {
              "index": 0,
              "message": {"role": "assistant", "content": "Prince Hamlet ..."},
              "finish_reason": "stop"
            }
          ],
          "usage": {"prompt_tokens": 16, "completion_tokens": 120, "total_tokens": 136}
        }
      },
      "error": null
    },
    {
      "id": "batch_req_1042",
      "custom_id": "q-2",
      "status": "failed",
      "response": null,
      "error": {"code": "invalid_request", "message": "...", "status": 400}
    }
  ],
  "has_more": false,
  "first_id": "batch_req_1041",
  "last_id": "batch_req_1042"
}

limit is 1–1000 (default 100). When has_more is true, ask for the next page with after set to this page’s last_id:

curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100&after=batch_req_1042" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Every request of the batch is listed, finished or not. A request that is still queued or running has no response yet; if it has already failed once and is being retried, its error shows what went wrong last time. So read the results once the batch has reached a final status, or read them again later.

Result status	Meaning
queued	Waiting to run, or waiting to be retried
running	Running now
completed	Succeeded. response.body is the chat completion, as /v1/chat/completions would have returned it
failed	Failed, or never ran because the batch failed. error says why
cancelled	Never ran: the batch was cancelled
expired	Never ran: the completion window closed first

An error has a code and a message, plus the HTTP status behind it when there was one:

error.code	Meaning
invalid_request	The request was rejected as invalid (400, 422)
model_not_found	The model is gone, or your key may no longer use it (404)
forbidden	The request was refused (403)
rate_limited	Still rate-limited after every retry (429)
timeout	Still timing out after every retry (504)
service_unavailable	No provider could serve it after every retry (503)
upstream_error	Another server error, after every retry
batch_failed	Not run: the batch failed
batch_expired	Not run: the completion window closed first

Python example

The OpenAI SDK’s batch helpers expect an uploaded input file, so call the endpoints over plain HTTP. This script submits a batch, waits for it to finish and collects every result, using requests:

import os
import time

import requests

API = "https://freeinference.org/v1"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['FREEINFERENCE_API_KEY']}"


def call(method, path, **kwargs):
    response = session.request(method, API + path, timeout=120, **kwargs)
    response.raise_for_status()
    return response.json()


prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = call(
    "POST",
    "/batches",
    json={
        "requests": [
            {
                "custom_id": f"q-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "glm-5.3-flash",
                    "messages": [{"role": "user", "content": prompt}],
                },
            }
            for i, prompt in enumerate(prompts)
        ]
    },
)

# Wait for a final status.
while batch["status"] not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = call("GET", f"/batches/{batch['id']}")

# Page through every result.
results, after = [], None
while True:
    params = {"limit": 1000}
    if after:
        params["after"] = after
    page = call("GET", f"/batches/{batch['id']}/results", params=params)
    results.extend(page["data"])
    if not page["has_more"]:
        break
    after = page["last_id"]

for result in results:
    if result["status"] == "completed":
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
    else:
        print(result["custom_id"], result["status"], result["error"])

Cancel a batch

POST /v1/batches/{batch_id}/cancel stops a batch and returns it:

curl -X POST https://freeinference.org/v1/batches/$BATCH_ID/cancel \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Requests that have not started are cancelled; running ones finish and keep their results. The batch is cancelling until they do, then cancelled. Cancelling a batch that has already finished returns it unchanged.

Delete a batch

DELETE /v1/batches/{batch_id} deletes a batch with its requests and results:

curl -X DELETE https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

{"id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0", "object": "batch.deleted", "deleted": true}

A batch can be deleted in any status, but not while one of its requests is running: the answer is then 409. Cancel the batch, wait for its running requests to finish, then delete it.

FreeInference also deletes a finished batch on its own, 7 days after it finished by default, so collect its results before then.

List your batches

GET /v1/batches lists your batches, newest first:

curl "https://freeinference.org/v1/batches?limit=20" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

limit is 1–100 (default 20). The answer has the shape of a page of results, with batch objects in data; for the next page, pass after set to the previous page’s last_id.

Reading, listing, cancelling and deleting your batches work even if your account can no longer create them, so you can always collect and remove what you submitted. A batch that belongs to another account reads as not found.

Limits

Limit	Default
Requests in one batch	50,000
Size of the create request’s body	32 MiB
Unfinished batches per account (validating, in_progress or cancelling)	10
Completion window: how long a batch’s requests have to start	24 hours
How long a finished batch and its results are kept	7 days

These are the defaults, and FreeInference may change them. Split a larger job into several batches.

Errors

Status	When
400	Creating: the body is not valid JSON or not a batch; a request is invalid, uses an unknown, unavailable or auto model, or repeats a custom_id; or the batch holds too many requests. Reading results: after is not a result ID such as batch_req_42
401	The API key is missing, invalid or expired
403	Creating: your account may not create batches (internal accounts only, for now)
404	The batch does not exist, was deleted, or belongs to another account; or batch inference is unavailable at the moment. The message says which
409	Deleting: one of the batch’s requests is still running. Cancel the batch and wait for it to finish first
413	Creating: the body is larger than the size limit
429	Creating: you already have the most unfinished batches allowed. Any call: your daily quota is used up until it renews at 00:00 UTC

Coming from OpenAI’s Batch API

• Put the requests in the create call’s requests array, each entry shaped like a line of an OpenAI batch input file. There is no file upload: no input_file_id, and no output_file_id or error_file_id to download. Read the results from /v1/batches/{batch_id}/results.

• The OpenAI SDK’s batches.create needs an uploaded file, so it cannot create a batch here; use plain HTTP, as in the Python example.

• The completion window is FreeInference’s (24 hours by default); a completion_window you send is ignored.

• Only /v1/chat/completions can be batched, and without streaming.

• The status finalizing never occurs, and DELETE /v1/batches/{batch_id} exists.

• There is no batch discount: batch requests count against your quota like any other.

See also

• Available Models — the model IDs a batch can use

• API Headers Reference — authentication headers

• IDE & Coding Agent Integrations — send chat requests directly with the OpenAI SDK or curl

• Quick Start — create an API key