Measurements you can inspect.

Fixed workload

Select one to three models. Each model completes a separate round and its server closes before the next starts; models never run concurrently. The runner tokenizes a fixed synthetic text with each model’s tokenizer, then uses exactly 512 or 2,048 input token IDs. Each workload generates 128 tokens with temperature 0, seed 42, EOS ignored and prompt cache disabled. One warm-up is discarded; three measured repetitions are retained. Context is 4,096, KV cache f16, flash attention off, batch and microbatch 512, one slot, GPU layer request 999, CPU threads equal to detected logical cores.

Concurrent jobs

TokFire Bench also provides a separate concurrent profile: one GGUF or MLX model, 1–3 free simultaneous jobs, or up to 20 with Pro. It measures three synchronized rounds and reports each job’s latency and speed separately from aggregate throughput. These results are not mixed with the sequential standard profile.

Windows preview

Windows 10/11 x64 uses llama.cpp and GGUF with 512 input tokens, 128 output tokens and three synchronized measured rounds after warm-up. Each job shares one model server, with 4,096 context tokens per slot. Reports use local-ai-windows-jobs-v1 and include CPU, computer model, GPU names, RAM and Windows version. They are stored separately from the Mac standard comparison table. GPU names do not prove that GPU offloading succeeded. The Windows executable is unsigned and has been cross-compiled and unit-tested on macOS; Windows GUI, login, cancellation and CPU/GPU inference tests remain pending.

Quick trial

The optional MiniCPM trial uses one model, 512 input and 32 output tokens, one warm-up and one measurement. Its local-ai-trial-v1 report stays local and cannot enter the standard comparison database.

What is measured

TTFT is client wall-clock time from starting the loopback HTTP request to receiving the first generated token ID. It includes local HTTP overhead. Prefill and decode rates come from the runtime’s final timing report. Load time is process launch to a healthy server, without clearing the operating system’s file cache. Peak process RSS is sampled every 250 ms across that model session; it is not total unified or Metal memory.

Comparison rules

Exact model bytes and runtime binary are SHA-256 hashed. A comparison group also includes every runtime setting, specification version, and input/output length. Hardware/OS combinations remain separate rows. Each report contributes a median of three runs; community rows average these medians. No global score, quality score, predicted speed or verified badge is generated.

Known limitations

These are community submissions, not attested measurements. Format checks cannot prove the hardware claim or prevent a modified client. Power, temperature, swap, actual layer-offload verification, total GPU memory and sustained thermal testing are not measured yet. Model-load timing is affected by OS caches. Ignoring EOS is a throughput workload, not a quality assessment. Do not infer answer quality or broad application compatibility from speed alone.

Privacy and publication

The benchmark runner makes loopback requests only and saves reports locally. The optional MiniCPM launcher downloads the official model from Hugging Face and verifies its checksum. Browser import does not upload. Saving requires explicit collection consent and sign-in; publication is a separate opt-in. The account ID is retained privately for ownership and deletion. Published comparisons omit identity but include hardware and hashes. Published comparisons are visible to site visitors. Private reports require the account that uploaded them.

Native testing status

The source includes a SwiftUI macOS interface and a real llama-server runner. Python protocol tests run in this development environment. Swift compilation, real Metal inference, GUI lifecycle tests and ad-hoc signing have passed on an M2 Max. The developer DMG is not Developer ID signed or Apple notarized.