Frequently asked questions

Only questions we can answer with a measurement. If we could not put a number with a machine and a date next to it, the question is not on this page — not even with a well-phrased opinion.

About your hardware

• Can I run a 35-billion-parameter model on an 11 GB card? Yes. We keep one alive on a GTX 1080 Ti from 2017, at 23.7 tokens per second, measured over seven hours on more than 500 real answers. It works because it is an MoE: out of 35 billion parameters, only about 3 work on each token, and the rest can live in ordinary system memory. See what comes out for your card.
• Then why does the VRAM calculator say it is not possible? Because it only computes „parameters × bits = memory needed". For a 35B model at Q4 that is ~21 GB, your card has 11, so: no. Those calculators do not know about experts spilling into system memory, which is exactly why they are wrong on old hardware.
• Does it work with no graphics card at all? It does, but slowly: 2.5 tokens per second for the same model, measured on a 12-core server. A normal answer takes three minutes. That is a demo, not a tool.
• Why does it get slower after a few hours? On CPU it degrades with uptime. Measured on the same day, same context: from 3.99 down to 2.05 tokens per second over 31.8 hours. After a restart: 4.77. On a GPU it does not happen — there we measured 23.86 at 20:00 and 23.73 at 02:00, which is flat. The fix on CPU is a scheduled nightly restart.
• Should I let the server handle two people at once? No. With two slots each destroys the other's cache and speed jumps around: measured on a live site, 30, 10, 15, 8, 10, 6, 7, 9, 13 seconds. With a single slot it settles at 2-3 seconds and stays there.
• Does quantising the KV cache help? On the small case yes, on the real one no. Measured: +15% on an empty context and −40% on a 12,288-token context. Decompression costs more than the memory it saves. If you test with „hi, how are you", you are testing something other than what you use.
• How many threads should I use? Depends on what hurts. Prefill doubles with threads (from 15 to 36 tokens per second between 4 and 12 threads), but generation is flat — under 10% spread across the same range. Long prompt and short answers: many threads. Long answers: nothing to win there.
• Why is the first question in a conversation so slow? Because the server recomputes the prompt from scratch once the slot goes cold. It can be saved to disk: measured, 17,111 tokens read back in 0.45 seconds, instead of ~15 minutes of recomputation.

About the marketplace

• What does it mean that you „measure" the goods? That we run a standard benchmark ourselves, on fixed inputs, the same for the whole category, and publish the result next to the price with the method written down. The inputs are public and downloadable, so you can redo the benchmark yourself. Here is the whole procedure.
• And if the goods do badly in the benchmark? It says so. A shelf where every benchmark is green is not an honest shelf, and a benchmark that cannot say „no" is not a benchmark — which is why every input set contains a case we know in advance should fail.
• Are you actually neutral? We have no platform of our own to make you build for, and we sell no product that competes with yours. That is what lets us benchmark other people's goods — a referee who is also a player is not a referee. Our own tools go through the same benchmark and their sheets look the same.
• Does it have to be built for your platform? No. It does not matter what it runs on. How goods reach the shelf.
• When does the shelf open to outside sellers? When there is someone to sell to. Buyers first, listings second — a marketplace that opens with four empty rooms opens with four empty rooms.

About the guide

• What exactly do I get? A PDF, 8 pages, in English. Plus future versions, free, without asking.
• Is it for me if I have a modern card? Probably not. It is written for old hardware and for day two. On a 24 GB card everything runs anyway, and the internet is already full of instructions for those.
• When can I buy it? Its page says „coming" because the payment path does not exist yet, and a button that leads nowhere is worse than no button.