I only use Qwen for Zoo Code, when chatting I just use Gemma

It’s MoE so you can use --n-cpu-moe

https://lemmus.org/post/24235317

Low number of active parameters (6B) means you don’t need much VRAM to get decent speeds

Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP

I bet you could run this in Q4 on 96GB RAM and 16GB VRAM, maybe even less. The benchmark scores seem good, beating 27b and DeepSeek Flash.

The n-gram embeddings sound very similar to Gemma 4 e4b embeddings. Need llama.cpp to support streaming n-grams from SSD, mmap would be less efficient than having explicit support.

GGUFs are starting to be available now

Yeah Ornith 1 35b seemed pretty good when I tried it, haven’t tried 1.5 yet. The only one from the 1.5 series I tested so far was 9b and it seems to punch above its weight

Have you tried Kat Coder? It’s also a fine-tune of Qwen 35b and I’ve heard good things about it

Yeah that one was over a torrent site, I think nyaa.si

https://lemmy.ca/post/69262372

https://www.techspot.com/news/113446-years-long-investigation-unmasked-major-torrent-pirate-how.html

8GB VRAM RTX 3070 + 32 GB DDR5 + 1TB NVMe SSD - AMD Ryzen 5 7600X CPU - putting this here in case it’s substantially better than the BC-250

Yes this can run Qwen 3.6 35b-a3b pretty nicely! And they might be releasing an updated version of that soon. Your BC-250 only has 16GB total which is not enough for 35b.

I also have 32GB RAM and 8GB VRAM, my computer is a little slower than yours, see my guide: https://lemmus.org/post/24235317

For the BC-250 you might try smaller models like Ling 3.0 Tiny, Ornith 1.5 9b, or Gemma 4 12b QAT

For your 8GB RAM devices, you can run Gemma 4 e4b QAT, Qwen 3.5 4b, or maybe Ling 3.0 Tiny

I’m a noob to local LLMs.

Use Unsloth Desktop or llama.cpp. Then you can connect Zoo Code to it, that’s a VSCode extension which I like for programming with my local LLMs.

They don’t tell us things lol. All they said was

Which sounds like there’s something “better” coming, but doesn’t deny the possibility of 35b-a3b. Which is weird because “better” is subjective and depends on your hardware. It could be smaller and smarter than 3.6 35b, but then people are gonna ask for a 3.8 35b because it should be even smarter.

They have a graph, differences are tiny at that high end

I feel like Ornith 1.0 9b was the best coding model at that size (since Qwen has neglected that size). Maybe now Ling Tiny is better but I’m curious to try Ornith 1.5

I would suggest using llama.cpp instead of Ollama, or maybe Unsloth Studio

https://sleepingrobots.com/dreams/stop-using-ollama/

!localllama@sh.itjust.works

Sounds like you could just replace the template to fix it, there’s a popular Qwen fixed template on hugging face, try that

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/blob/main/chat_template.jinja

https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF

just hoping for a new 35b a3b

my laptop is crappy, so like 5 tokens per second lol, prompt processing of like 20 tokens per second

I think a decent laptop nowadays, even running CPU only, could probably do like 5x faster

I’ve run Qwen 3.5 4b and Gemma 4 e2b on CPU only, this should be faster than those I think (fewer active parameters). If you have AVX512 or AVX10 then it should help a bit. Still slow compared to a GPU lol.

anyone try this? this might be good for my crappy laptop lol

is it good enough to use with Zoo Code? is it better than Qwen 3.5 4b?

EDIT: woa

https://artificialanalysis.ai/models/ling-3-0-tiny

But not yet supported in llama.cpp https://github.com/ggml-org/llama.cpp/pull/26608

Actually funny he’s not asking it to work harder (that would be system prompt or user message), he’s forcing it to think that it will work harder

That’s a really cool idea. It’s like inception for an LLM, you make it think it was the one that thought of this lol

Have you tried preserve thinking? https://lemmus.org/post/24365786

(Oops I got my Gemma and Qwen speeds mixed up, edited the post to fix it.)

But now with the new commits they added, with the same number of hot experts, Qwen is up to about 34. If I increase hot experts to 48 then I get around 37.

Gemma is still around 23 with just 10 hot experts. With 16 hot experts I get about 26 TPS. If I overprovision my VRAM (thanks to GGML_CUDA_ENABLE_UNIFIED_MEMORY=1) then 24 hot experts can give me 29 TPS, and 32 hot experts 34 TPS.