SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 months agoDo you host your own AI?message-squaremessage-square160linkfedilinkarrow-up159arrow-down17file-text
arrow-up152arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 months agomessage-square160linkfedilinkfile-text
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up0·4 months agoIf you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it. It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA. Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.