SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 months agoDo you host your own AI?message-squaremessage-square160linkfedilinkarrow-up159arrow-down17file-text
arrow-up152arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 months agomessage-square160linkfedilinkfile-text
minus-squareSuspiciousCarrot78@aussie.zoneOPlinkfedilinkEnglisharrow-up3·4 months agoLlama.cpp or death!
minus-squaretristynalxander@mander.xyzdeleted by creatorlinkfedilinkEnglisharrow-up2·edit-23 months agodeleted by creator
minus-squareBlackLaZoR@lemmy.worldlinkfedilinkEnglisharrow-up2·4 months agoI use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up1·4 months agoOr exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama.
Llama.cpp or death!
deleted by creator
I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm
Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama.