Seems more like ollama as approach. More easy but less optimized than llamacpp?

Yes, llama swap also give you a pretty web based statistics of all the calls and runs for every model with t/s and more statistics.

It’s pretty neat… You can also load and unload models manually, define groups for models that fit together in vram and so on.

It gives you that automation that people coming from ollama are used to.

under the hood all it does is running llama serve. You convert your models.ini to a yaml file 1:1 (plus a few more flexibility).

Yes but llama server does not unload models to fit a different model. At least, didn’t managed to have just llama.cpp swap between models that fill up my vram… The second one would fail load.

Llama.cpp of course! After I did the switch, never looked back. Add llama-swap in front of you switch between models often.

Yeah… It should pull at very least chromium I guess…

Technically it is open source if … the source is open source. Then, if it uses proprietary components under the hood as binaries, well… In that case any windows tool that uses Msvc redistributable would not be able to be open source…

You missed my point?

why can’t you just use a wireguard connection between your device and the server, or network?

I don’t get the need for those tunnels.

Wouldn’t a simple wireguard setup be enough un less you are behind CG-NAT?

Wireguard it’s there, free, open source, doesn’t rely on any external server or company…

What does these “tunnels” actually add?

Opened a ticket on your forgejo… Cannot pull the image… I don’t / can’t build it at the moment , so looking to try the pre built image

It’s like cats. Projects choose their maintainers…

This will never be exposed to net… I understand the risks, but at least fits a need I have, and nothing else come close.

Don’t worry, it’s ok.

That’s cool… Next week I will try it for sure …

Yeah ai slop? Welcome if it’s the one and only viable solution …

Hoping some maintenance will be there tough :)

This won’t work for Android windows and such devices. And it’s a pain to setup, won’t work 100% and some scanners are ven worse .

Scanservejs is the final solution for all platforms.

Op want to upload a file to a web GUI and print it from there, not share printers with other computer/ devices.

Also, scanservejs is the the sane web GUI to actually do the same with scanner.

For scanning it make sense as there are no shared scanning solutions across platforms. For printing, it’s of little use as other pointed out you can print to cups from anything basically.

But o get the need for a web printing page, as uploading a file and hit print button is a zero setup, and while simple even remote cups printing is not 0% extra setup, you have at the very least to select a printer or hit “share as” and select a print service (android). YMMV

48 disks seems a nightmare in power consumption, failure rate, and overall cabling management…

What was the total capacity?

I use e self made wood cabinet in the attic where also the PV inverter is located. I used Velcro to strip on the wall (brick wall) stuff like switch firewall etc, only to find those collapsed on the server itself every time. No matter which glue or adhesive i used.

At the end, a strong unbranded Chinese double sided tape was better and still strong after all big western brand failed over time.

Maybe a mix of heat and uneven surface was the culprit, who knows.

Uptime for home users is useless. More important is recovery, reboot and resume (the 3’Rs)

You server should reboot automatically, storage should restore itself from unsafe shutdown and your services should resume operations like nothing happened.

This means test your reboot, ensure stuff doesn’t break, and so on.

Failing storage after an hard reboot is the hardest.

For network, i have two ISP connections and a an auto switch script Incase one goes down. But honestly it’s overkill and not needed for 99% of home hosters.

Buying some kind of UPS like a battery powered power strip is also nice but keep in mind that require maintenance to replace batteries once every few years (long enough to forget) and so be useless. The ups shall only last enough to perform a controlled server shutdown using NUTS.

No cups does not print from web UI, it’s only for management. Cups is used to install a d manage the printer, all your computers can then print over the network using cups, but they still need to configure the (remote cups) printer locally.

I use scanservejs since at least one year and can confirm it’s the best.

As for printing, I went down the same rabbit hole and had zero luck. Seems such a stupid thing yet nobody created such a service, except for a couple of commercial ones.

So yeah, if you find something …

By the way, probably creating a simple web interface that accepts uploads and pipe to lpr is pretty easy to do, but I didn’t got around to do it. Printing feels… Outdated. And printing from computer or android is so easy anyway that probably useless.

Llama CPP can run models offloading with CPU (es MoE models), you have much more control over how you run your models, and overall it’s very much actively developed.

For starters and people without too much willingness to mess up with stuff, ollama is a great choice. Llama.cpp gives you that extra power and flexibility that is so much worth for people who like to tweak and do more.

My personal opinion, of course. But based on having used both and ditched ollama for llama.cpp, so I am also biased, keep in mind.

But I will hardly go back to ollama now :)