I have found a few so far:
Is there any reason to use one or another? What am I sacrificing choosing one or another?
2 Likes
DeepSeek R1 can also be run locally.
1 Like
Yes, the platforms I mentioned run individual llms such as llama, deepseek, and qwen.
1 Like
I’ve tried Jan, but with my 2080 Super, it runs slower than I was expecting for my first time actually messing with an LLM.
I think this very possibly relates more to the model you chose to use than the software you are using to run the model (unless you ran the same model using other software and experienced significantly quicker speeds).
Features, support for your OS, UI/UX, community/mindshare, where on the spectrum it falls between beginner friendly and advanced/feature rich, FOSS or not, and so forth, whether you want a desktop GUI app, CLI, or a service meant to be locally hosted on a server.
Personally I like and use ollama (backend) + open-webui (frontend). LM-studio and Jan are more beginner friendly, and more like traditional desktop GUI apps. Mozilla has a very different take on things that is also quite straightforward called a llamafile where the application and model are bundled as a single file
1 Like
Jan.ai runs really well for me personally, similar response speed to online services like ChatGPT and Gemini.
Which model did you try using? It could be that you’re running a model that’s too heavy for your system. You need quite a bit of RAM so that could be a limiting factor.
Take a look at Privacy Guides recommendations:
3 Likes
LM studio now supports RAG for Compatible models, which could be super useful, not sure about others.
There are also local AI clients for mobile:
Both IOS and Android: PocketPal
PocketPal provides some LLMs to download through the app and updates them automatically.
Only on android: ChatterUI
ChatterUI requires you to download the LLMs manually. It also allows you to use remote LLMs and use some remote LLM providers APIs such as ChatGPT.
I was running Mistral 7B Instruct Q4 and have 64Gigs of RAM.
1 Like
Be sure to check out Pinokio AI Browser. You can run all kind of local AI Apps (not just LLMs) with one-click-install.
iOS: Enclave AI
1 Like
Also see Add AI Chat tools
I wouldn’t trust it. For one, they are faking being open-source. Their github has only 13 commits, the latest being 27 days ago. Yet, their latest release was yesterday - meaning they publish proprietary builds. Second, pinokio sonds like Pinnochio, which is a liar.
1 Like
Interesting that runs fine for me and I have only 32GB of RAM, it might be an issue with your system perhaps? 
A 7B model should def rum very fast on a GPU, esp Nvidia.Does JanAi download libraries like Cuda?
I’m not sure! I only run Jan.ai on a Mac and it just works seamlessly without the need to install any additional software apart from downloading the models themselves.
1 Like
Maybe @Rasta try ollama or Kobold.cpp and compate the token/s you get?
if you can offload the whole model into GPU and VRAM then system ram and CPU is much less important.
Great idea for a thread!
I’m looking through your list now.
I prefer the idea of being disconnected to the Internet as much as possible.
1 Like
Honestly, I’ve had the best experience with LM Studio, though sadly its not really open source. Never heard of Jan AI, so I’ll give this a try instead.
Otherwise cheers for the list! I kinda hate the fact that AI companies even though you PAY THEM, will still data harvest everything you say to train more products for you to buy.