Hi, I want to use an on-device (Pixel 7 and MacBook M1) AI chatbot. I think there is a lot of options. Could someone guide me to a good option please?
Best
You need to tell us what exactly you’re looking for in a local AI and why and how you want to use it
Also, if it’s local and privacy is your concern - I don’t think it matters which one you use because… well, it’s local.
But I’m no expert and could be wrong. Let’s see what others have to say
How much memory do you have, and what version of M1 (base, pro, max, etc)?
Llama 3 8B quantized is one of the most popular models among hobbyists and home users at the moment. The hardware requirements are not excessively high, and the quality is good for the moderately small size of the model.
In addition to the model you’ll want an application/wrapper to interact with it. There are lots of options. ollama + openwebui is one of the most popular choices right now, ther are also more traditional applications like jan.ai There are also options that bundle both the model and the frontend like Mozilla’s Llamafiles
1 Like
uses GPU GitHub - mlc-ai/mlc-llm: Universal LLM Deployment Engine with ML Compilation
I didn’t try any of these, but heard ollama app is the best. Other members can test them and let us know their experience.
2 Likes
I think that the choice matters even for a local AI because even if the prompts don’t leave the device, there could be telemetry, personal information or usage information or network or device information that are collected.
The choice matters also because we want an app that is open source, actively developed, that don’t leave the conversations in plain text in the computer, etc.
Anyway, thank you for your answer
Ah I did not know that. I thought differently about it so wasn’t fully sure.
Thanks for clarifying.
I have the MacBook Air M1 with 8Gb of RAM
Thank you a lot for your suggestions. I am curious why there are so many applications and so many models released (((even if most seem purely inferior to the leading llama etc, why don’t just use Meta’s one or Mistral or one or two more ethical LLM instead of developing hundreds of them)))
Anyway, I looked at Jan and Ollama and they seem good thank you. What makes Ollama less “traditional” than Jan ?
Lastly, where can I find a good source of information or good guides about this subject please ? Because I also would be interested in an app like this on my smartphone
Well I don’t see how this could help me but Maid seem to be some kind of local AI assistant for Android so thank you (even if I don’t like the branding (name and logo)). The app also asks for Network access and nearby devices so I uninstalled. I also don’t see how it could use OpenAI or Gemini that easily…
2 Likes
Well, I wrote a guide here update!: Add AI chat recommendations by I-I-IT · Pull Request #2525 · privacyguides/privacyguides.org · GitHub
Discussion
Add AI Chat tools
From my experience webLLM is not that great, it doesn’t work on Linux and it’s working on Android phones but just not great because you’re constrained and can’t run models of more than 3 billion parameters.
Phones just aren’t ready to run AI models yet.
What I meant was just that Jan is like a traditional desktop GUI app. Something you just download, click to launch, and just use (after downloading a model). (I think GPT4ALL could be a good option in this category as well)
Whereas ollama is often/usually used in combination with an additional frontend like Open WebUI. You can use ollama directly from the terminal, but it seems most people (myself included) use a frontend in front of ollama. There are various ways to install ollama, I used docker (podman actually), and the OpenWebUI frontend which is accessed through any web browser.
I have the MacBook Air M1 with 8Gb of RAM
8GB total unified ram (only some % of that can be used as VRAM) is kind of towards the low end when it comes to current LLMs, but you should still have some options available. If not Llama 3 8B Q4 (which is about 5gb), then one of the smaller Phi or Gemma models. I think your first step should be determining how much of the unified RAM can be used as VRAM on your model macbook. Then you can experiment with models until you find one that is a good balance to you.