Add AI Chat tools

Site Development Tool Suggestions
204

I tried running it on both the Pixel 8 Pro and the iPhone 16 Pro. Although the iPhone has 8GB of memory, the largest model I was able to load was around 4.5GB. All models larger than that would not load at all.

In contrast, the Pixel has more memory than my iPhone, which allowed me to run larger models (around 9GB memory, 8B parameters). However, the inference time was painfully slow.

1 Like
205

I prefer to use the Duck chat now, but the app still seems very cool.

Perhaps when mobile hardware improves or larger models become more efficient, this will become a thing.

206

İPhone 15 a16.

The biggest Qwen model (Qwen2.5 3B(Quantisation5)) is sometimes a bit slower to load. But apart from that the app is working fine.

207

Interesting. How was the speed with 3B models?

I guess it’s better to stick with SLMs. When I have time, I will PR to add pocketpal and also explain what you can and can’t do on a smartphone AI.

On Pixel 9, I found Llama 3 8B speed to be OK actually. Like it’s not super fast but it basically is a fast as you can read.

Probably better to stick with Q4. The generation speed is OK then?

208

Yes. It approximately takes 10 seconds to load and generation speed is 8-10 tokens per second.

1 Like
209

I really like Opera’s Aria AI. I was able to create a fake account with a cock.li address and now use the service via a VPN.

The Aria AI is almost as strong as ChatGPT (I use it for business, or have to)

1 Like
210

Brave Leo and DuckDuckGo AI exist.

1 Like
211

Hugging chat is free to use and has a decent privacy policy.
For Firefox users, it can be used in browser just as Leo, with the recent updates.

2 Likes
212

Personally I’m using o1/o1-mini from OpenAI. I know It’s shitt* for privacy but I’m not aware of any other AI chatbot with good privacy and high performance for complex tasks and high reasoning (and that’s what I need in my studies). The last version of OpenAI is very, very good.

So efficient that It’s disturbing.

For common tasks I’m sure there are some chatbot respecting your privacy though

1 Like
213

Will this be also available for Firefox derivates? Is it fully Open Source?

214

Yes, and not really. Probably open source but only allows conenction to proprietary chatbots, not local one. (AFAIK)

215

That depends on the fork/derivative maintainer’s decision. The UI is open source, but it connects to proprietary models, except for hugging face.

1 Like
216
  1. And Mistral
  2. Additionally, for semi-advanced users, Firefox includes the ability to connect to your own locally hosted model OR to connect to a custom cloud provider of your choice. (these settings are found in about:config)

I’ve used Firefox’s AI sidebar with:

  1. Locally hosted open and open-ish models (e.g. Mistral Nemo, Llama 8B, ) [1]
  2. Cloud hosted open models from various hosting providers [2]
  3. I currently have it set to Duck.ai, and I use Llama 3 70B. Previously I was using locally hosted Llama 8B, but my system is too underpowered for that to be convenient.

In most cases, I assume so. Most Firefox derivatives are essentially “Firefox + a few changes”. So typically things that are available to Firefox are available to derivatives of Firefox unless the derivative takes steps to explicitly exclude something and prevent users from changing it. I know that Firefox’s AI sidebar feature can be used in both Zen and Librewolf, not sure about other browsers.

  1. Relevant about:config preference: browser.ml.chat.hideLocalhost set to false ↩︎

  2. relevant about:config preference: browser.ml.chat.provider set to the URL of your choice ↩︎

1 Like