I went deep in on this because I thought it sounded cool and I am a dev. Of course I used an LLM to help me understand the code as I wasn’t familiar with some of these technologies.
Basically Moxie open sourced his work for:
Confer image
This project creates a virtual machine image that will securely run local LLMs. You would need to find an LLM cloud provider to run this image as the hardware requirements for the underlying virtual machines are for corporations-only and you couldn’t buy them in store. You’d be running LLMs that you’d get from hugging face instead of the proprietary ones like ChatGPT. This is really not a hobbyist project, but for businesses like hospitals or banks that may have strict confidentiality requirements.
Confer proxy
This is a web server that will proxy client (ex. browser, phone) requests to one of two cloud providers: OpenAI (proprietary GPT stuff) or TogetherAI (hugging face open sourced models). You can also configure it to proxy to any other URL as long as that server supports the super popular OpenAI API format.
This server is written in Java (uggh) and it’s expecting the client to use this super niche “noise protocol” for encryption for your chat session (eg. WebSockets for bi-directional communication). There isn’t even an actively maintained client-side library to work with this protocol, so it looks like it would be a pain to implement. Once you beat this “noise protocol” boss you can basically create your own version of https://confer.to/
Free tier gives you 15 minutes of unlimited use before you have to stop using it or “re-register”. Once you pay it’s unlimited for however long their auth/billing (not open sourced here) system will let you. There are no model restrictions.