Running Your Own Stack: Open Source Models Without the Tears

Why this session

Lou had been getting the same question from members in different forms. Should I be running my own models? Where does the open source story actually pay off? The session was a tour of the options and the tradeoffs, with enough technical specificity that nobody had to walk away pretending to understand something they did not.

The cost of intelligence

The frame Lou opened with was a tradeoff, computation versus intelligence. A 20B parameter model needs roughly 20GB of memory to run. A 120B model needs around 128GB of unified RAM if you want it to be comfortable. Most laptops do not have that, which means most members are choosing between three real options:

1. Use the hosted version on the project’s website. Convenience, with a small confidentiality risk that the open source community generally polices well.
2. Self-host on a virtual private server. Your hardware in someone else’s rack. Fine if you understand basic security hygiene, painful if you leave keys in the open.
3. Run on your own machine. Best privacy, requires a Mac with unified memory or a serious GPU, and a willingness to learn Docker and Python environments.

The license footnote that mattered

Lou spent more time than he expected on licenses. Apache and MIT let you do basically anything. GPL means your changes belong to the community. Open Web UI has a clause that flips at a user count threshold, which a few members had not noticed in their stack. The lesson, read the license carefully if you intend to honor it, which the cohort agreed they did.

The Docker shortcut

The thing that lowered the activation energy for the room was Docker. Find the project in the hub, install the container, click the arrow, you have a local instance running on a port. No Python environment fights, no dependency hell. For the members who wanted to experiment without committing to learning a new operating layer, Docker was the entry point.

The honest answer to “should I do this”

Lou closed with the answer the cohort came in looking for. If data confidentiality is paramount, run on your own server. If you just want to play with open source models without the friction, use the hosted versions. If you want to build something commercial on top of an open source project, read the license twice and then build it on a VPS you control.

The members who were already self-hosting compared notes on which models they were running. The members who were not left with a clearer sense of whether they needed to be.