HACKER Q&A
📣 fallinditch

Recommendations for Running LLMs Locally


Hi folks

I'm looking for hardware recommendations for running LLMs locally. Specifically, best bang for bucks at the following price points: $2,000 $5,000 $10,000

What more could you do with a 10k budget than with a 5k budget?

Thank you

  👤 sandreas Accepted Answer ✓
Well, this might be unpopular but for the 2000$ price point it may be the macOS ecosystem.

The most important part to run LLMs might be the VRAM-size. Since modern Macs use unified memory you simplified get RAM=VRAM, so a 64GB MacBook Pro might outperform an NVidia 4090 in LLMs.

However the specialized cards from nvidia have more VRAM and are generally faster.

The LLM of your choice should fit into vram, so if you only need 21gb, one 4090 will do the job

👤 vunderba
Previous thread from just a few days ago (5k budget):

https://news.ycombinator.com/item?id=42853287

👤 BMc2020
I have an old dell latitude 3420 with 16 gig of ram. It's running Pop!_OS. I used Alpaca to load the Mistral(7b) model. I ask it questions and it answers. Didn't cost anything.

On a different computer running Cinnamon Mint I used GPT4ALL to load a Llama 3 8b model. I use it to read a bunch of local docs for me. It also didn't cost anything.