What does your local LLM setup looks like?

Question

What models and hardware you are using? For what purpose? What were the challenges? Any tricks that helped you in doing this?This might help new users like me setup theirs.

xiaoyu2006 · Accepted Answer

V100 32G SXM2 adapted to PCIE. Running llamacpp with Q4KM Qwen 3.6 27B or Gemma 4 31B.

abahgat · Answer

I bought a pretty powerful desktop computer for gaming in late 2025. It came with an RTX 5080, which I started using to run some local LLMs and run some experiments (most recently, I was trying to get agents to get better at playing Zork I).
I've mostly enjoyed having WSL to leverage Linux dev tools, but it seems like it's still adding overhead that prevents me from taking advantage of the GPU in full, so I'll likely get another drive and install Linux.
I tried Qwen, Llama, Mistral and Gemma. Gemma 4 was pretty impressive.

aniokono · Answer

Claude.MD and other MD context files with guardrails and enforcement?Are you offering something in this space?

What does your local LLM setup looks like?

What models and hardware you are using? For what purpose? What were the challenges? Any tricks that helped you in doing this?
This might help new users like me setup theirs.

V100 32G SXM2 adapted to PCIE. Running llamacpp with Q4KM Qwen 3.6 27B or Gemma 4 31B.

Claude.MD and other MD context files with guardrails and enforcement?
Are you offering something in this space?