Why does my Node.js multiplayer game lag at 500 players with low CPU?

What are your processes waiting on? in Linux top, show the WCHAN field. In FreeBSD top, look at the STATE field. Ideally, your service processes are waiting on i/o (epoll, select, kqread, etc) or you're CPU limited.

Is there any cross-room communication? Can you spawn a process per room? Scaling limited at 25% CPU on a 4 vcpu node strongly suggests a locked section limiting you to effectively single threaded performance. Multiple processes serving rooms should bypass that if you can't find it otherwise, but maybe there's something wrong in your load balancing etc.

Personally, I'd rather run with fewer layers, because then you don't have to debug the layers when you have perf issues. Do matchmaking wherever with whatever layers, and let your room servers run in the host os, no containers. But nobody likes my ideas. :P

Edit to add: your network load is tiny. This is almost certainly something with your software, or how you've setup your layers. Unless those vCPUs are ancient, you should be able to push a whole lot more packets.

Are you buffering your output? Doing one syscall (write) for each client in a server for each keystroke is a significant amount of IO overhead and context switching.

Try buffering the outgoing keystrokes to each client. Then, someone typing "hello world" in a server of 50 people will use 50 syscalls instead of 550 syscalls.

Think Nagle's algorithm.

3000 pps / 6 Mbps is pretty much nothing for that server. I wouldn't change random network sysctl options.

> This suggests to me that the bottleneck isn’t CPU or app logic, but something deeper in the stack

Just a word of caution - I have seen plenty of people speed towards eg "it must be a bug in the kernel" when 98% of the time it is the app or some config.

Node gives access to event loop utilization stats that may be of value.

    import { performance, EventLoopUtilization } from 'node:perf_hooks'
    performance.eventLoopUtilization()

See the docs for how it works and how to derive some value from it.

We had a similar situation where our application was heavily IO bound (very little CPU) which caused some initial confusion with slowdown. We ended up added better metrics surrounding IO and the event loop which lead to us batch dequeuing our jobs in a more reasonable way that made the entire application much more effective.

If you crack the nut on this issue, I'd love to see an update comment detailing what the issue and solution was!

Don’t worry about the number of people users. Instead consider:

* the number of total sockets as I suspect there could be multiple sockets per user.

* investigate what socket.io does to serialize messages both on and off the wire. I wrote my own WebSocket library for Node and noticed the cost to process messages on the receiving end is about 11x greater than on the sending end. Normally that doesn’t matter until you push it past a critical point. At the critical point everything begins to super crawl because the message quantity per interval exceeds the garbage collection cycle and everything backs up. In my case this scenario didn’t realize until 180000 or 480000 messages per second depending upon the hardware. The critical difference from the hardware side was only about memory speed and cpu availability was largely irrelevant.

* also look at what socket.io does, if at all, to queue messages at each side of the socket. Message queued both on and off the wire will be a factor if not properly managed or if absent

Try cluster mode? I.e. use all cores.

Anyway please follow up or blog when you solve it. Sounds interesting.

Please also note that Hetzner is not providing CPU steal information inside of VMs. So there could be 75% steal and you wouldn't notice! It's unlikely for CCX instances, but it happened for me a lot with regular instances.

Any monitoring/logging in you nodejs code?

I noticed, for example, adding a newrelic agent drops http throughput almost 10x.

It sounds like you want to coalesce the outbound updates otherwise everyone typing is accidentally quadratic.

Are you awaiting anywhere, such that you might be better off doing fire n forget instead?

25% CPU usage could indicate that your I/O throughput is bottlenecked.

are you using uwebsockets.js?

You say you use Docker via Docker Swarm, so maybe this is about the Docker network setup?

I haven't tried Swarm, but to some degree assume it can give the same effects as Docker Compose with several services. I also less sure of the effects if you never have communication between containers, but I think perhaps there may still be the same or similar issue.

What I experienced when doing not exactly a load test, but just processing a large dataset through multiple docker containers started from a docker compose config, was that the default docker network loopback (docker0) was saturated. After creating a docker network that the various nodes were configured to use, things got a lot better.

So this is the question for you, do all the containers in the swarm talk via docker0? If yes, read up on docker networks in relation to swarm in particular.

Big thanks to everyone who commented so far, I wasn't able to reply to everyone (busy trying to fix the issue!) but grateful for everyone's insights.

I ended up figuring out a fix but it's a little embarrassing... Optimizing certain parts of socket.io helped a little (eg installing bufferutil: https://www.npmjs.com/package/bufferutil), but the biggest performance gain I found was actually going from 2 node.js containers on a single server to just 1! To be exact I was able to go from ~500 concurrent players on a single server to ~3000+. I feel silly because had I been load-testing with 1 container from the start, I would've clearly seen the performance loss when scaling up to 2 containers. Instead I went on a wild goose chase trying to fix things that had nothing to do with the real issue[0].

In the end it seems like the bottleneck was indeed happening at the NIC/OS layer rather than the application layer. Apparently the NIC/OS prefers to deal with a single process screaming `n` packets at it rather than `x` processes screaming `n/x` packets. In fact it seems like the bigger `x` is, the worse performance degrades. Perhaps something to do with context switching, but I'm not 100% sure. Unfortunately given my lacking infra/networking knowledge this wasn't intuitive to me at all - it didn't occur to me that scaling down could actually improve performance!

Overall a frustrating but educational experience. Again, thanks to everyone who helped along the way!

TLDR: premature optimization is the root of all evil

[0] Admittedly AI let me down pretty bad here. So far I've found AI to be an incredible learning and scaffolding tool, but most of my LLM experiences have been in domains I feel comfortable in. This time around though, it was pretty sobering to realize that I had been effectively punked by AI multiple times over. The hallucination trap is very real when working in domains outside your comfort zone, and I think I would've been able to debug more effectively had I relied more on hard metrics.