Potree is a self-contained, performant library for handling point clouds. This is great if you run into needing this with a client! However, it can be hard to take total control of Potree to use it for just your own purposes, so roll up your hacking sleeves.
https://github.com/potree/potree