Well, there are a few things you can do, and a few things you might not be able to.
One is to stop users from upload executables or anything weird.
Another is Cloudflare which magically blocks countless categories of attacks, although it isn't omnipotent, so don't use that to compensate for non-existent security.
You should keep everything on your server / site up-to-date, so that people can't take advantage of security holes in those. A major big board with what was it a quarter of a million users got hacked and had their database dumped because they ran an old version of vB.
One thing I do which may or may not help is that I do not allow them access to any folder directory whenever possible, otherwise they might be able to enumerate over the files looking for flaws or do path tricks to break out.
The trick is to serve requests directly with a forum instance in memory (ala NodeBB, Gosora, Discourse, etc.), or to have a reverse-proxy like Nginx hand over requests to it, but disable any /www/ folders, etc. if possible, it might not be possible with some software.
It isn't impossible with PHP, but no software supports it to my knowledge, the trick is to disable the webserver's capability to run any PHP file it finds or serve files you don't want served but are in strange places, e.g. software configuration files with your database user inside or attachments in private forums).
You can theoretically do this with .htaccess, but not every webserver supports .htaccess (e.g. Nginx which runs circles around Apache, which is a snail by comparison) and you run a risk of things being misconfigured due to the more obscure syntax of the thing.
For Nginx, there is a central configuration file, but it only takes one mistake to make the whole thing moot. These aren't solutions however, they are band-aids for a greater problem, and they sometimes fall over due to this.
Additionally, you ideally want a software which isn't vulnerable to SQL Injections (
#1 way sites get hacked, which is sad because it's a solved issue with prepared statements which many software don't use), or failing that, a web application firewall to at-least eliminate some classes of attacks, but make no mistake, there are many ways to bypass a web application firewall.
Remember, the only valid response to, "Do you use prepared statements?" from a forum development team is "Yes." Failing that, "Working on it."
I also know that PHP / MyBB can be a little chatty and give away a bit too much information about your server / site in error messages, but I believe there are settings to disable that and keep those messages exclusively to error log files.
You might also want to lock-down the file permissions, so they can't be modified by strange processes and you probably want to use something other than root for your database user. You also want to make sure a password is set on root and your database user, something secure, that is.
I say this, because
ten thousand MongoDB instances got hacked one day because the admins left them with the default settings (no password), so that might be something to watch out for.
It winded up getting pretty bad.
