Gosora Forum Software: Supremely Fast and Feature Packed

[Azareal]

There, we go, I'm finally running into 0 / 0'.
I dunno why it randomly started turning up now, maybe less turbulence in Windows?

0 means that it's too fast for the timer to track, so somewhere around 200 microseconds and under.
If you're not familiar with microseconds, there are a thousand microseconds (μs) in a millisecond (ms) and a thousand milliseconds in a second.

The fastest unit in all of computing (nothing can really go faster) is the nanosecond. There are a thousand nanoseconds in a microsecond, however it's physically impossible to render a page in less than a microsecond (that doesn't mean you can't render them in parallel, the future is multi-core!).

This doesn't mean that you will get the page in the time specified though, that is a violation of the laws of physics, I wish I could break those laws because they are very annoying, but sadly, I am incapable of doing so at this time, maybe when we get one of those sci-fi faster-than-light communication devices.

Anyway, asides from the page generation, there is a little bit of overhead from the HTTP Stack and, of course, there's the TCP protocol which powers just about everything across the entire web. On-top of that, on occasion, is the database which can be quite slow (in comparison), but luckily, I can largely predict where users are most likely to go and cache those spots.

If the site is small enough, I can cache the entire thing in one go and serve it entirely out of memory, well almost, lots of things to do.

On another note, there seems to be a lot of discussion among the community about Go 2.
The Go Team are forming draft plans on what they want to change in the standard library and are also pushing discussions on ways to help simplify error handling and adding generics.

They did some great work on dependency management this year and I'm looking forward to where they go next. (I haven't forgotten about WebAssembly, but the browser spec is very immature, to the point of not being that useful, maybe in a year or two we'll be able to do away with JavaScript.)

Wow I think the level system is attractive! 🙂

About the dicenroll I think it should be a premium plugin for your income sir.
Its a great idea for gaming forums, like fighting a Boss, all players will just post anything,
then in their post there is random dice that will tell how much damage to inflict to the boss.
Also in those roll they can also get damage from boss, I saw that system in terrarp.net before.

Sir how about a Rating system?
when they post a reply to your thread, they must choose a number of stars for your rating, just like
google play, they require you to choose your stars for rating before adding your message.

About the footer sir I think it will look great if you can make a footer that is really cool and unique?

Thank you very much 🙂

You want some sort of system where the OP is an item to be reviewed and the replies are reviews? I suppose it could be some sort of plugin, it wouldn't be too hard to do, although I'm slightly sceptical whether people would actually use it lol

A RPG system is interesting, I think I made a crude awful one for MyBB... When was it... Around six years ago. Good times. I'll see what I can do with that, although it'll be a fair bit of work.
I'm not sure what to do with the footer, I'm keeping it simple for now, although I might play around with it later.

 

[Azareal]

A second post to help break up the already lengthy one.

One thing which has always bugged me about forums is the permissions.
They are just. Painful. To the point that admins don't really want anything to do with them, if they can help it.

Every-time you touch them, they are a drama. You have to run through a several sub-panes, checking through checkboxes to get what you want for each and every group and forum you have.

In Gosora however, I've tried a different approach from the mainstream.

Here, group permissions are split into global permissions and local permissions.
By default, if you don't set permissions for a forum for a specific group, then it will inherit from the local permissions for a group, otherwise it will use the permission set specified.

The benefit of this is that you don't have to play the little game where every-time you create a new forum, you have to set the permissions for every single group you have created.
90% of the time, I have found that I'm just using the same basic template: Admins can moderate a forum, ditto for mods, banned can't view it, members can post, and guests can only view.

How often do you really customise that further? Even when you do, how often do you customise the permissions for each and every group on that forum? So, I thought. Why not do away with that nonsense?

This isn't all either. I also have a system of presets. Rather than throw you into a sea of checkboxes for a forum-group pair (you can do that if you want), you can select a common preset like Read Only or Can Post or Can Moderate for that forum-group pair and the software will deal with setting the permissions associated with it.

There is still work to be done, however like porting all of the forum perm UIs to Nox and making things a little friendlier, so that admins can see how it cascades down, but I think it works fairly nicely.

 

[MrDangem]

here is my latest suggestion sir from original to my suggestion:

 

[Azareal]

I've been doing a fair bit of theme work including splitting generated templates into halves. One optimised for guests and one optimised for members for each route which has a generated template.

I'm slightly concerned about the potential effects on the CPU cache, but things seem to be working well so far. Now, if only I could get rid of some of these interface related indirections, maybe with package unsafe to punch a hole in the net/http's abstraction? Might be an interesting area to explore.

There is a compiler optimisation which the Go Team can make which I believe people call devirtualization (sort of bad yet simple example) which might also help with those indirect calls, essentially flattening them out into normal function calls. It also opens the door to a host of other potential optimisations like inlining.

FastHTTP is a good alternative which over ten times faster than net/http, although it's going to be a while before they get HTTP/2 support and Go 2 seems to be practically around the corner, and it's likely to shake things up in net/http.

I'm not in too much of a rush to get rid of the indirect calls in the templates yet, so I'll just watch how things pan out. Other theme work includes upping the number of items visible on control panel pages and revamping some of the interfaces to make them look nicer.

I'm also playing around with making some things more mobile friendly like the topic list, profile, account manager, and the control panel. This is a work in progress though for the last three.

I wasn't originally going to include the control panel, but it shares styles with the account manager, so it isn't as much work as I originally expected. I am going to have to figure out how to fix that header though to better mesh with it.

Another thing I'm thinking of is a feature to force all staff to use 2FA, although it would have to be optional. The last thing you need is one of your mods or admins picking a stupid password or falling for a phishing scam or one of many other problems.

I don't think I've ever shown the 2FA interface, so:

As you can see, I have 2FA setup, although don't expect me to show you my keys.

It is such an important feature that I've put it in plain view the moment you enter the Account Manager rather than hidden away, it isn't SMS 2FA either, but the more secure and arguably more flexible, authenticator based SMS.

Asides from those things, I've also fixed a bug in the permissions system.

Don't worry though, it hid random forums rather than showing things you shouldn't be able to see and I've further hardened the software against threats demoed in security conferences a few months ago.

I might add Subresource Integrity at some point to harden things even further, particularly when dealing with pesky little CDNs and caches, but we should be generally fine for now.

here is my latest suggestion sir from original to my suggestion:

What anime is that?

That is an interesting suggestion.

I've tried to keep the UI simple as tall boxes takes away from room for actual discussion, but maybe some variety might be nice. I believe it can be done via templates without a change to the actual code, although not the extra features and the bar, maybe I can add a template for bars.

 

[Azareal]

I'm experimenting with reusing blocks of memory where possible.

I was a little worried that the data wouldn't end up in contiguous blocks of memory, but it seems that transferring less data between the CPU and Main RAM makes up for the cache misses, as it winds up making the topic route 50% faster for guests.

I will have to do some more tests though.

 

[Azareal]

Should I let admins delete modlogs / adminlogs? They could make a good trail if there's a security incident, so I'm hesitant to do so.

 

[Azareal]

https://github.com/Azareal/Gosora/commit/bdf7fa40d59f9ad46f492e81e07d8a96def06319
Pushed another commit. You can play around with that or use the live site.

One thing I'd like to add is login logs, so users can see who has been in their account and when.

 

[Azareal]

https://github.com/Azareal/Gosora/commit/543ad8a0182e3773d974f2df6106a6569a1fc6a4
Another update, although some of these features are very new.

Login logs have been added. You can now see every IP which has logged into your account.

Added better pagination for topics, not just Tempra Simple style pagination.

You can now jump to the last page in a topic by clicking on the date on the topics list and forum page. Further refinements may be made here to jump to the last post on that page to save you a bit more work.

I've also made some improvements to the UI, made a few more optimisations, turned the "experimental optimisation" back on for further analysis and some other little changes.

You will need to run the updater or patcher for this update, as it includes two new tables: updates and login_logs

 

[Azareal]

I've made all routes about 50% faster and significantly reduced the number of allocations by using package unsafe to remove some unnecessary string copies. This should help to avoid spinning the GC, which in turn improves performance further.

This may cause issues with some environments which don't support unsafe like Google Appengine, but we'll see what we can do to fix that, most likely a fall-back to the previous less efficient behaviour.

The little change with the updates table, by the way, is part of simplifying the update process to make it a little less brittle for advanced users and maybe to open the door for Docker deployment.

I'm also reducing the amount of duplicated code for the attachments and we might see some improved attachment handling making itself in. Attachments as they are now work, but they're fairly rudimentary without any functionality for managing them.

I'm largely backing away from making template generator changes for now, as I hope that the recent changes will be largely sufficient performance-wise, but we'll see how things go in production and think of ways to make things faster, if we can.

Also, as noted in the previous post, I'd like to improve those last poster links a little more, perhaps by storing the last reply's ID on the topics table, so we can have a link pointing there without an additional query? A page that simply redirects the user might work too.

Also, does anyone want a who's online like thing? It might make a good widget and it's extremely easy to throw in.
I'm slightly concerned for the implications for privacy, particularly the legislation Europe has pushed relating to it though.

 

[Azareal]

https://gosora-project.com/topic/patch-11.63
I've got a Christmas present for you all.

I added an attachment manager on topic edit, improved the update system a bit to make it less brittle, improved some of the UIs, added the reltime template function so you can convert time.Times to relative times on demand in templates, fixed a bunch of things, made sure that the times in the topics / forum pages now takes you to the last post rather than the top of the page that said last post is on, etc.

There's more to see in the changelogs.

 

[Azareal]

https://gosora-project.com/topic/patch-13.65
Apparently, I have acquired a taste for writing announcements lol

Patches here refer to the patcher, the thing which installs database schema changes, these schema changes have their own internal version numbers, but right now, it's useful for grouping a bunch of related changes together into a micro release for announcement sake.

The announcement manager is now complete, covering both topics (aka opening posts) and replies. In Gosora's internal architecture, the opening post *is* the topic, in fact you can have the opening post appear at the top of every page with zero additional cost, as the data is all there.

I've also made a number of performance improvements, because we all love those. I've added some database indices to help avert o(n) disasters, greatly reduced the number of allocations for the user agent parser, and expanded the benchmark suite to help find more areas for improvement for performance in the future.

I have also fixed a major bug where the second ticker would sometimes get hijacked by the Watchdog Resource Manager which lords over the topic and user caches, making sure that they don't grow too stale while hopefully not getting in the way too much either.

This meant that certain tasks didn't run like the ones required for ban expiries, the live topic ticks (I mention that I've fixed it but didn't test it and that happens, just my luck, those words tempt fate lol), and the thawer which regulates the topic list cache.

So that people don't forget that we have a widget system, I'm thinking of pushing an optional who's online widget up the list, although for variety of reasons, it won't cough up someone's precise location on the site, simply that they are.

I might toss in an invisible mode, I dunno. Guests are invisible, as we don't want to waste precious resources on them. They simply. Are.
The administrator can get a vague guest count in the Control Panel.

Unlike many systems, Gosora knows exactly how many people are visiting the site at any given time, as users are continuously connected via WebSockets to get their live alerts and live topic list.

Unfortunately, this also means that a user can be simultaneously viewing twenty pages at once creating a conundrum for any would-be stalker or any other individual. All in all, people don't really need this data anyway.

I might also push @MrDangem 's ideas in a future child theme Midnight, but not in Cosora or Nox. The simplicity is nice there.

 

[Azareal]

Experimenting with making a user interface for the Widget System.
We have a widget system, I use it all the time, but not a proper UI for admins to play with.

Two important widgets will likely come up. The Search and Filter Widget which will finally introduce a search system (perhaps with ElasticSearch or maybe what was that library called? Riot?)
And also, the WOL Context Widget, so that you can see who else is viewing a topic. It does not give you any information beyond that, except maybe their avatar.

The WOL Global Widget might also be added as a proof of concept, although it would simply show who is online and not much beyond that. It's a simple one, as all I have to do is iterate over a lightly sharded map (WsHub.evenOnlineUsers and WsHub.oddOnlineUsers), generate a bunch of HTML off that, and then, reuse that block of HTML every-time someone views the widget.

The location field in the widgets allows you to specify which pages you want it to show up on. E.g. the topic list, the control panel, everywhere, everywhere not a certain page (by putting a ! in-front which means not), or even multiple specific pages by separating them with |.

I've been waiting six months to implement the search system, so this should be fun.
And then, we'll see about revamping the theme system under the covers for easier template overrides.

 

[Teg]

I’m very impressed! Kudos to you! I’m actually in the process of building my own CMS with LMS which will have a combination of learning management systems, blogs, forums and the likes coded in PHP. I’m about six months in and nowhere near an alpha. Super awesome job you’ve done so far and I can’t wait to see it once you implement everything. Keep me updated.

 

[Azareal]

I’m very impressed! Kudos to you! I’m actually in the process of building my own CMS with LMS which will have a combination of learning management systems, blogs, forums and the likes coded in PHP. I’m about six months in and nowhere near an alpha. Super awesome job you’ve done so far and I can’t wait to see it once you implement everything. Keep me updated.

Thank you very much. What do you mean by LMS?

 

[Teg]

@Azareal Learning Management System. It's kinda going to be an all in one CMS, at least that's what I'm hoping for. 😛

 

[Azareal]

https://github.com/Azareal/Gosora/commit/8f2f47e8aaa5844933b3fec70364eea653b44aeb
Rolled it out a bit early due to a potential security issue in the simple widget where an admin might decide to put sensitive information like passwords.

A lot of rewriting, refactoring the Online Users Widget and part of the Widget Management UI, so you don't need to fiddle with the database to manage widgets. The other widgets (e.g. the Search and Filter Widget) are part-done, although not ready for prime time.

I plan to use Elasticsearch for the search system with an inferior fall-back. It seems that it'll allow me to do little suggestions as you type in the search box and it's nowhere as pitifully slow as fulltext search.

A pure Go alternative might also make a good alternative fall-back for Elasticsearch which is written in Java which is *usually* a slower language and it'll help simplify things, although it won't be as feature rich.

 

[Azareal]

https://gosora-project.com/topic/patch-14.68
I made a proper topic for Patch 14.

Online Users Context has been rolled out now too, so users will know when someone else is browsing a topic which they are. Like Online Users, this is capped to thirty users and after that, the widget simply displays the number of users to avoid spewing out a zillion users we probably don't care about.

 

[Azareal]

I'm doing a bit of refactoring with the theme system to reduce the maintenance burden. Unfortunately, there a global two indirect costs of I dunno ten nanoseconds each? But, a smarter template generator could probably get that down to one or none.

It'll also be easier to override templates when you have multiple default themes, which just happens to be useful to core, so that's always nice. I've also tweaked the arguments on the handlers for plugins, so we can remove some unnecessary boilerplate when registering and unregistering hooks.

Templates are essentially pure functions, so I might be able to get away with pretty aggressive arguments in some places where I can just use the results of a subtemplate twice if it's computing the same thing, although it may involve using a map which well, when you call it, it looks really simple, right? Something like blah["haha"], right?

That is an illusion. This is what a map really looks like under the surface:
https://github.com/golang/go/blob/master/src/runtime/map.go#L396
Every-time you call a map, you're actually doing *that*.

So, I'll have to do some experimenting to make sure the map doesn't backfire.
One of the fundamental laws of performance. Always profile.

Another thing I could do to speed up the topic page is to avoid interpolating the post IDs into so many spots like the buttons and just use a <form> instead with the post ID in one place.

 

[Elli]

This is very cool! Especially that it's written in Go, which I truthfully don't know much about, but have heard a bit about here and there from other developers who seem quite excited about it.

I have to ask, though: why jQuery on a custom software? Is it an interim solution, or is it running a dependency?

 

[Azareal]

I've done a few things since my last post and I'll probably push this and that in a commit soon, so people can play with it.

One of these is having Gosora spit out a new log file pair when it starts up rather than using the previous one. Putting everything in one log file resulted in it eventually growing ridiculously big (e.g. 50MB / 200MB) which in turn made Gosora slow down (likely because the OS was paging the log file in and out of memory *just in case* I decided to access earlier portions of it).

This will also have the benefit of simplifying the update process, as we won't run the risk of one log file ending up with different permissions from Gosora itself when running under a "paranoid setup".

I'm also doing a bit more work on the Search and Filter Widget like figuring out what styling I'm going to use exactly for it. I have some ideas in mind, but we'll see what comes out of it. It will naturally use AJAX, so the moment you select a forum or enter a search query, it'll swap the results in on the left.

I will have to see how this feature interacts with widget docks however, as you might want different widgets for the forum page than with the topics list. Edge cases, always a damper on one's mood lol

This is very cool! Especially that it's written in Go, which I truthfully don't know much about, but have heard a bit about here and there from other developers who seem quite excited about it.

I have to ask, though: why jQuery on a custom software? Is it an interim solution, or is it running a dependency?

It helps simplify some things over just writing pure JavaScript. Convenience, I guess.
It does block the page render though, so I'll see what I can do about it eventually, I have a trick to move some initialisation work before it though.

Also, the WYSIWYG Editor Trumboyg relies on it in Cosora Theme, although that one is not quite the style I'd like for Nox Theme, so I'll see what editor looks nice for that.

And Go is always fun, some say it's the best language for web development in 2019.

Easy deployment (I push the boundaries a little with Gosora, so it's probably not as *elegant* as one would usually see), great performance, security audited by security experts at Google, you can even theoretically use it on the client-side, etc.

It's also very good for concurrent applications which is probably what everything should be these days and doesn't involve the "dance" of countless caches, third party processes, etc. which you need to get similar out-of the-box functionality in other languages for things like scheduled tasks, storing things in persistent memory across requests, etc.

There is plenty more as-well.