Gosora Forum Software: Supremely Fast and Feature Packed

[Azareal]

Implemented the registration logs, which logs both successful and attempted registrations, with information about every field that didn't validate. Useful to see where users or bots drop off early on.

I've also fixed some bugs blocking installation, so that should proceed smoothly without trouble and I'll likely throw together a notifications page next to cover more alerts than the alert drop-down (e.g. with pagination), although I have had thoughts of possibility revolutionising the alert interface.

In addition to that, I've refactored the modlogs and friends, which should help ease development in that area and to help phase out legacy coding styles.

 

[Azareal]

I'm prototyping Nox Theme, although it might not make it for v0.1 (0.2 shouldn't be that far off from it tbh) and I'm doing a number of refactors to make Code Climate (a code quality tool) happy.

https://godoc.org/golang.org/x/crypto/argon2
I'm also thinking of adding support for Argon2 (for hashing passwords), maybe keep bcrypt around as an alternative.

 

[Azareal]

You probably suspected it when I added a registration attempt log which collected data on registration attempts, how they failed, etc. but I have added an additional two anti-spam measures, including a JS gate to eliminate 99.99% of spambots.

If you want to go even further than Gosora's capabilities, you can slap down Cloudflare which is so amazingly effective, that I didn't see a single spambot until fairly recently, which was immediately smacked down by my admin activation measures.

To contrast, I've seen sites which were hidden away get completely wrecked with one spam post per minute without an adequate spam defence, this was not Gosora but an incomplete build of MyBB.

I've also dramatically revamped the setting manager to make it far more refined, although that's not all that I have in mind there.

In addition to that, I have begun work on the Nox Theme, although it is currently hidden until you flip a certain flag in the theme.json file, added the ability to do bulk moderation with the Shadow Theme (it was always in Cosora), fixed many bugs and cleaned up a great number of things to help pave the way for the immediate future.

As always, you can update by running the update script, or patcher, if you prefer to manually run `git pull origin`

 

[Azareal]

I'm extremely depressed, so I somehow winded up writing 1600 lines of code in four days, or so Github says. Anyway, I've added a page manager, so you can create, edit and delete custom pages in the Control Panel without going through the old laborious process.

I also made a lot of progress on the Nox theme, began work on two-factor authentication to comply with the General Data Protection Regulation (GDPR), fixed bugs and refactored a huge amount of code (not finished there, this is a checkpoint commit).

 

[Azareal]

Not depressed anymore, yay. Well, a lot less so, I seem normal.

I've implemented two-factor authentication for all users, added a new account dashboard, made big advances in the Nox theme, switched from using config.go to using config.json as it's a little more intuitive for users, fixed a bunch of bugs, and refactored a load of things.

 

[Azareal]

While I work on a live topic list (don't worry, you can just turn it off and it won't have any impact on performance or otherwise), I'll share some juicy data I dug up. These are just casual benchmarks on my computer, but they're remarkably stable (give or take 10% per run).

I wanted to benchmark NodeBB, but I couldn't get Redis to work onto Windows and I don't want to fiddle around with that on my VPS right now, I'll see if I can get that together at a later date.

The one I'll compare Gosora to today is MyBB which claims to be one of the fastest forum software, according to it's official site and one of the giants of the industry, I'm sure we all know who they are.

My computer is a tad slow, so things might be a little on the lower end, but I did throw MyBB onto a SSD for fairness. If you have any ideas of how to optimise MyBB to make it run faster, please give me a holler.

Gosora:
1107 req/s HTTPS Topic List Index (full list, 25 topics)
1275 req/s HTTP Topic List Index (full list, 25 topics)
4448 req/s HTTP Forum List (7 forums)
2161 req/s HTTP Topic #2 (6 posts)
2099 req/s HTTP Redirect

MyBB w/ APC
105 req/s - HTTP Forum List Index (1 forum)

MyBB w/o APC:
15 req/s - HTTP Forum List Index (1 forum)

HTTP and HTTPS speeds are remarkably similar with a 10% deviation, but you do have to keep in mind that I'm pounding it with a benchmark tool called Bombardier which doesn't bring the most out HTTP/2's ability to serve many files over one connection.

For things like topics, etc. MyBB clocks out at about 0.1 req/s, this is likely because the index is being served out of the memory cache APC (it was likely optimised to the point of being almost static due to how heavily that route is hit), while those pages involve round-trips to the database.

Also, fun fact. You can change the index in Gosora from the topic list to the forum list via config.json. More benchmarks coming very soon.

 

[Azareal]

The Live Topic List works now, although only on the Cosora Theme (default) and I've also added the Nox Theme to the theme selector at the bottom of the page, although it's still under construction.

 

[Cosmic]

Written in Golang? This is very cool. Nice project. Keep it up!

 

[Azareal]

I'm adding some of the grind-ier features like proxy IP support, even better anti-spam and more progress on Nox (who would have thought that a theme would take so long).

As far as the spambots are concerned, a few rodents seem to have found some weaknesses, so I'm treating them like lab rats by tweaking heuristics and algorithms to see which work best without shaking the boat too much yet. They're very curious creatures.

One keeps ramming it's head into the error page trying to enter, but it's thwarted by... Having a weak password, so it keeps receiving an error there. I require numbers, uppercase letters, lowercase, etc. in passwords.

Another keeps trying the same username but changes a single character over and over. I setup a heuristic which locked it out, but it keeps trying.
The third one is a little trickier, but I have a couple mitigations in the pipeline to deal with it.

They all ram through classical captchas, probably humans filling them out.
Exactly one appears per day, probably part of the same botnet.

Written in Golang? This is very cool. Nice project. Keep it up!

Thank you very much 🙂

 

[Azareal]

Rolled the update a few days ago and now the little machines are tangled on my net. Obfuscating the scripts does wonders, plus some other little tricks. All without inconveniencing the user with puzzles or distorted images.

I'll monitor them and see how things go however. With that out of the way for now, I'll return to the other features which need my attention.

 

[Azareal]

The Nox Theme, it's a big work in progress though, so is subject to big changes.

https://gosora-project.com/topic/why-are-the-templates-so-fast.49 Also, if you're curious as to the history of my template engine and sort of how it works, I have this.

 

[edenwax]

Wow this is actually really cool! I had never heard of GoLang before now and I'm definitely looking into it now.

Word of advice, don't use Internet Explorer, it does not support CSS Variables.

I mean that's just common sense right there! 😛

 

[Azareal]

I'm going to see if I can start breaking future commits up more now, as sometimes they are huge and sometimes they are tiny.
For instance, I've had single commits after weeks of silence where half of the software gets rewritten, five features get implemented, and the UI gets revamped.

In more progress related news, I'm working on the bulk moderation UI for Nox now and I've improved the handling of database connection drops, so it doesn't log an error for each request until the database comes back up.

Instead, it will log the time the database went down and the time it came back up, it's currently done in `logs/ops.log` but maybe I can add some sort of chart or something.

If anyone's worried about it having an impact on performance, it's just an atomic load for an int32 and a single conditional after /static/, so it should be fine.

Wow this is actually really cool! I had never heard of GoLang before now and I'm definitely looking into it now.

That's great. Go is very easy to setup just about anywhere, in fact @Rick Ace managed to get Gosora working on shared, although I'd recommend a VPS for now (~$3/month).

I mean that's just common sense right there! 😛

Mm, it is a little dated, unfortunately some people still use it, although I'm pushing the requirements even higher, so more browsers might break, but just about anything modern should do fine.

The other one which is really far behind is UCBrowser, it's apparently very popular in China, we will have to see what we can do about that, hopefully they'll just implement the more modern features and I won't have to worry too much about that.

 

[Deleted member 5665]

Interesting design on how you make this software so far. It seems similar to bootstrap, but I assume it isn't based what I have seen from the source.

I am curious what your inspiration may have been to make it? The first appearance automatically made me think of Reddit, but then again I also thought of something else I forgot what it was called. I hope to see more of it in the future. 🙂 I also will send you a PM on something you may be interested in.

 

[Azareal]

It seems similar to bootstrap, but I assume it isn't based what I have seen from the source.

It is wholly custom. All the JavaScript (minus the libraries), CSS, back-end, etc.
There are no CSS frameworks, although RRL's Kana did suggest using Bootstrap as I stumbled in a few places with the styling along the way before getting my bearings better.

The first appearance automatically made me think of Reddit

I was trying to go for something original and interesting, I liked how the original theme didn't squish tall avatars up or otherwise distort them, it was kind of like an interesting side background.

The newer themes are slightly less creative, but more polished in a way and part of a series of compromises to get more functionality into the thing (it initially didn't have the last replyer in the topic list) to make it more usable.

The quick topic feature was inspired by the Steam Forums, they work in a similar way, XenForo gave an idea for the click in text-field one in Cosora, but that one isn't in the older or newest themes.

It's also inspired by the new age software like Discourse, NodeBB, and Flarum. Although, I take exception at their emphasis on minimalism and lack of experimentation.

Experimentation is the best part of building something to figure out new interesting things I can do, if it's not interesting, then what's the point lol
I'm always open to new ideas, UI-wise, code-wise, etc. and I'm always looking at other things to see what I can do better.

Absorb new knowledge and build greater and better things. Survey many fields and look for inspiration there. Learn new programming languages, look at how sites are done in other cultures, and explore new areas. Always be moving forward and not looking back on the old days.

It's that sort of philosophy. If you have any ideas for features, improvements, and what-not, then they're welcome 🙂

I am curious what your inspiration may have been to make it?

There are several reasons.

One was seeing the major real world scalability issues in MyBB and how RRL ultimately solved it by going custom with C#. I know an owner there and it was horrible, every-day the cache would crash and MyBB would start spewing out errors all over the page and they had three dedicated servers constantly on brink of collapsing from the high server load.

Ever since then, they've been free and soared into the sky past the traffic of those 60 million post sites.
This was like a wake-up call. Modern forum software are simply not scalable or performant. At all.

But that didn't make me decide to write Gosora. At the time, I said something along the lines of... Well, surely XenForo / IPB / phpBB / etc. are better optimised. And well, they apparently suffered from the same sort of problems, particularly with the edge case they hit which was having too many forums.

A couple of years after that, I was fairly curious about the Go Programming Language and casually wrote a little bug tracker in it. Nothing elaborate or complex, just the basics.

And it was a surprisingly easy to use, highly performant, etc. language, so I decided to rewrite that into a forum software, possibly to use in a site somewhere and to get some more practice in, and so, I experimented, tweaked, tumbled, and created this thing.

 

[Deleted member 5665]

Thanks for answering my questions. Looks very interesting may download it and try it out. 🙂

 

[Azareal]

I'm trying to move some of the load from the server to the client (other than the first load which out of necessity needs to be done on there or the performance on mobile will go down dramatically).

Client rendering to some extent is important, as the end-user might be on a poor mobile connection, be on the other side of the world, or the origin server might be overloaded / have a saturated connection.
However, client rendering also increases CPU / memory usage which leads to the battery being drained faster (in Discourse's case, it can lead to severe lag on anything but the newest iPhone).

There is also the issue with advertisements when relying too heavy on AJAX loading, as advertisements (except for Google who support AJAX loading) sometimes mess with document.write and the like in ways in which things are prevented from loading properly.

To rectify this, I'm experimenting with serving transpiled templates (depending on the size of the template, they seem to vary from 20 to a hundred lines, I've tried to avoid serving the really complex ones), so that the software can avoid full loads in some areas and to load important pages before the user needs them in others.

However, to handle the situation with the ads, I have introduced a new setting called "Fast Loading" which does more aggressive optimisations where possible at the expense of the ads.

The toughest part is going to be dealing with underpowered phones which have little capability to do anything, I have some friends afflicted by those sorts of problems, and it would be nice if things moved as smoothly as possible there too.

 

[Azareal]

I added some new installation instructions revolving around setting up a systemd service (you'll need root on a VPS / dedicated server for this), I'll also see about adding something which doesn't require root there.

 

[Azareal]

Closed off a couple of potential timing attacks, added some notes in the installation instructions to try to stop admins from making mistakes which may compromise their security, and hardened a few things.

I also began rolling out some changes which automatically re-encode avatars in a far more compact form to help save bandwidth and to make pages load faster on mobile.

 

[Azareal]

I've delayed v0.1's deadline to the end of August. It was supposed to be on my birthday and all that, but well, not ready. The thing should be fairly stable and secure to service your needs, although plugin developers / theme developers / people who like modifying things might find things moving out from under them on a regular basis.

I've also started playing an addictive little game with Code Climate again where it gives you little grades and stats whenever you improve the quality of your code or reduce the number of bugs in Gosora, so that's fun.

It's a sort of hardcore game though where it's never happy, but it's always nice to see the little stats go up.