How Discord stores billions of messages

[Azareal]

https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7
It's always fun to read Discord's engineering blog and the things they're doing to get the service to scale.

This one involves their database, Cassandra.
I've never actually used Cassandra, to be honest, but it seems particularly useful when you have dozens of database servers.

 

[Page]

Keeping messages is just wards on a white paper and so one paper is nothing or even just one discord server that has 20 members with active discussions will have used up 50 papers. So 50 papers adds up to 900kb maybe?

But once you added all servers together and all white papers of wards together it adds up allot, 50GB, 100GB and whatnot. So how long this will go on and will it ever be saved and keeped

 

[Azareal]

Keeping messages is just wards on a white paper and so one paper is nothing or even just one discord server that has 20 members with active discussions will have used up 50 papers. So 50 papers adds up to 900kb maybe?

But once you added all servers together and all white papers of wards together it adds up allot, 50GB, 100GB and whatnot. So how long this will go on and will it ever be saved and keeped

If this was YouTube, then maybe it would be problematic, but 100GB is essentially toilet paper.
Modern hard drives can hold many terabytes and servers cost nothing compared to the thirty programmers.

And these aren't cheap intern quality XenForo / MyBB / Wordpress plugin developer grade programmers, but actual professionals. They are not cheap. But then again, that is the power of venture capital.

I am curious what they're doing now though, this article was written quite a while ago.

 

[Cam]

It's incredible to read how much thought goes into selecting a database. It's also interesting to see all the different platforms that use Cassandra for their services.

 

[Carlos X]

Modern hard drives can hold many terabytes and servers cost nothing compared to the thirty programmers.

You have this backwards. Servers cost more than programmers. Let's talk about scale: To hold 20 Million gamers on a Call of Duty server is 100 Million. You have hundreds of instances every waking hour, in every waking day. This is across all game modes. And each one has their own "market." Right now it's 500k each game mode. Peak times, or days it's in millions of players logging in. 30 Programmers is nothing to the C.E.O. Now, try 100 to 200 contracts (programmers), now you'll be sweating the C.E.O.

Each programmer is contracted at 10k to 20k to 50k depending on the job, how well he does the job, and/or his expertise in the field.

And these aren't cheap intern quality XenForo / MyBB / Wordpress plugin developer grade programmers, but actual professionals. They are not cheap. But then again, that is the power of venture capital.

That's a good thing. You could even take your little project that you're doing (your forum software), ask for capital, and get your own team. Yes! That's what venture capital is for - creating opportunities for you - the next generation of coders!

It's not easy, but you can do it if you put your mind to it.

I am curious what they're doing now though, this article was written quite a while ago.

They're probably doing custom databases or coding, much like Facebook.

 

[Azareal]

You have this backwards. Servers cost more than programmers. Let's talk about scale: To hold 20 Million gamers on a Call of Duty server is 100 Million. You have hundreds of instances every waking hour, in every waking day. This is across all game modes. And each one has their own "market." Right now it's 500k each game mode. Peak times, or days it's in millions of players logging in. 30 Programmers is nothing to the C.E.O. Now, try 100 to 200 contracts (programmers), now you'll be sweating the C.E.O.

Game servers require significantly more resources than ferrying text messages back and forth.

There's an ad server which uses Go's FastHTTP library to serve a quarter of a million requests per second. That's not even breaking out C++.

 

[Carlos X]

You have this backwards. Servers cost more than programmers. Let's talk about scale: To hold 20 Million gamers on a Call of Duty server is 100 Million. You have hundreds of instances every waking hour, in every waking day. This is across all game modes. And each one has their own "market." Right now it's 500k each game mode. Peak times, or days it's in millions of players logging in. 30 Programmers is nothing to the C.E.O. Now, try 100 to 200 contracts (programmers), now you'll be sweating the C.E.O.

Game servers require significantly more resources than ferrying text messages back and forth.

A single HTTP server can service a quarter of a million requests per second.

Have you been to a busy Discord? Have you seen the amount of messages that gets posted every second?

Let me put it this way; the amount of traffic a site gets adds to the overall 'server space' so all those billions of messages, and millions of users are added to the final total of packet size, so discord would have to either cut corners, or buy a larger server, which venture capital provides.

120 million messages sent per day? That's not toilet paper. That's a lot of gigs or terabytes of data every month.

 

[Azareal]

Have you been to a busy Discord? Have you seen the amount of messages that gets posted every second?

Let me put it this way; the amount of traffic a site gets adds to the overall 'server space' so all those billions of messages, and millions of users are added to the final total of packet size, so discord would have to either cut corners, or buy a larger server, which venture capital provides.

Mail.ru used Go to service a million WebSockets connections for their mailboxes.
On a single server.

 

[Carlos X]

Have you been to a busy Discord? Have you seen the amount of messages that gets posted every second?

Let me put it this way; the amount of traffic a site gets adds to the overall 'server space' so all those billions of messages, and millions of users are added to the final total of packet size, so discord would have to either cut corners, or buy a larger server, which venture capital provides.

Mail.ru used Go to service a million WebSockets connections for their mailboxes.
On a single server.

Right, but as of May 2018: 130 million unique users. Billions of messages sent every month. That doesn't sound like easy peasy. You are not looking at the big picture.

 

[Azareal]

Right, but as of May 2018: 130 million unique users. Billions of messages sent every month. That doesn't sound like easy peasy. You are not looking at the big picture.

Even a billion messages per day is a mere 11k requests per second and keep in mind that WebSockets is far lighter weight than HTTP. Any server that can't handle 11k requests per second shouldn't be called a server.

And if you read the article, they did weigh the option of compression, but they weren't terribly concerned. Compression can reduce the size of messages by a factor of ten.

As far as Discord is concerned, I believe they use a mix of Go, C++, Elixir, Erlang, Python, C and JavaScript. There are probably more I missed, they use a lot of languages, they have some pretty good developers.

The reason Facebook has a ridiculous number of servers is because of PHP (getting phased out) and machine learning. They love feeding your data through their algorithms to see what they can find, they would never settle for being a mere intermediary.

I wouldn't sneeze at Discord's budget for servers, but it's nowhere as bad as you think it is.

Adding more programmers isn't really a solution, programmers are expensive and existing programmers would have to bring them up to speed.
That said, Discord seems fairly unconcerned about performance, it takes many, many months before they really deal with performance problems staring them in the face.

 

[Azareal]

They're probably doing custom databases or coding, much like Facebook.

I'm fairly sure they still use Cassandra, it's fairly well-known that they do so among the Discord Community.

 

[Carlos X]

They're probably doing custom databases or coding, much like Facebook.

I'm fairly sure they still use Cassandra, it's fairly well-known that they do so among the Discord Community.

I don't know what Cassandra is, but I read about it in the blog post. But I've marked your post as "informative." 🙂

 

[Azareal]

That's a good thing. You could even take your little project that you're doing (your forum software), ask for capital, and get your own team. Yes! That's what venture capital is for - creating opportunities for you - the next generation of coders!

It's not easy, but you can do it if you put your mind to it.

That is called selling your soul to the devil. Unless it can grab $5 million right off the bat, it's a waste of time and I would basically have to turn around and immediately destroy XenForo and wring cash from their little souls.

Venture capital firms are very greedy people who do want returns.
Returns that are about ten times bigger than what they invested.

There is no money in the forum market anyway, the top players probably make $50 million combined over many years.

 

[Carlos X]

That is called selling your soul to the devil. Unless it can grab $5 million right off the bat, it's a waste of time and I would basically have to turn around and immediately destroy XenForo and wring cash from their little souls.

Venture capital firms are very greedy people who do want returns.
Returns that are about ten times bigger than what they invested.

There is no money in the forum market anyway, the top players probably make $50 million combined over many years.

I don't know why you're always so negative. Investors and Venture Capital firms give you working capital to do what you need to do. Some of them are evil, some of them aren't. It depends squarely on if you met the right investor, or you do a good job.

You think xenForo was started without any capital...? Psft. No. They had money saved up from the Internet Brands employment, and convinced a few people to invest in their company. Otherwise, that lawsuit they went through at the beginning of their startup...? All that money was not from sales. Oh no. That was investor or VC money to burn through.