Use a free CDN like CloudFlare, minify ALL CSS/JS/HTML files, cache images at lower resolution that they were uploaded with (unless specifically requested at original res./size), setup some type of internal cache for your website to send over commonly requested files that never change, set your Auto TTL is something decent (normally around 2-4 weeks), use a good hosting company to provide you with fast connection speeds/resources, etc. There are tons of things to do, but you always will have those that lie on the outside of really getting fast speeds no matter what you do.