Submitting to Search Engines

[Gizmo]

So I have a Xenforo forum and have added a robots.txt file with this in it:

User-agent: PetalBot User-agent: AspiegelBot User-agent: AhrefsBot User-agent: SemrushBot User-agent: SemRush User-agent: DotBot User-agent: MauiBot User-agent: MJ12bot Disallow: / User-agent: Amazonbot Disallow: /threads/*/reply User-agent: * Disallow: /whats-new/ Disallow: /account/ Disallow: /members/ Disallow: /attachments/ Disallow: /goto/ Disallow: /help/ Disallow: /posts/ Disallow: /login/ Disallow: /search/ Disallow: /admin.php Allow: / Sitemap: https://mediaplanetforum.com/sitemap.xml

Is there anything else I need to do to get the search engines to add links? Or is this all these days?

 

[Page]

I have not blocked any bots on my end I mean no idea why but then at the same time doesn't make any difference to me anyway. But you can add more over time 🙂

Since you running XF not all will be workable but you get the idea...

# Disallow directory
Disallow: /api/
Disallow: /applications/
Disallow: /cgi-bin/
Disallow: /datastore/
Disallow: /plugins/
Disallow: /system/

# Block pages with no unique content
Disallow: /startTopic/
Disallow: /*?do=add
Disallow: /*?do=submit
Disallow: /discover/unread/
Disallow: /markallread/
Disallow: /staff/
Disallow: /online/
Disallow: /discover/
Disallow: /search/
Disallow: /*?advancedSearchForm=
Disallow: /register/
Disallow: /lostpassword/
Disallow: /login/

# Block faceted pages and 301 redirect pages
Disallow: /*?sortby=
Disallow: /*?filter=
Disallow: /*?tab=comments
Disallow: /*?do=email
Disallow: /*?do=findComment
Disallow: /*?do=getLastComment
Disallow: /*?do=getNewComment

#Disallow files
Disallow: /403error.php
Disallow: /404error.php
Disallow: /500error.php
Disallow: /Credits.txt
Disallow: /error.php
Disallow: /upgrading.html

 

[Gizmo]

I have not blocked any bots on my end I mean no idea why but then at the same time doesn't make any difference to me anyway. But you can add more over time 🙂

Since you running XF not all will be workable but you get the idea...

Yeah no most if not all of those won't do anything 😛 but thanks.

 

[DaniWeb]

It is recommended to block faceted pages such as:

Disallow: /*?sortby=
Disallow: /*?filter=

Ideally, you don't want Googlebot wasting your site's crawl budget crawling the same thread listing pages sorted and filtered all different ways, or crawling search result pages. With large forums (e.g. 500,000+ threads), you might end up having crawl budget wasted with googlebot spending all its time going down a rabbit hole of duplicate/thin content, and never discovering a large portion of your quality content.

 

[Nomad]

After making these changes, has your site been indexed on search engines? Do search engines show your site?