TYPO3 v9.5.0 - Error message: Requested page does not exist /robots.txt

Question

TYPO3 v9.5.0 - Error message: Requested page does not exist /robots.txt

I have a TYPO3 9.5.0LTS and use the bootstrap package theme. It seems to be all working ... but quite often I get such error messages:

Core: Exception handler (WEB): Uncaught TYPO3 Exception: #1518472189: The requested page does not exist | TYPO3\CMS\Core\Error\Http\PageNotFoundException thrown in file /is/www/typo3_src-9.5.0/typo3/sysext/frontend/Classes/Controller/ErrorController.php in line 82. Requested URL: domain/robots.txt

What causes this and how to prevent this? Or how do I create a robots.txt in v.9.5 ?

Answer 1

In TYPO3 9.5 you can add a robots.txt in your Sites module.

Sites -> Choose your site -> Static Routes -> Create new.

Static Route Name: select "robots.txt"
Route Type: select "Static Text"
Static Text: Select "robots.txt Example Content"

Save. Should be fixed now.

Answer 2

This will work for all TYPO3 versions. For TYPO3 V9.x use the solution by Thomas Löffler.

Your server configuration (apache? .htaccess?) will hand over any request to a source that is no file and no directory and no symbolic link to the index.php file which is TYPO3.

In your case, you do not have a file robots.txt. So TYPO3 wants to handle it, but has no resource with that name. This creates a 404 error in TYPO3.

To prevent this, jst create the robots.txt file on your webserver in the DOCUMENT_ROOT folder

So what is a robots.txt file anyway.
This is a method to tell search engines how to behave on your server. It contains recomendations to the search engines' crawlers, when to stop crawling (like typo3_src folder). It is requested by the crawlers automatically and regularly.