More advanced "no-index" checker

Currently we're only checking the meta tags to see if a page can be indexed or not.

The better more robust solution would be to also validate against the robots.txt file (to see if there is a disallow somewhere) and to check the HTTP x-robots-tag.

Comments

To leave a comment, please authenticate.

No comments yet
Voters
Status
Completed
Board
Feature
Submitted
3 months ago, Hans