Database of robots.txt-derived permissions
This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <https://www.robotstxt.org/robotstxt.html>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.
Release | Stable | Testing |
---|---|---|
Fedora Rawhide | 6.02-30.fc35 | - |
Fedora 35 | 6.02-30.fc35 | - |
Fedora 34 | 6.02-28.fc34 | - |
You can contact the maintainers of this package via email at
perl-WWW-RobotRules dash maintainers at fedoraproject dot org
.