How to disallow nested folders with robots.txt?

37 views Asked by At

In my robots.txt I have a disallowing rule like:

User-agent: *
Disallow: /_hcms/

I want to disallow anything placed inside of _hcms and in its nested folders, like /_hcms/a/ or /_hcms/b/.

Should I add a disallowing rule for such subfolders, like

User-agent: *
Disallow: /_hcms/
Disallow: /_hcms/a/
Disallow: /_hcms/b/

Or is the existing rule Disallow: /_hcms/ makes subfolders disallowed? Or should it even be Disallow: /_hcms, without closing slash?

1

There are 1 answers

0
Sahil Ahmad On BEST ANSWER

You can use the existing rule Disallow: /_hcms/ to disallow anything placed inside of _hcms and in its nested folders. The trailing slash is optional and will not affect the behavior of the rule. Therefore, you do not need to add additional rules for subfolders.