Allow access to robots.txt in Azure Web application Firewall

Question

I have a public facing angular site hosted is AKS which is behind Azure WAF. But the WAF is blocking calls to robots.txt as a result the google search engine crawler is not indexing the site.

I don't see an option to exclude the policy based on URL. It can only be done based on Headers, Query params and Arguments.

How can I configure WAF to allow access to robots.txt

Answer 1

You can configure an exclusion list for the Azure Application Gateway Web Application Firewall (WAF) to allow access to robots.txt. The exclusion list allows you to omit certain request attributes from a WAF evaluation. To do this, you can create a per-URI policy to exclude the robots.txt URL path. You can follow the instructions in the "Web Application Firewall exclusion lists" document to create the exclusion list. Once you have created the exclusion list, you can add the robots.txt URL path to the exclusion list.

Here are the steps to create a per-URI policy:

1. In the Azure portal, navigate to your WAF policy.
2. Select Rules engine, then select Custom rules.
3. Select Add.
4. In the Add custom rule page, enter a name for the rule.
5. Under Match conditions, select String

6. In the match values put the URI path, and set other parameters accordingly
7. Select Add.
8. Select Save.

After you have created the custom-URI policy, you can associate it with your Application Gateway. You can follow the instructions in the "Create Web Application Firewall policies for Application Gateway" document to associate the policy with your Application Gateway.

References:

• Web Application Firewall exclusion lists
• Create Web Application Firewall policies for Application Gateway