next.js/examples/with-sitemap-and-robots-express-server/static/robots.txt

mirror of https://github.com/terribleplan/next.js.git synced 2024-01-19 02:48:18 +00:00
User-agent: *
Allow: /a
Allow: /b
Allow: /posts/*
Disallow: /
Disallow: /a/*
Disallow: /b/*
Disallow: /posts

Sitemap: https://sitemap-robots.now.sh/sitemap.xml
sitemap.xml and robots.txt example (#4163) This example app shows you how to set up sitemap.xml and robots.txt files for proper indexing by search engine bots. 2018-05-25 12:01:32 +00:00			`User-agent: *`
			`Allow: /a`
			`Allow: /b`
			`Allow: /posts/*`
			`Disallow: /`
			`Disallow: /a/*`
			`Disallow: /b/*`
			`Disallow: /posts`

			`Sitemap: https://sitemap-robots.now.sh/sitemap.xml`