Remove Pages from XML Sitemap and rebuilding index [solved]

**Justin** · 01/21/2010, 20:15

I have a "News" archive that I want to link directly to but do not want it to be searchable on both my site and in search engines. It was easy to make it not searchable within my typolight site by clicking the setting for the news reader page to "Do not search".

How do I make it so it's not being indexed when I rebuild my index? Is it possible to have the news not be archived? I was thinking choosing "noindex, nofollow" under the site structure for the news reader page would not include the news articles in the index, but this is not the case.

Thanks

**Justin** · 01/21/2010, 20:21

I was also wondering if anyone knows how the Robots tags work in Typolight? I assumed choosing "noindex" would take the page out of the index when you rebuilt your index under maintenance. This doesn't seem to be the case however

Site Structure > Regular Page > Robots Tag
- index, follow
- index, nofollow
- noindex, follow
- noindex, nofollow

**Medianomaly** · 01/21/2010, 22:05

I believe these are 2 different things --

The robots.txt file tells search engines like Google how to treat your site -- if you don't want Google to index your site (or a page tree) you'd choose "noindex, nofollow". Google looks for a robots.txt file in your site root. But I don't think it affects how TL treats your site and builds its index.

Look for separate options to allowing searching, and also an exclude page from navigation option if that is in fact what you want.

**Justin** · 01/21/2010, 22:14

Medianomaly, thanks for clarifying. So, you've answered how the noindex, nofollows works with the robots.txt file.

My issue is, when you go and "rebuild index" in typolight, it adds all the pages. So if you have a page you don't want added to your index or your sitemap.xml file, there is no way to not have it show up. It seems weird that I would not want a page included in the robot.txt but would then archive it in my sitemap.xml file? Does that make sense? Does it matter?