For anyone who may find this topic when looking for the answer to this problem, if you see your pages in Google's search index with no meta description and with the message from Google saying that the description isn't available because of your site's robots.txt file, here's the fix:
Generally this is because you accientally checked "Hide this page from search engines" in the SEO settings of a specific page, or for your entire site.
On the page level in the Weebly editor click into "SEO Settings" and scroll to the bottom to find the check box:
Uncheck it, and then republish your site.
For your entire site, go into the "Settings" menu, click "SEO" and then scroll down until you see that toggle:
These options in the Weebly Editor allow you to block your page or your site from search engine crawling via your site's robots.txt file (yourdomain.com/robots.txt) ... when you have both toggled off, your site is available for crawling by search engine spiders, and all of the information on them is available.
As for the other blocked resources (ajax, etc.), as the moderator said these are back-end scripts that you don't need to worry about, handled by Weebly. In some themes there won't be an issue with blocked resources by Google, so if it bothers you then your best option may be to choose a different theme.
The reasons why you might want to block a webpage from search engines vary, but might include:
- It's a template page that you've created with place-holder content, to make it very easy to make a copy and add content for landing pages for different products or guides. There's no real content on the place-holder page, so you don't want it crawled and indexed.
- It's a page that you're still working on but have not yet finished, but you need to push changes you've made to another page on your website live by publishin the site.
- It's a page that you need to have (for example privacy policy or advertising policy), but that you don't necessarily want people entering your site on (they can find it via site-search or in the site's footer).
- It's an old page without much content and it's creating a duplicate content warning with other thin pages on your website.
In addition to using these built-in toggle features to block your webpage from being crawled via robots.txt, if you'd like to make sure that it is not indexed by search engines at all, you can do that by adding a meta tag to the header of the page in the page's SEO Settings. This tag would be:
<meta name="robots" content="noindex, nofollow">
Mos has a good guide about the Meta Robots tag here.
Hope this is helpful to the next person with this issue!
I'm apparently having a similar problem with a robot.txt file - and cannot find it. It appears to be blocking Google's access to my first page, which is where my Nabble forum is embedded, so Google isn't finding the content of the forum, but does all the other pages of the site.
I've checked the SEO settings and I do not have the page hidden from search engines. I've checked the header and footer code, to no avail. But here's what Google's Analytics say. What am I missing?
Well, I now appear to have another problem somewhat along these lines. And, while I'm not sure this is the place to post it, I'll try here and y'all tell me if not.
In using Google Analytics I've realized that about a month ago I started getting "blocked resources" messages. The graph is below and it shows that I'm now up to 273 pages with blocked resources. And, when I delve deeper it always says the problem is this statement in the robots.txt file: Disallow: /ajax/
Is there really a problem? If not, why did Google start seeing it as a problem a month ago when most of those pages have been static for a year or two? Who changed what? And, what can I do about it?
Thanks,
Gary
