x

How can I modify the robots.txt file of my Weebly site?

How can I modify the robots.txt file of my Weebly site?
I’m trying to get my site indexed by Google, but I get the coverage warning of “Indexed, though blocked by robots.txt”. I had previously already unchecked the ‘hide site from search engines’ button (in Settings > SEO), but even after checking and unchecking it to test it, the robots.txt file still says:
      “
        Sitemap: https://maprun-parks-qld.weebly.com/sitemap.xml

        User-agent: NerdyBot
        Disallow: /


        User-agent: *
        Disallow: /ajax/
        Disallow: /apps/
      “

I understand that I need to get it to say

        Sitemap: https://maprun-parks-qld.weebly.com/sitemap.xml
        User-agent: *
        Disallow:

for full crawler access, but how else can I change it?

1,707 Views
Message 1 of 4
Report
3 REPLIES 3

@gabrielle 

Take a look at this thread which you should find helpful...

https://community.weebly.com/t5/SEO/Robots-txt/td-p/4760

1,700 Views
Message 2 of 4
Report

Thanks @NJRFTF , but it doesn’t help.

I have already submitted my sitemap to Google Web Master tools, none of my pages are hidden from search engines, and the Settings > SEO hide from search engines button is definitely not enabled.

Google Search Console gives the following error for all pages:
     “Indexed, though blocked by robots.txt”

And it recommends
     “If you do not want to block this page, update your robots.txt file to unblock your page.”

What else can I try?

1,684 Views
Message 2 of 4
Report
Square

All Weebly sites block nerdybot so it sounds like the error you see is a common one. Here is another thread with some more information that will hopefully clear up your question. https://community.weebly.com/t5/SEO/Blocked-Resources-and-robot-txt/td-p/1856

1,678 Views
Message 2 of 4
Report