Optional@lemmy.world to Technology@lemmy.worldEnglish · 29 days agoGoogle is no longer asking — feed the AI or you’re not in search resultspivot-to-ai.comexternal-linkmessage-square102fedilinkarrow-up1534arrow-down110file-textcross-posted to: techtakes@awful.systems
arrow-up1524arrow-down1external-linkGoogle is no longer asking — feed the AI or you’re not in search resultspivot-to-ai.comOptional@lemmy.world to Technology@lemmy.worldEnglish · 29 days agomessage-square102fedilinkfile-textcross-posted to: techtakes@awful.systems
minus-squareColinHayhurst@lemmy.worldlinkfedilinkEnglisharrow-up29·edit-228 days agoYou should put these entries into your robots.txt file. To block the Google search crawler use for all of your site: User-agent: Googlebot Disallow: / To block the Google AI crawler use: User-agent: Google-Advanced Disallow: /
minus-squareJamyang@lemmy.worldlinkfedilinkEnglisharrow-up2·28 days agoWhat if I made a static site using Github pages hosting? Will having a robots.txt in my root folder ward off Google bhoots (devils)?
You should put these entries into your robots.txt file.
To block the Google search crawler use for all of your site:
User-agent: Googlebot
Disallow: /
To block the Google AI crawler use:
User-agent: Google-Advanced
Disallow: /
You rock, thank you!
What if I made a static site using Github pages hosting? Will having a robots.txt in my root folder ward off Google bhoots (devils)?
Yes.