KarlHeinzSchwuke@feddit.org to Technology@lemmy.worldEnglish · 22 days agoI was wrong about robots.txtevgeniipendragon.comexternal-linkmessage-square22linkfedilinkarrow-up192arrow-down116
arrow-up176arrow-down1external-linkI was wrong about robots.txtevgeniipendragon.comKarlHeinzSchwuke@feddit.org to Technology@lemmy.worldEnglish · 22 days agomessage-square22linkfedilink
minus-squareGeneral_Effort@lemmy.worldlinkfedilinkEnglisharrow-up82arrow-down3·22 days agoWhat did he think a crawler is? Why was he surprised that not allowing companies to use his data lead to them not using his data? Looks like he has another surprise coming when he notices that search engines no longer index his blog.
minus-squareArchr@lemmy.worldlinkfedilinkEnglisharrow-up17arrow-down1·edit-222 days agoI feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article. Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.
minus-squareGeneral_Effort@lemmy.worldlinkfedilinkEnglisharrow-up7·22 days ago that is not how general news media has been talking about robots.txt. Ahh, yes. I think there is a lesson there.
What did he think a crawler is? Why was he surprised that not allowing companies to use his data lead to them not using his data? Looks like he has another surprise coming when he notices that search engines no longer index his blog.
I feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article.
Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.
Ahh, yes. I think there is a lesson there.