Apni website ke liye robots.txt file banayein.
Apni website ke liye robots.txt file banayein.
robots.txt aapki website ke root par rakhi jaane wali chhoti si text file hoti hai, jo search engine crawlers ko batati hai ki site ke kaunse hisse crawl karne hain aur kaunse nahi. Is generator me Allow/Disallow paths, sitemap URL aur optional crawl-delay daalkar aap kuch seconds me ready-to-use file bana sakte hain. Sab kuch aapke browser me hi banta hai — aapka data kahin upload nahi hota.
robots.txt sirf crawling ko guide karti hai. Ye kisi page ko search results se hatane ki guarantee nahi deti, aur private ya sensitive content chhupane ka tarika bhi nahi hai — uske liye password protection ya noindex ka use karein. File hamesha domain ke root par honi chahiye, aur upload ke baad Google Search Console me check kar lein ki wo sahi padhi ja rahi hai.
Website ke root par, yaani https://yourwebsite.com/robots.txt. Kisi subfolder me rakhi file crawlers use nahi karte.
Nahi. robots.txt sirf crawling rokti hai. Page ko search results se hatane ke liye noindex ya password protection use karein. Agar page robots.txt se block hai to crawler noindex tag padh hi nahi paata.
Kuch crawlers, jaise Bing, ise do requests ke beech ka wait-time maante hain. Google is directive ko support nahi karta.