垃圾UA(userAgent)信息

屏蔽垃圾ua,可以大大节省服务器带宽成本,以下是懒人工具管理员多年的总结和收集的毫无意义的垃圾UA爬虫机器人Bot,拦截掉可以让正规的搜索引擎更快速的抓取你的网站内容.
headlessbai.comCrawlerBarkrowlerCakePHP
GarlikCrawlerGo-http-clientias_crawlerICC-CrawlerPotPlayer
RiddlerScrapyWINAMPviz/vizZXing
CastroJakarta Commonsltx71NativeHostSalesIntelligent
Xenu Link SleuthY!J-ASRBUbiNGCRAZYWEBCRAWLERhttp Cnrdn
LavfNSPlayerspray-canstagefrightvoltron
LibVLCA6-Indexercrawler4jwsr-agentDigitalPebble Crawler
MBCrawlerAhrefsBotGrapeshotCrawlerproximicSemrushBot
ahoy!alkalineananzianthillarachnophilia
aralearaneoarethaariadnearks
askjeevesatn worldwideauresysbackrubbig brother
bjaalandblackwidowbloodhoundcalifcassandra
christcrawler.comchurlcienciaficcion.netcmc/0.01collective
combine systemcomputingsite robi/1.0crawler.feedbackcuscocyberspyder link test
katalog/indexdie blinde kuhdiggerdirect hit grabberdownload express
dwcpebinesse-collectoremacs-w3 search engineesculapio
estherevliya celebifastcrawlerfelix idefetchrover
fidofish searchfouineurfreecrawlfunnelweb
gazzgcreepgetterroboplus puugeturlgolem
grapnel/0.01 experimentgriffongromitGlutenh?m?h?kki
harvesthavindexhi (html index) searchhku www octopusht://dig
html_analyzerhtmlgobblehyper-decontextualizeria_archiveribm_planetwide
image.kapsi.netimagelockincywincyinformantinfoseek sidewinder
ingridinktomi slurpinspector webintelliagentinternet shinchakubin
iron33israeli-searchjavabeejcrawlerjumpstation
katipokdd-explorerkilroykit-fireballlabelgrabber
larbinlegslink validatorlinkscanlinkwalker
lockonlogo.gif crawlerlycosmac wwwwormmagpie
marvin/infoseekmattiemediafoxmerzscopemindcrawler
mnogosearch search engine softwaremogetmonstermotormuncher
muninnmuscat ferretmwd.searchnec-meshexplorernederland.zoek
netcarta webmap enginenetmechanicnetscoopnewscan-onlinenhse web forager
nomadnorthern light gullivernzexplorerobjectssearchoccam
OOZBOTopenfind data gathererorb searchpack ratpageboy
parasitepatricpegasusperlcrawler 1.0pgp key agent
phpdigpiltdownmanpioneerplumtreewebaccessorpoppi
popular iconoclastraven searchroadhouse crawling systemrobofoxrobozilla
rulesscootersearch.aus-au.comsearchprocesssenrigan
sg-scoutshagseekersiftsite searchersite valet
sitetech-roverskymob.comslcrawlersleeksnooper
sukesuntek search enginesvensygoltach black widow
tarantulatempletonthe peregrinatorthe web moosethe web wombat
the world wide web wandererthe world wide web wormtitantitinucsd crawl
udmsearchunnamedurl checkvalkyrieverticrawl
victoriavision-searchvoyagerw3m2w3mir
walhello appiewallpaper (alias crawlpaper)web core / rootswebcatcherwebcopy
webfetcherwebinatorweblayersweblinkerweblog monitor
webmirrorwebquestwebreaperwebsnarfwebstolperer
webvacwebwalkwebwalkerwebwatchwebzinger
wgetwhatuseek winonawild ferret web hopperwired digitalwwwc ver
xgetdaumoajoboecho!linkchecker
bloglinestwicelerappiesun4uhttrack
sisirobiwebster prowebsterzeus
sciruspicosearchpluckerdisco pumpgulliver
emailsiphonteleport profetchpamukwebcopier
webcapturemass downloaderawv0.8dcrescent internet toolpakwebstripper
sitesuckerwebduppython-urllibpythonfranklin locator
ck-sillydogpockethttpjavakototoi.orgteragramwebcrawler
vagabondonogoop-httpclientmyoperatbmyoperatbaccoona-ai-agent
arachmob-l-i-t-z-b-o-tboitho.com-dccerberian drtrscharlotte
converacrawlercosmoscovario idsdataparksearchearthcom.info
fast enterprise crawlerfast-webcrawlerfindlinksg2crawlerholmes
htdigiccrawlerichiroigdespyderissuecrawler
l.webislwp-trivialmabontlandmagpie-crawlermnogosearch
mogimogimorning papermvaclientnetresearchservernetseer crawler
newsgatorng-searchnutchcvsnymesisoegp
orbiterpeewpompospostpostpycurl
qseeroradian6sandcrawlersbiderscoutjet
scrubbysearchsightseekbotsemanticdiscoverysensis web crawler
shim-crawlershopwikisnappysqwormstackrambler
teomatineyetruwogpsupdatedvortex
vyu2webcollagewebsquash.comwf84womlpefactory
yacyyahooseekeryahooseeker-testingyandeximagesyandexmetrika
yetiyooglifetchagentzyborgwordpressa6-indexer
wsr-agentMicrosoft OfficeJDatabaseDriverfacebookexternalhitThe+Knowledge+AI
TwitterbotVenusCrawleraria2GetCodeCCBot
NetTrackTurnitinIAS crawlerPOE-ComponentVelenPublicWebCrawler
www.ruNutch Master TestWotboxorion-semantics.comlwp-request
ShortLinkTranslatemj12botWinHttpRequestExabotAuto Spider
DuckDuckGoSeznamBotmoatbotDotBotSurdotlyBot
28logsSpiderzgrabWindows-Media-PlayerspbotMail.RU_Bot
BacklinkSiteExplorerSEOkickslinkdexbotQwantify
DataXuExtLinksBotgvfs/evc-batchCliqzbot
YandexBotYandexMobileBotnewspaperClickagyChicken laser
coccocbotMicrosoft Windows Network Diagnosticsspuhex.comsmtbotDataprovider
HybridBotSky-WapproxySafeDNSBotHatenaBookmarkMeta_Bot
ToutiaoSpiderHttpComponentsips-agentyandex.com/bots(ziva)
JerseyAuto Shell SpiderUser-AgentcurlMPlayer
internal requestGrammarlypackageTrendsmapResolverPaperLiBot
startmebotWebFuckGStreamerhttpsrcAntennaPod
panscient.comwebscanScreaming FrogWFilter Livetrendictionbot
nsrbotPlurkBotMojoliciousAlphaBottracemyfile
VCTestClientheritrixMiniRedirIframelyrest-client
CappuccinoFirmsBotBOT for JCENimbostratus-BotEmacs-w3m
WordupinfoSearchDispatchParacrawlMr.4×3axios
Typhoeustools.randomWhatCMSBotInetURLNetpeakCheckerBot
Gooselua-restyWhatWebspecial_archiverXoviBot
WappalyzerOK-Search-BotabotMechanizeuipbot
GnowitNewsbotPostmanRuntimeHoneyBeegobusterBidtellect
SonosRankingBotUptimebotSynapseRe-re Studio
MappyStatasticoLinguee BotPocketImageCachecolly
YunSecurityBotarchive.org_botCheckMarkNetworkJooblebotZoomBot
LinkbotStreamline3BotLetsearchBotLinguee-BotThither.Direct
Bose/PPBotIndeedBotEveryonedomainsbotPPBot
MixnodeCacheNetpeakSpiderBotTagVisitRestSharpSymfony
NeedlekubectlvuhuvBotStaddlebotddline.cn
AdsrvrContextual_zbotPagePeekerOutclicksBotKozmosbot
PicoFeedMediatoolkitbotnetdiskESP32Traackr.com
DiscordbotPinkBotValidatorSemanticaiHitBot
Zoxh.Comfoobar2000bitlybotbeegoServerMFC_Tear_Sample
QuantcastbotHeiKeManicTimeNewsWindows 95
Windows 98WebPicturesSBL-BOTDreamPassportBlazer
RealMediaLiberate DTVCyberdogFuzz Fasterportalmmm
WannaBebluefishUtopia WebWasherOffline ExplorerVisicom
BarcaANTFrescoHotzonuWfuzzDillo
iSiloXCommerce Browser CenterW3CLineModePandalyticsLinkpadBot
daum.netNewTVGigablastOpenSourceMAZBotpilicanbot
EchoboxBotCincrawScraperBotadmantxAspiegelBot
BDCbotLogStatisticMAZBotCheckHost7Siters
BorneoBotCincrawHuaweiWebCatBotPetalBotZoominfoBot
PinterestbotMojeekBotSeoBotLogStatisticl9explore
FMODStudioNutch SpiderDomainStatsseostaromgili
webprosbotThinkChaosWellKnownPunkspiderDataForSeo
KeybotBaispiderMegaIndex.ruMauiBotBLEXBot
digextMagiBotAdsbotNmapvip0.ru
thetradedeskApache-HttpClienttrendkite-akashic-crawlerphantomjsAmazonbot
semantic-visions.comSkypeUriPreviewserpstatbotPubMatic\xB2\xBB\xCA\xCA\xD3\xC3UA
libcurl-agentNeevabot