Changeset 33550

Timestamp:

2019-10-04T19:06:51+13:00 (5 years ago)

Author:

ak19

Message:

First stage of introducing sites-too-big-to-exhaustively-crawl.tx: split url-greylist-filter.txt into true greylisted sites (product sites so far) and the existing top sites urls that simply represent sites too big to crawl in entirety.

Location:

gs3-extensions/maori-lang-detection/conf

Files:

• sites-too-big-to-exhaustively-crawl.txt (added)
• url-greylist-filter.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/conf/url-greylist-filter.txt

@@ -13 +13 @@
 /product-page/
 /product-category/
-
-# Add alexa top sites to greylist
-
-youtube.com
-tmall.com
-baidu.com
-qq.com
-sohu.com
-facebook.com
-taobao.com
-#login.tmall.com
-wikipedia.org
-yahoo.com
-360.cn
-jd.com
-amazon.com
-Sina.com.cn
-weibo.com
-#pages.tmall.com
-live.com
-vk.com
-netflix.com
-alipay.com
-office.com
-okezone.com
-csdn.net
-instagram.com
-xinhuanet.com
-babytree.com
-twitter.com
-ebay.com
-stackoverflow.com
-naver.com
-aliexpress.com
-twitch.tv
-tribunnews.com
-apple.com
-soso.com
-tianya.cn
-microsoftonline.com
-yandex.ru
-
-# Remaining top sites from https://en.wikipedia.org/wiki/List_of_most_popular_websites
-
-ok.ru
-paypal.com
-t.co
-pinterest.com
-sogou.com
-espn.com
-walmart.com
-bitly.com
-ampproject.org
-sm.cn
-
-
-
-# UNSURE - what if these contain translated pages?
-google.com
-bing.com
-amazon.co
-msn.com
-microsoft.com
-accuweather.com
-
-#nasa.gov
-# w3schools.com
-# quora.com
-#reddit.com
-#blogspot.com
-#yahoo.co.
-
-
-## TODO: Get more from https://moz.com/top500