Changeset 33551

Timestamp:

2019-10-04T19:35:06+13:00 (5 years ago)

Author:

ak19

Message:

Added in top 500 urls from moz.com/top500 and removed duplicates, and removed subdomain variants keeping just main site variant, and sorted alphabetically again.

File:

• gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt

@@ -1 +1 @@
 
-# Add alexa top sites to greylist
-# Remaining top sites from https://en.wikipedia.org/wiki/List_of_most_popular_websites
-## TODO: Get more from https://moz.com/top500
+# Add alexa top sites (only 50 visible)
+# Added further top sites from https://en.wikipedia.org/wiki/List_of_most_popular_websites
+## Finally also got the CSV from https://moz.com/top500 and added it to the list and added them in.
+# Then used LibreOffice's Calc spreadsheet software to sort alphabetically and remove duplicates.
+# Then in Gedit, used regex search and replace to remove <subdomain>.<site>.ext to keep just <site>.ext
+# And resorted alphabetically
 
+
+000webhost.com
 360.cn
+4shared.com
+a8.net
+abc.es
+abc.net.au
+abcnews.go.com
+about.com
+about.me
+aboutads.info
+abril.com.br
+academia.edu
 accuweather.com
-amazon.co
-amazon.com
-ampproject.org
+addthis.com
+addtoany.com
+adobe.com
+airbnb.com
+akamaihd.net
+alexa.com
+alibaba.com
 aliexpress.com
 alipay.com
+aljazeera.com
+allaboutcookies.org
+allrecipes.com
+amazon.
+ampproject.org
+android.com
+aol.com
+ap.org
+apache.org
+apachefriends.org
 apple.com
+archive.org
+arstechnica.com
+arxiv.org
+asahi.com
+ask.fm
+asus.com
+axs.com
 babytree.com
 baidu.com
+bandcamp.com
+bbc.co.uk
+bbc.com
+berkeley.edu
+biblegateway.com
+biglobe.ne.jp
+billboard.com
 bing.com
+bit.ly
 bitly.com
+blackberry.com
+blogger.com
 blogspot.com
+bloomberg.com
+booking.com
+box.com
+britannica.com
+bt.com
+bund.de
+businessinsider.com
+businesswire.com
+buydomains.com
+buzzfeed.com
+ca.gov
+cambridge.org
+cbc.ca
+cbsnews.com
+cdc.gov
+change.org
+channel4.com
+chicagotribune.com
+cisco.com
+clickbank.net
+cloudflare.com
+cnbc.com
+cnet.com
+cnn.com
+cocolog-nifty.com
+columbia.edu
+cornell.edu
+corriere.it
+cpanel.com
+cpanel.net
+creativecommons.org
 csdn.net
+csmonitor.com
+dailymail.co.uk
+dailymotion.com
+dan.com
+daum.net
+dell.com
+depositfiles.com
+detik.com
+digg.com
+disney.com
+disqus.com
+doubleclick.net
+dreniq.com
+dribbble.com
+dropbox.com
+dropboxusercontent.com
+dw.com
+e-recht24.de
+ea.com
+ebay.co.uk
 ebay.com
+economist.com
+eff.org
+ehow.com
+elmundo.es
+elpais.com
+engadget.com
+entrepreneur.com
+eonline.com
 espn.com
+espn.go.com
+etsy.com
+europa.eu
+eventbrite.com
+example.com
+excite.co.jp
+express.co.uk
 facebook.com
+fandom.com
+fastcompany.com
+fb.com
+fb.me
+fda.gov
+fedoraproject.org
+feedburner.com
+fifa.com
+files.wordpress.com
+flickr.com
+forbes.com
+fortune.com
+foursquare.com
+foxnews.com
+ft.com
+ftc.gov
+gen.xyz
+geocities.jp
+gesetze-im-internet.de
+ggpht.com
+github.com
+gizmodo.com
+globo.com
+gmail.com
+gnu.org
+godaddy.com
+gofundme.com
+goo.gl
+goo.ne.jp
+goodreads.com
+google.
+googleblog.com
+googleusercontent.com
+gooyaabitemplates.com
+gov.uk
+gravatar.com
+greenpeace.org
+gstatic.com
+guardian.co.uk
+harvard.edu
+hatena.ne.jp
+histats.com
+hm.com
+hollywoodreporter.com
+home.pl
+house.gov
+howstuffworks.com
+hp.com
+huffingtonpost.com
+huffpost.com
+hugedomains.com
+ibm.com
+ibtimes.com
+icann.org
+ieee.org
+ietf.org
+ig.com.br
+ign.com
+ikea.com
+imageshack.us
+imdb.com
+imgur.com
+inc.com
+independent.co.uk
+indiatimes.com
+indiegogo.com
 instagram.com
+intel.com
+issuu.com
+istockphoto.com
+iubenda.com
 jd.com
-google.com
+joomla.org
+jquery.com
+jstor.org
+kickstarter.com
+kinja.com
+last.fm
+latimes.com
+lefigaro.fr
+lemonde.fr
+line.me
+linkedin.com
+list-manage.com
 live.com
+livejournal.com
+livescience.com
+loc.gov
+lycos.com
+mail.ru
+marketwatch.com
+marriott.com
+mashable.com
+mediafire.com
+medium.com
+mega.nz
+mercurynews.com
+merriam-webster.com
+metro.co.uk
 microsoft.com
 microsoftonline.com
+mirror.co.uk
+mit.edu
+mixcloud.com
+mlb.com
+mozilla.com
+mozilla.org
 msn.com
+myspace.com
+mysql.com
+namecheap.com
+narod.ru
+nasa.gov
+nationalgeographic.com
+nature.com
 naver.com
-nasa.gov
+naver.jp
+nbcnews.com
+ndtv.com
 netflix.com
+netsons.com
+netvibes.com
+networkadvertising.org
+news.com.au
+newscientist.com
+newsweek.com
+nginx.com
+nginx.org
+nhk.or.jp
+nicovideo.jp
+nifty.com
+nih.gov
+nikkei.com
+noaa.gov
+nokia.com
+npr.org
+nvidia.com
+nydailynews.com
+nypost.com
+nytimes.com
+nyu.edu
+odnoklassniki.ru
 office.com
 ok.ru
 okezone.com
+opera.com
+oracle.com
+orange.fr
+oreilly.com
+oup.com
+over-blog.com
+ovh.co.uk
+ovh.com
+ovh.net
+ox.ac.uk
+parallels.com
+pastebin.com
 paypal.com
+pbs.org
+people.com
+photobucket.com
+php.net
 pinterest.com
+pixabay.com
+playstation.com
+plesk.com
+politico.com
+prezi.com
+princeton.edu
+privacyshield.gov
+prnewswire.com
+psychologytoday.com
 qq.com
+quantcast.com
 quora.com
+rakuten.co.jp
+rambler.ru
+rapidshare.com
 reddit.com
-Sina.com.cn
+repubblica.it
+reuters.com
+ria.ru
+rottentomatoes.com
+rt.com
+rtve.es
+samsung.com
+sapo.pt
+sciencedaily.com
+sciencedirect.com
+sciencemag.org
+scientificamerican.com
+scribd.com
+seattletimes.com
+secureserver.net
+sedo.com
+seesaa.net
+sendspace.com
+sfgate.com
+shopify.com
+shutterstock.com
+siemens.com
+sina.com.cn
+sky.com
+skype.com
+skyrock.com
+slideshare.net
 sm.cn
+smh.com.au
+so-net.ne.jp
+softonic.com
 sogou.com
 sohu.com
+soratemplates.com
 soso.com
+soundcloud.com
+spiegel.de
+spotify.com
+springer.com
+sputniknews.com
 stackoverflow.com
+stanford.edu
+state.gov
+steamcommunity.com
+steampowered.com
+storage.canalblog.com
+stores.jp
+storify.com
+stuff.co.nz
+surveymonkey.com
+symantec.com
+t-online.de
 t.co
+t.me
+tabelog.com
 taobao.com
+target.com
+techcrunch.com
+ted.com
+telegram.me
+telegraph.co.uk
+terra.com.br
+theglobeandmail.com
+theguardian.com
+themeforest.net
+thestar.com
+thesun.co.uk
+thetimes.co.uk
+theverge.com
+thoughtco.com
 tianya.cn
+time.com
+tinyurl.com
 tmall.com
-#pages.tmall.com
-#login.tmall.com
+tmz.com
 tribunnews.com
+tripadvisor.com
+trustpilot.com
 twitch.tv
 twitter.com
+ucoz.ru
+uiuc.edu
+umich.edu
+un.org
+undeveloped.com
+unesco.org
+uol.com.br
+urbandictionary.com
+usatoday.com
+usgs.gov
+usnews.com
+uspto.gov
+ustream.tv
+utexas.edu
+variety.com
+venturebeat.com
+vice.com
+viglink.com
+vimeo.com
 vk.com
+vkontakte.ru
+vox.com
+w3.org
 w3schools.com
+wa.me
 walmart.com
+washington.edu
+washingtonpost.com
+wattpad.com
+web.fc2.com
+webmd.com
+weebly.com
 weibo.com
+welt.de
+whatsapp.com
+whitehouse.gov
+who.int
+wikia.com
+wikihow.com
+wikimedia.org
 wikipedia.org
+wikipedia.org
+wikipedia.org
+wiktionary.org
+wiley.com
+windowsphone.com
+wired.com
+wix.com
+wordpress.org
+worldbank.org
+wp.com
+wsj.com
+xbox.com
 xinhuanet.com
+yadi.sk
 yahoo.co.
 yahoo.com
+yahoo.com
+yale.edu
 yandex.ru
+yelp.com
+youku.com
+youronlinechoices.com
+youtu.be
 youtube.com
+ytimg.com
+zdnet.com
+zendesk.com
 
 
-
-
-
-
-
-