Changeset 33553

Timestamp:

2019-10-04T22:19:20+13:00 (5 years ago)

Author:

ak19

Message:

Comments

File:

• gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt

@@ +1 @@
+# URL blacklist
+# FORMAT:
+# precede URL by ^ to blacklist urls that match the given prefix
+# succeed URL by $ to blacklist urls that match the given suffix
+# ^url$ will blacklist urls that match the given url completely
+# Without either ^ or $ symbol, urls containing the given url will get blacklisted
 
-# Add alexa top sites (only 50 visible)
+# Contains alexa top sites (where only the first 50 were visible)
 # Added further top sites from https://en.wikipedia.org/wiki/List_of_most_popular_websites
-## Finally also got the CSV from https://moz.com/top500 and added it to the list and added them in.
+# Finally also added https://moz.com/top500 by downloading its CSV file and
+# adding its URLs to the existing listing here from alexa/wiki.
 # Then used LibreOffice's Calc spreadsheet software to sort alphabetically and remove duplicates.
-# Then in Gedit, used regex search and replace to remove <subdomain>.<site>.ext to keep just <site>.ext
-# And resorted alphabetically
+# Then in Gedit, used regex search and replace to remove <subdomain>.<site>.ext variants, keeping
+# just <site>.ext
+# And finally, re-sorted the reduced list alphabetically and pasted into here.