|
Last change
on this file since 33559 was 33559, checked in by ak19, 5 years ago |
- Special string COPY changed to SUBDOMAIN-COPY after Dr Bainbridge explained why it was more accurate to the behaviour. 2. Comments to explain how the sites-too-big-to-exhaustively-crawl.txt should be formatted, what values are expected and how they work. 3. Special blacklisting and whitelisting of urls on yale.edu, coupled with special treatment in topsites file too.
|
|
File size:
2.0 KB
|
| Line | |
|---|
| 1 | # URL blacklist
|
|---|
| 2 | # FORMAT:
|
|---|
| 3 | # precede URL by ^ to blacklist urls that match the given prefix
|
|---|
| 4 | # succeed URL by $ to blacklist urls that match the given suffix
|
|---|
| 5 | # ^url$ will blacklist urls that match the given url completely
|
|---|
| 6 | # Without either ^ or $ symbol, urls containing the given url will get blacklisted
|
|---|
| 7 |
|
|---|
| 8 |
|
|---|
| 9 | # manually adjusting for irrelevant topsite hits
|
|---|
| 10 | # Rapa-Nui is related to Easter Island
|
|---|
| 11 | ^http://codex.cs.yale.edu/avi/silberschatz/gallery/trips-photos/South-America/Rapa-Nui/
|
|---|
| 12 |
|
|---|
| 13 | # We will blacklist this yale.edu domain except for the subportion that gets whitelisted
|
|---|
| 14 | # then in the sites-too-big-to-exhaustively-crawl.txt, we have a mapping for an allowed url
|
|---|
| 15 | # pattern in case elements on the page are stored elsewhere
|
|---|
| 16 | ^http://korora.econ.yale.edu/
|
|---|
| 17 |
|
|---|
| 18 | # wikipedia pages in
|
|---|
| 19 | # ksh (a German dialect), ilo (Filippino), ty Tahitian, wa for Walons/Walloon,
|
|---|
| 20 | # io (Ido version of Esperanto) and zh-min-nan (Min-Nan-Chinese) are not in the Maori language
|
|---|
| 21 | # Not sure why Commoncrawl had found them for language code MRI
|
|---|
| 22 | ksh.wikipedia.org
|
|---|
| 23 | ilo.wikipedia.org
|
|---|
| 24 | wa.wikipedia.org
|
|---|
| 25 | ty.m.wikipedia.org
|
|---|
| 26 | io.m.wikipedia.org
|
|---|
| 27 | zh-min-nan.wikipedia.org
|
|---|
| 28 | zh-min-nan.wiktionary.org
|
|---|
| 29 |
|
|---|
| 30 | # unwanted domains
|
|---|
| 31 | .video-chat.
|
|---|
| 32 | .videochat.
|
|---|
| 33 | 3chat.ru
|
|---|
| 34 | livevideochatting.org
|
|---|
| 35 | lovewebcam.net
|
|---|
| 36 |
|
|---|
| 37 | cherrybabe.biz
|
|---|
| 38 | dreamsbabes.com
|
|---|
| 39 | adultfantasyboutique.com
|
|---|
| 40 | adultterra.com
|
|---|
| 41 |
|
|---|
| 42 | leatherdyke.porn
|
|---|
| 43 | hornyteenharlots.com
|
|---|
| 44 | adultviewsex.com
|
|---|
| 45 | adultsexualvideo.com
|
|---|
| 46 | ctbererotica.sexe-traque.com
|
|---|
| 47 | cybererotia.porn234.com
|
|---|
| 48 | cybereroticz.adultsupermart.com
|
|---|
| 49 | freegaywebcams.info
|
|---|
| 50 | lesbiansinmysoup.com
|
|---|
| 51 | videopornoxx.online
|
|---|
| 52 | sexandplay.com
|
|---|
| 53 | sexynakedselfies.info
|
|---|
| 54 | barebabez.com
|
|---|
| 55 | britnudes.net
|
|---|
| 56 | camaporno.com
|
|---|
| 57 | webxvideo.com
|
|---|
| 58 | gayspornosex.com
|
|---|
| 59 | jasminreviews.com
|
|---|
| 60 | sexchatlines4u.com
|
|---|
| 61 | sexybabeworld.org
|
|---|
| 62 | sexyleaks.info
|
|---|
| 63 | uniqueporno.com
|
|---|
| 64 | wildsexsluts.com
|
|---|
| 65 | xxxblacknudes.com
|
|---|
| 66 |
|
|---|
| 67 | # sounds like some pirating site
|
|---|
| 68 | ^http://pirateguides.com/
|
|---|
| 69 |
|
|---|
| 70 | # from alexa topsites at https://www.alexa.com/topsites
|
|---|
| 71 | livejasmin.com
|
|---|
| 72 | pornhub.com
|
|---|
| 73 | # listed as a similar topsite at https://en.wikipedia.org/wiki/List_of_most_popular_websites
|
|---|
| 74 | redtube.com
|
|---|
| 75 | xvideos.com
|
|---|
| 76 | xhamster.com
|
|---|
| 77 | xnxx.com
|
|---|
Note:
See
TracBrowser
for help on using the repository browser.
