|
Last change
on this file since 33556 was 33556, checked in by ak19, 5 years ago |
|
Blacklisted wikipedia pages that are actually in other languages which had found their way into commoncrawl MRI results.
|
|
File size:
1.6 KB
|
| Line | |
|---|
| 1 | # URL blacklist
|
|---|
| 2 | # FORMAT:
|
|---|
| 3 | # precede URL by ^ to blacklist urls that match the given prefix
|
|---|
| 4 | # succeed URL by $ to blacklist urls that match the given suffix
|
|---|
| 5 | # ^url$ will blacklist urls that match the given url completely
|
|---|
| 6 | # Without either ^ or $ symbol, urls containing the given url will get blacklisted
|
|---|
| 7 |
|
|---|
| 8 |
|
|---|
| 9 | # wikipedia pages in
|
|---|
| 10 | # ksh (a German dialect), ilo (Filippino), ty Tahitian, wa for Walons/Walloon,
|
|---|
| 11 | # io (Ido version of Esperanto) and zh-min-nan (Min-Nan-Chinese) are not in the Maori language
|
|---|
| 12 | # Not sure why Commoncrawl had found them for language code MRI
|
|---|
| 13 | ksh.wikipedia.org
|
|---|
| 14 | ilo.wikipedia.org
|
|---|
| 15 | wa.wikipedia.org
|
|---|
| 16 | ty.m.wikipedia.org
|
|---|
| 17 | io.m.wikipedia.org
|
|---|
| 18 | zh-min-nan.wikipedia.org
|
|---|
| 19 | zh-min-nan.wiktionary.org
|
|---|
| 20 |
|
|---|
| 21 | # unwanted domains
|
|---|
| 22 | .video-chat.
|
|---|
| 23 | .videochat.
|
|---|
| 24 | 3chat.ru
|
|---|
| 25 | livevideochatting.org
|
|---|
| 26 | lovewebcam.net
|
|---|
| 27 |
|
|---|
| 28 | cherrybabe.biz
|
|---|
| 29 | dreamsbabes.com
|
|---|
| 30 | adultfantasyboutique.com
|
|---|
| 31 | adultterra.com
|
|---|
| 32 |
|
|---|
| 33 | leatherdyke.porn
|
|---|
| 34 | hornyteenharlots.com
|
|---|
| 35 | adultviewsex.com
|
|---|
| 36 | adultsexualvideo.com
|
|---|
| 37 | ctbererotica.sexe-traque.com
|
|---|
| 38 | cybererotia.porn234.com
|
|---|
| 39 | cybereroticz.adultsupermart.com
|
|---|
| 40 | freegaywebcams.info
|
|---|
| 41 | lesbiansinmysoup.com
|
|---|
| 42 | videopornoxx.online
|
|---|
| 43 | sexandplay.com
|
|---|
| 44 | sexynakedselfies.info
|
|---|
| 45 | barebabez.com
|
|---|
| 46 | britnudes.net
|
|---|
| 47 | camaporno.com
|
|---|
| 48 | webxvideo.com
|
|---|
| 49 | gayspornosex.com
|
|---|
| 50 | jasminreviews.com
|
|---|
| 51 | sexchatlines4u.com
|
|---|
| 52 | sexybabeworld.org
|
|---|
| 53 | sexyleaks.info
|
|---|
| 54 | uniqueporno.com
|
|---|
| 55 | wildsexsluts.com
|
|---|
| 56 | xxxblacknudes.com
|
|---|
| 57 |
|
|---|
| 58 | # sounds like some pirating site
|
|---|
| 59 | ^http://pirateguides.com/
|
|---|
| 60 |
|
|---|
| 61 | # from alexa topsites at https://www.alexa.com/topsites
|
|---|
| 62 | livejasmin.com
|
|---|
| 63 | pornhub.com
|
|---|
| 64 | # listed as a similar topsite at https://en.wikipedia.org/wiki/List_of_most_popular_websites
|
|---|
| 65 | redtube.com
|
|---|
| 66 | xvideos.com
|
|---|
| 67 | xhamster.com
|
|---|
| 68 | xnxx.com
|
|---|
Note:
See
TracBrowser
for help on using the repository browser.
