source: main/trunk/model-sites-dev/eurovision-lod/collect/eurovision/prepare/README.txt@ 35940

# Local Config Settings

Create _local_prepare_config.sh by copying its '.in' file:

    cp _local_prepare_config.sh.in _local_prepare_config.sh

Then open in a text editor, for example:

    emacs _local_prepare_config.sh

review and edit appropriately.

If you plan to use the Spotify based 'Track a' for MIR-based features,
then create an appropriate Spotify API credentials through:

    https://developer.spotify.com/dashboard/applications

and then update SPOTIPY_CLIENT_ID and SPOTIPY_CLIENT_SECRET with
your specific credentials.

Otherwise you can use 'Track b' which -- proxied through the
eurovisionworld.com fan web site -- page scrapes YouTube content of
the songs, and then goes through a process using ffmpeg and essentia
to generate audio features.  No key setup is needed if Track b.


# Setup a Virtual Python3 Environment

Next create a virtual Python environment:

    ./CREATE-VENV-PYTHON3.sh my-python3-dev

And then install required Python packages:

  * For processing Excel Voting data spreadsheet
    pip3 install openpyxl

  * For parsing Wikipeida pages for missing category entries
    pip3 install wikipedia beautifulsoup4
 
  * For Spotify/MusicBrainz musically computed audio content (Track 'a')
     pip3 install -r  spotify-musicbrainz/requirements.txt

  * For Essentia Audio Features (Track 'b')
     
     git clone https://github.com/davidbwaikato/eurovision-dataset essentia-audio-features

     pip3 install -r essentia-audio-features/requirements.txt

# One-time Errata Triplestore Setup

The (personal) convention of using ALL-CAPS is to signify that these
scripts can be run without any arguments and they will do something
meaningful, with the caveat that the current working directory must be
the directory where the scripts are located.

Below you will learn about the sequence of 'prepare' scripts that
reach out to external resoures -- initially Linked Data through
DBpedia, but then topped up with some page scraping -- but first a
local (Greenstone3) triple store graph needs to set up with some
errata values that help the SPARQL query run against DBpedia to yield
more accurate results.

Check your Greenstone3 Triplestore server is running:

  sudo systemctl status greenstone3-triplestore

Then run:

   ./UPLOAD-TTL-EUROVISION-ERRATA-GRAPH.sh

This currently talks to the Apache Jena v1 Triplestore version that the
Greenstone3 extension operates on port 3030


# Running the 'prepare' ALL-CAPS scripts


The prefix to the scripts to run are sequentially numbers 01-...,
02-... and so on.


To aid development and testing, there is a 'small' set of files that
can be prepared, based on only the Eurovision entries for 2015.

You can run all the steps to generate the small version with:

  ./PREPARE-ALL-SMALL.sh

Or else generate the data for the full collection with

  ./PREPARE-ALL.sh

Look inside the scripts, and copy and paste just the bits you want to
run smaller segements of the prepare process.

With the a complete run through done, move up one directory to the
main collection directory and build the collection.