source: main/trunk/model-sites-dev/multimodal-mdl/collect/salami-ground-truth/transform/pages/about.xsl@ 28747

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE videocollection [
  <!ENTITY ndash "&#8211;">
  <!ENTITY mdash "&#8212;">
]>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:java="http://xml.apache.org/xslt/java"
    xmlns:util="xalan://org.greenstone.gsdl3.util.XSLTUtil"
    xmlns:gslib="http://www.greenstone.org/skinning"
    extension-element-prefixes="java util"
    exclude-result-prefixes="java util">

    <!-- use the 'main' layout -->
    <xsl:include href="layouts/main.xsl"/>

    <!-- set page title -->
    <xsl:template name="pageTitle"><gslib:collectionName/></xsl:template>

    <!-- set page breadcrumbs -->
    <xsl:template name="breadcrumbs"><gslib:siteLink/><gslib:rightArrow/></xsl:template>

    <!-- the page content -->
    <xsl:template match="/page">
  
        <!--Display the description text of the current collection,
        and if some services are available then create a list
        of links for each service within a <ul id="servicelist"> element.-->

    <xsl:variable name="httpPath"> 
      <xsl:value-of select="/page/pageResponse/collection/metadataList/metadata[@name='httpPath']" />
    </xsl:variable>


    <style>
      p.about { text-align: justify; }
      h2 { /* line-height: 135%; */ }
      h3 { margin-top: 4pt; margin-bottom: 7pt; line-height: 135%; }

        a.mashup {text-decoration: none; } 
    a.mashup:link    {color:#000080;}  /* unvisited link */
    a.mashup:visited {color:#303080;}  /* visited link */
    a.mashup:hover   {color:#000040;}  /* mouse over link */
    a.mashup:active  {color:#0000a0;}

    </style>


    <script src="{$httpPath}/script2/browser-detect.js" type="text/javascript">
    /* space filler needed */
    </script>


<div style="width: 800px; margin-left: auto; margin-right: auto;">


<h2>Structured Audio in a Music Digital Library</h2>

<!--
<div style="background-color: #DD3611; color: white; padding: 4pt; margin-bottom: 6pt;">
<table>

  <tr>
    <td>
       <img src="{$httpPath}/images/firefox-128.png" />
    </td>
    <td>
    <p class="about">
      <i><font size="+2">
        An experiment with Firefox's Advanced
        Audio Processing Extension ...
      </font></i>
    </p>

    <p class="about">
      <i><font size="+2">
... featuring a mash-up of 
<a class="mashup" href="http://www.greenstone.org" target="_blank">Greenstone</a>,
<a class="mashup" href="http://www.omras2.org/audioDB" target="_blank">AudioDB</a>, and
<a class="mashup" href="http://seasr.org/meandre/" target="_blank">Meandre</a>.
      </font></i>
    </p>


          <div id="bd-version" style="display:none;" class="about">
        <p>
          <img src="{$httpPath}/images/warning-icon.png" />
            Version of Firefox detected:
              <script type="text/javascript">
                document.write(BrowserDetect.version);
              </script>
        </p>
        <p>
            You need to upgrade to a more recent version of Firefox 
            (v4.0 or greater) to access all the audio features of 
            this demonstration.
        </p>
          </div>

          <div id="bd-browser" style="display:none;" class="about">

        <p>
          <img src="{$httpPath}/images/warning-icon.png" />
Browser detected:
          <script type="text/javascript">
            document.write(BrowserDetect.browser);
          </script>
        </p>
            <p>You need to access this web-site using Firefox 
            (v4.0 or greater) to access all the audio features provided.
        </p>
          </div>




  <script>
    <xsl:text disable-output-escaping="yes">

if (BrowserDetect.browser == "Firefox") {
  if (BrowserDetect.version &lt; 4) {
    document.getElementById("bd-version").style.display="block";

  }
}
else {
  document.getElementById("bd-browser").style.display="block";

}
      </xsl:text>
    </script>




    </td>
  </tr>
</table>
</div>
-->

<p class="about">
This digital library collection is derived from the American workset,
from the <a href="http://salami.lis.illinois.edu">SALAMI</a> project.

</p>

<p class="about">This music DL embodies the following:

<style>
  li { padding-bottom: 10px; }

</style>


<ol>
  <li>
  <i>Starting point:</i> Manual (command-line) aggregation of disparate
    resources prior to building the DL.  A set of bespoke Greenstone
    document processing plugins corrals the heterogeneous gathered
    data into a unified (homogeneous) canonical form that the DL can
    access and display.  Everything presented in the DL is
    either pre-computed (such as the self-similarity heat-maps) or else
    computed at build-time.
  </li>

   <li>
     <i>Audio-fingerprinting:</i> as before, but now metadata about the audio
     songs is now enriched through a set of audio-fingerprinting web
     services.  Everything presented in the DL is still pre-computed
     or else computed at build-time—however,  the inclusion in the document
     view of a &quot;Discovery&quot; block allows a user to begin to
     access and explore through linked-data, information related to
     the song.
   </li>

  <li>
    <i>Client-side audio processing (and visualization):</i> the
    Pre-computed self-similarity heat-maps are dropped from the
    collection building process in preference for the same information
    being computed through Javascript running in the user's web
    browser.
  </li>

  <li>
    <i>Embedded Meandre workflows:</i> the Meandre Workbench is
    integrated into Greenstone.  Audio documents in the Greenstone
    digital library can now be dispatched and be processed by the
    selected Meandre, and output from the workflow returned to the
    Greenstone document view—for example, playing audio that has been
    processed and output from the workflow.
  </li>

  <li>
    <i>Client-side workflows:</i> the client-side Javascript
    processing code is refactored to follow the same methodology as
    Meandre components.
  </li>

  <li>
    <i>Client/Server hybrid workflows:</i> the Greenstone/Meandre
    integration is extended to support the dynamic 
    transmission of <tt>executeCallBack()</tt> methods written 
    in the user's browser in Javascript to be run on the
    Meandre server as part of the active workflow.
  </li>

  <li>
    <i>Forging ahead:</i> the next area to be worked on is upgrading
    the level of Greenstone/Meandre integration so that data produced
    by the workflows can be incorporated back into the underlying
    digital library itself, rather than being (as it currently stands)
    transitory data that only lives for the duration of the web page
    being viewed.  This will form an implementation stepping-stone to
    a more generalized ability to have data retrieved from other
    external resources (located through the Discovery block
    linked-data portion of the document view) ingested in to the DL
    collection.
  </li>


</ol>

</p>



<!--

<p class="about">This digital library collection demonstrates the
integration of a variety of sources of structural metadata from both
automatically derived content analysis and manually labelled
ground-truth data to form a rich interactive web application suitable
for musicologists (as the target end-users) to explore a collated set
of music files.</p>

<p class="about">While the base digital library system, <a
href="http://www.greenstone.org" target="_blank">Greenstone</a>, is
designed to operate with any web browser, the audio features this
particular DL collection demonstrates relies on audio processing
features currently only available in Firefox.  Using one of the other
web browsers you will be able to access most features of the digital
library collection, but when you access a particular song you may find
playing the audio—in particular the panned audio
effect—does not work.</p>
-->


<!--
<p class="about">The developed system combines the open source capabilities of
Greenstone, \cite{Greenstone}, NEMA \cite{Nema}, Salami \cite{Salami},
and AudioDB \cite{AudioDB}.

%%
NEMA and Salami have been previously described in this article, and contribute 
manual and audio content based metadata.
%%
AudioDB is a raw-audio content based searching algorithm based on Local-Sensitivity Hashing \cite{LSH}.
%%
Greenstone is a versatile digital library architecture with an
extensible service-based architecture that.  In
addition to the text-based searching and browsing capability to
organize content, Greenstone provides the framework in which to harness the
structural metadata from NEMA and Salami, and the audio-content based
search functionality of audioDB.

-->



<h3>A Walkthrough</h3>


<p class="about">The following walkthrough is for the initial incarnation of the DL collection, where all
the information presented is either precomputed or computed at build time.</p>

<p class="about">Taking as a starting point a set of music files identified as worthy
of study, the <a href="#browse">figure below</a> shows the result of browsing
the formed digital library collection <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by title</a> from a web browser.
The figure is a useful snapshot in which to orientate ourselves with
the main structure and features to the digital library.  Functionality that
persistently reoccurs is accessible through the header to the page.</p>

<p class="about">This includes:
<ul>

<li>help and preferences (top-right);</li>
<li>a quick-search option (located just
below) with links to more sophisticated searching options; and</li>
<li>pin-pointing where within the site a user is currently located
(top-left).</li>
</ul>


<a name="browse" />
<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
  <tr>
    <td style="border: solid 1px;">
      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-browse.png" />
    </td>
  </tr>
  <tr style="background-color: #bbeebb">
    <td>
      <i>Browsing in the digital library
  <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by titles</a>.</i>
    </td>
  </tr>
</table>


The specific content to this location within the site (in this case
browsing by title) is shown beneath the main banner.  Various
groupings of title can be accessed by clicking on the bookshelf icons
vertically aligned as the main part of the page: currently <a
href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1#CL1.2">C–D</a>
is open, with the remaining letters to the alphabet below this,
accessed through scrolling.</p>




<p class="about">Interested in the song <i>Candela</i> our curious musicologist clicks
on <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;p.sa=&amp;p.s=ClassifierBrowse" target="_blank">the link for this</a>.  This brings up the document view to this song:</p>

<a name="self-similarity" />
<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
  <tr>
    <td style="border: solid 1px;">
      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-self-similarity2.png" />
    </td>
  </tr>
  <tr style="background-color: #bbeebb">
    <td>
      <i>The <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;p.sa=&amp;p.s=ClassifierBrowse" target="_blank">musicologically enriched document view</a> for</i> Candela.
    </td>
  </tr>
</table>

<p class="about">Normally in a digital
library the document view brings up a page that is strongly derived
from textual metadata.  If the document viewed was a text-document,
some summary information such as title and author is typically
presented, say in tabular form, before the main text is presented.
Even in the case of multimedia digital libraries, the view presented
is still strongly derived from textual metadata: this time including
details such as the length of the video, the TV company that produced
it, whether captions are available, and so forth, accompanied with an
embedded video player for viewing the content—essentially more
textual metadata (in this case the URL to the video content) which in
terms of the user-interface is largely divorced from the other
elements displayed on the page.</p>

<p class="about">This contrasts sharply with the document view developed in this
digital library.
Naturally it allows the song to be played (akin to the embedded video player),
however this is largely of secondary importance to the 
other functionality available this is much more closely
integrated.</p>

<p class="about">The most striking visual component to the document view is a
self-similarity &quot;heat map&quot; where the duration of the song forms both
the <i>x</i>- and <i>y</i>-axis, and a red pixel located at a given <i>(x,y)</i>
co-ordinate in the map represents a location where two parts of the
song are strongly similar, proportionally shifting to blue to
represent dissimilar.  Given this configuration, the leading diagonal
to the matrix (<i>x=y</i>) is always coloured red as this represents the
comparison of one part of the song with itself.</p>

<p class="about">When the user moves the mouse cursor around the self-similarity map
a highlighting circle is shown to emphasize the area the user is over,
with a black dot at the centre (visible in in
the <a href="#self-similarity">above figure</a>); annotated vertically
and horizontally are the two time-offsets in seconds that that point
in the map corresponds to.  Clicking the cursor at this point results
in the audio being played <i>simultaneously</i> from these two points.
To aid the musicologist in listening to the two parts of the song, one
part is panned to the left speaker, and the other to the right (this
was implemented using the extended audio API provided by Firefox, and
so this particular feature only work when viewing the collection with
this browser—see implementation details below).  In our figure,
the musicologist has zeroed in the location <i>x=33</i>, <i>y=97</i>
which corresponds to the start of a strong red diagonal that occurs
some distance off the leading diagonal.  Listening to the two sounds
played (most reliably done with headphones on), they hear that these
two sections of the song are indeed repeating sections of the guitar
piece <i>Candela</i> with a minor variation in the latter section
where a recorder is also playing in the arrangement.</p>

<p class="about">The structured audio time-lines (labelled A, B, ..., and 6, 5, 2,
... in the figure) located above the self-similarity map are another
area of enriched musical content in the digital library. The upper
line shows the ground-truth data for this song generated by
the <a href="http://www.music-ir.org/?q=node/14" target="_blank">Salami project</a>;
the lower line is generated by an algorithmically based content
analysis algorithm.</p>

<p class="about">While there is some agreement between these two lines, there are
also significant differences.  The play and search buttons within the
structured time-lines (the latter represented by a magnifying glass)
allow the user to investigate further these structures.  We shall
return to the search functionality shortly (which is content based,
using <a href="http://omras2.doc.gold.ac.uk/software/audiodb/" target="_blank">AudioDB</a>),
but in the meantime note that with the time-lines positioned above the
self-similarity map, there is further opportunity to study the
differences between the two structured time-line.  It is certainly the
case that there are strong visual cues in the map that line up with
the algorithmic time-line, even though they do not align with a
boundary in the ground-truth data, and the user can click on these
parts of the similarity map to hear what is occurring at these
points.</p>



<a name="resultspage" />
<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
  <tr>
    <td style="border: solid 1px;">
      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search-resultspage-halfcut.png" />
    </td>
    <td style="border: solid 1px;">
      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-structured-search-resultspage-halfcut.png" />
    </td>
  </tr>
  <tr style="background-color: #bbeebb">
    <td colspan="2">
      <i>Audio content based search results: left) the 
    <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=AudioQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.maxDocs=100&amp;s1.hitsPerPage=20&amp;q=D206.dir&amp;s1.query=D206.dir&amp;s1.offset=263&amp;s1.length=24&amp;mysongWindowDuration=6" target="_blank">result list</a> from
      an audio content based query taken from an extract of </i>
      Michelle; <i>and right)
      the <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=TextQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.query=%22b+b+c+b+c%22&amp;s1.index=JS" target="_blank">result of a structured music search</a> for
      content containing &quot;b b c b a&quot; as a sequence.</i>
    </td>
  </tr>
</table>

<p class="about">Returning to the search capability provided by the structured
time-lines, the <a href="#resultspage">above figure (left)</a> show the result of
using this feature while study the song <i>Michelle</i> by The
Beatles.  In this case the user selected the section of the
ground-truth time-line corresponding to section starting &quot;I want you
...&quot;, but could have equally used the algorithmically calculated
time-line, or in fact paused the song playing at any point, and
started a match from there.</p>


<p class="about">Not surprisingly <i>Michelle</i> is returned as the top hit (at
92.1%)—we shall see shortly that this is because the system found
several sections of the song that matched this—the next hit being
<i>Bigger Than JC</i> at 74.4%, and so on down, where only one hit
per song occurs.  Clicking on the top hit,
the <a href="#audiodb">figure below</a> shows the document view that is
displayed, focusing in on the key area to this screen.  This time the
time-line area has an additional bar: the points within the song that
AudioDB found to be similar.  Like the other time-lines, a play button
is present on these segments so the user can play the matching points
directly.  In this case
clicking on them reveals the matching sections found correspond to
melodically repeating sections of the song, only with different lyrics
(&quot;I love you ...&quot; and &quot;I need you ...&quot;).</p>


<p class="about">A further form of musical content-based searching is available through
the main header to the digital library.  Instead of searching by title
or artist (which are also available) the user can click on this
&quot;search by&quot; menu next to the quick-search text box and select
&quot;ground-truth structure&quot; instead.
The <a href="#resultspage">figure above</a> (right) shows the result of using this
option, where the user has entered &quot;b b c b c&quot; as the query, in
other words searching for songs that have two sections in a row the
same, then a new section, then returning to the original section,
before progressing to a recurrence of the second distinct section.  In
pop and rock music this is a popular sequence corresponding to verse,
verse, chorus, verse, chorus.  A more intriguing framing of a query
along these lines—equally possible in the digital library—would be
to go to the fielded search page (accessible through the <i>advanced
  search</i> link below the quick-search box, and entered the same
music structure query, but this time combine it with other fielded
query terms (Genre=Blues OR Jazz, Year&lt;1950).  Results from this
query would let the musicologist explore potential evidence of this
pattern of sections being used in two of the key musical genres that
influenced the development of Rock and Roll music.</p>


<a name="audiodb" />
<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
  <tr>
    <td style="border: solid 1px;">
      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search2.png" />
    </td>
  </tr>
  <tr style="background-color: #bbeebb">
    <td>
      <i>The document view for</i> Michelle <i><a href="dev?a=d&amp;c=salami-audioDB&amp;d=D206&amp;dt=simple&amp;p.frameOffset=133,393,264,262,392&amp;p.frameLength=24&amp;p.a=q&amp;p.s=AudioQuery&amp;hl=on&amp;ed=1#D206" target="_blank">augmented with the locations of the matches</a> of the audio query.</i>
    </td>
  </tr>
</table>




<h3>Implementation details</h3>

<p class="about">The core part of the interactive elements in the document view were
implemented using SVG combined with Javascript.  The left- and
right-panning interactively available from the self-similarity map was
implemented by processing the raw audio stream, made accessible by the
Firefox Audio extension
<a href="https://wiki.mozilla.org/Audio_Data_API" target="_blank">API</a>.</p>

<p class="about">AudioDB content based searching was integrated into Greenstone through
two components of the digital library software architecture: its
build-time document processing plugin system, and its runtime
message-passing service-base framework.  The developed plugin accepts
a wide range of audio formats (including OGG and MP3), and converts
them to WAV, the format needed by AudioDB for processing.  The new
search service took the form of a proxy, accepting messages in the
XML syntax used by Greenstone, turning them into the necessary calls
to the AudioDB command-line interface, and then converting the output
from AudioDB back into the XML syntax expected by the digital library
architecture.  Finally, the two parts were packaged to operate as a
Greenstone extension; the software is available at:
<a href="http://trac.greenstone.org/gs3-extensions/audioDB/trunk/src" target="_blank">http://trac.greenstone.org/gs3-extensions/audioDB/trunk/src</a>.</p>

</div>


    </xsl:template>


</xsl:stylesheet>