root/main/trunk/model-sites-dev/multimodal-mdl/collect/salami-ground-truth/transform/pages/about.xsl @ 28433

Revision 28433, 21.9 KB (checked in by davidb, 6 years ago)

Further collections to add to the set

  • Property svn:executable set to *
  • Property svn:mime-type set to application/xml
1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE videocollection [
3  <!ENTITY ndash "&#8211;">
4  <!ENTITY mdash "&#8212;">
6<xsl:stylesheet version="1.0"
7    xmlns:xsl=""
8    xmlns:java=""
9    xmlns:util="xalan://org.greenstone.gsdl3.util.XSLTUtil"
10    xmlns:gslib=""
11    extension-element-prefixes="java util"
12    exclude-result-prefixes="java util">
14    <!-- use the 'main' layout -->
15    <xsl:include href="layouts/main.xsl"/>
17    <!-- set page title -->
18    <xsl:template name="pageTitle"><gslib:collectionName/></xsl:template>
20    <!-- set page breadcrumbs -->
21    <xsl:template name="breadcrumbs"><gslib:siteLink/><gslib:rightArrow/></xsl:template>
23    <!-- the page content -->
24    <xsl:template match="/page">
26        <!--Display the description text of the current collection,
27        and if some services are available then create a list
28        of links for each service within a <ul id="servicelist"> element.-->
30    <xsl:variable name="httpPath">
31      <xsl:value-of select="/page/pageResponse/collection/metadataList/metadata[@name='httpPath']" />
32    </xsl:variable>
35    <style>
36      p.about { text-align: justify; }
37      h2 { /* line-height: 135%; */ }
38      h3 { margin-top: 4pt; margin-bottom: 7pt; line-height: 135%; }
40        a.mashup {text-decoration: none; }
41    a.mashup:link    {color:#000080;}  /* unvisited link */
42    a.mashup:visited {color:#303080;}  /* visited link */
43    a.mashup:hover   {color:#000040;}  /* mouse over link */
44    a.mashup:active  {color:#0000a0;}
46    </style>
49    <script src="{$httpPath}/script/browser-detect.js" type="text/javascript">
50    /* space filler needed */
51    </script>
54<div style="width: 800px; margin-left: auto; margin-right: auto;">
57<h2>A Musical Web Observatory: </h2>
59<h3><i><font size="+2">Connecting People with Data, Documents, and Algorithms</font></i></h3>
61<div style="background-color: #DD3611; color: white; padding: 4pt; margin-bottom: 6pt;">
64  <tr>
65    <td>
66       <img src="{$httpPath}/images/firefox-128.png" />
67    </td>
68    <td>
69    <p class="about">
70      <i><font size="+2">
71        An experiment with Firefox's Advanced
72        Audio Processing Extension ...
73      </font></i>
74    </p>
76    <p class="about">
77      <i><font size="+2">
78... featuring a mash-up of
79<a class="mashup" href="" target="_blank">Greenstone</a>,
80<a class="mashup" href="" target="_blank">AudioDB</a>, and
81<a class="mashup" href="" target="_blank">Meandre</a>.
82      </font></i>
83    </p>
86          <div id="bd-version" style="display:none;" class="about">
87        <p>
88          <img src="{$httpPath}/images/warning-icon.png" />
89            Version of Firefox detected:
90              <script type="text/javascript">
91                document.write(BrowserDetect.version);
92              </script>
93        </p>
94        <p>
95            You need to upgrade to a more recent version of Firefox
96            (v4.0 or greater) to access all the audio features of
97            this demonstration.
98        </p>
99          </div>
101          <div id="bd-browser" style="display:none;" class="about">
103        <p>
104          <img src="{$httpPath}/images/warning-icon.png" />
105Browser detected:
106          <script type="text/javascript">
107            document.write(BrowserDetect.browser);
108          </script>
109        </p>
110            <p>You need to access this web-site using Firefox
111            (v4.0 or greater) to access all the audio features provided.
112        </p>
113          </div>
118  <script>
119    <xsl:text disable-output-escaping="yes">
121if (BrowserDetect.browser == "Firefox") {
122  if (BrowserDetect.version &lt; 4) {
123    document.getElementById("bd-version").style.display="block";
125  }
127else {
128  document.getElementById("bd-browser").style.display="block";
131      </xsl:text>
132    </script>
137    </td>
138  </tr>
142<p class="about">This project is currently work in progress.  The sequence of development so far has been:
145  li { padding-bottom: 10px; }
151  <li>
152  <i>Starting point:</i> Manual (command-line) aggregation of disparate
153    resources prior to building the DL.  A set of bespoke Greenstone
154    document processing plugins corrals the heterogeneous gathered
155    data into a unified (homogeneous) canonical form that the DL can
156    access and display.  Everything presented in the DL is
157    either pre-computed (such as the self-similarity heat-maps) or else
158    computed at build-time.
159  </li>
161   <li>
162     <i>Audio-fingerprinting:</i> as before, but now metadata about the audio
163     songs is now enriched through a set of audio-fingerprinting web
164     services.  Everything presented in the DL is still pre-computed
165     or else computed at build-time—however,  the inclusion in the document
166     view of a &quot;Discovery&quot; block allows a user to begin to
167     access and explore through linked-data, information related to
168     the song.
169   </li>
171  <li>
172    <i>Client-side audio processing (and visualization):</i> the
173    Pre-computed self-similarity heat-maps are dropped from the
174    collection building process in preference for the same information
175    being computed through Javascript running in the user's web
176    browser.
177  </li>
179  <li>
180    <i>Embedded Meandre workflows:</i> the Meandre Workbench is
181    integrated into Greenstone.  Audio documents in the Greenstone
182    digital library can now be dispatched and be processed by the
183    selected Meandre, and output from the workflow returned to the
184    Greenstone document view—for example, playing audio that has been
185    processed and output from the workflow.
186  </li>
188  <li>
189    <i>Client-side workflows:</i> the client-side Javascript
190    processing code is refactored to follow the same methodology as
191    Meandre components.
192  </li>
194  <li>
195    <i>Client/Server hybrid workflows:</i> the Greenstone/Meandre
196    integration is extended to support the dynamic
197    transmission of <tt>executeCallBack()</tt> methods written
198    in the user's browser in Javascript to be run on the
199    Meandre server as part of the active workflow.
200  </li>
202  <li>
203    <i>Forging ahead:</i> the next area to be worked on is upgrading
204    the level of Greenstone/Meandre integration so that data produced
205    by the workflows can be incorporated back into the underlying
206    digital library itself, rather than being (as it currently stands)
207    transitory data that only lives for the duration of the web page
208    being viewed.  This will form an implementation stepping-stone to
209    a more generalized ability to have data retrieved from other
210    external resources (located through the Discovery block
211    linked-data portion of the document view) ingested in to the DL
212    collection.
213  </li>
224<p class="about">This digital library collection demonstrates the
225integration of a variety of sources of structural metadata from both
226automatically derived content analysis and manually labelled
227ground-truth data to form a rich interactive web application suitable
228for musicologists (as the target end-users) to explore a collated set
229of music files.</p>
231<p class="about">While the base digital library system, <a
232href="" target="_blank">Greenstone</a>, is
233designed to operate with any web browser, the audio features this
234particular DL collection demonstrates relies on audio processing
235features currently only available in Firefox.  Using one of the other
236web browsers you will be able to access most features of the digital
237library collection, but when you access a particular song you may find
238playing the audio—in particular the panned audio
239effect—does not work.</p>
244<p class="about">The developed system combines the open source capabilities of
245Greenstone, \cite{Greenstone}, NEMA \cite{Nema}, Salami \cite{Salami},
246and AudioDB \cite{AudioDB}.
249NEMA and Salami have been previously described in this article, and contribute
250manual and audio content based metadata.
252AudioDB is a raw-audio content based searching algorithm based on Local-Sensitivity Hashing \cite{LSH}.
254Greenstone is a versatile digital library architecture with an
255extensible service-based architecture that.  In
256addition to the text-based searching and browsing capability to
257organize content, Greenstone provides the framework in which to harness the
258structural metadata from NEMA and Salami, and the audio-content based
259search functionality of audioDB.
265<h3>A Walkthrough</h3>
268<p class="about">The following walkthrough is for the initial incarnation of the DL collection, where all
269the information presented is either precomputed or computed at build time.</p>
271<p class="about">Taking as a starting point a set of music files identified as worthy
272of study, the <a href="#browse">figure below</a> shows the result of browsing
273the formed digital library collection <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by title</a> from a web browser.
274The figure is a useful snapshot in which to orientate ourselves with
275the main structure and features to the digital library.  Functionality that
276persistently reoccurs is accessible through the header to the page.</p>
278<p class="about">This includes:
281<li>help and preferences (top-right);</li>
282<li>a quick-search option (located just
283below) with links to more sophisticated searching options; and</li>
284<li>pin-pointing where within the site a user is currently located
289<a name="browse" />
290<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
291  <tr>
292    <td style="border: solid 1px;">
293      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-browse.png" />
294    </td>
295  </tr>
296  <tr style="background-color: #bbeebb">
297    <td>
298      <i>Browsing in the digital library
299  <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by titles</a>.</i>
300    </td>
301  </tr>
305The specific content to this location within the site (in this case
306browsing by title) is shown beneath the main banner.  Various
307groupings of title can be accessed by clicking on the bookshelf icons
308vertically aligned as the main part of the page: currently <a
310is open, with the remaining letters to the alphabet below this,
311accessed through scrolling.</p>
316<p class="about">Interested in the song <i>Candela</i> our curious musicologist clicks
317on <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;;p.s=ClassifierBrowse" target="_blank">the link for this</a>.  This brings up the document view to this song:</p>
319<a name="self-similarity" />
320<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
321  <tr>
322    <td style="border: solid 1px;">
323      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-self-similarity2.png" />
324    </td>
325  </tr>
326  <tr style="background-color: #bbeebb">
327    <td>
328      <i>The <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;;p.s=ClassifierBrowse" target="_blank">musicologically enriched document view</a> for</i> Candela.
329    </td>
330  </tr>
333<p class="about">Normally in a digital
334library the document view brings up a page that is strongly derived
335from textual metadata.  If the document viewed was a text-document,
336some summary information such as title and author is typically
337presented, say in tabular form, before the main text is presented.
338Even in the case of multimedia digital libraries, the view presented
339is still strongly derived from textual metadata: this time including
340details such as the length of the video, the TV company that produced
341it, whether captions are available, and so forth, accompanied with an
342embedded video player for viewing the content—essentially more
343textual metadata (in this case the URL to the video content) which in
344terms of the user-interface is largely divorced from the other
345elements displayed on the page.</p>
347<p class="about">This contrasts sharply with the document view developed in this
348digital library.
349Naturally it allows the song to be played (akin to the embedded video player),
350however this is largely of secondary importance to the
351other functionality available this is much more closely
354<p class="about">The most striking visual component to the document view is a
355self-similarity &quot;heat map&quot; where the duration of the song forms both
356the <i>x</i>- and <i>y</i>-axis, and a red pixel located at a given <i>(x,y)</i>
357co-ordinate in the map represents a location where two parts of the
358song are strongly similar, proportionally shifting to blue to
359represent dissimilar.  Given this configuration, the leading diagonal
360to the matrix (<i>x=y</i>) is always coloured red as this represents the
361comparison of one part of the song with itself.</p>
363<p class="about">When the user moves the mouse cursor around the self-similarity map
364a highlighting circle is shown to emphasize the area the user is over,
365with a black dot at the centre (visible in in
366the <a href="#self-similarity">above figure</a>); annotated vertically
367and horizontally are the two time-offsets in seconds that that point
368in the map corresponds to.  Clicking the cursor at this point results
369in the audio being played <i>simultaneously</i> from these two points.
370To aid the musicologist in listening to the two parts of the song, one
371part is panned to the left speaker, and the other to the right (this
372was implemented using the extended audio API provided by Firefox, and
373so this particular feature only work when viewing the collection with
374this browser—see implementation details below).  In our figure,
375the musicologist has zeroed in the location <i>x=33</i>, <i>y=97</i>
376which corresponds to the start of a strong red diagonal that occurs
377some distance off the leading diagonal.  Listening to the two sounds
378played (most reliably done with headphones on), they hear that these
379two sections of the song are indeed repeating sections of the guitar
380piece <i>Candela</i> with a minor variation in the latter section
381where a recorder is also playing in the arrangement.</p>
383<p class="about">The structured audio time-lines (labelled A, B, ..., and 6, 5, 2,
384... in the figure) located above the self-similarity map are another
385area of enriched musical content in the digital library. The upper
386line shows the ground-truth data for this song generated by
387the <a href="" target="_blank">Salami project</a>;
388the lower line is generated by an algorithmically based content
389analysis algorithm.</p>
391<p class="about">While there is some agreement between these two lines, there are
392also significant differences.  The play and search buttons within the
393structured time-lines (the latter represented by a magnifying glass)
394allow the user to investigate further these structures.  We shall
395return to the search functionality shortly (which is content based,
396using <a href="" target="_blank">AudioDB</a>),
397but in the meantime note that with the time-lines positioned above the
398self-similarity map, there is further opportunity to study the
399differences between the two structured time-line.  It is certainly the
400case that there are strong visual cues in the map that line up with
401the algorithmic time-line, even though they do not align with a
402boundary in the ground-truth data, and the user can click on these
403parts of the similarity map to hear what is occurring at these
408<a name="resultspage" />
409<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
410  <tr>
411    <td style="border: solid 1px;">
412      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search-resultspage-halfcut.png" />
413    </td>
414    <td style="border: solid 1px;">
415      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-structured-search-resultspage-halfcut.png" />
416    </td>
417  </tr>
418  <tr style="background-color: #bbeebb">
419    <td colspan="2">
420      <i>Audio content based search results: left) the
421    <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=AudioQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.maxDocs=100&amp;s1.hitsPerPage=20&amp;q=D206.dir&amp;s1.query=D206.dir&amp;s1.offset=263&amp;s1.length=24&amp;mysongWindowDuration=6" target="_blank">result list</a> from
422      an audio content based query taken from an extract of </i>
423      Michelle; <i>and right)
424      the <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=TextQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.query=%22b+b+c+b+c%22&amp;s1.index=JS" target="_blank">result of a structured music search</a> for
425      content containing &quot;b b c b a&quot; as a sequence.</i>
426    </td>
427  </tr>
430<p class="about">Returning to the search capability provided by the structured
431time-lines, the <a href="#resultspage">above figure (left)</a> show the result of
432using this feature while study the song <i>Michelle</i> by The
433Beatles.  In this case the user selected the section of the
434ground-truth time-line corresponding to section starting &quot;I want you
435...&quot;, but could have equally used the algorithmically calculated
436time-line, or in fact paused the song playing at any point, and
437started a match from there.</p>
440<p class="about">Not surprisingly <i>Michelle</i> is returned as the top hit (at
44192.1%)—we shall see shortly that this is because the system found
442several sections of the song that matched this—the next hit being
443<i>Bigger Than JC</i> at 74.4%, and so on down, where only one hit
444per song occurs.  Clicking on the top hit,
445the <a href="#audiodb">figure below</a> shows the document view that is
446displayed, focusing in on the key area to this screen.  This time the
447time-line area has an additional bar: the points within the song that
448AudioDB found to be similar.  Like the other time-lines, a play button
449is present on these segments so the user can play the matching points
450directly.  In this case
451clicking on them reveals the matching sections found correspond to
452melodically repeating sections of the song, only with different lyrics
453(&quot;I love you ...&quot; and &quot;I need you ...&quot;).</p>
456<p class="about">A further form of musical content-based searching is available through
457the main header to the digital library.  Instead of searching by title
458or artist (which are also available) the user can click on this
459&quot;search by&quot; menu next to the quick-search text box and select
460&quot;ground-truth structure&quot; instead.
461The <a href="#resultspage">figure above</a> (right) shows the result of using this
462option, where the user has entered &quot;b b c b c&quot; as the query, in
463other words searching for songs that have two sections in a row the
464same, then a new section, then returning to the original section,
465before progressing to a recurrence of the second distinct section.  In
466pop and rock music this is a popular sequence corresponding to verse,
467verse, chorus, verse, chorus.  A more intriguing framing of a query
468along these lines—equally possible in the digital library—would be
469to go to the fielded search page (accessible through the <i>advanced
470  search</i> link below the quick-search box, and entered the same
471music structure query, but this time combine it with other fielded
472query terms (Genre=Blues OR Jazz, Year&lt;1950).  Results from this
473query would let the musicologist explore potential evidence of this
474pattern of sections being used in two of the key musical genres that
475influenced the development of Rock and Roll music.</p>
478<a name="audiodb" />
479<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
480  <tr>
481    <td style="border: solid 1px;">
482      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search2.png" />
483    </td>
484  </tr>
485  <tr style="background-color: #bbeebb">
486    <td>
487      <i>The document view for</i> Michelle <i><a href="dev?a=d&amp;c=salami-audioDB&amp;d=D206&amp;dt=simple&amp;p.frameOffset=133,393,264,262,392&amp;p.frameLength=24&amp;p.a=q&amp;p.s=AudioQuery&amp;hl=on&amp;ed=1#D206" target="_blank">augmented with the locations of the matches</a> of the audio query.</i>
488    </td>
489  </tr>
495<h3>Implementation details</h3>
497<p class="about">The core part of the interactive elements in the document view were
498implemented using SVG combined with Javascript.  The left- and
499right-panning interactively available from the self-similarity map was
500implemented by processing the raw audio stream, made accessible by the
501Firefox Audio extension
502<a href="" target="_blank">API</a>.</p>
504<p class="about">AudioDB content based searching was integrated into Greenstone through
505two components of the digital library software architecture: its
506build-time document processing plugin system, and its runtime
507message-passing service-base framework.  The developed plugin accepts
508a wide range of audio formats (including OGG and MP3), and converts
509them to WAV, the format needed by AudioDB for processing.  The new
510search service took the form of a proxy, accepting messages in the
511XML syntax used by Greenstone, turning them into the necessary calls
512to the AudioDB command-line interface, and then converting the output
513from AudioDB back into the XML syntax expected by the digital library
514architecture.  Finally, the two parts were packaged to operate as a
515Greenstone extension; the software is available at:
516<a href="" target="_blank"></a>.</p>
521    </xsl:template>
Note: See TracBrowser for help on using the browser.