root/main/trunk/model-sites-dev/multimodal-mdl/collect/salami-ground-truth/transform/pages/about.xsl @ 28747

Revision 28747, 21.9 KB (checked in by davidb, 6 years ago)

Mime type change to help with future diffs. Could be some other changes, but until the svn property is set, hard to see what they might be

  • Property svn:executable set to *
  • Property svn:mime-type set to text/plain
Line 
1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE videocollection [
3  <!ENTITY ndash "&#8211;">
4  <!ENTITY mdash "&#8212;">
5]>
6<xsl:stylesheet version="1.0"
7    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
8    xmlns:java="http://xml.apache.org/xslt/java"
9    xmlns:util="xalan://org.greenstone.gsdl3.util.XSLTUtil"
10    xmlns:gslib="http://www.greenstone.org/skinning"
11    extension-element-prefixes="java util"
12    exclude-result-prefixes="java util">
13
14    <!-- use the 'main' layout -->
15    <xsl:include href="layouts/main.xsl"/>
16
17    <!-- set page title -->
18    <xsl:template name="pageTitle"><gslib:collectionName/></xsl:template>
19
20    <!-- set page breadcrumbs -->
21    <xsl:template name="breadcrumbs"><gslib:siteLink/><gslib:rightArrow/></xsl:template>
22
23    <!-- the page content -->
24    <xsl:template match="/page">
25 
26        <!--Display the description text of the current collection,
27        and if some services are available then create a list
28        of links for each service within a <ul id="servicelist"> element.-->
29
30    <xsl:variable name="httpPath">
31      <xsl:value-of select="/page/pageResponse/collection/metadataList/metadata[@name='httpPath']" />
32    </xsl:variable>
33
34
35    <style>
36      p.about { text-align: justify; }
37      h2 { /* line-height: 135%; */ }
38      h3 { margin-top: 4pt; margin-bottom: 7pt; line-height: 135%; }
39
40        a.mashup {text-decoration: none; }
41    a.mashup:link    {color:#000080;}  /* unvisited link */
42    a.mashup:visited {color:#303080;}  /* visited link */
43    a.mashup:hover   {color:#000040;}  /* mouse over link */
44    a.mashup:active  {color:#0000a0;}
45
46    </style>
47
48
49    <script src="{$httpPath}/script2/browser-detect.js" type="text/javascript">
50    /* space filler needed */
51    </script>
52
53
54<div style="width: 800px; margin-left: auto; margin-right: auto;">
55
56
57<h2>Structured Audio in a Music Digital Library</h2>
58
59<!--
60<div style="background-color: #DD3611; color: white; padding: 4pt; margin-bottom: 6pt;">
61<table>
62
63  <tr>
64    <td>
65       <img src="{$httpPath}/images/firefox-128.png" />
66    </td>
67    <td>
68    <p class="about">
69      <i><font size="+2">
70        An experiment with Firefox's Advanced
71        Audio Processing Extension ...
72      </font></i>
73    </p>
74
75    <p class="about">
76      <i><font size="+2">
77... featuring a mash-up of
78<a class="mashup" href="http://www.greenstone.org" target="_blank">Greenstone</a>,
79<a class="mashup" href="http://www.omras2.org/audioDB" target="_blank">AudioDB</a>, and
80<a class="mashup" href="http://seasr.org/meandre/" target="_blank">Meandre</a>.
81      </font></i>
82    </p>
83
84
85          <div id="bd-version" style="display:none;" class="about">
86        <p>
87          <img src="{$httpPath}/images/warning-icon.png" />
88            Version of Firefox detected:
89              <script type="text/javascript">
90                document.write(BrowserDetect.version);
91              </script>
92        </p>
93        <p>
94            You need to upgrade to a more recent version of Firefox
95            (v4.0 or greater) to access all the audio features of
96            this demonstration.
97        </p>
98          </div>
99
100          <div id="bd-browser" style="display:none;" class="about">
101
102        <p>
103          <img src="{$httpPath}/images/warning-icon.png" />
104Browser detected:
105          <script type="text/javascript">
106            document.write(BrowserDetect.browser);
107          </script>
108        </p>
109            <p>You need to access this web-site using Firefox
110            (v4.0 or greater) to access all the audio features provided.
111        </p>
112          </div>
113
114
115
116
117  <script>
118    <xsl:text disable-output-escaping="yes">
119
120if (BrowserDetect.browser == "Firefox") {
121  if (BrowserDetect.version &lt; 4) {
122    document.getElementById("bd-version").style.display="block";
123
124  }
125}
126else {
127  document.getElementById("bd-browser").style.display="block";
128
129}
130      </xsl:text>
131    </script>
132
133
134
135
136    </td>
137  </tr>
138</table>
139</div>
140-->
141
142<p class="about">
143This digital library collection is derived from the American workset,
144from the <a href="http://salami.lis.illinois.edu">SALAMI</a> project.
145
146</p>
147
148<p class="about">This music DL embodies the following:
149
150<style>
151  li { padding-bottom: 10px; }
152
153</style>
154
155
156<ol>
157  <li>
158  <i>Starting point:</i> Manual (command-line) aggregation of disparate
159    resources prior to building the DL.  A set of bespoke Greenstone
160    document processing plugins corrals the heterogeneous gathered
161    data into a unified (homogeneous) canonical form that the DL can
162    access and display.  Everything presented in the DL is
163    either pre-computed (such as the self-similarity heat-maps) or else
164    computed at build-time.
165  </li>
166
167   <li>
168     <i>Audio-fingerprinting:</i> as before, but now metadata about the audio
169     songs is now enriched through a set of audio-fingerprinting web
170     services.  Everything presented in the DL is still pre-computed
171     or else computed at build-time—however,  the inclusion in the document
172     view of a &quot;Discovery&quot; block allows a user to begin to
173     access and explore through linked-data, information related to
174     the song.
175   </li>
176
177  <li>
178    <i>Client-side audio processing (and visualization):</i> the
179    Pre-computed self-similarity heat-maps are dropped from the
180    collection building process in preference for the same information
181    being computed through Javascript running in the user's web
182    browser.
183  </li>
184
185  <li>
186    <i>Embedded Meandre workflows:</i> the Meandre Workbench is
187    integrated into Greenstone.  Audio documents in the Greenstone
188    digital library can now be dispatched and be processed by the
189    selected Meandre, and output from the workflow returned to the
190    Greenstone document view—for example, playing audio that has been
191    processed and output from the workflow.
192  </li>
193
194  <li>
195    <i>Client-side workflows:</i> the client-side Javascript
196    processing code is refactored to follow the same methodology as
197    Meandre components.
198  </li>
199
200  <li>
201    <i>Client/Server hybrid workflows:</i> the Greenstone/Meandre
202    integration is extended to support the dynamic
203    transmission of <tt>executeCallBack()</tt> methods written
204    in the user's browser in Javascript to be run on the
205    Meandre server as part of the active workflow.
206  </li>
207
208  <li>
209    <i>Forging ahead:</i> the next area to be worked on is upgrading
210    the level of Greenstone/Meandre integration so that data produced
211    by the workflows can be incorporated back into the underlying
212    digital library itself, rather than being (as it currently stands)
213    transitory data that only lives for the duration of the web page
214    being viewed.  This will form an implementation stepping-stone to
215    a more generalized ability to have data retrieved from other
216    external resources (located through the Discovery block
217    linked-data portion of the document view) ingested in to the DL
218    collection.
219  </li>
220
221
222</ol>
223
224</p>
225
226
227
228<!--
229
230<p class="about">This digital library collection demonstrates the
231integration of a variety of sources of structural metadata from both
232automatically derived content analysis and manually labelled
233ground-truth data to form a rich interactive web application suitable
234for musicologists (as the target end-users) to explore a collated set
235of music files.</p>
236
237<p class="about">While the base digital library system, <a
238href="http://www.greenstone.org" target="_blank">Greenstone</a>, is
239designed to operate with any web browser, the audio features this
240particular DL collection demonstrates relies on audio processing
241features currently only available in Firefox.  Using one of the other
242web browsers you will be able to access most features of the digital
243library collection, but when you access a particular song you may find
244playing the audio—in particular the panned audio
245effect—does not work.</p>
246-->
247
248
249<!--
250<p class="about">The developed system combines the open source capabilities of
251Greenstone, \cite{Greenstone}, NEMA \cite{Nema}, Salami \cite{Salami},
252and AudioDB \cite{AudioDB}.
253
254%%
255NEMA and Salami have been previously described in this article, and contribute
256manual and audio content based metadata.
257%%
258AudioDB is a raw-audio content based searching algorithm based on Local-Sensitivity Hashing \cite{LSH}.
259%%
260Greenstone is a versatile digital library architecture with an
261extensible service-based architecture that.  In
262addition to the text-based searching and browsing capability to
263organize content, Greenstone provides the framework in which to harness the
264structural metadata from NEMA and Salami, and the audio-content based
265search functionality of audioDB.
266
267-->
268
269
270
271<h3>A Walkthrough</h3>
272
273
274<p class="about">The following walkthrough is for the initial incarnation of the DL collection, where all
275the information presented is either precomputed or computed at build time.</p>
276
277<p class="about">Taking as a starting point a set of music files identified as worthy
278of study, the <a href="#browse">figure below</a> shows the result of browsing
279the formed digital library collection <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by title</a> from a web browser.
280The figure is a useful snapshot in which to orientate ourselves with
281the main structure and features to the digital library.  Functionality that
282persistently reoccurs is accessible through the header to the page.</p>
283
284<p class="about">This includes:
285<ul>
286
287<li>help and preferences (top-right);</li>
288<li>a quick-search option (located just
289below) with links to more sophisticated searching options; and</li>
290<li>pin-pointing where within the site a user is currently located
291(top-left).</li>
292</ul>
293
294
295<a name="browse" />
296<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
297  <tr>
298    <td style="border: solid 1px;">
299      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-browse.png" />
300    </td>
301  </tr>
302  <tr style="background-color: #bbeebb">
303    <td>
304      <i>Browsing in the digital library
305  <a href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1" target="_blank">by titles</a>.</i>
306    </td>
307  </tr>
308</table>
309
310
311The specific content to this location within the site (in this case
312browsing by title) is shown beneath the main banner.  Various
313groupings of title can be accessed by clicking on the bookshelf icons
314vertically aligned as the main part of the page: currently <a
315href="dev?a=b&amp;rt=s&amp;s=ClassifierBrowse&amp;c=salami-audioDB&amp;cl=CL1#CL1.2">C–D</a>
316is open, with the remaining letters to the alphabet below this,
317accessed through scrolling.</p>
318
319
320
321
322<p class="about">Interested in the song <i>Candela</i> our curious musicologist clicks
323on <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;p.sa=&amp;p.s=ClassifierBrowse" target="_blank">the link for this</a>.  This brings up the document view to this song:</p>
324
325<a name="self-similarity" />
326<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
327  <tr>
328    <td style="border: solid 1px;">
329      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-self-similarity2.png" />
330    </td>
331  </tr>
332  <tr style="background-color: #bbeebb">
333    <td>
334      <i>The <a href="dev?a=d&amp;ed=1&amp;book=off&amp;c=salami-audioDB&amp;d=D145&amp;dt=simple&amp;sib=1&amp;p.a=b&amp;p.sa=&amp;p.s=ClassifierBrowse" target="_blank">musicologically enriched document view</a> for</i> Candela.
335    </td>
336  </tr>
337</table>
338
339<p class="about">Normally in a digital
340library the document view brings up a page that is strongly derived
341from textual metadata.  If the document viewed was a text-document,
342some summary information such as title and author is typically
343presented, say in tabular form, before the main text is presented.
344Even in the case of multimedia digital libraries, the view presented
345is still strongly derived from textual metadata: this time including
346details such as the length of the video, the TV company that produced
347it, whether captions are available, and so forth, accompanied with an
348embedded video player for viewing the content—essentially more
349textual metadata (in this case the URL to the video content) which in
350terms of the user-interface is largely divorced from the other
351elements displayed on the page.</p>
352
353<p class="about">This contrasts sharply with the document view developed in this
354digital library.
355Naturally it allows the song to be played (akin to the embedded video player),
356however this is largely of secondary importance to the
357other functionality available this is much more closely
358integrated.</p>
359
360<p class="about">The most striking visual component to the document view is a
361self-similarity &quot;heat map&quot; where the duration of the song forms both
362the <i>x</i>- and <i>y</i>-axis, and a red pixel located at a given <i>(x,y)</i>
363co-ordinate in the map represents a location where two parts of the
364song are strongly similar, proportionally shifting to blue to
365represent dissimilar.  Given this configuration, the leading diagonal
366to the matrix (<i>x=y</i>) is always coloured red as this represents the
367comparison of one part of the song with itself.</p>
368
369<p class="about">When the user moves the mouse cursor around the self-similarity map
370a highlighting circle is shown to emphasize the area the user is over,
371with a black dot at the centre (visible in in
372the <a href="#self-similarity">above figure</a>); annotated vertically
373and horizontally are the two time-offsets in seconds that that point
374in the map corresponds to.  Clicking the cursor at this point results
375in the audio being played <i>simultaneously</i> from these two points.
376To aid the musicologist in listening to the two parts of the song, one
377part is panned to the left speaker, and the other to the right (this
378was implemented using the extended audio API provided by Firefox, and
379so this particular feature only work when viewing the collection with
380this browser—see implementation details below).  In our figure,
381the musicologist has zeroed in the location <i>x=33</i>, <i>y=97</i>
382which corresponds to the start of a strong red diagonal that occurs
383some distance off the leading diagonal.  Listening to the two sounds
384played (most reliably done with headphones on), they hear that these
385two sections of the song are indeed repeating sections of the guitar
386piece <i>Candela</i> with a minor variation in the latter section
387where a recorder is also playing in the arrangement.</p>
388
389<p class="about">The structured audio time-lines (labelled A, B, ..., and 6, 5, 2,
390... in the figure) located above the self-similarity map are another
391area of enriched musical content in the digital library. The upper
392line shows the ground-truth data for this song generated by
393the <a href="http://www.music-ir.org/?q=node/14" target="_blank">Salami project</a>;
394the lower line is generated by an algorithmically based content
395analysis algorithm.</p>
396
397<p class="about">While there is some agreement between these two lines, there are
398also significant differences.  The play and search buttons within the
399structured time-lines (the latter represented by a magnifying glass)
400allow the user to investigate further these structures.  We shall
401return to the search functionality shortly (which is content based,
402using <a href="http://omras2.doc.gold.ac.uk/software/audiodb/" target="_blank">AudioDB</a>),
403but in the meantime note that with the time-lines positioned above the
404self-similarity map, there is further opportunity to study the
405differences between the two structured time-line.  It is certainly the
406case that there are strong visual cues in the map that line up with
407the algorithmic time-line, even though they do not align with a
408boundary in the ground-truth data, and the user can click on these
409parts of the similarity map to hear what is occurring at these
410points.</p>
411
412
413
414<a name="resultspage" />
415<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
416  <tr>
417    <td style="border: solid 1px;">
418      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search-resultspage-halfcut.png" />
419    </td>
420    <td style="border: solid 1px;">
421      <img style="width: 350px" src="{$httpPath}/images/figs/cropped/salami-structured-search-resultspage-halfcut.png" />
422    </td>
423  </tr>
424  <tr style="background-color: #bbeebb">
425    <td colspan="2">
426      <i>Audio content based search results: left) the
427    <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=AudioQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.maxDocs=100&amp;s1.hitsPerPage=20&amp;q=D206.dir&amp;s1.query=D206.dir&amp;s1.offset=263&amp;s1.length=24&amp;mysongWindowDuration=6" target="_blank">result list</a> from
428      an audio content based query taken from an extract of </i>
429      Michelle; <i>and right)
430      the <a href="dev?a=q&amp;sa=&amp;rt=rd&amp;s=TextQuery&amp;c=salami-audioDB&amp;startPage=1&amp;s1.query=%22b+b+c+b+c%22&amp;s1.index=JS" target="_blank">result of a structured music search</a> for
431      content containing &quot;b b c b a&quot; as a sequence.</i>
432    </td>
433  </tr>
434</table>
435
436<p class="about">Returning to the search capability provided by the structured
437time-lines, the <a href="#resultspage">above figure (left)</a> show the result of
438using this feature while study the song <i>Michelle</i> by The
439Beatles.  In this case the user selected the section of the
440ground-truth time-line corresponding to section starting &quot;I want you
441...&quot;, but could have equally used the algorithmically calculated
442time-line, or in fact paused the song playing at any point, and
443started a match from there.</p>
444
445
446<p class="about">Not surprisingly <i>Michelle</i> is returned as the top hit (at
44792.1%)—we shall see shortly that this is because the system found
448several sections of the song that matched this—the next hit being
449<i>Bigger Than JC</i> at 74.4%, and so on down, where only one hit
450per song occurs.  Clicking on the top hit,
451the <a href="#audiodb">figure below</a> shows the document view that is
452displayed, focusing in on the key area to this screen.  This time the
453time-line area has an additional bar: the points within the song that
454AudioDB found to be similar.  Like the other time-lines, a play button
455is present on these segments so the user can play the matching points
456directly.  In this case
457clicking on them reveals the matching sections found correspond to
458melodically repeating sections of the song, only with different lyrics
459(&quot;I love you ...&quot; and &quot;I need you ...&quot;).</p>
460
461
462<p class="about">A further form of musical content-based searching is available through
463the main header to the digital library.  Instead of searching by title
464or artist (which are also available) the user can click on this
465&quot;search by&quot; menu next to the quick-search text box and select
466&quot;ground-truth structure&quot; instead.
467The <a href="#resultspage">figure above</a> (right) shows the result of using this
468option, where the user has entered &quot;b b c b c&quot; as the query, in
469other words searching for songs that have two sections in a row the
470same, then a new section, then returning to the original section,
471before progressing to a recurrence of the second distinct section.  In
472pop and rock music this is a popular sequence corresponding to verse,
473verse, chorus, verse, chorus.  A more intriguing framing of a query
474along these lines—equally possible in the digital library—would be
475to go to the fielded search page (accessible through the <i>advanced
476  search</i> link below the quick-search box, and entered the same
477music structure query, but this time combine it with other fielded
478query terms (Genre=Blues OR Jazz, Year&lt;1950).  Results from this
479query would let the musicologist explore potential evidence of this
480pattern of sections being used in two of the key musical genres that
481influenced the development of Rock and Roll music.</p>
482
483
484<a name="audiodb" />
485<table style="width: 700px; margin-left: auto; margin-right: auto; margin-bottom: 6pt;">
486  <tr>
487    <td style="border: solid 1px;">
488      <img style="width: 700px" src="{$httpPath}/images/figs/cropped/salami-audiodb-search2.png" />
489    </td>
490  </tr>
491  <tr style="background-color: #bbeebb">
492    <td>
493      <i>The document view for</i> Michelle <i><a href="dev?a=d&amp;c=salami-audioDB&amp;d=D206&amp;dt=simple&amp;p.frameOffset=133,393,264,262,392&amp;p.frameLength=24&amp;p.a=q&amp;p.s=AudioQuery&amp;hl=on&amp;ed=1#D206" target="_blank">augmented with the locations of the matches</a> of the audio query.</i>
494    </td>
495  </tr>
496</table>
497
498
499
500
501<h3>Implementation details</h3>
502
503<p class="about">The core part of the interactive elements in the document view were
504implemented using SVG combined with Javascript.  The left- and
505right-panning interactively available from the self-similarity map was
506implemented by processing the raw audio stream, made accessible by the
507Firefox Audio extension
508<a href="https://wiki.mozilla.org/Audio_Data_API" target="_blank">API</a>.</p>
509
510<p class="about">AudioDB content based searching was integrated into Greenstone through
511two components of the digital library software architecture: its
512build-time document processing plugin system, and its runtime
513message-passing service-base framework.  The developed plugin accepts
514a wide range of audio formats (including OGG and MP3), and converts
515them to WAV, the format needed by AudioDB for processing.  The new
516search service took the form of a proxy, accepting messages in the
517XML syntax used by Greenstone, turning them into the necessary calls
518to the AudioDB command-line interface, and then converting the output
519from AudioDB back into the XML syntax expected by the digital library
520architecture.  Finally, the two parts were packaged to operate as a
521Greenstone extension; the software is available at:
522<a href="http://trac.greenstone.org/gs3-extensions/audioDB/trunk/src" target="_blank">http://trac.greenstone.org/gs3-extensions/audioDB/trunk/src</a>.</p>
523
524</div>
525
526
527    </xsl:template>
528
529
530</xsl:stylesheet> 
531
Note: See TracBrowser for help on using the browser.