| Independent | |
| Master | |
| Slave[0] | |
| Slave[1] |
In previous tutorials, we looked at how multiple files can be played simultaneously using multiple audio controllers. However, little was said about synchronizing the "music". We will look at that issue in this tutorial.
If one reads the proposed HTML5 standard, one might expect this to be easy because the standard specifies a mediaGroup attribute for the <audio ...> tag and a MediaController interface for Javascript (and hence Processing.js). The purpose of these items are to synchronize multiple media elements. Unfortunately as of the time this tutorial was written, none of the 5 major browsers has implemented mediaGroup. Only Chrome recognizes the MediaController interface but it doesn't seem to work correctly. Hopefully these elements will eventually be implemented and simplify synchronization. But in the mean time, is there any way to achieve this goal?
This tutorial suggests a technique where one media controller is designated as the "master" and additional controllers as "slaves". Seemingly, this should be straight forward. For example, one could listen for a play event on the master controller which would call a method whenever the master started to play. That method could then tell the slaves to start playing. The pause event could be used in a similar manner. The master's currentTime could be used to set the slaves' current time to make sure that they are in sync. Unfortunately, it is not quite that simple in practice. In particular, assigning the master's currentTime to the slaves may not synchronize them as well as expected. This is probably due to the asynchronous nature of the Audio controllers and/or time delays while caring out the procedure.
KISA 8 uses a fairly simple MediaSynchronizer class that attempts to keep the controllers in sync. Unfortunately it is not perfect but may be satisfactory especially if perfect synchronization is not essential. It success may depend on several factors including the browser, the CPU and computer, the operating system, and other undetermined factors. It generally works very well in Internet Explorer 9. It is adequate in many situations for the other 4 browsers because synchronization may not be as critical as the test used in this demo. In any case, it is presented in a "use at your own risk" manner.
The KISA 8 demo may help you decide if the concept is adequate. The first "independent" player uses a single controller to play an audio file. The master controller plays the same file on itself and 3 additional slave controllers. The first two slaves are declared in HTML5. The third slave, "slave[2]", is declared in Processing.js. Its limited controls appear at the bottom of the Processing.js canvas. To play or pause the file using master/slave system, you can use the "Master" controller or the "play" and "pause" buttons. Likewise, the volume and current time can be set using the master controller or the corresponding blue buttons.
Lets begin by looking at the MediaSynchronizer class. Even though the present code applies to the Audio class, the word "Media" was used because it believed that the class would applied to video controls by just replacing "Audio" by "Video" and "audio" by "video". (The HTML5 video controller uses the same audio interface).
The MediaSynchronizer allows the master to control the slaves playing, pausing, ending, volume, and currentTime. In many applications would be desirable to add mute to the list of controlled items. It was not done in this case because it was useful to be able to mute individual audio controllers while testing. (At one point during testing, it was discovered that slaves were muted for unknown reasons.) Loading the Audio components was not controlled because it is assumed that the audio file will be automatically load using the default auto setting for preload attribute. Additional items could be controlled, if needed.
The class begins be declaring a master Audio component and an array of slaves. These values will be passed in. The slaves array was initially declared to have only 1 component as we will be taking advantage of Javascript's (and hence Processing.js') ability to increase the size of an array when more slaves are needed.
176 class MediaSynchronizer { 177 Audio master; 178 Audio[] slaves = new Audio[1];
A tolerance value closeEnough is also declared. The components will be resynced only if the difference between the master and a slave is greater than this value in seconds. The value 0.150 seconds is arbitrary and may seem too large. Surprisingly, testing has shown than values less than 0.100 seconds may actually make matters worse instead of better in some browsers. We will discuss this value later.
181 // number of milliseconds variance allowed before resyncing a slave
The final 3 class variables are maxDelta, sumDelta, and countDelta. In this program, delta refers to the difference between the currentTime of the master and of a slave. These differences are measured each time a timeupdate event occurs. Experience has shown that the timeupdate event typically occurs between 4 and 5 times a second although the standard would allow a larger range. The variable maxDelta is the largest (in magnitude) "delta" that occurs at any one timeupdate. Variable countDelta counts the number of such events and sumDelta sums the maxDeltas. The getAveDelta function determines the average of the maximums. These values were added for testing purposes and could be deleted in an actual production program.
The constructor has two arguments, the master audio component and the first slave which are stored appropriately. Additional slaves can be added with setSlave method. The other job of the constructor is to add several event listeners that respond to events in the master Audio component.
188 189 MediaSynchronizer(Audio m, Audio s) { 190 // The constructor. m is the master controller, 191 // s is the first slave. Additional slaves can be added 192 // using the "setSlave' method. 193 master = m; 194 slaves[0] = s; 195 master.addEventListener("play", playEH, false); 196 master.addEventListener("pause", pauseEH, false); 197 master.addEventListener("ended", endedEH, false); 198 master.addEventListener("timeupdate", updateEH, false); 199 master.addEventListener("volumechange", volumeEH, false); 200 } // constructor MediaSynchronizer
The setSlave method substitutes for 3 different methods. It can be used to replace, add, or delete a slave. The first parameter, item, specifies the slave number - the subscript in the array of slaves. (The subscripts begin with 0.) To replace a given slave, just specify the item number and the new slave. To add a slave, use a unused item number. To delete an existing slave, specify null as the second parameter.
202 void setSlave(int item, Audio s) { ... 209 if (item < 0 || item > slaves.length) 210 item = slaves.length; 211 slaves[item] = s; 212 } // setSlave
The "add a new Audio component" operation takes advantage of Javascript's (and hence Processing.js's) ability increase the size of an array. Because this method is very simple, it should be used very carefully. It does make sure that new slaves are added immediately at the end of the array. For efficiency sake, users are encouraged to replace any null components before adding new elements to the array.
The getMaxDelta and getsumDelta methods simply return statistical values that can be used to determine how well the components are synchronized.
The remaining methods are suffixed with "EH" to signify that they are event handlers and should only be called in response to events.
The playEH, pauseEH, endedEH and volumeEH event handlers handle the corresponding master events. All of these ignore any slaves that have been deleted by being replaced by "null". Whenever the master starts to play, playEH starts the slaves playing. It also resets that statistical variables. pauseEH pauses the slaves when the master is paused. In case the master ends before a slave, endedEH pauses all the slaves. While testing this routine, at one time it also set the currentTime for both the master and slaves back to zero after pausing the slaves in hopes that the next play would be better synchronized but it didn't seem to help on a regular basis. The playEH and endedEH methods are shown here because they somewhat more complicated than the other two.
219 void playEH() { 220 // Used internally in response to a play event on the master 221 // component. The slave components will start to play. 222 int j; 223 for (j = 0; j < slaves.length; j++) { 224 if (slaves[j] != null) { 225 slaves[j].play(); 226 sumDelta = 0; 227 countDelta = 0; 228 } 229 } 230 } // playEH ... 242 void endedEH() { 243 // Used internally to respond to a ended event on the master. 244 // Pauses all of the components (master and slaves) 245 // in response to a "ended" event on the master. 246 int j; 247 master.pause(); 248 for (j = 0; j < slaves.length; j++) { 249 if (slaves[j] != null) { 250 slaves[j].pause(); 251 } 252 } 253 } // endedEH
The updateEH is one of the most critical methods and also the one most subject to experimentation to see what seems to work best. It is called in response to a timeupdate event which typically happens 4 or 5 times a second. Unfortunately the variation between browsers and a host of other factors make it difficult to determine the "best" method. Initially it was thought the slaves should be synched to the master on a regular basis. However experimentation showed that regular synchronization often made the problem worse. The current version only resyncs the slaves if they are significantly off. This is most likely to happen when the currentTime of the master is changed using its control or the "Time down" or "Time up" buttons.
Each time this method is called, it iterates though all of the slaves. The variable delta is used to measure the difference in the current time of the master and a particular slave (in seconds). If this difference is too large, then the slave is resynced. The question is what is "too large"? The value closeEnough specifies the maximum delta (in seconds) that is allowed before resyncing. Intuition tells us that it should be small - for example, 0.050 seconds or less. However, experimentation suggests that vales less than 0.100 are apt to increase the problem. On the other hand, larger values of closeEnough tend to make it difficult to reset the slaves when the master's control current time is reset manually. The current value 0.150 is an arbitrary compromise.
255 void updateEH() { 256 // Used internally to attempt to resyncronize the slaves with 257 // the master if they become out of sync. 258 // Experimentally it appears best not to resync the slaves unless 259 // they are badly out of sync. 260 int j; 261 double mTime = master.currentTime; 262 263 maxDelta = 0; 264 for (j = 0; j < slaves.length; j++) { 265 if (slaves[j] != null) { 266 double sTime = slaves[j].currentTime; 267 double delta = mTime - sTime; 268 double absDelta = abs(delta); 269 maxDelta = max(maxDelta, absDelta); 270 if ((absDelta > closeEnough) && (mTime < slaves[0].duration)) { 271 slaves[j].currentTime = master.currentTime; 272 } 273 } 274 } 275 sumDelta += maxDelta; 276 countDelta++; 277 } // updateEH
Previous KISA tutorials were built by revising and expanding the previous tutorial. That didn't seem appropriate for testing the MediaSynchronizer. So KISA 8 starts over although it uses many of the concepts from previous tutorials. It seemed desirable to verify that the class worked with both HTML5 audio controls and Audio components declared in Processing.js. The KISA 8 web page declares the master and the first 2 slaves. The 3rd slave is declared in the .pde file.
Another serious question was what test file(s) to use. In most applications one would want to synchronize two or more distinct but related files. Unfortunately, I don't have any appropriate files available. So instead, I decided to use one file played simultaneously on multiple Audio components. The "groove" file was selected as it is relatively short and had sound that requires adequate synchronization to avoid "echos" It is intended to allow determining if the synchronizing was adequate by just listing to the result. It is felt that using this file provides a very stringent test for this concept.
The HTML file provides one separate control called "Independent" so one can tell what the file sound file should sound like when played by itself so it can be compared to the combined sound produced by master and 3 slave components playing at the same time.
Because the interface should be familiar despite it differences, we will only touch on significant parts.
In the .html file, four audio controls are declared in a similar manner - the "Independent", the "Master", and the first two slaves, Slave[0] and Slave[1]. As in the first tutorial, the internal <source ...> tags are used so the audio component can pick between a .ogg and .mp3 file as required by the browser. An distinct id is specified for master and each slave so they can be accessed by .pde code. To avoid loading delays as much as possible, preload = "auto" attribute is specified. (Actually it is the default.) For example, the master is specified as follows in KISA8Tutorial.html:
31 <audio id = "master" preload = "auto" controls> 32 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 33 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 34 Your browser does not support HTML 5 audio. You may want to update your 35 browser to a current version. 36 </audio>
The .pde file declares four Audio components and an array into which they can be put. The first three are obtained from the .html page by specifying their id. The last one is declared entirely in the .pde file. The MediaSynchronizer is also declared as the class variable ms. (The .pde file does not interface with the independent audio element.)
19 Audio audioMaster = document.getElementById("master"); 20 Audio audioSlave0 = document.getElementById("slave0"); 21 Audio audioSlave1 = document.getElementById("slave1"); 22 Audio audioSlave2 = new Audio(); 23 Audio[] audioArray = new Audio[4]; ... 35 MediaSynchronizer ms = new MediaSynchronizer(audioMaster, audioSlave0);
The setup method includes code to set the source of the last slave and add the 2nd and 3rd slaves to the MediaSynchronizer ms. The audio components are then added to the audioArray as a convenience and so they can be used with the display. (The same display used in tutorials 6 and 7.)
47 setSource(audioSlave2, source); // specify source for audioSlave2" 48 ms.setSlave(1, audioSlave1); // adds audioSlave1 to MediaSynchronizer 49 ms.setSlave(2, audioSlave2); // adds audioSlave2 50 51 audioArray[0] = audioMaster; 52 audioArray[1] = audioSlave0; 53 audioArray[2] = audioSlave1; 54 audioArray[3] = audioSlave2;
Event listeners are established for the 3rd slave in setup. These listeners allow the current time for this slave to be displayed and turn on and off the play toggle button light for this slave. (Note: One should not specify event listeners for the master Audio component in the main sketch because they are set in the MediaSunchronizer class code.)
56 audioArray[3].addEventListener("play", toggle2On, false); 57 audioArray[3].addEventListener("pause", toggle2Off, false); 58 audioArray[3].addEventListener("ended", toggle2Off, false);
The draw procedure is pretty much as expected except for statistical testing output which will be discussed later.
The sketch shows the "Volume" of the Master and the 3 slaves. The values can range from 0 to 1. This coding was added to insure that changing the master volume was adjusting the volume of the 3 slaves correctly. The 4 numbers should always be the same.
The toggle2On and toggle2Off methods turn the slave2PlayBtn "light" button on and off in response to slave[2] starting to play or being paused or ended.
The code in mouseClicked method is pretty much routine. Please note that the code for the Play, Pause, Volume down, Volume up, Time down, and Time up buttons controls the audioMaster while the code for the Slave[2] Play and Slave[2].Mute buttons only applies to the last slave.
A considerable amount of time was spend determining ways to test the MediaSynchronizer. As mentioned earlier, some intuition about things that should have helped did not pan out. Problems include the fact that the results are different in different browsers and often depend on unexplained factors.
Three techniques have been used during testing:
We assume that files are loaded before they are played. Internet Explorer often synchronizes playback completely. What happens in other situations? The MediaSynchronizer uses the timeupdate event to check the synchronization of the Audio components. This typically happens 4 to 5 times a second. Recall that in our discussion of the MediaSynchronizer we defined delta to be the difference between current time of master and a slave. The sketch shows the maximum delta for most recent of those checks. It also shows the average of those maximums. Typically when the max delta is less than 0.030 seconds we can't hear any problem. If the max delta is over 0.050 seconds we normally can hear that synchronization is not good. Fortunately that doesn't happen to often.
Is the master/slave technique in Tutorial 8 better than those used in Tutorial 7? It depends. If one wants to the user to control multiple audio players using a default controller provided by HTML5's <audio controls ...> tag, the techniques in this tutorial may be the only option. On the other hand if the Processing.js code is being used to declare and provide access to the audio players, the techniques used in Tutorial 7 might be preferred. The code could just make the changes for each of the audio components directly instead of trying to use the master/slave concept. It is apt to provide better synchronization.
These tutorials have not illustrated all the possible options in the HTML 5 audio API. If you want additional possibilities, you can look for techniques used in Javascript. You should find they can be used in Processing.ps with minimal changes. The page HTML5 Audio Demo may be useful. This page demonstrates many of the a Audio API. The demo is just a much glorified version of KISA 5 except the "loop" button just sets the Audio component's loop attribute.
1 <!DOCTYPE html> 2 <html> 3 <head> 4 <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 5 <title>KISA Part 8 (Keep It Simple Audio)</title> 6 <script type="text/javascript" src="processing.min.js"></script> 7 </head> 8 <body> 9 <h1>Keep It Simple Audio 8<br> 10 Synchronizing Audio Components</h1> 11 12 <table> 13 <tr> 14 <td> 15 Independent 16 </td> 17 <td> 18 <audio preload="auto" controls> 19 <source src = "http://brinkje.com/KISA_Sounds//groove.ogg" type="audio/ogg" /> 20 <source src = "http://brinkje.com/KISA_Sounds//groove.mp3" type="audio/mpeg" /> 21 Your browser does not support HTML 5 audio. You may want to update your 22 browser to a current version. 23 </audio> 24 </td> 25 </tr> 26 <tr> 27 <td> 28 Master 29 </td> 30 <td> 31 <audio id = "master" preload = "auto" controls> 32 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 33 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 34 Your browser does not support HTML 5 audio. You may want to update your 35 browser to a current version. 36 </audio> 37 </td> 38 </tr> 39 <tr> 40 <td> 41 Slave[0] 42 </td> 43 <td> 44 <audio id ="slave0" preload = "auto" controls> 45 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 46 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 47 Your browser does not support HTML 5 audio. You may want to update your 48 browser to a current version. 49 </audio> 50 </td> 51 </tr> 52 <tr> 53 <td> 54 Slave[1] 55 </td> 56 <td> 57 <audio id ="slave1" preload = "auto" controls> 58 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 59 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 60 Your browser does not support HTML 5 audio. You may want to update your 61 browser to a current version. 62 </audio> 63 </td> 64 </tr> 65 </table> 66 <canvas id="KISA" data-processing-sources="KISA8.pde Display.pde CircleButton.pde"> 67 Your browser does not support HTML 5. You may want to update your 68 browser. 69 </canvas> 70 <p> 71 </p> 72 <p> 73 </p> 74 <p>Updated 11/1/14</p> 75 </body> 76 </html>
1 <!DOCTYPE html> 2 <html> 3 <head> 4 <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 5 <title>KISA Part 8 (Keep It Simple Audio)</title> 6 <script type="text/javascript" src="processing.min.js"></script> 7 </head> 8 <body> 9 <h1>Keep It Simple Audio 8<br> 10 Synchronizing Audio Components</h1> 11 12 <table> 13 <tr> 14 <td> 15 Independent 16 </td> 17 <td> 18 <audio preload="auto" controls> 19 <source src = "http://brinkje.com/KISA_Sounds//groove.ogg" type="audio/ogg" /> 20 <source src = "http://brinkje.com/KISA_Sounds//groove.mp3" type="audio/mpeg" /> 21 Your browser does not support HTML 5 audio. You may want to update your 22 browser to a current version. 23 </audio> 24 </td> 25 </tr> 26 <tr> 27 <td> 28 Master 29 </td> 30 <td> 31 <audio id = "master" preload = "auto" controls> 32 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 33 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 34 Your browser does not support HTML 5 audio. You may want to update your 35 browser to a current version. 36 </audio> 37 </td> 38 </tr> 39 <tr> 40 <td> 41 Slave[0] 42 </td> 43 <td> 44 <audio id ="slave0" preload = "auto" controls> 45 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 46 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 47 Your browser does not support HTML 5 audio. You may want to update your 48 browser to a current version. 49 </audio> 50 </td> 51 </tr> 52 <tr> 53 <td> 54 Slave[1] 55 </td> 56 <td> 57 <audio id ="slave1" preload = "auto" controls> 58 <source src = "http://personalpages.tds.net/~jimdani74/groove.ogg" type="audio/ogg" /> 59 <source src = "http://personalpages.tds.net/~jimdani74/groove.mp3" type="audio/mpeg" /> 60 Your browser does not support HTML 5 audio. You may want to update your 61 browser to a current version. 62 </audio> 63 </td> 64 </tr> 65 </table> 66 <canvas id="KISA" data-processing-sources="KISA8.pde Display.pde CircleButton.pde"> 67 Your browser does not support HTML 5. You may want to update your 68 browser. 69 </canvas> 70 <p> 71 </p> 72 <p> 73 </p> 74 <p>Updated 11/1/14</p> 75 </body> 76 </html>
[+/-] The CircleButton.pde file
Updated 11/6/14