Music play along with FollowTheScore (FTS)

How large is your personal effective playing time on such a day? Surely you would have liked to see how your instrument fits into the sound of the whole orchestra. And there is that transition where you would have liked to see how the conductor might take you safely from the lento in the introduction to the lively allegro with quarter notes = 168.
May-be next time ...

Enjoy the sound of a professional orchestra while turning the tempo a little bit down in the allegro until your fingers know what to do. Then watch the conductor in the video how he manages the transition. Still not quite clear? A small upward movement of his hand seems to be sufficient for these pro´s. Well, FTS offers a second video where the conductor gives the whole four beats at 168 during the fermata of the introduction. Is that really better?

The point is not what these conductors were doing or what your conductor will do in the next rehearsal. The point is that FTS gives you the chance to compare. After some practice you will have the faster tempo in your mind before any conductor starts to move!

FollowTheScore (FTS)

Sceptic? – see, what FTS can do for you.

Playing Music with the computer is boring.
: Probably you are thinking of schematic midi sound and mathematical exactness. With FTS you will be playing with Leonard Bernstein or with the Tokyo Kosei Wind Orchestra.
You are NOT "playing with a computer" - you are playing with the best artists and orchestras in the world!
And that is definitely great fun!

As a conductor I don´t like my musicians to train with audio recordings.
: You think that they will "burn in" a single recording? They will narrow their mind and develop fixed ideas of tempi, of ritardandi?
Well, FTS can vary the tempo slightly while playing - just as you would. And it can switch between different recordings seamlessly, from phrase to phrase if you like. Used in a proper way FTS can even promote the flexibility of your musicians!; Practicing only with the score has some risks: If a musician initially has a false perception of a certain rhyhtm and begins to practice then he may have a very hard time later to extinguish the misconceptions from his brain. If you really feel distracted by listening to an interpretation you still can use the intelligent metronome of FTS. It has an adjustable base tempo and it handles all the nifty things like changes in metre, accelerandi, ritardandi and fermatas.

Such a program will make my musicians stare at the score all the time.
: Sure, FTS creates a temptation to stare at the moving bar highlight while you are playing. But there is also a "natural mode" where the program does not indicate the current bar. And there is something in between, where the program helps the player at critical points in case he gets lost - typically when a new study symbol is reached. Very close to reality, isn´t it?

I am still not convinced. My musicians have their heads in the notes far too often.
: FTS has a mode where it hides the complete score page for two or three seconds at the right moment. It shows a conductor symbol instead and expects the musician to know the next measure by heart so he can fully concentrate on the correct moment to start playing. Thus the ability to look at a different place and to return to the correct position on the score leaf explicitly gets trained. Apart from that FTS also has a mode for read-ahead training where the current measure becomes invisible shortly before reaching its end. This also helps to create free time slots for the musicians where they can have a glance at the conductor.



I never heard of a computer program that can do all this. Is it true?
: There is no magic involved. But you are right: The problem of matching arbitrary paper scores and audio recordings is very hard to solve. A fully automated approach does not work perfectly. Therefore our concept for editing documents in FTS is semi-automatic. We use an optical recognition program to identify bars on score pages. An intuitive user interface allows to correct remaining errors (typically less than 5%).
Then you listen to the music and hit the enter key once at the beginning of each measure. This means that you have to go through the music once manually. If you don´t get all bar begin times right at the first attempt you can easily correct the timing for a single bar or for a phrase. Thus you create a master track. The same master track is then used for all instrument scores of the title. If you want to link another audio recording to the score you can repeat the manual process or try our automatic beat finder which tries to find analogies between different recordings. This utility is still somewhat experimental, however.

What can I do with FTS?

The user interface for watching, listening and editing

FTS has a play mode for users who want to view a score, listen to the music and play along with their instruments.

FTS has an edit mode for users who want to contribute titles to the public FTS library (or to their private library), provided they are willing to respect copyrights.
To create a title you need to upload the media files (pdf score, MXML score or Lilypond score, MP3 recording, link to YouTube or MIDI file) and to create an FTS "index" which ties them together. The index is a JSON file which contains (A) time stamps for the audio resources and (B) bar positions for the scores and (C) defines their interrelationships, including repetitions and jumps.
It is especially easy to add a title if the score is available on IMSLP and if one or more videos of that title can be found on YouTube.

The screenshots below give an impression of the user interface.

Our example shows the horn score together with a cello recording. You can fluently switch to Radek Baborak playing the french horn. The program is playing the first beat of measure number 3 (highlighted in blue). A practice loop is defined for measures 12..15.

FTS supports two types of annotations: single score (yellow) and full score (orange). When you scroll through the annotations in the right hand window the corresponding position will be highlighted (in this case: measure 40).

FTS comes with an extensive tutorial which explains playing and editing features. Even as a guest you can try out editing (without saving).

When creating a title you start in the structure editor defining parts/movements (red), audio recordings (green) and score documents (blue). Special import features help you to easily compose a title from IMSLP scores and YouTube recordings. But you can also upload your own media as long as you comply with copyright restrictions. For compatibility with existing scores FTS can also import MusicXML format or LilyPond format. If you have a midi file FTS can take it and convert it to mp3 (although this is not exactly what FTS was made for ;-)

A spectrum based on FFT (fast Fourier transform). The large bars are measures, the small yellow bars are auto-detected onsets where the length of the yellow line is an indication for the significance of the onset. FTS can fine-adjust your bar settings based on these onsets.

Looking for pilot editors ..

Looking for partners ..

We are also looking for the best way to establish FollowTheScore in the market. Currently we see the following marketing scenarios:

FTS could be sold as a product without any content, like score editors, sequencers etc.
FTS could become part of the offering of a major player in the traditional music publishing business, creating a competitive edge for that publisher. FTS could become part of his web platform but there is also another possibility: The offline variant of FTS could be added as a "second session" to audio CDs which are typically enclosed with play-along booklets.
FTS could become the work horse of an independent company which sells scores and audio recordings on a web platform. Ideally such a company would have partnership contracts with traditional publishers and act as an e-commerce sales agent for them. Such a platform would probably also allow users to upload content and share it with others - provided that copyright issues are handled thoroughly.
FTS could become a product which helps YouTube publishers to create videos. There are many videos on youtube which present static score pictures while the music is playing. FTS would be kind of a quantum leap for such videos. In that case FTS could be extended in a way which supports other scenarios where visual information correlates to audio material. The core engine of FTS is generic, working on abstract optical spots which are linked to certain points on the time axis of a video or audio stream.
FTS could become an open source product, maybe in co-operation with IMSLP. In that case one would hope for the crowd effect.

How would I use FTS? — Questions and Answers

My computer is in room A and I use to play with my instrument in room B.
: No one said that you should sit in front of your computer when practicing. Put a tablet computer on your note stand in room B, for example an iPad or a 10 inch Android device. A good choice is also a Windows-based notebook where the display can be bent by 180° or even by 360°.
You could also mount a monitor to your wall and connect it to your laptop when you want to play with FTS. There are ways to transmit the video signal via air, so you need not connect a cable each time.

I need my hands for the instrument, even a touch screen is somewhat uncomfortable.
: Use a foot pedal! FTS supports the most important functions like start/stop/forward/back with only two foot pedals.
Another possibility are ultra flat wireless keyboards. They look like the keyboard of a notebook and are very light-weight. Place such a keyboard near you in a comfortable height.

The sound of a tablet or notebook is lousy. I will not even hear it when I play mezzo forte on my trombone.
: Use an open headset or a good loudspeaker. Position the speaker in a way that resembles the spatial arrangement in your orchestra. If you are sitting in the back of the orchestra, put the speaker in front of you, heading to a virtual audience. If you are sitting in the first row put it behind you and have it play into your ears ;-)

I am afraid of cables hanging around. Eventually the tablet will drop to the floor.
: Use Bluetooth components! There is no need for any cable. You can find BT pedals, headsets and speakers for affordable money.

I practice in a cellar room, there is no WiFi connection in that room.
: FTS allows you to download a title. Then you can play it completely offline (audio sound track only, videos are not suported offline). Offline use is also quite useful if you have to spend some time travelling. Lean back in the chair, put good headphones on and go through a title several times. This can be even more efficient than practicing with your instrument.

We are playing a piece where I have long rests. I don´t want to waste time listening all the time while others have to play.
: Use the quick jump mode of FTS. The program will cut out most of the parts where you do not have to play. But you will be put on track one or two measures before you have to step in again so that you can learn to know the right moment.

As a conductor I put great effort in finding the right pieces for our next concert. But far too seldom the players seem to be interested in the background of the pieces.
: Help them to discover details about the pieces, about their origin, about the story behind, about the composer, about the film - if you are playing titles from movies. If you chose an arrangement of an original work, give them a link to the original and tell them why you prefer that very arrangement you selected over others. FTS easily connects to Wikipedia, to YouTube to music dictionaries etc. Prepare some snippets of information and your musicians might become more educted and curious next time.

In many rehearsals I have to tell my musicians the same things again and again. I could well tell you in advance the five greatest mistakes that will be made.
: O.k., then take your courage and put those essential remarks into the score. You can add them to the full score and they will be shown in each instrument´s score at the correct position - unless the instrument has multiple rests at that measure.
You - or the musicians themselves - can also add remarks (or finger settings) which are specific to certain instruments.

Gradually our orchestra is improving. I might try an ambitious piece next time with many changes in metre and with lots of irregular measures like 5/8, 7/8, 9/8, 11/8ths.
: FTS can be taught to support meter changes. If you have, for instance, a piece with 5/8ths where the emphasis changes from 2-3 to 3-2 after several measures you can add a hint to FTS and its built-in metronome will show such changes. Even more: When the tempo gets slower, the metronome will change its color to red, acceleration is shown in blue. Changes in meter will be highlighted very clearly. Thus your musicians will be prepared when they come to the first rehearsal (well, let´s be honest: to the second).

That new piece has some phrases which are quite fast for the clarinets. How do I get them to speed and still play correctly?
: With FTS they can mark a group of measures for endless repetition. They can reduce the tempo to, say, 60% and each run of the loop will automatically become slightly faster until the final tempo is reached. Make sure that you select a good recording for that kind of practicing as a tempo of 60% may also unveal some weaknesses in the recordings.

Recently we took some juniors into our orchestra. A little bit of fundamental daily practice would be good for them.
: Take a look at the add-on components of FTS. There are dictates for rhythm, intervals and melody. Scale practicing is also supported. And there is a module where they can check how good they are at recognizing minimal differences in pitch.

We have some musicians in our orchestra who have a problem with intonation. They understand that playing the correct note is not enough, but they do not know what to do when a chord sounds "rough".
: That´s a point we are working on. A future version of FTS will have a module which allows to practice intonation, eg. by FTS playing two (purely tuned) tones of a tonic triple chord. It will be the user´s part to place the major or minor third until he hears a perfect sound.

Still not convinced?
: Then let us know. May-be we can find a solution. FTS becomes better by critical users telling us what they dislike or what they miss.