How can I quickly locate a specific piece of music in my vast library of videos? Can I match a specific beat to a new production I’m working on? Which programs in my archive are the most successful? These are typical questions that a broadcaster might pose to Fraunhofer IDMT’s Semantic Music Technologies Group, whose big picture is that when working with the content in an audiovisual archive, you need to be able to retrieve information by making it searchable.
Headed by Hanna Lukashevich, the group focuses on solving complex problems for music technology companies and professional media organizations involved in all kinds of audiovisual production. Use cases include locating a precise audio clip or specific type of sound, matching a particular mood, or enabling search across entire music libraries such as the broadcast archives of a TV or radio station, and analyzing programs statistically, based on acoustic analysis techniques.
“Our technologies are designed to give broadcasters an edge over their competition at a moment when competition forviewers is incredibly strong. When you can identify content and make informed business decisions quickly, this can drive new creative and business possibilities, says Hanna Lukashevich. “One cannot manually locate the content in a big archive with any degree of speed, and in today’s business environment, that just doesn’t scale.”
The group uses classical signal processing algorithms together with AI/ML and recommendation technologies in a “multimodal approach” that is very effective in bringing new solutions to common, but very challenging problems.
Its professional consulting services are used, for example, by big music archives that offer specialized search engines to professional producers. When enhancements are required, the group will first fully assess and then start to build the right components that improve performance or more fully automate the search processes.
On delivery, the system is not only customized for a specific set of tasks, but future enhancements can be seamlessly added.
Professional producers always need fast access to many music options. They depend on such search engines to find specific sounds, or specific beats and loops that perfectly match and complement a given music production.
This is the business of Jamahook, a Swiss music technology company. Jamahook operates a vast database of loops and sounds and offers a plugin that allows music producers to find the most suitable audio loops for a production, based on examples extracted from an audio mix. The plugin is the result of a long-term technology cooperation with Fraunhofer IDMT.
Jamahook’s latest feature is called “Pitch Shifted Matching“. After analyzing an example track the software recommends a collection of loops from the Jamahook database and can even shift the loops into the pitch that matches the example. This provides tremendous flexibility enabling access to a larger selection of loops and sounds to find the perfect match for a creative work.
Another common use case is music replacement. Often in a production placeholder music is used until licensing and rights issues are resolved. “Our multi-modal approach enables us to match the overall feeling of a production by training our models with granular attributes like rhythms, tempo and instrumentation,” says Hanna Lukashevich. This “learned” knowledge can then be applied to a specific task, such as the need to identify the most important musical instruments in a score, or by making them searchable at the precise moment they are played.
The model keeps on learning. It can infer and make an interpretation of new, unseen music data on its own. Over time, the models get even better at their tasks. In the case of music detection, accuracy rates of 99% for foreground music and over 80% for the detection of low-volume background music can be achieved.
“Customers can win big in terms of time and efficiency savings¨, says Hanna Lukashevich. “There’s just no time to manually sift through all of that material, so you need smarter tools when you are looking for something specific.”
Like finding a needle in a haystack, Fraunhofer IDMT’s Semantic Music Technologies Group addresses a class of work that simply cannot be done manually.
“We even help audience measurement companies acoustically analyze the performance of their programs and do similar work analyzing ads, to help them determine those that are the most successful,” says Hanna Lukashevich.
Fraunhofer IDMT’s Semantic Music Technologies Group helps its customers improve their archive management processes and optimize their businesses. Based on careful analysis and then, by delivering the components that are right for the job, customers are enabled to work in a highly accurate and automated way, at scale, using the multi-modal approach which is the strength of Fraunhofer IDMT.
Fraunhofer-Gesellschaft