Music Analysis & Classification with the Logitech Media Server using Essentia


Essentia is

a open-source C++ library for audio analysis and audio-based music information retrieval. It contains an extensive collection of algorithms including audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors.

This plugin enables importing Essentia's analysis results (currently statistical classification of music) into the LMS database, making them available to be used by LMS-based applications / plugins (currently, extGUI4LMS and the LMS Playlist Editor).


Essentia now supports analysis using Deep Learning (courtesy of TensorFlow) in addition to Gaia/SVM, which significantly improves accuracy (but probably will be much slower, unless GPGPU is used). LMS Essentia has been updated to take advantage of this, and the following documentation has been adapted to reflect this by marking sections as "Gaia/SVM only" or as "TensorFlow only". It's highly recommended to choose one method only and not mix them.


  • A mechanism for uploading the analysis data to the LMS database
  • A simple Web UI for statistics, data display, DB administration
  • A interface for other applications to query the data
  • A helper script to create/upload the analysis data
  • Integration with extGUI4LMS and the LMS Playlist Editor, enabling searches for tracks based on analysis data like mood, genre, ...


Available data

The Essentia music extractor generates a large number of so-called "music descriptors" (see here for details). However, for this application only the high-level classifier models are used.
Examples include:

  • mood (happy, relaxed, ...)
  • genre (classical, rock, ...)
  • danceability
  • vocal / instrumental
  • timbre (dark / bright)
  • rhythm (samba, waltz, ...)

Details (including the accuracy of the classification) can be found here for SVM and here for TensorFlow.


beta (i.e. expect bugs)


The following packages only need to be installed on the system the analysis is run from, the LMS plugin itself doesn't need them (so it is possible e.g. to install them on a fast workstation, mount the music files residing on a slow NAS, run the analysis of these files on the workstation, and upload the resulting analysis data files to LMS on a Raspberry Pi).

  • jq (available as a repository package for most Linux distributions, binaries available for Linux, OS X and Windows)
  • curl (available as a repository package for most Linux distributions, binaries available for most platforms)
  • The helper scripts (including the analysis profile and a music extractor supporting TensorFlow)
for Gaia/SVM only
  • The Essentia music extractor
    • The official binaries are missing one required feature (Gaia) and therefore cannot be used.
    • A binary for Linux x86_64 is available here. If your CPU doesn't support AVX and SSE4 , use this one instead.
    • A binary for Linux arm64 / aarch64 is available here. However, non-x86 platforms are not officially supported by Essentia, and while building the program on aarch64 was possible, it crashes on some SVM models. You'll have to use the highlevel-aarch64.profile, which doesn't include these models.
  • The high-level models (file essentia-extractor-svm_models-v2.1_beta1.tar.gz)
for TensorFlow only
  • Python 3
  • Essentia with Tensorflow support
  • The models

Note that while the basic components listed above are available for a number of platforms, the helper script to run the analysis (for Gaia/SVM) and upload the data to LMS is written in bash, and therefore will only work on *nix-like systems. The script, however, is rather simple and likely could be easily implemented in e.g. Windows Power Shell.


The application consist of two components: The LMS plugin, and the package for the music analysis, which can be run on any system with access to the music files and LMS.


In the LMS standard web interface in Settings/Plugins, add to the list of Additional Repositories and activate the plugin.
You'll find an entry to launch the web interface in the Extras menu, which is just a link (to http://lmsserver:9000/plugins/LMSessentiaIntegration/html/index.html?player=xx:xx:xx:xx:xx) , so you can bookmark it - it doesn't have to be launched from the regular web interface.

  • Install curl and jq (e.g. using your distribution's package management)
  • Download the helper script package and unpack it somewhere
  • For Gaia/SVM:
    • Download the binary music extractor and the high-level models and unpack them somewhere
    • Edit highlevel.profile and adjust the paths to the svm models
  • For TensorFlow:
  • Proceed to "Creating and uploading the analysis data" below
Compilation from sources

Note: only relevant if you haven't installed the binaries as documented above.

  • Install curl and jq (e.g. using your distribution's package management)
  • Download the helper script package and unpack it somewhere
  • Basic installation instructions are listed here for Essentia and here for Gaia (which must be enabled in Essentia)
  • Get the sources for Essentia and Gaia
  • Make sure you have installed the required dependencies
  • When compiling Gaia, this issue became pertinent. Applying the solution described there fixed it.

Example - Install in /opt

Note: In order to keep non-repository packages separate from the main system, this uses the prefix /opt.


    CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" ./configure -prefix /opt/qt-4.8.7 -no-qt3support -no-phonon -no-phonon-backend -no-declarative -no-cups -no-nis -no-accessibility -no-sql-sqlite -no-audio-backend -no-svg -no-webkit -no-script -no-scripttools -no-dbus -no-gtkstyle -no-opengl -no-openvg -no-gui -no-fontconfig -opensource -nomake examples -nomake demos -nomake docs -nomake translations -no-iconv -no-glib -no-xkb -no-xinput -no-fontconfig -no-mitshm -no-xrender -no-xrandr -no-xfixes -no-xcursor -no-xinerama -no-xsync -no-xvideo -no-xshape -no-sm -no-pch -no-openssl -no-libjpeg -no-libmng -no-libpng -no-libtiff -no-gif -no-svg -no-xmlpatterns -qt-zlib -static -opensource -confirm-license
    make install


    PKG_CONFIG_PATH=/opt/qt-4.8.7/lib/pkgconfig CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" ./waf configure --prefix=/opt/essentia-2.1b4 --qtdir=/opt/qt-4.8.7 --mode=release
    ./waf install


    PKG_CONFIG_PATH=/opt/qt-4.8.7/lib/pkgconfig:/opt/essentia-2.1b4/lib/pkgconfig CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -I/opt/essentia-2.1b4/include" CXXFLAGS="${CFLAGS}" LDFLAGS="-L/opt/essentia-2.1b4/lib -L/opt/qt-4.8.7/lib -Wl,-rpath,/opt/essentia-2.1b4/lib -Wl,-rpath,/opt/qt-4.8.7/lib" ./waf configure --prefix=/opt/essentia-2.1b4 --with-gaia --with-examples --mode=release
    ./waf install

Finally, delete /opt/qt-4.8.7 and copy the svm models extracted from essentia-extractor-svm_models-v2.1_beta1.tar.gz to /opt/essentia-2.1b4/share/svm_models/


Building Essentia with TensorFlow is described in this post.
Another option not mentioned is to install a TensorFlow package provided by the OS (e.g. Gentoo has sci-libs/tensorflow) and compile Essentia as described above (without Gaia, but --with-tensorflow). Note that if Essentia is installed outside of the regular Python search path, you'll have to adjust the line sys.path.append( "/opt/essentia-200404-tf/lib64/python3.6/site-packages") in

Creating and uploading the analysis data

  • The general workflow is to analyse the music and upload the resulting data files to the plugin, which will save the data to the LMS database. This can be done using the (for Gaia/SVM) or (for TensorFlow) helper scripts (see below).
  • It is important that the absolute file paths of the analysed files match the absolute file paths on the LMS server. If the analysis is done on a different system with different paths, use the -s option of the script to modify the path written to the data files.
  • The generated analysis data files are just text files (JSON format), so they can be easily manipulated after creation (e.g. if the music files have been moved on the LMS server, run a search/replace over them to adjust the file paths (look for the "file_path" entry)).
  • Since analysis takes a lot of time, the analysis data files should be kept, in case they need to be uploaded again for whatever reason.
  • After the upload, you can check in the web interface how many Essentia tracks are in the database, and how many match with the regular LMS tracks. If there are fewer matches than expected, the most likely cause is that the absolute file paths are different (since this is the criteria used to recognize which Essentia and LMS tracks are the same).

Helper Scripts Usage Instructions

  • for Gaia/SVM, is used for both analysis and upload. For analysis, it will run the Essentia music extractor.
  • for TensorFlow, is used for analysis and for upload. The Essentia music extractor is not used.
Analysis of a directory tree and upload (in one operation) (Gaia/SVM only)
    ./ -o all -s "s#/home/user/tmp/sshfs/music#/data/music#" -p ~/essentia/highlevel.profile -v -j ~/essentia/jsons -m ~/sshfs/music/ -l lmsserver:9000
Analysis of music files read from a text file and upload (in two operations)


    ./ -o analyze -s "s#/home/luser/tmp/sshfs#/data/music#" -v -j ~/essentia/jsons -i ~/tmp/playlist.m3
    ./ -o submit -v -j ~/essentia/jsons -l lmsserver:9000


    ./ --model-dir ~/essentia/models --replace-pathname "/home/luser/tmp/sshfs:/data/music" ~/tmp/playlist.m3 ~/essentia/jsons/ 
    ./ -o submit -v -j ~/essentia/jsons -l lmsserver:9000
Options for
Option Description Example
-o all|analyze|submit Operation mode:
  • analyze: analyze music and create analysis data files (SVM only)
  • submit: upload data files to LMS database
  • all: analyze and upload (SVM only)
    -o all
    -l LMS server (including port) -l lmsserver:9000
    -p Analysis profile (SVM only) -p /home/user/essentia/highlevel.profile
    -j Directory for analysis data files: destination for analysis / source for upload -j /home/user/essentia/jsondata
    -m Music files directory tree to be analysed (cannot be combined with -i) (SVM only) -m /home/user/sshfs
    -i Text file to read paths to music files to be analysed from (Format: one file per line. Cannot be combined with -m)(SVM only) -i /home/user/essentia/playlist.m3u
    -s sed string: The absolute file paths of the anayzed files have to match the absolute file paths on the LMS server. This option can be used to change the file path written to the data file. -s "s#/home/user/sshfs#/data/music#" will substitute the server path (/data/music) for the local path (/home/user/sshfs)
    -x Essentia extractor to use (only needed if essentia_streaming_extractor_music is not in search path)(SVM only) -x /opt/essentia-2.1b4/bin/essentia_streaming_extractor_music
    -c Overwrite existing output files (SVM only) -c
    -v Verbose output -v
    Options for

    Run --help

    The Analysis Profile (Gaia/SVM only)

    defines which classifications will be done during the analysis phase by specifying SVM model files to be used (e.g. mood_happy.history to analyse if the mood of the music is happy). The included highlevel.profile specifies nearly all available classifications. If you are not interested in a classification, you can remove it from the profile.
    Note: For TensorFlow, all models found in the specified directory will be applied.



    The analysis is is rather slow, and the extractor only runs single-threaded (i.e. uses one CPU core). It's possible to run several instances of the helper script at the same time (even on the same directory tree), provided analysis and upload are done separately.

    • Using GPGPU (i.e. CUDA for NVIDIA cards) will speed up the analysis significantly. Note that the version of TensorFlow used has to support this.
    • musiCNN models are ~6 times faster then VGGish (on my setup, YMMV), but usually less accurate


    Adding the plugin repository to LMS results in some errors / doesn't work. Why?
    Make sure your LMS installation is SSL-enabled (e.g. on *nix, the perl modules Net::SSLeay and IO:Socket:SSL need to be installed)
    The music extractor crashes with a 7716 Illegal instruction (core dumped) error. Why?
    Your CPU doesn't support certain instructions (AVX, SSE4), either because it is very old (<2008 (SSE) / < 2011 (AVX) ), or low-end (some Atoms / Celerons / Pentiums). Use the non-AVX version.
    Is it possible to use other classifications?
    Yes, but you will have to train your own classifier models (Gaia/SVM only)


    The API is documented here.


    GPL v3

    Reporting Issues

    Use the discussion thread in the LMS forum.