Introduction

Essentia is

a open-source C++ library for audio analysis and audio-based music information retrieval. It contains an extensive collection of algorithms including audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors.

This plugin enables importing Essentia's analysis results (currently statistical classification of music) into the LMS database, making them available to be used by LMS-based applications / plugins (currently, extGUI4LMS and the LMS Playlist Editor).

Features

• A mechanism for uploading the analysis data to the LMS database
• A simple Web UI for statistics, data display, DB administration
• A interface for other applications to query the data
• A helper script to create/upload the analysis data
• Integration with extGUI4LMS and the LMS Playlist Editor, enabling searches for tracks based on analysis data like mood, genre, ...

Status

Available data

The Essentia music extractor generates a large number of so-called "music descriptors" (see here for details). However, for this application only the high-level classifier models are used.
Examples include:

• mood (happy, relaxed, ...)
• genre (classical, rock, ...)
• danceability
• vocal / instrumental
• timbre (dark / bright)
• rhythm (samba, waltz, ...)

Details (including the accuracy of the classification) can be found here for SVM and here for TensorFlow.

Plugin

beta (i.e. expect bugs)

Prerequisites

The following packages only need to be installed on the system the analysis is run from, the LMS plugin itself doesn't need them (so it is possible e.g. to install them on a fast workstation, mount the music files residing on a slow NAS, run the analysis of these files on the workstation, and upload the resulting analysis data files to LMS on a Raspberry Pi).

• jq (available as a repository package for most Linux distributions, binaries available for Linux, OS X and Windows)
• curl (available as a repository package for most Linux distributions, binaries available for most platforms)
• The helper scripts (including the analysis profile and a music extractor supporting TensorFlow)

for Gaia/SVM only

• The Essentia music extractor
  • The official binaries are missing one required feature (Gaia) and therefore cannot be used.
  • A binary for Linux x86_64 is available here. If your CPU doesn't support AVX and SSE4 , use this one instead.
  • A binary for Linux arm64 / aarch64 is available here. However, non-x86 platforms are not officially supported by Essentia, and while building the program on aarch64 was possible, it crashes on some SVM models. You'll have to use the highlevel-aarch64.profile, which doesn't include these models.
• The high-level models (file essentia-extractor-svm_models-v2.1_beta1.tar.gz)

for TensorFlow only

• Python 3
• Essentia with Tensorflow support
• The models

Note that while the basic components listed above are available for a number of platforms, the helper script to run the analysis (for Gaia/SVM) and upload the data to LMS is written in bash, and therefore will only work on *nix-like systems. The script, however, is rather simple and likely could be easily implemented in e.g. Windows Power Shell.

Installation

The application consist of two components: The LMS plugin, and the package for the music analysis, which can be run on any system with access to the music files and LMS.

Plugin

In the LMS standard web interface in Settings/Plugins, add https://www.nexus0.net/pub/sw/lmsessentia/repo.xml to the list of Additional Repositories and activate the plugin.
You'll find an entry to launch the web interface in the Extras menu, which is just a link (to http://lmsserver:9000/plugins/LMSessentiaIntegration/html/index.html?player=xx:xx:xx:xx:xx) , so you can bookmark it - it doesn't have to be launched from the regular web interface.

Binaries

• Install curl and jq (e.g. using your distribution's package management)
• Download the helper script package and unpack it somewhere
• For Gaia/SVM:
  • Download the binary music extractor and the high-level models and unpack them somewhere
  • Edit highlevel.profile and adjust the paths to the svm models
• For TensorFlow:
  • Install Essentia with TensorFlow as described in this post
  • Download the models you want to use (but do not rename them)
• Proceed to "Creating and uploading the analysis data" below

Compilation from sources

Note: only relevant if you haven't installed the binaries as documented above.

• Install curl and jq (e.g. using your distribution's package management)
• Download the helper script package and unpack it somewhere

Gaia/SVM

• Basic installation instructions are listed here for Essentia and here for Gaia (which must be enabled in Essentia)
• Get the sources for Essentia and Gaia
• Make sure you have installed the required dependencies
• When compiling Gaia, this issue became pertinent. Applying the solution described there fixed it.

Example - Install in /opt

Note: In order to keep non-repository packages separate from the main system, this uses the prefix /opt.

qt

    CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" ./configure -prefix /opt/qt-4.8.7 -no-qt3support -no-phonon -no-phonon-backend -no-declarative -no-cups -no-nis -no-accessibility -no-sql-sqlite -no-audio-backend -no-svg -no-webkit -no-script -no-scripttools -no-dbus -no-gtkstyle -no-opengl -no-openvg -no-gui -no-fontconfig -opensource -nomake examples -nomake demos -nomake docs -nomake translations -no-iconv -no-glib -no-xkb -no-xinput -no-fontconfig -no-mitshm -no-xrender -no-xrandr -no-xfixes -no-xcursor -no-xinerama -no-xsync -no-xvideo -no-xshape -no-sm -no-pch -no-openssl -no-libjpeg -no-libmng -no-libpng -no-libtiff -no-gif -no-svg -no-xmlpatterns -qt-zlib -static -opensource -confirm-license
    make
    make install

Gaia

    PKG_CONFIG_PATH=/opt/qt-4.8.7/lib/pkgconfig CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" ./waf configure --prefix=/opt/essentia-2.1b4 --qtdir=/opt/qt-4.8.7 --mode=release
    ./waf
    ./waf install

Essentia

    PKG_CONFIG_PATH=/opt/qt-4.8.7/lib/pkgconfig:/opt/essentia-2.1b4/lib/pkgconfig CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -I/opt/essentia-2.1b4/include" CXXFLAGS="${CFLAGS}" LDFLAGS="-L/opt/essentia-2.1b4/lib -L/opt/qt-4.8.7/lib -Wl,-rpath,/opt/essentia-2.1b4/lib -Wl,-rpath,/opt/qt-4.8.7/lib" ./waf configure --prefix=/opt/essentia-2.1b4 --with-gaia --with-examples --mode=release
    ./waf
    ./waf install

Finally, delete /opt/qt-4.8.7 and copy the svm models extracted from essentia-extractor-svm_models-v2.1_beta1.tar.gz to /opt/essentia-2.1b4/share/svm_models/

TensorFlow

Building Essentia with TensorFlow is described in this post.
Another option not mentioned is to install a TensorFlow package provided by the OS (e.g. Gentoo has sci-libs/tensorflow) and compile Essentia as described above (without Gaia, but --with-tensorflow). Note that if Essentia is installed outside of the regular Python search path, you'll have to adjust the line sys.path.append( "/opt/essentia-200404-tf/lib64/python3.6/site-packages") in esstf-extractor.py.

Creating and uploading the analysis data

• The general workflow is to analyse the music and upload the resulting data files to the plugin, which will save the data to the LMS database. This can be done using the ess-analyze.sh (for Gaia/SVM) or esstf-extractor.py (for TensorFlow) helper scripts (see below).
• It is important that the absolute file paths of the analysed files match the absolute file paths on the LMS server. If the analysis is done on a different system with different paths, use the -s option of the script to modify the path written to the data files.
• The generated analysis data files are just text files (JSON format), so they can be easily manipulated after creation (e.g. if the music files have been moved on the LMS server, run a search/replace over them to adjust the file paths (look for the "file_path" entry)).
• Since analysis takes a lot of time, the analysis data files should be kept, in case they need to be uploaded again for whatever reason.
• After the upload, you can check in the web interface how many Essentia tracks are in the database, and how many match with the regular LMS tracks. If there are fewer matches than expected, the most likely cause is that the absolute file paths are different (since this is the criteria used to recognize which Essentia and LMS tracks are the same).

Helper Scripts Usage Instructions

• for Gaia/SVM, ess-analyze.sh is used for both analysis and upload. For analysis, it will run the Essentia music extractor.
• for TensorFlow, esstf-extractor.py is used for analysis and ess-analyze.sh for upload. The Essentia music extractor is not used.

Examples

Analysis of a directory tree and upload (in one operation) (Gaia/SVM only)

    ./ess-analyze.sh -o all -s "s#/home/user/tmp/sshfs/music#/data/music#" -p ~/essentia/highlevel.profile -v -j ~/essentia/jsons -m ~/sshfs/music/ -l lmsserver:9000

Analysis of music files read from a text file and upload (in two operations)

Gaia/SVM:

    ./ess-analyze.sh -o analyze -s "s#/home/luser/tmp/sshfs#/data/music#" -v -j ~/essentia/jsons -i ~/tmp/playlist.m3
    ./ess-analyze.sh -o submit -v -j ~/essentia/jsons -l lmsserver:9000

TensorFlow

    ./esstf-extractor.py --model-dir ~/essentia/models --replace-pathname "/home/luser/tmp/sshfs:/data/music" ~/tmp/playlist.m3 ~/essentia/jsons/ 
    ./ess-analyze.sh -o submit -v -j ~/essentia/jsons -l lmsserver:9000

Options for ess-analyze.sh

Option	Description	Example
-o all|analyze|submit	Operation mode: analyze: analyze music and create analysis data files (SVM only) submit: upload data files to LMS database all: analyze and upload (SVM only)	-o all
-l	LMS server (including port)	-l lmsserver:9000
-p	Analysis profile (SVM only)	-p /home/user/essentia/highlevel.profile
-j	Directory for analysis data files: destination for analysis / source for upload	-j /home/user/essentia/jsondata
-m	Music files directory tree to be analysed (cannot be combined with -i) (SVM only)	-m /home/user/sshfs
-i	Text file to read paths to music files to be analysed from (Format: one file per line. Cannot be combined with -m)(SVM only)	-i /home/user/essentia/playlist.m3u
-s	sed string: The absolute file paths of the anayzed files have to match the absolute file paths on the LMS server. This option can be used to change the file path written to the data file.	-s "s#/home/user/sshfs#/data/music#" will substitute the server path (/data/music) for the local path (/home/user/sshfs)
-x	Essentia extractor to use (only needed if essentia_streaming_extractor_music is not in search path)(SVM only)	-x /opt/essentia-2.1b4/bin/essentia_streaming_extractor_music
-c	Overwrite existing output files (SVM only)	-c
-v	Verbose output	-v

Options for esstf-extractor.py

Run esstf-extractor.py --help

The Analysis Profile (Gaia/SVM only)

defines which classifications will be done during the analysis phase by specifying SVM model files to be used (e.g. mood_happy.history to analyse if the mood of the music is happy). The included highlevel.profile specifies nearly all available classifications. If you are not interested in a classification, you can remove it from the profile.
Note: For TensorFlow, all models found in the specified directory will be applied.

Performance

Gaia/SVM

The analysis is is rather slow, and the extractor only runs single-threaded (i.e. uses one CPU core). It's possible to run several instances of the helper script at the same time (even on the same directory tree), provided analysis and upload are done separately.

TensorFlow

• Using GPGPU (i.e. CUDA for NVIDIA cards) will speed up the analysis significantly. Note that the version of TensorFlow used has to support this.
• musiCNN models are ~6 times faster then VGGish (on my setup, YMMV), but usually less accurate

FAQ

Adding the plugin repository to LMS results in some errors / doesn't work. Why?

Make sure your LMS installation is SSL-enabled (e.g. on *nix, the perl modules Net::SSLeay and IO:Socket:SSL need to be installed)

The music extractor crashes with a 7716 Illegal instruction (core dumped) error. Why?

Your CPU doesn't support certain instructions (AVX, SSE4), either because it is very old (<2008 (SSE) / < 2011 (AVX) ), or low-end (some Atoms / Celerons / Pentiums). Use the non-AVX version.

Is it possible to use other classifications?

Yes, but you will have to train your own classifier models (Gaia/SVM only)

API

The API is documented here.

License

GPL v3

Reporting Issues

Use the discussion thread in the LMS forum.