cuesplitflac : a mass flac splitting script for synology and ubuntu

-
Read the disclaimer

This bash script searches recursively directories and splits .flac or .ape file in separate tracks, based on the information provided in the .cue files.
A text database keeps an inventory of split files, enabling the script to run on a schedule.
The original file tree structure is mirrored to the converted file tree structure.

You can get the script here : https://github.com/youpla/bash_cuesplitflac

Since DSM 6, synology provides a bash implementation that is close to what you would find on Linux systems, enabling easily bash code porting.
This script uses different compiled tools to achieve its goal.

There are GUI tools that manage flac files splitting, example : https://flacon.github.io/
Most of those tools achieve what this script does not, they are multi-threaded, optimized for speed and highly customizable.
cuesplitflac on the other hand is more of a slow background process you can run on a synology or linux shell.

You can generate .flac and .cue with software like EAC (Exact Audio Copy) : http://www.exactaudiocopy.de

The script has been tested on Synology and Ubuntu (18).

Prerequisities

In order to work, this script need the following tools to perform the different tasks
  • flac
  • mac (monkey's audio)
  • cpulimit
  • shntool
  • cuetools
  • imagemagick (is builtin in synology)
  • file
  • iconv (libiconv)

Synology

You can compile those tools following those different walkthrough:

Ubuntu

Depending on the version, some tools may already be installed

Run from a terminal :
sudo apt install cpulimit flac shntool cuetools imagemagick

awk : the gawk implementation is used

Customization

The script is piloted by a settings file named "init.txt"

There is no " quotation mark to use in init.txt , even if one of your path contains a space character.

The .txt extension is being preferred to allow the use of the built in synology gui text editor.

flac split synology ubuntu dsm
Please note the file format : utf-8

Settings

  • pathcueflacsource : full source path, no relative path
  • pathflacdest : full destination path, no relative path
  • pathexclus-x : optional exclusion, can be deleted
  • coverlist : cover file names, case insensitive, can be deleted
  • coverfolderlist : cover folder names, case insensitive, can be deleted
  • filethumbnail : thumbnail file name to be added to the folder 
  • waittime : comparison delay, can be deleted
  • valcpulimit : cpulimit threshold
  • flaccomplevel : lossless flac compression level, can be deleted
  • smbpattern-x smbreplace-x: is a set of pattern to replace unix file name by windows file names, can be deleted

More details

pathexclus-x  :
this is a setting that removes the paths starting with a pattern
  • source path : /home/userprofile/demo
    • subdirs:
      • /home/userprofile/demo/household-member-1
      • /home/userprofile/demo/household-member-2
  • pathexclus-1=household-member-2
  • result:
    • only "/home/userprofile/demo/household-member-1" is processed
Can be deleted. Can be incremented up to 10. 

coverlist :
Is a list of file names that can be used as a front cover for the flac file.
The search order is:
  • flac file root (folder containing the flac file)
  • then coverfolderlist (folders)
  • then the flac file root folder recursively
  • as soon a match is found the search stops
    • this means 
      • the order of coverlist matters
      • the order of coverfolderlist matters
coverfolderlist :
Are the subfolders to be searched within the folder containing the flac file. Can be deleted. 

filethumbnail :
Has a default value of "Folder.jpg" if omitted.
Folder.jpg is only added if a cover file is found

waittime :
Before processing any file, the script checks if the file size is the same over a short elapsed time. This parameter provides the time gap between the first and the second check.
It can be set to 0.
If unset the waittime is 5 seconds

flaccomplevel :
Allows to change the flac compression level.
If unset, the value is 8, which is the highest value with the highest cpu load.

You can also set a negative value, it will then perform the split with a different command line.
smbpattern-x smbreplace-x :
Allows to change the file name in order to be compatible with windows systems and windows smb shares.
It will only replace the incompatible characters in the file name. The file tags will remain the same.
You can specify up to 6 patterns.
Note: the double \\ in the sample "init.txt" file is not a typo. (see : https://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_03.html )

Running the script on synology.


Steps:
  • Compile and install all the prerequisites
  • Create a directory for your script. You can create the directory in the web gui.
  • Then in the shell navigate to it : /volume1/yourdir
  • download the script
    • curl https://raw.githubusercontent.com/youpla/bash_cuesplitflac/master/cuesplitflac.sh -O
    • curl https://raw.githubusercontent.com/youpla/bash_cuesplitflac/master/init.txt -O
    • curl https://raw.githubusercontent.com/youpla/bash_cuesplitflac/master/cpulimitpid.sh -O
  • check if the script is executable with : ls -al cuesplitflac.sh
    • has to be (at least)  : -r--r--r-x 
    • where x is the executable for your owner / group / user
    • user should be executable
  • to render it executable (if it is not) use : chmod u+x cuesplitflac.sh
  • adapt the init.txt file (see previous section)
  • run the script ./cuesplitflac.sh

Script detail

File tree

Source

flac split synology ubuntu dsm

Destination

flac split synology ubuntu dsm

  • Source variable : 
    • pathcueflacsource : /home/userprofile/demo
    • example of processed source path : /home/userprofile/demo/household-member-1/artist/album
  • Destination variable : 
    • pathflacdest : /home/userprofile/tmp/flac-no-cue
    • example of processed destination path : /home/userprofile/tmp/flac-no-cue/household-member-1/artist/album

Database

Each processed file is written in the database.
Only the short path is written in the database. The shortpath consist of the full path minus the pathcueflacsource variable
  • full path : /home/userprofile/demo/household-member-1/artist/album/cuefile.cue
  • source : /home/userprofile/demo
  • short path : /household-member-1/artist/album/cuefile.cue
If the short path already exist within the database, the script ignores the cue file.
The database is a text file and can be edited. You can delete an entry in order to reprocess an album.
  • scriptdir/db/base.txt
There is also a list for unprocessed files
  • scriptdir/db/basenoprocess.txt

Directories

The script creates 3 sub-directories
  • db : stores the text databases
  • tmp : stores the temp files
  • log : stores the log file

Notes

  • cue file has to have the same name as flac file (or ape file)
  • no cue file are copied in the destination directory

Code

There are two main functions
  • fileprocessing
    • lists all .cue files in pathcueflacsource 
    • substracts the files already listed in base.txt
    • processes the delta list
  • processingcore
    • extracts covers
    • converts cue files to utf-8
    • splits
    • add cover
    • add tags
    • changes smb names

Notes

smb compatibility, utf-8 conversion.
The script handles the following file types : 
  • ISO-8859 text
  • UTF-8 Unicode (with BOM)
  • unknown-8bit : generates an entry in the unprocessed database
    • If you stumble on an unknown file type, you'll have to convert it manually with a text editor.

cpulimitpid.sh

In order to reduce the load on low powered synology CPU. The script uses cpulimit.
  • The "cpulimit --exe flac --limit 50" would block the flac process at 50% cpu usage.
  • shnsplit starts 2 flac processes. The aforementioned command only limits one.
    • cpulimitpid.sh : is a script that searches (flac) pids and limits the processes based on their pids.
    • cpulimitpid.sh is started the same way as cpulimit and halted the same way once the process is over.

Notes

"Exact audio copy"'s cue files come in a variety of encoding:
  • text/plain; charset=utf-8
  • text/plain; charset=unknown-8bit
  • text/plain; charset=us-ascii
  • etc...


Popular posts from this blog

Compiling "file" with spksrc for synology NAS

-
Read the disclaimer This article describes how to compile "file" for synology devices, package it and install it. You can get "file" sources at: https://github.com/file/file ftp://ftp.astron.com/pub/file/ The package described in this guide is will download the content from that repository. The "file" command is "a file type guesser", that is, a command-line tool that tells you in words what kind of data a file contains. The original spksrc already contains a file package, that only compiles the binaries : https://github.com/SynoCommunity/spksrc/tree/master/cross/file This package fills the gap and provides the command line utilities from the shell: file This guide is the same that https://dayandnightcoding.blogspot.com/2020/04/compiling-cpulimit-with-spksrc-for.html You can follow the different steps there. I'll document only the steps referring specifically to "file". Clone the repository and make setup ...
Read more

Compiling cpulimit with spksrc for synology NAS

-
Image
Read the disclaimer This article discribes how to compile cpulimit for synology devices, package it and install it. I've set up a forked repository of spksrc at  https://github.com/youpla/spksrc There I publish some tools. You can get cpulimit sources at https://github.com/opsengine/cpulimit The package described in this guide is will download the content from that repository. cpulimit is a simple program which attempts to limit the cpu usage of a process expressed in percentage, not in cpu time. The apprehensible quality of this software is allowing you to run demanding software without rendering the NAS unusable. In order to compile the software there are two possibilities. a virtual machine a docker container In this example I'll rely on a virtual machine running Ubuntu . There are many steps involved in getting a virtual machine ready that are beyond this article scope, here are some basic steps: chose an hypervisor  install Ubuntu install th...
Read more