Speed up thumbnail generation on Synology DS212J
I recently bought a Synology 212J to store all my pictures on, and of course also to play with. I always had some crappy tiny non-modifyable NAS/Media Center solution that I really hated. cough Plextor cough
So, I started adding pictures to the NAS and I wanted to play with the photo station application that is included. However, I noticed that as soon as I added pictures to the drive, it completely stalled. CPU utilisation went up to 100% 24/7. This also meant that it never went into idle mode and consumed more energy.
Online I found a number of tutorials to speed up thumbnail generation, some useful, others complete crap. I added the ones I found useful in the source list below.
What are we going to do?
What follows is my experience with Synology DS212J DSM4.1 & OSX Mountain Lion with Brew installed. We will go with the method of speeding up your thumbnail generation by doing all the processing on your local computer. In my case, that is a macbook pro i7. I'll show you three possible ways.
- Using the local convert binary on your mac to convert the image from the NAS to 4 smaller images. This will require 4 reads and 4 writes over your network and is a little slower
- Using a custom compiled c++ program to fetch the image once, and use this image in the memory to write the 4 thumbnails back to the NAS. This will require 1 read and 4 writes.
- Using this custom compiled c++ program and running this in parallel. Speedup++!
Speed?
Using this script I am able to convert more or less 1 image a second. This means it generates 4 new images a second and writes them to the NAS. Compared to the build-in thumbnail generation, this is lightning fast. I was already generating thumbnails on the nas for over 2 weeks, and it had generated less than 10000 photos. Now, in less than an hour, I am able to process more than 4000 pictures +-. I guess it is clear that this new method is WAY faster. Additional benefit is that my NAS nor my macbook utilise their CPU for the full 100%. Looks like I can optimise it even more. When I finish generating thumbnails for my 94.000 photos I will post the results here. Stay tuned!
Enable SSH access to your NAS & Mount your drive without locking
- Log in to your drive
ssh root@192.168.*.*<crubyde> * Type your admin password * <ruby>vi /etc/exports
- edit the /etc/exports file, and change all the "insecure_locks" entries to simply read "insecure". If you don't know vi: press the INS[SERT] key, use arrows to go to the line to edit, and delete/edit the line. Then press the ESC key, and type :x to 'Save & Exit'.
- More detailed explanation can be found on the web
Prepare your mac
if you have linux, it should be similar. Yum or apt-get will get you on your way.
brew install imagemagick
Mount your drive with your pictures to your Machine
mkdir /mnt && mkdir /mnt/photo sudo mount DS_IP:/volume1/photo /mnt/photo/
Create the most basic script
Create the following script and save it in /mnt/mkthumb_seq.sh
#!/bin/bash pushd "$1" shopt -s nocaseglob options_sharp="-unsharp 0.5x0.5+1.25+0.0" options="" if [[ ! -d @eaDir ]] then mkdir @eaDir echo "Thumbnail dir created" fi for f in *.jpg ; do if [[ "$f" == "*.jpg" ]] then break fi if [[ ! -d @eaDir/$f ]] then echo "$1 - $f : Making thumb dir" mkdir @eaDir/$f fi if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]] then convert $f -resize 1280x1280\> -quality 90 $options_sharp @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg echo "converted XL for $f" fi if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]] then convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_L.jpg echo "converted L for $f" fi if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]] then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_M.jpg echo "converted M for $f" fi if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]] then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_S.jpg echo "converted S for $f" fi if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]] then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_B.jpg echo "converted B for $f" fi done popd
Save this file and run it against a directory This should already succeed, but as I told you, this is not optimal yet due to the 4 network reads.
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_seq.sh {} \;
The C++ moment
Creating and compiling our C++ code. The following piece of code I found on a gist in github. While I did adjust it, it is in essence still the same. It basically does the following
- takes in a path to a picture
- loads the picture
- creates the 4 thumbnails
- creates the directory where the file should be stored
- writes them back to the proper directory structure where the Synology NAS expects them
Save the following code as /mnt/convertn.cpp
#include <iostream> #include <string> #include <libgen.h> #include <Magick++.h> #include <sys/stat.h> using namespace std; using namespace Magick; string thumbDir( ) { return "@eaDir"; } string outdir( string fn ) { return thumbDir() + "/" + fn; } string out( string fn, string sz ) { return outdir( fn ) + "/SYNOPHOTO:THUMB_" + sz + ".jpg"; } int main( int, char **argv) { InitializeMagick(*argv); Image im; string src = argv[1]; string fn = string( basename( strdup( src.c_str() ) ) ); string xl_path = out( fn, "XL" ); string path = outdir( fn ); string thumddir = thumbDir( ); struct convertn { string geometry; string size; }; convertn sizes [4] = { { "800x800>", "L" }, { "640x640>", "B" }, { "320x320>", "M" }, { "120x120>", "S" } }; // Create the directory mkdir ( thumddir.c_str(), 0755); mkdir ( path.c_str(), 0755); try { im.read( src ); im.scale("1280x1280>"); im.unsharpmask(0.5, 0.5, 1.25, 0.0); im.write( xl_path ); } catch (const std::exception &e) { cerr << e.what() << endl; } for ( int j=0; j<4; j++) { //cout << j << endl; try { im.scale( sizes[j].geometry ); //cout << out( fn, sizes[j].size ) << endl; im.write( out( fn, sizes[j].size ) ); } catch (const std::exception &e) { cerr << e.what() << endl; } } }
Compiling and using our C++ program
cd /mnt g++ `Magick++-config --cxxflags --cppflags --ldflags --libs` convertn.cpp -o convertn chmod +x convertn
Try out your C++ program! It should create @eaDir/imagefilename/SYNOPHOTO:THUMB_{XL/L/M/S].jpg files.
./convertn path_to_test_image.jpg
Using the C++ in the bash script
Let's use this program now in our bash script. Save the following as /mnt/mkthumb_cplusplus.sh
#!/bin/bash pushd "$1" shopt -s nocaseglob options_sharp="-unsharp 0.5x0.5+1.25+0.0" options="" if [[ ! -d @eaDir ]] then mkdir @eaDir echo "Thumbnail dir created" fi for f in *.jpg ; do if [[ "$f" == "*.jpg" ]] then break fi if [[ ! -d @eaDir/$f ]] then echo "$1 - $f : Making thumb dir" mkdir @eaDir/$f fi if [[ -z `ls -A "@eaDir/$f"` ]] then echo "Making Thumbs for $1 - $f" /mnt/convertn $f fi done popd
Clear thumbnail directory and execute & test the new script
rm -rf /mnt/photo/YOURTESTDIR/@eaDir find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;
Running in parallel
As a final step, let's make this run in parallel. the --jobs +8 stands for allowing the script to open up your amount of cores + 8 more jobs in parallel. In my case, this is 16. You should play a bit with this setting.
#!/bin/bash pushd "$1" shopt -s nocaseglob options_sharp="-unsharp 0.5x0.5+1.25+0.0" options="" parallel_command="sem --jobs +8 --quote " if [[ ! -d @eaDir ]] then mkdir @eaDir echo "Thumbnail dir created" fi for f in *.jpg ; do if [[ "$f" == "*.jpg" ]] then break fi if [[ ! -d @eaDir/$f ]] then echo "$1 - $f : Making thumb dir" mkdir @eaDir/$f fi if [[ -z `ls -A "@eaDir/$f"` ]] then echo "Making Thumbs for $1 - $f" $parallel_command /mnt/convertn $f fi done popd
Clear thumbnail directory and execute & test the new script
Enjoy speedy thumbnail generation ;-)
rm -rf /mnt/photo/YOURTESTDIR/@eaDir find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;
Comments
Thanks a ton for this! I
Thanks a ton for this! I couldn't get the parallel command part to work -- no "sem" on Mac OS X -- but this is still really handy.
If I wanted to run this against other filetypes (i.e. raw files) would I just change the file extension in the bash script?
Yes, just change the
Yes, just change the extension or improve the bash script. I actually made a modification to support jpg, JPG, jpeg, JPEG, png, PNG but I'm sure there are more as my synology had 1000 pictures left to convert (and I had no clue which ones they were). To install sem on osx -> brew install parallel
Nice work. Requires a lot of
Nice work. Requires a lot of unix knowledge but that never has hurt anyone.
Some observations:
GNU parallel/sem can be found here: http://www.gnu.org/software/parallel/
I wonder why you don't mount the volume via AFP? Could also be done via the finder for less experienced people. I find the AFP support adequate for these things.
I prefer to run scripts like these from xargs instead of through find -exec, so your commands can look like this:
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -print0 | xargs -0 /mnt/mkthumb_seq.sh
I tried both the script and
I tried both the script and the C++ program (Xubuntu 12.04), but could not get it working.
Running the basic script gives: "/mnt/mkthumb_seq.sh: Toegang geweigerd"
Compiling the C++ program gives:
"convertn.cpp: In functie ‘int main(int, char**)’:
convertn.cpp:27:55: fout: ‘strdup’ was not declared in this scope"
I am a complete programming-noob, btw...
Call me stupid, but on OS X
Call me stupid, but on OS X Mountain Lion, you can't do a mkdir @eaDir In fact you can't create any directory that starts with an @ symbol. It can contain an @ symbol, but not start with one. I would love to do this all form my Mac, but am fearful, I'll need to do this through a Linux VM, which is not what I want to do. Any idea on how to get OS X to let me create directories that start with @
nvm... that was a stupid
nvm... that was a stupid moment... Just had to do it as root :)
SOOOO HAPPY!!! This is
SOOOO HAPPY!!! This is working now, and fast. I have well over 50K photos, and my DS212 couldn't handle it. I did need to make some minor mods to this.
1) I added an im.quality(70); to .cpp file after line 51.
I'm okay with the thumbs being of lower quality, and given I have so many pics, I didn't want to consume uneeded space. Wouldn't hurt to make this a command line arg, but hardcoding it is fine. If I ever want a different quality, I'll just recompile it.
2) Many of my pics have spaces in the file name, and the script was puking on it Here is an updated script file that works with spaces in the file names for the pics.
!/bin/bash
pushd "$1"
shopt -s nocaseglob
parallel_command="sem --jobs +8 --quote "
if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi
for f in .jpg ; do
if [[ "$f" == ".jpg" ]]
then
break
fi
if [[ ! -d "@eaDir/$f" ]]
then
echo "$1 - $f : Making thumb dir"
mkdir "@eaDir/$f"
fi
if [[ -z
ls -A "@eaDir/$f"
]]then
echo "Making Thumbs for $1 - $f"
$parallel_command /mnt/convertn "$f"
fi
done
popd
Hi,
Hi,
I am complete noob in this. I have few questions.
1. I have a macbook air 2012 with 8 gigs of RAM and latest processor at that time. Will this be okay running on that?
2. I also have a windows desktop with 8 gigs of RAM and E8400 processor. Which one would be better and in case of windows how would I run this on windows?
3. If running on macbook is okay, then I modified the exports file on my synology. Then installed brew on my macbook, then the mkdir and all other commands should run on mac or synology? By Synology I mean logging in on synology using mac terminal. I am getting confused can someone pls help?
Thanks
Correcting the 3rd point. I
Correcting the 3rd point. I tried running the sudo command by giving my ip address for the synology, at this time I was logged in as a root user on my mac. I am getting permission denied error. What should I do?
I had the problem, that there
I had the problem, that there were already folders names @eaDir but with only the XL thumbnail created.
The solution was to test for the small thumbnails being there. You have to change the condition in the shell script from
if [[ -z `ls -A "@eaDir/$f"` ]]
to
lsdir=`ls -A "@eaDir/$f"`
if [[ -z $lsdir || !($lsdir =~ .*THUMB_S.*)]] #empty or not all thumbs created
change line-2 to:
change line-2 to:
#include <string.h>
Thanks for sharing excellent
Thanks for sharing excellent informations. Your web-site is very cool. I am impressed by the details that you’ve on this blog. It reveals how nicely you perceive this subject. Bookmarked this website page :)
Using Ubuntu, I managed to
Using Ubuntu, I managed to compile the C++ using :
g++ `Magick++-config --cxxflags --cppflags` -O2 -Wall -o convertn thumbnail_generator.cpp `Magick++-config --ldflags --libs`
I also had to #include <string.h> instead of <string>
My problem is on the sem side : sem is always giving me errors :
$ sem "echo foo"
/bin/bash: -c: option requires an argument
It seems like sem is calling bash using the -c flag but without the command line parameter. I digged in the man pages, googled a lot and I seem to be the only one with this issue, so I guess the problem is somewhere in my config but I cannot find out.
An idea someone ?
I too used Ubuntu (v13.04 64
I too used Ubuntu (v13.04 64 bit) running in a VM (Oracle Virtual Box) on a Windows 7 machine to handle this. After a lot of irritating issues, I finally got it to work and others might be interested in my findings so they don't have to reinvent the wheel.
Nick does not make it clear that NFS has to be installed on Ubuntu - it isn't standard. Maybe it is on OS-X? Until you do this, you cannot mount the Synology directory.
I also had problems with Ubuntu not recognising pushd, popd and shopt commands in Bash. It appears that by default, the Ubuntu shell is 'dash' I needed to 'sudo rm -f /bin/sh' and then 'sudo ln -s /bin/bash bin/sh' from the Linux command line. In addition, the use of double [[ is NOT needed in Ubuntu - change every instance of [[ to [ in the shell scripts. I also added the -auto-orient option to the convert commands, as suggested in other blogs. This ensures that the picture orientation is correct:
convert -auto-orient @eaDir etc etc
Initially, I had BIG problems with trying to compile the C++ code. It was as though the compiler did not 'know' about the Magick++ libraries. In fact, I had given up - and then, after restarting the VM the next day, and giving it one more try, it all worked!! Why? I don't know. But - if your compiler complains about not "understanding" magick++, it might be worth powering off and on again......
I also agree with other comments on here - it needs to be #include <string.h> and the compile command MUST be 'split':
g++ `Magick++-config --cxflags -cppflags` convertn.cpp -o convertn `Magick++-config --ldflags --libs`
Also - VERY IMPORTANT - the ` around the magick++ components are backward quotes - the one above the tilde key.
Now, with all these irritations out of the way, I can produce the thumbnails for each photo at a rate of about 1 second per picture - about 3600 per hour. I haven't yet tried the parallel processing - which may speed it up further.
So, one last question. The -auto-orient option in the shell script is needed to ensure the thumbnails have correct orientation. Is there a similar flag required in the C++ code to do the same thing? If so, what? (It appears from a trial run of a couple of hundred photos that they are mostly correctly oriented - so maybe it is built in?).
Thanks for a great way to handle such a poor implementation of photo processing!
I have the exact same problem
I have the exact same problem on ubuntu - it has something to do with calling "parallel --bg". If you remove the --bg switch the error disappears, but then you're not really running this script in parallel.
So this sucks - I tried to google and struggle for it for some hours, but think I must give up. I suspect it's a bug, I just don't understand why nobody else found this problem...
Am talking about the sem side
Am talking about the sem side: /bin/bash: -c: option requires an argument
Aaah, I think I finally found
Aaah, I think I finally found out what the problem (/bin/bash: -c: option requires an argument) is and succeeded!
You have to use --gnu in the command line for parallel because it defaults to something "--tollef" for some reason and you do not want that!
Hi, first of all thank you
Hi, first of all thank you for your really helpful tool.
At a first glance it works great for me - all my thumbs (about 30 k, DNGs, jgps) etc. have been created successfully. Nevertheless, the Synology creation (synomkthumd) immediately starts when I start the Synology Photo Station afterwards. Do you have any ideas?
Thank you so much.
Perhaps for video thumbnail
Perhaps for video thumbnail creation?
add a line at the beginning
add a line at the beginning of the cpp file
#include <cstring>
If you (like me) get errors compiling like
... undefined "Magick::InitializeMagick(char const*)"
use this command to compile
g++ `Magick++-config --cxxflags --cppflags` -O2 -Wall -o convertn convertn.cpp `Magick++-config --ldflags --libs`
Is this still valid?
Is this still valid?
I've upgraded to DSM 5.0, using DS212J.
The conversion progress is just impossibly slow!
Here's my Python alternative
Here's my Python alternative if anyone is interested:
https://github.com/mbrrg/synology-thumbgen
Ran into a bunch of problems,
Ran into a bunch of problems, but was able to solve them all. But I think that I'm still having issues either with files that have spaces or ()'1:
/mnt/photo/from Everpix/JPG's from Everpix - DSC01692 (1).jpg : Making thumb dir
mkdir: (1).jpg: File exists
ls: @eaDir/DSC01692 (1).jpg: No such file or directory
Making Thumbs for /mnt/photo/from Everpix/JPG's from Everpix - DSC01692 (1).jpg
/mnt/photo/from Everpix/JPG's from Everpix - DSC01692.jpg : Making thumb dir
Making Thumbs for /mnt/photo/from Everpix/JPG's from Everpix - DSC01692.jpg
convertn: unable to open image `DSC01692': No such file or directory @ error/blob.c/OpenBlob/2709
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_L.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_B.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_M.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_S.jpg' @ error/cache.c/OpenPixelCache/3534
I tried to change the original script, to add the space fix that Pierre mentioned, but ran into errors.
Any idea?
THANK YOU
getting a bunch of these:
getting a bunch of these:
Warning: Semaphore stuck for 30 seconds. Consider using --semaphoretimeout.parallel
any idea?
On my DiskStation, mounting
On my DiskStation, mounting via AFP does not expose the @eaDir directories that were already created, I'm assuming NFS was used because of that.
Thank you so much for sharing
Thank you so much for sharing words of the wisdom. It was exactly what I was looking for because synology was taking ages to convert pictures. But your bash script wasn't working for me out of the box
I was always getting follow error message:
convert: not authorized `@eaDir/IMG_4036.JPG/SYNOPHOTO:THUMB_XL.jpg' @ error/blob.c/OpenBlob/2468.
Therefore, I did a dirty work around:
#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""
if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi
for f in *.jpg ; do
if [[ "$f" == "*.jpg" ]]
then
break
fi
if [[ ! -d @eaDir/$f ]]
then
echo "$1 - $f : Making thumb dir"
mkdir @eaDir/$f
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]]
then
cd @eaDir/$f
convert ../../$f -resize 1280x1280\> -quality 90 $options_sharp ./SYNOPHOTO:THUMB_XL.jpg
cd ../../
echo "converted XL for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options ./SYNOPHOTO:THUMB_L.jpg
cd ../../
echo "converted L for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options ./SYNOPHOTO:THUMB_M.jpg
cd ../../
echo "converted M for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options ./SYNOPHOTO:THUMB_S.jpg
cd ../../
echo "converted S for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options ./SYNOPHOTO:THUMB_B.jpg
cd ../../
echo "converted B for $f"
fi
done
popd
Best alternative! No bash/cpp
Best alternative! No bash/cpp headaches and works as charm!