Speed up thumbnail generation on Synology DS212J

I recently bought a Synology 212J to store all my pictures on, and of course also to play with. I always had some crappy tiny non-modifyable NAS/Media Center solution that I really hated. cough Plextor cough

So, I started adding pictures to the NAS and I wanted to play with the photo station application that is included. However, I noticed that as soon as I added pictures to the drive, it completely stalled. CPU utilisation went up to 100% 24/7. This also meant that it never went into idle mode and consumed more energy.

Online I found a number of tutorials to speed up thumbnail generation, some useful, others complete crap. I added the ones I found useful in the source list below.

What are we going to do?

What follows is my experience with Synology DS212J DSM4.1 & OSX Mountain Lion with Brew installed. We will go with the method of speeding up your thumbnail generation by doing all the processing on your local computer. In my case, that is a macbook pro i7. I'll show you three possible ways.

  • Using the local convert binary on your mac to convert the image from the NAS to 4 smaller images. This will require 4 reads and 4 writes over your network and is a little slower
  • Using a custom compiled c++ program to fetch the image once, and use this image in the memory to write the 4 thumbnails back to the NAS. This will require 1 read and 4 writes.
  • Using this custom compiled c++ program and running this in parallel. Speedup++!

Speed?

Using this script I am able to convert more or less 1 image a second. This means it generates 4 new images a second and writes them to the NAS. Compared to the build-in thumbnail generation, this is lightning fast. I was already generating thumbnails on the nas for over 2 weeks, and it had generated less than 10000 photos. Now, in less than an hour, I am able to process more than 4000 pictures +-. I guess it is clear that this new method is WAY faster. Additional benefit is that my NAS nor my macbook utilise their CPU for the full 100%. Looks like I can optimise it even more. When I finish generating thumbnails for my 94.000 photos I will post the results here. Stay tuned!

Enable SSH access to your NAS & Mount your drive without locking

  • Log in to your drive
  1. ssh root@192.168.*.*<crubyde>
  2. * Type your admin password
  3. * <ruby>vi /etc/exports
  • edit the /etc/exports file, and change all the "insecure_locks" entries to simply read "insecure". If you don't know vi: press the INS[SERT] key, use arrows to go to the line to edit, and delete/edit the line. Then press the ESC key, and type :x to 'Save & Exit'.
  • More detailed explanation can be found on the web

Prepare your mac

if you have linux, it should be similar. Yum or apt-get will get you on your way.

  1. brew install imagemagick

Mount your drive with your pictures to your Machine

  1. mkdir /mnt && mkdir /mnt/photo
  2. sudo mount DS_IP:/volume1/photo /mnt/photo/

Create the most basic script

Create the following script and save it in /mnt/mkthumb_seq.sh

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6.  
  7. if [[ ! -d @eaDir ]]
  8.   then
  9.   mkdir @eaDir
  10.   echo "Thumbnail dir created"
  11. fi
  12.  
  13. for f in *.jpg ; do
  14.   if [[ "$f" == "*.jpg" ]]
  15.     then
  16.     break
  17.   fi
  18.  
  19.   if [[ ! -d @eaDir/$f ]]
  20.     then 
  21.     echo "$1 - $f : Making thumb dir"
  22.     mkdir @eaDir/$f 
  23.   fi
  24.  
  25.   if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]] 
  26.     then
  27.     convert $f -resize 1280x1280\> -quality 90 $options_sharp @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg
  28.     echo "converted XL for $f"
  29.   fi
  30.  
  31.   if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
  32.     then
  33.     convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_L.jpg
  34.     echo "converted L for $f"
  35.   fi
  36.  
  37.   if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
  38.     then
  39.     convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_M.jpg
  40.     echo "converted M for $f"
  41.   fi
  42.  
  43.   if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
  44.     then 
  45.     convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_S.jpg
  46.     echo "converted S for $f"
  47.   fi
  48.  
  49.   if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
  50.     then
  51.     convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_B.jpg
  52.     echo "converted B for $f"
  53.   fi
  54. done
  55. popd

Save this file and run it against a directory This should already succeed, but as I told you, this is not optimal yet due to the 4 network reads.

  1. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_seq.sh {} \;

The C++ moment

Creating and compiling our C++ code. The following piece of code I found on a gist in github. While I did adjust it, it is in essence still the same. It basically does the following

  • takes in a path to a picture
  • loads the picture
  • creates the 4 thumbnails
  • creates the directory where the file should be stored
  • writes them back to the proper directory structure where the Synology NAS expects them

Save the following code as /mnt/convertn.cpp

  1. #include <iostream>
  2. #include <string>
  3. #include <libgen.h>
  4. #include <Magick++.h>
  5. #include <sys/stat.h>
  6.  
  7. using namespace std;
  8. using namespace Magick;
  9.  
  10. string thumbDir( ) {
  11.   return "@eaDir";
  12. }
  13.  
  14. string outdir( string fn ) {
  15.   return thumbDir() + "/" + fn;
  16. }
  17.  
  18. string out( string fn, string sz ) {
  19.   return outdir( fn ) + "/SYNOPHOTO:THUMB_" + sz + ".jpg";
  20. }
  21.  
  22.  
  23. int main( int, char **argv) {
  24.   InitializeMagick(*argv);
  25.   Image im;
  26.   string src = argv[1];
  27.   string fn = string( basename( strdup( src.c_str() ) ) );
  28.   string xl_path =  out( fn, "XL" );
  29.   string path = outdir( fn );
  30.   string thumddir = thumbDir( );
  31.  
  32.   struct convertn {
  33.     string geometry;
  34.     string size;
  35.   };
  36.  
  37.   convertn sizes [4] = {
  38.     { "800x800>", "L" },
  39.     { "640x640>", "B" },
  40.     { "320x320>", "M" },
  41.     { "120x120>", "S" }
  42.   };
  43.  
  44.   // Create the directory
  45.   mkdir ( thumddir.c_str(), 0755);
  46.   mkdir ( path.c_str(), 0755);
  47.  
  48.   try {
  49.     im.read( src );
  50.     im.scale("1280x1280>");
  51.     im.unsharpmask(0.5, 0.5, 1.25, 0.0);
  52.     im.write( xl_path );
  53.   } catch (const std::exception &e) {
  54.     cerr << e.what() << endl;
  55.   }
  56.   for ( int j=0; j<4; j++) {
  57.     //cout << j << endl; 
  58.     try {
  59.     im.scale( sizes[j].geometry );
  60.     //cout << out( fn, sizes[j].size ) << endl;
  61.     im.write( out( fn, sizes[j].size ) );
  62.     } catch (const std::exception &e) {
  63.       cerr << e.what() << endl;
  64.     }
  65.   }
  66. }

Compiling and using our C++ program

  1. cd /mnt
  2. g++ `Magick++-config --cxxflags --cppflags --ldflags --libs` convertn.cpp -o convertn
  3. chmod +x convertn

Try out your C++ program! It should create @eaDir/imagefilename/SYNOPHOTO:THUMB_{XL/L/M/S].jpg files.

  1. ./convertn path_to_test_image.jpg

Using the C++ in the bash script

Let's use this program now in our bash script. Save the following as /mnt/mkthumb_cplusplus.sh

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6.  
  7. if [[ ! -d @eaDir ]]
  8.   then
  9.   mkdir @eaDir
  10.   echo "Thumbnail dir created"
  11. fi
  12.  
  13. for f in *.jpg ; do
  14.   if [[ "$f" == "*.jpg" ]]
  15.     then
  16.     break
  17.   fi
  18.  
  19.   if [[ ! -d @eaDir/$f ]]
  20.     then 
  21.     echo "$1 - $f : Making thumb dir"
  22.     mkdir @eaDir/$f 
  23.   fi
  24.   if [[ -z `ls -A "@eaDir/$f"` ]]
  25.     then
  26.     echo "Making Thumbs for $1 - $f"
  27.     /mnt/convertn $f
  28.   fi
  29. done
  30. popd

Clear thumbnail directory and execute & test the new script

  1. rm -rf /mnt/photo/YOURTESTDIR/@eaDir
  2. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Running in parallel

As a final step, let's make this run in parallel. the --jobs +8 stands for allowing the script to open up your amount of cores + 8 more jobs in parallel. In my case, this is 16. You should play a bit with this setting.

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6. parallel_command="sem --jobs +8 --quote "
  7.  
  8. if [[ ! -d @eaDir ]]
  9.   then
  10.   mkdir @eaDir
  11.   echo "Thumbnail dir created"
  12. fi
  13.  
  14. for f in *.jpg ; do
  15.   if [[ "$f" == "*.jpg" ]]
  16.     then
  17.     break
  18.   fi
  19.  
  20.   if [[ ! -d @eaDir/$f ]]
  21.     then 
  22.     echo "$1 - $f : Making thumb dir"
  23.     mkdir @eaDir/$f 
  24.   fi
  25.   if [[ -z `ls -A "@eaDir/$f"` ]]
  26.     then
  27.     echo "Making Thumbs for $1 - $f"
  28.     $parallel_command /mnt/convertn $f
  29.   fi
  30. done
  31. popd

Clear thumbnail directory and execute & test the new script

Enjoy speedy thumbnail generation ;-)

  1. rm -rf /mnt/photo/YOURTESTDIR/@eaDir
  2. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Sources

Nick
Sunday, November 25, 2012 - 20:42

Comments

On , Neil said...

Thanks a ton for this! I couldn't get the parallel command part to work -- no "sem" on Mac OS X -- but this is still really handy.

If I wanted to run this against other filetypes (i.e. raw files) would I just change the file extension in the bash script?

On , Nick said...

Yes, just change the extension or improve the bash script. I actually made a modification to support jpg, JPG, jpeg, JPEG, png, PNG but I'm sure there are more as my synology had 1000 pictures left to convert (and I had no clue which ones they were). To install sem on osx -> brew install parallel

On , Hendrik said...

Nice work. Requires a lot of unix knowledge but that never has hurt anyone.
Some observations:

GNU parallel/sem can be found here: http://www.gnu.org/software/parallel/

I wonder why you don't mount the volume via AFP? Could also be done via the finder for less experienced people. I find the AFP support adequate for these things.

I prefer to run scripts like these from xargs instead of through find -exec, so your commands can look like this:
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -print0 | xargs -0 /mnt/mkthumb_seq.sh

On , Bert said...

I tried both the script and the C++ program (Xubuntu 12.04), but could not get it working.
Running the basic script gives: "/mnt/mkthumb_seq.sh: Toegang geweigerd"
Compiling the C++ program gives:
"convertn.cpp: In functie ‘int main(int, char**)’:
convertn.cpp:27:55: fout: ‘strdup’ was not declared in this scope"

I am a complete programming-noob, btw...

On , Pierre said...

Call me stupid, but on OS X Mountain Lion, you can't do a mkdir @eaDir In fact you can't create any directory that starts with an @ symbol. It can contain an @ symbol, but not start with one. I would love to do this all form my Mac, but am fearful, I'll need to do this through a Linux VM, which is not what I want to do. Any idea on how to get OS X to let me create directories that start with @

On , Pierre said...

nvm... that was a stupid moment... Just had to do it as root :)

On , Pierre said...

SOOOO HAPPY!!! This is working now, and fast. I have well over 50K photos, and my DS212 couldn't handle it. I did need to make some minor mods to this.

1) I added an im.quality(70); to .cpp file after line 51.
I'm okay with the thumbs being of lower quality, and given I have so many pics, I didn't want to consume uneeded space. Wouldn't hurt to make this a command line arg, but hardcoding it is fine. If I ever want a different quality, I'll just recompile it.

2) Many of my pics have spaces in the file name, and the script was puking on it Here is an updated script file that works with spaces in the file names for the pics.

!/bin/bash

pushd "$1"
shopt -s nocaseglob
parallel_command="sem --jobs +8 --quote "

if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi

for f in .jpg ; do
if [[ "$f" == ".jpg" ]]
then
break
fi

if [[ ! -d "@eaDir/$f" ]]
then
echo "$1 - $f : Making thumb dir"
mkdir "@eaDir/$f"
fi
if [[ -z ls -A "@eaDir/$f" ]]
then
echo "Making Thumbs for $1 - $f"
$parallel_command /mnt/convertn "$f"
fi
done
popd

On , AJ said...

Hi,
I am complete noob in this. I have few questions.
1. I have a macbook air 2012 with 8 gigs of RAM and latest processor at that time. Will this be okay running on that?
2. I also have a windows desktop with 8 gigs of RAM and E8400 processor. Which one would be better and in case of windows how would I run this on windows?
3. If running on macbook is okay, then I modified the exports file on my synology. Then installed brew on my macbook, then the mkdir and all other commands should run on mac or synology? By Synology I mean logging in on synology using mac terminal. I am getting confused can someone pls help?
Thanks

On , AJ said...

Correcting the 3rd point. I tried running the sudo command by giving my ip address for the synology, at this time I was logged in as a root user on my mac. I am getting permission denied error. What should I do?

On , Anonymous said...

I had the problem, that there were already folders names @eaDir but with only the XL thumbnail created.

The solution was to test for the small thumbnails being there. You have to change the condition in the shell script from

if [[ -z `ls -A "@eaDir/$f"` ]]

to

lsdir=`ls -A "@eaDir/$f"`
if [[ -z $lsdir || !($lsdir =~ .*THUMB_S.*)]] #empty or not all thumbs created

On , Anonymous said...

change line-2 to:
#include <string.h>

On , number book said...

Thanks for sharing excellent informations. Your web-site is very cool. I am impressed by the details that you’ve on this blog. It reveals how nicely you perceive this subject. Bookmarked this website page :)

On , Heecks said...

Using Ubuntu, I managed to compile the C++ using :
g++ `Magick++-config --cxxflags --cppflags` -O2 -Wall -o convertn thumbnail_generator.cpp `Magick++-config --ldflags --libs`

I also had to #include <string.h> instead of <string>

My problem is on the sem side : sem is always giving me errors :
$ sem "echo foo"
/bin/bash: -c: option requires an argument

It seems like sem is calling bash using the -c flag but without the command line parameter. I digged in the man pages, googled a lot and I seem to be the only one with this issue, so I guess the problem is somewhere in my config but I cannot find out.

An idea someone ?

On , Mike said...

I too used Ubuntu (v13.04 64 bit) running in a VM (Oracle Virtual Box) on a Windows 7 machine to handle this. After a lot of irritating issues, I finally got it to work and others might be interested in my findings so they don't have to reinvent the wheel.

Nick does not make it clear that NFS has to be installed on Ubuntu - it isn't standard. Maybe it is on OS-X? Until you do this, you cannot mount the Synology directory.

I also had problems with Ubuntu not recognising pushd, popd and shopt commands in Bash. It appears that by default, the Ubuntu shell is 'dash' I needed to 'sudo rm -f /bin/sh' and then 'sudo ln -s /bin/bash bin/sh' from the Linux command line. In addition, the use of double [[ is NOT needed in Ubuntu - change every instance of [[ to [ in the shell scripts. I also added the -auto-orient option to the convert commands, as suggested in other blogs. This ensures that the picture orientation is correct:

convert -auto-orient @eaDir etc etc

Initially, I had BIG problems with trying to compile the C++ code. It was as though the compiler did not 'know' about the Magick++ libraries. In fact, I had given up - and then, after restarting the VM the next day, and giving it one more try, it all worked!! Why? I don't know. But - if your compiler complains about not "understanding" magick++, it might be worth powering off and on again......

I also agree with other comments on here - it needs to be #include <string.h> and the compile command MUST be 'split':

g++ `Magick++-config --cxflags -cppflags` convertn.cpp -o convertn `Magick++-config --ldflags --libs`

Also - VERY IMPORTANT - the ` around the magick++ components are backward quotes - the one above the tilde key.

Now, with all these irritations out of the way, I can produce the thumbnails for each photo at a rate of about 1 second per picture - about 3600 per hour. I haven't yet tried the parallel processing - which may speed it up further.

So, one last question. The -auto-orient option in the shell script is needed to ensure the thumbnails have correct orientation. Is there a similar flag required in the C++ code to do the same thing? If so, what? (It appears from a trial run of a couple of hundred photos that they are mostly correctly oriented - so maybe it is built in?).

Thanks for a great way to handle such a poor implementation of photo processing!

On , someone too said...

I have the exact same problem on ubuntu - it has something to do with calling "parallel --bg". If you remove the --bg switch the error disappears, but then you're not really running this script in parallel.

So this sucks - I tried to google and struggle for it for some hours, but think I must give up. I suspect it's a bug, I just don't understand why nobody else found this problem...

On , someone said...

Am talking about the sem side: /bin/bash: -c: option requires an argument

On , someone said...

Aaah, I think I finally found out what the problem (/bin/bash: -c: option requires an argument) is and succeeded!

You have to use --gnu in the command line for parallel because it defaults to something "--tollef" for some reason and you do not want that!

On , Tom said...

Hi, first of all thank you for your really helpful tool.
At a first glance it works great for me - all my thumbs (about 30 k, DNGs, jgps) etc. have been created successfully. Nevertheless, the Synology creation (synomkthumd) immediately starts when I start the Synology Photo Station afterwards. Do you have any ideas?
Thank you so much.

On , Anonymous said...

Perhaps for video thumbnail creation?

On , giucas said...

add a line at the beginning of the cpp file
#include <cstring>

If you (like me) get errors compiling like
... undefined "Magick::InitializeMagick(char const*)"

use this command to compile
g++ `Magick++-config --cxxflags --cppflags` -O2 -Wall -o convertn convertn.cpp `Magick++-config --ldflags --libs`

On , Gordon said...

Is this still valid?

I've upgraded to DSM 5.0, using DS212J.

The conversion progress is just impossibly slow!

On , Martin said...

Here's my Python alternative if anyone is interested:
https://github.com/mbrrg/synology-thumbgen

On , Ziv said...

Ran into a bunch of problems, but was able to solve them all. But I think that I'm still having issues either with files that have spaces or ()'1:

/mnt/photo/from Everpix/JPG's from Everpix - DSC01692 (1).jpg : Making thumb dir
mkdir: (1).jpg: File exists
ls: @eaDir/DSC01692 (1).jpg: No such file or directory
Making Thumbs for /mnt/photo/from Everpix/JPG's from Everpix - DSC01692 (1).jpg
/mnt/photo/from Everpix/JPG's from Everpix - DSC01692.jpg : Making thumb dir
Making Thumbs for /mnt/photo/from Everpix/JPG's from Everpix - DSC01692.jpg
convertn: unable to open image `DSC01692': No such file or directory @ error/blob.c/OpenBlob/2709
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_L.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_B.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_M.jpg' @ error/cache.c/OpenPixelCache/3534
convertn: no pixels defined in cache `@eaDir/DSC01692/SYNOPHOTO:THUMB_S.jpg' @ error/cache.c/OpenPixelCache/3534

I tried to change the original script, to add the space fix that Pierre mentioned, but ran into errors.

Any idea?

THANK YOU

On , Ziv said...

getting a bunch of these:

Warning: Semaphore stuck for 30 seconds. Consider using --semaphoretimeout.parallel

any idea?

On , Chris said...

On my DiskStation, mounting via AFP does not expose the @eaDir directories that were already created, I'm assuming NFS was used because of that.

On , Navneet Tohan said...

Thank you so much for sharing words of the wisdom. It was exactly what I was looking for because synology was taking ages to convert pictures. But your bash script wasn't working for me out of the box

I was always getting follow error message:

convert: not authorized `@eaDir/IMG_4036.JPG/SYNOPHOTO:THUMB_XL.jpg' @ error/blob.c/OpenBlob/2468.

Therefore, I did a dirty work around:

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""

if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi

for f in *.jpg ; do
if [[ "$f" == "*.jpg" ]]
then
break
fi

if [[ ! -d @eaDir/$f ]]
then
echo "$1 - $f : Making thumb dir"
mkdir @eaDir/$f
fi

if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]]
then
cd @eaDir/$f
convert ../../$f -resize 1280x1280\> -quality 90 $options_sharp ./SYNOPHOTO:THUMB_XL.jpg
cd ../../
echo "converted XL for $f"
fi

if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options ./SYNOPHOTO:THUMB_L.jpg
cd ../../
echo "converted L for $f"
fi

if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options ./SYNOPHOTO:THUMB_M.jpg
cd ../../
echo "converted M for $f"
fi

if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options ./SYNOPHOTO:THUMB_S.jpg
cd ../../
echo "converted S for $f"
fi

if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
then
cd @eaDir/$f
convert ./SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options ./SYNOPHOTO:THUMB_B.jpg
cd ../../
echo "converted B for $f"
fi
done
popd

On , mrchief said...

Best alternative! No bash/cpp headaches and works as charm!