Speed up thumbnail generation on Synology DS212J

I recently bought a Synology 212J to store all my pictures on, and of course also to play with. I always had some crappy tiny non-modifyable NAS/Media Center solution that I really hated. cough Plextor cough

So, I started adding pictures to the NAS and I wanted to play with the photo station application that is included. However, I noticed that as soon as I added pictures to the drive, it completely stalled. CPU utilisation went up to 100% 24/7. This also meant that it never went into idle mode and consumed more energy.

Online I found a number of tutorials to speed up thumbnail generation, some useful, others complete crap. I added the ones I found useful in the source list below.

What are we going to do?

What follows is my experience with Synology DS212J DSM4.1 & OSX Mountain Lion with Brew installed. We will go with the method of speeding up your thumbnail generation by doing all the processing on your local computer. In my case, that is a macbook pro i7. I'll show you three possible ways.

  • Using the local convert binary on your mac to convert the image from the NAS to 4 smaller images. This will require 4 reads and 4 writes over your network and is a little slower
  • Using a custom compiled c++ program to fetch the image once, and use this image in the memory to write the 4 thumbnails back to the NAS. This will require 1 read and 4 writes.
  • Using this custom compiled c++ program and running this in parallel. Speedup++!

Speed?

Using this script I am able to convert more or less 1 image a second. This means it generates 4 new images a second and writes them to the NAS. Compared to the build-in thumbnail generation, this is lightning fast. I was already generating thumbnails on the nas for over 2 weeks, and it had generated less than 10000 photos. Now, in less than an hour, I am able to process more than 4000 pictures +-. I guess it is clear that this new method is WAY faster. Additional benefit is that my NAS nor my macbook utilise their CPU for the full 100%. Looks like I can optimise it even more. When I finish generating thumbnails for my 94.000 photos I will post the results here. Stay tuned!

Enable SSH access to your NAS & Mount your drive without locking

  • Log in to your drive
  1. ssh root@192.168.*.*<crubyde>
  2. * Type your admin password
  3. * <ruby>vi /etc/exports
  • edit the /etc/exports file, and change all the "insecure_locks" entries to simply read "insecure". If you don't know vi: press the INS[SERT] key, use arrows to go to the line to edit, and delete/edit the line. Then press the ESC key, and type :x to 'Save & Exit'.
  • More detailed explanation can be found on the web

Prepare your mac

if you have linux, it should be similar. Yum or apt-get will get you on your way.

  1. brew install imagemagick

Mount your drive with your pictures to your Machine

  1. mkdir /mnt && mkdir /mnt/photo
  2. sudo mount DS_IP:/volume1/photo /mnt/photo/

Create the most basic script

Create the following script and save it in /mnt/mkthumb_seq.sh

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6.  
  7. if [[ ! -d @eaDir ]]
  8. then
  9. mkdir @eaDir
  10. echo "Thumbnail dir created"
  11. fi
  12.  
  13. for f in *.jpg ; do
  14. if [[ "$f" == "*.jpg" ]]
  15. then
  16. break
  17. fi
  18.  
  19. if [[ ! -d @eaDir/$f ]]
  20. then
  21. echo "$1 - $f : Making thumb dir"
  22. mkdir @eaDir/$f
  23. fi
  24.  
  25. if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]]
  26. then
  27. convert $f -resize 1280x1280\> -quality 90 $options_sharp @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg
  28. echo "converted XL for $f"
  29. fi
  30.  
  31. if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
  32. then
  33. convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_L.jpg
  34. echo "converted L for $f"
  35. fi
  36.  
  37. if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
  38. then
  39. convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_M.jpg
  40. echo "converted M for $f"
  41. fi
  42.  
  43. if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
  44. then
  45. convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_S.jpg
  46. echo "converted S for $f"
  47. fi
  48.  
  49. if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
  50. then
  51. convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_B.jpg
  52. echo "converted B for $f"
  53. fi
  54. done
  55. popd

Save this file and run it against a directory This should already succeed, but as I told you, this is not optimal yet due to the 4 network reads.

  1. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_seq.sh {} \;

The C++ moment

Creating and compiling our C++ code. The following piece of code I found on a gist in github. While I did adjust it, it is in essence still the same. It basically does the following

  • takes in a path to a picture
  • loads the picture
  • creates the 4 thumbnails
  • creates the directory where the file should be stored
  • writes them back to the proper directory structure where the Synology NAS expects them

Save the following code as /mnt/convertn.cpp

  1. #include <iostream>
  2. #include <string>
  3. #include <libgen.h>
  4. #include <Magick++.h>
  5. #include <sys/stat.h>
  6.  
  7. using namespace std;
  8. using namespace Magick;
  9.  
  10. string thumbDir( ) {
  11. return "@eaDir";
  12. }
  13.  
  14. string outdir( string fn ) {
  15. return thumbDir() + "/" + fn;
  16. }
  17.  
  18. string out( string fn, string sz ) {
  19. return outdir( fn ) + "/SYNOPHOTO:THUMB_" + sz + ".jpg";
  20. }
  21.  
  22.  
  23. int main( int, char **argv) {
  24. InitializeMagick(*argv);
  25. Image im;
  26. string src = argv[1];
  27. string fn = string( basename( strdup( src.c_str() ) ) );
  28. string xl_path = out( fn, "XL" );
  29. string path = outdir( fn );
  30. string thumddir = thumbDir( );
  31.  
  32. struct convertn {
  33. string geometry;
  34. string size;
  35. };
  36.  
  37. convertn sizes [4] = {
  38. { "800x800>", "L" },
  39. { "640x640>", "B" },
  40. { "320x320>", "M" },
  41. { "120x120>", "S" }
  42. };
  43.  
  44. // Create the directory
  45. mkdir ( thumddir.c_str(), 0755);
  46. mkdir ( path.c_str(), 0755);
  47.  
  48. try {
  49. im.read( src );
  50. im.scale("1280x1280>");
  51. im.unsharpmask(0.5, 0.5, 1.25, 0.0);
  52. im.write( xl_path );
  53. } catch (const std::exception &e) {
  54. cerr << e.what() << endl;
  55. }
  56. for ( int j=0; j<4; j++) {
  57. //cout << j << endl;
  58. try {
  59. im.scale( sizes[j].geometry );
  60. //cout << out( fn, sizes[j].size ) << endl;
  61. im.write( out( fn, sizes[j].size ) );
  62. } catch (const std::exception &e) {
  63. cerr << e.what() << endl;
  64. }
  65. }
  66. }

Compiling and using our C++ program

  1. cd /mnt
  2. g++ `Magick++-config --cxxflags --cppflags --ldflags --libs` convertn.cpp -o convertn
  3. chmod +x convertn

Try out your C++ program! It should create @eaDir/imagefilename/SYNOPHOTO:THUMB_{XL/L/M/S].jpg files.

  1. ./convertn path_to_test_image.jpg

Using the C++ in the bash script

Let's use this program now in our bash script. Save the following as /mnt/mkthumb_cplusplus.sh

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6.  
  7. if [[ ! -d @eaDir ]]
  8. then
  9. mkdir @eaDir
  10. echo "Thumbnail dir created"
  11. fi
  12.  
  13. for f in *.jpg ; do
  14. if [[ "$f" == "*.jpg" ]]
  15. then
  16. break
  17. fi
  18.  
  19. if [[ ! -d @eaDir/$f ]]
  20. then
  21. echo "$1 - $f : Making thumb dir"
  22. mkdir @eaDir/$f
  23. fi
  24. if [[ -z `ls -A "@eaDir/$f"` ]]
  25. then
  26. echo "Making Thumbs for $1 - $f"
  27. /mnt/convertn $f
  28. fi
  29. done
  30. popd

Clear thumbnail directory and execute & test the new script

  1. rm -rf /mnt/photo/YOURTESTDIR/@eaDir
  2. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Running in parallel

As a final step, let's make this run in parallel. the --jobs +8 stands for allowing the script to open up your amount of cores + 8 more jobs in parallel. In my case, this is 16. You should play a bit with this setting.

  1. #!/bin/bash
  2. pushd "$1"
  3. shopt -s nocaseglob
  4. options_sharp="-unsharp 0.5x0.5+1.25+0.0"
  5. options=""
  6. parallel_command="sem --jobs +8 --quote "
  7.  
  8. if [[ ! -d @eaDir ]]
  9. then
  10. mkdir @eaDir
  11. echo "Thumbnail dir created"
  12. fi
  13.  
  14. for f in *.jpg ; do
  15. if [[ "$f" == "*.jpg" ]]
  16. then
  17. break
  18. fi
  19.  
  20. if [[ ! -d @eaDir/$f ]]
  21. then
  22. echo "$1 - $f : Making thumb dir"
  23. mkdir @eaDir/$f
  24. fi
  25. if [[ -z `ls -A "@eaDir/$f"` ]]
  26. then
  27. echo "Making Thumbs for $1 - $f"
  28. $parallel_command /mnt/convertn $f
  29. fi
  30. done
  31. popd

Clear thumbnail directory and execute & test the new script

Enjoy speedy thumbnail generation ;-)

  1. rm -rf /mnt/photo/YOURTESTDIR/@eaDir
  2. find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Sources