Synology NAS

I recently bought a Synology 212J to store all my pictures on, and of course also to play with. I always had some crappy tiny non-modifyable NAS/Media Center solution that I really hated. cough Plextor cough

So, I started adding pictures to the NAS and I wanted to play with the photo station application that is included. However, I noticed that as soon as I added pictures to the drive, it completely stalled. CPU utilisation went up to 100% 24/7. This also meant that it never went into idle mode and consumed more energy.

Online I found a number of tutorials to speed up thumbnail generation, some useful, others complete crap. I added the ones I found useful in the source list below.

What are we going to do?

What follows is my experience with Synology DS212J DSM4.1 & OSX Mountain Lion with Brew installed. We will go with the method of speeding up your thumbnail generation by doing all the processing on your local computer. In my case, that is a macbook pro i7. I’ll show you three possible ways.

  • Using the local convert binary on your mac to convert the image from the NAS to 4 smaller images. This will require 4 reads and 4 writes over your network and is a little slower
  • Using a custom compiled c++ program to fetch the image once, and use this image in the memory to write the 4 thumbnails back to the NAS. This will require 1 read and 4 writes.
  • Using this custom compiled c++ program and running this in parallel. Speedup++!

Speed?

Using this script I am able to convert more or less 1 image a second. This means it generates 4 new images a second and writes them to the NAS. Compared to the build-in thumbnail generation, this is lightning fast. I was already generating thumbnails on the nas for over 2 weeks, and it had generated less than 10000 photos. Now, in less than an hour, I am able to process more than 4000 pictures +-. I guess it is clear that this new method is WAY faster. Additional benefit is that my NAS nor my macbook utilise their CPU for the full 100%. Looks like I can optimise it even more. When I finish generating thumbnails for my 94.000 photos I will post the results here. Stay tuned!

Enable SSH access to your NAS & Mount your drive without locking

  • Log in to your drive
  • ssh root@192.168.*.*
  • Type your admin password
  • vi /etc/exports
  • edit the /etc/exports file, and change all the “insecure_locks” entries to simply read “insecure”. If you don’t know vi: press the INSERT key, use arrows to go to the line to edit, and delete/edit the line. Then press the ESC key, and type :x to ‘Save & Exit’.
  • More detailed explanation can be found on the web

Prepare your mac

if you have linux, it should be similar. Yum or apt-get will get you on your way.

brew install imagemagick

Mount your drive with your pictures to your Machine

mkdir /mnt && mkdir /mnt/photo
sudo mount DS_IP:/volume1/photo /mnt/photo/

Create the most basic script

Create the following script and save it in /mnt/mkthumb_seq.sh

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""

if [[ ! -d @eaDir ]]
  then
  mkdir @eaDir
  echo "Thumbnail dir created"
fi

for f in *.jpg ; do
  if [[ "$f" == "*.jpg" ]]
    then
    break
  fi

  if [[ ! -d @eaDir/$f ]]
    then 
    echo "$1 - $f : Making thumb dir"
    mkdir @eaDir/$f 
  fi

  if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]] 
    then
    convert $f -resize 1280x1280\> -quality 90 $options_sharp @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg
    echo "converted XL for $f"
  fi

  if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
    then
    convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_L.jpg
    echo "converted L for $f"
  fi

  if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
    then
    convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_M.jpg
    echo "converted M for $f"
  fi

  if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
    then 
    convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_S.jpg
    echo "converted S for $f"
  fi

  if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
    then
    convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_B.jpg
    echo "converted B for $f"
  fi
done
popd

Save this file and run it against a directory This should already succeed, but as I told you, this is not optimal yet due to the 4 network reads.

find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_seq.sh {} \;

The C++ moment

Creating and compiling our C++ code. The following piece of code I found on a gist in github. While I did adjust it, it is in essence still the same. It basically does the following

  • takes in a path to a picture
  • loads the picture
  • creates the 4 thumbnails
  • creates the directory where the file should be stored
  • writes them back to the proper directory structure where the Synology NAS expects them

Save the following code as /mnt/convertn.cpp

<cpp>
#include <iostream>
#include <string>
#include <libgen.h>
#include <Magick++.h>
#include <sys/stat.h>

using namespace std;
using namespace Magick;

string thumbDir( ) {
  return "@eaDir";
}

string outdir( string fn ) {
  return thumbDir() + "/" + fn;
}

string out( string fn, string sz ) {
  return outdir( fn ) + "/SYNOPHOTO:THUMB_" + sz + ".jpg";
}


int main( int, char **argv) {
  InitializeMagick(*argv);
  Image im;
  string src = argv[1];
  string fn = string( basename( strdup( src.c_str() ) ) );
  string xl_path =  out( fn, "XL" );
  string path = outdir( fn );
  string thumddir = thumbDir( );

  struct convertn {
    string geometry;
    string size;
  };

  convertn sizes [4] = {
    { "800x800>", "L" },
    { "640x640>", "B" },
    { "320x320>", "M" },
    { "120x120>", "S" }
  };

  // Create the directory
  mkdir ( thumddir.c_str(), 0755);
  mkdir ( path.c_str(), 0755);

  try {
    im.read( src );
    im.scale("1280x1280>");
    im.unsharpmask(0.5, 0.5, 1.25, 0.0);
    im.write( xl_path );
  } catch (const std::exception &e) {
    cerr << e.what() << endl;
  }
  for ( int j=0; j<4; j++) {
    //cout << j << endl; 
    try {
    im.scale( sizes[j].geometry );
    //cout << out( fn, sizes[j].size ) << endl;
    im.write( out( fn, sizes[j].size ) );
    } catch (const std::exception &e) {
      cerr << e.what() << endl;
    }
  }
}
</cpp>

Compiling and using our C++ program

cd /mnt
g++ `Magick++-config --cxxflags --cppflags --ldflags --libs` convertn.cpp -o convertn
chmod +x convertn

Try out your C++ program! It should create @eaDir/imagefilename/SYNOPHOTO:THUMB_{XL/L/M/S].jpg files.

./convertn path_to_test_image.jpg

Using the C++ in the bash script

Let’s use this program now in our bash script. Save the following as /mnt/mkthumb_cplusplus.sh

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""

if [[ ! -d @eaDir ]]
  then
  mkdir @eaDir
  echo "Thumbnail dir created"
fi

for f in *.jpg ; do
  if [[ "$f" == "*.jpg" ]]
    then
    break
  fi

  if [[ ! -d @eaDir/$f ]]
    then 
    echo "$1 - $f : Making thumb dir"
    mkdir @eaDir/$f 
  fi
  if [[ -z `ls -A "@eaDir/$f"` ]]
    then
    echo "Making Thumbs for $1 - $f"
    /mnt/convertn $f
  fi
done
popd

Clear thumbnail directory and execute & test the new script

rm -rf /mnt/photo/YOURTESTDIR/@eaDir
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Running in parallel

As a final step, let’s make this run in parallel. the –jobs +8 stands for allowing the script to open up your amount of cores + 8 more jobs in parallel. In my case, this is 16. You should play a bit with this setting.

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""
parallel_command="sem --jobs +8 --quote "

if [[ ! -d @eaDir ]]
  then
  mkdir @eaDir
  echo "Thumbnail dir created"
fi

for f in *.jpg ; do
  if [[ "$f" == "*.jpg" ]]
    then
    break
  fi

  if [[ ! -d @eaDir/$f ]]
    then 
    echo "$1 - $f : Making thumb dir"
    mkdir @eaDir/$f 
  fi
  if [[ -z `ls -A "@eaDir/$f"` ]]
    then
    echo "Making Thumbs for $1 - $f"
    $parallel_command /mnt/convertn $f
  fi
done
popd

Clear thumbnail directory and execute & test the new script

Enjoy speedy thumbnail generation ;-)

rm -rf /mnt/photo/YOURTESTDIR/@eaDir
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;

Sources