I recently bought a Synology 212J to store all my pictures on, and of course also to play with. I always had some crappy tiny non-modifyable NAS/Media Center solution that I really hated. cough Plextor cough
So, I started adding pictures to the NAS and I wanted to play with the photo station application that is included. However, I noticed that as soon as I added pictures to the drive, it completely stalled. CPU utilisation went up to 100% 24/7. This also meant that it never went into idle mode and consumed more energy.
Online I found a number of tutorials to speed up thumbnail generation, some useful, others complete crap. I added the ones I found useful in the source list below.
What are we going to do?
What follows is my experience with Synology DS212J DSM4.1 & OSX Mountain Lion with Brew installed. We will go with the method of speeding up your thumbnail generation by doing all the processing on your local computer. In my case, that is a macbook pro i7. I’ll show you three possible ways.
- Using the local convert binary on your mac to convert the image from the NAS to 4 smaller images. This will require 4 reads and 4 writes over your network and is a little slower
- Using a custom compiled c++ program to fetch the image once, and use this image in the memory to write the 4 thumbnails back to the NAS. This will require 1 read and 4 writes.
- Using this custom compiled c++ program and running this in parallel. Speedup++!
Speed?
Using this script I am able to convert more or less 1 image a second. This means it generates 4 new images a second and writes them to the NAS. Compared to the build-in thumbnail generation, this is lightning fast. I was already generating thumbnails on the nas for over 2 weeks, and it had generated less than 10000 photos. Now, in less than an hour, I am able to process more than 4000 pictures +-. I guess it is clear that this new method is WAY faster. Additional benefit is that my NAS nor my macbook utilise their CPU for the full 100%. Looks like I can optimise it even more. When I finish generating thumbnails for my 94.000 photos I will post the results here. Stay tuned!
Enable SSH access to your NAS & Mount your drive without locking
- Log in to your drive
ssh root@192.168.*.*- Type your admin password
vi /etc/exports- edit the /etc/exports file, and change all the “insecure_locks” entries to simply read “insecure”. If you don’t know vi: press the INSERT key, use arrows to go to the line to edit, and delete/edit the line. Then press the ESC key, and type :x to ‘Save & Exit’.
- More detailed explanation can be found on the web
Prepare your mac
if you have linux, it should be similar. Yum or apt-get will get you on your way.
brew install imagemagick
Mount your drive with your pictures to your Machine
mkdir /mnt && mkdir /mnt/photo
sudo mount DS_IP:/volume1/photo /mnt/photo/
Create the most basic script
Create the following script and save it in /mnt/mkthumb_seq.sh
#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""
if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi
for f in *.jpg ; do
if [[ "$f" == "*.jpg" ]]
then
break
fi
if [[ ! -d @eaDir/$f ]]
then
echo "$1 - $f : Making thumb dir"
mkdir @eaDir/$f
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ]]
then
convert $f -resize 1280x1280\> -quality 90 $options_sharp @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg
echo "converted XL for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ]]
then
convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_L.jpg
echo "converted L for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ]]
then
convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_M.jpg
echo "converted M for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ]]
then
convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 120x120\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_S.jpg
echo "converted S for $f"
fi
if [[ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ]]
then
convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 $options @eaDir/$f/SYNOPHOTO:THUMB_B.jpg
echo "converted B for $f"
fi
done
popd
Save this file and run it against a directory This should already succeed, but as I told you, this is not optimal yet due to the 4 network reads.
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_seq.sh {} \;
The C++ moment
Creating and compiling our C++ code. The following piece of code I found on a gist in github. While I did adjust it, it is in essence still the same. It basically does the following
- takes in a path to a picture
- loads the picture
- creates the 4 thumbnails
- creates the directory where the file should be stored
- writes them back to the proper directory structure where the Synology NAS expects them
Save the following code as /mnt/convertn.cpp
<cpp>
#include <iostream>
#include <string>
#include <libgen.h>
#include <Magick++.h>
#include <sys/stat.h>
using namespace std;
using namespace Magick;
string thumbDir( ) {
return "@eaDir";
}
string outdir( string fn ) {
return thumbDir() + "/" + fn;
}
string out( string fn, string sz ) {
return outdir( fn ) + "/SYNOPHOTO:THUMB_" + sz + ".jpg";
}
int main( int, char **argv) {
InitializeMagick(*argv);
Image im;
string src = argv[1];
string fn = string( basename( strdup( src.c_str() ) ) );
string xl_path = out( fn, "XL" );
string path = outdir( fn );
string thumddir = thumbDir( );
struct convertn {
string geometry;
string size;
};
convertn sizes [4] = {
{ "800x800>", "L" },
{ "640x640>", "B" },
{ "320x320>", "M" },
{ "120x120>", "S" }
};
// Create the directory
mkdir ( thumddir.c_str(), 0755);
mkdir ( path.c_str(), 0755);
try {
im.read( src );
im.scale("1280x1280>");
im.unsharpmask(0.5, 0.5, 1.25, 0.0);
im.write( xl_path );
} catch (const std::exception &e) {
cerr << e.what() << endl;
}
for ( int j=0; j<4; j++) {
//cout << j << endl;
try {
im.scale( sizes[j].geometry );
//cout << out( fn, sizes[j].size ) << endl;
im.write( out( fn, sizes[j].size ) );
} catch (const std::exception &e) {
cerr << e.what() << endl;
}
}
}
</cpp>
Compiling and using our C++ program
cd /mnt
g++ `Magick++-config --cxxflags --cppflags --ldflags --libs` convertn.cpp -o convertn
chmod +x convertn
Try out your C++ program! It should create @eaDir/imagefilename/SYNOPHOTO:THUMB_{XL/L/M/S].jpg files.
./convertn path_to_test_image.jpg
Using the C++ in the bash script
Let’s use this program now in our bash script. Save the following as /mnt/mkthumb_cplusplus.sh
#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""
if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi
for f in *.jpg ; do
if [[ "$f" == "*.jpg" ]]
then
break
fi
if [[ ! -d @eaDir/$f ]]
then
echo "$1 - $f : Making thumb dir"
mkdir @eaDir/$f
fi
if [[ -z `ls -A "@eaDir/$f"` ]]
then
echo "Making Thumbs for $1 - $f"
/mnt/convertn $f
fi
done
popd
Clear thumbnail directory and execute & test the new script
rm -rf /mnt/photo/YOURTESTDIR/@eaDir
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;
Running in parallel
As a final step, let’s make this run in parallel. the –jobs +8 stands for allowing the script to open up your amount of cores + 8 more jobs in parallel. In my case, this is 16. You should play a bit with this setting.
#!/bin/bash
pushd "$1"
shopt -s nocaseglob
options_sharp="-unsharp 0.5x0.5+1.25+0.0"
options=""
parallel_command="sem --jobs +8 --quote "
if [[ ! -d @eaDir ]]
then
mkdir @eaDir
echo "Thumbnail dir created"
fi
for f in *.jpg ; do
if [[ "$f" == "*.jpg" ]]
then
break
fi
if [[ ! -d @eaDir/$f ]]
then
echo "$1 - $f : Making thumb dir"
mkdir @eaDir/$f
fi
if [[ -z `ls -A "@eaDir/$f"` ]]
then
echo "Making Thumbs for $1 - $f"
$parallel_command /mnt/convertn $f
fi
done
popd
Clear thumbnail directory and execute & test the new script
Enjoy speedy thumbnail generation ;-)
rm -rf /mnt/photo/YOURTESTDIR/@eaDir
find /mnt/photo/YOURTESTDIR -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec /mnt/mkthumb_cplusplus.sh {} \;
