How to Use Queue with Rsync

Having more than 120K 5MB+ images that should be moved to the server is a great oportunity for some automatic bash processing. It might be good idea to use ImageMagick convert tool to make images smaller in a simple for loop. GNU Parallel can significantly increase the performance by running one job per CPU core.

parallel --verbose convert {} -quality 40% {} ::: *.jpg

The parallel modifies several images per second. Uploading these right away seems to be the next step. But how do you tell rsync to check for modified files every now and then? Another for loop mixed with sleep would work, but it just doesn’t feel right.

Luckily, there’s a inotifywait tool capable of watching changes to files and taking actions based on those changes.

inotifywait -e close_write -m --format '%f' . | \
while read file
do
    echo $file
    rsync -OWRD0Pq --ignore-existing $file data@localhost
done

By default, inotifywait stops after receiving a single event, while -m flag runs it indefinitely. -e flag defines an event to watch for, in my case that’s a close_write event. The inotifywait output can be piped to rsync that takes care of syncing local files to remote server.

The last step, as usual, is profit.

SSH GRASS Processing Status Check

I’ve been running some GRASS/PostGIS computations on a remote server that were taking hours to finish. Once in a while I checked for their state by issuing tail log_XX.log from my laptop to see if they were ready yet. It suddenly became pretty annoying to check five different logs every ten minutes.

Instead of waiting and checking the logs, I thought it would be great to automate this. And it would be awesome if checking was fun. So I wrote a simple routine that takes log number as an argument (every process logs to a separate logfile) and checks it every minute until it says done. Right after that notify-send gives me a neat popup and Queen starts playing their We are the champions thanks to mpg123.

#!/usr/bin/env bash
item=$1

while true; do
    echo "############ ${item} ############"
    x=$(ssh [email protected] "tail -n 30 path/to/my/log_${item}.log")

    if [[ $x == *"done"* ]]
        then
            notify-send -u critical "Finally ${item}"
            mpg123 -n 250 ~/Music/queen-we_are_the_champions.mp3
            exit
        else echo "Not yet"
    fi
    sleep 60
done

What seemed to be really frustrating makes me happy right now. Unless Freddie starts singing in the middle of the night.

Bash: Prepend To Filename

for f in *; do mv "$f" "prepend_$f"; done

Whenever you need to prepend anything to your files.