[seriesfiler] Series Auto-filer

Sam Widges

Active Member
Now I've got telnet access to my Hummy, I want to write an auto-filer script that will allow us to file series in our subdirectories for 'His', 'Hers' and 'Both of Us' and for new episodes to be automatically moved to those directories as they are recorded. This is a feature that would really clear up the top level of the Media directory.

I'm not far off having a proof-of-concept, except I'm not sure how and when thumbnails are created. There doesn't seem to be any particular time that they appear, so I'm wondering if I move a fileset to another directory, will the thumbnailer process still do its job and create a thumbnail in the new directory? I'm running some test recordings at the moment (thank goodness for all the Top Gear repeats on Dave) but it would be interesting to hear what other people's experiences are.

EDIT: For people wanting to install the basic script see http://www.hummy.tv/forum/threads/customised-hdf-file-released.232/page-5#post-2712.

EDIT: There is an advanced version of the script which incorporates renaming of the files during the transfer as well, see http://www.hummy.tv/forum/threads/series-auto-filer.278/page-2#post-4900
 
Well, it seems to be working well enough now, so if anyone is interested, here's the script

Code:
#!/bin/sh

#Needed to stop the list using spaces as delimiters
IFS=$'\n'

#Find all the top level directories in "My Video" that contain
#a .series file and iterate through them
for dir in `find /media/My\ Video/ -maxdepth 2 -mindepth 2 -name '.series' | sed 's/\/.series$//g'`

do
        #Strip out the series name from the directory name and look for
        #other directories with the same name
        series=`basename "$dir"`
        echo "Checking $dir"
        location=`find /media/My\ Video/ -mindepth 2 -type d -name "$series"`

        #$location is only set if another directory has been found with the same name
        if [ -n "$location" ]; then
                echo "Filing for $location"
                #Use the hmt files to identify the episode base filename (strip off the suffix using sed)
                #If the file is in use ignore it (mmin +5 means ignore anything that has changed in the last 5 minutes)
                for episodefile in `find "$dir/" -name '*.hmt' -mmin +5 | sed 's/.hmt$//'`
                do
                        #Find and move all the files for this episode
                        episode=`basename "$episodefile"`
                        echo "Episode $episode"
                        find "$dir" -name "$episode.*" -exec mv '{}' $location \;
                done
                if [ "$(ls $dir)" ]; then
                        #There is probably another episode being recorded
                        #we don't want to interfere, so do nothing
                        echo "Series directory not empty"
                else
                        #.series is a hidden file, and the only hidden file that should exist
                        #if there are no other files in the directory, move .series as well
                        #and then remove the series directory at the top level - the filing is now done
                        mv $dir/.series $location
                        echo "Removing empty series directory"
                        rmdir "$dir"
                fi

        fi
done

I saved it as /mod/bin/seriesfiler and made it executable (chmod 755 /mod/bin/seriesfiler).

To use it, all you have to do is move a series directory to a new location, e.g. move the "Doctor Who" folder into another folder, say called "Family Viewing". After the next episode in the series has been broadcast, a new series directory will be created in the usual place, as usual. Just run the script and it will automatically copy the new programme's files into the destination directory and remove the now empty top level directory. There's no harm running the script during a recording because it won't touch any files until 5 minutes after the recording has finished.

I've checked and there doesn't seem to be a problem with moving files while watching a programme, but if you stop watching it won't remember your place. However, once the files have been moved, everything should work again exactly as normal.

I'll probably set this to run every 10 or 15 minutes, using cron.

By the way, shell scripting isn't my thing, so feel free to suggest any improvements!

Edit - I've just found a bug, if you move a programme while it is being watched, the Humax actually creates a new hmt file in the old place, even recreating the directory. It's probably best to not run this script from cron until that problem is sorted and it won't be simple.
 
Nifty.. I can see myself having a use for this!

Edit - I've just found a bug, if you move a programme while it is being watched, the Humax actually creates a new hmt file in the old place, even recreating the directory. It's probably best to not run this script from cron until that problem is sorted and it won't be simple.

You could watch the .ts file to make sure it isn't growing prior to moving, or check that the file isn't currently open by the humaxtv process. That last one will also stop things being moved while they are being watched or otherwise used by the box.

By the way, shell scripting isn't my thing, so feel free to suggest any improvements!

Just a couple for you:

for dir in `find /media/My\ Video/ -maxdepth 2 -mindepth 2 -name '.series' | sed 's/\/.series$//g'`

This will only work as long as the output of the find command is less than the shell's defined line length. A better way to do it, which will allow for longer lists of files is:

Code:
find "/media/My Video/" -maxdepth 2 -mindepth 2 -name .series | while read line; do
    dir="`dirname $line`"
...
done

This way doesn't require changing IFS either.

find "$dir" -name "$episode.*" -exec mv '{}' $location \;

or just

Code:
mv "$dir/$episode.*" "$location"
 
Nifty.. I can see myself having a use for this!

You could watch the .ts file to make sure it isn't growing prior to moving, or check that the file isn't currently open by the humaxtv process. That last one will also stop things being moved while they are being watched or otherwise used by the box.

Glad you like it, I was beginning to wonder if anyone was listening :)

I had assumed that lsof wasn't installed but I've realised that it's there by default, so I now have a way of detecting that a file is being watched, which is excellent.

Just a couple for you:



This will only work as long as the output of the find command is less than the shell's defined line length. A better way to do it, which will allow for longer lists of files is:

Code:
find "/media/My Video/" -maxdepth 2 -mindepth 2 -name .series | while read line; do
    dir="`dirname $line`"
...
done

This way doesn't require changing IFS either.

That's better. I prefer pipes in situations like this.

or just

Code:
mv "$dir/$episode.*" "$location"

I did try that initially but the versions that I tried barfed on spaces and other characters, so I resorted to the find command.

Right, time for some tidying up...
 
I had assumed that lsof wasn't installed but I've realised that it's there by default...

It's required by the webif package so if you've installed that then lsof will have been installed too.

I did try that initially but the versions that I tried barfed on spaces and other characters, so I resorted to the find command.

I got the quotes in the wrong place in the example:

Code:
mv "$dir/$episode".* "$location"

Handling of file paths containing whitespace is one of the main pains with shell. It's refreshing to be writing the web interface in tcl as it just handles them :)
 
It's required by the webif package so if you've installed that then lsof will have been installed too.

Is there something about this implentation of lsof that is different? Here is the output from a couple of different commands

Code:
humax# lsof | grep "Top Gear_20110627_0900.nts"
humax#
Which shows that the nts file for that episode of Top Gear isn't in use (correct)

However, looking for the file being open, specifically,
Code:
humax# lsof "/media/My Video/Top Gear/Top Gear_20110627_0900.nts"
COMMAND  PID USER  FD  TYPE DEVICE      SIZE      NODE NAME
humaxtv  124 root  17  0000    8,2 1309283750 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
humaxtv  124 root  18  0000    8,2 1309283755 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
humaxtv  124 root  21  0000    8,2 1304283597 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
humaxtv  124 root  23  0000    8,2 1304283597 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
mediatomb 423 root    5  0000    8,2 1309154305 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
mediatomb 423 root  10  0000    8,2 1309281263 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
...
Which erroneously seems to show that the humaxtv and mediatomb processes have this file open.

I'm trying to check whether the humaxtv process has a file open and the behaviour of the second command seems a bit strange but I suspect that this may be down to some aspect of the humaxtv and mediatomb processes. Obviously, I can resort to grep, but it would be good to be able to check a return value and do the job properly for once :cool:
 
Same behaviour here.. odd!

If it helps, I usually use something like the following to see what is being watched/recorded:

Code:
humax# lsof -p `pgrep humaxtv`  -a /mnt/hd2
COMMAND PID USER   FD   TYPE DEVICE       SIZE       NODE NAME
humaxtv 124 root   31   0000    8,2 1309289373 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.ts
humaxtv 124 root   32   0000    8,2 1309289374 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.nts
 
Something I noticed this morning, the HDR-T2 can shut down pretty suddenly following a recording. I don't think the seriesfiler will be affected because any files being moved will either fail or I'll find out very quickly that there's a problem, no doubt due to physical feedback from the other half. However, any file moves that haven't been completed at shutdown should simply complete the next time the script is run. Is there a way to temporarily stop the box shutting down? I imagine that a lot of scripts/services would like this option.

Of course, once we have found a way of stopping the box from shutting down, we would then have to find out how to make all the scripts play nicely together, which is probably even more difficult.
 
Same behaviour here.. odd!

If it helps, I usually use something like the following to see what is being watched/recorded:

Code:
humax# lsof -p `pgrep humaxtv`  -a /mnt/hd2
COMMAND PID USER  FD  TYPE DEVICE      SIZE      NODE NAME
humaxtv 124 root  31  0000    8,2 1309289373 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.ts
humaxtv 124 root  32  0000    8,2 1309289374 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.nts

OK, so it's not just me then, that's a relief!

Lsof and grep seem to work fairly well to detect programmes that are in use and I prefer that method to trying to track files that are growing, which is more complicated to program around. I'll upload another version of the seriesfiler when I've used it for a while and I'm happy that it's likely to work fairly well.
 
I like the idea of this, could it be integrated into the opkg system to make it easy to install and use when it is complete?

This has given me and idea which I might investigate. Copying new recordings from the default folder to one with a custom name.
E.G.

The F1 has the name Canadian Grand Prix (for example). When you set the series link it always uses this name for any recordings. I tried to rename the folder to Formula 1 however the Humax just recreates the folder with the original name. I am thinking that it should be possible to create a script which checks recordings and moves them to a folder with other episodes of the same series in. Any thoughts on this?
 
OK, the revised script seems to be working as intended now. I've removed the check for the .hmt file having been accessed within the last 5 minutes and now simply check for the humaxtv process having the file open.

There are plenty of comments if people want to work out is going on at each point.

Code:
#!/bin/sh

#Set these here otherwise path problems break the script when run from cron
FIND=/mod/bin/busybox/find
GREP=/mod/bin/busybox/grep
SED=/mod/bin/busybox/sed
LSOF=/mod/bin/lsof
BASENAME=/mod/bin/busybox/basename
RMDIR=/mod/bin/busybox/rmdir
MV=/mod/bin/busybox/mv

#Needed to stop the list using spaces as delimiters
IFS=$'\n'
#Despite the pipe following the next find, the IFS still needs to be set

#Find all the top level directories in "My Video" that contain
#a .series file and iterate through them
$FIND /media/My\ Video/ -maxdepth 2 -mindepth 2 -name '.series' | $SED 's/\/.series$//g' | while read dir; do

        #Strip out the series name from the directory name and look for
        #other directories with the same name
        series=`$BASENAME "$dir"`
        echo "Checking $dir"
        location=`$FIND /media/My\ Video/ -mindepth 2 -type d -name "$series"`

        #$location is only set if another directory has been found with the same name
        if [ -n "$location" ]; then
                echo "Filing for $location"
                #Use the hmt files to identify the episode base filename and strip off the suffix using sed
                for episodefile in `$FIND "$dir/" -name '*.hmt' | $SED 's/.hmt$//'`
                do
                        #Find and move all the files for this episode, if they are not in use
                        episode=`$BASENAME "$episodefile"`
                        inuse=`$LSOF | $GREP "$episode".ts`
                        if [ -n "$inuse" ]; then
                                echo "$episodefile is in use - skipping"
                        else
                                echo "Moving $episode to $location"
                                $MV "$dir/$episode".* "$location"
                        fi
                done
                if [ "$(ls $dir)" ]; then
                        #There is probably another episode being recorded
                        #we don't want to interfere, so do nothing
                        echo "Series directory not empty"
                else
                        #.series is a hidden file, and the only hidden file that should exist
                        #if there are no other files in the directory, move .series as well
                        #and then remove the series directory at the top level - the filing is now done
                        #.series has to be moved last, in case the box powers down mid script
                        #in which case, re-running it should still cleanly finish the process
                        $MV $dir/.series $location
                        echo "Removing empty series directory"
                        $RMDIR "$dir"
                fi

        fi
done

#And show all the series that have been processed in the last day
echo "Recently filed:"
$FIND /media/My\ Video/ -mindepth 3 -mtime -1 -name '.series' | $SED 's/\/.series$//g'

Save the script in /mod/bin/seriesfiler and use 'chmod 755' to make it executable.

You need to install the lsof package to use the script, if it hasn't already been installed.

I have added the cron package and have set it to run using the following crontab entry in /mod/var/spool/cron/crontabs/root

Code:
0,15,30,45 * * * * /mod/bin/seriesfiler 2>&1 > /mod/tmp/seriesfiler.log

Once that is done, it's a matter of running
Code:
 /mod/etc/init.d/S01crond restart
but after the initial configuration it should then start automatically.

One other thing that came up follows on from the analysis of the .series file posted on here last night, where it is now clear that there are details of the number of viewed files etc. in the .series file - copying this file into the destination directory is obviously broken, but the only side-effect is whether the 'new file' behaviour would be correct. I will see what I can do to fix it if it matters to enough people.

And finally, there is no reason that this can't be packaged up the same way as dedup is, although it might be better to package multiple scripts such as this into one as they don't exactly take up a lot of space.

Use of this script is 'At Your Own Risk' - don't blame me if it eats your video collection, but it has passed the first hurdle which is running for a day on my own box without any domestic violence, so it's unlikely to do any major or irreversible damage.

Edit: I had problems running the script from cron but I think it was path related. I've updated the script again to code in paths for find, basename, grep etc.
 
There were a few more problems with the path (rmdir wasn't working but I hadn't spotted it) so I have now coded in paths for rmdir and mv as well in case it spontaneously broke in the future as well. I've updated the script in the post above.
 
I'm having trouble running the script. I just get

sh: ./seriesfiler: not found
when i try and run it using ./seriesfiler or

/bin/sh: /mod/bin/seriesfiler: not found
when run with cron.
Any ideas? I have lsof installed and i have done a chmod 755.
Thanks
 
I'm having trouble running the script. I just get

sh: ./seriesfiler: not found
when i try and run it using ./seriesfiler or cron.
Any ideas? I have lsof installed and i have done a chmod 755.
Thanks

You probably have DOS/Windows newlines in the file.
Try running it through dos2unix.

Code:
humax# dos2unix < seriesfiler > seriesfiler2
humax# chmod 755 seriesfiler2
humax# ./seriesfiler2
 
You probably have DOS/Windows newlines in the file.
Try running it through dos2unix.

Code:
humax# dos2unix < seriesfiler > seriesfiler2
humax# chmod 755 seriesfiler2
humax# ./seriesfiler2

Unbelievably fast response!

Didn't work first time I tried the command but tried resaving it again from Linux specifying LF and then ran it through the dos2unix again and it is now working!

Thanks
 
Unbelievably fast response!

Didn't work first time I tried the command but tried resaving it again from Linux specifying LF and then ran it through the dos2unix again and it is now working!

Thanks

No problem - it's a very misleading error message but it is because the shell reads the first line of the script to find out how to execute it and it takes everything after the shebang (!#) as the interpreter name, including the newline character so it ends up trying to execute '/bin/sh^M' which doesn't exist.
 
Great script, works well. I'm going to look at extending it to allow for renamed folders using the details in the series file. If I come up with anything useful I'll post it up of course.
 
Once it's stable enough, I'll roll it into a package on hummypkg if you want. I'm going to release a package that makes it easy to maintain packages too so that it is easier for people to submit packages to the repository.
 
Once it's stable enough, I'll roll it into a package on hummypkg if you want. I'm going to release a package that makes it easy to maintain packages too so that it is easier for people to submit packages to the repository.
Sounds great. Maybe you could flag new packages/scripts as beta so that people know which items are not as well tested. The could then be promoted if feedback is good? Thanks
 
Back
Top