1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[seriesfiler] Series Auto-filer

Discussion in 'HD/HDR-FOX T2 Customised Firmware' started by Sam Widges, Jun 26, 2011.

  1. Sam Widges

    Sam Widges Active Member

    Now I've got telnet access to my Hummy, I want to write an auto-filer script that will allow us to file series in our subdirectories for 'His', 'Hers' and 'Both of Us' and for new episodes to be automatically moved to those directories as they are recorded. This is a feature that would really clear up the top level of the Media directory.

    I'm not far off having a proof-of-concept, except I'm not sure how and when thumbnails are created. There doesn't seem to be any particular time that they appear, so I'm wondering if I move a fileset to another directory, will the thumbnailer process still do its job and create a thumbnail in the new directory? I'm running some test recordings at the moment (thank goodness for all the Top Gear repeats on Dave) but it would be interesting to hear what other people's experiences are.

    EDIT: For people wanting to install the basic script see http://www.hummy.tv/forum/threads/customised-hdf-file-released.232/page-5#post-2712.

    EDIT: There is an advanced version of the script which incorporates renaming of the files during the transfer as well, see http://www.hummy.tv/forum/threads/series-auto-filer.278/page-2#post-4900
     
  2. Sam Widges

    Sam Widges Active Member

    Well, it seems to be working well enough now, so if anyone is interested, here's the script

    Code:
    #!/bin/sh
    
    #Needed to stop the list using spaces as delimiters
    IFS=$'\n'
    
    #Find all the top level directories in "My Video" that contain
    #a .series file and iterate through them
    for dir in `find /media/My\ Video/ -maxdepth 2 -mindepth 2 -name '.series' | sed 's/\/.series$//g'`
    
    do
            #Strip out the series name from the directory name and look for
            #other directories with the same name
            series=`basename "$dir"`
            echo "Checking $dir"
            location=`find /media/My\ Video/ -mindepth 2 -type d -name "$series"`
    
            #$location is only set if another directory has been found with the same name
            if [ -n "$location" ]; then
                    echo "Filing for $location"
                    #Use the hmt files to identify the episode base filename (strip off the suffix using sed)
                    #If the file is in use ignore it (mmin +5 means ignore anything that has changed in the last 5 minutes)
                    for episodefile in `find "$dir/" -name '*.hmt' -mmin +5 | sed 's/.hmt$//'`
                    do
                            #Find and move all the files for this episode
                            episode=`basename "$episodefile"`
                            echo "Episode $episode"
                            find "$dir" -name "$episode.*" -exec mv '{}' $location \;
                    done
                    if [ "$(ls $dir)" ]; then
                            #There is probably another episode being recorded
                            #we don't want to interfere, so do nothing
                            echo "Series directory not empty"
                    else
                            #.series is a hidden file, and the only hidden file that should exist
                            #if there are no other files in the directory, move .series as well
                            #and then remove the series directory at the top level - the filing is now done
                            mv $dir/.series $location
                            echo "Removing empty series directory"
                            rmdir "$dir"
                    fi
    
            fi
    done
    
    I saved it as /mod/bin/seriesfiler and made it executable (chmod 755 /mod/bin/seriesfiler).

    To use it, all you have to do is move a series directory to a new location, e.g. move the "Doctor Who" folder into another folder, say called "Family Viewing". After the next episode in the series has been broadcast, a new series directory will be created in the usual place, as usual. Just run the script and it will automatically copy the new programme's files into the destination directory and remove the now empty top level directory. There's no harm running the script during a recording because it won't touch any files until 5 minutes after the recording has finished.

    I've checked and there doesn't seem to be a problem with moving files while watching a programme, but if you stop watching it won't remember your place. However, once the files have been moved, everything should work again exactly as normal.

    I'll probably set this to run every 10 or 15 minutes, using cron.

    By the way, shell scripting isn't my thing, so feel free to suggest any improvements!

    Edit - I've just found a bug, if you move a programme while it is being watched, the Humax actually creates a new hmt file in the old place, even recreating the directory. It's probably best to not run this script from cron until that problem is sorted and it won't be simple.
     
  3. af123

    af123 Administrator Staff Member

    Nifty.. I can see myself having a use for this!

    You could watch the .ts file to make sure it isn't growing prior to moving, or check that the file isn't currently open by the humaxtv process. That last one will also stop things being moved while they are being watched or otherwise used by the box.

    Just a couple for you:

    This will only work as long as the output of the find command is less than the shell's defined line length. A better way to do it, which will allow for longer lists of files is:

    Code:
    find "/media/My Video/" -maxdepth 2 -mindepth 2 -name .series | while read line; do
        dir="`dirname $line`"
    ...
    done
    
    This way doesn't require changing IFS either.

    or just

    Code:
    mv "$dir/$episode.*" "$location"
     
  4. Sam Widges

    Sam Widges Active Member

    Glad you like it, I was beginning to wonder if anyone was listening :)

    I had assumed that lsof wasn't installed but I've realised that it's there by default, so I now have a way of detecting that a file is being watched, which is excellent.

    That's better. I prefer pipes in situations like this.

    I did try that initially but the versions that I tried barfed on spaces and other characters, so I resorted to the find command.

    Right, time for some tidying up...
     
  5. Black Hole

    Black Hole Usain the Whippet

    Oh yes, we're listening.... doesn't necessarily mean we understand! Carry on the good work!
     
  6. af123

    af123 Administrator Staff Member

    It's required by the webif package so if you've installed that then lsof will have been installed too.

    I got the quotes in the wrong place in the example:

    Code:
    mv "$dir/$episode".* "$location"
    
    Handling of file paths containing whitespace is one of the main pains with shell. It's refreshing to be writing the web interface in tcl as it just handles them :)
     
  7. Sam Widges

    Sam Widges Active Member

    Is there something about this implentation of lsof that is different? Here is the output from a couple of different commands

    Code:
    humax# lsof | grep "Top Gear_20110627_0900.nts"
    humax#
    
    Which shows that the nts file for that episode of Top Gear isn't in use (correct)

    However, looking for the file being open, specifically,
    Code:
    humax# lsof "/media/My Video/Top Gear/Top Gear_20110627_0900.nts"
    COMMAND  PID USER  FD  TYPE DEVICE      SIZE      NODE NAME
    humaxtv  124 root  17  0000    8,2 1309283750 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    humaxtv  124 root  18  0000    8,2 1309283755 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    humaxtv  124 root  21  0000    8,2 1304283597 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    humaxtv  124 root  23  0000    8,2 1304283597 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    mediatomb 423 root    5  0000    8,2 1309154305 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    mediatomb 423 root  10  0000    8,2 1309281263 4295000484 /mnt/hd2/My Video/Top Gear/Top Gear_20110627_0900.nts
    ...
    
    Which erroneously seems to show that the humaxtv and mediatomb processes have this file open.

    I'm trying to check whether the humaxtv process has a file open and the behaviour of the second command seems a bit strange but I suspect that this may be down to some aspect of the humaxtv and mediatomb processes. Obviously, I can resort to grep, but it would be good to be able to check a return value and do the job properly for once :cool:
     
  8. af123

    af123 Administrator Staff Member

    Same behaviour here.. odd!

    If it helps, I usually use something like the following to see what is being watched/recorded:

    Code:
    humax# lsof -p `pgrep humaxtv`  -a /mnt/hd2
    COMMAND PID USER   FD   TYPE DEVICE       SIZE       NODE NAME
    humaxtv 124 root   31   0000    8,2 1309289373 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.ts
    humaxtv 124 root   32   0000    8,2 1309289374 4295000484 /mnt/hd2/My Video/CSI_ Crime Scene Investigation_20110628_2029.nts
    
     
  9. Sam Widges

    Sam Widges Active Member

    Something I noticed this morning, the HDR-T2 can shut down pretty suddenly following a recording. I don't think the seriesfiler will be affected because any files being moved will either fail or I'll find out very quickly that there's a problem, no doubt due to physical feedback from the other half. However, any file moves that haven't been completed at shutdown should simply complete the next time the script is run. Is there a way to temporarily stop the box shutting down? I imagine that a lot of scripts/services would like this option.

    Of course, once we have found a way of stopping the box from shutting down, we would then have to find out how to make all the scripts play nicely together, which is probably even more difficult.
     
  10. Sam Widges

    Sam Widges Active Member

    OK, so it's not just me then, that's a relief!

    Lsof and grep seem to work fairly well to detect programmes that are in use and I prefer that method to trying to track files that are growing, which is more complicated to program around. I'll upload another version of the seriesfiler when I've used it for a while and I'm happy that it's likely to work fairly well.
     
  11. neilleeds

    neilleeds Member

    I like the idea of this, could it be integrated into the opkg system to make it easy to install and use when it is complete?

    This has given me and idea which I might investigate. Copying new recordings from the default folder to one with a custom name.
    E.G.

    The F1 has the name Canadian Grand Prix (for example). When you set the series link it always uses this name for any recordings. I tried to rename the folder to Formula 1 however the Humax just recreates the folder with the original name. I am thinking that it should be possible to create a script which checks recordings and moves them to a folder with other episodes of the same series in. Any thoughts on this?
     
  12. Sam Widges

    Sam Widges Active Member

    OK, the revised script seems to be working as intended now. I've removed the check for the .hmt file having been accessed within the last 5 minutes and now simply check for the humaxtv process having the file open.

    There are plenty of comments if people want to work out is going on at each point.

    Code:
    #!/bin/sh
    
    #Set these here otherwise path problems break the script when run from cron
    FIND=/mod/bin/busybox/find
    GREP=/mod/bin/busybox/grep
    SED=/mod/bin/busybox/sed
    LSOF=/mod/bin/lsof
    BASENAME=/mod/bin/busybox/basename
    RMDIR=/mod/bin/busybox/rmdir
    MV=/mod/bin/busybox/mv
    
    #Needed to stop the list using spaces as delimiters
    IFS=$'\n'
    #Despite the pipe following the next find, the IFS still needs to be set
    
    #Find all the top level directories in "My Video" that contain
    #a .series file and iterate through them
    $FIND /media/My\ Video/ -maxdepth 2 -mindepth 2 -name '.series' | $SED 's/\/.series$//g' | while read dir; do
    
            #Strip out the series name from the directory name and look for
            #other directories with the same name
            series=`$BASENAME "$dir"`
            echo "Checking $dir"
            location=`$FIND /media/My\ Video/ -mindepth 2 -type d -name "$series"`
    
            #$location is only set if another directory has been found with the same name
            if [ -n "$location" ]; then
                    echo "Filing for $location"
                    #Use the hmt files to identify the episode base filename and strip off the suffix using sed
                    for episodefile in `$FIND "$dir/" -name '*.hmt' | $SED 's/.hmt$//'`
                    do
                            #Find and move all the files for this episode, if they are not in use
                            episode=`$BASENAME "$episodefile"`
                            inuse=`$LSOF | $GREP "$episode".ts`
                            if [ -n "$inuse" ]; then
                                    echo "$episodefile is in use - skipping"
                            else
                                    echo "Moving $episode to $location"
                                    $MV "$dir/$episode".* "$location"
                            fi
                    done
                    if [ "$(ls $dir)" ]; then
                            #There is probably another episode being recorded
                            #we don't want to interfere, so do nothing
                            echo "Series directory not empty"
                    else
                            #.series is a hidden file, and the only hidden file that should exist
                            #if there are no other files in the directory, move .series as well
                            #and then remove the series directory at the top level - the filing is now done
                            #.series has to be moved last, in case the box powers down mid script
                            #in which case, re-running it should still cleanly finish the process
                            $MV $dir/.series $location
                            echo "Removing empty series directory"
                            $RMDIR "$dir"
                    fi
    
            fi
    done
    
    #And show all the series that have been processed in the last day
    echo "Recently filed:"
    $FIND /media/My\ Video/ -mindepth 3 -mtime -1 -name '.series' | $SED 's/\/.series$//g'
    
    Save the script in /mod/bin/seriesfiler and use 'chmod 755' to make it executable.

    You need to install the lsof package to use the script, if it hasn't already been installed.

    I have added the cron package and have set it to run using the following crontab entry in /mod/var/spool/cron/crontabs/root

    Code:
    0,15,30,45 * * * * /mod/bin/seriesfiler 2>&1 > /mod/tmp/seriesfiler.log
    
    Once that is done, it's a matter of running
    Code:
     /mod/etc/init.d/S01crond restart
    
    but after the initial configuration it should then start automatically.

    One other thing that came up follows on from the analysis of the .series file posted on here last night, where it is now clear that there are details of the number of viewed files etc. in the .series file - copying this file into the destination directory is obviously broken, but the only side-effect is whether the 'new file' behaviour would be correct. I will see what I can do to fix it if it matters to enough people.

    And finally, there is no reason that this can't be packaged up the same way as dedup is, although it might be better to package multiple scripts such as this into one as they don't exactly take up a lot of space.

    Use of this script is 'At Your Own Risk' - don't blame me if it eats your video collection, but it has passed the first hurdle which is running for a day on my own box without any domestic violence, so it's unlikely to do any major or irreversible damage.

    Edit: I had problems running the script from cron but I think it was path related. I've updated the script again to code in paths for find, basename, grep etc.
     
    Drutt likes this.
  13. Sam Widges

    Sam Widges Active Member

    There were a few more problems with the path (rmdir wasn't working but I hadn't spotted it) so I have now coded in paths for rmdir and mv as well in case it spontaneously broke in the future as well. I've updated the script in the post above.
     
  14. neilleeds

    neilleeds Member

    I'm having trouble running the script. I just get

    sh: ./seriesfiler: not found
    when i try and run it using ./seriesfiler or

    /bin/sh: /mod/bin/seriesfiler: not found
    when run with cron.
    Any ideas? I have lsof installed and i have done a chmod 755.
    Thanks
     
  15. af123

    af123 Administrator Staff Member

    You probably have DOS/Windows newlines in the file.
    Try running it through dos2unix.

    Code:
    humax# dos2unix < seriesfiler > seriesfiler2
    humax# chmod 755 seriesfiler2
    humax# ./seriesfiler2
    
     
  16. neilleeds

    neilleeds Member

    Unbelievably fast response!

    Didn't work first time I tried the command but tried resaving it again from Linux specifying LF and then ran it through the dos2unix again and it is now working!

    Thanks
     
  17. af123

    af123 Administrator Staff Member

    No problem - it's a very misleading error message but it is because the shell reads the first line of the script to find out how to execute it and it takes everything after the shebang (!#) as the interpreter name, including the newline character so it ends up trying to execute '/bin/sh^M' which doesn't exist.
     
  18. neilleeds

    neilleeds Member

    Great script, works well. I'm going to look at extending it to allow for renamed folders using the details in the series file. If I come up with anything useful I'll post it up of course.
     
  19. af123

    af123 Administrator Staff Member

    Once it's stable enough, I'll roll it into a package on hummypkg if you want. I'm going to release a package that makes it easy to maintain packages too so that it is easier for people to submit packages to the repository.
     
  20. neilleeds

    neilleeds Member

    Sounds great. Maybe you could flag new packages/scripts as beta so that people know which items are not as well tested. The could then be promoted if feedback is good? Thanks