Sweeper / Network Shares Automount Issue

D

darlo

New Member
I have recently added a USB drive to my router with the aim of using it to move files from 1 HDR T2 to another. My plan was to use sweep to move the files once decrypted onto the drive and then next time the second unit is turned on have sweeper move them down to the Humax disk.

Everything worked until I tried to setup a sweeper rule on the automount drive. Once created (and tested successfully moving 4 files) hitting save seemed to work but on coming out of the folder and going back in rule was not saved

Is it a case that sweeper rules cannot be setup on a network mount?
 
Last edited:
Here's what I think you've done:
  1. Use the WebIF media browser to navigate to /media/<external drive>;
  2. Use the "Sweeper Rules" button at the bottom of the page to open the Sweeper GUI;
  3. This then allows the sandboxed creation and testing of a rule set prior to saving a Sweeper config file;
  4. Click "Save changes", then a message appears briefly (you probably didn't notice) saying "Bad directory".
Your problem is that Sweeper rules get processed by scanning the My Video file system (and only the My Video file system) for folders containing a .sweeper configuration file, during the Auto Processing cycle (by default every 10 minutes). The Sweeper GUI knows this and refuses to save a config outside My Video (although it would be handy if it issued a warning before creating rules – there is a use-case for being able to run ad-hoc rules from the GUI, and therefore for not disabling the Sweeper Rules button outside My Video).

(For historical reasons, the Sweeper config which applies to My Video is /mod/etc/sweeper.conf)

For folders (including My Video) with rules attached, the WebIF media browser displays a broom icon against the folder name.

There is a work-around which might achieve what you want, but which comes with potential side-effects to be on the alert for:

As said above, Sweeper rules will only get actioned in My Video, but the mvdisks package actually mounts USB drives (and virtual USB drives) under My Video so that it is not necessary to go via the Media >> Storage >> USB menu to play recordings – instead (or as well) the external drive is accessible in Media >> Storage >> HDD alongside the internal recordings.

The consequence ought to be that mvdisks then fools the rest of the system into including an external drive in the auto-processing, and the Sweeper GUI into accepting the external drive as a valid location. Be aware I have not tested this.

The side-effects to be aware of are that by mounting external storage into My Video, the standard Humax firmware probably won't know the difference so the Media button might slow down while it scans the network drive, free disk space calculations might go awry, shut-down processes might take a long time before the unit goes into stand-by (scanning for DLNA indexing and thumbnail generation). It would be useful if you report back observations.

Be aware you try this at your own risk. There may be unanticipated consequences. Also be aware that as the external drive will appear to be a folder under My Video, you (or another member of the household) might simply delete the folder as unrecognised/not required, using the standard UI – this will systematically delete the entire (video) content of the external drive. THIS HAS HAPPENED, SOMETIMES SEVERAL TIMES BEFORE THE LESSON GOT LEARNED.
 
Last edited:
Hi,

Many thanks for the feedback.


In terms of what I was doing. I was going into the folder and then using the opt+ icon to setup sweeper and you're right never saw any error messages when attempting to save the rule.


I have installed mvdisks and it has created folders matching the drives listed under media (drive1, NAS & Traveller). Traveller is the usb drive connected to the router. However neither the NAS or Traveller folders are showing any of the files within the actual folders, drive1 which I'm assuming is the virtual drive shows correctly. The files within the NAS and Traveller shares are all Humax recordings
 
darlo said:
I was going into the folder and then using the opt+ icon to setup sweeper
Click to expand...
Same difference.

darlo said:
However neither the NAS or Traveller folders are showing any of the files within the actual folders, drive1 which I'm assuming is the virtual drive shows correctly.
Click to expand...
"drive1" is (most likely) the result of the virtual-disk2 package, and it will identify as "VirtualUSB" in the eject list (button at top left on the WebIF - note it takes two clicks to pop up the list due to a bug which thinks that area is also clickable for the WebIF short-cuts). It's there to force the Humax software to recognise network-shares-automount (fake) USB storage is present; if you have real USB storage permanently fitted you don't need virtual-disk2 (although it's not doing any harm).

I'm not a mvdisks user, so I'm at a loss to explain why this isn't working. I have reviewed the mvdisks topic for clues. Perhaps somebody active in that topic can comment @xyz321.

It's worth noting that (found in the mvdisks topic), as well as mvdisks folders potentially confusing settop (the Humax HDR-FOX application) in terms of space calculations and DLNA scanning etc, you also can't use the Humax UI to copy content to/from the mvdisks folders – the Humax UI is (deliberately) being fooled into thinking these areas are on the HDD, so it offers move not copy. Move won't work because all that does is update file system pointers and the actual data payload remains where it is. Whether that will pose a similar problem to the WebIF/sweeper (which are hopefully able to distinguish between actual data and a mounted hard link) remains to be seen.
 
I think this might be barking up the wrong tree. Instead of fudging things (potentially unsuccessfully) to make sweeper work with an external folder as the source, try putting an rsync command in a script on the destination unit. It's not a problem using sweeper to send data to the intermediate location, more of an issue getting it to retrieve data from the intermediate location. (Of course, the second HDR-FOX could be automounted without relying on the intermediate location, but that relies on both HDR-FOXes being turned on at the same time.)

Information about setting up scripts to auto-run at boot can be found here (although you might want to delay execution until the automount has kicked in), or a script call can be added to the cron schedule WebIF >> Diagnostics >> File Editor >> Commonly edited files >> ...crontab... if you want to run it, say, every 15 minutes. Adding another call in anacrontab can make the script run just after boot, before cron gets around to it.
 
Last edited:
Thankyou
Correct the point of this is so the 2 units don’t have to be on at the same time as this is rare. I use to ftp down to a PC and then back again but figured there must be a way to just have it happen.

I’ve had a read of the link I think it will take me a while to sort it. I have a vague recollection of doing something similar in windows many years ago but can’t remember why I did it. So it will be fun playing. I’ll feedback when I either get stuck or hopefully have some success!
 
Made some progress. I've created a script:
#!/bin/sh
[ -d "/media/Traveller" ] && ln -s "/media/Traveller" "/media/My Video/Traveller"

and made it into an executable within the /mod/etc/init.d directory. This creates the link to the mapped disk and allows me to then create a sweeper rule to move any files to the local hard disk. When manually run from the webif interface this works

I have a method that works.

Chanlenges:
The USB drive is mapped using SMB which gives the same issues as @Black Hole , Webif can see the folder but the humax can't.
In itself this doesn't impact on what I am trying to do as long as sweeper works without webif running on a folder the humax can't see/display but must exist

I am unable to connect by NFS (the ASUS router the drive is connected to doesn't support). I have tried using the NAS without success and will try again.

I have left the setup with a file ready to transfer from humas2 to USB to humax1 and will see where we stand tomorrow.

Many thanks @Black Hole once I read your linked post again it made alot more sense.
 
Are you doing anything in Sweeper more than just copy across anything it finds? If you don't need the filtering rules, I recommend you have another look at using rsync to do the fetch instead.
 
I’ll take a look I have it installed,
I have added a rule to ensure file has been decrypted before moving, although this should be redundant as I also check it’s decrypted before moving it from the other box.

On a positive note sweeper has successfully move the recording to the local disk
 
I've setup a NFS share (my issue was with the NAS drive configuration and not the Humax box) and added a line to create a symbolic link. Whilst I can see both of these via Webif (NFS & SMB shares) neither are visible on the box. I've removed the SMB link without any change to the visibility of the NFS link

I have removed "[ -d "/media/Traveller" ] &&" from the script as this seems to be stopping the line from working. Linking seems to be persistence without the need for a script to run on startup. I have removed the lines for both links and they remain after a reboot.

I've done some reading on rsync and I'm struggling on the benefits over sweeper. Whilst there are some hints at it being faster under some circumstances the ease of setup sweeper gives and the filtering rules (even If I don't end up needing them), I think out way this (my aim here is the files just move when I'm not around)
@Black Hole what are your thoughts on the benefits of rsync over sweeper?
 
Simple really. You can put an rsync command into a script to move data around between local and network drives – in this case to take whatever it finds on a network drive and copy it to a local drive. Source and destination paths are declared in the command, and are unrestricted.

Sweeper rules act on the files in the folder containing the rules, and are actioned only if the folder appears to be local. That means jumping through hoops to make a folder on a network drive appear as if it is local,

Thus, for what you are doing, and particularly with complex filtering not required, the retreival of recordings from the intermediate store is very much a force-fit for sweeper, whereas command line file transfer commands are not impeded by them being on a network drive.
 
Makes sense, I think I should use rsync to move from the original humax to the usb too.. Leave sweeper to handle the check for decrypted then move to a folder for rsync to move from there.

I've been playing with the structure between folders on the humax and have copied and then deleted the source files (and folders).
Once I modify it to point the destination to the network I am asked for a password. I have enable ssh on the router and installed dropbear on the humax. Pasted the public key (generated when dropbear was installed) but to no effect. Keep getting asked to enter a password but the only key that works is "enter"

This is what I have "/mod/bin/rsync -avP --remove-source-files '/mnt/hd2/My Video/Transfer/' media@192.XXX.X.XX:Traveler/"

I'm I right that sorting the ssh conection is what I'm missing. I can remember seeing a guide within 1 of the links on how to key new keys somewhere but I suspect this will just create another key which I can't get to work!!
 
darlo said:
I think I should use rsync to move from the original humax to the usb too.. Leave sweeper to handle the check for decrypted then move to a folder for rsync to move from there.
Click to expand...
Seems fair enough.

darlo said:
I have enable ssh on the router and installed dropbear on the humax.
Click to expand...
Must you complicate things that way?

Code: 
media@192.XXX.X.XX:Traveler/
Click to expand...
Like it would matter if we knew your internal IP address!
 
darlo said:
media@192.XXX.X.XX
Click to expand...
Ooooh, only 180 things to try, in all probability. XXX is 168, X is either 0 or 1, and XX is 10 to 99. That's significantly better than 65024 (assuming you're using a /24 mask).

And rsync via ssh will be very slow - about 500kpbs max. (using it over NFS is a much better idea).
 
prpr said:
Ooooh, only 180 things to try, in all probability. XXX is 168, X is either 0 or 1, and XX is 10 to 99. That's significantly better than 65024 (assuming you're using a /24 mask).

And rsync via ssh will be very slow - about 500kpbs max. (using it over NFS is a much better idea).
Click to expand...
Great the mask worked. The address falls outside the range you suggest.
 
The router doesn’t support NFS. My original goal was to use a usb connected to an access point local to the two humax’s. I can share via samba.
Any advice on connecting to a samba share using rsync or is NFS the only realistic option and I need to use the NASfor temp storage?

I have it working using network shares automount and symbolic link (thanks to @Black Hole) but looking to improve the method
 
You must log in or register to reply here.
Back
Top