[undelete] package released!

Ezra Pound · Feb 5, 2012

Is there a undelete.log or any debug available for viewing as I am trying to sort out why only some of the files in my [Deleted] directory are being auto deleted

af123 · Feb 5, 2012

There's a log file for the main module (that changes deletes into renames to the dustbin folder) but not one for the empty process, but you can run /mod/sbin/empty_dustbin by hand from telnet/ssh if it helps.
I should probably add a log file...

Ezra Pound · Feb 5, 2012

O.K. I had 6 items (all in sub folders) in my [Deleted] folder all deleted over 2 days ago, The Undelete is set up is set to 1 day. I manually ran empty_dustbin and 3 disapeared, The remaining 3 have folder dates more recent than the time I deleted them, I can't explain why as yet, But it looks like the folders are being re-visited which is changing the folder dates. I will do some more investigation

af123 · Feb 5, 2012

Ok, thanks. The timestamp on any .hmt files is already ignored as long as there are no other files with the same prefix left - that's necessary because auto-unprotect can be a bit keen sometimes and modifies .hmt files within the dustbin.

Ezra Pound · Feb 5, 2012

I have emptied the [Deleted] directory and deleted some new files, I will keep a check on the time stamps, Could it be that Undelete is taking the time stamp from the Series directory that contains the deleted files rather than the files themselves?

af123 · Feb 5, 2012

No, effectively all the script does is this:

Code:

find . -type f -mtime +$keep -exec trm {} \; -print

which deletes any files with a modification time > $keep days. It uses the trm command rather than just rm because it is nicer to the Humax..

MartinLiddle · Feb 6, 2012

af123 said:
It uses the trm command rather than just rm because it is nicer to the Humax..

I am not familiar with trm and it isn't implemented on Centos 5.7 which my servers are currently using. Purely as a matter of interest, how does it differ from rm and where can I read about it?

Ezra Pound · Feb 6, 2012

trm is Truncate ReMove, It deletes in 'chunks', There is a bit of info. on the WiKi

MartinLiddle · Feb 6, 2012

Ezra Pound said:
trm is Truncate ReMove, It deletes in 'chunks', There is a bit of info. on the WiKi

Thanks. Is it part of Busybox or the linux distribution that the Humax uses or specially written?

Ezra Pound · Feb 6, 2012

MartinLiddle said:
Thanks. Is it part of Busybox or the linux distribution that the Humax uses or specially written?

It's not part of Busybox, I think it was written / Complied by af123 it uses Truncate64 some info HERE. There had been some problems in the past while deleting files because of the load it puts on the system trm is less demanding

MartinLiddle · Feb 6, 2012

Ezra Pound said:
It's not part of Busybox, I think it was written / Complied by af123 it uses Truncate64 some info HERE.

Thanks.

af123 · Feb 6, 2012

Ezra's right, it's a utility I wrote to mimic the way in which the Humax software removes large files. It successively truncates the file in steps down to 40MiB then deletes it. If you remove a large file using rm then it can cause problems with HD recording and playback (it's easy to cause HD playback to jitter then stop by doing this). Using trm doesn't have the same effect.

I'm not entirely sure why it is so much better but ext3 isn't great at storing large files and removing a large file causes a lot of work to traverse the block pointers and free everything - performing multiple truncation operations probably spreads this work across multiple transactions.

Ezra Pound · Feb 6, 2012

While looking into Undelete I saw a request to look in /var/log/humaxtv.log to see the Version of 'Dustbin'. Is this file always present on the HDR as I don't seem to have this file

Code:

humax# cd /var/log
humax# ls -al
drwxrwxrwt    2 root    root          280 Feb  6 19:09 .
drwxr-xr-x  18 root    root          192 Jan 10 09:10 ..
-rw-r--r--    1 root    root            0 Jan  1  2000 .humaxtv.running
-rw-rw-rw-    1 root    root            0 Feb  6 13:06 .modinit.done
-rw-r--r--    1 root    root            4 Jan  1  2000 dnsmasq.pid
-rw-rw-rw-    1 root    root          2103 Feb  6 18:58 epgd.log
-rw-rw-rw-    1 root    root            37 Feb  6 13:06 hosts
-rw-rw-rw-    1 root    root            0 Feb  6 13:06 if-up
-rw-r--r--    1 root    root            6 Jan  1  2000 ifstate
-rw-rw-rw-    1 root    root          4614 Feb  6 13:06 modinit.log
-rw-rw-rw-    1 root    root          240 Feb  6 17:02 mongoose_error.log
-rw-rw-rw-    1 root    root          2392 Feb  6 13:06 rag.log
-rw-r--r--    1 root    root          311 Jan  1  2000 rsvsync.log
-rw-r--r--    1 root    root            0 Jan  1  2000 xinit.log
humax#

af123 · Feb 6, 2012

You need to enable humaxtv debugging to get that log file (very useful for debugging portal applications!):

Code:

humax# touch /mod/boot/debugtv

I have it on all the time, it doesn't get particularly large and clears on reboot since it is in memory.

Ezra Pound · Feb 7, 2012

I still seem to have files in my [Deleted] that are more than 1 day old, In order to understand why, I did the following, Question Why doesn't 'find' find these files?

Code:

humax# ls -al
drwxr-xr-x    2 root    root          4096 Feb  5 23:28 .
drwxr-xr-x  11 root    root          4096 Feb  7 00:40 ..
-rw-r--r--    1 root    root          2072 Feb  5 23:28 Click_20120204_0130.hmt
-rw-r--r--    1 root    root      1434496 Feb  5 23:28 Click_20120204_0130.nts
-rw-r--r--    1 root    root        43680 Feb  5 23:28 Click_20120204_0130.thm
-rw-r--r--    1 root    root    788127744 Feb  5 23:28 Click_20120204_0130.ts
humax# date
Tue Feb  7 16:17:07 GMT 2012
humax# find . -type f -mtime +1
humax#

Could it be that although the files are old enough the [Deleted] directory that they are in is is not?

Code:

drwxr-xr-x  11 root    root          4096 Feb  7 00:40 .
drwxr-xr-x  11 root    root          4096 Feb  7 00:40 ..
drwxr-xr-x    2 root    root          4096 Feb  6 21:26 An Island Parish
drwxr-xr-x    2 root    root          4096 Feb  5 23:28 Click
drwxr-xr-x    2 root    root          4096 Feb  6 21:53 Cornwall with Caroline Quentin
drwxr-xr-x    2 root    root          4096 Feb  6 19:28 Helen's Polar Challenge for___
drwxr-xr-x    2 root    root          4096 Feb  6 19:42 Ocean Giants
drwxr-xr-x    2 root    root          4096 Feb  7 00:40 The Diamond Queen
drwxr-xr-x    2 root    root          4096 Feb  5 20:04 The Jonathan Ross Show
drwxr-xr-x    2 root    root          4096 Feb  5 22:56 The Singing Detective
drwxr-xr-x    2 root    root          4096 Feb  6 20:39 University Challenge
humax#

af123 · Feb 7, 2012

It shouldn't be related to the directory. I presume if you take off the "-mtime +1" then it shows the files?
It's possible there is a bug in the busybox find command but it definitely does remove the files eventually in my experience.

Ezra Pound · Feb 7, 2012

Yes, Removing the -mtime +1 finds the files. I think that if I delete a file every day which up-dates the time / date of the [Deleted] directory that no files in the directory get deleted, But if I stop deleting new files then the time / date of the directory becomes more that 1 Day old and then files get deleted. Can't explain why

Code:

humax# find . -type f
./Click_20120204_0130.nts
./Click_20120204_0130.thm
./Click_20120204_0130.ts
./Click_20120204_0130.hmt
humax#

af123 · Feb 7, 2012

The find command should only be looking at files, not directories...

It's working ok for me, even if the current directory is dated today.

Code:

humax# touch .
humax# ls -ld .
drwxr-xr-x    2 root    root          4096 Feb  7 22:20 .
humax# ls -l
-rw-r--r--    1 root    root          2072 Feb  1 22:04 Suits_20120131_2100.hmt
-rw-r--r--    1 root    root      2873248 Feb  1 22:04 Suits_20120131_2100.nts
-rw-r--r--    1 root    root        43680 Feb  1 22:04 Suits_20120131_2100.thm
-rw-r--r--    1 root    root    1069817856 Feb  1 22:04 Suits_20120131_2100.ts
humax# find . -type f -mtime +5
./Suits_20120131_2100.hmt
./Suits_20120131_2100.ts
./Suits_20120131_2100.thm
./Suits_20120131_2100.nts
humax# find . -type f -mtime +6
humax#

Could you try touching one of those files into the past and see if the find command starts detecting it?

Code:

humax# opkg install coreutils
humax# /mod/bin/touch -t 201202052100 Suits_20120131_2100.ts
humax# ls -l
-rw-r--r--    1 root    root          2072 Feb  1 22:04 Suits_20120131_2100.hmt
-rw-r--r--    1 root    root      2873248 Feb  1 22:04 Suits_20120131_2100.nts
-rw-r--r--    1 root    root        43680 Feb  1 22:04 Suits_20120131_2100.thm
-rw-r--r--    1 root    root    1069817856 Feb  5 21:00 Suits_20120131_2100.ts
humax# find . -type f -mtime +5
./Suits_20120131_2100.hmt
./Suits_20120131_2100.thm
./Suits_20120131_2100.nts

(ok, that was into the future but you get the idea).

-t STAMP use [[CC]YY]MMDDhhmm[.ss] instead of current time

Ezra Pound · Feb 7, 2012

Yes that worked

Code:

humax# opkg install coreutils
Installing coreutils (8.11) to root...
Downloading http://hummypkg.org.uk/hdrfoxt2/base/coreutils_8.11_mipsel.opk.
Configuring coreutils.
humax# /mod/bin/touch -t 201202042328 Click_20120204_0130.ts
humax# ls -al
drwxr-xr-x    2 root    root          4096 Feb  5 23:28 .
drwxr-xr-x  11 root    root          4096 Feb  7 00:40 ..
-rw-r--r--    1 root    root          2072 Feb  5 23:28 Click_20120204_0130.hmt
-rw-r--r--    1 root    root      1434496 Feb  5 23:28 Click_20120204_0130.nts
-rw-r--r--    1 root    root        43680 Feb  5 23:28 Click_20120204_0130.thm
-rw-r--r--    1 root    root    788127744 Feb  4 23:28 Click_20120204_0130.ts
humax# find . -type f -mtime +1
./Click_20120204_0130.ts
humax#

EDIT:- I also 'touched' it back to the original date (Feb5) and find didn't find it

af123 · Feb 7, 2012

So it looks like +1 means more than 48 hours old.

That's borne out by the code:

Code:

ACTF(mtime)
{
    time_t file_age = time(NULL) - statbuf->st_mtime;
    time_t mtime_secs = ap->mtime_days * 24*60*60;
    if (ap->mtime_char == '+')
        return file_age >= mtime_secs + 24*60*60;
...

[undelete] package released!

Well-Known Member

Administrator

Well-Known Member

Administrator

Well-Known Member

Administrator

Super Moderator

Well-Known Member

Super Moderator

Well-Known Member

Super Moderator

Administrator

Well-Known Member

Administrator

Well-Known Member

Administrator

Well-Known Member

Administrator

Well-Known Member

Administrator