Insufficient disk space ... for what?

C

cloud9

Member
I've been wondering why my renaming (auto-dedup) hasn't been working well over the last few weeks. So eventually I thought to look in the auto.log and I see a large amount of insufficient space message s. In fact, it looks like a continual stream of them since
26/12/2013 01:10 - Insufficient disk space. Require=1073741824, Free=900935680
Click to expand...
So the questions are
  1. would that prevent the renaming? It doesn;t seem quite right since I thought there had been some renames a few weeks back, but perhaps I am misremembering.
  2. What operation is it that needs the 1073741824 bytes? I presume a decode or something, but is there any way to find out?
My other puzzle is that I know we are in a constant battle for space, but I thought I did have 40G+ free recently (which is a massive amount for us). I haven't checked that every day has a message, but certainly scrolling through, there is an unbroken list of those messages without anything intervening.
 
The surrounding lines in the log file would reveal more about what it was trying to do, but in general the WebIF auto process shuts down if there is less than a gigabyte of spare disk capacity.

It is bad policy to run that close to a full disk all the time. Make some considered decisions about all that junk that's stored up and will probably never be watched. Stuff that is being kept as a library of films (for example) can get shunted off to a cheap USB drive. Set a limit and insist on housekeeping when the limit is exceeded (100GB free would be a good figure). Joe Public seems to have the impression that because you can't see it, data is nothing. A PVR hard drive is no different from a cupboard - once it's full, you have to stop putting things into it or you have to declutter.

Something people seem to forget (or deliberately ignore): there are only so many hours in a day. If you record something because you are too busy to watch it live, you still have to find the time to watch it later - and if it is more than a week old there is a good chance you never will.
 
Deleted items folder and decryption process come to mind. I assume the HDR treats files in the Deleted Items folder in the same way as every other folder, i.e. used space, not available space, otherwise that could cause problems with reported free space. Clearly the background decryption process will need space to copy decrypted files to, before the encrypted versions can be deleted. If I've counted the digits correctly, the free space the HDR says it needs close to 1GB, which could easily be the size of a single recording to be decrypted.
 
cloud9 said:
I've been wondering why my renaming (auto-dedup) hasn't been working well over the last few weeks. So eventually I thought to look in the auto.log and I see a large amount of insufficient space message s. In fact, it looks like a continual stream of them ... So the questions are
would that prevent the renaming? It doesn't seem quite right since I thought there had been some renames a few weeks back, but perhaps I am misremembering.
What operation is it that needs the 1073741824 bytes? I presume a decode or something, but is there any way to find out?
Click to expand...

'auto-processing' stops working when there is very little disk space left, even for things which don't seem to need disk space to work (like renaming or filing processes)

I asked about this recently with reference to another auto-process 'sweeper' (see: http://hummy.tv/forum/threads/seriesfiler-series-auto-filer.278/page-4 post #78)

af123 explained the logic here: http://hummy.tv/forum/threads/seriesfiler-series-auto-filer.278/page-5 post #91

It may be auto-dedupe needs that space to work ...
What other auto-processes have you got set up beyond auto-dedupe?
Auto-decrypt?
Auto-shrink?
Sweeper?
You could try disabling other processes to see what effect it has ....
 
I don't think I've got any folders currently set to auto shrink or decrypt. I don't use sweeper at all

Part of the problem is that I presume it is using the web-if idea of free space, Which I believe excludes the reserved space of 5%. So what the Humax sees as 20G free, the web-if sees as 0.

Anyway, yes it would seem I have to be inventive or get my wife to clear put some stuff before renaming will work again.
 
Many thanks DelftBlue (now I fully read the thread, I was not at computer last night). So the answer is that af123 has an initial check that 1G is free before attempting any of the auto stuff.
The disk space check is in the automatic processing code - sweeper is a plugin to that. Automatic processing is suspended once there is less than 1GiB of free disk space remaining (that's the check you mentioned at line 114). Each function that requires disk space performs an additional check to make sure there is still 1GiB + 3 * <size of file being processed> available as disk space can come and go. The 3* is because the worst case is that you need the original file + temporary file + new temporary file in the video filesystem at the same time.
Those checks could be tweaked to reduce or eliminate that 1GiB buffer but if you're down to your last GiB then it feels right to suspend processing.
Click to expand...

Which all seems reasonable, except as I note above, I do have well over 1G free by the Humax measurement, but not by web-if measurement.
<sigh> I will just have to guess at what else can be archived early or run the renames manually.
 
Maybe there's something that needs looking into there. Unfortunately af123 seems to be busy elsewhere at the moment.
 
cloud9 said:
Many thanks DelftBlue (now I fully read the thread, I was not at computer last night). So the answer is that af123 has an initial check that 1G is free before attempting any of the auto stuff. Which all seems reasonable, except as I note above, I do have well over 1G free by the Humax measurement, but not by web-if measurement....
Click to expand...

You're welcome!

I was puzzling over sweeper and you over auto-dedup - but it's the same root issue.

re: the '1GB rule' that makes sense in its own right ... but doesn't seem to correlate with seeing 30GB or more free space on screen - I've just posted again on the other thread. http://hummy.tv/forum/threads/seriesfiler-series-auto-filer.278/page-5#post-64929
 
I have uploaded webif version 1.0.13-3 which has a new disk space check which will be automatically enabled if your custom firmware version is new enough. Unlike the previous check, this one ignores any reserved blocks on the filesystem. The reserved blocks are irrelevant anyway since everything runs as the super-user and it is for that user the blocks are reserved.
This should stop the automatic processing from being suspended when there is still over 5% of diskspace available.
On my box the web interface free space still doesn't exactly match the on-TV information, but it is fairly close.

You can tell if the new system is in use because the space at the top of the web interface screen will be shown with IEC suffixes, e.g. if the old version showed '1.8T', the new will show '1.8 TiB'
 
I have just upgraded one box to 1.0.13-3, and it is now showing 3% more used, and 3% less available space.
 
421GB on TV and 411.79GiB on webif.

Edit: 424GB on TV after emptying the dustbin.
 
I have recursive autodycrypt and shrink installed at the top folder level and they are working well.

I have noticed, however, that even though a recording has successfully been decrypted and shrunk, the menu option to shrink the recording is still 'clickable'. The decrypt option is correctly greyed out and is not 'clickable'.

If i do select the shrink option, I get a message that the recording has already been shrunk. Which it obviously has.

WebIf version 1.0.13-3
 
af123 said:
I have uploaded webif version 1.0.13-3 which has a new disk space check which will be automatically enabled if your custom firmware version is new enough. Unlike the previous check, this one ignores any reserved blocks on the filesystem. .... This should stop the automatic processing from being suspended when there is still over 5% of diskspace available.
Click to expand...

Many thanks for working on this.

Test results from here for feedback:

Humax HDR 'Box A'
Before - with webif 1.0.13-2
- via TV screen: 553 GB free space
- via web-if page: 509.7 G free space, 59% (Total: 906.1 G, Used: 350.3 G, 41%)
All auto-processing working fine.

After - with webif 1.0.13-3
- via TV screen: 553 GB free space
- via web-if page: 554.16 GiB free space, 61% (Total: 920.5 GiB, Used: 366.34 GiB, 39%)
All auto-processing working fine.

Humax HDR 'Box B'
Before - with webif 1.0.13-2
- via TV screen: 13 GB free space
- via web-if page: 0 free space, 0% (Total: 906.1 G, Used: 890.0 G, 100%)
Sweeper auto-processing suspended due to 'insufficient disk space'

After - with webif 1.0.13-3
- via TV screen: 13 GB free space
- via web-if page: 12.19 GiB free space, 2% (Total: 920.5 GiB, Used: 898.32 GiB, 98%)
Sweeper auto-processing working! Yay! Result!

From auto.log
28/05/2014 21:30 - Insufficient disk space. Require=1073741824, Free=0
28/05/2014 21:40 - Insufficient disk space. Require=1073741824, Free=0
28/05/2014 21:50 - Moving /media/My Video/The Twilight Saga_ Breaking Dawn____20140524_2058.ts to 02_SINGLE PROGS
28/05/2014 21:50 - Moving /media/My Video/Kick-Ass_20140525_2203.ts to 03_FILMS
28/05/2014 21:50 - Resetting unwatched recording flag for /media/My Video/02_SINGLE PROGS
28/05/2014 21:50 - Resetting unwatched recording flag for /media/My Video/03_FILMS

Feedback summary:
21:45 webif 1.0.13-3 installed ...
21:50 sweeper swept up

(Presumably cloud9's auto-dedup will activate too :) )

Cheers af123! Excellent stuff!
 
Fantastical fast work af123. And DelftBlue's results Look very promising as do my first look at the webif numbers. I need to check the TV numbers later.

One thing to note is that I think this will affect the deletion tidy up as well. I think I may need to tune my numbers now it thinks it has more space. (i.e. before 10G really meant 30 etc).


Sent from my Nexus 7 using Tapatalk
 
So 7G free showing on TV, Web-if show 12GiB which fits with my previous observations that it seems to keep ~5G back for something.

Total space: 454.75 GiB
Used: 442.71 GiB (97%)
Free: 12.04 GiB (3%)

(and yes, I am trying to free some up)

And the dedup/rename seems to be running OK.
 
af123 said:
I have uploaded webif version 1.0.13-3 which has a new disk space check which will be automatically enabled
Click to expand...

This worked immediately and auto-processing is now active - which is great. Thank you.

However, I noticed last night that the disk space data and pie chart up top in webif didn't seem to change in the way it used to, in response to deletion or addition of files, even after time elapsing and page refresh.

Is this a general bug or something local to me?

Perhaps other people can report their observations please.
 
Same here, I deleted several gig of recordings using WEBif but the pie chart did not update to reflect change.
It remained unchanged even when I:
- moved across menus
- exited webif (killed browser session) and then re entered webif.

I haven't rechecked any further yet, Hummy in use at time.
 
You must log in or register to reply here.
Back
Top