1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

WebIF "Shrink" option

Discussion in 'HD/HDR-FOX T2 Customised Firmware' started by Black Hole, Jun 15, 2012.

  1. Black Hole

    Black Hole Theloniuos Abbot

    For anyone who has not been following progress, the latest innovation in the WebIF command set is the shrink option (at the time of writing known as "squeeze" but due to change imminently). This removes redundant data from the recorded transport stream and thus the recording occupies less precious disk space, but it is a work-in-progress at this point. Savings are estimated at 20% for a StDef recording and 75% for radio.

    I'm sure Ezra will be along shortly to provide a reference to the relevant Wiki entry.

    I have pulled this out of the general WebIF topic to provide some subdivision of discussion. The relevant part of the WebIF topic starts HERE (click).
    af123 likes this.
  2. Black Hole

    Black Hole Theloniuos Abbot

    I have tried this on one of my radio files, starting at 990MB the initial savings estimate was 77%. The progress bar sat at zero for ages and then jumped up to 31% then 100% right at the end. The result was 220MB, in line with the estimate.

    Unfortunately the resulting file was unplayable (by network mount on HD-FOX).
  3. Scuttlebroom

    Scuttlebroom Member

    Looks like a good package.

    I have tested one film I had already watched, and this is the results shown on WebIf:

    Processing The Hoax_20120604_0002 Moving recording to /media/My Video/Films/_original The Hoax_20120604_0002.thm The Hoax_20120604_0002.ts The Hoax_20120604_0002.hmt The Hoax_20120604_0002.nts WARNING: Found inter-frame packet. Saved: 2945389/17086698 packets, 539.32/-967.33 MiB (17.24%) Processed in: 594.01s Renaming file group to The Hoax_20120604_0002-1339750637 Time taken: 594.123

    Wondering what "WARNING: Found inter-frame packet" means?

    Nice to see it doesn't remove the EPG info that is seen when pressing "i".

    Also it keeps the place you were watching.
  4. Ezra Pound

    Ezra Pound Well-Known Member

    Why would you want to keep an audio file in a video file format, albeit a shrunken one? The TS / MPG contains an MP3 audio track, you are better off stripping out the MP3 if you want to reduce file size
  5. Black Hole

    Black Hole Theloniuos Abbot

    That's the point of patching up the nts file. The programme metadata is kept in the sidecar files too, so it is not part of the stripping process.

    I'm not aware resume play is available with a straight MP3?
  6. Ezra Pound

    Ezra Pound Well-Known Member

    Metadata can be stored in an MP3 ID3 Header and the file can then be played using any MP3 audio player e.g. no need for a Video player.

    That would be a function of the MP3 player
  7. Black Hole

    Black Hole Theloniuos Abbot

    Scuttlebroom is playing video.

    As far as audio is concerned - you do it your way, I'll do it mine (when shrink gets fixed).
  8. af123

    af123 Administrator Staff Member

    Thanks - good call : )

    xyz321 has noted in the webif thread that the rate at which EPG data are broadcast is fairly constant across the Muxes and so the savings will be around 200-300MiB per hour. That figure can be refined as we get more data.

    I hadn't tested shrink on audio recordings but will have a look to see what's going wrong there. It works fine on standard and high definition files although the web interface doesn't currently allow you to shrink high definition because I hadn't tested it at the time.

    The next webif release will include (at least):
    • shrinking of high definition recordings (without the inter-frame packet warnings and statistic overflows);
    • shrinking of audio recordings (if it proves possible!);
    • option to set folders as auto-shrink, auto-dedup and auto-decrypt. When recordings are automatically shrunk or decrypted and you have the undelete package installed, they old versions will be placed in the dustbin and subsequently automatically deleted in line with the retention period you have configured. If you don't use undelete then they are just removed (would people prefer them to be placed in a _original/ sub-folder as per manual shrink through the web interface?);
    • icons against recordings to show if they are shrunk or not (a bit tricky because I have to scan the first few thousand packets of the .ts file to see whether there are any EIT frames in there, but I prefer that approach at the moment to re-purposing a flag in the .hmt file). That's something that happens asynchronously from the web interface display so is filled in shortly after load like the folder sizes and folder new flags.
    • more recording in-use checking to avoid multiple processes accessing the same recording at once - I'm also going to provide an API to this so that other packages can easily hook into it (I'm thinking of flatten, nicesplice, seriesfiler, unencrypt, ...) - it will be good to have everyone doing the same checks.
    * Automatic processing has proven interesting so far. I have a robust locking mechanism in place so that only one copy can run at once (more than simple lock files) which could be used by other processes too (and, for example, I will be extending to the package management scripts so that you can get a Package management is busy message rather than a hang followed by an opkg error). I'm currently working on making it as safe as possible on a system which can power off any moment. There are only a few critical sections while sets of files are being moved around so I'm adding recovery and cleanup mechanisms (basic journalling) to handle these. For people who use redring I'm also adding a general mechanism where it can be instructed to delay the power off command until after a critical section has completed.

    You can see it's quite involved so it may take a few days before I'm ready to publish an update!
    oijonesey likes this.
  9. af123

    af123 Administrator Staff Member

    It means that the .ts file contains a packet which is after the end of one video frame and before the start of the next. Nothing to worry about. It's a fairly rare occurrence with SD recordings but very common in HD ones. Shrink already deals with that case and the next version won't print the warning.

    As Black Hole has noted, it doesn't modify the .hmt sidecar file which is where this is stored.

    Should preserve bookmarks too!
  10. af123

    af123 Administrator Staff Member

    I found some time to work on this so I've released an update this evening.​
    It now works on radio recordings for me - (~75% saving!)​
    The automatic folder processing is there - I do suggest you also install undelete as the automatic processing will move the old recordings into the dustbin initially and you can get it back if something goes wrong. I've been using it all week and it's happily chewed through lots of my archived recordings so I'm happy to publish it now.​
  11. Ezra Pound

    Ezra Pound Well-Known Member

    Just a small detail, the new 0.9.12-1 Web-If has shrink instead of squeeze, but the 'do it' icon still says squeeze, It also referes to TIME : squeezing and squeezing = XX%

    shrink-squeeze.jpg shrink-squeeze2.jpg
  12. Brian

    Brian Administrator Staff Member

    I have just updated my boxes to the latest version of webif (0.9.12-1), and see that most of my recordings are now shown as "shrunk" even though I have not done anything to them. Hopefully this is just a bug which can be fixed.
  13. xyz321

    xyz321 Well-Known Member

    I am not sure if the new auto shrink function is actually doing anything at the moment. Some of the files are shown as being shrunk but the list does not seem to grow. Also there are no additional files in [Deleted Items], is this correct? If not, then am I in danger of running out of disk space with only 60GB free? The file /tmp/webif-auto.log only shows "Got lock." for each time the cron job has been run.
  14. Brian

    Brian Administrator Staff Member

    I suspect that the marked files have not been shrunk, there must be a bug which is causing the icon to show incorrectly. I have not had any of these files added to my dustbin.
  15. xyz321

    xyz321 Well-Known Member

    Yes, they have definitely not been shrunk.
  16. xyz321

    xyz321 Well-Known Member

    It seems that auto-shrink is enabled by clicking on OPT+ for a sub-directory in the webif browser.
  17. Michaelnuk

    Michaelnuk Member

    Missed this one looking through webif also looks like Autodedup has made an appearance.

    Sweet! :)
  18. xyz321

    xyz321 Well-Known Member

    On an HDR if an external disk is connected, the 'auto-' options still appear in the OPT+ dropdown for directories on this disk. However, they are ineffective because the background task only looks at the internal disk. Perhaps the menu options should be greyed out or made to work (whichever's the easiest).
  19. af123

    af123 Administrator Staff Member

    There's a bug that causes some HD files to show as shrunk. It will be fixed later.
    brian likes this.
  20. af123

    af123 Administrator Staff Member

    Yes, you enable it selectively on directories then the contents are shrunk in the background and any new recordings that appear in there are also shrunk automatically. I'd start with something small and see how it goes.
    Black Hole likes this.