Automatic edit point bookmarks following crop

Matthew · Dec 30, 2018

The current cropping creates a cropped file with no bookmarks, if your wishing to keep the file for a while before watching its useful to check the edit points haven't messed up the program before you delete the original file, but it can be awkward and time consuming to find those points.

The following automatically calculates the bookmarks for the edit points from the cropping bookmarks, the bookmark is adjusted by 3 secs to ensure its just past the edit point so it can be used if there is excessive sound disturbance over a crucial bit of dialogue.

A save button is added so the new bookmarks can be saved post crop.

Firstly copy the file save.jim from mod/webif/html/browse/bookmarks/ to mod/webif/html/browse/crop/ and ensure the file has execute permission following upload.

Edit:
See post https://hummy.tv/forum/threads/automatic-edit-point-bookmarks-following-crop.9057/#post-128315 for updated code

Black Hole · Dec 30, 2018

The above appears to be a solution to my request:

Black Hole said:
Feature request: when cropping, place a bookmark at the start of each saved section (end of each cropped section). This may have to be an option so as to preserve previous behaviour, but I see no disadvantage in having the bookmarks there.

Section bookmarks would provide a means to skip to "chapters", something sorely lacking compared with DVD playback. If the cropping is the result of ad-detection, the chapters will have been identified by the programme makers deciding where to have the ad breaks. If the cropping is the result of the user picking out highlights, again the chapter markers will be appropriate.

(Edit: I put the wrong quote in)

MontysEvilTwin · Dec 31, 2018

I am probably being a bit dim but how can I make these changes? There are three blocks of code above and reference to a file 'save.jim' but it is not obvious what to do.

Black Hole · Dec 31, 2018

Unified Diff Format

This is a standard representation, and commonly used on this forum for discussing code changes. If you are going to get into hand-modifying the CF files, you need to become familiar it.

The content is auto-generated by comparing a backup of the original file with the updated file, on the Linux command line

diff -u <original> <updated>

(the -u switch stands for "unified"; without that switch diff defaults to its original, less compact, format).

In the output, lines are prefixed by indicators which specify how to interpret that particular line:

--- : the original file (documented for the record);

+++ : the updated file (documented for the record);

@@ : location and number of lines in each file described by the subsequent text (until the next @@ line), as follows:

@@ -<start line number in original file>,<number of lines> +<start line number in updated file>,<number of lines> @@

<space> : unaltered line, unnecessary but provided for visual context (three lines lead in and three lines lead out typical, including blank lines);

- : a line present in the original file and not present in the updated file (in its identical form);

+ : a line present in the updated file and not present in the original file (in its identical form).

To implement the code changes described:

Locate the original file in the file system and make a backup copy;
Open the original file in an editor, eg nano on the command line or WebIF >> Diagnostics >> File Editor;
Locate the section of the file described by the @@ string, confirm visually using the context lines;
Remove the lines prefixed "-" and replace with lines prefixed "+" (often this involves a small edit rather than complete replacement;
Repeat from 3 until all the @@ sections have been done;
Save the file;
(Optional) confirm the changes have been implemented correctly by running diff -u on the backup and your new file, and compare the output with the recipe.

For more information, google "unified diff format".

Applying the Diff Automatically (Linux command patch)

Please see below before implementing

Create a file containing the Unified Diff Format description of the changes required (or obtain a ready-made one). Apply the changes on the command line (Telnet, or WebIF >> Diagnostics >> Command Line):

patch -b < <changes file>

The "-b" switch creates a backup of the original file. The file to be edited does not need to be specified, as this is already defined in the changes file.

I will also verify whether the changes file can contain all the diffs in one, or if they need a separate patch operation for each file to be modified.

Yes, it is possible to concatenate all the diffs into one file, but (1) the Busybox version of patch (as installed in the CF) doesn't seem to accept the -b flag, and (2) won't act on its own output!

prpr said:
The version of patch in Busybox is old and stupid. It doesn't complain about things it should complain about. If you copy the diffs as posted on to a sane system and attempt it, then it tells you they are all garbage.

MymsMan · Jan 1, 2019

Similar code could be added to detectads to set bookmarks in the cropped output file when autocropping

Also in Join to preserve bookmarks when joining files together

Matthew · Jan 1, 2019

MymsMan said:
Similar code could be added to detectads to set bookmarks in the cropped output file when autocropping

I`ve already done that, I`ll post it in the thread https://hummy.tv/forum/threads/detect-ads.8923/ later

MymsMan said:
Also in Join to preserve bookmarks when joining files together

Hadn't thought of that, good idea.

Matthew · Jan 1, 2019

MontysEvilTwin said:
I am probably being a bit dim but how can I make these changes? There are three blocks of code above and reference to a file 'save.jim' but it is not obvious what to do.

The posted changes are of a patch file created by Meld

This site gives some useful info https://linuxacademy.com/blog/linux/introduction-using-diff-and-patch/

Matthew · Jan 1, 2019

MontysEvilTwin said:
I am probably being a bit dim but how can I make these changes? There are three blocks of code above and reference to a file 'save.jim' but it is not obvious what to do.

I should have added, if you don't copy the save file it won't break anything, the save function simply won't work, but you can still copy the displayed new bookmarks (eg 0 647 1391) and then paste those into the bookmaks bar in 'bookmarks' for the file.

/df · Jan 1, 2019

Black Hole said:
...

patch -b < <changes file>

The "-b" switch creates a backup of the original file. The file to be edited does not need to be specified, as this is already defined in the changes file.
...

In case it helps, the patch utility is included in the busybox CF package from the repository but not in the version of busybox built into CF 3.13.

The package version appears as /mod/bin/busybox/busybox while the CF version is /bin/busybox. The difference in available utilities is especially relevant for scripts that have to run when /mod may not be mounted, eg maintenance mode. For instance. instead of tr or awk, you have to work up an equivalent script for sed (which is in the CF).

This is the current situation (CF bb1.20 vs repo bb1.20-1). But, utilities in the Both column may not be quite the same build (eg ash).

Code:

Both        CF only     Repo only
====        =======     =========
[
[[
                        ar
                        arp
                        arping
ash
                        awk
                        base64
                        basename
                        bunzip2
                        bzcat
                        bzip2
                        cal
 cat
                        catv
                        chgrp
 chmod
 chown
                        chroot
                        cksum
 clear
                        cmp
                        comm
                        conspy
 cp
                        cpio
                        crond
                        crontab
 cut
 date
                        dc
 dd
 df
            dmesg
            dnsdomainname
                        diff
                        dirname
                        dos2unix
 du
 echo
                        ed
 egrep
 env
            fdisk
                        ether-wake
                        expr
                        false
 fgrep
 find
                        fold
 free
 fsync
            getty
                        ftpget
                        ftpput
                        fuser
 grep
            halt
                        gunzip
                        gzip
                        hd
                        head
 hexdump
            hostname
            id
 ifconfig
            ifdown
            ifup
            init
                        inotifyd
 insmod
            ip
                        install
                        iostat
 kill
 killall
                        killall5
                        less
 ln
            login
                        logname
                        losetup
 ls
 lsmod
            mdev
                        lsof
                        lsusb
                        lzcat
                        lzma
                        lzop
                        lzopcat
                        md5sum
 mkdir
                        mkdosfs
 mkfifo
                        mkfs.vfat
 mknod
 mkswap
 mktemp
                        modinfo
 modprobe
 more
            mount
                        mpstat
 mv
                        nc
                        netstat
 nice
                        nmeter
                        nohup
                        nslookup
                        od
                        patch
                        pgrep
 ping
            pipe_progress
            poweroff
                        pkill
                        pmap
                        printenv
                        printf
 ps
                        pstree
 pwd
            reboot
                        pwdx
                        readlink
                        realpath
 renice
                        resize
 rm
 rmdir
 rmmod
 route
            run-parts
                        rpm
                        sendmail
 seq
 sh
                        sha1sum
                        sha256sum
                        sha512sum
 sleep
                        sort
                        split
                        stat
                        strings
 stty
            su
                        sum
 swapoff
 swapon
 sync
 sysctl
 tail
                        tar
 tee
                        telnet
 test
 time
                        timeout
                        top
 touch
            udhcpc
            umount
                        tr
                        traceroute
                        true
                        tty
                        uname
                        uniq
                        unix2dos
                        unlzma
                        unlzop
                        unxz
                        unzip
                        uptime
                        usleep
                        uudecode
                        uuencode
 vi
            watchdog
                        watch
                        wc
 wget
 which
            zcip
                        who
                        whoami
                        xargs
                        xz
                        xzcat
                        yes
                        zcat

The following shell function parses the output of the busybox command:

Code:

bbfn() {
if [ "$1" = "--help" ]; then
    printf "Usage: bbfn [command_name]\nList supported Busybox commands, or search (regexp) for specified command.\n"
    return
fi
(while read line; do
    if [ "$line" = "Currently defined functions:" ]; then
        xargs -r -n 1 echo | tr -d ',' | if [ -n "$1" ]; then grep -E "$1"; else cat; fi;
        return $?
    fi
    done;
}

Eg:

Code:

humax# busybox | bbfn
[
[[
...
yes
zcat
humax# busybox | bbfn zcat
bzcat
lzcat
xzcat
zcat
humax# busybox | bbfn "^zcat$"
zcat
humax#

Matthew · Jan 1, 2019

MymsMan said:
Also in Join to preserve bookmarks when joining files together

I had time so here's the mod for join

Diff:

--- /mod/webif/html/browse/join/execute_orig.jim
+++ /mod/webif/html/browse/join/execute.jim
@@ -4,10 +4,20 @@
source /mod/webif/lib/setup
require ts.class pretty_size

+proc times {time} {
+  return [clock format $time -format "%T"]
+}
+
httpheader

cgi_input 1

+array set bookmrksarray {}
+array set len {}
+set finalsize 0
+set grow 0
+set newbookmark ""
+set i 1
set joinstart [clock milliseconds]

set cmd {/mod/bin/nicesplice}
@@ -17,6 +27,11 @@
     set file [cgi_unquote_input $file]
     lappend cmd "-in" [file rootname $file]
     set dir [file dirname $file]
+    set ts [ts fetch $file]
+    set bookmrksarray($i) [$ts bookmarks 1]
+    set len($i) [$ts duration 1]
+    incr finalsize $len($i)
+    incr i 1
}
lappend cmd "-out" "$dir/$dst"

@@ -28,6 +43,30 @@
set ts [ts fetch "$dir/$dst.ts"]
$ts settitle $dst

+set newsize [$ts duration 1]
+
+foreach {k v} $bookmrksarray {
+  if {$k == 1} {
+      append newbookmark "$v "
+  } else {
+      set lk $($k -1)
+      incr grow $($len($lk)+1)
+      foreach b $v {
+          if {[expr $b + $grow] <= $newsize} {
+              if {[lindex $newbookmark end] > [expr $b + $grow - 30]} {
+                  set n $([llength $newbookmark] - 1)
+                  set newbookmark [lreplace $newbookmark $n $n]
+              }   
+              append newbookmark " $($b + $grow) "
+          }
+      }
+  }
+}
+
+$ts setbookmarks $newbookmark
set jointime [expr [expr [clock milliseconds] - $joinstart] / 1000.0]
puts "Time taken: $jointime"
-
+puts "\nSource combined length = $finalsize ([times $finalsize]), $dst.ts joined length = $newsize ([times $newsize])"
+set n [llength $newbookmark]
+if {$n} {
+set bmtimes [join [lmap i $newbookmark {times $i}] ", "]
+puts "\n$n combined bookmarks: $newbookmark \n( $bmtimes )\nSaved to $dst.ts"
+} else {
+puts "\nNo bookmarks found on source files to combine."
+}

Edit: Minor change, previous version could create bookmarks close together which it seems the Humax doesn't like, now if bookmarks are within 30secs only the latter is included.

Matthew · Jan 12, 2019

Having used this mod for a while I find I keep forgetting to click the 'save new bookmarks' button, so this is a minor change adding a tickbox option so new bookmarks are auto saved following crop.

This change affects crop.js, crop.jim and execute.jim.

Diff:

--- /mod/webif/html/browse/crop/crop_orig.jim
+++ /mod/webif/html/browse/crop/crop.jim
@@ -23,13 +23,13 @@
<tr><th>File:</th><td>$rfile</td></tr>
<tr><th>Length:</th><td>[clock format $len -format \"%T\"]</td></tr>
<tr><th>Size:</th><td>[pretty_size [$ts size]] ([$ts get definition])</td></tr>
-<tr><th>Bookmarks:</th><td>[$ts get bookmarks] @ "
+<tr><th>Bookmarks:</th><td id=originalbookmarks>[$ts get bookmarks] @ "

puts [join [lmap i [$ts bookmarks 1] {
     clock format $i -format "%T"
}] ", "]

-puts "</td></tr><tr><th>File</th><td>"
+puts "</td></tr><tr><th>File</th><td id=cutplan>"

puts "<div style=\"position: relative; top: -10px\">"

@@ -86,7 +86,42 @@
</td></tr>
<tr><th>Keeping:</th><td>
[clock format $keeping -format "%T"] ([format "%.2f" $keepperc]%)
-</td></tr>
+</td></tr>"
+
+set end -1
+set removing 0
+set i 0
+set newbookmark ""
+foreach b $bookmarks {
+    if {$i == 0} {
+        set removing $b
+        append newbookmark "0 "
+        incr i 1
+        continue
+    }
+    if {$end == -1} {
+        set start $($b - $removing)
+        append newbookmark "$($start +3) "
+        set end 0
+    } else {
+        set end $($b - $removing)
+        incr removing $($end - $start)
+        set end -1
+    }
+}
+ set i $([llength $newbookmark] - 1)
+ if {[lindex $newbookmark end] > [expr $keeping - 8] && [expr $keeping - 8] > 0} {
+    set newbookmark [lreplace $newbookmark $i $i]
+    append newbookmark " $($keeping - 8)"
+ }
+
+puts "<tr><th title=\"Calculated bookmarks for edit points following crop.\">New Bookmarks:</th><td id=newbookmarks>"
+puts "<span id=\"bookmarks\" val=\"$newbookmark\">$newbookmark</span>&nbsp;&nbsp;&nbsp;&nbsp;( "
+puts [join [lmap i $newbookmark {
+    clock format $i -format "%T"
+}] ", "]
+
+puts " )</td></tr>
<tr><th>Time:</th>
<td>Cropping will take around [clock format $esttime -format "%T"]</td></tr>
</table>
@@ -100,9 +135,11 @@
<div id=cropdiv style=\"padding: 1em\">
<button id=invert invert=$invert>Invert selection</button>
<button id=cropit>Perform crop operation</button>
+<input id=saveIt type=checkbox name=saveIt checked /> Then Save New Bookmarks?
+<label id=saveItLabel for=saveIt>&nbsp;</label>

</div>
<button id=back>Back to media list</button>
-
+<button id=save class=\"hidden\">Save New Bookmarks</button>
+<div id=results class=\"hidden blood\"></div>
<div id=progressdiv class=hidden>
Cropping: <div id=progressbar></div>
</div>

Diff:

--- /mod/webif/html/browse/crop/crop_orig.js
+++ /mod/webif/html/browse/crop/crop.js
@@ -3,8 +3,8 @@
function update()
{
     var perc = $('#params').attr('perc');
-    var file = $('#params').attr('file');
-
+    var file = escapestring($('#params').attr('file'));
+ 
     $.get('progress.jim' + '?perc=' + perc + '&file=' + file,
         function(data) {
         if (handle)
@@ -12,15 +12,28 @@
     });
}

+function escapestring (str) {
+  str = JSON.stringify(String(str));
+  str = str.substring(1, str.length-1);
+  return str;
+}
+
$(document).ready(function() {
+ 
+  $('#saveIt').hide();
+  $('#progressbar').reportprogress(0);
+  $('#saveItLabel').css({'height':'22px','width':'70px','background-image':'url("/img/on.png")',
+    'display':'inline-block','background-size':'cover'});
+  $("#saveIt").change(function(){
+    if($(this).is(":checked")) $(this).next("label").css('background-image','url("/img/on.png")');
+    else $(this).next("label").css('background-image','url("/img/off.png")');
+  });

-$('#progressbar').reportprogress(0);
+  $('#back').button().click(function() {
+    window.location = '/go/browse?dir=' + $('#params').attr('dir');
+  });

-$('#back').button().click(function() {
-    window.location = '/go/browse?dir=' + $('#params').attr('dir');
-});
-
-$('#cropit').button().click(function() {
+  $('#cropit').button().click(function() {
     $('#cropdiv').hide('slow');
     $('#progressdiv').show('slow');
     $('#back').hide();
@@ -31,15 +44,43 @@
         function() {
         clearInterval(handle);
         handle = 0;
-        $('#back').show();
+        $('#back').show('slow');
+        $('#save').show('slow');
+        $('#cutplan').html('File cropping complete');
+        $('#originalbookmarks').html('Empty');
         $('#progressbar').reportprogress(100);
+        if($('#saveIt').prop('checked')){
+            $.post('save.jim', {
+              file: $('#fileparams').attr('file'),
+              bookmarks: $('#bookmarks').attr('val')
+              }, function(data) {
+                $('#results').html(data)
+                .slideDown('slow').delay(6000).slideUp('slow')
+                .css('color','green');
+                $('#originalbookmarks').html($('#newbookmarks').clone());     
+                $('#progressdiv').delay(3000).text('Cropping complete.');
+             });
+        }
     });
+  });
+
+  $('#invert').button().on('click', function() {
+    window.location = 'crop.jim?file=' + $('#params').attr('file') +
+       '&invert=' + ($(this).attr('invert') == '1' ? '0' : '1');
+  });
+
+  $('#save').button({icons: {primary: "ui-icon-disk"}})
+    .on('click', function() {
+    $('#progressdiv').hide();
+    $('#output').hide();
+    $.post('save.jim', {
+        file: $('#fileparams').attr('file'),
+        bookmarks: $('#bookmarks').attr('val')
+    }, function(data) {
+        $('#results').html(data)
+            .slideDown('slow').delay(6000).slideUp('slow')
+            .css('color','green');
+        $('#originalbookmarks').html($('#newbookmarks').clone());     
+    });
+  });
});

-$('#invert').button().on('click', function() {
-    window.location = 'crop.jim?file=' + $('#params').attr('file') +
-       '&invert=' + ($(this).attr('invert') == '1' ? '0' : '1');
-});
-
-});
-

Diff:

--- /mod/webif/html/browse/crop/execute_orig.jim
+++ /mod/webif/html/browse/crop/execute.jim
@@ -68,6 +68,10 @@

set newname "$shname-[clock seconds]"
puts "Renaming file group to $newname"
+puts "<span class=hidden id=fileparams
+ file=\"$dir/$newname.ts\"
+></span>"
ts renamegroup "$dir/$shname.ts" $newname
exec /mod/bin/hmt "+setfilename=$newname" "$dir/$newname.hmt"
# New nicesplice shrinks whilst cropping.

Edit: modified to correct bug when file-name contains a quote char

Black Hole · Sep 28, 2019

I've just tried to implement this, and found the alterations to execute.jim failed:

Code:

HDRFOX1# patch < edits3                                                                                               
patching file execute.jim                                                                                             
Hunk 1 FAILED 68/68.

I also found patch -b doesn't work (BusyBox v1.20.2).

I don't know what to do now, I presume execute.jim has been updated since the patches were published. I have reverted the files.

Any chance we can have these mods built into the standard CF files?

prpr · Sep 28, 2019

Black Hole said:
I don't know what to do now, I presume execute.jim has been updated since the patches were published.

It hasn't, but it's easy enough to do manually - just 3 lines to add.
For the record, it's just a spacing thing in the diff, possibly caused by the forum editor:

Diff:

--- /mod/webif/html/browse/crop/execute_orig.jim
+++ /mod/webif/html/browse/crop/execute.jim
@@ -68,6 +68,10 @@

 set newname "$shname-[clock seconds]"
 puts "Renaming file group to $newname"
+puts "<span class=hidden id=fileparams
+ file=\"$dir/$newname.ts\"
+></span>"
 ts renamegroup "$dir/$shname.ts" $newname
 exec /mod/bin/hmt "+setfilename=$newname" "$dir/$newname.hmt"
 # New nicesplice shrinks whilst cropping.

Black Hole · Sep 28, 2019

I take it this still applies?:

Matthew said:
Firstly copy the file save.jim from mod/webif/html/browse/bookmarks/ to mod/webif/html/browse/crop/ and ensure the file has execute permission following upload.

/df · Sep 30, 2019

Post #8 applies.

You could just link it if you use the command line, and then the permissions come free.

Code:

ln /mod/webif/html/browse/bookmarks/save.jim /mod/webif/html/browse/crop/

But really someone ought to offer these changes to the Webif codebase as a pull request. In that case it might be better not to duplicate save.jim. I think a /mod/webif/html/browse/crop/save.jim something like this should do:

Code:

#!/mod/bin/jimsh
source /mod/webif/html/browse/bookmarks/save.jim

Black Hole · Sep 30, 2019

I've been trying to get @af123's attention, but he might have drowned watching the cycling.

Black Hole · Oct 17, 2019

I don't know why, but having made the mods and used it once, the "save new bookmarks" button no longer appears after a crop. Any guesses?

Matthew · Oct 17, 2019

Does the 'New Bookmarks' entry show?

The button is shown by the script, if something is causing a script error that would prevent it working.

Have you tried with different files? (maybe file-name issue)

Black Hole · Oct 17, 2019

Nothing special about the filename (Strictly - same structure as the example which worked). Anything I can do to diagnose?

Matthew · Oct 17, 2019

Does the New Bookmarks entry show?

Which browser, have you cleared the cache.

Does the progress bar disappear on completion?

If your browser has an inspect option click on inspect over the back button, is <button id="back" class="ui-button ui-corner-all ui-widget" style="display: inline-block;">Back to media list</button> followed by:

<button id="save" class="hidden ui-button ui-corner-all ui-widget" style="display: inline-block;"><span class="ui-button-icon ui-icon ui-icon-disk"></span><span class="ui-button-icon-space"> </span>Save New Bookmarks</button>

PS I have been using the last posted version without issue ever since it was posted.

Automatic edit point bookmarks following crop

Active Member

May contain traces of nut

Well-Known Member

May contain traces of nut

Ad detector

Active Member

Active Member

Active Member

Well-Known Member

Active Member

Active Member

May contain traces of nut

Well-Known Member

May contain traces of nut

Well-Known Member

May contain traces of nut

May contain traces of nut

Active Member

May contain traces of nut

Active Member