Bug & workaround found in WebIf schedule - missing series events

martxw

Active Member
Hi there...

I've been working on a plug-in on top of the WebIf that includes displaying series events, and been finding some series show only the first event. Examining the raw database view shows that there is actually data in the szEventToRecord and aulEventToRecordInfo columns.
I've debugged it, and found that the field has a sensible size immediately after querying the DB in the rsv.class list proc. But after it's been passed through "new" to create an instance, and gets read again in the rsv method aul, its length appears to be just 1 byte.

As a workaround, I've cloned the code from the rsv method aul to pre-build the aul list prior to creating the new rsv instance. The aul method then just returns this pre-built aul.

The two areas of code I changed are:
Code:
class rsv {
  ulslot      -1
  ersvtype    0
  hsvc      0
  nsttime    0
  szsttime    "00000000000000"
  nduration  0
  erepeat    0
  usevtid    0
  szevtname  {}
  ulPreOffset  0
  ulPostOffset    0
  ulProgramId  0
  ulSeriesId    0
  ucVolume    0
  ucInputMode  0
  usChNum    0
  ucRecKind  0
  ucCRIDType    0
  szCRID      {}
  szFPBRecPath    {}
  szRecordedProgCrid  {}
  szEventToRecord  {}
  aulEventToRecordInfo  {}
  bRecomRsv  0
  usLastRecordedEvtId 0
  eReady      0
  szSvcName  {}
  usLcn    0
  sort      0
  action      0
#BEGIN WORKAROUND
  pre_made_aul {}
#END WORKAROUND
}
 
rsv method aul {} {
  if {![exists -proc binary]} { package require binary }
  set aul {}
  for {set i 0} {$i < [string length $aulEventToRecordInfo]} {incr i 16} {
    binary scan [string range $aulEventToRecordInfo $i $($i + 15)] \
        iiii service start end event_id
    catch {lappend aul [list $service $start $end $event_id]}
  }
#BEGIN WORKAROUND
  if { "$aul" != "$pre_made_aul" } {
    puts "WARNING: $aul != $pre_made_aul"
  }
  return $pre_made_aul
#END WORKAROUND
}

and
Code:
proc {rsv list} {{table tbl_reservation} {extra ""}} {
  set qstring "
    select $table.*,
    channel.TBL_SVC.szSvcName, channel.TBL_SVC.usLcn,
        case when ersvtype > 3 then 1 else 0 end as sort1,
        case when nsttime + nduration < [clock seconds]
      then 0 else 1 end as sort2
    from $table
    left join channel.TBL_SVC
    on $table.hSvc = channel.TBL_SVC.hSvc
  "
  if {$extra ne ""} { append qstring $extra }
  append qstring "
    order by sort1, sort2 desc, nsttime
  "
 
  #puts "QSTRING: ($qstring)"
 
  set res [$::rsvdb query $qstring]
  set records {}
  foreach rec $res {
 
#BEGIN WORKAROUND
    if {![exists -proc binary]} { package require binary }
    set aul {}
    set aulEventToRecordInfo [dict get $rec aulEventToRecordInfo]
    for {set i 0} {$i < [string length $aulEventToRecordInfo]} {incr i 16} {
      binary scan [string range $aulEventToRecordInfo $i $($i + 15)] \
          iiii service start end event_id
      catch {lappend aul [list $service $start $end $event_id]}
    }
    dict set rec pre_made_aul $aul
#END WORKAROUND
    lappend records [rsv new $rec]
  }
 
  return $records
}
Obviously, I'm only leaving the binary to list conversion in twice to provide some logging to highlight the issue.
Going to the WebIf schedule now puts some messy but informative logging messages near the top of the page showing where series events would otherwise have been missing. The table beneath now shows more accurate information. For me, an example of a program that had this problem is Time Team on More 4 Tues 13:30 and onwards. I don't know what causes it - perhaps particular byte sequences??

cheers...
martxw...
 
That's interesting, thanks. I'll have to see if I can work out what the class instantiation is doing to that field.
 
Cheers.
I'd also like to backtrack from suggesting it's a proper workaround, as the insert, backup and restore methods also access aulEventToRecordInfo, and my code would be unhelpful to those. It only helps with list.
 
The class module is written in TCL so it should be possible to debug that - I/we could also log something with the Jim developer who is very responsive.
 
I think I've taken it as far down as I can go, and tracked it to a single line in the oo.tcl constructor proc
Code:
set instvars [dict merge $classvars $instvars]
Printing $classvars and $instvars before this line shows the length of aulEventToRecordInfo is correct before, but afterwards it's sometimes truncated to 1.
Changing the code to save the value of aulEventToRecordInfo before and restoring it afterwards makes it work correctly.
Code:
  # Constructor
  proc "$classname new" {{instvars {}}} {classname classvars} {
    #BEGIN WORKAROUND
    # Saving aulEventToRecordInfo before if it exists
    if {[dict exists $instvars aulEventToRecordInfo]} {
      set saved_aulEventToRecordInfo [dict get $instvars aulEventToRecordInfo]
    }
    #END WORKAROUND
 
    set instvars [dict merge $classvars $instvars]
 
    #BEGIN WORKAROUND
    # Restore aulEventToRecordInfo after if it exists
    if {[dict exists $instvars aulEventToRecordInfo]} {
      if {[string length [dict get $instvars aulEventToRecordInfo]] != [string length $saved_aulEventToRecordInfo]} {
        puts "WORKAROUND"
        dict set instvars aulEventToRecordInfo $saved_aulEventToRecordInfo]
      }
    }
    #END WORKAROUND
 
    # This is the object dispatcher for $classname.
    # Store the classname in both the ref value and tag, for debugging
    # ref tag (for debugging)
    proc [ref $classname $classname "$classname finalize"] {method args} {classname instvars} {
      if {![exists -proc "$classname $method"]} {
        return -code error "$classname, unknown method \"$method\": should be [join [$classname methods] ", "]"
      }
      "$classname $method" {*}$args
    }
  }
The Tcl code looks fine, so the bug must be down in the C++ of Jim itself.
I'm finding the Humax a surprisingly nice development environment for Tcl, but I'm not set up at all to do C++ development, and so I think I'll have to call it a day there. Do you know the Jim C++ better?
Cheers...
 
I've saved hex encoded versions of the binary data strings that both work and don't work, and put together a standalone script that demonstrates the problem.
Code:
#!/mod/bin/jimsh
package require binary
 
proc test_dict_merge { expect_pass_fail hex_value } {
  set binary_value [binary format H* $hex_value]
 
  set length_before [string length $binary_value]
 
  set d1 {}
  dict set d1 value {}
 
  set d2 {}
  dict set d2 value $binary_value
 
  # This is where the bug seems to lie.
  set d3 [dict merge $d1 $d2]
 
  set length_after [string length [dict get $d3 value]]
 
  if { $length_after == $length_before } {
    set pass_fail "PASS"
  } else {
    set pass_fail "FAIL"
  }
  puts "$expect_pass_fail $pass_fail length_before=$length_before length_after=$length_after"
}
 
test_dict_merge FAIL 28000200d8f39552140396524ad400002800020014039652501296524bd400002800020058459752945497525dd4000028000200d896985214a6985270d400002800020014a6985250b5985271d4000028000200588b9c52949a9c52a1e5000028000200949a9c52d0a99c52a2e5000028000200d8dc9d5214ec9d52b4e500002800020014ec9d5250fb9d52b5e50000
test_dict_merge FAIL 3100030068729652c87b9652cd6a0000
test_dict_merge FAIL 78000600505d9652606b965210950000
test_dict_merge FAIL 78000600d07d9b52e08b9b524b8e0000
test_dict_merge FAIL 79000600a8c995528cd69552542e000079000600b8d79552c8e59552662e000079000600281b97520c2897526f2f0000790006003829975248379752fd2f000079000600a86c98528c799852b131000079000600b87a9852c8889852c33100007900060028619c520c6e9c525534000079000600386f9c52487d9c526634000079000600a8b29d528cbf9d522b35000079000600b8c09d52c8ce9d523c350000
 
test_dict_merge PASS 25000200a0e19452a8e8945212e000002500020020339652283a965213e0000025000200a0849752a88b975214e000002500020020d6985228dd985216e0000025000200cc289a52242b9a522ce200002500020054819b52ac839b522de2000025000200a0ca9c52a8d19c522ee2000025000200201c9e5228239e522fe20000
test_dict_merge PASS 27000200606b96529c7a965236c80000
test_dict_merge PASS 2c000200c0e69c5258f49c5296cd0000
test_dict_merge PASS 2e000300d8b59752e0bc97525ac80000
test_dict_merge PASS 310003009447955278549552c8600000
test_dict_merge PASS 43000400a8e89452b0ef9452598f000043000400283a9652304196525c8f000043000400a88b9752b09297525f8f00004300040028dd985230e49852628f000043000400f8299a52242b9a52658f000043000400787b9b52d07d9b52678f000043000400a8d19c52b0d89c52c59700004300040028239e52302a9e52c8970000
test_dict_merge PASS 46000400c8ed9c52d0f49c52aab40000
test_dict_merge PASS 66000500b4c89952bccf9952288d000066000500bccf995298d59952298d00006600050098d59952ccdd99522a8d00006600050018729b5220799b52318d0000
test_dict_merge PASS 7600060040959b5250a39b525c940000
test_dict_merge PASS 7600060054629c5230689c526a940000
 
Superb, thank you. I will have a look at the Jim source code later and see if I can find the problem. If not I'll send your test case onto the developer if that's ok.
 
I'm finding the Humax a surprisingly nice development environment for Tcl,
I find it nice too. I didn't know Tcl (nor Javascript/AJAX come to that) before I started this project but it's come together pretty well. You can still spot the early code in places..
 
You're welcome. I sometimes let my curiosity get the better of me & can't put put things down, but I need to switch over to some Google Maps integration now (nothing to do with Humax).
I sent an email about the add-on I've been writing to 'packages@hummypkg.org.uk', but I'm not sure if it's been seen yet.
 
It turns out that {dict merge} is implemented in Jim - a lot of internal commands are implemented natively. This is the implementation:

Code:
proc {dict merge} {dict args} {
       foreach d $args {
               foreach {k v} $d {
                       dict set dict $k $v
               }
       }
       return $dict
}

If I replace it with this:

Code:
proc {dict merge} {dict args} {
        foreach d $args {
                foreach k [dict keys $d] {
                        dict set dict $k [dict get $d $k]
                }
        }
        return $dict
}

then everything works as expected:

Code:
humax# ./dictmerge
FAIL PASS length_before=144 length_after=144
FAIL PASS length_before=16 length_after=16
FAIL PASS length_before=16 length_after=16
FAIL PASS length_before=16 length_after=16
FAIL PASS length_before=160 length_after=160
PASS PASS length_before=128 length_after=128
PASS PASS length_before=16 length_after=16
PASS PASS length_before=16 length_after=16
PASS PASS length_before=16 length_after=16
PASS PASS length_before=16 length_after=16
PASS PASS length_before=128 length_after=128
PASS PASS length_before=16 length_after=16
PASS PASS length_before=64 length_after=64
PASS PASS length_before=16 length_after=16
PASS PASS length_before=16 length_after=16

so it looks like there is something wrong with the

Code:
foreach {k v} <dict>

construct in these cases.
 
A temporary fix would be to put the new implementation of {dict merge} into the /mod/webif/lib/setup file.
 
webif 1.0.7-8 contains this temporary fix (and I have no more non-linked events on my schedule page : ) ).
I've contacted the Jim developer about the potential {foreach} issue.
 
I already have a test patch for Jim back from the developer! I'll give it a go tomorrow.
 
Back
Top