[Solved] WebIF Bug: Folder Names Containing "}"

[Black Hole]

Due to a typo, I set a recording target folder to "[MP3}" instead of "[MP3]". The recording created the folder (under My Video) and recorded into it, but it doesn't show up in the WebIF media browser so I didn't see it and didn't realise the recording existed. It's only because I was hacking around on the command line I've spotted it!

 

[prpr]

Well it stuffs the Browse Media Files page too, so the answer is found is /tmp/webif-error.log:

/mod/webif/lib/utils:104: Error: extra characters after close-brace
in procedure 'entry' called at file "/mod/webif/html/browse/index.jim", line 370
in procedure '::js::escape' called at file "/mod/webif/html/browse/index.jim", line 118
at file "/mod/webif/lib/utils", line 104

 

[Black Hole]

I take it the } is being treated as a delimiter?

 

[/df]

/mod/webif/lib/utils:104: Error: missing close-brace
in procedure 'entry' called at file "/mod/webif/html/browse/index.jim", line 37
0
in procedure '::js::escape' called at file "/mod/webif/html/browse/index.jim", 
line 118
at file "/mod/webif/lib/utils", line 104

Looks like the escaping is being done for JS but not for Jim.

This patch to /mod/webif/lib/utils:104 allowed me to display the previously invisible folder It has a { there that I had created.

-                [regsub -all -- {[^A-Za-z0-9@*_+./-]+} $str \ 
+                [regsub -all -- {[^A-Za-z0-9@*_+./-]+} [regsub -all -- {[{}]} $str {\\&}] \

However, some further testing is probably indicated in case other TCL special characters need to be handled.

This happens because of a hack (mine?) that escapes the filename for JS by replacing each special character with an expression that escapes the character and then performing a limited evaluation of the resulting string to replace those characters with the escaped value. The string being evaluated has to be valid as Jim TCL. The hack is used because, unlike JS and Python, Jim's regular expression replace command doesn't support specifying a replacing function to be called for each match, but only a replacement string.

 

[prpr]

I take it the } is being treated as a delimiter?

It's effectively a code injection attack. It's really horrid in interpreted languages like Jim, unless you're very careful.

However, some further testing is probably indicated in case other TCL special characters need to be handled.

I found the whole thing very fragile. Hopefully this is better:

proc ::js::escape {str} {
    set re1 {[\;\[\]\{\}]}
    set re2 {[^A-Za-z0-9@*_+./-]+}
    regsub -all -- $re1 $str {\\&} str1
#puts $str1
    regsub -all -- $re2 $str1 \
        {[subst -nobackslashes -novariables \
            [::js::_escape [subst {&}]]]} str2
#puts $str2
    return [subst -nobackslashes -novariables $str2]
#    return [subst -nobackslashes -novariables \
#        [regsub -all -- {[^A-Za-z0-9@*_+./-]+} $str \
#            {[::js::_escape {&}]}]]
}

I haven't been able to break it yet. Do you want to see if you can?

# jimsh
Welcome to Jim version 0.82
. source /mod/webif/lib/utils
jsunescape
. set str Del\{et\}\$str\ ed\ i\[t\;ems\]
Del{et}$str ed i[t;ems]
. jsescape $str
Del%7Bet%7D%24str%20ed%20i%5Bt%3Bems%5D
. jsunescape [jsescape $str]
Del{et}$str ed i[t;ems]
. exit

 

[prpr]

Updated package in the Beta repository.

 

[Black Hole]

Yay!



Now I can delete it!

 

[/df]

Could a literal \ be present in the str passed in here (ie, not one that just appears to escape some other character, like ;)?

... Jim's regular expression replace command doesn't support specifying a replacing function to be called for each match ...

But TCL 8.7 has regsub -command ... which causes the "replacement" to be applied as a proc to a list of the match and submatches. According to Jim TCL issue #170 this is an "inessential" feature.

 

[prpr]

Could a literal \ be present in the str passed in here?

That makes it fail

 

[prpr]

But easily fixed:

humax ~ # jimsh
Welcome to Jim version 0.82
. source /mod/webif/lib/utils
jsunescape
. set str a\\b
a\b
. jsescape $str
a%5Cb
. jsunescape [jsescape $str]
a\b

 

[prpr]

But TCL 8.7 has regsub -command ... which causes the "replacement" to be applied as a proc to a list of the match and submatches. According to Jim TCL issue #170 this is an "inessential" feature.

jim 0.82-3 now in the Beta repository with this feature implemented. Seems to work:

humax# jimsh
Welcome to Jim version 0.82
. source /mod/webif/lib/utils
jsunescape
. proc enc {ch} {format %%%02X [scan $ch %c]}
enc
. set str 12\[\}\]\{\;\\ab
12[}]{;\ab
. regsub -command -all {[^0-9A-Za-z]} $str enc
12%5B%7D%5D%7B%3B%5Cab
. regsub -command -all {[^0-9A-Za-z]} $str { apply {ch {format %%%02X [scan $ch %c]}} }
12%5B%7D%5D%7B%3B%5Cab
. jsescape $str
12%5B%7D%5D%7B%3B%5Cab
. jsunescape 12%5B%7D%5D%7B%3B%5Cab
12[}]{;\ab

 

[prpr]

And this:

proc ::js::escape {str} {
        set re1 {[\;\[\]\{\}\\]}
        set re2 {[^A-Za-z0-9@*_+./-]+}
        regsub -all -- $re1 $str {\\&} str1
        regsub -all -- $re2 $str1 \
                {[subst -nobackslashes -novariables \
                        [::js::_escape [subst {&}]]]} str2
        return [subst -nobackslashes -novariables $str2]
}

proc ::js::unescape {str} {
        return [subst -nobackslashes -novariables \
                [regsub -all -- {%(u[[:xdigit:]]{2})?[[:xdigit:]]{2}} $str \
                        {[::js::_unescape {&}]}]]
}

simplifies considerably to this:

proc ::js::escape {str} {
    regsub -command -all -- {[^A-Za-z0-9@*_+./-]} $str ::js::_escape
}

proc ::js::unescape {str} {
    regsub -command -all -- {%(u[[:xdigit:]]{2})?[[:xdigit:]]{2}} $str ::js::_unescape
}

 

[/df]

Indeed, what I would have hoped for. Generally I'm not in favour of tweaking the language to solve a problem but this is entirely reasonable. No doubt Steve will main-line the feature in due course. Perhaps unsurprisingly, the linked example for TCL8.7 covers a very similar application.