Is the recording you used for this test viewable? I would expect this one, and any others decrypted this way before you set the custom key, to be unviewableupdate 2:
I just experimented and manually 'decrypted' a file using Opt+ on webif.
it doesn't play, instead it says 'The channel is scrambled or not available'
But the original file will have been stored in the _original folder so you can recover it easily.