SCART should be no contest for HDMI, although you might come close with RGB and a very good quality (or a very short) lead (unlike a digital interface, the transmission line characteristics of an analogue cable does matter).
As to the appropriate HDMI setting - I imagine the TV is full HD - ie a native 1920x1080 pixel screen? In which case if you output an SD signal from the Humax at 576 the TV will upscale it to 1080, and if you output from the Humax at 1080 the Humax will upscale it, so it depends which upscales it more to your liking.
Not sure what happens if you have the Humax output set to 576 and you get 1080 from an HD channel though - you don't want to be downscaling to 576 over HDMI and then back up to 1080 to fit the display. Personally I would set the output to 1080 and leave it there.