RS emails - spurious characters

peterworks

peterworks

Ye Olde Bowler
I received the attached RS email alert today and noticed that there are some spurious characters where the apostrophe should be.
I use Thunderbird and know that it is having problems with emails from Yahoo (see: https://forums.yahoo.net/t5/Sending...g-8bit-capability-incorrectly-and/td-p/449275).
I am uncertain whether it is Thunderbird being overly fussy with character translation or if it is the SMTP server configuration.
Is anyone else seeing this ?
Capture01-03-2018-18.09.34.jpg
 
I don't have that service in my set. I'll PM my address and you can forward the email to me for inspection.
 
Here's how it looks using webmail in Chrome/Win7:

grab1.png

This is iOS:

grab2.png

The actual raw text looks like this:

Code: 
International Open from Blackpool’s Newton
Hall.

..or in Hex:

grab3.png

Over to af123!
 
I tried the following:
Reset Auto to get the emails again
Set Thunderbird mail.strictly_mime to true (as suggested in the forums.yahoo.net link above)
Unfortunately it made no difference
 
I have little doubt this is caused by a close-quote symbol (seems more likely than an apostrophe) not being passed through to the output because it isn't supported in the relevant encoding. It has already been mentioned that the EPG system uses some weird encoding that is different from encodings used elsewhere in the Freeview broadcast system, so if the close-quote symbol in the EPG encoding doesn't match the default encoding used for email content delivery or web representation it would need to be picked up in the input data and converted to the required output representation.
 
I have done some analysis on this (just to see if I can prove where the problem is, or isn't!).

Looking at the raw EPG data, the character causing the trouble is a Right single quotation mark which has the Unicode designation U+2019. This is correctly UTF-8 encoded in the data stream and turns into "e2 80 99".
This has then been UTF-8 encoded again (by something for unknown reasons) to turn it into "c3 a2 c2 80 c2 99". At this point something else has interpreted this as a Windows-1252 codepage sequence - that maps 80 to U+20AC (Euro symbol); 99 to U+2122 (TM symbol); all others remaining as they are, so we get "c3 a2 c2 20 ac c2 21 22". It should have been UTF-8 decoded at this point to get back to "e2 80 99".

This is then UTF-8 encoded yet again (probably by the forward to BH!) and ends up as "c3 83 c2 a2 c3 82 e2 82 ac c3 82 e2 84 a2" which is what was reported above.

I set up an RS email notification to see what I received. I can confirm that Gmail receives the correct UTF-8 sequence "e2 80 99" as in the original data and it is not being munged by the RS server. So the problem above must be being caused by the seemingly ignorant people who run Yahoo's mail system.
See below.
 
Last edited:
Thanks for you analysis prpr however these messages do not 'touch' Yahoo's mail system.
Here is the message source:
Code: 
From - Fri Mar  2 09:58:17 2018
X-Account-Key: account1
X-UIDL: UID6650-1422463305
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-path: <prvs=159902c340=noreply@hpkg.tv>
Envelope-to: humax@xxxxxxxxxx.com
Delivery-date: Fri, 02 Mar 2018 08:38:46 +0000
Received: from fallback.citrus-it.net ([89.248.55.126]:56926)
   by hulk.freezone.co.uk with esmtps (TLSv1.2:ECDHE-RSA-AES256-SHA:256)
   (Exim 4.87)
   (envelope-from <prvs=159902c340=noreply@hpkg.tv>)
   id 1ergDF-001Ont-UL
   for humax@xxxxxxxxxx.com; Fri, 02 Mar 2018 08:38:45 +0000
Received: from mail.citrus-it.net (mail.citrus-it.net [89.248.55.103])
   by fallback.citrus-it.net with ESMTPS id w228c2lI019527
   (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=OK)
   for <humax@xxxxxxxxxx.com>; Fri, 2 Mar 2018 08:38:04 GMT
Authentication-Results: citrusmail; dkim=pass header.i=@hpkg.tv
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hpkg.tv; s=base;
   t=1519978607; bh=U9Xmc9mHK2IEvs6oKUeLE20Ai3mHse1nXAVnblZURiA=;
   h=Date:Message-Id:From:To:Subject:MIME-Version:Content-Type;
   b=krvzo9ufOgamI174W15JWToDDOuFXJjZ00FliAs3+URIqbnBtEt328DC8txxQ1s9O
    Dknu3z4yty8nwim2P1NAc9LwiRG99dMbAjt2fyE8rWiC3GL2rOkhz8Bg2+XAVv+/uU
    tmotQE9kHQvmDwFUS+ITGyd8C3QQslf8g/k++HQA=
Received: from mail.citrus-it.net (mail.citrus-it.net [89.248.55.103])
   by mail.citrus-it.net with ESMTP id w228Glri017236
   for <humax@xxxxxxxxxx.com>; Fri, 2 Mar 2018 08:16:47 GMT
X-Citrus-Remote-IP: 89.248.55.84
Date: Fri, 2 Mar 2018 08:16:47 GMT
Message-Id: <201803020816.w228GkUg017233@mail.citrus-it.net>
From: "Hummypkg Remote Scheduling" <noreply@hpkg.tv>
To: "Hummypkg User" <humax@xxxxxxxxxx.com>
Subject: RS Auto / HDR-Fox T2 (!Humax Two - dc:d3:21:81:8e:75)
MIME-Version: 1.0
Content-Type: multipart/alternative;
   boundary="----_=_17f150c033b7f01fffc9a428eb58448e"
X-NS: 15ebeeb87da3a4c14b1ca37c0ffaf6bd
X-Citrus-Version: 5.2.4757
X-Citrus-ID: 5w228Glri017236
X-Citrus-Spam: Whitelisted
X-Citrus-Trust: Trusted
X-Antivirus: Avast (VPS 180302-0, 02/03/2018), Inbound message
X-Antivirus-Status: Clean

This is a multi-part message in MIME format.

------_=_17f150c033b7f01fffc9a428eb58448e
Content-Type: text/plain; charset="ISO-8859-15"
Content-Transfer-Encoding: 8bit

FreeSports [Fri, 02 Mar 2018 09:00:00 +0000 / 01:00]
  Test Match Bowls
  England v Wales: Top bowlers from England and Wales go head
  to head in triples, pairs and singles matches over a best of
  11 game Test Match format.
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

Channel 4 HD [Sun, 04 Mar 2018 14:25:00 +0000 / 00:30]
  The Simpsons
  Hello Gutter, Hello Fadder: Homer bowls a perfect 300 and
  becomes a celebrity, even appearing on the Springfield
  Squares with Bumble Bee Man, Sideshow Mel and his old
  nemesis Ron Howard.  [AD,S]
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

FreeSports [Wed, 07 Mar 2018 13:00:00 +0000 / 05:00]
  Live: Bowls: International Open
  Coverage of round two of one of the biggest events on the
  Bowls World Tour, the Co-op Funeralcare International Open
  from Blackpool’s Newton Hall.
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

FreeSports [Thu, 08 Mar 2018 08:00:00 +0000 / 01:00]
  Test Match Bowls
  England v Wales: Top bowlers from England and Wales go head
  to head in triples, pairs and singles matches over a best of
  11 game Test Match format.
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

FreeSports [Thu, 08 Mar 2018 10:00:00 +0000 / 01:30]
  Live: Bowls: International Open
  Quarter-final coverage of one of the biggest events on the
  Bowls World Tour, the Co-op Funeralcare International Open
  from Blackpool’s Newton Hall.
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

FreeSports [Thu, 08 Mar 2018 13:00:00 +0000 / 05:00]
  Live: Bowls: International Open
  Quarter-final coverage of one of the biggest events on the
  Bowls World Tour, the Co-op Funeralcare International Open
  from Blackpool’s Newton Hall.
    === Matched email alert (bowls / bowls) ===
   http://hpkg.tv/a14595

Drama [Fri, 09 Mar 2018 22:00:00 +0000 / 02:30]
  Waking the Dead
  Blind Beggar: Trevor Eve and Sue Johnston star in another
  compelling forensics mystery. A skeleton found in a crypt
  awakens secrets that force Boyd to look into his own past.
  [S]
    === Matched email alert (waking the dead / ) ===
   http://hpkg.tv/a14717



--
This message is sent from an unmonitored account and replies will not
be read. For help and support, please join the community at
http://hummy.tv/ and search the forums to see if your question has been
asked before. If not, then post a new thread in the customised firmware
section and someone will be along to help you.

------_=_17f150c033b7f01fffc9a428eb58448e
Content-Type: text/html


<html>
<head>
<title>RS Auto Scheduler</title>
<style>
.event { margin: 1em 0; }
.date { font-style: italic; font-size: 0.9em; }
.channel, .title { font-size: 1.1em; font-weight: bold; }
</style>
</head>
<body>
   
<div class=event>
<div class=channel>FreeSports</div>
<div class=date>
[Fri, 02 Mar 2018 09:00:00 +0000 / 01:00]
</div>
<div class=title><a href="http://hpkg.tv/xTest+Match+Bowls">
Test Match Bowls</a></div>
<div class=epg>England v Wales: Top bowlers from England and Wales go head to head in triples, pairs and singles matches over a best of 11 game Test Match format.</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>Channel 4 HD</div>
<div class=date>
[Sun, 04 Mar 2018 14:25:00 +0000 / 00:30]
</div>
<div class=title><a href="http://hpkg.tv/xThe+Simpsons">
The Simpsons</a></div>
<div class=epg>Hello Gutter, Hello Fadder: Homer bowls a perfect 300 and becomes a celebrity, even appearing on the Springfield Squares with Bumble Bee Man, Sideshow Mel and his old nemesis Ron Howard.  [AD,S]</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>FreeSports</div>
<div class=date>
[Wed, 07 Mar 2018 13:00:00 +0000 / 05:00]
</div>
<div class=title><a href="http://hpkg.tv/xLive%3A+Bowls%3A+International+Open">
Live: Bowls: International Open</a></div>
<div class=epg>Coverage of round two of one of the biggest events on the Bowls World Tour, the Co-op Funeralcare International Open from Blackpool’s Newton Hall.</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>FreeSports</div>
<div class=date>
[Thu, 08 Mar 2018 08:00:00 +0000 / 01:00]
</div>
<div class=title><a href="http://hpkg.tv/xTest+Match+Bowls">
Test Match Bowls</a></div>
<div class=epg>England v Wales: Top bowlers from England and Wales go head to head in triples, pairs and singles matches over a best of 11 game Test Match format.</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>FreeSports</div>
<div class=date>
[Thu, 08 Mar 2018 10:00:00 +0000 / 01:30]
</div>
<div class=title><a href="http://hpkg.tv/xLive%3A+Bowls%3A+International+Open">
Live: Bowls: International Open</a></div>
<div class=epg>Quarter-final coverage of one of the biggest events on the Bowls World Tour, the Co-op Funeralcare International Open from Blackpool’s Newton Hall.</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>FreeSports</div>
<div class=date>
[Thu, 08 Mar 2018 13:00:00 +0000 / 05:00]
</div>
<div class=title><a href="http://hpkg.tv/xLive%3A+Bowls%3A+International+Open">
Live: Bowls: International Open</a></div>
<div class=epg>Quarter-final coverage of one of the biggest events on the Bowls World Tour, the Co-op Funeralcare International Open from Blackpool’s Newton Hall.</div>
<div class=match>
<a href=http://hpkg.tv/a14595>= Matched email alert (bowls / bowls) =</a>
</div></div>
<div class=event>
<div class=channel>Drama</div>
<div class=date>
[Fri, 09 Mar 2018 22:00:00 +0000 / 02:30]
</div>
<div class=title><a href="http://hpkg.tv/xWaking+the+Dead">
Waking the Dead</a></div>
<div class=epg>Blind Beggar: Trevor Eve and Sue Johnston star in another compelling forensics mystery. A skeleton found in a crypt awakens secrets that force Boyd to look into his own past. [S]</div>
<div class=match>
<a href=http://hpkg.tv/a14717>= Matched email alert (waking the dead / ) =</a>
</div></div></body></html>
<br><br>
--
<p>
This message is sent from an unmonitored account and replies will not
be read. For help and support, please join the community at
<a href=http://hummy.tv/>hummy.tv</a> and search the forums to see if your
question has been asked before. If not, then post a new thread in the
customised firmware section and someone will be along to help you.


------_=_17f150c033b7f01fffc9a428eb58448e--

My incoming mail is set to Western (ISO-8859-1). I have recently started using Avast so whether that might have an impact I am not sure.
I have changed my incoming mail to Unicode (UTF-8) and reset auto to resend the alerts. I suspect there is a good chance that this will temporarily solve the problem.
Tomorrow I will turn off Avast email checking to see if this could be affecting it.
 
peterworks said:
Thanks for you analysis prpr however these messages do not 'touch' Yahoo's mail system.
Click to expand...
Oh, OK. You confused me by the two references in post #1 then.
In any case, I checked Yahoo's mail servers and the "8 BITMIME" / "8BITMIME" problem in the link you quoted seems to have been solved.
Looking closer at the email:
Code: 
Content-Type: text/plain; charset="ISO-8859-15"
Content-Transfer-Encoding: 8bit

Content-Type: text/html
it seems wrong to me. It says the plain text encoding is 8859-15 yet it's sending UTF-8. And the html doesn't specify anything and again is sending UTF-8, so perhaps your client just displays whatever default it's set to - I don't use Thunderbird. Either way, it explains the results you're seeing.
It does look like RS is at fault then - twice. Back to af123.
 
Nice analysis prpr. If they stuck to the proper apostrophe instead of the fashionable close-quote there wouldn't be a problem! (Although, I admit, I also substitute the close-quote in word processing because the apostrophe glyph rarely looks right.)
prpr said:
So the problem above must be being caused by the seemingly ignorant people who run Yahoo's mail system.
Click to expand...
peterworks said:
Thanks for you analysis prpr however these messages do not 'touch' Yahoo's mail system.
Click to expand...
I may have added to the confusion here because I do indeed use BT Yahoo!. I would quite like to move to an email system under my control, but it needs to be done right, once, and then I have the problem of migrating all my correspondents no matter how infrequently they contact me.
 
prpr said:
It probably is worth changing your incoming setting to UTF-8 (even if temporarily), just to see whether it solves the problem.
Click to expand...
Change to UTF-8 worked - no 'funny' characters instead of the apostrophe (or whatever it is). Have reverted to Western (ISO-8859-1) as, apparently, UTF-8 will not be able to display all characters and, from what I have read, can make emails much bigger (though please don't ask me why).
One other thing I noticed is that the RS email uses ISO-8859-15. Could this perhaps make a difference as some characters were changed when '15' was introduced ?
 
Black Hole said:
I would quite like to move to an email system under my control, but it needs to be done right, once, and then I have the problem of migrating all my correspondents no matter how infrequently they contact me.
Click to expand...
It is fairly easy to do using Thunderbird - your biggest problem would be the migration from Yahoo of all your emails. I have done it for a friend who had nearly 5Gb of messages and it took me 36 hours on fibre. If I remember correctly you have a slow/iffy connection so could take a very long time...
 
peterworks said:
Change to UTF-8 worked - no 'funny' characters instead of the apostrophe (or whatever it is). Have reverted to Western (ISO-8859-1) as, apparently, UTF-8 will not be able to display all characters
Click to expand...
You have that the wrong way round: ISO 8859-1 can only display a limited set of characters; UTF-8 can display anything.
and, from what I have read, can make emails much bigger (though please don't ask me why).
Click to expand...
I think that emails will only be bigger where they are using non ASCII characters.
 
Last edited:
peterworks said:
I will set incoming mails to Unicode (UTF-8) and see how it goes...
This should also solve the problem of spurious characters...
Click to expand...
Character set encoding is a nightmare but it will get easier as there are now sensible standards (eg UTF-8) that are slowly taking over. No single setting will, at this point in time, cope with all eventualities.

I first battled with character sets in the the late 1980s when I was providing computer facilities for the World Veterans Orienteering Championship in Scotland. The event had a substantial entry from around the world with a lot of Eastern Europeans with all sorts of diacritical marks in their names. This was long before UTF-8 was invented and the only solution that the printers I was using supported was Windows code pages. No single code page covered everything I needed so each competitor record had to include the code page used to encode the name. Not pretty but it worked. More recently I have been trying to understand the current options to display diacritical marks and I am using a mix of UTF-16 and UTF-8 for what I need to do.
 
You must log in or register to reply here.
Back
Top