View Issue Details

IDProjectCategoryView StatusLast Update
0002177SOGoWeb Mailpublic2013-01-30 20:50
Reporterwaschmittel Assigned Tofrancis  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Product Version2.0.3a 
Fixed in Version2.0.4b 
Summary0002177: Folder not showing when mail subject contains unicode SUB character
Description

A folder's contents are not displayed in the web interface when it contains an e-mail whose subject contains a unicode SUB character.
Amazon did this to me, so this may not be as unusual as it sounds.

The folders work correctly from Thunderbird. So is must be a SOGo problem.

I have attached a UTF-8 text file that contains such a subject for copy-pasting.

Additional Information

This is perfectly reproducible with any browser I have. Whenever I create a draft in the drafts folder, for example with a subject that contains SUB, the drafts folder appears to be empty.

I am using the current version from http://inverse.ca precise/precise.

TagsNo tags attached.

Relationships

duplicate of 0002079 resolvedfrancis The email tab is empty if a specific email is in INBOX 

Activities

2013-01-12 23:57

 

example_subject_for_copy_paste.txt (23 bytes)   
Bestellung Cast Marble
waschmittel

waschmittel

2013-01-13 00:03

reporter   ~0005163

I just checked: Amazon did this in every delivery notification they sent me in december. So this is really serious for me.

francis

francis

2013-01-14 15:35

administrator   ~0005165

Can you attach the full message source? Thanks.

2013-01-14 16:16

 

38 (378 bytes)   
User-Agent: SOGoMail 2.0.3
X-Forward: 87.139.142.241
MIME-Version: 1.0
from: "Daniel Flassak" <daniel@flubba.de>
subject: =?utf-8?q?Bestellung_=1ACast_Marble?=
message-id: <5995-50f42e80-33b-593b2600@232652330>
to: test@test.com
content-type: text/plain; charset="utf-8"
date: Mon, 14 Jan 2013 17:11:52 +0100
content-length: 6
content-transfer-encoding: 7bit

foobar
38 (378 bytes)   
waschmittel

waschmittel

2013-01-14 16:17

reporter   ~0005166

I have created a message with the web interface and saved it in the drafts folder. Now my draft folder appears to be empty.

I have attached the source as it is accessible in thunderbird and as it resides in /var/spool/cyrus/mail/d/user/daniel/Drafts.

waschmittel

waschmittel

2013-01-14 16:20

reporter   ~0005167

By the way: The XHR response in the web interface is not empty. But somehow, it does not display:

[["To", "Attachment", "Flagged", "Subject", "From", "Unread", "Priority", "Date", "Size", "rowClasses", "labels", "rowID", "uid"], ["test@test.com", "", "<img src=\"\/SOGo.woa\/WebServerResources\/dot.png?lm=1357682453\">", "<span>Bestellung Cast Marble<\/span>", "Daniel Flassak", "<img src=\"\/SOGo.woa\/WebServerResources\/dot.png?lm=1357682453\" class=\"mailerReadIcon\" title=\"Als ungelesen markieren\" title-markread=\"Als gelesen markieren\" title-markunread=\"Als ungelesen markieren\" id=\"readdiv_38\"\/>", "", "Montag, 14. Januar 2013 17:11 CET", "0.4 KB", "mailer_readmailsubject", "", "row_38", 38]]

bernhard

bernhard

2013-01-14 18:29

reporter   ~0005168

I have the same problem since December.
Please see my attached amazon-mail in Bug 2079.

My system is: Ubuntu 12.04
dovecot 2.0.19
sogo 2.0.3a-1

Best regards,
Bernhard

ludovic

ludovic

2013-01-14 20:19

administrator   ~0005172

I don't have an issue with the mail above (38).

2013-01-15 08:57

 

badmail_amazon2.eml (41,051 bytes)
bernhard

bernhard

2013-01-15 09:02

reporter   ~0005178

I have uploaded another mail with this problem: badmail_amazon2.eml
The bug is easy reproducibly at my machine.

I'm guessing: It has something to do with dovecot or with German settings.

What can i do (special logging?) to provide more informations?

waschmittel

waschmittel

2013-01-15 12:18

reporter   ~0005183

I don't think it has something to do with German settings. If I change the language to English, I have the same problem.

Also, I am using cyrus and not dovecot.

What can I do to provide more information? Since I only have my private server, I could give one of you admins an account so you can take a look at it from the web frontend side.
I think the problem is on the web frontend side, because the mail list XHR response comes through, but is not displayed.

The SUB character is not escaped in any way in the XHR response. I think it's supposed to be JSON.
Here's the relevant part:
3], ["foo@test.test", "", "<img src=\"\/SOGo.woa\/WebServerResources\/dot.png?lm=1357682453\">", "<span>Bestellung Cast Marble<\/span>",

Just in case the Bugtracker software filters it: It's "Bestellung <SUB>Cast Marble"

In correct JSON encoding, it should be:
"Bestellung \u001aCast Marble"

Maybe that's the problem.

waschmittel

waschmittel

2013-01-15 12:31

reporter   ~0005184

Last edited: 2013-01-15 12:37

I just checked by inserting a Unicode RLO mark for changing the writing direction. It is escaped properly:
from the response from
https://*/SOGo/so/daniel/Mail//0/folderINBOX/folderDrafts/unseenCount

So i guess that the list of unicode characters that should be escaped id is incomplete and missing SUB.

PS: The bugtracker seems to have a problem with this post. When I edit it, I edit backwards text instead of seeing the escaped text. Funny how UTF8 is confusing in so many places. :-)

I have added the response's text as an attachment. I can't insert it correctly anymore.

2013-01-15 12:38

 

correct response example with RLO mark.txt (38 bytes)   
<span>&#8238;Daniel Flassak<\/span>
ludovic

ludovic

2013-01-15 13:29

administrator   ~0005186

Just tried badmail_amazon2.eml and it also works for me.

ludovic

ludovic

2013-01-15 13:37

administrator   ~0005187

If you compiled things from source, have a look at:

SOPE/sope-json/SBJson/Classes/SBJsonWriter.m - more specifically, the -appendString:into: method.

You could also install the debug symbols and set a breakpoint and see if it encodes the output properly.

bernhard

bernhard

2013-01-15 15:01

reporter   ~0005190

I downloaded the file 38 as 38.eml and could reproduce the problem of waschmittel on my machine.

waschmittel

waschmittel

2013-01-15 17:16

reporter   ~0005194

Sorry, I have not compiled from source. I am using the official Ubuntu repository on 12.04.

waschmittel

waschmittel

2013-01-17 12:28

reporter   ~0005208

BTW, the output is definitely not encoded properly. It's probably not in the range of kEscapeChars.

SUB is U+001A according to http://en.wikipedia.org/wiki/List_of_Unicode_characters

I don't think I understand the code, because I am not even familiar with the language.

I have now reproduced this with Firefox 18, IE 9 and Chrome 24.

Christian Mack

Christian Mack

2013-01-17 13:17

developer   ~0005211

That is a nonprintable control character.

It should never be part of a subject or body in the first place.
But if it is, then SOGo should ignore it.

waschmittel

waschmittel

2013-01-17 13:50

reporter   ~0005213

I agree that it should not be part of a subject or body. Unfortunately, Amazon has been sending me several of these mails. And I can't make them stop.

Additionaly, anyone knowing this could mailciously break my webmailer.

herve

herve

2013-01-17 18:00

reporter   ~0005221

I have the same problem with this character in the subject
U+001C File Separator FS

Thanks.

francis

francis

2013-01-30 20:50

administrator   ~0005288

Fixed. Please try the next nightly build tomorrow.

See https://github.com/inverse-inc/sogo/commit/0408415c61f572ab859c58f44c1d5a028b3fbef7

Issue History

Date Modified Username Field Change
2013-01-12 23:57 waschmittel New Issue
2013-01-12 23:57 waschmittel File Added: example_subject_for_copy_paste.txt
2013-01-13 00:03 waschmittel Note Added: 0005163
2013-01-14 15:35 francis Note Added: 0005165
2013-01-14 16:16 waschmittel File Added: 38
2013-01-14 16:17 waschmittel Note Added: 0005166
2013-01-14 16:20 waschmittel Note Added: 0005167
2013-01-14 18:29 bernhard Note Added: 0005168
2013-01-14 20:19 ludovic Note Added: 0005172
2013-01-15 08:57 bernhard File Added: badmail_amazon2.eml
2013-01-15 09:02 bernhard Note Added: 0005178
2013-01-15 12:18 waschmittel Note Added: 0005183
2013-01-15 12:31 waschmittel Note Added: 0005184
2013-01-15 12:33 waschmittel Note Edited: 0005184
2013-01-15 12:34 waschmittel Note Edited: 0005184
2013-01-15 12:35 waschmittel Note Edited: 0005184
2013-01-15 12:37 waschmittel Note Edited: 0005184
2013-01-15 12:38 waschmittel File Added: correct response example with RLO mark.txt
2013-01-15 13:29 ludovic Note Added: 0005186
2013-01-15 13:37 ludovic Note Added: 0005187
2013-01-15 15:01 bernhard Note Added: 0005190
2013-01-15 17:16 waschmittel Note Added: 0005194
2013-01-17 12:28 waschmittel Note Added: 0005208
2013-01-17 13:17 Christian Mack Note Added: 0005211
2013-01-17 13:50 waschmittel Note Added: 0005213
2013-01-17 18:00 herve Note Added: 0005221
2013-01-30 20:46 francis Relationship added duplicate of 0002079
2013-01-30 20:50 francis Note Added: 0005288
2013-01-30 20:50 francis Status new => resolved
2013-01-30 20:50 francis Fixed in Version => 2.0.5
2013-01-30 20:50 francis Resolution open => fixed
2013-01-30 20:50 francis Assigned To => francis