View Issue Details

IDProjectCategoryView StatusLast Update
0001110SOGo Connectorwith external serverpublic2012-11-21 21:17
Reporterfedel Assigned To 
PrioritynormalSeverityfeatureReproducibilityalways
Status closedResolutionfixed 
Product Version3.104 
Summary0001110: Various patches for synchronizing Thunderbird address books with iPhone, iPad etc. and DAViCal
Description

Thought I'd share the results of my Sogo Thunderbird connector debugging exercises. I am running the latest Sogo connector (3.104 nightly 1/30/2011) on Thunderbird 3.1.7 with a DAViCal server version 0.9.9.4 running on Debian. I am trying to synchronize my Thunderbird address books with all my iToys.

With the original Sogo connector from http://inverse.ca/downloads/extensions/nightly/ this did NOT work at all. Calendar syncing worked fine, but address book sync was broken all over the place. After hours of debugging I can now synchronize TB3 with my iToys.

The attached series of patches may be applied against the current Sogo connector nightly (http://inverse.ca/downloads/extensions/nightly/sogo-connector-3.104.xpi at the time of this submission). I'm assuming you know how to patch a Mozilla add-on.

All patches are individually explained in the "Additional Information" field.

Additional Information

(1) sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch

The Sogo connector is unable to receive any updates from compliant DAV servers (except Sogo server :-)) because the Sogo connector expects deprecated DAV response tokens. For more details, see:

http://www.sogo.nu/bugs/bug_view_advanced_page.php?bug_id=857
http://www.sogo.nu/bugs/view.php?id=942
http://wiki.davical.org/w/Configuration/settings/use_old_sync_response_tag

This patch lets the Sogo connector accept "response" tags in addition to "sync-response", thus removes the need for setting "use_old_sync_response_tag = true" in DAViCal.


(2) sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch

Apple uses an optional grouping feature defined in RFC2426, chapter 3.1.2. For example, iOS sends content lines prefixed with "item1." as in

item1.ADR;type=WORK;type=pref:;;1 Infinite Loop;Cupertino;CA;95014;USA

Unfortunately, the current Sogo connector does not understand this syntax. It only tries matching parameter names against the entire first field until ; and ignores any VCARD records with unknown parameter names.

Now, in real life this means that any contacts touched on an Apple toy will cause TB contacts without any addresses. Next time such a contact is modified in TB, this contact's address information will be gone forever. Sigh!

This patch lets the Sogo connector ignore the group prefix and luckily import addresses with or without group prefix.


(3) sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch

Apple seems to incorrectly encode multi-line address values using line-feeds (instead of using COMMAs as defined in RFC2426). This patch makes the Sogo connector accept such misformed VCARDs by translating line feeds within address fields into COMMAs. Please note that this patch eventually concatenates multi-line addresses into comma separated single-line address fields which is not perfect but still better than loosing precious address data. It would have been better to cleanly fix Apple's VCARD writer but that seems out of reach.


(4) sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch

The Sogo connector does not seem to escape values correctly when encoding VCARDs (see RFC2426). For example, COMMAs are escaped twice, some line breaks are not escaped at all (e.g. in notes), commas in values allowing for comma separated lists (e.g. in ADR, NICKNAME...) were still escaped and so on. This patch tries correcting the escaping logic in the Sogo connector.


Instead of individually applying separate patches, you may also apply all patches at once by using sogo-connector-3.104-nightly-20110130-all.patches.

TagsNo tags attached.

Relationships

related to 0001098 resolvedwsourdeau GroupDavSynchronizer: obsolete DAV:sync-response after DAV:sync-collection request? 

Activities

2011-01-31 16:04

 

2011-01-31 16:05

 

sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch (1,442 bytes)   
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js
--- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js	Sun Jan 30 07:44:44 2011
+++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js	Mon Jan 31 13:11:45 2011
@@ -814,6 +814,15 @@
                 this.newWebdavSyncToken
                     = jsonResponse["multistatus"][0]["sync-token"][0];
                 let responses = jsonResponse["multistatus"][0]["sync-response"];
+                /* WebDAV Sync Draft 3+ propose regular "DAV:response" instead of 
+                   "DAV:sync-response" in response to "DAV:sync-collection" */
+                if (typeof responses == "undefined") {
+                    responses = jsonResponse["multistatus"][0]["response"];
+                    for each (let response in responses) {
+                        if (typeof response["status"] == "undefined")
+                            response["status"] = ["HTTP/1.1 200 OK"];
+                    }
+                }
                 for each (let response in responses) {
                     let href = response["href"][0];
                     let keyArray = href.split("/");

2011-01-31 16:05

 

sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch (7,047 bytes)   
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js
--- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js	Sun Jan 30 07:44:44 2011
+++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js	Mon Jan 31 15:23:20 2011
@@ -38,14 +38,19 @@
 jsInclude(["chrome://inverse-library/content/uuid.js",
            "chrome://inverse-library/content/quoted-printable.js"]);
 
+/* Individually escape COMMA separated components of value list */
+function escapedForCardList(theString) {
+    let theList = theString.split(",");
+    for (let k in theList)
+        theList[k] = escapedForCard(theList[k]);
+    return theList.join(",");
+}
+
 function escapedForCard(theString) {
     theString = theString.replace(/\\/g, "\\\\");
     theString = theString.replace(/,/g, "\\,");
     theString = theString.replace(/;/g, "\\;");
-    theString = theString.replace(/,/g, "\\,");
-
-    // theString.replace(/\n/g, "\\n,");
-    // theString.replace(/\r/g, "\\r,");
+    theString = theString.replace(/\n|\r\n/g, "\\n");
 
     return theString;
 }
@@ -72,17 +77,6 @@
     return multiValue.split("\u001A");
 }
 
-function unescapedFromCard(theString) {
-    theString = theString.replace(/\\/g, "\\");
-    theString = theString.replace(/\,/g, ",");
-    theString = theString.replace(/\;/g, ";");
-    theString = theString.replace(/\,/g, ",");
-    // theString.replace(/\\n/g, "\n,");
-    // theString.replace(/\\r/g, "\r,");
-
-    return theString;
-}
-
 /* this method parses a versit directory:
  - normalizing the charset and encodings;
  - returning the lines as hashes filled with the tag, parameters and values
@@ -408,33 +405,7 @@
         props["PreferMailFormat"] = value;
     },
     categories: function(props, parameters, values) {
-        let commaValues = values[0];
-        let newValues = [];
-        if (commaValues.length > 0) {
-            let escaped = false;
-            let currentValue = "";
-            for (let i = 0; i < commaValues.length; i++) {
-                if (escaped) {
-                    currentValue += commaValues[i];
-                    escaped = false;
-                }
-                else {
-                    if (commaValues[i] == "\\") {
-                        escaped = true;
-                    }
-                    else if (commaValues[i] == ",") {
-                        newValues.push(currentValue);
-                        currentValue = "";
-                    }
-                    else {
-                        currentValue += commaValues[i];
-                    }
-                }
-            }
-            newValues.push(currentValue);
-        }
-        props["Categories"] = newValues.join("\u001A");
-        // props["Categories"] = arrayToMultiValue(values);
+        props["Categories"] = values[0].split(",").join("\u001A");
     },
     note: function(props, parameters, values) {
         props["Notes"] = values.join(";");
@@ -496,6 +467,11 @@
         }
     };
 
+    /* Remove address grouping, if any, for known tags */
+    let tagparts = tag.split(".");
+    if ((tagparts.length == 2) && (typeof _insertCardMethods[tagparts[1]] != "undefined"))
+        tag = tagparts[1];
+
     if (typeof _insertCardMethods[tag] != "undefined")
         _insertCardMethods[tag](properties, parameters, values);
     else
@@ -614,8 +590,8 @@
     let lastName = card.getProperty("LastName", "");
     let firstName = card.getProperty("FirstName", "");
     if (lastName.length || firstName.length)
-        vCard += foldedLine("N:" + escapedForCard(lastName)
-                            + ";" + escapedForCard(firstName)) + "\r\n";
+        vCard += foldedLine("N:" + escapedForCardList(lastName)
+                            + ";" + escapedForCardList(firstName)) + "\r\n";
 
     let displayName = card.getProperty("DisplayName", "");
     if (displayName.length)
@@ -629,11 +605,11 @@
 
     let nickName = card.getProperty("NickName", "");
     if (nickName.length)
-        vCard += foldedLine("NICKNAME:" + escapedForCard(nickName)) + "\r\n";
+        vCard += foldedLine("NICKNAME:" + escapedForCardList(nickName)) + "\r\n";
 
     let categories = card.getProperty("Categories", "");
     if (categories.length)
-        vCard += foldedLine("CATEGORIES:" + categories.split("\u001A").join(",")) + "\r\n";
+        vCard += foldedLine("CATEGORIES:" + escapedForCardList(categories.split("\u001A").join(","))) + "\r\n";
 
     let workAddress = card.getProperty("WorkAddress", "");
     let workAddress2 = card.getProperty("WorkAddress2", "");
@@ -643,8 +619,9 @@
     let workCountry = card.getProperty("WorkCountry", "");
     if ((workAddress + workAddress2 + workCity + workState + workZipCode
          + workCountry).length)
-        vCard += foldedLine("ADR;TYPE=work:;" + escapedForCard(workAddress2)
-                            + ";" + escapedForCard(workAddress)
+        vCard += foldedLine("ADR;TYPE=work:" 
+                            + ";" + escapedForCardList(workAddress2)
+                            + ";" + escapedForCardList(workAddress)
                             + ";" + escapedForCard(workCity)
                             + ";" + escapedForCard(workState)
                             + ";" + escapedForCard(workZipCode)
@@ -658,8 +635,9 @@
     let homeCountry = card.getProperty("HomeCountry", "");
     if ((homeAddress + homeAddress2 + homeCity + homeState + homeZipCode
          + homeCountry).length)
-        vCard += foldedLine("ADR;TYPE=home:;" + escapedForCard(homeAddress2)
-                            + ";" + escapedForCard(homeAddress)
+        vCard += foldedLine("ADR;TYPE=home:"
+                            + ";" + escapedForCardList(homeAddress2)
+                            + ";" + escapedForCardList(homeAddress)
                             + ";" + escapedForCard(homeCity)
                             + ";" + escapedForCard(homeState)
                             + ";" + escapedForCard(homeZipCode)
@@ -722,7 +700,7 @@
 
     let jobTitle = card.getProperty("JobTitle", "");
     if (jobTitle.length)
-        vCard += foldedLine("TITLE:" + jobTitle) + "\r\n";
+        vCard += foldedLine("TITLE:" + escapedForCard(jobTitle)) + "\r\n";
 
     let birthYear = card.getProperty("BirthYear", 0);
     let birthMonth = card.getProperty("BirthMonth", 0);
@@ -742,7 +720,7 @@
     let notes = card.getProperty("Notes", "");
     if (notes.length) {
         vCard += foldedLine("NOTE:"
-                            + escapedForCard(notes.replace(/\n/g, "\r\n"))) + "\r\n";
+                            + escapedForCard(notes)) + "\r\n";
     }
 
     let aimScreenName = card.getProperty("_AimScreenName", "");

2011-01-31 16:05

 

sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch (943 bytes)   
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js
--- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js	Sun Jan 30 07:44:44 2011
+++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js	Mon Jan 31 11:52:10 2011
@@ -344,6 +338,9 @@
     },
     adr: function(props, parameters, values) {
         let types = this._upperTypes(parameters["type"]);
+        /* Concat multi-line(feed) address field with commas (quirk for iOS) */
+        values[1] = values[1].split("\n").join(", ");
+        values[2] = values[2].split("\n").join(", ");
         if (types.indexOf("WORK") > -1) {
             props.extend({ "WorkAddress2": values[1],
                            "WorkAddress": values[2],

2011-01-31 16:06

 

sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch (858 bytes)   
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js
--- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js	Sun Jan 30 07:44:44 2011
+++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js	Mon Jan 31 10:40:28 2011
@@ -496,6 +467,11 @@
         }
     };
 
+    /* Remove address grouping, if any, for known tags */
+    let tagparts = tag.split(".");
+    if ((tagparts.length == 2) && (typeof _insertCardMethods[tagparts[1]] != "undefined"))
+        tag = tagparts[1];
+
     if (typeof _insertCardMethods[tag] != "undefined")
         _insertCardMethods[tag](properties, parameters, values);
     else
tigono

tigono

2011-03-08 19:29

reporter   ~0002223

I am not familiar with patching, but the patches are that one i need to use my iphone address book.
Is there a place where to get the patched version or when will it be merged to the main sogo connector?

wsourdeau

wsourdeau

2011-03-08 20:16

viewer   ~0002227

Hi Tigono,

Don't worry. I am currently having a look at this bugreport. Most of the code should be integrated in the next nightlies...

wsourdeau

wsourdeau

2011-03-09 20:18

viewer   ~0002239

Issues 2 and 4 above are fixed. Please try the nightlies. The issue with webdav sync will be fixed later. I need yet to double-check issue 3.

Thanks for the patches!

wsourdeau

wsourdeau

2011-03-09 20:20

viewer   ~0002240

Further on webdav-sync: since it requires changes to the SOGo server, it's postponed as I want to do it cleanly. Thanks nonetheless.

wsourdeau

wsourdeau

2011-03-09 20:51

viewer   ~0002241

Issue 3 has been patched as well. So this bugreport is mostly fixed.
Tigono: if you wish, please use the new nightlies that I will regenerate in a few minutes...

tigono

tigono

2011-03-10 07:41

reporter   ~0002243

I tested (2) and what I saw it is fixed, this helps a lot :-)

But there is another new behavior in concern to Thunderbird (only using thunderbird, sogo connector and in this case apple address book server).
(1) Adding a contact works well!
(2) Deleting a contact works well!

=> Changing some properties of the contact does not update the contact anymore!
I have to retest this, but I think it worked with pre4?! I don't know if this
is in fact of the changes made above and I will make more tests in the
afternoon. Only give you a short feedback fast ...

tigono

tigono

2011-03-11 04:50

reporter   ~0002248

After making some additional tests I see that changing contacts is not possible anymore with Thunderbird3.19, SoGo Connector Nightly Build and Apple Addressbook Server.

The Server gives the following error message:

2011-03-11 05:45:31+0100 [-] [carddav-8802] [-] [twistedcaldav.method.put_addressbook_common#error] Invalid vcard data: More than one VERSION property in component <Component: "<VCARD| [<VERSION{}3.0>, <VERSION{}3.0>, <PRODID{}-//Inverse inc.//SOGo Connector 1.0//EN>, <PRODID{}-//Inverse inc.//SOGo Connector 1.0//EN>, <UID{}C4B05D06-42E0-0001-A6CB-1900166319E7>, <ADR{u'TYPE': [u'work']}Ich weiss nicht 10\nStadt, 33055>, <ADR{u'TYPE': [u'home']}Das ist eine Strasse 44\n, >, <FN{}Dieter Wudel>, <N{} Dieter Wudel >, <X-MOZILLA-HTML{}FALSE>]>">
2011-03-11 05:45:31+0100 [-] [carddav-8801] [PooledMemCacheProtocol,client] [twistedcaldav.method.put_addressbook_common#error] Cannot overwrite vcard resource C4B0A621-5660-0001-B45B-66216B351A4C.vcf with different UID C4B0A621-56A0-0001-95D3-1E6F30406570

Please let me know if I should open a new bug?

wsourdeau

wsourdeau

2011-03-11 19:12

viewer   ~0002251

Hi Tigono,

No it's fine. I'll have a look right away. Thanks for reporting this!

wsourdeau

wsourdeau

2011-03-11 20:29

viewer   ~0002256

Hi Tigono,

That bug is now fixed. Please try again with the latest nightlies:

http://inverse.ca/downloads/extensions/nightly/

tigono

tigono

2011-03-12 06:45

reporter   ~0002257

Last edited: 2011-04-11 12:35

Hi, the first error message is gone but the second remains ->

[2011-03-12 07:30:22+0100 [-] [carddav-8801] [PooledMemCacheProtocol,client] [twistedcaldav.method.put_addressbook_common#error] Cannot overwrite vcard resource C4B0A621-5660-0001-B45B-66216B351A4C.vcf with different UID C4B0A621-56A0-0001-95D3-1E6F30406570

AND: It works like a charme in the pre4 version (but not with my iphone:-)

I hope this has no influence: the thunderbird version with pre4 is on a windows pc and the test producing the error is on a mac (only for information)

REMARK:
This has no influence, I tested it with Sogo Connector 3.105. Creating contacts in thunderbird works, but updating does not work anymore; no influence of the OS of the client found.

ludovic

ludovic

2012-11-21 21:17

administrator   ~0004964

From reading the comments, this bug seems fixed.

Issue History

Date Modified Username Field Change
2011-01-31 16:04 fedel New Issue
2011-01-31 16:04 fedel File Added: sogo-connector-3.104-nightly-20110130-all.patches
2011-01-31 16:05 fedel File Added: sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch
2011-01-31 16:05 fedel File Added: sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch
2011-01-31 16:05 fedel File Added: sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch
2011-01-31 16:06 fedel File Added: sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch
2011-03-08 19:29 tigono Note Added: 0002223
2011-03-08 20:16 wsourdeau Note Added: 0002227
2011-03-09 20:18 wsourdeau Note Added: 0002239
2011-03-09 20:19 wsourdeau Relationship added related to 0001098
2011-03-09 20:20 wsourdeau Note Added: 0002240
2011-03-09 20:51 wsourdeau Note Added: 0002241
2011-03-10 07:41 tigono Note Added: 0002243
2011-03-11 04:50 tigono Note Added: 0002248
2011-03-11 19:12 wsourdeau Note Added: 0002251
2011-03-11 20:29 wsourdeau Note Added: 0002256
2011-03-12 06:45 tigono Note Added: 0002257
2011-03-12 06:47 tigono Note Edited: 0002257
2011-04-11 12:35 tigono Note Edited: 0002257
2012-11-21 21:17 ludovic Note Added: 0004964
2012-11-21 21:17 ludovic Status new => closed
2012-11-21 21:17 ludovic Resolution open => fixed