BOMbin’ the L (aka Wierd Errors with XmlWriter)

[Title from a track on Come Find Yourself– Fun Lovin’ Criminals]

Have you experienced strange errors when working with XML generated by an XMLWriter? If so, this post may be helpful to you. I just spent a few hours trying to figure out why I was getting a weird error when I was sending some XML to the Windows Azure Notification Hubs client api (post on that coming up, just as soon as I have my sample working).

There’s a page here Get Started with Notification Hubs with tutorials on using this technology with Windows Store C#, Windows Phone, Android and iOS clients. In the Windows Phone tutorial, it walks through creating a Windows Phone client (needless to say) and a console app which simulates your backend logic that you use to send Tile, Toast or Raw notification messages to the mobile clients. In the instructions for the console app it includes the following:

In your Program class add the following method:

private static async void SendNotificationAsync()
{
    NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString("<connection string with full access>", "<hub name>");
    string toast = "<?xml version="1.0" encoding="utf-8"?>" +
        "<wp:Notification xmlns:wp="WPNotification">" +
           "<wp:Toast>" +
                "<wp:Text1>Hello from a .NET App!</wp:Text1>" +
           "</wp:Toast> " +
        "</wp:Notification>";
    await hub.SendMpnsNativeNotificationAsync(toast);
}

Make sure to insert the name of your hub and the connection string called DefaultFullSharedAccessSignature that you obtained in the section “Configure your Notification Hub.” Note that this is the connection string with Full access, not Listen access.

Then add the following line in your Main method:

SendNotificationAsync();
 Console.ReadLine();

Easy enough! So, using this as my guide, I created my own app that followed these principles, the main difference being that instead of creating a string with the XML payload in it, I thought I’d use an XmlWriter instead, because that’s neater:

        private static string prepareToastPayload(string text1, string text2)
        {
            MemoryStream stream = new MemoryStream();
            XmlWriterSettings settings = new XmlWriterSettings() 
                                            { 
                                                Indent = false,
                                                Encoding = Encoding.UTF8   
                                            };
            XmlWriter writer = XmlWriter.Create(stream, settings);
            writer.WriteStartDocument();
            writer.WriteStartElement("wp", "Notification", "WPNotification");
            writer.WriteStartElement("wp", "Toast", "WPNotification");
            writer.WriteStartElement("wp", "Text1", "WPNotification");
            writer.WriteValue(text1);
            writer.WriteEndElement();
            writer.WriteStartElement("wp", "Text2", "WPNotification");
            writer.WriteValue(text2);
            writer.WriteEndElement();
            writer.WriteEndElement();
            writer.WriteEndDocument();
            writer.Close();

            return Encoding.UTF8.GetString(stream.ToArray());
        }

Wierd exception

When I ran this, I got the following exception:

System.ArgumentException: The payload is not in accepted XML format. The first node should be Tile/Toast. If want to send raw notification, please set “X-NotificationClass” to “3” in header.rn   at Microsoft.ServiceBus.Notifications.RegistrationSDKHelper.DetectMpnsTemplateRegistationType(String body, String errorMsg)rn   at Microsoft.ServiceBus.Notifications.MpnsNotification.OnValidateAndPopulateHeaders()rn   at Microsoft.ServiceBus.Notifications.Notification.ValidateAndPopulateHeaders()rn   at Microsoft.ServiceBus.Notifications.NotificationHubManager.BeginSendNotification(Notification notification, Boolean testSend, AsyncCallback callback, Object state)rn   at Microsoft.ServiceBus.Notifications.NotificationHubManager.<>c__DisplayClass1.<SendNotificationAsync>b__0(AsyncCallback c, Object s)rn   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)rn   at System.Threading.Tasks.TaskF
actory`1.FromAsync(Func`3 beginMethod, Func`2 endMethod, Object state)rn   at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CreateTask[T](Func`3 begin, Func`2 end)… in c:\Development\AzurePushNotificationsDemo\NotificationSenderUtility\NotificationSenderUtility.cs:line 139

Now as it turns out, the error message here is rather misleading. “The first node should be Tile/Toast” seems to be referring to the expected XML format if you are sending toast messages using the (very useful) template capability of Notification Hubs, which allows you to send a generic message to Notification Hubs just once, and it will build the appropriate payloads for your actual clients, whether they are Windows Phone, Windows 8, Android or iOS, all of which have different payloads. But “The payload is not in accepted XML format” is a bit more interesting and forced me to take a look at the XML I was generating.

And here it gets really confusing. I swapped out the XmlWriter code and replaced it with just simply setting a string variable directly, similar to the original tutorial – that works! So what is different about the string generated by my prepareToastpayload method? I then put the two methods one after the other and stored the results in two different string variables which I then examined when stopped at a breakpoint – identical!! So, what gives?

Examining the string bytes

Of course, the strings WEREN’T identical, despite what the Visual Studio debugger was showing me! A bit of a web search led me to this useful post by Jon Skeet, author of the book “C# In Depth”. His post Strings in C# and .NET includes a warning about the Visual Studio debugger not showing you the whole story, and he includes some useful code for dumping the bytes of a string to output. I plugged in the suggested methods and the output revealed the following at the beginning of the XML string I was sending to SendMpnsNativeNotifiactionAsync():

image

Aha! What are these (Possibly non-printable) xFEFF bytes at the start? Next web search was for FEFF and I came upon a few posts explaining that hex FEFF is something called a BOM or Byte Order Mark, which is something that is usually inserted into UTF-8 encoded files that are stored on disk, which XML parsers can look for to figure out how to interpret the file contents. However, if you are formatting XML to send as payload in a web service call as we are here, then you don’t want it. Best post that explain this is a six year old blog by Rick Strahl, XmlWriter, Strings and Byte Order Marks .

Un-BOMbin’ the L

Rick gives the solution as well. It’s actually very difficult to stop the XmlWriter for outputting XML without a BOM, but you can get it to behave by creating a custom UTF-8 encoding where you can specify you don’t want a BOM by passing false into an override of the constructor, and then use that in your XmlWriterSettings object:

        private static string prepareToastPayload(string text1, string text2)
        {
            // Create encoding manually in order to prevent
            // creation of leading BOM (Byte Order Mark) xFEFF at start
            // of string created from the XML
            Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
            MemoryStream stream = new MemoryStream();
            XmlWriterSettings settings = new XmlWriterSettings() 
                                            { 
                                                Indent = false,
                                           //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
                                                Encoding = Utf8,    // Use manually created UTF8 encoding
                                            };
            XmlWriter writer = XmlWriter.Create(stream, settings);
            writer.WriteStartDocument();
            writer.WriteStartElement("wp", "Notification", "WPNotification");
            writer.WriteStartElement("wp", "Toast", "WPNotification");
            writer.WriteStartElement("wp", "Text1", "WPNotification");
            writer.WriteValue(text1);
            writer.WriteEndElement();
            writer.WriteStartElement("wp", "Text2", "WPNotification");
            writer.WriteValue(text2);
            writer.WriteEndElement();
            writer.WriteEndElement();
            writer.WriteEndDocument();
            writer.Close();

            var payload =  Encoding.UTF8.GetString(stream.ToArray());
            DisplayString(payload); // Inspect bytes
            return payload;
        }

Conclusion

So there you have it. Once you know what the problem is – an unwanted BOM – it’s quite easy to find posts that discuss it and give you a solution. But many developers who are having difficulty with XML they have created in code will not know what the problem is they are trying to fix. No amount of studying the code, from inspecting strings in Visual Studio, or quite likely the exception message will tell you that maybe there is a hidden and unwanted character at the start of the generated XML. For you, hopefully this post will help you find a solution.

Leave a Reply

Your email address will not be published. Required fields are marked *