In soap, byte arrays are encoded as base64 strings so it can look like this:
or with line breaks after each 73 characters, like this:
both options are valid according to the base64 RFC:
Ok so it does not really advocate this... But it is a fact that many soap stacks still use this MIME-originated format and also Wcf supports it.
So what is the problem?
It seems that when Wcf gets a message which contains base64 with CRLF, the processing is slower in a few seconds(!). A drill down shows that the problem is in the DataContract serializer. Take a look at this program:
For those of you who are interested to test this, the files are here and here.
The output is:
This clearly reveals a performance problem.
Why does this happen?
While debugging the .Net source code, I have found this in the XmlBaseReader class (code comments were in the source - they are not mine):
So the data contract serializer tries to read the base64 string, but for some reason succeeds only if the string does not have white spaces inside it (we can further debug to see how that happens but it is exhausting for one post :). The serializer then removes all the white spaces (which requires copying the buffer again) and tries again. This is definitely a performance issue.
Notes:
I have reported this in Microsoft connect, you are welcome to vote this issue up.
Workarounds
There a few workarounds. The trade-offs are generally convenience of API (or "where you prefer to put the 'dirty' stuff").
1. As Valery noticed, you can change the contract to use String instead of byte[]. Then Convert.FromBase64String will give you the byte array.
2. Change your contracts to use the XmlSerializer instead of DataContract serializer. The former does not experience this issue. The XmlSerializer is generally slower (when base64 does not appear that it) so this is what you loose. You get a better API here as clients do not need to manipulate the base64 string.
3. The best of course is to change the service implementation to return base64 without line breaks. Also if large binaries are returned anyway it may be a better idea to employ MTOM.
4. A Wcf custom encoder can strip the spaces from the message before it is deserialized. However this also involves copy of strings and this is beneficial only in rare cases.
13 comments:
Wow
Wow, it's a really deep dive!!!
Thank you!!!
This is really a great analysis!
Thanks a lot for investigating so far this issue and reporting your findings.
You suggest good alternatives to solve the immediate performance issue. But long term, for a company with 100s of developers creating business applications which consume web services, I still hope a fix in WCF to fully benefit the out-of-the-box automatic code generation.
Great job debugging this.
I'm surprised that problem like that have been surviving for a long time. Especially keeping in mind that their workaround is so obviously hacky.
Thanks Man for the great investigations... !! You Rock !!
-Honest Indian
A fix won't be provided by Microsoft before the release of .Net 4.5.
Here is the answer from a Microsoft's Distributed Services Team: "The current release of 4.0 and 3.5 does not include a fix for this issue, and the dev team is looking into this for a future release which would be either 4.5 or post-4.5 release. I don’t have a date for this though."
Thanks Valéry for the follow up!
Has anyone tried the XmlSerializer workaround? For me it didn't work. One megabyte of base64 encoded data takes more than a minute to deserialize anyway.
To assure I did things right: I simply add XmlSerializer as the contents of the tag in the svcmap file.
Sim
When you add the service reference (or use svcutil) you need to specify that the xml serializer should be used.
But isn't it exactly what I do by adding "XmlSerializer as the contents of the Serializer tag in the svcmap file?
Reference.svcmap:
...
XmlSerializer
...
that file is not meant to be hackable so even if it is related there is another way. In particular some attributes need to be added to the proxy.
try to use svcutil with this flag:
svcutil "w.wsdl" /serializer:XmlSerializer
Yaron
I generated the proxy with svcutil and I can see all the XmlSerializer Attributes in the generated code. However it still takes multiple minutes to decode just one megabyte of base64 data. I can see in Fiddler that the reply is already sent and I can see in VS that 'System.Runtime.Serialization.resources.dll' is being loaded just in that moment.
Anyways, we will stick with MTOM soon, when our server will be capable of doing so.
4 Years leater this performance problem is still alive and rocking.
Found the reason thanks to your post. The String-Workaround worked well. A 600kb binary data took ~3min. With the workaround <1sek
Post a Comment