Thursday, April 30, 2015

Running Terminal Dashboards on Windows


With blessed-contrib it is easy to create terminal dashboards using ascii-art:

blessed-contrib uses Braille fonts which are available by default on Linux and Mac distributions but not on Windows:

In order to run such dashboards on windows you need to perform the following steps:

1. Download, open and install the FreeMono font.
2. Follow these instructions in order to to configure FreeMono in the windows terminal.
3. In the command line type: "set TERM=windows-ansi".
4. Clone & Run the blessed-contrib project according the regular guidelines (or run any other dashboard that you need)

Enjoy :)


What's next? get this blog rss updates or register for mail updates!

Wednesday, January 14, 2015

Building Terminal Dashboards Using Ascii/Ansi Art


Developers are most productive when working at the command prompt. A lot of applications have command line utilities to facilitate that. However those cli's restrict themselves to textual based data. When it is time to show any kind of graphics, a browser or desktop applications are preferred. For me this is a huge miss of the terminal full potential. Having a native terminal way to create graphics would allow developers keeping the same aesthetics across the board, rather than switching back and forth to the browser experience. It would also open opportunities for better ssh experience and utilities. And as it is always available a command line away, it also provides a better experience for any quick visualization we require (think dashboards, monitoring etc.)

This is why I have built a Node.JS library to build dashboards using ascii/ansi art. I am using blessed, which is an NCurses-like javascript library, and drawille, which provides an HTML5 canvas abstraction over the terminal. The full details are in the github repository.

Check it out on github.


What's next? get this blog rss updates or register for mail updates!

Thursday, June 12, 2014

eMedNy Web Services in .NET (Guest Post)


This is a guest post by Isaac Kleinman about web services interoperability with eMedNy

I was recently tasked with building a SOAP Client to consume some services provided by eMedNy, New York State Dept. of Health's electronic Medicaid system. While eMedNy provides a number of web services for providers such as medical and prescription history etc., my project focuses on their subscriber (patient) eligibility service. Once you manage to successfully communicate with the service, the actual exchange consists of submitting an X12-formatted 270 file which is an eligibility request and receiving a 271 (eligibility response) in return.

Working with this arcane format is challenging in its own right and is more than worthy of its own post. However, configuring the certificates and structuring the SOAP security message headers proved to be even more challenging. I still haven't completely gotten my mind around the concept, but apparently the service was written in Java and uses WS-Security to define its security policy. It's possible that building a client in Java is a smooth process, but doing so in .NET proved to be a quite complicated ordeal. While I can't fully explain all the details of the issues at hand, I can, at least, describe the steps I took to get my project up and running. Yaron Naveh was extremely helpful at each step of the way; hat-tip to him.

Officially, Microsofts's `svcutil` utility was supposed to do all the magic for me: just provide the WSDL URL and the appropriate proxy and config files get generated and you're good to go. This was far from the case. Here's what happened when I ran `svcutil`:

The generated `output.config` file contains a similar error:

Now, I haven't even gotten clarity yet as to what exactly this error message means or what is causing it, but as you can see, due to the error, not much configuration is happening here at all. Thus, I don't include this file in my project.

As an aside, the proxy file, `MHService.cs`, contains a whopping 55K loc. Most of this is not relevant to the services I'm using. (I suspect I only need about 30 of those lines, but I haven't gotten to sifting it yet.)

Let's begin with the steps toward putting together a functional outbound SOAP request message.

Here's a sample request message provided by eMedNy:

While it's true that a message doesn't have to conform entirely to this sample, it gives us something to work towards.Here is an outline of the architecture we can derive from this sample:

  • A random temporary session key is generated.
  • Message body is encrypted (Triple-Des CBC) with this key.
  • The temporary key gets encrypted using eMedNy's server certificate.
  • The encrypted key is included in the request header.
  • (Encrypted) Message body gets canonicalized (C14N).
  • Digest (SHA1) is produced from canonicalized message.
  • A digest (SHA1) of all digests (only message body here) is produced.
  • This final digest is encrypted with client's private key.
  • The hash and signature are included in the request header.
  • Username token: includes `username`, `password`, `nonce` and `timestamp` (I've found that the request is accpeted just fine even without the `nonce` and `timestamp`)

  • So here's a first draft of my code which generates a working request (based on Yaron's gist):

    And here what the generated message looks like:

    As you can see, the message I generate differs slightly from the sample they provide.
    In particular, my version does not have:

    1. the `nonce` and `timestamp`
    2. the server certificate ( it only gets referenced by `Subject Key Identifier`

    but.. it works.

    Now, let's proceed to dealing with the response message.

    If you try running your program using the above code, you'll get the following error:

    The issue here is that eMedNy's server is presenting a certificate which is not valid according to the client's trust chain. Yaron addresses this issue here. You incorporate his code as follows:

    1. add the following callback to the program:
    static bool OnValidationCallback( object sender, X509Certificate cert,
    X509Chain chain, SslPolicyErrors errors)
    return true;

    2. and, in your configuration code, perform the following assignment:
    ServicePointManager.ServerCertificateValidationCallback = new
    After making that change, running the program should give you this error:

    Unhandled Exception: System.ServiceModel.Security.MessageSecurityException: The incoming message was signed with a token which was different from what used to encrypt the body. This was not expected.

    For some reason, WCF is not properly identifying the server certificate token. Hard as I tried, I have not (yet) been able to figure out how to tweak the configuration to overcome this issue. As a last resort, I had no choice but to roll my own custom encoder. The code is based on the examples provided on MSDN, but I've tried to remove a lot of the parts that are not needed for our case, so that it's somewhat more obvious what the code does.

    So here is the CustomTextMessageBindingElement class:

    here is the CustomTextMessageEncoderFactory class:

    and, finally, here is the actual CustomTextMessageEncoder class. The `ReadMessage` method is where the decryption takes place. I intend to make the method a bit neater using the `EncryptedXml` class, but here it is for now:

    At this point, we can go back to the configuration code and replace the use of the default encoder:

    binding.Elements.Add(new TextMessageEncodingBindingElement(MessageVersion.Soap11, Encoding.UTF8));

    with our custom encoder as follows:

    binding.Elements.Add(new CustomTextMessageBindingElement());

    As a relative n00b, getting this to work was quite challenging, so I hope this can help some others in the same situation.


    What's next? get this blog rss updates or register for mail updates!

    Friday, May 23, 2014

    Gifw00t! A pure javascript web recorder


    Recently I took a break from all the angle brackets I typically deal with. I decided to build a web recorder in pure javascript (demo, github). You can now record your interaction with a web page into an animated gif without installing anything native. This method works in devices as well. One immediate use case that comes to mind is instant replay of HTML5 game moves:

    (live demo)

    There are tons of other usages as well.

    The recorder internally relies on libraries such as html2canvas by @Niklasvh and jsgif by @antimatter15, and the demos use excelent html5 games written by @daleharvey. In the next few days I'll publish some cool usage samples in my twitter. I can't promise I'll switch to curly braces for good but this was a fun experiment!


    What's next? get this blog rss updates or register for mail updates!

    Friday, January 24, 2014

    More EBS-EDT errors


    In my recent post I have discussed some of the basic errors you may get when trying to consume EBS-EDT with WCF. Here are some more errors you should be aware of:

    Error 1:

    policy rejected

    If the server returns the above error it typically means that you have issues with some values of elements of your request. Either you are using an older WSDL version, or you have given wrong values in some of the fields (e.g. auditID). This might also be related to using a mix of authenticaiton formats (MSA and IDP) instead of just using one of them and setting the other to null.

    Error 2:

    The algorithm '' is not accepted for operation 'Digest' by algorithm suite Basic128Sha256Rsa15


    The algorithm '' is not accepted for operation 'Digest' by algorithm suite Basic128Rsa15

    This error is thrown by WCF when it tries to validate the response signature. This happens because the signature uses a mix of SHA1 and SHA256 hash algorithms. There is nothing you can do to make WCF accept this. What you should do is implement a custom encoder (which you probably do anyway if you have read my last post) and in the encoder validate the signature by yourself and then remove it from the SOAP.
    EDIT: Dwayne McKnight has commented that there is a way around a custom encoder.


    What's next? get this blog rss updates or register for mail updates!

    Wednesday, December 18, 2013

    Consuming EBS-EDT SOAP service from WCF


    If you want personal guidance with EBS-EDT feel free to mail me at

    A while ago the Ontario Ministry of Health and Long-Term Care published this document, which explains how to consume their new SOAP web service. (In favor of Google the exact title is "Technical Specification for Medical Claims Electronic Data Transfer (MCEDT) Service via Electronic Business Services (EBS) Ministry of Health and Long-Term Care"). I have received over a dozen of questions about how to consume this service with WCF. Unfortunately it is not a simple task since the service uses a complex configuration which is not available in any of the built-in WCF bindings. However it is possible to do it with some custom code. Bellow I describe the general scheme for this to work. I know some community members are preparing a simple wrapper for this so I will publish it here once ready.

    The Errors
    Depending on which path you chose for implementation, the most common error message you are likely to receive is the dreadful:

    The incoming message was signed with a token which was different from what used to encrypt the body. This was not expected.

    There are other possible errors as well or some consumers may not know where to start.

    The Solution
    1. Since the client needs to send both username token and an X.509 certificate (and sign with the latter) we need to write a code binding:

    One thing you want to notice in this code is that it contains the username and password, so change them according to your credentails.
    Another thing to notice is that the client certificate is loaded from disk. You could change that to the windows certificate store if you wish. As for the server certificate, you could put any dummy certificate there, including the same one as the client certificate (it will not be used but WCF needs something in this setting).
    Also note the EnableUnsecuredResponse=true. It is a key for the next steps.

    2. Since the request needs to be signed only (not encrypted) let's configure the contract in reference.cs with the ProtectionLevel attribute:

    3. WCF is reluctant to decrypt the response. For this reason we need to do the decryption manually. This is the hardest part but I give most of the code here so hopefully it will be easier. You need to implement a custom message encoder and configure the binding above to use your encoder instead of text message encoder. Read here on how to implement an encoder.

    4. You need to override the ReadMessage method of the encoder and decrypt the response message in it.

    This code shows how to decrypt a message (not necessarily in the context of an encoder):

    This code needs access to your private key so it could extract the session key in the message and it also needs some elements from the response. Once you get the decypted message you can replace the encypted body part in the message provided by the encoder with the decrypted message.

    5. The last mission to accomplish in the encoder is to delete the <security> element (and all of its child nodes) from the response message before you return it to WCF. Otherwise WCF will try to decrypt the message which is redundant since we just unencrypted it now (WCF decryption would fail anyway). Remember the EnableUnsecuredResponse flag from step #2? It tells WCF not to expect any security, so stripping the elements out is safe.

    Information on some possible errors in this process is available here.

    MIME Attachments

    Hopefully by now you have a working client. Some of the operations also receive an attachment from the service. This attachment in SwA (Soap with Attachments) which is a MIME format a little different than the MTOM whcih WCF knows about.To extract this attachment you could use some kind of a mime parser library as the first step of your encoder (apply it over the raw bytes from the network). Copy the first MIME part to the Message object (this is the SOAP). The second part will be the attachment which you can keep on the custom encoder as a property or on some other context available to your application code.

    Fault Contract
    Since there is no formal fault contract in the WSDL you should inspect any incoming soap fault using a custom message inspector.

    To sum up, consuming EBS-EDT from WCF is not easy but doable, good luck!


    What's next? get this blog rss updates or register for mail updates!

    Tuesday, June 25, 2013

    Validating Windows Mobile App Store Receipts Using Node.js


    When your windows phone user performs an in-app purchase you can get its receipt using the API:

    You need to send this receipt to your backend system to validate the signature. If that backend system happens to be in C# your life is easy as the official documentation provides exact validation instructions. If you use another platform then be aware that there are a few gotchas to this validation process.

    Based on several requests I have checked the feasibility to use my xml-crytpo node.js module to perform this validation. There were two challenges that required to patch xml-crypto. Both of them caused the signature digest calculation to differ from the stated one by the store. This had failed the validation process.

    White spaces
    As you can see in the sample receipt above it contains white spaces. White spaces in Xml are meaningful and once an Xml has been signed it is not legal to remove or alter the white spaces. However take a look at these lines from Microsoft C# validation instructions:

    The receipt is loaded into an XmlDocument. Since not defined otherwise, the PreserveWhitespace property defaults to false. This means that the actual validaiton code later on is performed not on the actuall Xml received from the network/disk but on an altered version of it which strips all white space. This is not standard and confusing. Anyway if you use node.js remember to first remove the whitespace before processing the document further. Unfortunately xmldom, the module which xml-crypto uses for xml processing, does not provide this utility. I did a quick patch for it and for now have put it in my private xmldom repo. Just initialize the parser with the ignoreWhiteSpace flag:

    Debugging this was quite painfull which is why I posted this tweet shortly after:

    Implicit Exclusive Xml Canonicalization
    Canonicalization is probably one of the most confusing topics regarding xml digital signature. In a nutshell, before processing Xml (either when signing it or when validating the signature) we need to transform it to a canonical form in terms of attribute order, namespace definition, whitespace and etc. There are multiple standard to do so and they can be chained together so the signed xml should state which standard(s) it uses:

    This is the transformation stated by the windows store receipt

    In practice when trying to validate according to this transformation xml-crypto shows this validation error:

    I have built a simple C# app to do this validation according to the Microsoft sample - it worked. The sample uses the high level SignedXml class. I have then built a C# snippet to do the validation in a more low level way by applying the enveloped-signature transformation myself - this failed.

    I had to dig deep in the reflector to find that internally the SignedXml .net class always applies the Exclusive Xml Canonicalization standard in addition to any explicitly defined transformation. This happens in TransformChain.TransformToOctetStream() method which is used internally in the signing process when it starts with SignedXml:

    The CanonicalXml class has an internal property m_c14nDoc which holds the canonicalized version.

    Once I have forced xml-crypto to use c14n at all times the validation was successful.

    I have not found any evidence in the xml digital signature or exclusive xml canonicalization specs that the way SignedXml works complies with the definitions.

    All Together Now
    I have committed the changes to the xml-crytpo windows-store branch and have not checked them in to npm since this is not necessarily the correct behavior with other platforms. So you should use xml-crypto from there (a quick way is to "npm install xml-crypto" and then override the created folder with the windows-store branch zip. There is a way to tell npm to install directly from github but not sure if this can happen from a branch. This is the usage example:

    The pem file contains the raw content of the certificate which you get from the windows store provided url:

    I hope your app will be successful and you will use this procedure a lot...


    What's next? get this blog rss updates or register for mail updates!