One Half Done

After a week of coding, i finally have a working MTOM unwrapper. Getting this was relatively easy, and involved parsing the message for the boundary, seperating the message parts, recombining them and encoding the binary octets as base64. See the code below if you are interested. Included is also a simple wrapper for the SOAP4r Libary so I can use it through rails.
http://ross-assoc.net/ruby4mtom.rb

Next up is figuring out a way to hook the MTOM handler into the generation of the SOAP message. The main thing to know is which parts are defined/restricted to base64 or hex, otherwise it is very hard to tell what needs to be seperated out into parts and un-encoded.

Any thoughts or bugs are much appreciated.

More Thoughts

Spend some of today doing more research on the feasibility of implementing MTOM in ruby using the rails framework. As expected, there really isn't any action happening with MTOM outside of the Java and .NET worlds. But I did find a lot of interesting analysis of the MTOM spec that should be a big help. First off, this article was quite helpful:

http://www.devx.com/xml/Article/34797

I was originally pretty worried that MTOM includes some weird formatting of the HTTP header, which I wouldn't have access to using the rails framework, but except for the content-type (which one can get access to through the @headers instance variable in the ActionController class), the HTTP header is essentially the same. FYI - for setting a unique HTTP content type in rails, check out this article. That means that the transformation occurs on the HTTP body only.

Basically, there are four things to set/read in a SOAP message to indicate that MTOM has been used:

1. The application/xop+xml media is present
2. The media type of the HTTP message is multipart/related.
3. The media type of the root part of the MIME Multipart/Related package is application/xop+xml.
4. The start-info parameter indicates a content type of application/soap+xml.

The other really interesting thing I found out is that MTOM can be implemented on top of SOAP 1.1. These guys at IBM came up with a white paper on how to indicate an MTOM'd message using SOAP 1.1. I haven't had a chance to read it in depth, but it looks promising (and gives me an excuse to drop the SOAP1.2 part of this project :).

One more thing I'm noticing is that there aren't any MTOM test tools or validators out there. That is going to make testing any implementation I do in ruby pretty difficult (unless I want to set up a Java endpoint myself!) If anyone out there is interested in coding a basic Java/.NET MTOM endpoint, shoot me an email: mjreich at gmail dot com.

Direction

Life got crazy (moved to a new house), and I haven't been able to spend as much time on this as I would like. So, here's a dump of where I'm thinking for this project.

As I get into this more and more, it seems like the SOAP1.2 part of this project is pretty ambitious. Given the NaHi (the main developer of SOAP4R) hasn't touched SOAP 1.2 yet, I'm skeptical that I'll be able to integrate anything nicely into his library that will handle the new spec. That said, for my project, I've been able to generate stub code from the SOAP4R library, which means that we aren't doing anything that is technically new in SOAP1.2. Additionally, I'm really only interested in the MTOM attachment mechanism (now that DIME is officially DOA), not the down and dirty details of SOAP. This all leads me to think that the value added here would be to create a wrapper for the SOAP4R library that runs a transform on the incoming and outgoing messages and:

1. Changes the namespaces to indicate SOAP1.2 messages and trigger the correct parser on the receiving end.
2. Makes any other changes to a SOAP 1.1 message to make it semantically acceptable to a SOAP 1.2 parser.
3. Extracts the Base64Encoded message parts, optimizes them and creates the correct envelope structure (the actual implementation of the MTOM specification).

Again, from the specification, I'm not sure what the functional relationship between MTOM and SOAP1.2 is, other than they were both released at the same time. Given that MTOM and XOP operate outside of the SOAP message spec, and don't alter the non Base64Encoded parts in any appreciable way, making a message completely compliant with the SOAP1.2 spec may not be necessary at all (which would reduce the amount of work on my end by an order of magnitude.)

Stay tuned: next up is creating the wrapper class and trying to clean this up so that it might be useful to someone on another project.

HTTP header content type

One thing I ran into problems with when testing was that Mongrel was returning a HTTP content type of application/xml rather than text/xml. After a little searching online, I found an easy (and pretty darn useful) fix. In the 'environment.rb' file in the config folder is a section for defining MIME types that returned through the rails 'respond_to' method. I simply added this line of code and got the result I was looking for:

Mime::Type.register "text/xml", :xml

Hope this helps!

One more thing: boot.rb

After doing some testing and getting this up on my hosted server, I found one problem that others have run into. When using SOAP4r as a gem, you may get an error along the lines of 'uninitialized constant SOAP::Mapping::EncodedRegistry(nameError)'. For some reason, the gem load order is wonky, so there is an easy fix involving boot.rb. Check out the link for more info:

http://dev.ctor.org/soap4r/wiki/RailsAndSoap4R

Ruby on Rails + SOAP4R server

And I thought it would be hard! After scouring the web and finding nothing on how to integrate the SOAP4r stub code onto rails (to create a SOAP server on top of the rails framework), I finally figured out a way to do it. The trick was in the server/servant files that are generated by wsdl2ruby; by diving down into the class definitions I was able to pull out the atomic components of the servant and stick them in a rails project. Here's the breakdown:

First, I created an empty rails project using the scaffold generator. I set it up with a base rails class that will act as a router to grab the post data and pass it through to the SOAP router and parser. What we are interested in is the 'create' method in the applicationController that rails generates (which using the REST architecture is where the HTTP POST command is captured.) I'll walk through the raw code first, but keep in mind that I plan to wrap all of this into a helper class for cleanliness.

def create
  @router = ::SOAP::RPC::Router.new('rubynode')
  servant = NetworkNodePortType2.new
  res = ''
  req = request

@router is the SOAP4R router that takes an input post dump and parses it using the type definitions created by WSDL2RUBY. The 'servant' is an instance of the basic class generated by the WSDL2RUBY script. The servant class contains the application logic for generating the return data from the parameters in the request message (this class is dummy generated by the WSDL2RUBY script, and must be filled in; check out the doc/literal example in the soap4r install folder for the specific example.) The 'res' variable will be used to hold the SOAP response message, but for now is nilled out, and the 'req' variable is a copy of the rails generated 'request' data structure that contains the raw POST dump and other HTTP header info needed later on.

  NetworkNodePortType2::Methods.each do definitions
    opt = definitions.last
    if opt[:request_style] == :document
      @router.add_document_operation(servant, *definitions)
    else
      @router.add_rpc_operation(servant, *definitions)
    end
  end

The above section of code is the intialization stub taken from the generated server app class. This simply initializes the operation definitions from the WSDL and binds them to the servant methods through the router. That is, the 'definitions' include the input and output message types and bindings for each operation defined in the WSDL. This code sets up the path to the application logic (in the servant class) when a defined message is recieved.

  conn_data = ::SOAP::StreamHandler::ConnectionData.new
  setup_req(conn_data, req)
  @router.external_ces = nil
  conn_data = @router.route(conn_data)
  input = setup_res(conn_data, req, res)

  respond_to do |format|
    format.xml { render :xml => input }
  end
end

def setup_req(conn_data, req)
  conn_data.receive_string = req.raw_post
  conn_data.receive_contenttype = req.content_type
  conn_data.soapaction = nil
end

def setup_res(conn_data, req, res)
  res = conn_data.send_string
end

Here's the meat of the code. First, the 'conn_data' variable contains the streamhandler connection data type, which is used to house the raw POST dump and the HTTP header info. Next, we map the 'request' information into the 'conn_data' structure and then pass it into the router. We then grab the response from the conn_data structure and pass it to rails XML handler to be returned.

And there you go: a decent first shot at integrating the SOAP4R library/generated stub code into a working Ruby on Rails web service implementation. Incidentally, a wrapper class using these basic tools would be a perfect place to put the MTOM hooks: all that would need to be done is grab the raw SOAP response message and run a transform on the 'base64encoded' parts.

Rails + SOAP4R != Fun

The next step of this project (now that I have figured out how to bind to the WSDL) is to put up a test service implementation of the WSDL and then validate it using either a SOAP 1.2 message validator, or barring that, a test client that a colleague is developing in .NET.

Consuming web services through rails is easy.

I'm learning that serving anything through rails is hard.

I may be able to use ActionWebService (I've heard from some that this uses soap4r behind the scenes), but my guess is not. So, next up is to figure out how to bind the soap4r servant stub to Rails - possibly through the native rest interface. SOAP operates over HTTP using the POST method, so if I can find a way to trigger the SOAP4R servant stub and pass it the input, and trap/send the output, this may work. Possibly through routes.rb.

Two Steps Back, Three Steps Forward

Alrighty, well I had a change of plans. After thinking about this for a while and doing some research, the wsdl2ruby tool finally generates some code. I ended up changing the WSDL in a couple of places to get it to generate code successfully:

1. Changed xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" namespace decleration to xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
   
2. Changed corresponding 'soap12:' tags to 'soap:'

So here's the updated WSDL. Aside from the SOAP12 changes, I had to add back in some fault messages that I'd removed in error, ooops. Turns out that was generating a lot of the errors.

Now it is clear that we aren't doing much in the WSDL that isn't supported by SOAP4r - or at least enough to generate the WSDL. So, the real questions is: how much has the functional SOAP layer changed, and how much of a pain is it going to be to alter the existing soap4r code to add these things? At this point, I'm not too worried about making this packageable, or fully implementing the spec: just what I would need to add in order for MTOM to work and allow me to bind to our .NET node running the WSDL.

One outstanding question: will the standalone server allow more than one operation in the portType def?

Diving Deeper

I've been diving into the SOAP4R code a little to get a lay of the land and figure out where any MTOM hooks would be possible. BTW, if you haven't checked out the ruby-debug gem, I highly recommend it. Here's a short screencast on basic useage (if you are familiar with gdb, then this is cake).

After running through the code a few times with the debugger, I have a basic sense as to what the wsdl2ruby.rb file is doing (and where it's generating errors). I've been using this document/literal WSDL which includes the SOAP12 schema (just to try out). Given that SOAP4R doesn't have support for SOAP1.2, I didn't really expect this to work - and it doesn't. Keeps bombing out when it is building the definitions:

ignored element: {http://schemas.xmlsoap.org/wsdl/soap12/}binding
ignored element: {http://schemas.xmlsoap.org/wsdl/soap12/}operation
ignored element: {http://schemas.xmlsoap.org/wsdl/soap12/}body
ignored element: {http://schemas.xmlsoap.org/wsdl/soap12/}fault
ignored element: {http://schemas.xmlsoap.org/wsdl/soap12/}address

Also, wsdl2ruby doesn't seem to like it when the service and definition elements are named the same: it is using the names of the elements to name the files, and runs into a collision when it creates the 'service' library because the definitions library (of the same name) has already been created. I'd be curious if there is anything in the WS-I profile (or WSDL spec for that matter) that prohibits naming the defs and service the same.

The other interesting tidbit is that if I replace the SOAP12 namespace to SOAP and rename the definitions element so that it does not conflict, wsdl2ruby binds fine and generates all the correct code without errors! So, I'm at least working with a WSDL that is bindable, aside from some minor details. However, I would like to keep the SOAP12 refs in there, as MTOM is technically only supported by SOAP12 (although I don't know of any technical issue in the MTOM/XOP spec that would preclude using it with SOAP11).

So, here's my overview of the WSDL2ruby.rb file (the version in lib\ruby\1.8\wsdl\soap, rather than the wrapper version included in the soap4r distribution. I'm running soap4r 1.5.6.

This file does two main things: 1) parses the WSDL using the XSD::XMLparser engine and, 2) dumps the class definitions, type mappings, and client/server stub code (including the driver and servant) into files. The WSDL parsing is the interesting part, and where it keeps barfing. Specifically, when trying to import the 'binding' element, it compares the imported element name with the the DefinitionsName variable and throws an unknown element error (line 115 of lib/ruby/1.8/wsdl/parser.rb). The value of DefinitionsName is 'definitions' which is obviously not 'binding'. Next up is figuring out where DefinitionsName comes from, or is defined, and what it means.

Kickoff

Welcome all! This blog will document my efforts to create/extend MTOM (Message Transmission and Optimization Mechanism) functionality in Ruby (mainly through the SOAP4R library). MTOM was added as a W3C recommendation with SOAP 1.2 back almost a year ago and was created to allow for inline attachment of payloads in a SOAP message, without having to hack the stack with DIME or SwA. MTOM (and the optimization component XOP) are pretty cool, as the user at either end sees only a unified infoset, with binary represented by the xsd:base64encoded type. There are no hooks, one simply has stick the binary data in a base64encoded element and MTOM/XOP takes over the rest. Currently, MTOM is implemented in .NET 2 and Xfire, thought I haven't explored any other toolkits, so it may be elsewhere too.

This blog will document my progress, and hopefully provide some needed assistance to people who are stuck down the black hole called ruby SOAP support.