wso2 esb customURI / serviceURI with expression

Question

I am trying to implement ESB solution to replace a web-tier part of a complex web-service architecture. The web-tier just receives the request from clients and distributes to a given set of app-tier instances.

Since the end-points are already defined and used by clients, I had to build the ESB proxy using the "CustomURI" approach as described here - http://achala11.blogspot.com/2012/08/access-wsdl-for-customuri-wso2-esb.html - to expose the end-point and wsdl.

Note that ServiceURI only allows to specify a fixed context path (correct me if I am wrong). For example: If WSDL end-point is http:// localhost:8280/CustomURL/Part1/Part2?wsdl, the corresponding ServiceURI entry is

<parameter name="ServiceURI">/CustomURL/Part1/Part2</parameter> 

What I want to use is an expression like this:

<parameter name="ServiceURI">/CustomURL/*</parameter>

to indicate that all requests that starts with /CustomURL in the context path will be handled by the proxy. Inside the proxy, I like to propagate the context URI to the endpoint defined in the send block. I think I found a RESTful way (didn't get it to work completely) of doing what I am trying to do here. As you can see, I can specify a context in the api and then use url-mapping option of resource to route all request to the api block. Later I am trying to use the "http endpoint" approach to construct the endpoint with the contextURI appended.

<api xmlns="http://ws.apache.org/ns/synapse" name="customService1" context="/CustomServices">
<resource methods="POST" url-mapping="/*">
  <inSequence>
     <log level="custom">
        <property name="uri.var.servicepath" expression="get-property('To')"/>
     </log>
     <send>
        <endpoint name="HTTPEndpoint">
           <http method="POST" uri-template="http://localhost:8001/{uri.var.servicepath}"/>
        </endpoint>
     </send>
  </inSequence>
  <outSequence>
     <send/>
  </outSequence>
</resource>
</api>

I hope you can help me with a similar solution for SOAP services - a way to specify a contextURI and then include the contextURI to the final endpoint that the proxy will send the request to. Note that the endpoints will be a list of entries (load balanced entries), but to keep things simple, I just kept one endpoint entry above.

Answer 1

I am new to WSO2 esb and so correct me if you find any mistakes.

Use case: Check this out - http://achala11.blogspot.com/2012/08/access-wsdl-for-customuri-wso2-esb.html . It shows a way to allow custom wsdl URL for SOAP services in WSO2. Well, in my case, the services are already in place and are consumed by clients. So I couldn't use the "generated" wsdl endpoint from WSO2 and had to make sure that the existing wsdl and service URLs are going to work through WSO2. Note that I am dealing with SOAP here and there are plenty of examples for RESTful services(including REST_URL_POSTFIX to handle endpoint manipulation).

The key information you need is that there are two sequences in WSO2 that all requests go through - namely "main" and "fault" (fault is used only when there is a failure scenario). As explained in my original question, getting the proxy service to do what I want became very challenging. Proxy service doesn't offer much in terms of propagating context to the endpoint or specifying expressions for ServiceURI variable. Then I looked at the "main" sequence entry as explained in the link I posted above. What WSO2 team is doing is just using regular expression to detect the incoming context and if it has "wsdl", they are routing the requests to a fixed wsdl endpoint and stopping the flow further. Well, I figured that we don't have to keep that code in the main sequence itself. Sequences can be chained. What that means is that you can call another sequence from main where you can do lot of magic (I didn't want to add lot of logic into the main sequence itself - my c/c++/java experiences might be stopping me from that :)) - If you are familiar with programming, sequences are like method/function calls. Main is the entry point and then you call whatever method you want inside and make further chained sequences from it etc).

First is the main sequence (filter is not needed - basically, .

<sequence name="main">
      <in>
         <log level="full"/>
         <sequence key="ISP_seq"/>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>

Now the ISP_seq where the routing of requests based on context (note the regular expressions used).

    <sequence xmlns="http://ws.apache.org/ns/synapse" name="ISP_seq">
   <in>
      <log level="custom">
         <property xmlns:ns="http://org.apache.synapse/xsd" name="Current URL" expression="get-property('To')"/>
      </log>
      <conditionalRouter continueAfter="false">
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
               <or>
                  <match type="url" regex="/firstService/10\.06/.*"/>
                  <match type="url" regex="/firstServiceVariant/.*"/>
               </or>
            </condition>
            <target sequence="firstService_seq"/>
         </conditionalRoute>
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
                  <match type="url" regex="/secondService.*"/>
            </condition>
            <target sequence="second_seq"/>
         </conditionalRoute>
      </conditionalRouter>
   </in>
</sequence>

Now inside the firstService_seq, you will receive all requests that has a context URI - /firstService/10.06/ (followed by anything - including ?wsdl :)) or /firstServiceVariant/ (followed by anything).

              <match type="url" regex="/firstService/10\.06/.*"/>
              <match type="url" regex="/firstServiceVariant/.*"/>

Ok - now to the Sequence that is going to handle the firstService and the other service.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="firstService_seq">
   <in>

      <!-- FIRST let us handle the WSDL requests. -->

      <property name="REST_URL_POSTFIX" action="remove" scope="axis2"/>
      <switch xmlns:ns="http://org.apache.synapse/xsd" source="get-property('To')">
         <case regex="/firstService/10\.06/service\?[Ww][Ss][Dd][Ll]">
            <send>
               <endpoint>
                  <address uri="http://myappServer:10011/firstService10.06?wsdl" format="get"/>
               </endpoint>
            </send>
         </case>
      </switch>
      <send>

         <!-- below here, we will handle the actual SOAP requests -->
         <endpoint>
            <loadbalance algorithm="org.apache.synapse.endpoints.algorithms.RoundRobin">
               <endpoint name="firstFarm_7011">
                  <address uri="http://hostA:7011/firstService/10.06/service"/>
               </endpoint>
               <endpoint name="firstFarm_7021">
                  <address uri="http://hostA:7021/firstService/10.06/service"/>
               </endpoint>
               <endpoint name="secondFarm_7011">
                  <address uri="http://hostX:7011/firstService/10.06/service"/>
               </endpoint>
            </loadbalance>
         </endpoint>
      </send>
   </in>
   <out>
      <send/>
   </out>
</sequence>

I am sorry for the long explanation. Somehow, I was not able to figure this out from the existing documentation. I wish the same features are available through Proxy Service as well as that is what you will think of (at least I was) when you are considering to use a ESB for the use case in my question.

Hope it helps someone. As I said earlier, correct me if I made any mistakes or if there is a better approach.

Answer 2

I found a cleaner/better solution - it is just an enhanced version of the above answer.

The main challenge is to get the SOAP WSDL and the Service endpoint (Context URI) work without forcing it like it happens on a proxy service approach. Note that if you can work with the contextURI WSO2 team suggests (/services//), you don't have to do any of these. The standard documentation will have all the details you need.

The logic is:

Define 3 sequences (I split them into three, but you can put all these in just one sequence as well).

main sequence (the standard built-in one - this is the sequence that gets all traffic if a request doesn't follow proxy service rules already defined in ESB). In main sequence, we just route all the incoming requests to another sequence where we do the filtering/conditinal routing.

<sequence name="main">
  <in>
     <log level="full"/>
     <sequence key="routing_seq"/>
  </in>
  <out>
     <send/>
  </out>
  <description>The main sequence for the message mediation</description>
</sequence>

*routing_sequence* Just like in the first answer, we are going to route the incoming requests to other special sequences based on the context URI in the request.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="routing_seq">
   <in>
      <log level="custom">
         <property xmlns:ns="http://org.apache.synapse/xsd" name="Current URL" expression="get-property('To')"/>
      </log>
      <conditionalRouter continueAfter="false">
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
               <or>
                  <match type="url" regex="/firstService/10\.06/.*"/>
                  <match type="url" regex="/firstServiceVariant/.*"/>
               </or>
            </condition>
            <target sequence="firstService_seq"/>
         </conditionalRoute>
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
                  <match type="url" regex="/secondService.*"/>
            </condition>
            <target sequence="second_seq"/>
         </conditionalRoute>
      </conditionalRouter>
   </in>
</sequence>

*firstservice_seq*

Now we are ready to handle the incoming request - note that we identified the application as "firstservice" in the previous step. There are two kinds of requests you can receive here - one is for WSDL and the other one is a soap request

<sequence xmlns="http://ws.apache.org/ns/synapse" name="firstService_seq">
   <in>

     <property name="REST_URL_POSTFIX" scope="axis2" action="remove"/>

     <!-- We are checking whether the request ends with a ?wsdl or .xsd -->
     <!-- For that we are using the context URI present in the 'To' field -->
     <!-- if it is a wsdl or xsd request, we are converting it to a HTTP GET method -->
     <!-- and sending to the final endpoint. All soap operation requests are sent as HTTP POST -->


     <switch xmlns:ns="http://org.apache.synapse/xsd" source="get-property('To')">
        <case regex=".*(?:\?[Ww][Ss][Dd][Ll]|\.[Xx][Ss][Dd])\s*$">
           <property name="HTTP_METHOD" value="GET"/>
           <property name="messageType" value="text/xml"/>
           <send receive="wsdl_transformer_seq">
              <endpoint key="local-enrty-firstservice-ep-key"/>
           </send>
           <drop/>
        </case>
        <!-- default means non-wsdl/non-xsd - which means a regular soap operation on the service -->
        <default>
           <property name="HTTP_METHOD" value="POST"/>
           <send>
              <endpoint key="local-enrty-firstservice-ep-key"/>
           </send>
        </default>
     </switch>

   </in>
   <out>
      <send/>
   </out>
</sequence>

Hope you read the comments in the sequence definition. As you can see, I am converting the wsdl and xsd requests into HTTP GET to avoid any confusion at the app-tier as otherwise, there could be some junk SOAP body parts get injected from somewhere in the flow).

So basically, we check the 'To' property which contains a Context URI like /firstservice/10.06/service?WSDL (if it was a WSDL request) or /firstservice/10.06/service - if it was a SOAP request. Based on the value, we decide what to do with the request.

Note the below section in the wsdl logic:

<send receive="wsdl_transformer_seq">
              <endpoint key="local-enrty-firstservice-ep-key"/>
           </send>
           <drop/>
 </send>

What happens is that when we pull the wsdl from the endpoint (which I will explain shortly), schemaLocation and soap:address fields contain the information (hostname and port) of actual server instance that processed the request. This is an unwanted result as you are exposing your internal details to the clients and others. So we should mask it. The way to do that is to use a special feature in WSO2. In a send, you can specify another sequence that will receive the results of the send before it goes to the client. It is more or less like you define an <out> portion in a sequence. Since we don't want this particular manipulation happen to all requests, we are not defining a special sequence that gets applied to only wsdl/xsd reqeusts. In the wsdl_transformer_seq, we use XSLT to change the hostname and port present in the wsdl or xsd response.

wsdl_transformer_seq

<sequence name="wsdl_transformer_seq">
  <xslt xmlns:ns="http://org.apache.synapse/xsd"
        key="xslt-url-manipulator"
        source="/"/>
  <send/>
</sequence>

Note that I externalized these entries (xslt transformer for example) and get them loaded through a local-entry registry.

  <localEntry key="xslt-url-manipulator"
           src="file:repository/myapp/resources/transform/url-in-wsdl-transform.xslt"/>

Now the contents of url-in-wsdl-transform.xslt

<?xml version="1.0" encoding="ISO-8859-1"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                  xmlns:xs="http://www.w3.org/2001/XMLSchema"
                  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
                  version="2.0">
     <xsl:output method="xml" encoding="UTF-8" indent="yes" omit-xml-declaration="yes"/>
     <xsl:param name="newURL">https://services.mycompany.com</xsl:param>
     <xsl:template match="@*|node()">
        <xsl:copy>
           <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
     </xsl:template>
     <xsl:template match="soap:address/@location">
        <xsl:attribute name="location">
           <xsl:value-of select="replace(.,'https?://[^/]*',$newURL)"/>
        </xsl:attribute>
     </xsl:template>
     <xsl:template match="xs:import/@schemaLocation">
        <xsl:attribute name="schemaLocation">
           <xsl:value-of select="replace(.,'https?://[^/]*',$newURL)"/>
        </xsl:attribute>
     </xsl:template>
  </xsl:stylesheet>

Note the name - https://services.mycompany.com - in the xslt above. You can change it to whatever you want and the schemaLocation and soap:address fields in the wsdl and xsd will now use that in the host:port area

And finally the endpoint local-enrty-firstservice-ep-key Note that this as well is an externally loaded file - just like the xslt entry above. So I will only post the content of the actul external file.

This was a tough cookie and I had to dig through WSO2 source code and figure out some "undocumented" features. Well, it is "undocumented" in WSO2 world, but documented otherwise as WSO2 uses a third-party library which has many features that are not published in WSO2. Note that the endpoint points to the app-tier instances and we use the roundrobin algorithm to distribute the traffic to the avilalble app-tier instances.

<endpoint xmlns="http://ws.apache.org/ns/synapse">
  <loadbalance algorithm="org.apache.synapse.endpoints.algorithms.RoundRobin">
     <endpoint name="firsthost_5012">
        <http uri-template="http://firsthost.com:5012{+uri.var.servicepath}"/>
     </endpoint>
     <endpoint name="firsthost_5022">
        <http uri-template="http://firsthost.com:5022{+uri.var.servicepath}"/>
     </endpoint>
  </loadbalance>
  <property xmlns:ns="http://org.apache.synapse/xsd"
            name="uri.var.servicepath"
            expression="get-property('To')"/>
</endpoint>

There is lot of magic happens here - First we save the context uri to a special variable called uri.var.servicepath Now we will use that with the new "http endpoint" feature available from WSO 4.7.0 onwards.

http endpoint uses uri-template - where you can dynamically construct the endpoint. So I don't have to specify the context URI in the configuration - So let us say the original request we got into ESB was like this:

http://esbhost.com:8280/firstservice/10.06/service

The context uri here is "/firstservice/10.06/service" - so we want this to be appended to the actual app-tier server url. In other words, I want it to become http://firsthost.com:5022/firstservice/10.06/service

In http endpoint, it allows us to use the special variable - uri.var.servicepath that we defined earlier.

      <http uri-template="http://firsthost.com:5022{+uri.var.servicepath}"/>

since uri.var.servicepath already contains a '/' at the beginning, I am not specifying separately. But note the + sign in the brackets. What is that for? hmmm - well, it turned out that WSO2 uses the interesting third-party library - "damn good uri processor" - it is mainly used for RESTful APIs. Unfortunately, if you plainly use the field like this, http://firsthost.com:5022{uri.var.servicepath}, the 3rd party library will convert the special characters present in the context uri to their http safe equivalent - something like %20f or whatever. So our url now becomes http ://firsthost:5022%20ffirstservice%20f10.06.... - it is not good. Well here comes the special feature in the 3rd party library that saves the day. If you put a + at the beginning of the special variable, this translation is turned off ;) - voila - we got what we wanted.

So that is it folks. I will try to post the full configuration below (minus the externalized entries that you can find in the above sections)

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
      <parameter name="cachableDuration">15000</parameter>
   </registry>
   <localEntry key="xslt-url-manipulator"
               src="file:repository/myapp/resources/transform/url-in-wsdl-transform.xslt"/>
   <localEntry key="local-enrty-firstservice-ep-key"
               src="file:repository/myapp/resources/endpoint/firstservice-endpoints.xml">
   </localEntry>
   <sequence xmlns="http://ws.apache.org/ns/synapse" name="routing_seq">
   <in>
      <log level="custom">
         <property xmlns:ns="http://org.apache.synapse/xsd" name="context URI" expression="get-property('To')"/>
      </log>
      <conditionalRouter continueAfter="false">
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
               <or>
                  <match type="url" regex="/firstService/10\.06/.*"/>
                  <match type="url" regex="/firstServiceVariant/.*"/>
               </or>
            </condition>
            <target sequence="firstService_seq"/>
         </conditionalRoute>
         <conditionalRoute breakRoute="false" asynchronous="false">
            <condition>
                  <match type="url" regex="/secondService.*"/>
            </condition>
            <target sequence="second_seq"/>
         </conditionalRoute>
      </conditionalRouter>
   </in>
   </sequence>
   <sequence name="wsdl_transformer_seq">
      <xslt xmlns:ns="http://org.apache.synapse/xsd"
            key="xslt-url-manipulator"
            source="/"/>
      <send/>
   </sequence>
   <sequence name="fault">
      <log level="full">
         <property name="MESSAGE" value="Executing default 'fault' sequence"/>
         <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
         <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
      </log>
      <drop/>
   </sequence>
   <sequence name="firstservice_seq">
      <in>
         <property name="REST_URL_POSTFIX" scope="axis2" action="remove"/>
         <switch xmlns:ns="http://org.apache.synapse/xsd" source="get-property('To')">
            <case regex=".*(?:\?[Ww][Ss][Dd][Ll]|\.[Xx][Ss][Dd])\s*$">
               <property name="HTTP_METHOD" value="GET"/>
               <property name="messageType" value="text/xml"/>
               <send receive="wsdl_transformer_seq">
                  <endpoint key="local-enrty-firstservice-ep-key"/>
               </send>
               <drop/>
            </case>
            <default>
               <property name="HTTP_METHOD" value="POST"/>
               <send>
                  <endpoint key="local-enrty-firstservice-ep-key"/>
               </send>
            </default>
         </switch>
      </in>
   </sequence>
   <sequence name="main">
      <in>
         <filter xmlns:ns="http://org.apache.synapse/xsd"
                 source="get-property('To')"
                 regex="http://localhost:9000.*">
            <then>
               <send/>
            </then>
            <else/>
         </filter>
         <sequence key="routing_seq"/>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>
</definitions>

I know it is a very lengthy post - but wanted to explain the details for those who care. Hope it helps someone.