[WSO2][ESB][MB] Transport message types are differ

Environment - WSO2 ESB 5.0.0 Alpha
                     WSO2 MB 3.1.0 (JMS broker)
                          jdk1.8.0_72

Preconditions: ESB should be configured with MB. (How to ?)
                          MB up and running (Which having 1 offset)
                          ESB should be up and running.
                          Created JMS message store in ESB.

Sample Configuration:

 <messageStore  
     class="org.apache.synapse.message.store.impl.jms.JmsStore" name="JMSStore">  
     <parameter name="store.jms.destination">JMSMS</parameter>  
     <parameter name="store.failover.message.store.name">JMSStore</parameter>  
     <parameter name="store.producer.guaranteed.delivery.enable">false</parameter>  
     <parameter name="store.jms.cache.connection">true</parameter>  
     <parameter name="java.naming.factory.initial">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter>  
     <parameter name="java.naming.provider.url">repository/conf/jndi.properties</parameter>  
     <parameter name="store.jms.JMSSpecVersion">1.1</parameter>  
   </messageStore>  

                                
                             Created Proxy to publish messages to JMS message store.

Sample Configuration:

 <proxy name="JMSStoreProxy" startOnLoad="true" transports="https http">  
     <description/>  
     <target>  
       <inSequence>  
         <log/>  
         <property name="FORCE_SC_ACCEPTED" scope="axis2" value="true"/>  
         <property name="OUT_ONLY" value="true"/>  
         <log level="custom">  
           <property name="STATUS" value="StoringMessage"/>  
         </log>  
         <store messageStore="JMSStore"/>  
       </inSequence>  
     </target>  
   </proxy>  

Situation : Created a queue in MB called "JMSMS" 
               Going to publish messages to the proxy of ESB. Published SOAP message to proxy
  
Error:

 [2016-04-11 11:12:57,127] ERROR - RelayUtils Error while building Passthrough stream  
 org.apache.axiom.soap.SOAPProcessingException: Transport level information does not match with SOAP Message namespace URI  
      at org.apache.axis2.builder.BuilderUtil.validateSOAPVersion(BuilderUtil.java:745)  
      at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:68)  
      at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:148)  
      at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:137)  
      at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:100)  
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:72)  
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:49)  
      at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:160)  
      at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:197)  
      at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)  
      at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)  
      at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)  
      at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)  
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
      at java.lang.Thread.run(Thread.java:745)  
 [2016-04-11 11:12:57,128] ERROR - SequenceMediator Error while building message  
 org.apache.axis2.AxisFault: Error while building Passthrough stream  
      at org.apache.synapse.transport.passthru.util.RelayUtils.handleException(RelayUtils.java:287)  
      at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:146)  
      at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:100)  
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:72)  
      at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:49)  
      at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:160)  
      at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:197)  
      at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)  
      at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)  
      at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)  
      at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)  
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
      at java.lang.Thread.run(Thread.java:745)  
 Caused by: org.apache.axiom.soap.SOAPProcessingException: Transport level information does not match with SOAP Message namespace URI  
      at org.apache.axis2.builder.BuilderUtil.validateSOAPVersion(BuilderUtil.java:745)  
      at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:68)  
      at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:148)  
      at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:137)  
      ... 12 more  

Analysis:
This is JMS transport and mistakenly sent the SOAP message. So message transaction types are different and not the type expected by the mediator.

Solution:
Publish JMS messages to the proxy.

In general terms, check your message type. You may get following error because of publishing the incorrect type of messages.

Message Processor giving error while sending message to back-end service.

Environment - ESB 
                         jdk1.8.0_72

Preconditions: ESB should be started.
                        Axis2 Server should be started
                        Axis2 sample client used to publish messages
                                         
Situation - Configured a message processor(Scheduled Message Forwarding Processor) which targeting a message store and particular endpoint.

                  Sample config:

<messageProcessor class="org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor" messageStore="reg" name="storeProce" targetEndpoint="StockQuote">
    <parameter name="client.retry.interval">1000</parameter>
    <parameter name="throttle">false</parameter>
    <parameter name="max.delivery.attempts">4</parameter>
    <parameter name="member.count">1</parameter>
    <parameter name="max.delivery.drop">Enabled</parameter>
    <parameter name="interval">1000</parameter>
    <parameter name="message.processor.deactivate.sequence">fault</parameter>
    <parameter name="is.active">true</parameter>
    <parameter name="target.endpoint">StockQuote</parameter>
</messageProcessor>

            Store mediator created as follows:

<?xml version="1.0" encoding="UTF-8"?>
<sequence name="onstoreSeq">
    <store messageStore="reg" sequence="conf:/registrySeq" />
</sequence>

Then created a proxy and used Sample axis2Client ($ESB_HOME/samples/axis2Client/) to invoke proxy. Synapse config can find below[1]
       
         axis2Client Command:
         
ant stockquote -Daddurl=http://localhost:8280/services/regProxy -Dmode=placeorder - Dsymbol=MSFT
           
Error -  When publishing messages, Back end will receive it, But processor retries and give an error as follows: 

ERROR - ForwardingService BlockingMessageSender of message processor [storeProce] failed to send message to the endpoint

Analysis - It is about accepting message with different properties. So added property mediators to change message properties related to ESB.

Solution -  Add following properties before store mediator as follows:

<?xml version="1.0" encoding="UTF-8"?>
<sequence name="onstoreSeq">
    <property name="FORCE_SC_ACCEPTED" scope="axis2" value="true" />
    <property name="OUT_ONLY" value="true" />
    <store messageStore="reg" sequence="conf:/registrySeq" />
</sequence>

[1] - Synapse Config

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
    <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
        <parameter name="cachableDuration">15000</parameter>
    </registry>
    <taskManager provider="org.wso2.carbon.mediation.ntask.NTaskTaskManager" />
    <proxy name="regProxy" startOnLoad="true" trace="disable" transports="http https">
        <description />
        <target inSequence="onstoreSeq" />
    </proxy>
    <endpoint name="StockQuote">
        <address uri="http://127.0.0.1:9000/services/SimpleStockQuoteService" />
    </endpoint>
    <sequence name="onstoreSeq">
        <property name="FORCE_SC_ACCEPTED" scope="axis2" value="true" />
        <property name="OUT_ONLY" value="true" />
        <store messageStore="reg" sequence="conf:/registrySeq" />
    </sequence>
    <sequence name="fault">
        <!-- Log the message at the full log level with the ERROR_MESSAGE and the ERROR_CODE-->
        <log level="full">
            <property name="MESSAGE" value="Executing default 'fault' sequence" />
            <property expression="get-property('ERROR_CODE')" name="ERROR_CODE" />
            <property expression="get-property('ERROR_MESSAGE')" name="ERROR_MESSAGE" />
        </log>
        <!-- Drops the messages by default if there is a fault -->
        <drop />
    </sequence>
    <sequence name="main">
        <in>
            <!-- Log all messages passing through -->
            <log level="full" />
            <filter regex="http://localhost:9000.*" source="get-property('To')">
                <!-- Send the messages where they have been sent (i.e. implicit "To" EPR) -->
                <send />
            </filter>
        </in>
        <out>
            <send />
        </out>
        <description>The main sequence for the message mediation</description>
    </sequence>
    <messageStore name="reg" />
    <messageStore name="InStore" />
    <messageProcessor class="org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor" messageStore="reg" name="storeProce" targetEndpoint="StockQuote">
        <parameter name="client.retry.interval">1000</parameter>
        <parameter name="throttle">false</parameter>
        <parameter name="max.delivery.attempts">4</parameter>
        <parameter name="member.count">1</parameter>
        <parameter name="max.delivery.drop">Enabled</parameter>
        <parameter name="interval">1000</parameter>
        <parameter name="message.processor.deactivate.sequence">fault</parameter>
        <parameter name="is.active">true</parameter>
        <parameter name="target.endpoint">StockQuote</parameter>
    </messageProcessor>
    <!-- You can add any flat sequences, endpoints, etc.. to this synapse.xml file if you do

    *not* want to keep the artifacts in several files -->
</definitions>

Creating ESB Cluster - 502 Bad gateway Error

Environment - ESB Cluster with NGINX Plus

Situation - Creating a ESB cluster which are having separate cloud instances.
                  Load balancing will be done with NGINX Plus.
                  Configurations done and started the manager node.
                  It will indicate server is up correctly.

Error -  When clicking on the management console link, it will provide an error called "502, Bad Gateway"

Analysis - This error providing by NGINX and it indicate can not access to relevant manager and worker nodes. This can be happen because all servers are down.

If servers are up, it may not have access to those servers. In this situation, all servers running on different clod instances which are having Centos (May be having Ubuntu, Windows or any other OS). Those instances are blocking access via enabled firewalls. So have to disable firewalls.

Then you may not getting the error.

Solution - In Centos, have a command to disable firewalls as follows:
         
                 service iptables stop

ESB Cluster - Manager Node Error (JMS inbound endpoint)

Environment - ESB [500] Cluster with JMS inbound endpoint

Situation - After creating the cluster, you have to start manger node first.
                  When starting manager node, it may pop an long error.
                  (In this cluster, I made some configuration for JMS)

Error - Error is very long one and will be putting first part of it.

 ERROR - NTaskTaskManager Scheduling task[[NTask::-1234::jms_inbound - JMS--SYNAPSE_INBOUND_ENDPOINT]::JMS--SYNAPSE_INBOUND_ENDPOINT] FAILED.Error: No available task nodes  
 for resolving a task location  
 org.wso2.carbon.ntask.common.TaskException: No available task nodes  
 for resolving a task location  
 at org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.getTaskLocation(ClusteredTaskManager.java: 232)  
 at org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.locateMemberForTask(ClusteredTaskManager.java: 209)  
 at org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.getMemberIdFromTaskName(ClusteredTaskManager.java: 283)  
 at org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.scheduleTask(ClusteredTaskManager.java: 91)  
 at org.wso2.carbon.mediation.ntask.NTaskTaskManager.schedule(NTaskTaskManager.java: 103)  
 at org.wso2.carbon.mediation.ntask.NTaskTaskManager.init(NTaskTaskManager.java: 352)  
 at org.wso2.carbon.mediation.ntask.NTaskTaskManager.update(NTaskTaskManager.java: 365)  
 at org.wso2.carbon.mediation.ntask.internal.NtaskService.updateAndCleanupObservers(NtaskService.java: 103)  
 at org.wso2.carbon.mediation.ntask.internal.NtaskService.setConfigurationContextService(NtaskService.java: 96)  
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 62)  
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43)  
 at java.lang.reflect.Method.invoke(Method.java: 497)  
 at org.eclipse.equinox.internal.ds.model.ComponentReference.bind(ComponentReference.java: 376)  
 at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.bindReference(ServiceComponentProp.java: 430)  
 at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.bind(ServiceComponentProp.java: 218)  
 at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java: 343)  
 at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java: 620)  
 at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java: 197)  
 at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java: 343)  
 at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java: 222)  
 at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java: 107)  
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java: 861)  
 at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java: 230)  
 at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java: 148)  
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java: 819)  
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java: 771)  
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java: 130)  
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java: 214)  
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java: 433)  
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java: 451)  
 at  

Analysis : This will be occur as I configured some JMS transactions and there is no available workers. This not a harmful error and just giving warning of missing workers. This may also occur with other configurations (For different transactions). Because polling services will run on top of NTASK. JMS, VFS, KAFKA are polling inbound endpoints.

Solution : When start work nodes, it may not pop up the error.  May not giving special indication about resolved solution. But error will not pop up again as workers are available.

HornetQ IP/Hostname Conflict

Environment - ESB (as JMS client), HornetQ (as JMS broker)

Situation - Configure ESB with HornetQ (Refer doc: )
                  HornetQ in a separate machine/cloud instance
                  Refer HornetQ via its' IP.

Error -

TID: [-1234] [] [2016-02-24 15:30:13,451]  WARN {org.wso2.carbon.inbound.endpoint.protocol.jms.JMSPollingConsumer} -  Inbound JMS endpoint unable to get a connection. {org.wso2.carbon.inbound.endpoint.protocol.jms.JMSPollingConsumer}
TID: [-1234] [] [2016-02-24 15:30:14,451] ERROR {org.wso2.carbon.inbound.endpoint.protocol.jms.factory.JMSConnectionFactory} -  Connection cannot bb establish to the broke. Plese check the broker libs provided. {org.wso2.carbon.inbound.endpoint.protocol.jms.factory.JMSConnectionFactory}

Analysis: Even though this error talking about missing libraries, this is not an issue about libraries. This is about conflicting to handle IP/host name of HornetQ. For HornetQ, there is a special config to give its hostname/IP.

Solution: Must edit "run.sh" in <HornetQ_Home>/bin
             
                There is a commented line as follows:
                

# Use the following line to run with different ports
#export CLUSTER_PROPS="-Djnp.port=1099 -Djnp.rmiPort=1098 -Djnp.host=localhost -Dhornetq.remoting.netty.host=localhost -Dhornetq.remoting.netty.port=5445"
Uncomment it and give IP/hostname to "-Djnp.local" parameter
export CLUSTER_PROPS="-Djnp.port=1099 -Djnp.rmiPort=1098 -Djnp.host=192.*.*.*                  -Dhornetq.remoting.netty.host=localhost -Dhornetq.remoting.netty.port=5445"

Welcome to Bug corner

Hi Technical People,

This is my blog which will summarize bugs which are always create troubles for me. Hope this will be useful to everyone. :) 

Lets find out these lovely bugs :)