I have an Active-Active Deployment of WSO2 API Manager 2.5.0, also with WSO2 AM Analytics and WSO2 Identity Server as Key Manager (all in high availability).
I have enabled monetization following Documentation, deploying apim-billing-engine-1.3.0.war in /repository/deployment/server/webapps/, deploying subs-workflow-1.3.0.jar in /repository/component/lib/, and installing APIM_Billing_2.2.0.car in WSO2 AM Analytics.
I turn off one node to test the other node. Everything is as expected, but I have a issue with the Billing Engine. The use case is:
- A) User login in API Store (URL as https://10.90.60.100/store). OK
- B) User subscribes to an commercial API, and it is redirected to the billing engine (URL as https://10.90.60.100/apim-billing-engine-1.3.0/app/main?execution=e3s2). OK
- C) User fills the new user form, and clicks in "sign up". But it is not redirected to API Store, just stay "Processing..." foverer. In logs, it leaves the following error:
TID: [-1234] [] [2018-11-27 14:41:54,357] ERROR {org.apache.catalina.core.StandardWrapperValve} - Servlet.service() for servlet [Spring MVC Dispatcher Servlet] in context with path [/apim-billing-engine-1.3.0] threw exception [Request processing failed; nested exception is org.springframework.webflow.execution.ActionExecutionException: Exception thrown executing [AnnotatedAction@3af40bc0 targetAction = [EvaluateAction@761cb83b expression = workflowClient.activateSubscription(redirectBean, user), resultExpression = [null]], attributes = map[[empty]]] in state 'signUp' of flow 'main' -- action execution attributes were 'map[[empty]]'] with root cause {org.apache.catalina.core.StandardWrapperValve} sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker.process_record(Handshaker.java:987) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:290) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:259) at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:125) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:319) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.wso2.apim.billing.services.impl.WorkflowClientImpl.sendPOSTMessage(WorkflowClientImpl.java:104) at org.wso2.apim.billing.services.impl.WorkflowClientImpl.activateSubscription(WorkflowClientImpl.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69) at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:109) at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:57) at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102) at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:97) at org.springframework.binding.expression.spel.SpringELExpression.getValue(SpringELExpression.java:84) at org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:75) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) at org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) at org.springframework.webflow.engine.support.ActionTransitionCriteria.test(ActionTransitionCriteria.java:82) at org.springframework.webflow.engine.support.TransitionCriteriaChain.test(TransitionCriteriaChain.java:71) at org.springframework.webflow.engine.Transition.canExecute(Transition.java:195) at org.springframework.webflow.engine.Transition.execute(Transition.java:211) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:393) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:388) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ViewState.handleEvent(ViewState.java:232) at org.springframework.webflow.engine.ViewState.resume(ViewState.java:196) at org.springframework.webflow.engine.Flow.resume(Flow.java:545) at org.springframework.webflow.engine.impl.FlowExecutionImpl.resume(FlowExecutionImpl.java:258) at org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:169) at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183) at org.springframework.faces.webflow.JsfFlowHandlerAdapter.handle(JsfFlowHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80) at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91) at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60) at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99) at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57) at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48) at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62) at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
I know it is a cert issue, but I cannot find the exact point.
I have tested the billing engine is getting ok the jks file (datasource.properties file has: jksPath=/repository/deployment/server/webapps/apim-billing-engine-1.3.0/WEB-INF/classes/wso2-jks/wso2carbon.jks).
I use the default certificate in wso2store. And I have checked it is in:
- WSO2 API Manager private store:
/repository/resources/security/wso2carbon.jks - WSO2 API Manager client-trust store: /repository/resources/security/client-truststore.jks
- Billing machine store: /repository/deployment/server/webapps/apim-billing-engine-1.3.0/WEB-INF/classes/wso2-jks/wso2carbon.jks
- WSO2 AM Analytics private store: /repository/resources/security/wso2carbon.jks
- WSO2 AM Analytics client-trust store: /repository/resources/security/client-truststore.jks
- WSO2 IS as KM private store: /repository/resources/security/wso2carbon.jks
- WSO2 IS as KM client-trust store: /repository/resources/security/client-truststore.jks
I have added the WSO2 public certificate to Java Certificate Store following this post, with same result.
Some pages (this and this) say that similar error is shown when a WSO2 product cannot connect to a backend API, because a cert issue. I think this cannot be my problem because in my case, my WSO2 Api Manager dont throw the error trying to connect to a backend API, but trying to connect to billing engine (that uses wso2carbon cert, same as WSO2 Api Manager)
This post has the same problem (but he don't leave the log error)
