I recently updated my environment from WSO2 IS 5.0.0 to WSO2 IS 5.2.0. My environment consists of 2 machines that are creating a cluster (using the WKA membership scheme and Load Balancer(AWS ELB) with sticky session enabled). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
I am also using the WSO2 IS multitenancy. When I was using IS 5.0.0 I had several tenants which I thought I had migrated properly by following the instructions mentioned in:
https://docs.wso2.com/display/IS510/Upgrading+from+a+Previous+Release
https://docs.wso2.com/display/IS520/Upgrading+from+a+Previous+Release
Unfortunately when I tried to use the password grant type with one of the profiles in one of the tenant I received the following response:
{
"error": "server_error",
"error_description": "Internal Server Error."
}
instead receiving valid access token. The error in the console log is
[2016-10-21 15:41:34,941] ERROR {org.wso2.carbon.identity.oauth2.OAuth2Service} - Error occurred while issuing the access token for Client ID : klaFcti7TeDeEqFbzZr_HYKEEBga, User ID [email protected]@test
and Grant Type : password
java.lang.NullPointerException
at org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder.buildIDToken(DefaultIDTokenBuilder.java:147)
at org.wso2.carbon.identity.oauth2.token.AccessTokenIssuer.issue(AccessTokenIssuer.java:268)
at org.wso2.carbon.identity.oauth2.OAuth2Service.issueAccessToken(OAuth2Service.java:219)
at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.getAccessToken(OAuth2TokenEndpoint.java:246)
at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.issueAccessToken(OAuth2TokenEndpoint.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:188)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:104)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:249)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:289)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:209)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:265)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1082)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:623)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Then I tried creating brand new tenant and did the same case, but this time I was able to receive valid access token for the user.
Also when I try to login in the migrated from 5.0.0 tenants through the admin console, I receive the following error:
[IS]ERROR {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Error while updating wso2carbon-sts in STSDeploymentInterceptor
java.io.IOException: Invalid keystore format
Please, find attached the log file:
https://www.dropbox.com/s/jwstx6my3jcdjiz/Invalid%20keystore%20format.log?dl=0
It seems for some reason 5.2.0 is considering these keystores as invalid ones. I think these keystores are created upon tenant creation, so I am not quite sure what is causing this behavior and why WSO2 IS 5.2.0 considers the keystores of WSO2 IS 5.0.0 tenants as invalid.
The keystores exist.
Do you think this is related to some sort of misconfiguration or issue while migrating these tenants from 5.0.0 to 5.2.0? Is there a way to fix this behavior and not lose the tenants?
Thanks in advance
