A JWT token is a self contained token. That means that it lives on its own until it expires and can't be revoked. So by definition it must expire. Because when it falls into the wrong hands, it'll give access to your resources without being able to revoke it. So yes, you should be worried with this implementation.
The problem here is that you trust the refresh token itself, because it's a JWT. You should in fact trust the server. Not because the JWT can't be trusted, but because the refresh token doesn't have to be a JWT.
Save refresh tokens in memory including the expiration time. You can remove expired tokens from memory. This means that only tokens that exist in memory can be used to request a new access token. And to be on the safe side, use one-time only refresh tokens.
The flow would be something like this:
- the user logs in, receives a JWT access token (5 minutes) and the refresh token 1 code (48 hours). Refresh token 1 is saved on the server.
- five minutes later: the access token expires
- a new access token is requested using refresh token 1.
- user receives a new access token (5 minutes) AND the refresh token 2 code (48 hours). Token 1 is removed from memory and token 2 is added to memory.
- and this goes on for several hours.
- For two days the user doesn't use the app
- 50 hours later: because both tokens are expired, the user has to login again. Resetting the flow.
On logout remove the refresh token from memory. And if in the meantime you wish to revoke access. Simply remove the refresh token from memory. Within 5 minutes the user has to login again.