Solving Spring Security OAuth2 Token Thundering Herd Problem

Author: Shazin Sadakath


"In computer science, the thundering herd problem occurs when a large number of processes waiting for an event are awoken when that event occurs, but only one process is able to proceed at a time. After the processes wake up, they all demand the resource and a decision must be made as to which process can continue. After the decision is made, the remaining processes are put back to sleep, only to all wake up again to request access to the resource." - Wikipedia

When Spring Security OAuth2 is used with a JdbcTokenStore which is backed by a Relational Database as in many cases, when a new token is generated on a user/microservice authentication an entry is added to oauth_access_token table. Also a refresh token is added to oauth_refresh_token table. 

Each Access Token and Refresh Token as an Time To Live (TTL) which is mentioned in seconds. By default a Spring Security OAuth2 Access Token is valid for 12 Hours and a Refresh Token is valid for 30 Days. 

But in an event of system outage lets say 24 hours, and when all user/microservice are forced to reauthenticate into the system, a Thundering herd problem can take place in the Database primarily on oauth_access_token table because the previous invalid Access Tokens need to be deleted from that table to make way for new Access Tokens. 

Lets say there are 1 Million users all logged in within relatively the same time, then 1 Million hits on Database need to take place because all the Access Tokens are expired after 12 hours. Even worse in another 12 hours all those will also expire meaning now the system is faced with a recurring thundering herd every 12 hours. 

The solution for this is to randomize the validity of the Access and Refresh token durations which spans into seconds within a maximum and minimum range. In Spring Security OAuth2 this can be done using implementing a custom token service which extends from org.springframework.security.oauth2.provider.token.DefaultTokenServices as following:

@Service
public class CustomTokenServices extends DefaultTokenServices {

    private int minimumAccessTokenValiditySeconds = 43200;
    private int maximumAccessTokenValiditySeconds = minimumAccessTokenValiditySeconds * 2;
    private int minimumRefreshTokenValiditySeconds = 2592000;
    private int maximumRefreshTokenValiditySeconds = minimumRefreshTokenValiditySeconds * 2;

    public CustomTokenServices(DataSource dataSource) {
        setTokenStore(new JdbcTokenStore(dataSource));
    }

    @Override
    protected int getAccessTokenValiditySeconds(OAuth2Request clientAuth) {
        return new Random(System.currentTimeMillis()).nextInt(maximumAccessTokenValiditySeconds) + minimumAccessTokenValiditySeconds;
    }

    @Override
    protected int getRefreshTokenValiditySeconds(OAuth2Request clientAuth) {
        return new Random(System.currentTimeMillis()).nextInt(maximumRefreshTokenValiditySeconds) + minimumRefreshTokenValiditySeconds;
    }

    public int getMinimumAccessTokenValiditySeconds() {
        return minimumAccessTokenValiditySeconds;
    }

    public void setMinimumAccessTokenValiditySeconds(int minimumAccessTokenValiditySeconds) {
        this.minimumAccessTokenValiditySeconds = minimumAccessTokenValiditySeconds;
    }

    public int getMaximumAccessTokenValiditySeconds() {
        return maximumAccessTokenValiditySeconds;
    }

    public void setMaximumAccessTokenValiditySeconds(int maximumAccessTokenValiditySeconds) {
        this.maximumAccessTokenValiditySeconds = maximumAccessTokenValiditySeconds;
    }

    public int getMinimumRefreshTokenValiditySeconds() {
        return minimumRefreshTokenValiditySeconds;
    }

    public void setMinimumRefreshTokenValiditySeconds(int minimumRefreshTokenValiditySeconds) {
        this.minimumRefreshTokenValiditySeconds = minimumRefreshTokenValiditySeconds;
    }

    public int getMaximumRefreshTokenValiditySeconds() {
        return maximumRefreshTokenValiditySeconds;
    }

    public void setMaximumRefreshTokenValiditySeconds(int maximumRefreshTokenValiditySeconds) {
        this.maximumRefreshTokenValiditySeconds = maximumRefreshTokenValiditySeconds;
    }
}

In the above code instead of a fixed Access and Refresh token validity seconds now there are configurable minimum and maximum values. By default it has twice the validity seconds so an Access Token can be valid anywhere from 12 hours to 24 hours and a Refresh Token from 30 days to 60 days to the second. This can eliminate the thundering herd problem as the validity periods are dispersed.

This CustomTokenServices can be made use of in Java Config as following:

@Configuration
@EnableAuthorizationServer
public class OAuth2AuthorizationServerConfigurer extends AuthorizationServerConfigurerAdapter {

    @Autowired
    private AuthenticationManager authenticationManagerBean;

    @Autowired
    private AuthorizationServerTokenServices customTokenServices;

    @Override
    public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
        clients.inMemory().withClient("web").secret("secret").authorizedGrantTypes("password").scopes("read,write");
    }

    @Override
    public void configure(AuthorizationServerEndpointsConfigurer endpoints) throws Exception {
        endpoints.authenticationManager(authenticationManagerBean)
                .tokenServices(customTokenServices) // Registered here
    }
}

And in XML as following:

<oauth2:authorization-server token-services-ref="customTokenServices">

 



Tags: SpringSecurity OAuth2 ThunderingHerd
Views: 601
Register for more exciting articles

Comments

Please login or register to post a comment.


There are currently no comments.