In March 2021, I observed troubling behavior in multiple applications I supported that are built using Spring Boot: they would occasionally stop responding. Eventually, I tracked down the root cause to a DoS (Denial of Service) vulnerability in Spring Security OAuth 2.0: a simple shell script could take down any affected web application. Respecting the practice of responsible disclosure, I reported the vulnerability which resulted in CVE-2021-22119: Denial-of-Service (DoS) attack via initiation of Authorization Request in Spring Security OAuth 2.0 Client Web and WebFlux Application. In the interest of speedy resolution to this high risk vulnerability, I immediately started to work with the project to fix the problem. In June 2021, Spring Security 5.5.1, 5.4.7, 5.3.10, and 5.2.11 were released with the fix for this vulnerability; Spring Boot users should upgrade to 2.5.2 or 2.4.8.
The Initial Outage
The tale begins with the report of an application outage. The cause of the outage is traced to the database server being out of disk space; the disk is full of MySQL Binary Logs.
MySQL Binary Logs
MySQL Binary Logs record all SQL that writes data. For example, it includes all UPDATE, DELETE, and INSERT statements. These logs are typically used for cluster replication.
In this case, the MySQL server was configured to persist all binary logs forever, so the log disk usage grew unbounded. The remediation is to configure MySQL to purge binary logs that are too old or they use too much disk space. This remediation is particularly logical since MySQL replication isn’t being used in this case, and even if it was, it would only need a finite amount of recent binary logs to function.
However, there is still an outstanding question: why was so much write-type SQL being executed that the binary logs grew to such size? Examination of the logs revealed many INSERT/DELETE/UPDATEs performed on the SPRING_SESSION table.
Spring Session JDBC
The SPRING_SESSION table is used by Spring Session JDBC, which is way to provide session replication so application servers can be stateless. By default, servlet session data is stored in the application server’s memory, meaning that if there are multiple servers running the application, each one has different session data. In practice, for example, that means if a user logs in via a request fulfilled by one server, and a subsequent request is routed to a different server, that different server won’t see the user as being logged in. By storing session data in a database as Spring Session JDBC does, all application servers will have the same session data eliminating that problem.
Since the SPRING_SESSION table is seeing an unexpected volume of write activity, something must be causing lots of sessions being created or updated. But what?
Out of Memory
With the MySQL Binary Logs now trimmed to a reasonable size, the issue was mitigated for the moment… but not for long. After a bit more time, the application servers started going down. Health monitoring brought them back up, eventually, but these intermittent outages are very problematic. Monitoring showed that the application servers were running out of memory, with the Java process consuming many gigabytes of RAM until it finally crashed.
By taking and studying a heap dump, the cause of the out of memory condition was identified, and it was almost entirely session data. Specifically, it was instances of the class org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest.
A Whole Lot of OAuth2AuthorizationRequests
org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest stores the information necessary for a user to login to the application via an OAuth 2.0 provider (ex, GitHub, Google, Okta). Each time a user tries to log in, Spring Security OAuth2 Client creates a new instance of OAuth2AuthorizationRequest and persists it to the session.
Identification and Reproduction
After much experimentation, I discovered that the issue could be reproduced by logging in then idling, doing nothing. After a while, the browser’s developer tool’s network monitor would show many XHR (AJAX) requests, and debugging revealed that each request wrote to the session. This application used sockjs-client to manage a WebSocket-type of connection (actual WebSockets wasn’t possible), emulating WebSockets using XHR. The server didn’t consider these requests as keeping the session alive, so a user idling on this page will eventually have their session expire, logging them out. Then, sockjs-client will trying to establish a new WebSocket-type connection, and that HTPTP request will be recognized as not being logged in, causing the server to attempt to log them in using OAuth, which creates and persists a OAuth2AuthorizationRequest. Seconds later, since it encountered an error (the user wasn’t logged in), sockjs-client tries again, and again, and again… each time resulting in a new OAuth2AuthorizationRequest being persisted.
The combination of sockjs-client + Spring Security should not behave in this way. I submitted a pull request to sockjs-client which sets the Accept header appropriately which will cause Spring Security to not attempt to login such requests, which would eliminate this issue. However, even in that case, it would still be possible for another bug, or a malicious user, to cause this issue.
Finally, it was time to create a minimal test case.
The Minimal Test Case
To reproduce the issue:
- Create a simple Spring Boot application. Use https://start.spring.io/ to create one using “OAuth2 Client”. Make sure to use a vulnerable version of Spring Boot if you want to actually see the issue.
- Configure the application to authenticate via oAuth2 as documented at https://docs.spring.io/spring-security/site/docs/current/reference/html5/#oauth2login-custom-provider-properties For reference, I’ll refer to the configured provider as “oauthprovider”. Here’s an example configuration (values must be changed as appropriate):
spring.security.oauth2.client.registration.oauthprovider.authorization-grant-type=authorization_code
spring.security.oauth2.client.registration.oauthprovider.provider=oauthprovider
spring.security.oauth2.client.registration.oauthprovider.client-id=VAUE-HERE
spring.security.oauth2.client.registration.oauthprovider.client-secret=VALUE-HERE
spring.security.oauth2.client.registration.oauthprovider.redirect-uri={baseUrl}/login/oauth2/code/{registrationId}
spring.security.oauth2.client.registration.oauthprovider.scope=user_info,profile,openid
spring.security.oauth2.client.provider.oauthprovider.user-name-attribute=sub
spring.security.oauth2.resourceserver.jwt.jwk-set-uri=VALUE-HERE
spring.security.oauth2.client.provider.oauthprovider.issuer-uri=VALUE-HERE
- Run the application
Now to experience the DoS issue:
- Use CURL to make request:
curl http://localhost/oauth2/authorization/oauthprovider -i -v - Note the session cookie name and value. In my case, it’s:
JSESSIONID=MjRiNWZiNDctN2E3Ny00OGRkLWJhMjEtZTI4MDQ1NGU4YTRl - Use curl to make many requests with the session cookie to this URL in a loop:
while true; do curl http://localhost/oauth2/authorization/oauthprovider --cookie "JSESSIONID=MjRiNWZiNDctN2E3Ny00OGRkLWJhMjEtZTI4MDQ1NGU4YTRl" -i -v; done
You’ll notice the JVM’s memory usage increases. Eventually, it will run out of memory. Note that there is no database involved, no Spring Session configuration, etc – those were all components I looked at while discovering this issue, they are not causes of it.
Responsible Disclosure
Responsible disclosure is all about giving the project time to mitigate the vulnerability before the vulnerability is disclosed. Without responsible disclosure, a vulnerability would be made public before the project in which the vulnerability exists even knows about it. Therefore, that vulnerability can be used instantly by malicious parties to attack targets that are completely unaware and without a mitigation available. In this case, it doesn’t seem right to tell the world how to very effectively DoS attack any Spring Security OAuth2 powered service before a fix is available.
Following the principles of responsible disclosure, I searched for how to privately contact Spring. I couldn’t find anything on their web site or in their GitHub repositories. I did, however, find Spring’s page on HackerOne which says to notify security@pivotal.io via email.
On April 5, 2021, I emailed security@pivotal.io. I never received a response.
In desperation, on April 9, I decided to email security@vmware.com since VMWare is the parent company of Pivotal. I immediately got a response. They acknowledged the issue tracking it internally as VSRC-4437.
Fixing the Issue
In March, before I fully appreciated the security implications of this issue, I created my first pull request “Expire OAuth2AuthorizationRequest when saving to the session”. Through collaboration with the Spring Security project, the approach changed, resulting in pull request “HttpSessionOAuth2AuthorizationRequestRepository storing one OAuth2AuthorizationRequest” which was merged on May 12, 2021.
Closing Thoughts
- Applications should have monitoring in place for metrics such as disk usage with alerts raised well before outages occur.
- Ensure that services that persist data don’t save infinite amounts of data like MySQL did in this case.
- Even widely used, well written, well tested, well maintained software can have vulnerabilities.
- Spring/Pivotal/VMWare should improve their responsible disclosure documentation, making sure that the email address provided is monitored, and that this information is readily available.
- Thank you to the Spring Security team for working with on this issue, specifically Joe Grandja, who tolerated me over the weeks it took to finally close out this effort.
Identifying, Reporting, and Fixing CVE-2021-22119: DoS Vulnerability in Spring Security OAuth 2.0 by Craig Andrews is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
