What jstack Misses When Your Java Service Uses Virtual Threads

When a Java service utilizes virtual threads, the traditional jstack tool may not accurately reflect the system's state, particularly if there are numerous requests in flight. For instance, consider a Spring Boot service with a HikariCP pool of 10 connections and around 5,000 requests in progress. Latency issues arise, and executing jstack yields 43 platform threads, primarily scheduler workers waiting in ForkJoinPool.awaitWork, with minimal CPU activity.

However, jstack fails to list virtual threads on JDK 21 through 25, making it difficult to identify the requests waiting for database access.

To capture virtual threads and their associated information, the jcmd command with the Thread.dump_to_file option can be used. This command has been available since JDK 21 and generates a text or JSON file containing thread stacks. Compared to jstack, jcmd provides additional details such as thread ID, name, a virtual marker, state, timestamps, and lock information in JDK 25. It also allows for capturing multiple virtual threads and their parent-child relationships, which can help identify the root cause of the bottleneck.

By analyzing the captured dump files, one can examine the thread stacks and identify the resources causing delays, such as HikariPool.getConnection(). For example, if each request holds a connection for approximately 18 milliseconds, ten connections could take around nine seconds to clear a backlog of 5,000 requests, assuming no new requests are arriving.

This information can guide further investigation into slow queries, transaction scope, and potential database performance issues. Ultimately, combining the dump analysis with system metrics can provide a comprehensive understanding of the performance issues and help optimize the service's resource utilization.