Potential deadlock with Vaadin #12272
Comments
|
I don't think this is a Jetty issue. Vaadin should not grab a lock and then perform a blocking operation like The other 2 stack traces are now waiting on Did you report this issue to Vaadin? |
Yes, I did. I referenced the issue in the beginning of the description.
That was my impression as well. I cannot say for sure that this is what causes the deadlock, but it might be a good idea to avoid this regardless. Do you think the same is true for writing back the response? |
Yes, if a blocking API is used. |
|
Alright, I will forward these suggestions to Vaadin, thanks! I am still interested, do you think that this is really the cause of the deadlock? Specifically, if you look at the threads waiting on the VaadinSession, are those the threads that would eventually release the BlockingContentProducer? Because to me, it looks like when content isn't immediately available, like it seems to be the case here, a read interested is registered in the selector, which should notify HttpInput when content becomes available. Can that really be affected by some request handler being blocked on a lock? I can see some threads are waiting in |
|
A blocker read will be eventually woken up, either when data is available, or when a timeout occurs. The problem is that in both cases the wait can be really long (10s of seconds, minutes or even more), so if something else happens in the system that requires access to the Vaadin session, then these threads will be blocked for a long time. The proper solution is to not perform blocking API calls with locks held. |
Does this mean that you don't see any reason for why either of the two threads waiting for the Vaadin session would prevent the first thread from eventually proceeding? This in turn would imply that shutdown would still be blocked by that first thread regardless of whether that thread holds the Vaadin session lock while reading the request body (except if the client for some reason delays sending more bytes until one of the other threads have made progress)? |
|
...or is this a form of head-of-line blocking with a TCP-level buffer that full of bytes for one of the threads waiting for the lock which prevents the demultiplexer from reaching the bytes that would allow the first thread to proceed? |
|
@Legioth I am not sure I understand your comments. A blocking operation performed with locks held is typically a mistake. In this case, there is nothing that Jetty can do, it's a Vaadin issue. Note that this will happen with any Servlet Container, not just Jetty, and for that matter with any |
|
I understand the theoretical issues with dining philosophers and so on. Holding a lock while doing a blocking operation (e.g. acquiring another lock or blocking I/O) is generally fine when it comes to deadlocks as long as that blocking operation is independent from the lock. That's usually the case when a high-level abstraction calls down to a lower-level abstraction since the lower-level abstraction doesn't have any direct code path that leads to that lock. It's a bit surprising that a HTTP request wouldn't be independent but I guess that's what TPC head-of-line blocking is all about. Could you confirm that this would be the likely underlying reason also from your perspective or is there some other factor at play here as well? If TCP head-of-line blocking is the expected cause, then this issue could be closed on my (and thus also Vaadin's) behalf. I'm not saying that Vaadin's implementation might not be a mistake, but it's then at least a mistake that has taken more than 10 years for anyone to discover 😄. I would just prefer to have a proper understanding of the underlying mechanisms before I start digging into whether anything could be changed on Vaadin's side. |
|
@Legioth I think you have the locking understanding the other way around.
They are.
TCP HoL blocking is just one of the multiple possible cases of blocking, but any blocking would show the problem. I would not concentrate on TCP HoL blocking, as this is a problem at a different level. The API called by Vaadin are blocking, so if the remote client is not sending data then the API will block, no matter whether the data is transported by a protocol that suffers from TCP HoL blocking. I would not close the issue on the Vaadin side without a fix, no matter if it took 10 years to discover. |
|
I see the possibility of TCP HoL blocking as "proof" that there's nothing Jetty could do in this case regardless of whether that's actually what goes on in this specific case. This also means that this issue could be closed (which can only be done by the original reporter or project maintainers) while keeping the Vaadin issue open. But... I'm also curious to gain a more in-depth understanding even though I also realize that it's not your duty to educate me. 😄 My statement about holding locks while calling a lower abstraction is maybe not accurate - it goes for calling any code where you are not sure about what shared resources it might try to use. A typical deadlock has two separate shared resources that are acquired in an inconsistent order. Vaadin's session lock is one of those resources but there also has to be some other shared resource for a deadlock to happen. Treating blocking I/O in general as a shared resource might be a good heuristic but I would like to have something more specific for my mental model. In the case of TCP HoL blocking, that specific other resource is the read buffer of the shared TCP connection which causes seemingly independent HTTP/2 requests to actually depend on each other. What other shared resources could there be that might cause a deadlock between concurrent HTTP requests or responses? |
|
@Legioth this is technically not a deadlock because there is no circular wait, but instead it is just a "hold and wait" situation, which would not happen if I/O reads/writes never block. For HTTP/1.1, the wait in reads/writes is caused by TCP HoL blocking and TCP congestion, respectively. As for your mental model, I/O operations and blocking access to bounded resources (e.g. a connection pool towards a JDBC database, but also adding items to a bounded queue) are the "wait" part of the "hold and wait" situation, and you should avoid to "hold" when performing those "wait" operations.
While it is possible that few requests cause the HTTP/2 session flow control blocking (for reads and writes), or TCP HoL blocking, or TCP flow control blocking, so that other requests on the same connection would depend on the few ones, it does not seem to be what happens here. This is just one request failing to provide the request content, and other requests from the same client waiting on the Vaadin session lock (which I assume be per-client). Other requests from other clients should be able to proceed without problem, so there is no dependency. |
|
Thanks for the clarification. I think we have partially talked across each other based on slightly different initial assumptions. I have been assuming that the first thread was stalled because the bytes it needed were held up by something from one of the two other threads, i.e. that the whole situation could be avoided if either of the two other threads could acquire the session lock and proceed to a point where they would indirectly release the bytes needed by the first thread. It seems like your assumption is that the first thread was stalled because the needed bytes never even reached the server, i.e. that the first stalled thread is inevitable but a better locking strategy in Vaadin could prevent the two other threads from also being stalled. |
I doubt this is the case, looking at the stack traces.
Correct, this seems likely the case from the stack traces. |
|
I realized the same now when looking more closely at the stack traces. Thread 2 is a websocket close frame which shouldn't have any payload that remains to be read while thread 3 is a timer rather than any request handling. |
|
I just realized that we also use virtual threads in some cases, and those will not show up in our handmade thread dump. This might only be remotely relevant for this particular case, since as you said, one of the threads is probably stuck inevitably. Still, it's quite possible that those virtual threads also wait on the VaadinSession, since we use them for background threads that need to access Vaadin components. |
|
@mperktold if you use Jetty 12, you can use |
A WebSocket CLOSE Frame has payload. |
Jetty version(s)
Jetty 12.0.13
Jetty Environment
core, ee10
Java version/vendor
(use: java -version)openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)
OS type/version
Windows 11
Description
This is a repost of vaadin/flow#19938
We have several reports of our application not being able to shutdown. Apparently, some VaadinSessions stay alive and cannot be destroyed. Here are the thread dumps of two such cases:
StackTraces1.txt
StackTraces2.txt
I found some common patterns in these dumps:
One of the threads blocks on a Jetty semaphore while reading the request content of an UIDL request. Note that this thread holds the lock on the VaadinSession while blocking.
A second thread blocks on the VaadinSession while trying to close the websocket:
A third thread also blocks on the VaadinSession while handling a connection loss, but this one comes from the HeartbeatInterception:
I'm not sure where things go wrong, but this does look a bit suspicious to me.
The blocked thread is waiting in BlockingContentProducer.nextChunk until content is available.
From what I see, the semaphore should eventually be released in BlockingContentProducer.onContentAvailable when new content is available, which should be called from HttpInput.run.
I'm not actually seeing a deadlock here, but I find it suspicious that some threads wait on the VaadinSession and another holds it while blocking for some other reason.
How to reproduce?
Unfortunately, I don't have a reproducer. However, we have several reports of this on shutdown, so that might have to do something with it. I hop you can do something with the thread dumps.
The text was updated successfully, but these errors were encountered: