AsyncMiddleManServlet response flushing

[iiliev2]

Jetty 12
ee8

Java 21

Continuing the discussion from #12294 (comment) as this is a separate topic.

The proxy servlets(in particular AsyncMiddleManServlet, as that is what I am looking at) do not seem to flush responses as the proxy client receives new bytes. Ideally it should auto flush in an optimal way(for ex. if http1.1 and chunked, it should flush on each full chunk). Otherwise there could be huge delays between when the data is available and when it is actually returned to the caller(due to buffering). Since this is a proxy, I expected this to be the default, otherwise I hope to be able to set it up that way.

The ContentTransformer does not seem to provide a way to control this(suggested in the previous github question). That abstraction is about mutating the raw data in some way.

The only way that I can see from the code of AsyncMiddleManServlet is to call flush on the ServletOutputStream right after ProxyWriter calls writeProxyResponseContent. Unfortunatly that method is package private so the only way is to override the entire onWritePossible and flush afterward.

What would you advise is the right way to do this?

[sbordet]

AsyncMiddleManServlet is a proxy servlet dedicated to mutate content.

If you don't need to mutate content, don't use it, and use instead ProxyHandler (recommended) or AsyncProxyServlet.

As for AsyncMiddleManServlet, did you actually follow the advice I gave in the comment?
Does it not work?
If it does not work, we need details, because it is supposed to work like you expect.

[iiliev2]

As for AsyncMiddleManServlet, did you actually follow the advice I gave in the comment?

AsyncMiddleManServlet already uses a default(no-op) ContentTransformer. I do not understand how overriding that will help me for this issue.

Does it not work?

I have created a unit test which illustrates the issue. It does not work.
Uncomment this to fix the test. Is this a bug?

[sbordet]

It's not a bug, perhaps a missing feature.

AsyncMiddleManServlet performs a ServletOutputStream.write() which may be buffered, and there is no explicit API in AsyncMiddleManServlet to allow applications to force the flush of the bytes.

Your current workaround would be to call HttpServletResponse.setBufferSize(<small-value>), so that your writes will not be buffered.

There actually exist AsyncMiddleManServlet.writeProxyResponseContent() but it is package-private.
If it was at least procteted, you could override it, call super and then call output.flush().

So your final solution would be:

class MyAMMS extends AsyncMiddleManServlet {
  protected void writeProxyResponseContent(ServletOutputStream output, ByteBuffer content) throws IOException {
    super.writeProxyResponseContent(output, content);
    if (output.isReady()) {
      output.flush();
    }
  }
}

@iiliev2 do you want to make this contribution?
That is, to create a PR making writeProxyResponseContent() to be protected.

[iiliev2]

output.flush();

I found this old thread where you are discussing how to do async flush:

You have to treat it exactly as a write, by calling isReady() before calling flush()

I can't find any information explaining why that is. Could you please clarify this? I don't see this being part of the servlet spec. I see in the implementation of HttpOutput.isReady that it mutates some state(so it's more like becomeReady).

Do I have to call isReady()? What happens if it returns false? Will it wait for the next chunk to arrive, or it will be notified by the framework when it becomes ready(I think that is what should happen). Currently my workaround(which you said I should not do) occasionally hits an error(without calling isReady). Will this not happen if we could override writeProxyResponseContent?:

java.lang.IllegalStateException: isReady() not called: s=OPEN,api=ASYNC,sc=false,e=null
    at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:712)
    at com.iviliev.Servlet$1.onWritePossible(Servlet.java:78)
    at org.eclipse.jetty.server.HttpOutput.run(HttpOutput.java:1494)
    at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1469)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:597)
    at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
    at 

Apart from that, I have added an additional test in my repo testChunkedTransferAbort to verify that if the client aborts while the streaming is going on, the proxy will kill its proxy client request in a reasonable and predictable amount of time.

All tests are now also executed both via http1.1 and http2.

For http1.1 the flushing is also needed for the new test to succeed. Otherwise the proxying continues even though the client may be long gone.
For http2 the new test works even without the flushing. Do you know why is that? Is it part of the protocol when one side aborts?

[sbordet]

In the email thread you linked, @gregw suggested that flushes are asynchronous, so yes you have to call isReady().

This is because if you did an asynchronous write, and the write did not complete, then isReady() would return false (and schedule for onWritePossible() to be called when the write completes).

If you perform the sequence:

write(large); // does not complete, isReady==false
flush(); // Throws IllegalStateException

I have correct the above comment with the right code, and yes you are occasionally hitting exactly this problem -- sorry I suggested you the incorrect solution.

If the call to isReady() before flush() returns false it means that the write has been initiated but has not yet finished.
For your case, this is fine because the write to the network is actually in progress.

When an HTTP/1 client aborts, the server is typically not informed. Only by forcing a write the server can detect that the connection is broken.

On the contrary, with HTTP/2 the server is informed that the request has been canceled, so it may cancel the request even when no writes are in progress.

Just to reiterate: I suggest you do not override ProxyWriter.onWritePossible().
Instead, either reduce the response buffer size, or contribute a PR to make writeProxyResponseContent() protected (and override that), or wait for us to do that.