Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vert.x stress test problem after 16xxx request sent #5255

Open
winter-lau opened this issue Jul 22, 2024 · 8 comments
Open

Vert.x stress test problem after 16xxx request sent #5255

winter-lau opened this issue Jul 22, 2024 · 8 comments
Labels

Comments

@winter-lau
Copy link

Questions

Vert.x stress test problem after 16xxx request sent

Version

4.5.x

Steps to reproduce

Server code:
`
package demo;

import io.vertx.core.AbstractVerticle;
import io.vertx.core.Vertx;
import io.vertx.core.http.HttpServer;
import io.vertx.core.http.HttpServerResponse;

public class SimpleHTTPServer extends AbstractVerticle {

@OverRide
public void start() {
HttpServer server = vertx.createHttpServer();

server.requestHandler(request -> {
HttpServerResponse response = request.response();
response.end("Hello, World!");
});

server.listen(8080, result -> {
if (result.succeeded()) {
System.out.println("Server started on port 8080");
} else {
System.err.println(STR."Server failed to start: {result.cause()}");
}
});
}

public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new SimpleHTTPServer());
}
}
`

  1. Test method:

we use ApacheBench to test, Here is the results:

  1. first round:

% ab -c 100 -n 10000 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

  1. second round:

% ab -c 100 -n 10000 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests

The test hung here.

We have test multi round, the program hung each time when 16xxx requests sent to http service

Extra

MacBook Pro with M1 Pro/16G mem/Java 21.0.2
Vert.x 4.5.x or any older 4.x version

@winter-lau winter-lau added the bug label Jul 22, 2024
@vietj
Copy link
Member

vietj commented Jul 22, 2024

what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ?

@winter-lau
Copy link
Author

what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ?

the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)"

i don't tune OS

@vietj
Copy link
Member

vietj commented Jul 22, 2024 via email

@winter-lau
Copy link
Author

can you verify you are not running out of file descriptors or port ?

On Mon, Jul 22, 2024 at 12:23 PM Winter Lau @.> wrote: what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ? the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)" i don't tune OS — Reply to this email directly, view it on GitHub <#5255 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCRYIFHF4SBFNOHBPFDZNTMSLAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGYYTINJUG4 . You are receiving this because you commented.Message ID: @.>

There are no error messages during the hang, and the issue automatically returns to normal after waiting a few seconds.

The validator code is very simple, you can try it and reproduce it 100%.

@vietj
Copy link
Member

vietj commented Jul 22, 2024 via email

@winter-lau
Copy link
Author

What more information can I provide? I've reproduced the issue on both my and my colleague's macs, the computers are the default system and haven't been tuned with any parameters

@winter-lau
Copy link
Author

In my windows11 laptop, the above issue doesn't happen

@surajkumar
Copy link

surajkumar commented Sep 8, 2024

@winter-lau if you're running your test using the same machine then it will not work. A common mistake that new developers make is attempting to benchmark a server using a client on the same machine. Your NIC may not be able to handle it or the OS is not able to support it. The server not responding is because the server does not receive anything from the underlying system.

Additionally, no single server can handle the volumes you are trying to test. In most real world scenarios, each server usually handles around 1000 connections (or % based depending on utilization of your system). This is when you need to start scaling.

If you want accurate results, consider using seperate machines or a cloud based solution designed for testing your traffic.

I am basing this on an assumption due to the lack of information provided in your issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants