Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Agent loses registration with server, does not re-attempt #295

Open
andrewazores opened this issue Dec 6, 2023 · 1 comment
Open

[Bug] Agent loses registration with server, does not re-attempt #295

andrewazores opened this issue Dec 6, 2023 · 1 comment
Labels
bug Something isn't working question Further information is requested

Comments

@andrewazores
Copy link
Member

Recently we observed a pair of Agent-equipped applications, deployed in OpenShift along with Cryostat,which after ~8 hours, lost their registration with the server and did not try to re-register. The logs were messy and inconclusive: the root cause may have been due to the cryostat.agent.authorization Bearer token expiring, or it may have been other networking related issues and hiccups. In any case, it indicates that the Agent should be more resilient to more failure modes, and "if in doubt, register again".

2023-12-06 04:11:54:174 +0000 [cryostat-agent-worker-2] ERROR io.cryostat.agent.CryostatClient - Failed to get credentials
java.util.concurrent.CompletionException: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:461)
	at io.cryostat.agent.CryostatClient.lambda$supply$28(CryostatClient.java:453)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:459)
	... 8 more
Caused by: java.net.SocketTimeoutException: Connect timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at io.cryostat.agent.shaded.org.apache.shaded.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 19 more
2023-12-06 04:11:54:175 +0000 [cryostat-agent-worker-2] ERROR io.cryostat.agent.WebServer - Could not submit credentials
java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.CryostatClient.lambda$queryExistingCredentials$10(CryostatClient.java:219)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.CompletionException: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:461)
	at io.cryostat.agent.CryostatClient.lambda$supply$28(CryostatClient.java:453)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 6 more
Caused by: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:459)
	... 8 more
Caused by: java.net.SocketTimeoutException: Connect timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at io.cryostat.agent.shaded.org.apache.shaded.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 19 more
2023-12-06 04:11:54:175 +0000 [cryostat-agent-worker-2] ERROR io.cryostat.agent.Registration - Failed to generate credentials
java.util.concurrent.CompletionException: Could not submit credentials
	at io.cryostat.agent.WebServer.lambda$generateCredentials$1(WebServer.java:140)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.CryostatClient.lambda$queryExistingCredentials$10(CryostatClient.java:219)
	... 10 more
Caused by: java.util.concurrent.CompletionException: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:461)
	at io.cryostat.agent.CryostatClient.lambda$supply$28(CryostatClient.java:453)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 6 more
Caused by: io.cryostat.agent.shaded.org.apache.shaded.http.conn.ConnectTimeoutException: Connect to clustercryostat-sample.openshift-console.svc:8181 [clustercryostat-sample.openshift-console.svc/172.30.196.125] failed: Connect timed out
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:459)
	... 8 more
Caused by: java.net.SocketTimeoutException: Connect timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at io.cryostat.agent.shaded.org.apache.shaded.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
	at io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 19 more
2023-12-06 04:11:59:003 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: UNREGISTERED
2023-12-06 04:11:59:003 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.CryostatClient - GET https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials HTTP/1.1
2023-12-06 04:11:59:040 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - GET https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials : 200
2023-12-06 04:11:59:042 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials HTTP/1.1
2023-12-06 04:11:59:088 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials : 201
2023-12-06 04:11:59:089 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - Defined credentials with id 9
2023-12-06 04:11:59:089 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: REFRESHING
2023-12-06 04:11:59:089 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery HTTP/1.1
2023-12-06 04:11:59:114 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.WebServer - GET /
2023-12-06 04:11:59:115 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.WebServer - Using 'deflate' encoding
2023-12-06 04:11:59:115 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.WebServer - GET / : 204 0ms
2023-12-06 04:11:59:122 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery : 201
2023-12-06 04:11:59:123 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.Registration - Registered as 8b8d0903-8bdb-4b76-ad9c-55829de580d2
2023-12-06 04:11:59:124 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.Registration - publishing self as [http://10.128.2.90:9977, service:jmx:rmi:///jndi/rmi://agent-test-55df9cbd79-zdzkm:9097/jmxrmi]
2023-12-06 04:11:59:124 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery/8b8d0903-8bdb-4b76-ad9c-55829de580d2?token=eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..8zgv7nG_nBGPX5bY.k3R00UhATLOAcWxQ6clOcS706yGqgJ2XgNvu-WsffY4jw8uQgWSmoOcCwf8i5ddo3jzjBa9magFXDO_ZmEWr__QyE5IlOVvv4q-_a7ZzVRZAC8C7jDe-MA915deuhTps30SPC92RylBenXVyB2_Sh0ovqpDe79DraNJaEJLSr2kKbS3g1Uj8ncauuTKC_1WY5mA7nalz7Bl3TouTeziYephjN4a7brVEARFaIkU0WVzmdTQBqCrPRPtX0Iq5loCB6alev1WC-k8CzFl78mQ8Eb6RQmvSgYbB0BJAMtFU-U9BFFI9y2AgkvAz3HJbe9S7v2U5hL-Fu6mj7VP9MaqOKqBl5YANLWJZpLGegEl4RrnkItFB_JPn8mnp9ap03YU_9kgz7qwBzGDlfJ-Zyr6b5mfgBvvzR68dQCB_Iu46BgcEuiB6rH-OhJA1aMGXfX217I3hzmPd4lduQM80ufijqhW_dadlyBh7BPuualVxvoQLrtlnOcNryV48ahBqbCFwRM0gCzUp8u6ccip-dy9fIDHLP3x82PbeBPOJc6_5_fQuM0ptnVjcLoz26wiQK8Cj7NpcHkty3tegb3qGP4HMxWbYz_othNCt7LyLQ7FOtD0z47GydLrIapNFXRdFXXSbsA_f1UTafDHQr9WM0IzgAmI6mMU90y8ZGT2kgUTPI7TIbyxckwzTm7m2bXFu-8Mg7kKlmSVG8rf-QySKbOUulytAXzKHZcMuIeThoniE5GHjFaCgZEgtCbnIle4vZeeeP0pEqZZzwSN6pn0V97u-9gViBJhMOlnxvk4D7kGoZbueVUrcD9NkXj0VWTd1IKqepbMTTA7rBHMaqqXRHJvHWFkWndJPij3E0iBiiI77-ynArf-x1svC9FSSJvRmGbgZuMgX3ThWTPLR8PVGPjEEnaRb_VTyvupDExgRG56mIGV4eBhTZqYhsY9D5GxwHiOj-B7SWuh4YgrYyrCAM1mfg1fRraI50NdLz8nbAKK2Y-BohfzGSFwQKuF9hxFYs4jWV6Zcp-rcPv6jNrd_Yu0ZLZX6_lyN-ZvdHOnku2CYndlS2yK6ptBAyxxqiD_w7AfaTjysWlySsk8DOrTE6YT5x9Myz2tAB1LLqPLRuvA1W0hK7-8LiEUguug.vkOWDiJipLkSNuGDl5DjUQ HTTP/1.1
2023-12-06 04:11:59:167 +0000 [cryostat-agent-worker-2] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery/8b8d0903-8bdb-4b76-ad9c-55829de580d2?token=eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..8zgv7nG_nBGPX5bY.k3R00UhATLOAcWxQ6clOcS706yGqgJ2XgNvu-WsffY4jw8uQgWSmoOcCwf8i5ddo3jzjBa9magFXDO_ZmEWr__QyE5IlOVvv4q-_a7ZzVRZAC8C7jDe-MA915deuhTps30SPC92RylBenXVyB2_Sh0ovqpDe79DraNJaEJLSr2kKbS3g1Uj8ncauuTKC_1WY5mA7nalz7Bl3TouTeziYephjN4a7brVEARFaIkU0WVzmdTQBqCrPRPtX0Iq5loCB6alev1WC-k8CzFl78mQ8Eb6RQmvSgYbB0BJAMtFU-U9BFFI9y2AgkvAz3HJbe9S7v2U5hL-Fu6mj7VP9MaqOKqBl5YANLWJZpLGegEl4RrnkItFB_JPn8mnp9ap03YU_9kgz7qwBzGDlfJ-Zyr6b5mfgBvvzR68dQCB_Iu46BgcEuiB6rH-OhJA1aMGXfX217I3hzmPd4lduQM80ufijqhW_dadlyBh7BPuualVxvoQLrtlnOcNryV48ahBqbCFwRM0gCzUp8u6ccip-dy9fIDHLP3x82PbeBPOJc6_5_fQuM0ptnVjcLoz26wiQK8Cj7NpcHkty3tegb3qGP4HMxWbYz_othNCt7LyLQ7FOtD0z47GydLrIapNFXRdFXXSbsA_f1UTafDHQr9WM0IzgAmI6mMU90y8ZGT2kgUTPI7TIbyxckwzTm7m2bXFu-8Mg7kKlmSVG8rf-QySKbOUulytAXzKHZcMuIeThoniE5GHjFaCgZEgtCbnIle4vZeeeP0pEqZZzwSN6pn0V97u-9gViBJhMOlnxvk4D7kGoZbueVUrcD9NkXj0VWTd1IKqepbMTTA7rBHMaqqXRHJvHWFkWndJPij3E0iBiiI77-ynArf-x1svC9FSSJvRmGbgZuMgX3ThWTPLR8PVGPjEEnaRb_VTyvupDExgRG56mIGV4eBhTZqYhsY9D5GxwHiOj-B7SWuh4YgrYyrCAM1mfg1fRraI50NdLz8nbAKK2Y-BohfzGSFwQKuF9hxFYs4jWV6Zcp-rcPv6jNrd_Yu0ZLZX6_lyN-ZvdHOnku2CYndlS2yK6ptBAyxxqiD_w7AfaTjysWlySsk8DOrTE6YT5x9Myz2tAB1LLqPLRuvA1W0hK7-8LiEUguug.vkOWDiJipLkSNuGDl5DjUQ : 200
2023-12-06 04:11:59:168 +0000 [cryostat-agent-worker-2] INFO io.cryostat.agent.Registration - Publish success
2023-12-06 04:11:59:168 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: REGISTERED
2023-12-06 04:11:59:168 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.insights.InsightsAgentHelper - Starting Red Hat Insights client
2023-12-06 04:11:59:168 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Started Red Hat Insights client
2023-12-06 04:11:59:169 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: PUBLISHED
2023-12-06 04:11:59:169 +0000 [cryostat-agent-harvester] INFO io.cryostat.agent.harvest.Harvester - Template not specified
2023-12-06 04:11:59:169 +0000 [cryostat-agent-harvester] INFO io.cryostat.agent.harvest.Harvester - JFR Harvester starting
2023-12-06 04:11:59:169 +0000 [cryostat-agent-harvester] INFO io.cryostat.agent.harvest.Harvester - JFR Harvester started, periodic uploads disabled (period -1 < 0)
2023-12-06 04:11:59:175 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: UNREGISTERED
2023-12-06 04:11:59:175 +0000 [cryostat-agent-harvester] INFO io.cryostat.agent.harvest.Harvester - Harvester stopping
2023-12-06 04:11:59:175 +0000 [cryostat-agent-harvester] INFO io.cryostat.agent.harvest.Harvester - Harvester stopped
2023-12-06 04:11:59:175 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.CryostatClient - GET https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials/9 HTTP/1.1
2023-12-06 04:11:59:208 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.CryostatClient - GET https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/credentials/9 : 200
2023-12-06 04:11:59:210 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.WebServer - Defined credentials with id 9
2023-12-06 04:11:59:210 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: REFRESHING
2023-12-06 04:11:59:210 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery HTTP/1.1
2023-12-06 04:11:59:239 +0000 [cryostat-agent-worker-2] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery : 201
2023-12-06 04:11:59:245 +0000 [cryostat-agent-worker-2] INFO io.cryostat.agent.Registration - Registered as 8b8d0903-8bdb-4b76-ad9c-55829de580d2
2023-12-06 04:11:59:245 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: REFRESHED
2023-12-06 04:16:52:258 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - POST /
2023-12-06 04:16:52:259 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - Using 'deflate' encoding
2023-12-06 04:16:52:259 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - POST / : 401 0ms

2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.headers - http-outgoing-606 << HTTP/1.1 201 Created
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.headers - http-outgoing-606 << location: /api/v2.2/discovery/34d3a1aa-a3d5-44b8-962c-7b3480a7197e
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.headers - http-outgoing-606 << content-type: application/json
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.headers - http-outgoing-606 << content-encoding: gzip
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.headers - http-outgoing-606 << content-length: 1178
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.CryostatClient - POST https://clustercryostat-sample.openshift-console.svc:8181/api/v2.2/discovery : 201
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec - Cancelling request execution
2023-12-06 04:12:09:379 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-606: Shutdown connection
2023-12-06 04:12:09:380 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.impl.execchain.MainClientExec - Connection discarded
2023-12-06 04:12:09:380 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 606][route: {s}->https://clustercryostat-sample.openshift-console.svc:8181][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2023-12-06 04:12:09:380 +0000 [cryostat-agent-worker-1] INFO io.cryostat.agent.Registration - Registered as 34d3a1aa-a3d5-44b8-962c-7b3480a7197e
2023-12-06 04:12:09:380 +0000 [cryostat-agent-registration] INFO io.cryostat.agent.Agent - Registration state: REFRESHED
2023-12-06 04:12:10 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot collect disk usage data because plugin CloudBees Disk Usage Simple is not installed: java.lang.NoClassDefFoundError: com/cloudbees/simplediskusage/QuickDiskUsagePlugin
2023-12-06 04:14:10 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot collect disk usage data because plugin CloudBees Disk Usage Simple is not installed: java.lang.NoClassDefFoundError: com/cloudbees/simplediskusage/QuickDiskUsagePlugin
2023-12-06 04:16:10 WARNING org.jenkinsci.plugins.prometheus.DiskUsageCollector collect Cannot collect disk usage data because plugin CloudBees Disk Usage Simple is not installed: java.lang.NoClassDefFoundError: com/cloudbees/simplediskusage/QuickDiskUsagePlugin
2023-12-06 04:17:02:366 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - POST /
2023-12-06 04:17:02:366 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - Using 'deflate' encoding
2023-12-06 04:17:02:366 +0000 [cryostat-agent-worker-0] INFO io.cryostat.agent.WebServer - POST / : 401 0ms

Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Using stored credentials id:8 referenced in ping callback userinfo
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Using stored credentials id:9 referenced in ping callback userinfo
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: POST http://storedcredentials:[email protected]:9977 status 401: Unauthorized
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"TargetJvmDiscovery","type":{"type":"application","subType":"json"}},"message":{"event":{"serviceRef":{"jvmId":"UK6E0ZmDZ-8CTyX84SLAqK4Gk0CyzAZt45rx6VgE3Wc=","connectUrl":"http://10.129.2.118:9977","alias":"agent-test-jenkins","labels":{},"annotations":{"platform":{"INSTANCE_ID":"a6ec9de7-6f5b-4739-8499-c7e19bff4ec4"},"cryostat":{"HOST":"agent-test-jenkins-5d468fd979-h45hx","PORT":"9977","JAVA_MAIN":"/usr/lib/jenkins/jenkins.war","PID":"14","START_TIME":"1701806492","REALM":"agent-test-jenkins"}}},"kind":"LOST"}}}
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Stale discovery service http://storedcredentials:[email protected]:9977 removed
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Removing cached connection for service:jmx:rmi:///jndi/rmi://10-128-2-60.achtung-baby.pod:9093/jmxrmi: EXPIRED
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: POST http://storedcredentials:[email protected]:9977 status 401: Unauthorized
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://10-128-2-60.achtung-baby.pod:9093/jmxrmi closed
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"TargetJvmDiscovery","type":{"type":"application","subType":"json"}},"message":{"event":{"serviceRef":{"jvmId":"H29te8-bqj6zBftWiCw3RtP3x1BzQuGL9TIM_b2_hWg=","connectUrl":"http://10.128.2.90:9977","alias":"agent-test","labels":{},"annotations":{"platform":{"INSTANCE_ID":"8c323eea-9547-4041-b64c-dd7beb2c3489"},"cryostat":{"HOST":"agent-test-55df9cbd79-zdzkm","PORT":"9977","JAVA_MAIN":"/deployments/quarkus-run.jar","PID":"1","START_TIME":"1701803115","REALM":"agent-test"}}},"kind":"LOST"}}}
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Stale discovery service http://storedcredentials:[email protected]:9977 removed
Dec 06, 2023 4:16:53 AM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"TargetJvmDiscovery","type":{"type":"application","subType":"json"}},"message":{"event":{"serviceRef":{"jvmId":"H29te8-bqj6zBftWiCw3RtP3x1BzQuGL9TIM_b2_hWg=","connectUrl":"service:jmx:rmi:///jndi/rmi://agent-test-55df9cbd79-zdzkm:9097/jmxrmi","alias":"agent-test","labels":{},"annotations":{"platform":{"INSTANCE_ID":"8c323eea-9547-4041-b64c-dd7beb2c3489"},"cryostat":{"HOST":"agent-test-55df9cbd79-zdzkm","PORT":"9097","JAVA_MAIN":"/deployments/quarkus-run.jar","PID":"1","START_TIME":"1701803115","REALM":"agent-test"}}},"kind":"LOST"}}}
@andrewazores andrewazores added bug Something isn't working question Further information is requested labels Dec 6, 2023
@ebaron
Copy link
Member

ebaron commented Dec 6, 2023

If we're still using OpenShift OAuth for 3.0, one possible improvement is to use the service account token under /var/run/secrets/kubernetes.io/serviceaccount if no authorization config property is defined. This token is automatically refreshed I believe. The only drawback is that users would have to use an elevated service account for their workloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andrewazores @ebaron