Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Trino-python-client] query failures on v454 Trino with Ingress-HAP as proxy in the middle #480

Open
vinay-kl opened this issue Sep 3, 2024 · 1 comment
Open

[Trino-python-client] query failures on v454 Trino with Ingress-HAP as proxy in the middle #480

vinay-kl opened this issue Sep 3, 2024 · 1 comment
Assignees
@damian3031

Comments

@vinay-kl
Copy link

vinay-kl commented Sep 3, 2024 •
edited
Loading

Queries submitted via Trino-python-client started failing when we migrated to Trino v454 version with no changes on client-side or server side, but the same was working as expected on our different version clusters v474 & v448

/Users/300070018/PycharmProjects/trino-python-client-main/venv/bin/python /Users/300070018/PycharmProjects/trino-python-client-main/local-tests/local-test.py 
Traceback (most recent call last):
  File "/Users/300070018/PycharmProjects/trino-python-client-main/local-tests/local-test.py", line 12, in <module>
    cur.execute("<query>")
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/dbapi.py", line 589, in execute
    self._iterator = iter(self._query.execute())
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 805, in execute
    status = self._request.process(response)
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 606, in process
    self.raise_response_error(http_response)
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 597, in raise_response_error
    raise exceptions.HttpError(
trino.exceptions.HttpError: error 400: b'
<html>

<head>
	<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1" />
	<title>Error 400 Invalid Authority</title>
</head>

<body>
	<h2>HTTP ERROR 400 Invalid Authority</h2>
	<table>
		<tr>
			<th>URI:</th>
			<td>https:/v1/statement</td>
		</tr>
		<tr>
			<th>STATUS:</th>
			<td>400</td>
		</tr>
		<tr>
			<th>MESSAGE:</th>
			<td>Invalid Authority</td>
		</tr>
	</table>
	<h3>Caused by:</h3>
	<pre>org.eclipse.jetty.http.BadMessageException: 400: Invalid Authority
	at org.eclipse.jetty.http.HttpCompliance.assertAllowed(HttpCompliance.java:421)
	at org.eclipse.jetty.http.HttpCompliance.checkHttpCompliance(HttpCompliance.java:407)
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.onRequest(HttpStreamOverHTTP2.java:97)
	at org.eclipse.jetty.http2.server.internal.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:145)
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:102)
	at org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1119)
	at org.eclipse.jetty.http2.server.internal.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
	at org.eclipse.jetty.http2.HTTP2Connection.onHeaders(HTTP2Connection.java:239)
	at org.eclipse.jetty.http2.parser.BodyParser.notifyHeaders(BodyParser.java:115)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.onHeaders(HeadersBodyParser.java:269)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.onHeaders(HeadersBodyParser.java:264)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.parse(HeadersBodyParser.java:208)
	at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
	at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
	at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:342)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:210)
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:442)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.MonitoredQueuedThreadPool$1.run(MonitoredQueuedThreadPool.java:73)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1570)
	Suppressed: org.eclipse.jetty.util.StaticException: Unconsumed request content
</pre>

</body>

</html>'

Our N/W stack is
client -> Ingress/HAP-> backend/server

Upon further investigation slack-thread: https://trinodb.slack.com/archives/CGB0QHWSW/p1724955298573889

We found out the following

These are the following steps which can be performed to make it work

  • within the requests lib of adapters.py class file, under send function of class HTTPAdapter(BaseAdapter):, changing the line to url = request.url makes it work as it internally specifies the url with absolute one instead of relative path
DEBUG:urllib3.connectionpool:https://host:443 "POST https://host:443/v1/statement HTTP/1.1"
  • Or make an explicit entry in Ingress Config to add http-header
haproxy.org/backend-config-snippet: |
      http-request set-uri https://%[req.hdr(host)]%[pathq]
  • Or add the following in Ingress YAML, which enforces HTTP/1.1 protocol by disabling default alpn h2,http1.1 negotiation
haproxy.org/server-proto: h1
@hashhar
Copy link
Member

hashhar commented Sep 3, 2024

We should force HTTP/1.1 in the Python client same as the JDBC/Trino-CLI. see trinodb/trino#22171 for equivalent change in CLI/JDBC.

cc: @damian3031

@hashhar hashhar transferred this issue from trinodb/trino Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@damian3031 damian3031

Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hashhar @vinay-kl @damian3031