Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daemon panics when describing jobs ran by 1.4.0 node #4510

Open
frrist opened this issue Sep 24, 2024 · 0 comments
Open

Daemon panics when describing jobs ran by 1.4.0 node #4510

frrist opened this issue Sep 24, 2024 · 0 comments
Labels
request/new Request: Indicates a new request that has been submitted and awaits initial triage type/bug Type: Something is not working as expected

Comments

@frrist
Copy link
Member

frrist commented Sep 24, 2024

Bug Description

Start a v1.4.0 bacalhau deamon and run a docker job against it.

  1. $ 140bacalhau serve --node-type=requester,compute
  2. $ 140bacalhau docker run ubuntu:latest echo hello world

Kill the deamon, start a 1.5.0 daemon, describe the job from above

  1. $ 150bacalhau serve --node-type=requester,compute
  2. $ 150bacalhau job describe j-784b1ca9-15f6-4d19-b05b-75d1ec8343e7

The CLI command returns: Internal server error
The 1.5.0 daemon shows this panic:

{
  "time": "2024-09-24T09:39:23.66901833-07:00",
  "level": "-",
  "prefix": "echo",
  "file": "recover.go",
  "line": "119",
  "message": "[PANIC RECOVER] runtime error: invalid memory address or nil pointer dereference goroutine 223 [running]:\ngithub.com/bacalhau-project/bacalhau/pkg/publicapi.NewAPIServer.Recover.RecoverWithConfig.func6.1.1()\n\tgithub.com/labstack/echo/[email protected]/middleware/recover.go:99 +0x156\npanic({0x239cc80?, 0x44d1040?})\n\truntime/panic.go:785 +0x132\nnet/http.(*timeoutHandler).ServeHTTP(0xc00065c240, {0x305a920, 0xc00072c760}, 0xc0003f2500)\n\tnet/http/server.go:3675 +0x768\ngithub.com/bacalhau-project/bacalhau/pkg/publicapi.NewAPIServer.TimeoutWithConfig.TimeoutConfig.ToMiddleware.func9.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/timeout.go:126 +0x1ed\ngithub.com/bacalhau-project/bacalhau/pkg/publicapi.NewAPIServer.RateLimiter.RateLimiterWithConfig.func8.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/rate_limiter.go:146 +0xba\ngithub.com/labstack/echo/v4/middleware.BodyLimitWithConfig.func1.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/body_limit.go:85 +0x15c\ngithub.com/bacalhau-project/bacalhau/pkg/publicapi.NewAPIServer.RequestID.RequestIDWithConfig.func7.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/request_id.go:68 +0x112\ngithub.com/bacalhau-project/bacalhau/pkg/publicapi.NewAPIServer.Recover.RecoverWithConfig.func6.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/recover.go:130 +0x114\ngithub.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/cors.go:203 +0x463\ngithub.com/labstack/echo/v4.(*Echo).ServeHTTP.func1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/echo.go:668 +0x127\ngithub.com/labstack/echo/v4/middleware.RewriteWithConfig.func1.1({0x30989f8, 0xc000c661e0})\n\tgithub.com/labstack/echo/[email protected]/middleware/rewrite.go:77 +0x7f\ngithub.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000978248, {0x3059a70, 0xc000274c40}, 0xc0003f2500)\n\tgithub.com/labstack/echo/[email protected]/echo.go:674 +0x327\nnet/http.serverHandler.ServeHTTP({0xc000a0a510?}, {0x3059a70?, 0xc000274c40?}, 0x6?)\n\tnet/http/server.go:3210 +0x8e\nnet/http.(*conn).serve(0xc00071a000, {0x3062568, 0xc000a2c480})\n\tnet/http/server.go:2092 +0x5d0\ncreated by net/http.(*Server).Serve in goroutine 180\n\tnet/http/server.go:3360 +0x485\n\ngoroutine 1 [chan receive]:\nmain.main()\n\tgithub.com/bacalhau-project/bacalhau/main.go:99 +0x1eb\n\ngoroutine 18 [select]:\ngithub.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc000122780)\n\tgithub.com/ipfs/[email protected]/writer/writer.go:71 +0x105\ncreated by github.com/ipfs/go-log/writer.NewMirrorWriter in goroutine 1\n\tgithub.com/ipfs/[email protected]/writer/writer.go:36 +0xbf\n\ngoroutine 19 [select]:\ngo.opencensus.io/stats/view.(*worker).start(0xc000428e00)\n\[email protected]/stats/view/worker.go:292 +0x9f\ncreated by go.opencensus.io/stats/view.init.0 in goroutine 1\n\[email protected]/stats/view/worker.go:34 +0x8d\n\ngoroutine 46 [syscall]:\nos/signal.signal_recv()\n\truntime/sigqueue.go:152 +0x29\nos/signal.loop()\n\tos/signal/signal_unix.go:23 +0x13\ncreated by os/signal.Notify.func1.1 in goroutine 1\n\tos/signal/signal.go:151 +0x1f\n\ngoroutine 47 [select]:\nos/signal.NotifyContext.func1()\n\tos/signal/signal.go:288 +0x66\ncreated by os/signal.NotifyContext in goroutine 1\n\tos/signal/signal.go:287 +0x156\n\ngoroutine 48 [chan receive]:\ngithub.com/bacalhau-project/bacalhau/cmd/cli/serve.serve(_, {{{0x2768c8b, 0x7}, 0x4d2, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...}, ...)\n\tgithub.com/bacalhau-project/bacalhau/cmd/cli/serve/serve.go:310 +0x17da\ngithub.com/bacalhau-project/bacalhau/cmd/cli/serve.NewCmd.func2(0xc000ea2f08, {0x275c5a7?, 0x4?, 0x275c48b?})\n\tgithub.com/bacalhau-project/bacalhau/cmd/cli/serve/serve.go:109 +0x299\ngithub.com/spf13/cobra.(*Command).execute(0xc000ea2f08, {0xc0009972f0, 0x1, 0x1})\n\tgithub.com/spf13/[email protected]/command.go:983 +0xaaa\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc000159b08)\n\tgithub.com/spf13/[email protected]/command.go:1115 +0x3ff\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\tgithub.com/spf13/[email protected]/command.go:1039\ngithub.com/bacalhau-project/bacalhau/\n"
@frrist frrist added type/bug Type: Something is not working as expected request/new Request: Indicates a new request that has been submitted and awaits initial triage labels Sep 24, 2024
frrist pushed a commit that referenced this issue Sep 24, 2024
wdbaruni added a commit that referenced this issue Sep 25, 2024
- related to #4510

---------

Co-authored-by: frrist <[email protected]>
Co-authored-by: Walid Baruni <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
request/new Request: Indicates a new request that has been submitted and awaits initial triage type/bug Type: Something is not working as expected
Projects
Status: Inbox
Development

No branches or pull requests

1 participant