Utilize 3rd GPU

[yen9912]

Followed solution from #7792 (reply in thread) for 2 GPUs which supported up to 60 camera feeds. Tried to add 10 more on a third gpu but appeared to have maxed out vram on all 3 gpus.

Streams divided between the 3 GPUs using config from #7792.
GPU0 = 30
GPU1 = 21
GPU2 = 18

ffmpeg:
  input_args: preset-rtsp-restream
  hwaccel_args: preset-nvidia-h264

go2rtc:
 ffmpeg:
   cuda0: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 0 -rtsp_transport tcp -i {input}"
   cuda1: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {input}"
   cuda2: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 2 -rtsp_transport tcp -i {input}"

  streams:
    cam1:
      - rtsp://xxx/Streaming/Channels/101
    cam1Rtc:
      - "ffmpeg:cam1#input=cuda1#video=h264#raw=-fpsmax 10 -gpu 1#hardware=cuda"

Also tried using ffmpeg directly.

    cam2Rtc:
      - "ffmpeg:rtsp://xxx/Streaming/Channels/101#input=cuda0#video=h264#raw=-fpsmax 10 -gpu 0#hardware=cuda"

Have I hit some sort of limit? The web gui only shows 2 gpus btw.

Errors

[AVHWDeviceContext @ 0x5d1d8eee53c0] cu->cuCtxCreate(&hwctx->cuda_ctx, desired_flags, hwctx->internal->cuda_device) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory

[h264 @ 0x63ba4df85000] No device available for decoder: device type cuda needed for codec h264.

Device creation failed: -542398533.

Device setup failed for decoder on input stream #0:0 : Generic error in an external library

[NickM-27]

What error are you getting?

[yen9912]

Ah right forgot about that. Updated post.
I didn't keep a copy of the logs when I had the full config. Switched to non gpu version to record for now ghcr.io/blakeblackshear/frigate:stable.
Some of the errors may not be relevant. I do remember seeing CUDA_ERROR_OUT_OF_MEMORY previously.

Errors

[AVHWDeviceContext @ 0x5d1d8eee53c0] cu->cuCtxCreate(&hwctx->cuda_ctx, desired_flags, hwctx->internal->cuda_device) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory

[h264 @ 0x63ba4df85000] No device available for decoder: device type cuda needed for codec h264.

Device creation failed: -542398533.

Device setup failed for decoder on input stream #0:0 : Generic error in an external library

[NickM-27]

Are you changing hwaccel args / global args too or trying to use the same GPU for decoding?

[yen9912]

Not familiar with ffmpeg or hwaccel args. Using whatever I see here or in the docs.

1. Is it better to use a different GPU for decoding?
2. Is it possible to use software decoding when I run out of GPU VRAM? eg. -hwaccel none

[NickM-27]

You can set hwaccel_args: [] to disable it. If you posted your full config it would be easy to make a suggestion