[Bug]: Video alignment/tearing and color issues in Birdseye post 0.14 upgrade

[jvallery]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

I use birdseye heavily. My use case is to have a TV which continuously displays a live feed. I have 33 cameras. It is important for me to only show cameras which have active objects/motion, otherwise the value is lost. I have a simple windows PC that boots up to VLC connected to my birdseye feed.

Every since the upgrade to 0.14 it has become unstable. The video feed often gets garbled, misaligned, and colors corrupted. Is this a known issue? Is there a plan to resolve? The only solution seems to be restart the container.

Recordings are fine. Interesting this is only showing up in 0.14. It was fine in 0.13. My config hasn't changed.

My goal is for the birdseye live view to be 4k, with each camera no less than 1080p (1920x1080). For a few cameras, I have them configured to 4k (3840x2160) in order to be able to decipher license plates. The rest are configured for 1080p. I have birdseye configured to 3840x2160 and detect configured to 1920x1080. The cameras are a mix of 3840x2160 and 1920x1080. My cameras do run at 15 FPS vs my configuration of 5 FPS for detect. I'm using go2rtc for my camera/birdseye streams.

My understanding is that the live view is based on the detect feed, not the record feed. Thus I'm running detect on the main stream instead of a sub-stream. Frigate is running on a beefy server with dual PCIe corals. I've not ran into performance issues with detect running at this resolution.

I should add, everything works fine for the first couple hours after a container restart. Only after some time does it get corrupted. Perhaps this is first motion/object event from one of my 4k cameras? I haven't actually correlated that.

Any suggestions on what I should do to impove this given my goals? All of these are divisible by 4.

Steps to reproduce

1. Upgrade to 0.14 (still present in 0.14.1)
2. Cameras configured to a mix of 1080p and 4k
3. Birdseye configured to 4k
4. Let run for a few hours

Version

0.14.1-f4f3cfa

In which browser(s) are you experiencing the issue with?

All (Chrome, Edge, VLC for RTSP)

Frigate config file

<snip>
birdseye:
  enabled: true
  restream: true
  width: 3840
  height: 2160
  quality: 8
  mode: objects
  inactivity_threshold: 5
  layout:
    scaling_factor: 2.0
    max_cameras: 4
detect:
  width: 1920
  height: 1080
  fps: 5
  enabled: true
</snip>

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.14.1
    runtime: nvidia
    shm_size: "4096mb" # update for your cameras based on calculation above
    devices:
      - /dev/apex_0:/dev/apex_0
      - /dev/apex_1:/dev/apex_1
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/surveillance/frigate/frigate.yml:/config/config.yml
      - /mnt/surveillance/frigate/db:/db
      - /mnt/surveillance:/media/frigate
      - type: tmpfs 
        target: /tmp/cache
        tmpfs:
          size: 8000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=all #<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video #<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      - PLUS_API_KEY=XXX
      - TZ=America/Denver

Relevant Frigate log output

NA

Relevant go2rtc log output

NA

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Amcrest

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[hawkeye217]

Posting the reddit conversation here for reference: https://www.reddit.com/r/frigate_nvr/comments/1f43cmt/birdseye_in_014/

[NickM-27]

This issue has always existed for birdseye restreaming. The reason is because we are encoding the stream from raw yuv in order to reduce CPU usage, but raw yuv does not have frame markers since it is not encoded so this error happens when the process reading the yuv gets out of sync. In most cases this is not a problem because the pipe is cleared when birdseye disconnects, but if you have the stream going continuously it will not be able to recover from this (and in general there is no way for frigate to know it has entered this error state).

The main solutions would be to encode the frames before decoding them to re-encode for birdseye restream which has higher CPU usage and is not viable for most users.

In your case you may be better off just using the jsmpeg stream via websocket depending on how your live view is displayed.

[NickM-27]

if you don't have global hwaccel_args set in your config it might be worth trying that or changing the arg and seeing if it helps speed up the encoding and avoid the issue.

In general I am not sure if we want to support multiple encoding modes as that will increase the complexity of the feature. This discussion here serves as the bug report

[jvallery]

if you don't have global hwaccel_args set in your config it might be worth trying that or changing the arg and seeing if it helps speed up the encoding and avoid the issue.

The switch from RTSP playback via VLC to using a browser configured for jsmpeg has worked. Unfortunately, that UI/UX isn't very good when the target use case is a fixed display TV.

How does global hwaccel_args impact the issue? I'm not sure we covered that? I do use global args.

ffmpeg: global_args: -hide_banner -loglevel warning -threads 2 hwaccel_args: auto input_args: preset-rtsp-restream output_args: detect: -threads 2 -f rawvideo -pix_fmt yuv420p record: preset-record-generic retry_interval: 5

[NickM-27]

The stream is encoded to h264, if you don't have hwaccel args set at the global level then it is using CPU to do that encoding.

[jvallery]

I assume the global "hwaccel_args: auto" solves this for me?

[NickM-27]

No, the auto is calculated after go2rtc config has been set. You will need to explicitly set your hwaccel_args based on the type

[hawkeye217]

In addition to what @NickM-27 said above, it's also worth mentioning that the additional activity that happens in Frigate 0.14 behind the scenes related to previews may be one possible cause why you see this more often, especially since you have so many cameras and are encoding birdseye at higher resolutions. One way to determine if this is related is that if the birdseye restream distortion happens right around the top of the hour.