disk is full #389

cben0ist · 2020-12-17T14:53:12Z

cben0ist
Dec 17, 2020

Hi everyone,

I really love using frigate. Since I found out about it about a week ago I have been extensively testing it and setup my 8 cameras through frigate now. I am still letting it running for a few days before I definitely shut down my shinobi instance.
I am successfully running frigate in a proxmox container following the directions from the readme page.
This morning I realized Frigate was not working because my 12Gb container disk was full.

I found this folder /var/lib/docker/overlay2/c9f6665d68f3d1657fc2e79bcf1d77b637b282cda6d17c8fc9f09a86c9152e8d/diff/tmp/cache contains a copy (?) of all my mp4 that I already have in my media folder.
Is there a reason for that ? What is the best way to deal with this ?

Here is my docker-compose file:

version: '3.6'
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    image: blakeblackshear/frigate:0.8.0-beta1-amd64
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config:/config
      - /opt/frigate/media:/media/frigate      
      - type: tmpfs # 1GB of memory, reduces SSD/SD Card wear
        target: /cache
        tmpfs:
          size: 100000000
    ports:
      - "5000:5000"
      - "1935:1935"
    environment:
      FRIGATE_CAM_PASSWORD: "xxx"
      FRIGATE_CAM_USERNAME: "yyy"
    healthcheck:
      test: ["CMD", "wget" , "-q", "-O-", "http://localhost:5000"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 3m

Thanks for your help!

blakeblackshear · 2020-12-17T14:58:01Z

blakeblackshear
Dec 17, 2020
Maintainer

Can you post your frigate config?

0 replies

cben0ist · 2020-12-17T15:02:21Z

cben0ist
Dec 17, 2020
Author

Here it is: https://pastebin.com/UV17R8p7

0 replies

walthowd · 2020-12-17T15:08:06Z

walthowd
Dec 17, 2020

Just left 0.8 running overnight and my /tmp/cache inside the container is also full.

root@ca6e44fe6f84:/tmp/cache# df -h
Filesystem                   Size  Used Avail Use% Mounted on
overlay                      457G  307G  127G  71% /
tmpfs                         64M     0   64M   0% /dev
tmpfs                         16G     0   16G   0% /sys/fs/cgroup
shm                          4.0G  5.7M  4.0G   1% /dev/shm
/dev/mapper/ubuntu--vg-root  457G  307G  127G  71% /config
tmpfs                        763M  763M     0 100% /tmp/cache
udev                          16G     0   16G   0% /dev/bus/usb

docker-compose.yml

  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    shm_size: '4g' # should work for 5-7 cameras
#    image: blakeblackshear/frigate:0.5.2
#    image: blakeblackshear/frigate:0.6.0-rc1
#    image: blakeblackshear/frigate:0.7.3-amd64
    image: blakeblackshear/frigate:0.8.0-beta1-amd64
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /opt/frigate:/config
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/clips:/media/frigate/clips
      - /opt/frigate/recordings:/media/frigate/recordings
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 800000000
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
    environment:
      - TZ=America/Chicago
    ports:
      - "5000:5000"
      - "1935:1935"
    environment:
      FRIGATE_RTSP_PASSWORD: "XXX"
    depends_on:
      - mosquitto
    healthcheck:
      test: ["CMD", "wget" , "-q", "-O-", "http://localhost:5000"]
      interval: 30s
      timeout: 10s
      retries: 5

Frigate config:
https://hastebin.com/etetigasug.yaml

Docker inspect:
https://hastebin.com/mijowedese.json

I manually cleared /tmp/cache inside the container and things started working again -- it had thousands of files and all zero byte files from today after it had filled.

Also not sure if it is related but debug endpoint (http://localhost:5000/doorbell) my cameras look like this:

0 replies

walthowd · 2020-12-17T15:25:42Z

walthowd
Dec 17, 2020

@cben0ist Looks like you need /tmp/cache for the 0.8 beta not /cache -- root cause seems to possibly be the cache directory filling up -- mine was full of old files this morning which led to FFMPEG continually restarting as it couldn't write any more data.

1 reply

cben0ist Dec 17, 2020
Author

@cben0ist Looks like you need /tmp/cache for the 0.8 beta not /cache -- root cause seems to possibly be the cache directory filling up -- mine was full of old files this morning which led to FFMPEG continually restarting as it couldn't write any more data.

I've updated my config with /tmp/cache, increased the size and also added shm.
Here is the df -h output now:

Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--103--disk--0   12G  8.5G  2.8G  76% /
/dev/loop0                        246G  193G   41G  83% /opt/frigate/media
none                              492K  4.0K  488K   1% /dev
tmpfs                             7.8G     0  7.8G   0% /dev/shm
tmpfs                             3.2G  172K  3.2G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                             4.0M     0  4.0M   0% /sys/fs/cgroup
overlay                            12G  8.5G  2.8G  76% /var/lib/docker/overlay2/a2b22aabc2c62315dca5e73d0135f77abec98116737df49a501dec43bed144a0/merged
shm                               4.0G  4.4M  4.0G   1% /var/lib/docker/containers/e2624158ebcfb38149e9bb8273f448e13b82c23f1afe83b87a313a3202b786ba/mounts/shm
tmpfs                             1.6G     0  1.6G   0% /run/user/0

Let's see if it does fill-up again.
Thanks for the tip.

version: '3.6'
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    shm_size: '4g' 
    image: blakeblackshear/frigate:0.8.0-beta1-amd64
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config:/config
      - /opt/frigate/media:/media/frigate      
      - type: tmpfs # 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 800000000
    ports:
      - "5000:5000"
      - "1935:1935"
    environment:
      FRIGATE_CAM_PASSWORD: "xxx"
      FRIGATE_CAM_USERNAME: "yyy"
    healthcheck:
      test: ["CMD", "wget" , "-q", "-O-", "http://localhost:5000"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 3m

walthowd · 2020-12-17T15:34:06Z

walthowd
Dec 17, 2020

A few minutes after manually clearing /tmp/cache it's full again

root@ca6e44fe6f84:/tmp/cache# rm *.mp4
root@ca6e44fe6f84:/tmp/cache# df -h
Filesystem                   Size  Used Avail Use% Mounted on
overlay                      457G  307G  127G  71% /
tmpfs                         64M     0   64M   0% /dev
tmpfs                         16G     0   16G   0% /sys/fs/cgroup
shm                          4.0G  6.1M  4.0G   1% /dev/shm
/dev/mapper/ubuntu--vg-root  457G  307G  127G  71% /config
tmpfs                        763M     0  763M   0% /tmp/cache
udev                          16G     0   16G   0% /dev/bus/usb
root@ca6e44fe6f84:/tmp/cache# ls
doorbell-20201217091836.mp4
root@ca6e44fe6f84:/tmp/cache# ls
backyard-nw-20201217091837.mp4  doorbell-20201217091836.mp4
root@ca6e44fe6f84:/tmp/cache# ls
backyard-nw-20201217091837.mp4  doorbell-20201217091836.mp4  garage-20201217091839.mp4
root@ca6e44fe6f84:/tmp/cache# ls
backyard-nw-20201217091837.mp4  doorbell-20201217091836.mp4  garage-20201217091839.mp4
root@ca6e44fe6f84:/tmp/cache# ls
backyard-nw-20201217091837.mp4  backyard-nw-20201217092938.mp4  doorbell-20201217092605.mp4  garage-20201217092237.mp4
backyard-nw-20201217091847.mp4  backyard-nw-20201217092948.mp4  doorbell-20201217092615.mp4  garage-20201217092247.mp4
backyard-nw-20201217091857.mp4  backyard-nw-20201217092958.mp4  doorbell-20201217092625.mp4  garage-20201217092257.mp4
backyard-nw-20201217091907.mp4  backyard-nw-20201217093008.mp4  doorbell-20201217092635.mp4  garage-20201217092307.mp4
backyard-nw-20201217091917.mp4  backyard-nw-20201217093018.mp4  doorbell-20201217092645.mp4  garage-20201217092317.mp4
backyard-nw-20201217091927.mp4  backyard-nw-20201217093028.mp4  doorbell-20201217092655.mp4  garage-20201217092327.mp4
backyard-nw-20201217091937.mp4  backyard-nw-20201217093038.mp4  doorbell-20201217092705.mp4  garage-20201217092337.mp4
backyard-nw-20201217091947.mp4  backyard-nw-20201217093048.mp4  doorbell-20201217092715.mp4  garage-20201217092347.mp4
backyard-nw-20201217091957.mp4  backyard-nw-20201217093058.mp4  doorbell-20201217092725.mp4  garage-20201217092357.mp4
backyard-nw-20201217092007.mp4  backyard-nw-20201217093108.mp4  doorbell-20201217092735.mp4  garage-20201217092407.mp4
backyard-nw-20201217092017.mp4  backyard-nw-20201217093118.mp4  doorbell-20201217092747.mp4  garage-20201217092417.mp4
backyard-nw-20201217092027.mp4  backyard-nw-20201217093128.mp4  doorbell-20201217092757.mp4  garage-20201217092427.mp4
backyard-nw-20201217092037.mp4  backyard-nw-20201217093138.mp4  doorbell-20201217092807.mp4  garage-20201217092437.mp4
backyard-nw-20201217092047.mp4  backyard-nw-20201217093148.mp4  doorbell-20201217092817.mp4  garage-20201217092447.mp4
backyard-nw-20201217092057.mp4  backyard-nw-20201217093158.mp4  doorbell-20201217092827.mp4  garage-20201217092457.mp4
backyard-nw-20201217092107.mp4  backyard-nw-20201217093208.mp4  doorbell-20201217092837.mp4  garage-20201217092507.mp4
backyard-nw-20201217092117.mp4  backyard-nw-20201217093218.mp4  doorbell-20201217092847.mp4  garage-20201217092517.mp4
backyard-nw-20201217092127.mp4  backyard-nw-20201217093228.mp4  doorbell-20201217092857.mp4  garage-20201217092527.mp4
backyard-nw-20201217092137.mp4  backyard-nw-20201217093238.mp4  doorbell-20201217092907.mp4  garage-20201217092537.mp4
backyard-nw-20201217092147.mp4  backyard-nw-20201217093248.mp4  doorbell-20201217092917.mp4  garage-20201217092547.mp4
backyard-nw-20201217092157.mp4  backyard-nw-20201217093258.mp4  doorbell-20201217092927.mp4  garage-20201217092557.mp4
backyard-nw-20201217092207.mp4  doorbell-20201217091836.mp4     doorbell-20201217092937.mp4  garage-20201217092607.mp4
backyard-nw-20201217092217.mp4  doorbell-20201217091845.mp4     doorbell-20201217092947.mp4  garage-20201217092617.mp4
backyard-nw-20201217092227.mp4  doorbell-20201217091855.mp4     doorbell-20201217092957.mp4  garage-20201217092627.mp4
backyard-nw-20201217092237.mp4  doorbell-20201217091905.mp4     doorbell-20201217093007.mp4  garage-20201217092637.mp4
backyard-nw-20201217092247.mp4  doorbell-20201217091915.mp4     doorbell-20201217093017.mp4  garage-20201217092647.mp4
backyard-nw-20201217092257.mp4  doorbell-20201217091925.mp4     doorbell-20201217093027.mp4  garage-20201217092657.mp4
backyard-nw-20201217092306.mp4  doorbell-20201217091935.mp4     doorbell-20201217093037.mp4  garage-20201217092707.mp4
backyard-nw-20201217092316.mp4  doorbell-20201217091945.mp4     doorbell-20201217093047.mp4  garage-20201217092717.mp4
backyard-nw-20201217092326.mp4  doorbell-20201217091955.mp4     doorbell-20201217093057.mp4  garage-20201217092727.mp4
backyard-nw-20201217092336.mp4  doorbell-20201217092005.mp4     doorbell-20201217093107.mp4  garage-20201217092737.mp4
backyard-nw-20201217092346.mp4  doorbell-20201217092015.mp4     doorbell-20201217093117.mp4  garage-20201217092748.mp4
backyard-nw-20201217092356.mp4  doorbell-20201217092025.mp4     doorbell-20201217093127.mp4  garage-20201217092758.mp4
backyard-nw-20201217092406.mp4  doorbell-20201217092035.mp4     doorbell-20201217093137.mp4  garage-20201217092810.mp4
backyard-nw-20201217092416.mp4  doorbell-20201217092045.mp4     doorbell-20201217093147.mp4  garage-20201217092820.mp4
backyard-nw-20201217092426.mp4  doorbell-20201217092055.mp4     doorbell-20201217093157.mp4  garage-20201217092830.mp4
backyard-nw-20201217092436.mp4  doorbell-20201217092105.mp4     doorbell-20201217093207.mp4  garage-20201217092840.mp4
backyard-nw-20201217092446.mp4  doorbell-20201217092115.mp4     doorbell-20201217093217.mp4  garage-20201217092850.mp4
backyard-nw-20201217092456.mp4  doorbell-20201217092125.mp4     doorbell-20201217093227.mp4  garage-20201217092900.mp4
backyard-nw-20201217092506.mp4  doorbell-20201217092135.mp4     doorbell-20201217093237.mp4  garage-20201217092908.mp4
backyard-nw-20201217092516.mp4  doorbell-20201217092145.mp4     doorbell-20201217093247.mp4  garage-20201217092919.mp4
backyard-nw-20201217092526.mp4  doorbell-20201217092155.mp4     doorbell-20201217093257.mp4  garage-20201217092928.mp4
backyard-nw-20201217092536.mp4  doorbell-20201217092205.mp4     garage-20201217091839.mp4    garage-20201217092939.mp4
backyard-nw-20201217092546.mp4  doorbell-20201217092215.mp4     garage-20201217091847.mp4    garage-20201217092948.mp4
backyard-nw-20201217092556.mp4  doorbell-20201217092225.mp4     garage-20201217091856.mp4    garage-20201217092958.mp4
backyard-nw-20201217092606.mp4  doorbell-20201217092235.mp4     garage-20201217091907.mp4    garage-20201217093008.mp4
backyard-nw-20201217092616.mp4  doorbell-20201217092245.mp4     garage-20201217091918.mp4    garage-20201217093019.mp4
backyard-nw-20201217092626.mp4  doorbell-20201217092255.mp4     garage-20201217091926.mp4    garage-20201217093032.mp4
backyard-nw-20201217092636.mp4  doorbell-20201217092305.mp4     garage-20201217091936.mp4    garage-20201217093038.mp4
backyard-nw-20201217092646.mp4  doorbell-20201217092315.mp4     garage-20201217091946.mp4    garage-20201217093048.mp4
backyard-nw-20201217092656.mp4  doorbell-20201217092325.mp4     garage-20201217091957.mp4    garage-20201217093058.mp4
backyard-nw-20201217092706.mp4  doorbell-20201217092335.mp4     garage-20201217092007.mp4    garage-20201217093108.mp4
backyard-nw-20201217092716.mp4  doorbell-20201217092345.mp4     garage-20201217092018.mp4    garage-20201217093118.mp4
backyard-nw-20201217092726.mp4  doorbell-20201217092355.mp4     garage-20201217092025.mp4    garage-20201217093129.mp4
backyard-nw-20201217092736.mp4  doorbell-20201217092405.mp4     garage-20201217092037.mp4    garage-20201217093138.mp4
backyard-nw-20201217092748.mp4  doorbell-20201217092415.mp4     garage-20201217092047.mp4    garage-20201217093148.mp4
backyard-nw-20201217092759.mp4  doorbell-20201217092425.mp4     garage-20201217092057.mp4    garage-20201217093158.mp4
backyard-nw-20201217092808.mp4  doorbell-20201217092435.mp4     garage-20201217092107.mp4    garage-20201217093208.mp4
backyard-nw-20201217092818.mp4  doorbell-20201217092445.mp4     garage-20201217092117.mp4    garage-20201217093218.mp4
backyard-nw-20201217092828.mp4  doorbell-20201217092455.mp4     garage-20201217092127.mp4    garage-20201217093228.mp4
backyard-nw-20201217092839.mp4  doorbell-20201217092505.mp4     garage-20201217092137.mp4    garage-20201217093240.mp4
backyard-nw-20201217092848.mp4  doorbell-20201217092515.mp4     garage-20201217092147.mp4    garage-20201217093248.mp4
backyard-nw-20201217092858.mp4  doorbell-20201217092525.mp4     garage-20201217092157.mp4    garage-20201217093258.mp4
backyard-nw-20201217092908.mp4  doorbell-20201217092535.mp4     garage-20201217092207.mp4
backyard-nw-20201217092918.mp4  doorbell-20201217092545.mp4     garage-20201217092217.mp4
backyard-nw-20201217092928.mp4  doorbell-20201217092555.mp4     garage-20201217092227.mp4
root@ca6e44fe6f84:/tmp/cache# du -h -c -s
763M	.
763M	total
root@ca6e44fe6f84:/tmp/cache#

EDIT: Restarted frigate and everything is working again -- assume the process that is responsible for drawing the frames around objects and cleaning up the cache died and never restarted on it's own.

0 replies

mario-tux · 2020-12-18T12:30:21Z

mario-tux
Dec 18, 2020

I noted something very similar: Frigate was stuck with HA saying that there were 26 people in my living room :-). I didn't check if my cache (tmpfs) was full but I took a screenshot and restarted Frigate. Maybe the same bug.

0 replies

walthowd · 2020-12-18T14:57:02Z

walthowd
Dec 18, 2020

I manually built 0.8.0-beta2 and I'm keeping an eye on it. I also took off the size of my tmpfs, volume and just left it the docker default unlimited (which I think is up to half of your RAM)

docker exec -it frigate bash
root@5a665557f2cf:/opt/frigate# while true; do date; du -h -c -s /tmp/cache; du -h -c -s /dev/shm; du -h -c -s /media/frigate; sleep 60; echo ''; done
Fri Dec 18 08:52:36 CST 2020
459M	/tmp/cache
459M	total
3.0M	/dev/shm
3.0M	total
459M	/media/frigate
459M	total

Fri Dec 18 08:53:36 CST 2020
454M	/tmp/cache
454M	total
3.9M	/dev/shm
3.9M	total
726M	/media/frigate
726M	total

Fri Dec 18 08:54:37 CST 2020
413M	/tmp/cache
413M	total
3.9M	/dev/shm
3.9M	total
995M	/media/frigate
995M	total

Fri Dec 18 08:55:37 CST 2020
459M	/tmp/cache
459M	total
3.9M	/dev/shm
3.9M	total
1.3G	/media/frigate
1.3G	total

0 replies

walthowd · 2020-12-18T21:33:59Z

walthowd
Dec 18, 2020

Running beta2, made it about 7hs then at 14:15 local time, around line 1430, /tmp/cache stopped being cleaned up and eventually filled to 16GB:

Here is the script where I was keep tracking of /tmp/cache and /dev/shm sizes:
https://hastebin.com/isutujusoz.apache

At the time the sizes started growing docker container logs had this error:

2020-12-18T20:14:57.672940716Z Exception in thread event_processor:
2020-12-18T20:14:57.672965178Z Traceback (most recent call last):
2020-12-18T20:14:57.672976943Z   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2020-12-18T20:14:57.673464391Z     self.run()
2020-12-18T20:14:57.673476152Z   File "/opt/frigate/frigate/events.py", line 184, in run
2020-12-18T20:14:57.673655871Z     self.create_clip(camera, event_data, save_clips_config.pre_capture)
2020-12-18T20:14:57.673663980Z   File "/opt/frigate/frigate/events.py", line 104, in create_clip
2020-12-18T20:14:57.673697695Z     while sorted_clips[-1]['start_time'] + sorted_clips[-1]['duration'] < event_data['end_time']:
2020-12-18T20:14:57.673718724Z IndexError: list index out of range

Full docker logs are here:
https://paste.c-net.org/GauzeCheque

1 reply

blakeblackshear Dec 18, 2020
Maintainer

Thanks for posting this. I can track this down and fix before the next beta release.

TheSleepySlee · 2020-12-29T11:44:49Z

TheSleepySlee
Dec 29, 2020

Hi

Was facing the same issue, here is how I fixed so far:

In docker compose, added this:

        - type: tmpfs # 1GB of memory, reduces SSD/SD Card wear
          target: /tmp/cache
          tmpfs:
            size: 1000000000 # mind number of zeroes

Here is how much ram used:

root@2f7fee5081df:/opt/frigate# while true; do date; du -h -c -s /tmp/cache; du -h -c -s /dev/shm; du -h -c -s /media/frigate; sleep 60; echo ''; done
Tue Dec 29 12:43:47 CET 2020
246M	/tmp/cache
246M	total
5.8M	/dev/shm
5.8M	total
3.4G	/media/frigate
3.4G	total

Seems this command in docker compose would limite the ram used if needed. Not tried yet:

  deploy:
      resources:
        limits:
          memory: 1024m

1 reply

TheSleepySlee Jan 6, 2021

Actually did not really fix... @blakeblackshear
Seeing errors after about one day

2021-01-06T20:48:44.099861243Z ffmpeg.z5.clips                ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2021-01-06T20:48:44.099941222Z ffmpeg.z5.clips                ERROR   : 
2021-01-06T20:48:44.720280748Z ffmpeg.japonais.clips          ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2021-01-06T20:48:44.720349977Z ffmpeg.japonais.clips          ERROR   : 
2021-01-06T20:48:45.082691102Z ffmpeg.garage.clips            ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2021-01-06T20:48:45.082770624Z ffmpeg.garage.clips            ERROR   : 
2021-01-06T20:48:45.209260911Z ffmpeg.garageptz.clips         ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2021-01-06T20:48:45.209333197Z ffmpeg.garageptz.clips         ERROR   :

Here is /tmp/cache status: size of 1G max

root@2f7fee5081df:/opt/frigate# while true; do date; du -h -c -s /tmp/cache; du -h -c -s /dev/shm; du -h -c -s /media/frigate; sleep 60; echo ''; done
Wed Jan  6 21:52:01 CET 2021
954M	/tmp/cache
954M	total
5.8M	/dev/shm
5.8M	total
35G	/media/frigate
35G	total

pdecat · 2021-01-09T14:25:32Z

pdecat
Jan 9, 2021

I regularly get filesystem full issues caused by /tmp/cache directory containing a very large number of mp4 files.

It seems the underlying cause is the EventCleanup thread is exiting after encountering an error:

2021-01-09T10:11:41.830256000Z Exception in thread event_cleanup:
2021-01-09T10:11:41.831611000Z Traceback (most recent call last):
2021-01-09T10:11:41.831881000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
2021-01-09T10:11:41.833546000Z     cursor.execute(sql, params or ())
2021-01-09T10:11:41.833658000Z sqlite3.OperationalError: database is locked
2021-01-09T10:11:41.833797000Z During handling of the above exception, another exception occurred:
2021-01-09T10:11:41.834135000Z Traceback (most recent call last):
2021-01-09T10:11:41.834290000Z   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-01-09T10:11:41.834868000Z     self.run()
2021-01-09T10:11:41.834949000Z   File "/opt/frigate/frigate/events.py", line 289, in run
2021-01-09T10:11:41.835304000Z     delete_query.execute()
2021-01-09T10:11:41.835657000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1898, in inner
2021-01-09T10:11:41.835841000Z     return method(self, database, *args, **kwargs)
2021-01-09T10:11:41.835941000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1969, in execute
2021-01-09T10:11:41.836279000Z     return self._execute(database)
2021-01-09T10:11:41.836429000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2465, in _execute
2021-01-09T10:11:41.836906000Z     cursor = database.execute(self)
2021-01-09T10:11:41.837108000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3142, in execute
2021-01-09T10:11:41.837655000Z     return self.execute_sql(sql, params, commit=commit)
2021-01-09T10:11:41.837751000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3136, in execute_sql
2021-01-09T10:11:41.839432000Z     self.commit()
2021-01-09T10:11:41.839592000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2902, in __exit__
2021-01-09T10:11:41.839934000Z     reraise(new_type, new_type(exc_value, *exc_args), traceback)
2021-01-09T10:11:41.840104000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 185, in reraise
2021-01-09T10:11:41.840290000Z     raise value.with_traceback(tb)
2021-01-09T10:11:41.840465000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
2021-01-09T10:11:41.840724000Z     cursor.execute(sql, params or ())
2021-01-09T10:11:41.840912000Z peewee.OperationalError: database is locked
2021-01-09T10:11:46.308663000Z Exception in thread event_processor:
2021-01-09T10:11:46.309859000Z Traceback (most recent call last):
2021-01-09T10:11:46.310128000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
2021-01-09T10:11:46.310503000Z     cursor.execute(sql, params or ())
2021-01-09T10:11:46.310906000Z sqlite3.OperationalError: database is locked
2021-01-09T10:11:46.311649000Z During handling of the above exception, another exception occurred:
2021-01-09T10:11:46.312576000Z Traceback (most recent call last):
2021-01-09T10:11:46.312899000Z   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2021-01-09T10:11:46.313619000Z     self.run()
2021-01-09T10:11:46.313795000Z   File "/opt/frigate/frigate/events.py", line 187, in run
2021-01-09T10:11:46.314089000Z     Event.create(
2021-01-09T10:11:46.314330000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 6338, in create
2021-01-09T10:11:46.314521000Z     inst.save(force_insert=True)
2021-01-09T10:11:46.314717000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 6548, in save
2021-01-09T10:11:46.314890000Z     pk = self.insert(**field_dict).execute()
2021-01-09T10:11:46.315077000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1898, in inner
2021-01-09T10:11:46.315280000Z     return method(self, database, *args, **kwargs)
2021-01-09T10:11:46.315437000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1969, in execute
2021-01-09T10:11:46.315607000Z     return self._execute(database)
2021-01-09T10:11:46.315810000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2730, in _execute
2021-01-09T10:11:46.316008000Z     return super(Insert, self)._execute(database)
2021-01-09T10:11:46.316173000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2465, in _execute
2021-01-09T10:11:46.316367000Z     cursor = database.execute(self)
2021-01-09T10:11:46.316558000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3142, in execute
2021-01-09T10:11:46.316805000Z     return self.execute_sql(sql, params, commit=commit)
2021-01-09T10:11:46.317001000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3136, in execute_sql
2021-01-09T10:11:46.317171000Z     self.commit()
2021-01-09T10:11:46.317359000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2902, in __exit__
2021-01-09T10:11:46.317537000Z     reraise(new_type, new_type(exc_value, *exc_args), traceback)
2021-01-09T10:11:46.317701000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 185, in reraise
2021-01-09T10:11:46.317892000Z     raise value.with_traceback(tb)
2021-01-09T10:11:46.318055000Z   File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
2021-01-09T10:11:46.318235000Z     cursor.execute(sql, params or ())
2021-01-09T10:11:46.318417000Z peewee.OperationalError: database is locked

Manually cleaning the /tmp/cache temporarily resolves this issue but as the cleanup thread is no longer running, it re-fills very fast.

The only solution is to restart the main frigate process.

I'll look into submitted a PR to catch such errors in the cleanup thread and continue processing events instead of exiting.

Looks like acquiring a lock to the db is missing somewhere, see coleifer/peewee#874 (comment)

12 replies

pdecat Jan 9, 2021

An alternative may be to use playhouse.sqliteq that extends the currently used SqliteExtDatabase:

SqliteQueueDatabase is designed to simplify things by sending all write queries through a single, long-lived connection. The benefit is that you get the appearance of multiple threads writing to the database without conflicts or timeouts. The downside, however, is that you cannot issue write transactions that encompass multiple queries – all writes run in autocommit mode, essentially.

I believe the mentioned downside does not affect frigate, does it?

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq

blakeblackshear Jan 9, 2021
Maintainer

Probably not.

pdecat Jan 9, 2021

Which branch should PRs target, release-0.8.0 or beta4?

blakeblackshear Jan 9, 2021
Maintainer

Beta4

cybertza Jan 11, 2021

Had the same issue on Unraid, Fixed it by editing the Template, have done a PR on the template but here is the change:

This is for : blakeblackshear/frigate:0.8.0-beta3-amd64nvidia

enable advanced view and then amend the following:

Extra Parameters:
--shm-size=4096m --runtime=nvidia --mount type=tmpfs,target=/tmp/cache,tmpfs-size=400000000

Update:
--shm-size=4096m --runtime=nvidia --mount type=tmpfs,target=/tmp/cache,tmpfs-size=4096m
made it human readable for convince

I believe 1024mb shm and 1024mb fs is no issue, my problem has now gone away.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disk is full #389

{{title}}

Replies: 10 comments 15 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

disk is full #389

Replies: 10 comments · 15 replies

blakeblackshear Dec 17, 2020 Maintainer

cben0ist Dec 17, 2020 Author

cben0ist Dec 17, 2020 Author

blakeblackshear Dec 18, 2020 Maintainer

blakeblackshear Jan 9, 2021 Maintainer

blakeblackshear Jan 9, 2021 Maintainer

Replies: 10 comments 15 replies

blakeblackshear
Dec 17, 2020
Maintainer

cben0ist
Dec 17, 2020
Author

cben0ist Dec 17, 2020
Author

blakeblackshear Dec 18, 2020
Maintainer

blakeblackshear Jan 9, 2021
Maintainer

blakeblackshear Jan 9, 2021
Maintainer