write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

cp: write error: No space left on device

Filesystem                Size      Used Available Use% Mounted on

/dev/root               463.9M    453.7M         0 100% /

root@tesla:/# du -sh *

3.0M    bin

5.1M    boot

0       dev

8.3M    etc

14.0K   factory

8.0K    first-install.log

232.1M  home

19.8M   lib

16.0K   lost+found

236.0K  media

4.0K    mnt

4.0K    node_app_slot

7.9M    opt

0       proc

536.0K  run

1.6M    sbin

1.7M    sketch

0       sys

568.0K  tmp

241.4M  usr

164.4M  var

Okay, since we have practically ZERO system partition space allocated to begin with, let's triage. /var/ has over 1/3 of my entire system space? Let's look...

root@tesla:/# du -sh /var/*

4.0K    /var/backups

28.0K   /var/cache

11.1M   /var/lib

4.0K    /var/local

0       /var/lock

153.2M  /var/log

0       /var/run

8.0K    /var/spool

0       /var/tmp

0       /var/volatile

root@tesla:/# ls -nAhsr /var/log

    52 -rw-rw-r--    1 0        43         45.4K Sep 25 13:35 wtmp

     0 -rw-r--r--    1 0        0              0 Sep  3 12:38 postinstall.log

     4 drwxr-sr-x   12 0        996         4.0K Dec 31  1999 journal

     4 -rw-------    1 0        43          1.1K Sep 24 20:50 btmp

So... where is that 164M being accounted for with that one whopping 45k file and those other ones?

It appears to be within journal:

root@tesla:/# du -sh /var/log/journal/

153.2M  /var/log/journal/

root@tesla:/# ls -nAhsr /var/log/journal/

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 d92da95e8c1e4cea86289da731526ec4

     4 drwxr-xr-x    2 0        0           4.0K Dec 31  1999 d6ac38bc69514167834078ea39252634

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 982c9b328dc344fd88f0981b670f0b05

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 89d860c4d16e4b87b7dd9777c9df560f

     4 drwxr-sr-x    2 0        996         4.0K Sep  9 00:09 64d0c722a29b41c3a8df8179a54e8da0

     4 drwxr-sr-x    2 0        996         4.0K Sep  4 18:02 5d665814dfbe45e5b5b98c458b8bbd11

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 37e35cc17194489e8ab236dce14586a7

     4 drwxr-sr-x    2 0        996         4.0K Sep 25 15:04 2f61975d399b4c229bf3573c1327c623

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 234259ec6ad343d2a2b4b3713f3b78f4

     4 drwxr-sr-x    2 0        996         4.0K Dec 31  1999 0218f3e851444c419d069686999a3876

root@tesla:/# du -sh /var/log/journal/*

5.8M    /var/log/journal/0218f3e851444c419d069686999a3876

5.8M    /var/log/journal/234259ec6ad343d2a2b4b3713f3b78f4

48.7M   /var/log/journal/2f61975d399b4c229bf3573c1327c623

5.8M    /var/log/journal/37e35cc17194489e8ab236dce14586a7

46.4M   /var/log/journal/5d665814dfbe45e5b5b98c458b8bbd11

17.4M   /var/log/journal/64d0c722a29b41c3a8df8179a54e8da0

5.8M    /var/log/journal/89d860c4d16e4b87b7dd9777c9df560f

5.8M    /var/log/journal/982c9b328dc344fd88f0981b670f0b05

5.8M    /var/log/journal/d6ac38bc69514167834078ea39252634

5.8M    /var/log/journal/d92da95e8c1e4cea86289da731526ec4

Yikes!!! That's a lotta logs! Given that this tiny thing has no space to begin with.

Can we get some log rotation baked in so we don't have to roll our own?

Furthermore, why are the journal logs in particular so large? Was this mostly for debugging purposes?