Cleanup not working

[tuxpeople]

I tried to reclaim some space and adjusted the numbers for keeping downloaded episodes. But the diskspace used was not going down. Digging further, I found following issue.

Given this configuration:

 [feeds.MyMechanics]
  url="https://www.youtube.com/channel/UCMrMVIBtqFW6O0-MWq26gqw"
  page_size = 5
  update_period = "18h"
  quality = "high"
  format = "video"
  clean = { keep_last = 5 }

The numbers have been higher earlier and got downsized various times. But:

[root@nuc4 runtime]# ls -1 /opt/docker/compose/data/podsync/data/MyMechanics/
6hVeWufC4E0.mp4
6iDJNGdMFls.mp4
9oWOu7tTBxo.mp4
ANsqM0tuXNo.mp4
dgZ41Rg1whs.mp4
FDAWzg0TgVo.mp4
fWikGne44F0.mp4
JmSXo0XdWoA.mp4
JXohCvh2omI.mp4
KahctElSnpw.mp4
kqLoC2U5u-g.mp4
ksCQqViwv2s.mp4
lV2T_IQOR24.mp4
nkzk-6lCSHc.mp4
nltNxh4PaZY.mp4
OyhfQYHnJuM.mp4
PbIj7v5oFUE.mp4
PiYLIXPacuk.mp4
PuBywc6Lnk8.mp4
qixylE1GOWs.mp4
R2M83iah9fo.mp4
R5r8_79x3Z0.mp4
tOVB9I4s0Cs.mp4
TtBeYsx0kEs.mp4
U2jNeObHnZY.mp4
UVkbYlQfNqE.mp4
YNnDGTTcdqw.mp4
YPpmIDlBQkI.mp4
zFbHfB2AnVQ.mp4
zgDVx9tg0lQ.mp4

This are 30 files (which was probably the initial value of page_size and clean). Podsync should have deleted 25 of them. Let's see whats inside the XML:

[root@nuc4 runtime]# grep enclosure /opt/docker/compose/data/podsync/data/MyMechanics.xml
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/JXohCvh2omI.mp4" length="508067476" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/JmSXo0XdWoA.mp4" length="488671169" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/KahctElSnpw.mp4" length="360988915" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/YPpmIDlBQkI.mp4" length="500192183" type="video/mp4"></enclosure>

Looks like there are not even the 5 I configured in it. Something is wrong (at least with the clean) but maybe also with the generation of the feed.

[tuxpeople]

I put the numbers back to 10. Now this happens:

[root@nuc4 runtime]# grep enclosure /opt/docker/compose/data/podsync/data/MyMechanics.xml
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/JXohCvh2omI.mp4" length="508067476" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/JmSXo0XdWoA.mp4" length="488671169" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/KahctElSnpw.mp4" length="360988915" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/PbIj7v5oFUE.mp4" length="342704742" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/UVkbYlQfNqE.mp4" length="507378680" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/YPpmIDlBQkI.mp4" length="500192183" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/kqLoC2U5u-g.mp4" length="477820624" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/lV2T_IQOR24.mp4" length="456287761" type="video/mp4"></enclosure>
      <enclosure url="http://podsync.eighty-three.me/MyMechanics/qixylE1GOWs.mp4" length="442324566" type="video/mp4"></enclosure>

Looks like a "off by one" error. Looks like same type of error for the downloader:

podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="-> updating https://www.youtube.com/channel/UCMrMVIBtqFW6O0-MWq26gqw" feed_id=MyMechanics format=video quality=high
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="downloading episodes" page_size=10
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="download count: 9"
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"JXohCvh2omI\" already exists on disk" episode_id=JXohCvh2omI index=0
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"JmSXo0XdWoA\" already exists on disk" episode_id=JmSXo0XdWoA index=1
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"KahctElSnpw\" already exists on disk" episode_id=KahctElSnpw index=2
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"PbIj7v5oFUE\" already exists on disk" episode_id=PbIj7v5oFUE index=3
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"UVkbYlQfNqE\" already exists on disk" episode_id=UVkbYlQfNqE index=4
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"YPpmIDlBQkI\" already exists on disk" episode_id=YPpmIDlBQkI index=5
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"kqLoC2U5u-g\" already exists on disk" episode_id=kqLoC2U5u-g index=6
podsync_1                | time="2020-09-07T12:30:27Z" level=info msg="episode \"lV2T_IQOR24\" already exists on disk" episode_id=lV2T_IQOR24 index=7
podsync_1                | time="2020-09-07T12:30:28Z" level=info msg="episode \"qixylE1GOWs\" already exists on disk" episode_id=qixylE1GOWs index=8
podsync_1                | time="2020-09-07T12:30:28Z" level=info msg="downloaded 0 episode(s)"
podsync_1                | time="2020-09-07T12:30:28Z" level=info msg="running cleaner" count=10 feed_id=MyMechanics
podsync_1                | time="2020-09-07T12:30:28Z" level=info msg="successfully updated feed in 595.263448ms"
podsync_1                | time="2020-09-07T12:30:28Z" level=info msg="next update of MyMechanics: 2020-09-08 06:30:17 +0000 UTC"

Still not cleaning up files, thought.

[sean89503]

I've had to keep my cleanup larger then the page size i think because clean is using '0' as a count. If you have problems after that try rebuilding the database file. (Delete '.DB' folder, then run again)

[feeds.V.....]
url="https://www.youtube.com/c......."
page_size = 6
update_period = "12h"
quality = "high"
format = "video"
max_height = 720
opml = true
clean = { keep_last = 10 } # Keep last 10 episodes (order desc by PubDate)

[tuxpeople]

I cannot find any .DB file. Where should it be? Also: If cleanup is off by one, because it starts at 0, it still makes no sense to to keep 30 files when cleanup is 5.

[tuxpeople]

using the script we had back in #44, I just freed 37GB data which should have been already deleted.

[tuxpeople]

Hmm... I tried to understand the code for cleanup. I can't really go, but as I understand it's going trough a db to decide what to delete and what not. This reminds me of the .DB @sean89503 mentioned. I'm using the docker container and I've no .DB in my persistence storage. Does it rebuild the DB when I restart the container? If not: could it be that it does not delete the episodes because it does not have it in the DB?

[sean89503]

I'm using Windows so I'm not sure how docker uses the .db. The folder would be hidden so you might have to use sudo. But not sure.

[tuxpeople]

Turns out: Without a database configuration, it does not make use uf the database at all. It just always reads 0 tables. And without a DB, no cleaning occurs (and also no error....).

I added a note in the readme for this: #169

However, maybe cleaner should trow an error when DB is empty.