Scans some mp3 files ok, and no flac files get scanned [SOLVED]

[grattle]

It doesn't scan the flac files from the app at all. Scattered tracks of mp3 albums are noticed.

Tried from the command line: sudo -u www-data ./occ audioplayer:scan admin -vv

Same result.

Added the flac decoder to my server, reboot, reset audio library, still no luck. And missing ~~3/4 of the mp3 files.

The music folder is local storage on the server, enabled in External Storage as Local.

Versions:

• Nextcloud: [21.0.0]
• Audioplayer [3.1.0]
• Server Debian Buster, up to date.
• php7.3

The apps music, folder player and audios are not enabled.

running the commands:
sudo -u www-data php occ files:scan admin

Starting scan for user 1 out of 1 (admin)
Entry "/admin/files/music/Paavo Järvi_ Deutsche Kammerphilharmonie Bremen" will not be accessible due to incompatible encoding
Entry "/admin/files/music/Pablo Casals/The 6 Cello Suites (disk 2)/11 - Suite No 4 in E flat major BWV 1010 - Bourée.flac" will not be accessible due to incompatible encoding
Entry "/admin/files/music/Pablo Casals/The 6 Cello Suites (disk 2)/05 - Suite No 3 in C major BWV 1009 - Bourée.flac" will not be accessible due to incompatible encoding
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 5393 | 35564 | 00:11:27 |
+---------+-------+--------------+

sudo -u www-data ./occ audioplayer:scan admin -vv

[about 700 files noticed, last lines below]
.
.
.
26903 - /admin/files/music/Jack Johnson/Unknown Album/07 Fortunate Fools.mp3 => already indexed
35023 - /admin/files/music/Compilations/Triple j's Hottest 100 Volume 5/2-04 Perfect Drug.mp3 => already indexed
Final audio files to be processed: 29099
Total stream files: 32
Checking stream files to be skipped
Final stream files to be processed: 0
Start processing of audio files
Audios found: 1
Duplicates found: 0
Written to library: 1
Albums found: 0
Errors: 0

It says there are 29K music files in total, that sounds right. But only ~700 mp3 files show up in the result of that command line, or in the app.

Since no one else seems to have this issue, it could well be something I have done with this new instance of Nextcloud. Everything else seems to work, but I'm just starting to fiddle with Nextcloud. I fully understand this could be a problem outside your scope as an app maker, but just wondering if you have seen this before.

The latter part of nextcloud.log below:

nextcloud-log-sample.txt

some command line things i tried from above:

command-line-scan.txt

[Rello]

Hello,
I think you have an issue with the line type.
Is flac recognized as media format? does it in the file explorer have a note-logo?

[Rello]

https://github.com/rello/audioplayer/wiki/audio-files-and-mime-types

[grattle]

Hello,
I think you have an issue with the line type.
Is flac recognized as media format? does it in the file explorer have a note-logo?

yes, both flac and mp3 files have a note logo in the file explorer. In file explorer, my collection seems complete.

In audio player, i can't see any flac files. Many albums with mp3 files only register 1 or 2 songs per album, when there are 8~10 songs available in file explorer. I can play the available mp3 files. So, 2 issues. Hoping they are two sides of the same problem...

Screenshots of the same album in audio player and file explorer below. This is an mp3 album, as there is no point showing absent flac albums.

I've again gone over all the prerequisite and optional php extensions and programs in the Nextcloud manual, and can't see anything I am missing. The music folder is read only, does your app try to write there?

I'm happy to keep giving whatever info I can to help sort out this issue, as this is a rather useful app.

[grattle]

I seem to have found the problem.

Tried scanning with a single artist music collection. Just 200 songs, some albums in mp3 and some in flac, works perfectly!

So, I have possibly hit a resource limit somewhere with my 30K songs?

I would like to help. Just know my PHP knowledge is at the level where I could follow adjusting PHP configs for Nextcloud installation without trauma, that's it. Do you have some suggestions for what order I should adjust various settings? Or even which settings could be relevant? Or am I barking up the wrong tree?

[Rello]

hello,
that is actually strange. there could be the php limit coming up when very big files are scanned. not masses because they are written in portions.
You can still scan your complete library. but restart again and again. would this work and scan more files every time and adding them up? this would be an indication that there is indeed a memory limit somewhere

[grattle]

Just did a reset and rescan in the GUI, and have the same total number of files, 798, as before. Those files include all the files in the one artist scan of before. So presumably lost other files in the scan.

On then rescanning without a reset, 1 new file is found. Each subsequent rescan says the same, 1 new file. Not keen on doing this 30K times 8)

FYI, here is my phpinfo. I'll have look at my settings versus the ones in the Nextcloud install manual again, in case I did some bad undersizing. It looks more and more like it is something at my end.

phpinfo.pdf

[Rello]

thats really strange 🤯

[grattle]

I feared you might say something like that.

Since this is obviously not happening to anyone else, some setting on my server is probably triggering the issue. Its an Intel NUC, 8GB RAM, currently about 5GB free, Debian Stable distribution, Apache for a trivial Wordpress site, and an ftp server. That all works. And now a NextCloud instance a few weeks ago that seems to work properly and passes all the tests, but not really explored yet.

I know bugger all about web languages, C++ is my hobby, but there are common debugging approaches to all code.

Perhaps if I have access to a debug version of your app, with logging statements somewhere after it looks at the next file, and before and after it accepts/rejects each file? Or something like that as a first step? I don't know your code, so that could be too simplistic, I'm the beginner here.

Perhaps as a branch in your git tree? And I can install it and give you feedback for the next iteration? The manual install instructions look pretty straightforward, although I've never installed anything but the master branch before. So if this seems like a good idea, tell me the modified git command to load the debug branch.

[Rello]

hello,
let me find some minutes and I will write some details. there is no diff between git and the app.
its working on my PI with less RAM and several 100 files without issues.
I will get back

[Rello]

Hello,
I did not write for a long time, but is the issue still existing?
did you find a solution?

as part of the new version I did a lot of testing and all works on my side in different directions.
can you give an update?

[grattle]

Hi Rello,

Till 2 months ago, the app was still doing the same thing, accepting only several hundred tracks scattered across many albums.

But I also had similar issues with another music cataloger on nextcloud.

Nobody else seems to have my issue with either program, so I suspect it is my personal nextcloud setup. Two days ago, I upgraded the OS, and also moved from php7.3 -> php7.4, nextcloud is now unresponsive. This is a good time to install nextcloud afresh, paying more attention to every single step, and try your lovely app again.

Right now, my attentions are focused on the Australian ski season, but that will soon end.

[grattle]

Hi Rello,
Reinstalled nextcloud, tried again. Fiddled with various php limits.

I can now import a folder of 11K songs (68GBytes) successfully! But I didn't try that number of songs before, so don't know if my previous setup would do this.

But my total collection of 30K songs (512GBytes) fails, although now after ~2000 songs instead of ~800 from before. Rescans apparently add 1 music file.

I'm happy to supply info or fiddle if you have any suggestions.

[Rello]

just got an idea from your old logs you attached.
do you use the cyrilic conversion activated in the settings?
because the old log has this line

Incorrect string value: '\\xFF\\xFE0' for column nextcloud.oc_audioplayer_genre.name``

and this would have been caught there.
could you activate it and then re-run the scan (don´t need to reset the library) and see if it continues?

[grattle]

activated cyrilic, sadly still appears to add only 1 new audio file.

[Rello]

still the same entry in the log-file?

[grattle]

I see 2 different repeated messages in the log file when I try and rescan.

1. Cannot modify header information - many of those. FYI, my storage has write enabled.
2. audioplay error, with a very deep stack. a small number of them.

I have enclosed a snippet of those, can send a larger log file if needed. Bloody ugly format in that log file, if you know a way of making it more human readable, I'm happy to do it from my end.

[Rello]

Hi,
you have one faulty Audio file in your collection. I would like to get it to enhance the scanner...

are you able to modify coding in your app/audioplayer folder?
I would like to have these lines

                  if ($this->timeForUpdate()) {
                      $this->updateProgress($counter, $audio->getPath(), $output);
                  }

https://github.com/Rello/audioplayer/blob/master/lib/Controller/ScannerController.php#L190-L192

moved up before the try statement
https://github.com/Rello/audioplayer/blob/master/lib/Controller/ScannerController.php#L176

I would make this change permanent, but to push a new version just for this would be much effort.
Result would be that during the occ scan, it should print out the last processed file before the error. this would help a lot.

let me know

[grattle]

Tried doing it, got a syntax error. It is breakfast time, not at my best in the morning and also unfamiliar with php.
Will look at it tonight. Thanks for digging into this.

[grattle]

Tried again, different errors.
Enclosed is my version of ScannerController.php in case I mucked it up, and the tail of nextcloud.log
I changed the file type from php to txt so it could be uploaded.
cyrillic is still enabled.
I do like your neat easy to read coding style.

[Rello]

Hello,
die the occ console (using -vv) print out anything between these 2 lines?

Start processing of audio files
Audios found: 1

[grattle]

using sudo -u www-data php occ audioplayer:scan admin -vv

this just shows ~2K mp3 files in my collection of 30K music files

[Rello]

ok, so now we have the issue, that only the MP3 files are scanned - but they seem to be ok/complete, correct?

then we need to see why its not scanning the others.
because it found them obviously in the overall count
https://github.com/Rello/audioplayer/blob/master/lib/Controller/ScannerController.php#L481-L487

[Rello]

ok, MP3 are the first ones to be scanned. so thats ok.
I see that your last run was executed on a clean database - but we need the re-run and not scann these files again.
Can you please do the following which should finally help (hopefully). do not reset the database in between.

1. sudo -u www-data php occ audioplayer:scan admin -vv
2. write down the Audios found: xxx
3. sudo -u www-data php occ audioplayer:scan admin -vv
4. send the console messages (scan result)
5. send the nextcloud log (the last error - not the "Cannot modify header information")

This will process as much as possible.
a second run should pick up where the last stopped.
then hopefully we get the error from the log

[grattle]

I reset the library.
sudo -u www-data php occ audioplayer:scan admin -vv
write down the Audios found: xxx
Audios found: 2858
Duplicates found: 0
Written to library: 2858
Albums found: 462
Errors: 0

**sudo -u www-data php occ audioplayer:scan admin -vv**
**send the console messages (scan result)**


**send the nextcloud log (the last error - not the "Cannot modify header information")**

I have sent the tail of the log. It may include errors from the 1st scan.

[Rello]

ok, getting there.
again - we stop at
/admin/files/music/Aerosmith/Unknown Album/I Don't Wanna Miss A Thing.mp3
can you move this file temporary? I think this is the file with the issue

[grattle]

MUCH closer now. up to 24K out of 30K tracks now seen. flac types play well.

I then tried to remove other problematic music files, but a scan after removing them fails at the same place because the music file is thought to be still there. Resets, redo the occ scan didn't help acknowledge that the files have been moved. Obviously a separate issue, but in the way. Rebooted hoping to clear some caches. Running occ scan again, but will finish well after my bedtime.

Hope to have some usable info for you tomorrow.

[grattle]

Whoo hoo! 30K tracks now cataloged.

There are issues with re scanning after a local disk is altered, but that is a different thread, and perhaps a Nextcloud issue.

As is being resilient against unacceptable files.

Not sure how to call this issue resolved, as not github savvy, so just changed the main title of this thread.

Rello, you followed up on an issue I thought was unresolved as too hard and thus dead and buried. It wasn't my nagging, but your prompting, that led to the resolution. Thank you.

[Rello]

Hello,
not that quick - we need to fix the issue :-)

1. there were some files which managed to kill my error handling. can you send the files ? I definately need to harden the scanner there!
2. regarding the removed files: if you have the recycle bin enabled, the files are still in place - NC only does not show them in the main display. could this be an issue? The reset in this case would not make any difference

...let me know...

[Rello]

Hello. thank you for the files.
i modified the scanner to handle these special cases of the Aerosmith title.
...I also removed all your log files in the thread here...