Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AccurateRip V2 support #187

Merged
merged 16 commits into from
Sep 15, 2017

Conversation

RecursiveForest
Copy link
Contributor

  • output path no longer has fallbacks
  • refactor accuraterip cache
  • use requests to download accuraterip entries
  • add tests for accuraterip functionality
  • remove gobject support from accuraterip-checksum calculation
  • default track template now includes extension
  • begin to remove support for continuing rip
  • begin to use print instead of sys.stdout.write() throughout

- output path no longer has fallbacks
- refactor accuraterip cache
- use requests to download accuraterip entries
- add tests for accuraterip functionality
- remove gobject support from accuraterip-checksum calculation
- default track template now includes extension
- begin to remove support for continuing rip
- begin to use print instead of sys.stdout.write() throughout
@RecursiveForest
Copy link
Contributor Author

I pressed 'return' by accident before I could add more text to the PR, so:

I've consolidated a lot of spread-out pieces of code into one location. I'm trying to incrementally move away from the current method of organisation into the current by writing code in a more function style, organised by subsystem. This means moving some code that potentially ought to belong on an object eventually into modules as functions in order to make the existing objects easier to modify or delete. Some files will have a mix of styles during the refactor, but I think this will do well to serve as a mark of what areas need attention.

I think we should adhere to the PEP8 79-character line limit. At the very least, it makes lining up code on my monitors into vertical panels extremely easy. ;)

I also think that henceforth we should either require or strive for unit tests with any new or reworked functionality.

@RecursiveForest
Copy link
Contributor Author

There are still more tests to be written, and some of the ideas and organisation in this PR haven't gone to their logical conclusion (in particular the offset find code hasn't really been stylistically updated like cd rip has), but I figured it was better to get the functionality out than to make it perfect.

@RecursiveForest
Copy link
Contributor Author

I should also note that this has a change which now actually requires pycdio as a full dependency.

@MerlijnWajer
Copy link
Collaborator

MerlijnWajer commented Sep 5, 2017

Good work. Lots of changes.

Notes from IRC:

09:43 Wizzup> Why does it change the default track template? We only support flac.
09:45 Wizzup> https://github.com/JoeLametta/whipper/pull/187/files#diff-a1d2c62449840797fa589e5cb1e480dbR73 does this really work?
09:45 Wizzup> I would think the eq is not nested
09:45 Wizzup> so then it would compare references of the lists
09:46 Wizzup> It's a bit hard to read the PR because it feels like there were quite a few other changes made that are not necessarily relevant
09:46 Wizzup> renaming functions, refactoring, etc
09:47 Wizzup> also, why reimplement accuraterip_ids ?
09:48 Wizzup> oh. it is just renamed.

@RecursiveForest
Copy link
Contributor Author

RecursiveForest commented Sep 6, 2017

09:43 Wizzup> Why does it change the default track template? We only support flac.

I didn't like hardcoding '.flac' to the end of the file. It just felt more right to include it in the track template, to make the behaviour more explicit and transparent for the user.

09:45 Wizzup> https://github.com/JoeLametta/whipper/pull/187/files#diff-a1d2c62449840797fa589e5cb1e480dbR73 does this really work?
09:45 Wizzup> I would think the eq is not nested
09:45 Wizzup> so then it would compare references of the lists

Correct, it just compares list references. I believe the only place it's used is comparing two variables potentially holding the same references, but I can make it a deep comparison if you want.

09:46 Wizzup> It's a bit hard to read the PR because it feels like there were quite a few other changes made that are not necessarily relevant
09:46 Wizzup> renaming functions, refactoring, etc

That was the point. It's not a direct "bolt on v2 support to the existing mess", but as stated above a step towards refactoring the program into something easier to work with overall. Some of the other changes, like removing getPath's disambiguation code, simplified some path handling logic in related functions regarding verification enough that I decided it was better to keep it in this branch than to try and factor it out.

@RecursiveForest
Copy link
Contributor Author

As an aside: I developed this changeset in a very experimental manner where I just started hacking away on my checkout of HEAD without trying to factor out different changes into different branches or tracking my progress along the way with individual commits in order to familiarise myself with some of the less common features of git. I don't think it made for a very readable PR (as Wizzup noted), and thus won't be developing like this again.

@MerlijnWajer
Copy link
Collaborator

Please don't take my comments as criticism. I'm happy with the PR and I think we should merge it after some testing. But splitting up the refactoring and actually adding the support for V2 might have made it a bit more readable. And I have to admin I only looked at the whole diff, not per commits, so maybe that paints a better picture. :)

@MerlijnWajer
Copy link
Collaborator

As for the eq, I think that if you actually mean the compare the values, then a deep comparison would make sense. The assumption that they point to the same list could break with any chance, and then there'll be extra unexpected breakage.

@RecursiveForest
Copy link
Contributor Author

I remember making the decision at the time that references were sufficient, but I'm not sure of that anymore.

Criticism is fine, it's what review is for! There's no per-commit history for this, unfortunately.

@RecursiveForest
Copy link
Contributor Author

I experimented around in the REPL and in the tests (__eq__() is only ever used in the unit tests), and it's a deep comparison. list.__eq__() is defined to check each element for lists, although there are optimisations to return True early if the two compared list references are the same.

@JoeLametta
Copy link
Collaborator

Here are the results of the static analysis. I've tried to filter out the warnings which aren't relevant to this PR or consist of FPs but a few surely slipped through...

whipper/command/accurip.py

  • Cannot find reference 'AccuCache' in 'accurip.py' (at line 42)

whipper/command/basecommand.py

  • Unresolved attribute reference 'description' for class 'BaseCommand' (at line 108)
  • Unresolved attribute reference 'subcommands' for class 'BaseCommand' (at line 128)
  • Unresolved attribute reference 'subcommands' for class 'BaseCommand' (at line 129)

whipper/command/cd.py

  • This code is unreachable (at line 340)
  • Statement seems to have no effect and can be replaced with function call to have effect (at line 340)
  • misplaced-bare-raise / The raise statement is not inside an except clause (at line 436, col 20) [pylint]

whipper/command/debug.py

  • Expected type 'str', got 'unicode' instead (at line 238)

whipper/command/image.py

  • Parameter 'itable' unfilled (at line 138)
  • Cannot find reference 'SafeRetagTask' in 'encode.py' (at line 97)
  • Cannot find reference 'AccuCache' in 'accurip.py' (at line 120)
  • Unresolved attribute reference 'getAccurateRipURL' for class 'Table' (at line 127)
  • Unresolved attribute reference 'getAccurateRipResults' for class 'Program' (at line 140)

whipper/common/cache.py

  • Type 'RipResult' doesn't have expected attribute 'eq' (at line 177)

whipper/common/mbngs.py

  • Unresolved attribute reference 'duration' for class 'DiscMetadata' (at line 310)

whipper/common/program.py

  • Unresolved attribute reference 'duration' for class 'DiscMetadata' (at line 323)
  • Unresolved attribute reference 'duration' for class 'DiscMetadata' (at line 334)
  • Unresolved attribute reference 'discid' for class 'DiscMetadata' (at line 429)
  • Cannot find reference 'ImageRetagTask' in 'image.py' (at line 558)
  • Expected type 'str' (matched generic type 'TypeVar('AnyStr', str, unicode)'), got 'unicode' instead (at line 603)
  • Expected type 'str' (matched generic type 'TypeVar('AnyStr', str, unicode)'), got 'unicode' instead (at line 607)

whipper/extern/task/task.py

  • Signature of method 'MultiSeparateTask.described()' does not match signature of base method in class 'ITaskListener' (at line 411)
  • Signature of method 'SyncRunner.schedule()' does not match signature of base method in class 'TaskRunner' (at line 512)

whipper/program/cdparanoia.py

  • using-constant-test / Using a conditional statement with a constant value (at line 162, col 8) [pycdio]
  • simplifiable-if-statement / The if statement can be replaced with 'var = bool(test)' (at line 603, col 8) [pycdio]

whipper/test/test_common_accurip.py

  • Statement seems to have no effect (at line 111)

whipper/test/test_program_cdparanoia.py

  • Call to init of super class is missed (at line 78)

@RecursiveForest
Copy link
Contributor Author

What is an FP?

@RecursiveForest
Copy link
Contributor Author

RecursiveForest commented Sep 6, 2017

Good reminder, though, that I broke the accuraterip command; I knew I was forgetting something… the lack of useful tests is misleading.

Anyways, almost all of those are definitely not relevant to the scope of this PR. It did find one leftover 'exit' from before I decided to raise an exception in that case though, so that's good.

@JoeLametta
Copy link
Collaborator

What is an FP?

I meant false positive.

@RecursiveForest
Copy link
Contributor Author

Understood. I believe I've addressed all the feedback relevant to this PR.

@JoeLametta
Copy link
Collaborator

JoeLametta commented Sep 8, 2017

I believe I've addressed all the feedback relevant to this PR.

Thanks, I'll field-test this PR later today and compute the updated coverage report.
Aren't the following warnings still relevant and not addressed?

whipper/command/cd.py

  • misplaced-bare-raise / The raise statement is not inside an except clause (at line 435, col 20) [pylint]

whipper/command/image.py

  • Parameter 'itable' unfilled (at line 138)
  • Cannot find reference 'AccuCache' in 'accurip.py' (at line 120)
  • Unresolved attribute reference 'getAccurateRipURL' for class 'Table' (at line 127)
  • Unresolved attribute reference 'getAccurateRipResults' for class 'Program' (at line 140)

whipper/common/cache.py

  • Type 'RipResult' doesn't have expected attribute 'eq' (at line 177)

@JoeLametta
Copy link
Collaborator

News

Updated coverage report

Coverage.py 4.4.1 text report against whipper (RecursiveForest's rewrite-accuraterip branch @ a72ae82f0df222f121180016faaafa688ba7ac4a)

$ coverage run --source=whipper --omit='whipper/test/*' -m unittest discover
$ coverage report -m

Name                              Stmts   Miss  Cover   Missing
---------------------------------------------------------------
whipper/__init__.py                  10      2    80%   9, 11
whipper/command/__init__.py           0      0   100%
whipper/command/accurip.py           44     44     0%   21-96
whipper/command/basecommand.py       58     42    28%   54-99, 106-115, 118, 121, 124, 127-130
whipper/command/cd.py               223    183    18%   75-87, 92-193, 196, 208, 231-287, 294-319, 322-501
whipper/command/debug.py            167    167     0%   21-293
whipper/command/drive.py             62     62     0%   21-122
whipper/command/image.py             74     74     0%   21-150
whipper/command/main.py              60     60     0%   4-105
whipper/command/offset.py           109    109     0%   21-227
whipper/common/__init__.py            0      0   100%
whipper/common/accurip.py           130      5    96%   119, 130, 139-141
whipper/common/cache.py             104     49    53%   66-90, 96, 99, 107-110, 113-114, 130, 142-147, 170-177, 201-206, 211-228
whipper/common/checksum.py           16      7    56%   38, 41-42, 45-49
whipper/common/common.py            133     15    89%   49-50, 117-118, 141-142, 165, 259-265, 301-305
whipper/common/config.py             85      9    89%   70, 95-96, 114-115, 121, 132, 134, 136
whipper/common/directory.py          21      6    71%   44-51
whipper/common/drive.py              31     20    35%   35-40, 44-50, 54-60, 64-71
whipper/common/encode.py             44     23    48%   37-38, 41-42, 45-46, 53-56, 59-60, 63-64, 76-77, 80-81, 84-91
whipper/common/mbngs.py             157     52    67%   37-38, 44, 89-95, 156-157, 162-163, 207, 210, 213, 234-237, 246, 266-318
whipper/common/path.py               24      0   100%
whipper/common/program.py           357    281    21%   88-90, 99-114, 122-154, 163-168, 171, 175-179, 219-224, 235-236, 238-242, 246, 258-272, 280-407, 418-468, 476-484, 488-503, 514-554, 557-559, 571-598, 601-616, 621-631, 634-642
whipper/common/renamer.py           102      2    98%   135, 158
whipper/common/task.py               77     19    75%   47-52, 87-88, 91-94, 102, 116-117, 124, 130, 136, 142, 148
whipper/extern/__init__.py            0      0   100%
whipper/extern/asyncsub.py          130     71    45%   15-17, 32, 37-38, 47-84, 89-102, 115, 122, 134, 145, 151, 156-160, 164-176
whipper/extern/task/__init__.py       0      0   100%
whipper/extern/task/task.py         275    117    57%   50, 54, 81, 84, 87, 153-155, 173-175, 183-199, 217-220, 240-241, 282-283, 286-292, 307-308, 316-318, 327-334, 340-356, 360, 363, 370-387, 398-399, 402-405, 409, 412, 427, 430-432, 448, 460, 505-510, 519-524, 535-543, 546-554, 557-558, 566, 571-573
whipper/image/__init__.py             0      0   100%
whipper/image/cue.py                 91      9    90%   99, 116-117, 132-134, 159, 187, 205
whipper/image/image.py              117     94    20%   49-57, 65-67, 74-107, 121-153, 156-172, 183-214
whipper/image/table.py              402     22    95%   238, 350-351, 503, 583, 669-670, 690-691, 700-703, 707-708, 755, 801-802, 804-805, 849-850, 855-857
whipper/image/toc.py                203     15    93%   134, 262-263, 279-282, 340-342, 364-366, 386, 410
whipper/program/__init__.py           0      0   100%
whipper/program/arc.py               38     15    61%   26-28, 32, 37-43, 52-58
whipper/program/cdparanoia.py       315    185    41%   48-50, 59-60, 124-126, 163-166, 199-200, 241-255, 258-310, 313-351, 354-358, 361-397, 452-504, 509-554, 587-590, 593, 600, 606, 611-616
whipper/program/cdrdao.py            51     29    43%   25-47, 54-60, 70-72, 76-78, 86, 93
whipper/program/flac.py               9      5    44%   12-19
whipper/program/sox.py               15      4    73%   18-19, 23-24
whipper/program/soxi.py              28      2    93%   36, 49
whipper/program/utils.py             16     10    38%   11-12, 19-20, 30-35
whipper/result/__init__.py            0      0   100%
whipper/result/logger.py            142    142     0%   1-235
whipper/result/result.py             56     13    77%   112-116, 134, 144-145, 154-161
---------------------------------------------------------------
TOTAL                              3976   1964    51%

Field-test report

I've caught two tracebacks: the former using DAE Drive Features Database's test CD-R including HTOA, the latter using a standard pressed CD (without HTOA).

CD with HTOA

$ LANG=en_US.UTF-8 whipper cd rip -p -o 6 -W . --cdr
Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
CDDB disc id: 0a00ea01
MusicBrainz disc id x5VvXYKDeFC.M0SGfeZXM6tlx8k-
MusicBrainz lookup URL https://musicbrainz.org/cdtoc/attach?toc=1+1+31833+14275&tracks=1&id=x5VvXYKDeFC.M0SGfeZXM6tlx8k-
Disc duration: 00:03:54.106, 1 audio tracks

Matching releases:

Artist  : [unknown]
Title   : DAE Drive Features Database
Duration: 00:03:54.106
URL     : https://musicbrainz.org/release/401a9ca2-cb62-4d9f-8f5f-20aa4ccb7e17
Release : 401a9ca2-cb62-4d9f-8f5f-20aa4ccb7e17
Type    : Album

creating output directory ./album/[unknown] - DAE Drive Features Database
found Hidden Track One Audio from frame 0 to 14124
Traceback (most recent call last):
  File "/usr/local/bin/whipper", line 9, in <module>
    load_entry_point('whipper==0.5.1', 'console_scripts', 'whipper')()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/main.py", line 32, in main
    ret = cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 190, in do
    self.doCommand()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 474, in doCommand
    _ripIfNotRipped(0)
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 363, in _ripIfNotRipped
    track_number=number)
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/common/program.py", line 249, in getPath
    return os.path.join(outdir, template % v)
KeyError: u't'

Pressed CD

All tracks have been ripped but it crashed after having generated the cue sheet (while generating the m3u file) so that the log file is missing.

Stdout + Stderr:

$ time LANG=en_US.UTF-8 whipper cd rip -o 6 -p -W .
Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
CDDB disc id: 9409710a
MusicBrainz disc id 6RUIx_6VkCmJ2VlmKb.IaINLagY-
MusicBrainz lookup URL https://musicbrainz.org/cdtoc/attach?toc=1+10+181450+150+22252+56042+70572+84872+97275+114635+131212+148077+163817&tracks=10&id=6RUIx_6VkCmJ2VlmKb.IaINLagY-
Disc duration: 00:40:17.333, 10 audio tracks

Matching releases:

Artist  : Lucio Battisti
Title   : Vol. 4
Duration: 00:40:18.903
URL     : https://musicbrainz.org/release/fd565c86-96a1-3ef6-9948-9f20d9aa5c7a
Release : fd565c86-96a1-3ef6-9948-9f20d9aa5c7a
Type    : Album
Barcode : 8003614148832
Cat no  : CDMRL 6484

creating output directory ./album/Lucio Battisti - Vol. 4
Ripping track 1 of 10: 01. Lucio Battisti - Le tre verità.flac
CRCs match for track 1                    
Peak level: 95.94% 
Rip quality: 100.00%
Ripping track 2 of 10: 02. Lucio Battisti - Dio mio no.flac
CRCs match for track 2                    
Peak level: 100.00% 
Rip quality: 100.00%
Ripping track 3 of 10: 03. Lucio Battisti - Adesso sì.flac
CRCs match for track 3                    
Peak level: 98.50% 
Rip quality: 100.00%
Ripping track 4 of 10: 04. Lucio Battisti - La mia canzone per Maria.flac
CRCs match for track 4                    
Peak level: 100.00% 
Rip quality: 100.00%
Ripping track 5 of 10: 05. Lucio Battisti - Luisa Rossi.flac
CRCs match for track 5                    
Peak level: 91.83% 
Rip quality: 100.00%
Ripping track 6 of 10: 06. Lucio Battisti - Pensieri e parole.flac
CRCs match for track 6                    
Peak level: 87.98% 
Rip quality: 100.00%
Ripping track 7 of 10: 07. Lucio Battisti - Mi ritorni in mente.flac
CRCs match for track 7                    
Peak level: 89.65% 
Rip quality: 100.00%
Ripping track 8 of 10: 08. Lucio Battisti - Insieme a te sto bene.flac
CRCs match for track 8                    
Peak level: 90.98% 
Rip quality: 100.00%
Ripping track 9 of 10: 09. Lucio Battisti - 29 settembre.flac
CRCs match for track 9                    
Peak level: 92.99% 
Rip quality: 100.00%
Ripping track 10 of 10: 10. Lucio Battisti - Io vivrò (senza te).flac
CRCs match for track 10                    
Peak level: 90.15% 
Rip quality: 100.00%
Traceback (most recent call last):
  File "/usr/local/bin/whipper", line 9, in <module>
    load_entry_point('whipper==0.5.1', 'console_scripts', 'whipper')()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/main.py", line 32, in main
    ret = cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 190, in do
    self.doCommand()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 490, in doCommand
    self.program.write_m3u(discName, htoapath)
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/common/program.py", line 618, in write_m3u
    common.FRAMES_PER_SECOND))
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/common/program.py", line 607, in writeFile
    f.write(u'#EXTINF:%d,%s\n' % (length, target_path))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 45: ordinal not in range(128)

real	19m26.203s
user	0m40.508s
sys	0m6.824s

Cue sheet:

REM DISCID 9409710A
REM COMMENT "whipper 0.5.1"
FILE "01. Lucio Battisti - Le tre verità.flac" WAVE
  TRACK 01 AUDIO
    INDEX 01 00:00:00
FILE "02. Lucio Battisti - Dio mio no.flac" WAVE
  TRACK 02 AUDIO
    INDEX 01 00:00:00
FILE "03. Lucio Battisti - Adesso sì.flac" WAVE
  TRACK 03 AUDIO
    INDEX 01 00:00:00
FILE "04. Lucio Battisti - La mia canzone per Maria.flac" WAVE
  TRACK 04 AUDIO
    INDEX 01 00:00:00
FILE "05. Lucio Battisti - Luisa Rossi.flac" WAVE
  TRACK 05 AUDIO
    INDEX 01 00:00:00
FILE "06. Lucio Battisti - Pensieri e parole.flac" WAVE
  TRACK 06 AUDIO
    INDEX 01 00:00:00
FILE "07. Lucio Battisti - Mi ritorni in mente.flac" WAVE
  TRACK 07 AUDIO
    INDEX 01 00:00:00
FILE "08. Lucio Battisti - Insieme a te sto bene.flac" WAVE
  TRACK 08 AUDIO
    INDEX 01 00:00:00
FILE "09. Lucio Battisti - 29 settembre.flac" WAVE
  TRACK 09 AUDIO
    INDEX 01 00:00:00
FILE "10. Lucio Battisti - Io vivrò (senza te).flac" WAVE
  TRACK 10 AUDIO
    INDEX 01 00:00:00

M3U file:

#EXTM3U

@solstice0
Copy link

I ran whipper offset find -o 6 and it failed to identify the offset as correct despite all tracks matching.

stdout:

Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
Trying read offset 6 ...
Offset of device is likely 6, confirming ...
Only 11 of 11 tracks matched, continuing ...
No matching offset found.
Consider trying again with a different disc.

stdout after running the same command on master codebase:

Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
Trying read offset 6 ...
Offset of device is likely 6, confirming ... 
                                             
Read offset of device is: 6.
Adding read offset to configuration file.

@@ -161,17 +146,18 @@ def match(archecksum, track, responses):

# now try and rip all other tracks as well, except for the
# last one (to avoid readers that can't do overread
for track in range(2, (len(table.tracks) + 1) - 1):
# for track in range(2, (len(table.tracks) + 1) - 1):
for track in range(2, (len(table.tracks) + 1)):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be reverted back to for track in range(2, (len(table.tracks) + 1) - 1):

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@@ -215,59 +215,38 @@ def getPath(self, outdir, template, mbdiscid, i, disambiguate=False):
v['x'] = 'flac'
v['X'] = v['x'].upper()
v['y'] = '0000'
if track_number:
Copy link

@solstice0 solstice0 Sep 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re the HTOA bug: when track_number == 0, this will evaluate to False. Thus line 222 is dead code. Maybe change to if track_number is not None: ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@RecursiveForest
Copy link
Contributor Author

Joe, can you let me know if the commit above fixes your unicode error?

I really need some CDs to test with...

@MerlijnWajer
Copy link
Collaborator

@RecursiveForest - sorry for offtopic, but I wanted to say that I forgot about the CDs that I was going to upload. I have about 20 packed in my bag, and I am planning to make a static page on my server with various kind of CDs. Let's coordinate (and push me!) to get that done ASAP. I have some interesting ones for AR code.

@JoeLametta
Copy link
Collaborator

Joe, can you let me know if the commit above fixes your unicode error?

Yeah, that seems to be fixed (tested with the same CD).

Pressed CD without HTOA

$ time LANG=en_US.UTF-8 whipper cd rip -o 6 -p -W .
Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
CDDB disc id: 9409710a
MusicBrainz disc id 6RUIx_6VkCmJ2VlmKb.IaINLagY-
MusicBrainz lookup URL https://musicbrainz.org/cdtoc/attach?toc=1+10+181450+150+22252+56042+70572+84872+97275+114635+131212+148077+163817&tracks=10&id=6RUIx_6VkCmJ2VlmKb.IaINLagY-
Disc duration: 00:40:17.333, 10 audio tracks

Matching releases:

Artist  : Lucio Battisti
Title   : Vol. 4
Duration: 00:40:18.903
URL     : https://musicbrainz.org/release/fd565c86-96a1-3ef6-9948-9f20d9aa5c7a
Release : fd565c86-96a1-3ef6-9948-9f20d9aa5c7a
Type    : Album
Barcode : 8003614148832
Cat no  : CDMRL 6484

creating output directory ./album/Lucio Battisti - Vol. 4
Ripping track 1 of 10: 01. Lucio Battisti - Le tre verità.flac
CRCs match for track 1                    
Peak level: 95.94% 
Rip quality: 100.00%
Ripping track 2 of 10: 02. Lucio Battisti - Dio mio no.flac
CRCs match for track 2                    
Peak level: 100.00% 
Rip quality: 100.00%
Ripping track 3 of 10: 03. Lucio Battisti - Adesso sì.flac
CRCs match for track 3                    
Peak level: 98.50% 
Rip quality: 100.00%
Ripping track 4 of 10: 04. Lucio Battisti - La mia canzone per Maria.flac
CRCs match for track 4                    
Peak level: 100.00% 
Rip quality: 100.00%
Ripping track 5 of 10: 05. Lucio Battisti - Luisa Rossi.flac
CRCs match for track 5                    
Peak level: 91.83% 
Rip quality: 100.00%
Ripping track 6 of 10: 06. Lucio Battisti - Pensieri e parole.flac
CRCs match for track 6                    
Peak level: 87.98% 
Rip quality: 100.00%
Ripping track 7 of 10: 07. Lucio Battisti - Mi ritorni in mente.flac
CRCs match for track 7                    
Peak level: 89.65% 
Rip quality: 100.00%
Ripping track 8 of 10: 08. Lucio Battisti - Insieme a te sto bene.flac
CRCs match for track 8                    
Peak level: 90.98% 
Rip quality: 100.00%
Ripping track 9 of 10: 09. Lucio Battisti - 29 settembre.flac
CRCs match for track 9                    
Peak level: 92.99% 
Rip quality: 100.00%
Ripping track 10 of 10: 10. Lucio Battisti - Io vivrò (senza te).flac
CRCs match for track 10                    
Peak level: 90.15% 
Rip quality: 100.00%
rip verified as accurate                          
track  1: rip accurate     (max confidence      3) v1 [0ac1a3e9], v2 [d3e7c200], DB [d3e7c200]
track  2: rip accurate     (max confidence      3) v1 [dc955a0e], v2 [13c63f55], DB [13c63f55]
track  3: rip accurate     (max confidence      3) v1 [004ea2b2], v2 [69d82eef], DB [69d82eef]
track  4: rip accurate     (max confidence      3) v1 [ad9f2dc0], v2 [66a0f024], DB [66a0f024]
track  5: rip accurate     (max confidence      3) v1 [3c944309], v2 [cf72d808], DB [cf72d808]
track  6: rip accurate     (max confidence      3) v1 [92634ec4], v2 [b97cf148], DB [b97cf148]
track  7: rip accurate     (max confidence      3) v1 [54415e90], v2 [89fda814], DB [89fda814]
track  8: rip accurate     (max confidence      3) v1 [3d545587], v2 [68f5c941], DB [68f5c941]
track  9: rip accurate     (max confidence      3) v1 [08b1d558], v2 [e7d98b7d], DB [e7d98b7d]
track 10: rip accurate     (max confidence      3) v1 [be671022], v2 [a25dab58], DB [a25dab58]

real	19m54.714s
user	0m49.680s
sys	0m7.084s

Here's the M3U file (file classifies it as "M3U playlist, UTF-8 Unicode text"):

#EXTM3U
#EXTINF:294,01. Lucio Battisti - Le tre verità.flac
01. Lucio Battisti - Le tre verità.flac
#EXTINF:450,02. Lucio Battisti - Dio mio no.flac
02. Lucio Battisti - Dio mio no.flac
#EXTINF:193,03. Lucio Battisti - Adesso sì.flac
03. Lucio Battisti - Adesso sì.flac
#EXTINF:190,04. Lucio Battisti - La mia canzone per Maria.flac
04. Lucio Battisti - La mia canzone per Maria.flac
#EXTINF:165,05. Lucio Battisti - Luisa Rossi.flac
05. Lucio Battisti - Luisa Rossi.flac
#EXTINF:231,06. Lucio Battisti - Pensieri e parole.flac
06. Lucio Battisti - Pensieri e parole.flac
#EXTINF:221,07. Lucio Battisti - Mi ritorni in mente.flac
07. Lucio Battisti - Mi ritorni in mente.flac
#EXTINF:224,08. Lucio Battisti - Insieme a te sto bene.flac
08. Lucio Battisti - Insieme a te sto bene.flac
#EXTINF:209,09. Lucio Battisti - 29 settembre.flac
09. Lucio Battisti - 29 settembre.flac
#EXTINF:235,10. Lucio Battisti - Io vivrò (senza te).flac
10. Lucio Battisti - Io vivrò (senza te).flac

Generated logfile:

Log created by: whipper 0.5.1 (internal logger)
Log creation date: 2017-09-10T09:52:38Z

Ripping phase information:
  Drive: TSSTcorpDVD+-RW SU-208GB (revision D100)
  Defeat audio cache: Yes
  Read offset correction: +6
  Overread into lead-out: No
  Gap detection: cdrdao 1.2.3
  CD-R detected: No

CD metadata:
  Album: Lucio Battisti - Vol. 4
  CDDB Disc ID: 9409710a
  MusicBrainz Disc ID: 6RUIx_6VkCmJ2VlmKb.IaINLagY-
  MusicBrainz lookup url: https://musicbrainz.org/cdtoc/attach?toc=1+10+181450+150+22252+56042+70572+84872+97275+114635+131212+148077+163817&tracks=10&id=6RUIx_6VkCmJ2VlmKb.IaINLagY-

TOC:
  01:
    Start: 00:00:00
    Length: 04:54:52
    Start sector: 0
    End sector: 22101

  02:
    Start: 04:54:52
    Length: 07:30:40
    Start sector: 22102
    End sector: 55891

  03:
    Start: 12:25:17
    Length: 03:13:55
    Start sector: 55892
    End sector: 70421

  04:
    Start: 15:38:72
    Length: 03:10:50
    Start sector: 70422
    End sector: 84721

  05:
    Start: 18:49:47
    Length: 02:45:28
    Start sector: 84722
    End sector: 97124

  06:
    Start: 21:35:00
    Length: 03:51:35
    Start sector: 97125
    End sector: 114484

  07:
    Start: 25:26:35
    Length: 03:41:02
    Start sector: 114485
    End sector: 131061

  08:
    Start: 29:07:37
    Length: 03:44:65
    Start sector: 131062
    End sector: 147926

  09:
    Start: 32:52:27
    Length: 03:29:65
    Start sector: 147927
    End sector: 163666

  10:
    Start: 36:22:17
    Length: 03:55:08
    Start sector: 163667
    End sector: 181299

Tracks:
  01:
    Filename: ./album/Lucio Battisti - Vol. 4/01. Lucio Battisti - Le tre verità.flac
    Peak level: 0.959351
    Pre-emphasis: No
    Extraction speed: 4.5 X
    Extraction quality: 100.00 %
    Test CRC: 89FAFD92
    Copy CRC: 89FAFD92
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: D3E7C200
      Remote CRC: D3E7C200
    Status: Copy OK

  02:
    Filename: ./album/Lucio Battisti - Vol. 4/02. Lucio Battisti - Dio mio no.flac
    Peak level: 0.999969
    Pre-emphasis: No
    Extraction speed: 5.2 X
    Extraction quality: 100.00 %
    Test CRC: AADD0BC2
    Copy CRC: AADD0BC2
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: 13C63F55
      Remote CRC: 13C63F55
    Status: Copy OK

  03:
    Filename: ./album/Lucio Battisti - Vol. 4/03. Lucio Battisti - Adesso sì.flac
    Peak level: 0.985016
    Pre-emphasis: No
    Extraction speed: 5.5 X
    Extraction quality: 100.00 %
    Test CRC: E03F29A4
    Copy CRC: E03F29A4
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: 69D82EEF
      Remote CRC: 69D82EEF
    Status: Copy OK

  04:
    Filename: ./album/Lucio Battisti - Vol. 4/04. Lucio Battisti - La mia canzone per Maria.flac
    Peak level: 0.999969
    Pre-emphasis: No
    Extraction speed: 6.0 X
    Extraction quality: 100.00 %
    Test CRC: 69EA0196
    Copy CRC: 69EA0196
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: 66A0F024
      Remote CRC: 66A0F024
    Status: Copy OK

  05:
    Filename: ./album/Lucio Battisti - Vol. 4/05. Lucio Battisti - Luisa Rossi.flac
    Peak level: 0.918335
    Pre-emphasis: No
    Extraction speed: 6.1 X
    Extraction quality: 100.00 %
    Test CRC: 21A40026
    Copy CRC: 21A40026
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: CF72D808
      Remote CRC: CF72D808
    Status: Copy OK

  06:
    Filename: ./album/Lucio Battisti - Vol. 4/06. Lucio Battisti - Pensieri e parole.flac
    Peak level: 0.879791
    Pre-emphasis: No
    Extraction speed: 6.6 X
    Extraction quality: 100.00 %
    Test CRC: DA2CDBC7
    Copy CRC: DA2CDBC7
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: B97CF148
      Remote CRC: B97CF148
    Status: Copy OK

  07:
    Filename: ./album/Lucio Battisti - Vol. 4/07. Lucio Battisti - Mi ritorni in mente.flac
    Peak level: 0.896484
    Pre-emphasis: No
    Extraction speed: 7.0 X
    Extraction quality: 100.00 %
    Test CRC: FB323CE6
    Copy CRC: FB323CE6
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: 89FDA814
      Remote CRC: 89FDA814
    Status: Copy OK

  08:
    Filename: ./album/Lucio Battisti - Vol. 4/08. Lucio Battisti - Insieme a te sto bene.flac
    Peak level: 0.909821
    Pre-emphasis: No
    Extraction speed: 7.0 X
    Extraction quality: 100.00 %
    Test CRC: BE846F72
    Copy CRC: BE846F72
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: 68F5C941
      Remote CRC: 68F5C941
    Status: Copy OK

  09:
    Filename: ./album/Lucio Battisti - Vol. 4/09. Lucio Battisti - 29 settembre.flac
    Peak level: 0.929871
    Pre-emphasis: No
    Extraction speed: 7.2 X
    Extraction quality: 100.00 %
    Test CRC: F17A1D13
    Copy CRC: F17A1D13
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: E7D98B7D
      Remote CRC: E7D98B7D
    Status: Copy OK

  10:
    Filename: ./album/Lucio Battisti - Vol. 4/10. Lucio Battisti - Io vivrò (senza te).flac
    Peak level: 0.901459
    Pre-emphasis: No
    Extraction speed: 7.9 X
    Extraction quality: 100.00 %
    Test CRC: 6C2FB879
    Copy CRC: 6C2FB879
    AccurateRip v1:
      Result: Track not present in AccurateRip database
    AccurateRip v2:
      Result: Found, exact match
      Confidence: 3
      Local CRC: A25DAB58
      Remote CRC: A25DAB58
    Status: Copy OK

Conclusive status report:
  AccurateRip summary: All tracks accurately ripped
  Health status: No errors occurred
  EOF: End of status report

SHA-256 hash: EF850632E76E7E04CD5B9D81BBA8DDAA03BB2749781A22A93D779A91654B8AEB

CD with HTOA

If I try to rip the CD which includes the HTOA, I get a new traceback (IndexError):

$ time LANG=en_US.UTF-8 whipper cd rip -p -o 6 -W . --cdr
Checking device /dev/sr0
eject: CD-ROM tray close command failed: Input/output error
CDDB disc id: 0a00ea01
MusicBrainz disc id x5VvXYKDeFC.M0SGfeZXM6tlx8k-
MusicBrainz lookup URL https://musicbrainz.org/cdtoc/attach?toc=1+1+31833+14275&tracks=1&id=x5VvXYKDeFC.M0SGfeZXM6tlx8k-
Disc duration: 00:03:54.106, 1 audio tracks

Matching releases:

Artist  : [unknown]
Title   : DAE Drive Features Database
Duration: 00:03:54.106
URL     : https://musicbrainz.org/release/401a9ca2-cb62-4d9f-8f5f-20aa4ccb7e17
Release : 401a9ca2-cb62-4d9f-8f5f-20aa4ccb7e17
Type    : Album

creating output directory ./album/[unknown] - DAE Drive Features Database
found Hidden Track One Audio from frame 0 to 14124
Ripping track 0 of 1: 00. [unknown] - Hidden Track One Audio.flac
CRCs match for track 0                   
Peak level: 100.00% 
Rip quality: 87.46%
Ripping track 1 of 1: 01. [unknown] - Music (In Pregap and Track).flac
CRCs match for track 1                   
Peak level: 100.00% 
Rip quality: 100.00%
Traceback (most recent call last):
  File "/usr/local/bin/whipper", line 9, in <module>
    load_entry_point('whipper==0.5.1', 'console_scripts', 'whipper')()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/main.py", line 32, in main
    ret = cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/basecommand.py", line 124, in do
    return self.cmd.do()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 190, in do
    self.doCommand()
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/command/cd.py", line 489, in doCommand
    self.program.write_m3u(discName, htoapath)
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/common/program.py", line 619, in write_m3u
    (self.result.table.getTrackLength(i + 1) /
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/image/table.py", line 238, in getTrackLength
    return self.getTrackEnd(number) - self.getTrackStart(number) + 1
  File "/usr/local/lib/python2.7/dist-packages/whipper-0.5.1-py2.7.egg/whipper/image/table.py", line 203, in getTrackStart
    track = self.tracks[number - 1]
IndexError: list index out of range

real	7m1.600s
user	0m14.220s
sys	0m1.684s

I really need some CDs to test with...

You can get the test one I'm using (which includes the HTOA) from here.

😉

@RecursiveForest
Copy link
Contributor Author

Okay, glad the UTF-8 problem is fixed. Now that I can test real HTOA (I only had a false positive disc) locally I'll make sure I properly fix the HTOA bug. I definitely made the assumption that we don't support other hidden tracks (which I thought we didn't), so I'm interested to see where the above breaks!

The command/image errors are indeed real problems which I should have under control today as well. I'm so glad we're testing this stuff instead of letting me break everything with each commit!

- accurip.py: raise exception if accuraterip db entry not found
- program.py: verify image with only one table, remove redundant check
@JoeLametta
Copy link
Collaborator

I've updated whipper from your branch and tested it against the previous CDs and it worked as expected. Then I've ran the static analysis check again:

whipper/command/cd.py

  • Expected type 'TypeVar('AnyStr', str, unicode)', got 'int' instead (at line 477)

whipper/common/cache.py

  • Type 'RipResult' doesn't have expected attribute '__eq__' (at line 177)

whipper/common/program.py

  • Expected type 'str' (matched generic type 'TypeVar('AnyStr', str, unicode)'), got 'unicode' instead (at line 593)

As you've slightly touched the code related to pycdio, I'm including the two following warnings too (the first one includes a TODO):

whipper/command/drive.py

  • '_' in try block with 'except ImportError' should also be defined in except block (at line 82)

whipper/common/drive.py

  • 'cdio' in try block with 'except ImportError' should also be defined in except block (at line 65)

Looking forward to merge this PR.

😜

@RecursiveForest
Copy link
Contributor Author

I don't believe the RipResult warning in common/cache or the two cdio warnings are within scope of this PR. I have a separate PR to fully require cdio/pycdio ready to go once this is merged into master, and plan on addressing the caching situation eventually. Next priority is the cdrdao & config regressions.

@JoeLametta
Copy link
Collaborator

JoeLametta commented Sep 15, 2017

@RecursiveForest All right. If everything is done for this PR, I'm going to merge it today: just let me know.

EDIT: Maybe you should add those two things ("output path no longer has fallbacks", "begin to remove support for continuing rip") here too.
About the continuing rip thing please check #136 too: are we going to support it for milestone 1.0?

Thanks

@RecursiveForest
Copy link
Contributor Author

That's a good question and a good idea. I think this PR is ready to merge though, especially if resuming a failed rip is already broken.

@JoeLametta JoeLametta merged commit a3e9260 into whipper-team:master Sep 15, 2017
@JoeLametta
Copy link
Collaborator

That's a good question and a good idea. I think this PR is ready to merge though, especially if resuming a failed rip is already broken.

Understood.
Merged, thanks for the PR.

@JoeLametta JoeLametta mentioned this pull request Jan 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants