Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better file export #3572

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

Better file export #3572

wants to merge 36 commits into from

Conversation

poelzi
Copy link
Contributor

@poelzi poelzi commented Jan 18, 2021

Greatly enhance the track export feature.

  • Added template functionality for formatting filenames
  • Better handling of errors while exporting
  • Remove now obsolete wizzard
  • Does not crash anymore when tracklist contains a nullptr or other problems

Done

  • add custom filters for formatting key and building folders for ranges (bpm)
  • fill CrateSummary in template context when exporting crate
  • fill PlaylistSummary in template context when exporting playlist (after Infobar #3406 ). Also playlist posittion as variable
  • Export tracks directly out of the library

TODO

  • Document all available properties

Future

Screenshot_20210118_024227

* major rewrite of the export dialog
* use a table to log all exported files and their errors
* use template language to format the output destination
* don't fail on errors, just log them
* remove now obsolete export wizzard
@daschuer
Copy link
Member

Build fails:

CMake Error at CMakeLists.txt:1806 (find_package):
  By not providing "FindGrantlee5.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "Grantlee5", but CMake did not find one.

  Could not find a package configuration file provided by "Grantlee5" with
  any of the following names:

did you miss to add the FindGrantlee5.cmake file?

You need also add libgrantlee5-dev to the tools/debian_buildenv.sh and to the packaging/debian/control.in file.

@daschuer
Copy link
Member

daschuer commented Jan 18, 2021

According to this we need no own FindGrandlee file: http://www.grantlee.org/apidox/using_and_deploying.html
So adding it to the tools file should do the trick.

TrackExportDlg(QWidget* parent,
UserSettingsPointer pConfig,
TrackPointerList& tracks,
Grantlee::Context* context = nullptr);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove the default nullptr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ? You can pass a custom context if you have additional information you want to pass. There are cases now that do not have any additional context to give, so they pass nothing.

src/library/export/trackexportdlg.cpp Outdated Show resolved Hide resolved
@daschuer
Copy link
Member

daschuer commented Jan 18, 2021

& is exported as &

grafik

It would be nice if we also export the playlist file to loose not the order of the tracks.

By default we should pick a sub folder that is named like the playlist. This can be made optional.

@github-actions github-actions bot added the ui label Jan 19, 2021
@poelzi
Copy link
Contributor Author

poelzi commented Jan 19, 2021

@daschuer you will be able to use the pattern like "{{playlist.name}}/{{index}} {{track.trackFilename}}" when exporting a playlist.
I have some rework in the infobar branch that needs to land first bevore playlist will be available. index already works
Now also {{track.key.openkey}} {{track.key.traditional}} and {{track.key.lancelot}} work

@daschuer
Copy link
Member

Ah cool. Can that be made default?

It requires some extra brain rounding to understand that the folder has to be placed in the file field. Probably to hard for non tech guys.

Can this be added to the folder field instead?

@poelzi
Copy link
Contributor Author

poelzi commented Jan 19, 2021

The Export folder is where the playlist mp3u will be placed and you can put your files anywhere. Also when djintorop library is landed and the export option active, the library will be searched there.
I will provide some more examples to choose from, so you instantly easily see how subfolder can be used.
The {{ index }} is always available and is the sorting the tracks where selected in. If you export the tracks from the library, that will be the order there.

@poelzi
Copy link
Contributor Author

poelzi commented Jan 19, 2021

I also try to get {{track.key}} to print something, but I don't know how to register a new type in grantlee that it has a transform handler.

@poelzi
Copy link
Contributor Author

poelzi commented Feb 3, 2021

@daschuer can you check if the hanging still occurs ? I was not able to reproduce that

@daschuer
Copy link
Member

daschuer commented Feb 3, 2021

Yes. Does it build? At least CI fails.

@poelzi
Copy link
Contributor Author

poelzi commented Feb 3, 2021

@daschuer yes. I will fix the clazy complains today

@daschuer
Copy link
Member

daschuer commented Feb 5, 2021

Thank you. It is getting better. Luckily I was able to catch the crasher in GDB.
There seems to be a memory issue somewhere. In this case it happens after I have deleted all text between the curly braces.
{{asdlasdlasfalsf}} to {{}}
I am using libGrantlee_Templates 5.1.0-2 on Ubuntu Bionic. The first crash reported above happens on Ubuntu focal using libGrantlee_Templates 5.2.0-0ubuntu2

free(): double free detected in tcache 2

Thread 1 "mixxx" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0  0x00007fffeee18fb7 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fffeee1a921 in __GI_abort () at abort.c:79
#2  0x00007fffeee63967 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fffeef90b0d "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fffeee6a9da in malloc_printerr (str=str@entry=0x7fffeef92778 "free(): double free detected in tcache 2") at malloc.c:5342
#4  0x00007fffeee7220d in _int_free (have_lock=0, p=0x55556f76dec0, av=0x7fffef1c5c40 <main_arena>) at malloc.c:4195
#5  0x00007fffeee7220d in __GI___libc_free (mem=0x55556f76ded0)
    at malloc.c:3134
#6  0x00007ffff70bc917 in Grantlee::Parser::takeNextToken() ()
    at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
#7  0x00007ffff70bd0d2 in  ()
    at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
#8  0x00007ffff70be031 in Grantlee::Parser::parse(Grantlee::TemplateImpl*, QStringList const&) () at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
#9  0x00007ffff70c5a70 in  ()
    at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
#10 0x00007ffff70c5b57 in Grantlee::TemplateImpl::setContent(QString const&) ()
    at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
#11 0x00007ffff70a8250 in Grantlee::Engine::newTemplate(QString const&, QString const&) const () at /usr/lib/x86_64-linux-gnu/libGrantlee_Templates.so.5
---Type <return> to continue, or q <return> to quit---
#12 0x0000555555e80d46 in TrackExportWorker::updateTemplate() (this=
    0x55556fc94f00)
    at /home/daniel/workspace/qt5mixxx/src/library/export/trackexportworker.cpp:137
#13 0x0000555555e7a7f5 in TrackExportDlg::updatePreview() (this=0x55556f959d90)
    at /home/daniel/workspace/qt5mixxx/src/library/export/trackexportdlg.cpp:224
#14 0x00007ffff062c66f in QtPrivate::QSlotObjectBase::call(QObject*, void**) (a=0x7fffffffcee0, r=0x55556fa66dc0, this=0x55556dd10100)
    at ../../include/QtCore/../../src/corelib/kernel/qobject_impl.h:101
#15 0x00007ffff062c66f in QMetaObject::activate(QObject*, int, int, void**) (sender=0x55556fa66dc0, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x7fffffffcee0) at kernel/qobject.cpp:3750
#16 0x00007ffff062cc27 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=<optimized out>, m=m@entry=0x7ffff4e08080 <QLineEdit::staticMetaObject>, local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x7fffffffcee0) at kernel/qobject.cpp:3629
#17 0x00007ffff487acf2 in QLineEdit::textChanged(QString const&) (this=<optimized out>, _t1=...) at .moc/moc_qlineedit.cpp:430
#18 0x00007ffff4881129 in QLineEdit::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=0x55556fa66dc0, _c=<optimized out>, _id=<optimized out>, _a=0x7fffffffd070) at .moc/moc_qlineedit.cpp:250
#19 0x00007ffff062c555 in QMetaObject::activate(QObject*, int, int, void**) (sen---Type <return> to continue, or q <return> to quit---
der=sender@entry=0x555565d41a00, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd070)
    at kernel/qobject.cpp:3767
#20 0x00007ffff062cc27 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=sender@entry=0x555565d41a00, m=m@entry=0x7ffff4e08140 <QWidgetLineControl::staticMetaObject>, local_signal_index=local_signal_index@entry=3, argv=argv@entry=0x7fffffffd070) at kernel/qobject.cpp:3629
#21 0x00007ffff4881c85 in QWidgetLineControl::textChanged(QString const&) (this=this@entry=0x555565d41a00, _t1=...) at .moc/moc_qwidgetlinecontrol_p.cpp:257
#22 0x00007ffff4884fc1 in QWidgetLineControl::finishChange(int, bool, bool) (this=this@entry=0x555565d41a00, validateFromState=<optimized out>, update=update@entry=false, edited=edited@entry=true) at widgets/qwidgetlinecontrol.cpp:736
#23 0x00007ffff48870a5 in QWidgetLineControl::insert(QString const&) (this=this@entry=0x555565d41a00, newText=...) at widgets/qwidgetlinecontrol.cpp:271
#24 0x00007ffff488814c in QWidgetLineControl::processKeyEvent(QKeyEvent*) (this=0x555565d41a00, event=event@entry=0x7fffffffd7b0)
    at widgets/qwidgetlinecontrol.cpp:1936
#25 0x00007ffff487ae4a in QLineEdit::keyPressEvent(QKeyEvent*) (this=0x55556fa66dc0, event=0x7fffffffd7b0) at widgets/qlineedit.cpp:1699
#26 0x00007ffff4778877 in QWidget::event(QEvent*) (this=this@entry=0x55556fa66dc0, event=event@entry=0x7fffffffd7b0) at kernel/qwidget.cpp:8933
#27 0x00007ffff48809b2 in QLineEdit::event(QEvent*) (this=0x55556fa66dc0, e=0x7fffffffd7b0) at widgets/qlineedit.cpp:1462
---Type <return> to continue, or q <return> to quit---
#28 0x00007ffff473983c in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=this@entry=0x55555657f200, receiver=receiver@entry=0x55556fa66dc0, e=e@entry=0x7fffffffd7b0) at kernel/qapplication.cpp:3722
#29 0x00007ffff4742642 in QApplication::notify(QObject*, QEvent*) (this=this@entry=0x7fffffffdb60, receiver=receiver@entry=0x55556fa66dc0, e=e@entry=0x7fffffffd7b0) at kernel/qapplication.cpp:3116
#30 0x000055555568020f in MixxxApplication::notify(QObject*, QEvent*) (this=0x7fffffffdb60, target=0x55556fa66dc0, event=0x7fffffffd7b0)
    at /home/daniel/workspace/qt5mixxx/src/mixxxapplication.cpp:135
#31 0x00007ffff05fd8d8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x55556fa66dc0, event=0x7fffffffd7b0)
    at kernel/qcoreapplication.cpp:1024
#32 0x00007ffff47957c5 in QWidgetWindow::event(QEvent*) (this=0x55556fc04410, event=0x7fffffffd7b0) at kernel/qwidgetwindow.cpp:243
#33 0x00007ffff473983c in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=this@entry=0x55555657f200, receiver=receiver@entry=0x55556fc04410, e=e@entry=0x7fffffffd7b0) at kernel/qapplication.cpp:3722
#34 0x00007ffff4741104 in QApplication::notify(QObject*, QEvent*) (this=this@entry=0x7fffffffdb60, receiver=receiver@entry=0x55556fc04410, e=e@entry=0x7fffffffd7b0) at kernel/qapplication.cpp:3481
#35 0x000055555568020f in MixxxApplication::notify(QObject*, QEvent*) (this=0x7fffffffdb60, target=0x55556fc04410, event=0x7fffffffd7b0)
    at /home/daniel/workspace/qt5mixxx/src/mixxxapplication.cpp:135
---Type <return> to continue, or q <return> to quit---
#36 0x00007ffff05fd8d8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=receiver@entry=0x55556fc04410, event=event@entry=0x7fffffffd7b0)
    at kernel/qcoreapplication.cpp:1024
#37 0x00007ffff38020f0 in QCoreApplication::sendSpontaneousEvent(QObject*, QEvent*) (event=0x7fffffffd7b0, receiver=0x55556fc04410)
    at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:236
#38 0x00007ffff38020f0 in QGuiApplicationPrivate::processKeyEvent(QWindowSystemInterfacePrivate::KeyEvent*) (e=0x7fffd40aa600)
    at kernel/qguiapplication.cpp:2078
#39 0x00007ffff3807035 in QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*) (e=e@entry=0x7fffd40aa600)
    at kernel/qguiapplication.cpp:1739
#40 0x00007ffff37de2eb in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) (flags=...)
    at kernel/qwindowsysteminterface.cpp:946
#41 0x00007fffdd350260 in userEventSourceDispatch(GSource*, GSourceFunc, gpointer) (source=<optimized out>) at qeventdispatcher_glib.cpp:77
#42 0x00007fffeaccf417 in g_main_context_dispatch ()
    at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#43 0x00007fffeaccf650 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#44 0x00007fffeaccf6dc in g_main_context_iteration ()
    at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#45 0x00007ffff065688f in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x5555566ad040, flags=...)
    at kernel/qeventdispatcher_glib.cpp:423
#46 0x00007ffff05fb90a in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=this@entry=0x7fffffffdac0, flags=..., flags@entry=...)
    at kernel/qeventloop.cpp:212
#47 0x00007ffff06049b4 in QCoreApplication::exec() ()
    at kernel/qcoreapplication.cpp:1297
#48 0x00005555556403b1 in (anonymous namespace)::runMixxx (args=..., app=0x7fffffffdb60, this=<optimized out>, this=<optimized out>)
    at /home/daniel/workspace/qt5mixxx/src/main.cpp:41
#49 0x00005555556403b1 in main(int, char**) (argc=<optimized out>, argv=<optimized out>) at /home/daniel/workspace/qt5mixxx/src/main.cpp:114

@daschuer
Copy link
Member

daschuer commented Feb 5, 2021

This is how the dialog pops up:

grafik

I think the buttons should go below the tabs, because they are valid for both.
Does the double v v on the drop down menu look as intended?

@poelzi
Copy link
Contributor Author

poelzi commented Feb 8, 2021

I found also a crash when '{{ }}' is entered and opened a ticket in grantlee:
steveire/grantlee#71

'{{}}' did not crash here, only as soon as a space is inserted

@poelzi
Copy link
Contributor Author

poelzi commented Feb 13, 2021

I think I will have to fix this myself and we have to ship with a copy of grentlee. Alternatively we could look into cutelee which is a fork with non specified small differences. Don't know if cutelee will also crash there and if it is more active then grantlee. I choose it only because it is commenly shipped with distros.

@poelzi
Copy link
Contributor Author

poelzi commented Feb 15, 2021

Interestingly, in the grantlee tests I found exactly this case as a test, seems I have to digg further what is going on.

@daschuer
Copy link
Member

It would be much more convenient to have a workaround in our code. This way we don't need to ship the patched grantlee library in our lib folder.
Is this possible?

@poelzi
Copy link
Contributor Author

poelzi commented Feb 19, 2021

@daschuer I was able to fix the crash without fixing grantlee.
I reverted back to a QComboBox because I had such a hard time getting a QListWidget into a popup and to behave as good as a combobox. I however implemented some tweaking in the item handling that should make the combobox good enough for this purpose.
The only thing missing is the vcpkg packages, I asked @Be-ing to help me with microsoft/vcpkg#15907 since I can't build on Windows .... would make more sense if sombody with windows build system finishes the package.

Copy link
Member

@daschuer daschuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments.

QString fullPattern;
if (m_pattern) {
QString trimmed = m_pattern->trimmed();
fullPattern = m_destDir + QDir::separator().toLatin1() + trimmed;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can omit .toLatin1() here and below.

@@ -30,6 +30,7 @@ const auto kResultCantCreateFile = QStringLiteral("Could not create file");
const auto kDefaultPattern = QStringLiteral(
"{{ track.basename }}{% if dup %}-{{dup}}{% endif %}"
".{{track.extension}}");
const auto kEmptyMsg = QLatin1String("");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can become just:
const QString kEmptyMsg;

@@ -23,6 +24,32 @@ PlaylistDAO::PlaylistDAO()
void PlaylistDAO::initialize(const QSqlDatabase& database) {
DAO::initialize(database);
populatePlaylistMembershipCache();

// create temporary views
QString queryString = QLatin1String(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
QString queryString = QLatin1String(
QString queryString = QStringLiteral(

if (QFileInfo::exists(playlistFilePath)) {
auto overwrite = QMessageBox::question(
nullptr,
tr("Overwrite File?"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sound techy to me. How about

How about:
"Playlist file ready exists"

"A playlist file with the name "%1" already exists.\n"
"The default "m3u" extension was added because none was specified.\n\n"
"Do you really want to replace it?"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have only the message box in case the extension was added?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jup, let's have a common way to deal with file overwrite.
For replacing an exitisting controller mapping there is:
https://github.com/mixxxdj/mixxx/blob/main/src/controllers/dlgprefcontroller.cpp#L559-L563

        QString overwriteTitle = tr("Mapping already exists.");
        QString overwriteLabel = tr(
                "<b>%1</b> already exists in user mapping folder.<br>"
                "Overwrite or save with a new name?");
        QString overwriteCheckLabel = tr("Always overwrite during this session");

} else {
//default export to M3U if file extension is missing
if (!playlistFilePath.endsWith(
QStringLiteral(".m3u"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The m3u playlist is Windows-1252 only.
Should we default to m3u8 to avoid data loss?

The attempt to export a m3u playlist with invalid charters fails with a message box.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

off-topic but related: I think we should have a helper class to check for valid filenames that alows to reduce redundancy (mapping export, playlist export, effect chain export and what not).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already added helper functions here to ensure a generated filename is readable on all platforms (afaik), as well as template generators and escape filters.
i would like to later consolidate the crates export (djinterop) to just be part of this export. later also transcoding into different fileformats while exporting (cdj compatible format (aiff/wav)), etc.

@poelzi poelzi mentioned this pull request May 12, 2021
@github-actions
Copy link

This PR is marked as stale because it has been open 90 days with no activity.

@github-actions github-actions bot added the stale Stale issues that haven't been updated for a long time. label Jul 13, 2021
@JoergAtGithub
Copy link
Member

The only thing missing is the vcpkg packages, I asked @Be-ing to help me with microsoft/vcpkg#15907 since I can't build on Windows .... would make more sense if sombody with windows build system finishes the package.

@poelzi I added a Grantlee port to VCPKG upstream: microsoft/vcpkg#28809

@daschuer
Copy link
Member

Cool, thank you for pushing the merge forward.
I will issue an upstream merge PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build code quality library stale Stale issues that haven't been updated for a long time. ui
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants