Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intro/Outro cues and silence detection #1242

Merged
merged 143 commits into from
Apr 24, 2019
Merged

Conversation

ninomp
Copy link
Contributor

@ninomp ninomp commented Apr 24, 2017

This PR introduces simple threshold-based silence detection

https://bugs.launchpad.net/mixxx/+bug/673515
https://bugs.launchpad.net/mixxx/+bug/1106813

Controls added: https://manual.mixxx.org/2.3/en/chapters/advanced_topics.html#control-[ChannelN]-intro_end_activate

Work items:

  • Implement silence detector that places special cue points on tracks
  • Allow user to edit these cue points
  • Tweak AutoDJ to make use of these cues
  • Place the main cue point to nearest beat to the detected begin point
  • CO for clearing main cue point
  • Visualize cue source (tells whether cue point was adjusted/corrected by user)
  • Make these cues range cues
  • Make AutoDJ use outro start cue as fade out start point
  • Add unit tests for new functionalities:
    • AnalyzerSilence
    • AutoDJProcessor
    • CueControl
  • Modify skins:
    • Display intro/outro start/end markers on waveforms and overviews
    • Add buttons for setting intro/outro start/end markers

Ideas:

  • Adjust silence detection threshold using the computed replay gain
  • Allow user to adjust detection threshold and to disable silence detection

Implement simple threshold-based silence detection.
Copy link
Member

@rryan rryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this @ninomp! I think you are also fixing this bug:
https://bugs.launchpad.net/mixxx/+bug/673515

}

bool AnalyzerSilence::initialize(TrackPointer tio, int sampleRate, int totalSamples) {
Q_UNUSED(tio);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we shouldn't enable the analyzer if the track already has an in/out cue (since the user can manually edit their cues, so we don't know if we are overwriting user data)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may consider her to move the cue points if they are finally independent from auto DJ an they are not user editable

m_iTotalSamples = totalSamples;
m_iFramesProcessed = 0;
m_bPrevSilence = true;
m_iSignalBegin = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would initialize with -1 to distinguish between not found and 0.


bool bBeginPointFoundAndSet = false;
bool bEndPointFoundAndSet = false;
QList<CuePointer> cues = tio->getCuePoints();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should edit cues since at the point we create them they become user data and we can't tell if they were edited by the user

AnalyzerSilence::AnalyzerSilence(UserSettingsPointer pConfig)
: m_pConfig(pConfig),
m_fThreshold(db2ratio(kSilenceThreshold)),
m_iSampleRate(0),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused, please remove m_iSampleRate and m_iTotalSamples

}

void AnalyzerSilence::finalize(TrackPointer tio) {
// If track didn't end with silence, place signal end marker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle m_iSignalBegin < 0 and m_iSignalEnd < 0 explicitly here ?

@@ -21,6 +21,8 @@ class Cue : public QObject {
BEAT = 3,
LOOP = 4,
JUMP = 5,
BEGIN = 6,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might make more sense to call these "fade in" and "fade out" cues (or autodj_in / autodj_out), since that's more aligned with the purpose.

Tracks may not have a beginning or end, and user's probably don't particularly care about having the beginning and end of the track annotated, but they do care about annotating where AutoDJ would fade in and out. Plus if they edit the cue to be located where they want AutoDJ to fade in, then it doesn't point at the track start anymore so it'd be misnamed.

@@ -102,6 +102,9 @@ CueControl::CueControl(QString group,
m_pCueIndicator = new ControlIndicator(ConfigKey(group, "cue_indicator"));
m_pPlayIndicator = new ControlIndicator(ConfigKey(group, "play_indicator"));

m_pSignalBeginPosition = new ControlObject(ConfigKey(group, "signal_begin_position"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

along with renaming the cue type, please call these

fade_in_cue_point
fade_out_cue_point

or similar?

@@ -41,6 +41,20 @@
<Color>#FF001C</Color>
<TextColor>#FFFFFF</TextColor>
</Mark>
<Mark>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we get closer to merging, please remember to update Deere and Shade.

@@ -0,0 +1,104 @@
#include "analyzer/analyzersilence.h"

const int kChannelCount = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use mixxx::AudioSource::kChannelCountStereo

#include "analyzer/analyzersilence.h"

const int kChannelCount = 2;
const float kSilenceThreshold = -60.0f;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please name it kSilenceThresholdDb to document the unit

@daschuer
Copy link
Member

I think we schould treat the silence start an silence end points independent from the auto DJ fade in and fade out points.
This allows us to distinguish between the crossfading and the jukebox use case.
For crossfading, the auto DJ in point is first downbeat aka THE CUE point. The out point is the last part that has usable beats before the noise starts that son tracks have.

In the jukebox use case, we want to play the tracks in a whole, with a equal gap of silence in between.

What do you think?

@daschuer
Copy link
Member

Do we need to consider the replay gain of a track or is -60 dB of full scale so quite that a replay gain does not matter? If yes, this feature should be part of the replay gain analysis:

The function form the ReplayGain2 analysis is

ebur128_loudness_momentary

int ebur128_loudness_momentary(ebur128_state* st, double* out);

it uses a 3 s window.

A momentary Loudness of replay1 gain is calculated here with a 50 ms window.

if ( ival >= (int)(sizeof(A)/sizeof(*A)) ) ival = (int)(sizeof(A)/sizeof(*A)) - 1;

I am not sure how much we should overcomplicate this feature that much.

@ninomp
Copy link
Contributor Author

ninomp commented Apr 26, 2017

Well, I was thinking about using the replay gain value to adjust the -60dB threshold value, but that means that we need multi-pass analysis (in first pass compute the RG, in second detect silence) which seems like an overkill (at least to me).

For now, I would like to continue testing this simple-threshold-on-a-raw-signal silence detection and see how it works with various tracks from my library. So far, this simple algorithm seems to be performing quite good.

Also, I would like to allow user to change silence detection threshold in preferences, but I'm afraid that it might cause problems if inexperienced users change it to very low or very high value. What do you think?

@ninomp
Copy link
Contributor Author

ninomp commented Apr 26, 2017

Regarding the cue points, previously all cue points were set/created by user and now I am introducing cue points that are set (and moved if threshold changes) by Mixxx and which need to be modifiable by user. As I see it, we have three options:

  1. Introduce a new field to Cue which will indicate whether that cue point was set by Mixxx or user (something similar to 'Source' field of CoverArt)
  2. Introduce a new cue point type for begin/end that is set (overriden) by user (AutoDJ will use detected begin/end if manual begin/end is not set)
  3. Make the detected begin/end points fields/attributes of Track. (This way all cue points are again only user editable, but we add two more fields in the 'library' table in the database.)

What do you think? Which one of these is better way?

@ninomp
Copy link
Contributor Author

ninomp commented Apr 26, 2017

One other thing I am also thinking of adding to this PR is automatic placement of main CUE point at the detected signal begin position if the main CUE point is not set. (I will quantize to nearest beat if quantization is enabled.) I don't think that we need a preference option for this. Do you agree?

@Pegasus-RPG
Copy link
Member

Pegasus-RPG commented Apr 26, 2017

automatic placement of main CUE point at the detected signal begin position if the main CUE point is not set.

I agree, this would be good to do. (Pioneer CDJs have been doing this for over a decade. It's about time Mixxx does it as well. ;) ) I would go further and set it to the detected begin point also if the main CUE point is set to the very beginning of the track.

@daschuer
Copy link
Member

Well, I was thinking about using the replay gain value to adjust the -60dB threshold value, but that means that we need multi-pass analysis (in first pass compute the RG, in second detect silence) which seems like an overkill (at least to me).

I agree two passes are overdone. A combined path may work.

Also, I would like to allow user to change silence detection threshold in preferences, but I'm afraid that it might cause problems if inexperienced users change it to very low or very high value. What do you think?

We schould try to go without a preference option. IMHO this option is somehow track depending and tweaking it for one track will destroy a good result for an other.

@daschuer
Copy link
Member

As I see it, we have three options: ...

since it is finally a user editable point, it does make sense to treat it as special cue point and not as a database column​.
To track a source field can work.

An alternative would to track it along with the lock feature of the beat grid, which is user editable as well.

Generally I think we need two sets of points the sound start point honoured by auto DJ in jukebox mode and the fade-in/out points used for crossfading/beatmatching.


void AnalyzerSilence::process(const CSAMPLE *pIn, const int iLen) {
for (int i = 0; i < iLen; i += kChannelCount) {
bool bSilence = math_max(fabs(pIn[i]), fabs(pIn[i + 1])) < m_fThreshold;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implicit assumption of 2 channels! Please make sure that your code is generic and works with any number of channels in kChannelCount. Otherwise document your assumptions explicitly with an assertion.

return false;
}

void AnalyzerSilence::process(const CSAMPLE *pIn, const int iLen) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document your assumptions explicitly:
DEBUG_ASSERT((iLen % kChannelCount) == 0);

Otherwise your code might read outside the bounds of the provided array! But this is actually a flaw in the Analyzer's design which needs to be migrated to frame indexing. It should always provide complete frames, that's why the assertion is sufficient here.

@ninomp
Copy link
Contributor Author

ninomp commented Apr 29, 2017

@daschuer How do you distinguish crossfading from jukebox use case?
Is it by transition duration (<= 0 jukebox, > 0 crossfading)?

@ninomp
Copy link
Contributor Author

ninomp commented May 2, 2017

Do we really need special begin cue point for AutoDJ? Cue points are starting to clutter on beginnings of tracks.

I suggest:

  • removing begin point and using the main cue point as start point for Auto DJ (in both jukebox and crossfading/beatmatching mode)
  • using end point (introduced by this PR) as stop point for AutoDJ in jukebox mode (although it can be used to limit transition in crossfading/beatmatching mode)
  • introduce new point for AutoDJ start fade out that will only be used in crossfading/beatmatching mode

Furtermore, I believe we are reaching the point when we need to introduce AutoDJ modes. Maybe something like:

  • jukebox (parameter is duration of silence between tracks)
  • crossfading (parameter is duration of transition in seconds)
  • beatmatching (parameter is duration of transition in beats)

@daschuer
Copy link
Member

daschuer commented May 2, 2017

Do we really need special begin cue point for AutoDJ?

No, AutoDJ can use one of the existing/upcoming cue points.
THE CUE for beatmatching/crossfading and sound start for jukebox mode.

I really think we need both, because in jukebox mode, the user want to listen a track from the first to the last tone unfortunately the first tone is unrelated to the first down beat.

But I agree, that showing all cue points at the same time is somehow cluttered, since the user normally needs only one of it. Coupling it with a kind of auto DJ mode might help.

This mode should select at which cue the track loaded. .. Is it finally a new or advanced track load mode?
Does it make sense to not show the not used other cue point?

@daschuer
Copy link
Member

daschuer commented May 2, 2017

How about using THE CUE to set either the sound start point or the first downbeat point, depending on the selected mode? This would work almost natural from controllers and without extra cluttering controls in the GUI .

@ninomp
Copy link
Contributor Author

ninomp commented May 3, 2017

So, we actually need four cue points for AutoDJ (all of them should be user-editable):

  • Begin point (for jukebox mode)
  • Start fade in point (for crossfading/beatmatching mode)
  • Start fade out point (for crossfading/beatmatching mode)
  • End point (for jukebox mode)

@daschuer
Copy link
Member

daschuer commented May 3, 2017

Unfortunately yes.

In the major manual DJ use case, only the "Start fade in point" (alias THE CUE) point is needed. Did you consider to hide the "Begin point" in this case, or add a preference option to use one of them as "THE CUE"?

We may also try to auto detect the "Start fade in point" one day. It is the first notable downbeat after the "Begin point". In the first version we may move both points to the same place.

@ninomp
Copy link
Contributor Author

ninomp commented May 3, 2017

Did you consider to hide the "Begin point" in this case, ...

Well, I'm trying to find a nice and simple solution that will cover all use cases (or at least most of them 😄).

Maybe we can leave everything visible for now. Furthermore, I'm concerned it might confuse users.

We may also try to auto detect the "Start fade in point" one day.

Maybe we can leave begin point non-quantized and quantize it as the main cue point. However, quantization will apply if user moves the begin point while quantization is enabled.

@rryan
Copy link
Member

rryan commented May 4, 2017

I don't think it's clear what problem this PR is trying to solve. Could you please make that clearer in the PR description? Ideally with some "user stories" or use cases?

I think we're a bit into the weeds here discussing AutoDJ modes. I'd also like us to keep in mind that Mixxx is not trying to be iTunes, so "jukebox" mode doesn't really make sense for to support. AutoDJ is a stand-in for the real DJ when they need to take a break, and I'm against scope creeping it into the music-player part of Mixxx.

@daschuer -- I don't think the main cue point should play a role in AutoDJ. The main cue has a purpose defined by the user that will vary depending on the user's style of DJing.

Also, if the main cue point is going to be automatically set, it should be set to the first detected beat, right? Not the first time non-silence frame in the track? (e.g. what would the benefit of that be over the first beat?)

What's wrong with this PR automatically creating AutoDJ in/out cues, as requested in
https://bugs.launchpad.net/mixxx/+bug/673515
that's a simple, well scoped concept that brings us to parity with other DJ software without adding preference options, etc. WDYT?

@Be-ing
Copy link
Contributor

Be-ing commented May 4, 2017

Also, if the main cue point is going to be automatically set, it should be set to the first detected beat, right? Not the first time non-silence frame in the track? (e.g. what would the benefit of that be over the first beat?)

No, I think it should be set to the beginning of non-silence. I think quantize should be responsible for starting playback on beat, which it already does. An example use case for starting playback at the beginning of non-silence rather than the first beat could be mixing ambient tracks with different tempos and not beatmatching.

What's wrong with this PR automatically creating AutoDJ in/out cues, as requested in
https://bugs.launchpad.net/mixxx/+bug/673515
that's a simple, well scoped concept that brings us to parity with other DJ software without adding preference options, etc. WDYT?

I think that is a good idea. These new markers shouldn't be exclusively for AutoDJ though. They are also useful as reminders set up when preparing tracks. But I'm not sure I'd want two autogenerated marks appearing on the waveform for every track...

@daschuer
Copy link
Member

daschuer commented May 6, 2017

IMHO the scope of this PR is just to auto detect silence at the very start
and end of the track.

The discussion is about how to store, use and tweak this information.

I think we are close to a conclusion, that these points schould be non
quantised, and user editable cue points. If we can agree on that, we need
to consider how to protect user edits.
IMHO a good solution is to do it along with the beatgrid lock feature. Do
you think this will work?


The other open issues is how to use these points.

"Load cue": currently we have a preference option to load a track either at
THE cue or the very start. The later fails, if the track has a notable
amount of silence. This issue can be fixed with the new start cue.

"First down beat": this is a highly desired feature for me, but cannot be
solved with this PR. We should just check if it will not become harder. In
my use case, wich schould be quite common, I use THE cue to mark this
point. Mixxx supports it already well, wenn we load a track at THE cue. So
we may improve this experience by initial set The cue to the first down
beat, once we can. This will not break any other Cue use case.

"End cue" this has only a value for Mixxx auto features. It can be used for
all end related features, like end of track warning and AutoDj, instead of
the track end.

"Jukebox mode": The AutoDj currently already support the jukebox mode using
a negative transition time. It is used for ballroom dance events. Currently
it fails for tracks with extra silence at the start or end. It also fails
if we load the track a on the first down beat. Thew cues can be used to
fix it. This is true for Auto DJ an manual mode. So I think we need a way
to tell Mix where to load the track. QQ

"Edit start point": normally this is done by The cue buttons. The
controllers don`t have an extra set of buttons to edit the "jukebox mode"
start. Without this feature, the user will probably cue the track by using
The Cue. One option could be to recognise the situation somehow and move
the start point along with the cue.

"Declutter the view" a disco user probably does not care about the start
point. On the other hand a ballroom user probably does not care about the
first down beat. So there is a chance to deal with it somehow. Any ideas?

@ninomp
Copy link
Contributor Author

ninomp commented May 8, 2017

@daschuer Thank you for summarizing the discussion.

IMHO a good solution is to do it along with the beatgrid lock feature. Do you think this will work?

I don't find this very intuitive. I would prefer to introduce 'source' field to cue and make AnalyzerSilence modify only those cues that have not been modified by user. However, in order to enable user to cancel his edits are make Mixxx detect these cues again, we need to make sure these cue points can be deleted.

But I'm not sure I'd want two autogenerated marks appearing on the waveform for every track...

Since I don't know how to solve this without introducing new modes or preference options, how about if we just continue using the main cue point as AutoDJ start? This way we are not only solving issues "edit start point" and "declutter the view", but we are also preserving backward compatibility for users who are currently using the main cue point for AutoDJ start. Besides, I don't think we have many users who are using same tracks for AutoDJ and manual Djing. (Am I wrong? I suppose these users are currently using Hotcues for manual Djing.)

What is the conclusion with AutoDJ cues and modes? Do we really need separate cue points for the two AutoDJ use cases/modes?

@daschuer
Copy link
Member

daschuer commented May 9, 2017 via email

@ronso0
Copy link
Member

ronso0 commented May 1, 2019

A bug:
pressing intro_start_clear makes the playposition jump to CUE.
Playposition should stay where it is so that the intro marker can easily be moved.

@ronso0
Copy link
Member

ronso0 commented May 1, 2019

I vote for creating a way to hide the Intro/Outro functionality before we release 2.3

Although this is a Waveform feature, I guess the option should rather be in Decks because it's there where we have the overview and the respective buttons.

@ninomp
Copy link
Contributor Author

ninomp commented May 3, 2019

@ronso0 We already have an option in all skins that allows users to hide Intro/Outro cues on waveforms and overviews as well as buttons for setting/clearing these cues. However, these cues will still be automatically placed on all tracks. Are you looking for option to disable automatic placement of these cues?

A bug:
pressing intro_start_clear makes the playposition jump to CUE.
Playposition should stay where it is so that the intro marker can easily be moved.

Thank you for reporting. I'll investigate this.

@ronso0
Copy link
Member

ronso0 commented May 3, 2019

We already have an option in all skins that allows users to hide Intro/Outro cues on waveforms and overviews as well as buttons for setting/clearing these cues.

...ahemm, jup..never mind! I completely missed that as I didn't expect it there...although I maybe even suggested it myself

Are you looking for option to disable automatic placement of these cues?

Nope, that's fine. An option would unnecessarily increase the complexity.

I rarely use AutoDJ and if I do it's mostly personal my private jukebox, therefore my question:
When AutoDJ is toggled, would it help users if the Intro/outro cues would be enabled (if hidden, and hidden again if they were hidden before)?

@ninomp
Copy link
Contributor Author

ninomp commented May 3, 2019

When AutoDJ is toggled, would it help users if the Intro/outro cues would be enabled (if hidden, and hidden again if they were hidden before)?

Yes, that makes sense. Without this, users will be confused why AutoDJ is behaving differently.

@ronso0
Copy link
Member

ronso0 commented May 4, 2019

pressing intro_start_clear makes the playposition jump to CUE.

not 100% reproducible though

@ronso0
Copy link
Member

ronso0 commented May 4, 2019

what's the purpose of setting the Intro cue before te track start?
As mentioned, I do this manually but is this intended for the AutoDJ use case?
intro-cue

@ninomp
Copy link
Contributor Author

ninomp commented May 4, 2019

The idea was to quantize start markers to previous beat and end markers to next beat, so that the range would be longer. I am not sure how intuitive this is. It is easy to change this, if we decide to do so.
Main cue point is quantized to closest beat.

@ronso0
Copy link
Member

ronso0 commented May 4, 2019

... and end markers to next beat

I wasn't aware we can play beyond the track end. I recall there was a bug that we can create loops that go beyond the end but they'd fail to play..
(screwed up my master build, so I can't test right now with Intro feature included)

@daschuer
Copy link
Member

daschuer commented May 5, 2019

Now Auto DJ always loads ad intro start, right? This works perfectly fot the Juke Box mode, but fails for all tracks which are already CUEed on the first beat for the DJ use case.
How can we adjust the setSeekOnLoadMode?

Sorry I have not followed the discussion, but I think it currently breaks the users use case.
Can we maintain two setSeekOnLoadModes In preferences? One in the Deck option and one the Auto DJ Prefences that allows to conditional override it?

@Be-ing
Copy link
Contributor

Be-ing commented May 6, 2019

I think it is up to the user to set the intro start point wherever they set the cue point previously. I am reluctant to add an option to revert to the old behavior. If this turns out to be a major issue for anyone, perhaps we could instruct them how to run a SQL query to set intro start points where they have the cue point for their whole library. But that should only be a one time issue when upgrading to 2.3, so I'm reluctant to add something to the GUI just for this.

However, I have realized it would be helpful for me to load tracks at the intro start point regardless of whether AutoDJ is in use. The durations shown on the overview waveforms calculate the length of the sections. Together with the automatic silence detection, this can be misleading if the track is loaded at the start point. For example, I have a track where the automatically placed intro start point was at about 0:01. The intro end point was at 0:24. The overview showed 0:23, as it should, but the track was loaded at 0:00. So if I pressed play thinking I would have 23 seconds to mix in the track, that could throw off the mix because the intro would actually end 24 seconds later. If the track was loaded at the intro start point, that would not be a problem.

@daschuer
Copy link
Member

daschuer commented May 6, 2019

We have clearly identified two use cases for Auto DJ. One is playing like a juke box with a small or no gap. The other is to let the tracks overlap for betmaching.

Currently only one is supported, before, we have supported both. This is a regression, which is required to fix.

The other issue that need to be fixed is the migration part.

All DJs like me used the load to cue option have cued the tracks to the first beat. Or their desired load point. We may argue that this was a workaround, but that is the current state. We cannot ignore.

The new feature is very nice and it frees up the CUE for temporary things, but we must not ignore the legacy. So let's find a solution instead of argue against it.
Ideas?

@Be-ing
Copy link
Contributor

Be-ing commented May 6, 2019

A fix that allows users to switch back to the old behavior is here: #2103.

@ronso0
Copy link
Member

ronso0 commented Jul 17, 2019

while playing with #2205 I expected that holding in/outro cues should jump to the respective position AND keep playing as long as the cue button is hold down

lp:1836886

@Be-ing
Copy link
Contributor

Be-ing commented Aug 11, 2019

I noticed the new intro/outro cue Controls had not been documented, so I added them to the wiki and mentioned them on https://mixxx.org/wiki/doku.php/updating_controller_mappings

@ywwg
Copy link
Member

ywwg commented Aug 12, 2019

We should also add information about intro/outtro markers to the mixxx manual, if that hasn't been done already.

@Be-ing
Copy link
Contributor

Be-ing commented Aug 12, 2019

Yes, certainly. But let's wait until we finish #2103 and reach a consensus on how AutoDJ should use them first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.