finished_at_start/end job filters #111

Xeitor · 2024-04-05T21:56:56Z

Changelog

added finished_at range filter support for jobs

Issue: #30

PD: I'll split the different filters in different PRs for easier review

Xeitor · 2024-04-09T23:48:14Z

Hi @rosa! Don't mean to bother you but in case you missed this this PR is ready for review :)

rosa · 2024-04-10T19:15:32Z

@Xeitor thanks for letting me know, I'll try to review soon!

lib/active_job/jobs_relation.rb

Xeitor · 2024-05-15T12:34:03Z

Hi @rosa, hope you are doing great

Sorry to bother you, wanted to know if you were able to take a look at the PR :)

app/controllers/concerns/mission_control/jobs/job_filters.rb

rosa · 2024-09-13T18:33:18Z

Hey @Xeitor, so sorry about this huge delay! I haven't had time to focus on Mission Control lately, but it's definitely in the plans for October, I need to spend a week or so just working on all pending PRs and missing features 😳

Xeitor · 2024-09-25T23:22:53Z

Hey @Xeitor, so sorry about this huge delay! I haven't had time to focus on Mission Control lately, but it's definitely in the plans for October, I need to spend a week or so just working on all pending PRs and missing features 😳

for sure no problem

lib/active_job/jobs_relation.rb

rosa · 2024-11-04T20:46:23Z

Hey @Xeitor, I'm finally back with Mission Control. Are you still interested in working on this one? I'd understand if not, given how long it's been, so sorry about that! If you're no longer interested, I'm happy to take it from here.

Xeitor · 2024-11-08T01:27:38Z

hi @rosa, glad to have you back. For sure, no worries, I 'll take a look at the comments in the next days

app/controllers/concerns/mission_control/jobs/job_filters.rb

app/views/mission_control/jobs/jobs/_filters.html.erb

rosa · 2024-11-17T19:26:07Z

lib/active_job/jobs_relation.rb

      worker_id: worker_id,
-      recurring_task_id: recurring_task_id
+      recurring_task_id: recurring_task_id,
    }.compact.collect { |key, value| [ key, value.to_s ] }.to_h

+    arguments.merge!(finished_at: finished_at) if finished_at.present? && !finished_at.to_s.match?(/^..$/)


Looks like Rails does the right thing when you provide a beginless and endless range in a condition value, so you don't need to worry about checking this here. This can be kept as:

arguments = { job_class_name: job_class_name, queue_name: queue_name, worker_id: worker_id, recurring_task_id: recurring_task_id, finished_at: finished_at, }.compact.collect { |key, value| [ key, value.to_s ] }.to_h

not sure about this, for example:

dummy(dev)> hash = { finished_at: nil..nil }.compact => {:finished_at=>nil..} dummy(dev)> hash.collect { |key, value| [ key, value.to_s ] }.to_h => {:finished_at=>".."}

will override finished_at to an empty range

I think that's not an empty range, but an "infinite" range, a range without start and end.

besides, I did not want not stringity the range, check out this behaviour:

dummy(dev)> ActiveJob.jobs.finished.where(finished_at: (..Time.now).to_s).first SolidQueue::Job Load (0.2ms) SELECT "solid_queue_jobs".* FROM "solid_queue_jobs" WHERE "solid_queue_jobs"."finished_at" IS NOT NULL AND "solid_queue_jobs"."finished_at" = ? ORDER BY "solid_queue_jobs"."finished_at" DESC LIMIT ? OFFSET ? [["finished_at", "2024-11-17 19:53:22"], ["LIMIT", 1000], ["OFFSET", 0]] => nil dummy(dev)> ActiveJob.jobs.finished.where(finished_at: ..Time.now).first SolidQueue::Job Load (0.5ms) SELECT "solid_queue_jobs".* FROM "solid_queue_jobs" WHERE "solid_queue_jobs"."finished_at" IS NOT NULL AND "solid_queue_jobs"."finished_at" <= ? ORDER BY "solid_queue_jobs"."finished_at" DESC LIMIT ? OFFSET ? [["finished_at", "2024-11-17 19:53:28.197854"], ["LIMIT", 1000], ["OFFSET", 0]] =>

Why would you convert the range into a string? That wouldn't work in the query.

mission_control-jobs/lib/active_job/jobs_relation.rb

Line 61 in a8073e7

}.compact.collect { |key, value| [ key, value.to_s ] }.to_h

this line will do the .to_s

and sorry yes you are right, we could just do: arguments.merge!(finished_at: finished_at) without the condition I wrote

Ah, sorry! Yes, we don't want to do that to the range. My comment was more addressed at this part:

if finished_at.present? && !finished_at.to_s.match?(/^..$/)

I meant we don't need that and we could just initialize arguments[:finished_at] with the value.

In any case, I think we can get rid of the to_s call on the values 🤔 It was added here because we didn't have the compact before, but I don't think we need it. You can try to remove it and see if any tests fail.

great!! thanks :)

rosa · 2024-11-17T19:26:27Z

lib/active_job/queue_adapters/solid_queue_ext.rb

@@ -40,7 +40,7 @@ def supported_job_statuses
  end

  def supported_job_filters(*)
-    [ :queue_name, :job_class_name ]
+    [ :queue_name, :job_class_name, :finished_at_start, :finished_at_end ]


I think we want finished_at here 🤔

Ah, no, this had to be split into supported_natively and supported_in_memory, but in any case, it should be just finished_at when added to supported_natively.

I had something like this in mind:

diff --git a/lib/active_job/jobs_relation.rb b/lib/active_job/jobs_relation.rb index e4e19ee..a92af46 100644 --- a/lib/active_job/jobs_relation.rb +++ b/lib/active_job/jobs_relation.rb @@ -23,7 +23,6 @@ class ActiveJob::JobsRelation include Enumerable STATUSES = %i[ pending failed in_progress blocked scheduled finished ] - FILTERS = %i[ queue_name job_class_name finished_at ] PROPERTIES = %i[ queue_name status offset_value limit_value job_class_name worker_id recurring_task_id finished_at ] attr_reader *PROPERTIES, :default_page_size @@ -276,8 +275,7 @@ class ActiveJob::JobsRelation end def filters - # @filetrs ||= queue_adapter.supported_job_filters(self).select { |property| public_send(property).present? && !queue_adapter.natively_supported_job_filters?(self, property) } - @filters ||= FILTERS.select { |property| public_send(property).present? && !queue_adapter.supports_job_filter?(self, property) } + @filters ||= queue_adapter.supported_job_filters(self).select { |property| public_send(property).present? && !queue_adapter.natively_supported_job_filters(self).include?(property) } end def ensure_failed_status diff --git a/lib/active_job/queue_adapters/resque_ext.rb b/lib/active_job/queue_adapters/resque_ext.rb index 17b2fbf..d7d28af 100644 --- a/lib/active_job/queue_adapters/resque_ext.rb +++ b/lib/active_job/queue_adapters/resque_ext.rb @@ -48,6 +48,10 @@ module ActiveJob::QueueAdapters::ResqueExt end def supported_job_filters(jobs_relation) + [ :queue_name, :job_class_name ] + end + + def natively_supported_job_filters(jobs_relation) if jobs_relation.pending? then [ :queue_name ] else [] end diff --git a/lib/active_job/queue_adapters/solid_queue_ext.rb b/lib/active_job/queue_adapters/solid_queue_ext.rb index 2db9ea1..b069504 100644 --- a/lib/active_job/queue_adapters/solid_queue_ext.rb +++ b/lib/active_job/queue_adapters/solid_queue_ext.rb @@ -39,8 +39,14 @@ module ActiveJob::QueueAdapters::SolidQueueExt SolidQueueJobs::STATUS_MAP.keys end - def supported_job_filters(*) - [ :queue_name, :job_class_name, :finished_at_start, :finished_at_end ] + def supported_job_filters(jobs_relation) + filters = [ :queue_name, :job_class_name ] + filters << :finished_at if jobs_relation.finished? + filters + end + + def natively_supported_job_filters(*) + [ :queue_name, :job_class_name, :finished_at ] end def jobs_count(jobs_relation)

We could also raise query errors if you try to use unsupported filters for a given job relation

I think natively_supported_job_filters should take a job_relation as well and do the check jobs_relation.finished?. Then you just use that in supported_job_filters 🤔

a little recap to see if we both are in the same page:
given that JobsRelations is the interface to the adapters and therefore knows all the generic filters that can be applied: FILTERS (with the extra parametrs in where method), I think the most reasonable way to select the filters that should be performed in memory is to first ask the adapter if the filter is supported but not natively.

translating this thoughts to code I was thinking something like this:

def filters @filters ||= FILTERS.select do |property| public_send(property).present? && should_perform_filter_in_memory?(filter) end end def should_perform_filter_in_memory?(filter) # example: finished_at is supported only if the status is finished queue_adapter.supports_job_filter?(self, property) && !queue_adapter.natively_supported_job_filters?(self).include?(property) end

Co-authored-by: Rosa Gutierrez <rosa.ge@gmail.com>

rosa · 2024-11-22T19:37:03Z

lib/active_job/jobs_relation.rb

@@ -274,6 +276,7 @@ def satisfy_filter?(job)
    end

    def filters
+      # @filetrs ||= queue_adapter.supported_job_filters(self).select { |property| public_send(property).present? && !queue_adapter.natively_supported_job_filters?(self, property) }


Suggested change

# @filetrs ||= queue_adapter.supported_job_filters(self).select { |property| public_send(property).present? && !queue_adapter.natively_supported_job_filters?(self, property) }

# @filetrs ||= queue_adapter.supported_job_filters(self).select { |property| public_send(property).present? && !queue_adapter.supports_job_filter_natively?(self, property) }

To be consistent with the other method, supports_job_filter?.

rosa · 2024-11-22T19:37:39Z

test/active_job/queue_adapters/adapter_testing/discard_jobs.rb

@@ -60,7 +60,7 @@ module ActiveJob::QueueAdapters::AdapterTesting::DiscardJobs
    5.times { FailingJob.perform_later }
    10.times { FailingReloadedJob.perform_later }
    perform_enqueued_jobs
-    ActiveJob.jobs.failed.where(queue_name: :queue_1).discard_all
+    ActiveJob.jobs.failed.where(queue_name: 'queue_1').discard_all


Why is this change? 🤔

oh that is because we got rid of the .to_s in the JobsRelation#where method and made the test fail

Ohhh, I see! In that case let's bring that back for queue_name. We want to be able to use symbols there! Sorry, I didn't think of that.

yep makes sense, no problem

Xeitor added 2 commits April 5, 2024 18:52

jobs_relation finished_at_start and finished_at_end filters

704cb86

natively_supported finished_at_* filters for solid_queue

5ca0682

Xeitor marked this pull request as draft April 5, 2024 21:57

Xeitor added 3 commits April 5, 2024 21:56

added filters to _filters partial and params

b469206

added timezone

2e4291d

show finished_at filters if jobs_status is 'finished'

5b41d16

Xeitor marked this pull request as ready for review April 7, 2024 14:38

added comments to jobs_relation where method

60eec4d

Xeitor added 2 commits April 10, 2024 16:16

clean finished_at_* filter values when clear button clicked

b2c0808

double instead of single quoted strings (rubocop)

f0753a9

Xeitor commented Apr 10, 2024

View reviewed changes

lib/active_job/jobs_relation.rb Outdated Show resolved Hide resolved

rosa reviewed Jun 6, 2024

View reviewed changes

app/controllers/concerns/mission_control/jobs/job_filters.rb Outdated Show resolved Hide resolved

enhacements set filters

62d5b3b

Xeitor requested a review from rosa August 18, 2024 13:34

rosa reviewed Nov 4, 2024

View reviewed changes

lib/active_job/jobs_relation.rb Outdated Show resolved Hide resolved

Xeitor added 2 commits November 11, 2024 21:52

Merge branch 'main' into jobs_finished_at_filters

ef2e964

finished_at range enhacement

a8073e7

rosa reviewed Nov 17, 2024

View reviewed changes

app/controllers/concerns/mission_control/jobs/job_filters.rb Outdated Show resolved Hide resolved

rosa reviewed Nov 17, 2024

View reviewed changes

app/views/mission_control/jobs/jobs/_filters.html.erb Outdated Show resolved Hide resolved

rosa reviewed Nov 17, 2024

View reviewed changes

Xeitor and others added 2 commits November 17, 2024 16:57

enhace date_with_time_zone method

3525169

Co-authored-by: Rosa Gutierrez <rosa.ge@gmail.com>

back to @job_filters in _filters

5503402

Xeitor added 3 commits November 17, 2024 16:58

god bye presence

38b3bf3

bye collect arguments with to_s for values

25142b7

fix tests

9d95d16

MacLove13 mentioned this pull request Nov 20, 2024

Filter by date #209

Open

rosa reviewed Nov 22, 2024

View reviewed changes

bringed back .to_s for queue_name parameters (JobsRelation#where)

4d7cb3e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

finished_at_start/end job filters #111

finished_at_start/end job filters #111

Xeitor commented Apr 5, 2024

Xeitor commented Apr 9, 2024

rosa commented Apr 10, 2024

Xeitor commented May 15, 2024

rosa commented Sep 13, 2024

Xeitor commented Sep 25, 2024

rosa commented Nov 4, 2024

Xeitor commented Nov 8, 2024 •

edited

Loading

rosa Nov 17, 2024

Xeitor Nov 17, 2024

rosa Nov 17, 2024

Xeitor Nov 17, 2024

rosa Nov 17, 2024

Xeitor Nov 17, 2024

Xeitor Nov 17, 2024

rosa Nov 17, 2024

Xeitor Nov 17, 2024

rosa Nov 17, 2024

rosa Nov 17, 2024

Xeitor Nov 17, 2024

Xeitor Nov 18, 2024 •

edited

Loading

rosa Nov 22, 2024

Xeitor Nov 24, 2024 •

edited

Loading

rosa Nov 22, 2024

rosa Nov 22, 2024

Xeitor Nov 22, 2024

rosa Nov 22, 2024 •

edited

Loading

Xeitor Nov 22, 2024

	# @filetrs \|\|= queue_adapter.supported_job_filters(self).select { \|property\| public_send(property).present? && !queue_adapter.natively_supported_job_filters?(self, property) }
	# @filetrs \|\|= queue_adapter.supported_job_filters(self).select { \|property\| public_send(property).present? && !queue_adapter.supports_job_filter_natively?(self, property) }

finished_at_start/end job filters #111

Are you sure you want to change the base?

finished_at_start/end job filters #111

Conversation

Xeitor commented Apr 5, 2024

Changelog

Xeitor commented Apr 9, 2024

rosa commented Apr 10, 2024

Xeitor commented May 15, 2024

rosa commented Sep 13, 2024

Xeitor commented Sep 25, 2024

rosa commented Nov 4, 2024

Xeitor commented Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xeitor Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xeitor Nov 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosa Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xeitor commented Nov 8, 2024 •

edited

Loading

Xeitor Nov 18, 2024 •

edited

Loading

Xeitor Nov 24, 2024 •

edited

Loading

rosa Nov 22, 2024 •

edited

Loading