Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lazy call_rcu() updates for v6.2 #24

Open
wants to merge 92 commits into
base: rcu-dev
Choose a base branch
from
Open

Lazy call_rcu() updates for v6.2 #24

wants to merge 92 commits into from

Conversation

kernel-patches-bot
Copy link

Pull request for series with
subject: Lazy call_rcu() updates for v6.2
version: 1
url: https://patchwork.kernel.org/project/rcu/list/?series=686916

@paulmckrcu
tools/memory-model: Document locking corner cases
d078eb7 
Most Linux-kernel uses of locking are straightforward, but there are
corner-case uses that rely on less well-known aspects of the lock and
unlock primitives.  This commit therefore adds a locking.txt and litmus
tests in Documentation/litmus-tests/locking to explain these corner-case
uses.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh note timeouts
d55a79c 
Currently, judgelitmus.sh treats timeouts (as in the "--timeout" argument)
as "!!! Verification error".  This can be misleading because it is quite
possible that running the test longer would have produced a verification.
This commit therefore changes judgelitmus.sh to check for timeouts and
to report them with "!!! Timeout".

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make cmplitmushist.sh note timeouts
3e0e0d9 
Currently, cmplitmushist.sh treats timeouts (as in the "--timeout"
argument) as "Missing Observation line".  This can be misleading because
it is quite possible that running the test longer would have produced
a verification.  This commit therefore changes cmplitmushist.sh to check
for timeouts and to report them with "Timed out".

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh identify bad macros
ce26c0d 
Currently, judgelitmus.sh treats use of unknown primitives (such as
srcu_read_lock() prior to SRCU support) as "!!! Verification error".
This can be misleading because it fails to call out typos and running
a version LKMM on a litmus test requiring a feature not provided by
that version.  This commit therefore changes judgelitmus.sh to check
for unknown primitives and to report them, for example, with:

	'!!! Current LKMM version does not know "rcu_write_lock"'.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh detect hard deadlocks
e450f7e 
If a litmus test specifies "Result: Never" and if it contains an
unconditional ("hard") deadlock, then running checklitmus.sh on it will
not flag any errors, despite the fact that there are no executions.
This commit therefore updates judgelitmus.sh to complain about tests
with no executions that are marked, but not as "Result: DEADLOCK".

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Fix paulmck email address on pre-existing scripts
c82c61e 
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Update parseargs.sh for hardware verification
4be1396 
This commit adds a --hw argument to parseargs.sh to specify the CPU
family for a hardware verification.  For example, "--hw AArch64" will
specify that a C-language litmus test is to be translated to ARMv8 and
the result verified.  This will set the LKMM_HW_MAP_FILE environment
variable accordingly.  If there is no --hw argument, this environment
variable will be set to the empty string.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh handle hardware verifications
a0ef691 
This commit makes the judgelitmus.sh script check the --hw argument
(AKA the LKMM_HW_MAP_FILE environment variable) and to adjust its
judgment for a run where a C-language litmus test has been translated to
assembly and the assembly version verified.  In this case, the assembly
verification output is checked against the C-language script's "Result:"
comment.  However, because hardware can be stronger than LKMM requires,
the judgelitmus.sh script forgives verification mismatches featuring
a "Sometimes" in the C-language script and an "Always" or "Never"
assembly-language verification.

Note that deadlock is not forgiven, however, this should not normally be
an issue given that C-language tests containing locking, RCU, or SRCU
cannot be translated to assembly.  However, this issue can crop up in
litmus tests that mimic deadlock by using the "filter" clause to ignore
all executions.  It can also crop up when certain herd arguments are
used to autofilter everything that does not match the "exists" clause
in cases where the "exists" clause cannot be satisfied.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Add simpletest.sh to check locking, RCU, and SRCU
8fca8d0 
This commit abstracts out common function to check a given litmus test
for locking, RCU, and SRCU in order to avoid duplicating code.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Fix checkalllitmus.sh comment
b483d17 
The checkalllitmus.sh runs litmus tests in the litmus-tests directory,
not those in the github archive, so this commit updates the comment to
reflect this reality.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Hardware checking for check{,all}litmus.sh
415343c 
This commit makes checklitmus.sh and checkalllitmus.sh check to see
if a hardware verification was specified (via the --hw command-line
argument, which sets the LKMM_HW_MAP_FILE environment variable).
If so, the C-language litmus test is converted to the specified type
of assembly-language litmus test and herd is run on it.  Hardware is
permitted to be stronger than LKMM requires, so "Always" and "Never"
verifications of "Sometimes" C-language litmus tests are forgiven.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh ransack .litmus.out files
a4da1dc 
The judgelitmus.sh script currently relies solely on the "Result:"
comment in the .litmus file.  This is problematic when using the --hw
argument, because it is necessary to check the hardware model against
LKMM even in the absence of "Result:" comments.

This commit therefore modifies judgelitmus.sh to check the observation
in a .litmus.out file, in case one was generated by a previous LKMM run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Split runlitmus.sh out of checklitmus.sh
b8e2918 
This commit prepares for adding --hw capability to github litmus-test
scripts by splitting runlitmus.sh (which simply runs the verification)
out of checklitmus.sh (which also judges the results).

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make runlitmus.sh generate .litmus.out for --hw
92e93ba 
In the absence of "Result:" comments, the runlitmus.sh script relies on
litmus.out files from prior LKMM runs.  This can be a bit user-hostile,
so this commit makes runlitmus.sh generate any needed .litmus.out files
that don't already exist.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Move from .AArch64.litmus.out to .litmus.AArch.out
07492ae 
When the github scripts see ".litmus.out", they assume that there must be
a corresponding C-language ".litmus" file.  Won't they be disappointed
when they instead see nothing, or, worse yet, the corresponding
assembly-language litmus test?  This commit therefore swaps the hardware
tag with the "litmus" to avoid this sort of disappointment.

This commit also adjusts the .gitignore file so as to avoid adding these
new ".out" files to git.

[ paulmck: Apply Akira Yokosawa feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Keep assembly-language litmus tests
1585dd5 
This commit retains the assembly-language litmus tests generated from
the C-language litmus tests, appending the hardware tag to the original
C-language litmus test's filename.  Thus, S+poonceonces.litmus.AArch64
contains the Armv8 assembly language corresponding to the C-language
S+poonceonces.litmus test.

This commit also updates the .gitignore to avoid committing these
automatically generated assembly-language litmus tests.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Allow herd to deduce CPU type
199d988 
Currently, the scripts specify the CPU's .cat file to herd.  But this is
pointless because herd will select a good and sufficient .cat file from
the assembly-language litmus test itself.  This commit therefore removes
the -model argument to herd, allowing herd to figure the CPU family out
itself.

Note that the user can override herd's choice using the "--herdopts"
argument to the scripts.

Suggested-by: Luc Maranget <luc.maranget@inria.fr>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make runlitmus.sh check for jingle errors
f85c750 
It turns out that the jingle7 tool is currently a bit picky about
the litmus tests it is willing to process.  This commit therefore
ensures that jingle7 failures are reported.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Add -v flag to jingle7 runs
d0dc9d3 
Adding the -v flag to jingle7 invocations gives much useful information
on why jingle7 didn't like a given litmus test.  This commit therefore
adds this flag and saves off any such information into a .err file.

Suggested-by: Luc Maranget <luc.maranget@inria.fr>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Implement --hw support for checkghlitmus.sh
660959d 
This commits enables the "--hw" argument for the checkghlitmus.sh script,
causing it to convert any applicable C-language litmus tests to the
specified flavor of assembly language, to verify these assembly-language
litmus tests, and checking compatibility of the outcomes.

Note that the conversion does not yet handle locking, RCU, SRCU, plain
C-language memory accesses, or casts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Fix scripting --jobs argument
935d88f 
The parseargs.sh regular expression for the --jobs argument incorrectly
requires that the number of jobs be at least 10, that is, have at least
two digits.  This commit therefore adjusts this regular expression to
allow single-digit numbers of jobs to be specified.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make checkghlitmus.sh use mselect7
8061f85 
The checkghlitmus.sh script currently uses grep to ignore non-C-language
litmus tests, which is a bit fragile.  This commit therefore enlists the
aid of "mselect7 -arch C", given Luc Maraget's recent modifications that
allow mselect7 to operate in filter mode.

This change requires herdtools 7.52-32-g1da3e0e50977 or later.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make history-check scripts use mselect7
ce208b6 
The history-check scripts currently use grep to ignore non-C-language
litmus tests, which is a bit fragile.  This commit therefore enlists the
aid of "mselect7 -arch C", given Luc Maraget's recent modifications that
allow mselect7 to operate in filter mode.

This change requires herdtools 7.52-32-g1da3e0e50977 or later.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Add "--" to parseargs.sh for additional arguments
c5cdcac 
Currently, parseargs.sh expects to consume all the command-line arguments,
which prevents the calling script from having any of its own arguments.
This commit therefore causes parseargs.sh to stop consuming arguments
when it encounters a "--" argument, leaving any remaining arguments for
the calling script.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Repair parseargs.sh header comment
8fe3019 
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Add checktheselitmus.sh to run specified litmus t…
b53862d 
…ests

This commit adds a checktheselitmus.sh script that runs the litmus tests
specified on the command line.  This is useful for verifying fixes to
specific litmus tests.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Add data-race capabilities to judgelitmus.sh
7458b3b 
This commit adds functionality to judgelitmus.sh to allow it to handle
both the "DATARACE" markers in the "Result:" comments in litmus tests
and the "Flag data-race" markers in LKMM output.  For C-language tests,
if either marker is present, the other must also be as well, at least for
litmus tests having a "Result:" comment.  If the LKMM output indicates
a data race, then failures of the Always/Sometimes/Never portion of the
"Result:" prediction are forgiven.

The reason for forgiving "Result:" mispredictions is that data races can
result in "interesting" compiler optimizations, so that all bets are off
in the data-race case.

[ paulmck: Apply Akira Yokosawa feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Make judgelitmus.sh handle scripted Result: tag
40652cf 
The scripts that generate the litmus tests in the "auto" directory of
the https://github.com/paulmckrcu/litmus archive place the "Result:"
tag into a single-line ocaml comment, which judgelitmus.sh currently
does not recognize.  This commit therefore makes judgelitmus.sh
recognize both the multiline comment format that it currently does
and the automatically generated single-line format.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
tools/memory-model: Use "-unroll 0" to keep --hw runs finite
5e12d26 
Litmus tests involving atomic operations produce LL/SC loops on a number
of architectures, and unrolling these loops can result in excessive
verification times or even stack overflows.  This commit therefore uses
the "-unroll 0" herd7 argument to avoid unrolling, on the grounds that
additional passes through an LL/SC loop should not change the verification.

Note however, that certain bugs in the mapping of the LL/SC loop to
machine instructions may go undetected.  On the other hand, herd7 might
not be the best vehicle for finding such bugs in any case.  (You do
stress-test your architecture-specific code, don't you?)

Suggested-by: Luc Maranget <luc.maranget@inria.fr>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@paulmckrcu
Merge branch 'nolibc.2022.08.31b' into HEAD
3d5bfdc 
nolibc.2022.08.31b: nolibc updates
@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

Frederic Weisbecker and others added 7 commits October 19, 2022 23:41
@chantra
rcu: Fix missing nocb gp wake on rcu_barrier()
266663d 
In preparation for RCU lazy changes, wake up the RCU nocb gp thread if
needed after an entrain.  This change prevents the RCU barrier callback
from waiting in the queue for several seconds before the lazy callbacks
in front of it are serviced.

Reported-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rcu: Make call_rcu() lazy to save power
bd3c443 
Implement timer-based RCU callback batching (also known as lazy
callbacks). With this we save about 5-10% of power consumed due
to RCU requests that happen when system is lightly loaded or idle.

By default, all async callbacks (queued via call_rcu) are marked
lazy. An alternate API call_rcu_flush() is provided for the few users,
for example synchronize_rcu(), that need the old behavior.

The batch is flushed whenever a certain amount of time has passed, or
the batch on a particular CPU grows too big. Also memory pressure will
flush it in a future patch.

To handle several corner cases automagically (such as rcu_barrier() and
hotplug), we re-use bypass lists which were originally introduced to
address lock contention, to handle lazy CBs as well. The bypass list
length has the lazy CB length included in it. A separate lazy CB length
counter is also introduced to keep track of the number of lazy CBs.

[ paulmck: Fix formatting of inline call_rcu_lazy() definition. ]

Suggested-by: Paul McKenney <paulmck@kernel.org>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rcu: Refactor code a bit in rcu_nocb_do_flush_bypass()
e075618 
This consolidates the code a bit and makes it cleaner. Functionally it
is the same.

Reported-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@chantra
rcu: Shrinker for lazy rcu
ec31536 
The shrinker is used to speed up the free'ing of memory potentially held
by RCU lazy callbacks. RCU kernel module test cases show this to be
effective. Test is introduced in a later patch.

Signed-off-by: Vineeth Pillai <vineeth@bitbyteword.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rcuscale: Add laziness and kfree tests
5635844 
This commit adds 2 tests to rcuscale.  The first one is a startup test
to check whether we are not too lazy or too hard working.  The second
one causes kfree_rcu() itself to use call_rcu() and checks memory
pressure. Testing indicates that the new call_rcu() keeps memory pressure
under control roughly as well as does kfree_rcu().

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
percpu-refcount: Use call_rcu_flush() for atomic switch
b4515a3 
call_rcu() changes to save power will slow down the percpu refcounter's
"per-CPU to atomic switch" path. The primitive uses RCU when switching to
atomic mode. The enqueued async callback wakes up waiters waiting in the
percpu_ref_switch_waitq. Due to this, per-CPU refcount users will slow down,
such as blk_pre_runtime_suspend().

Use the call_rcu_flush() API instead which reverts to the old behavior.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rcu/sync: Use call_rcu_flush() instead of call_rcu
032a3b3 
call_rcu() changes to save power will slow down rcu sync. Use the
call_rcu_flush() API instead which reverts to the old behavior.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

joelagnel and others added 5 commits October 19, 2022 23:41
@joelagnel @chantra
rcu/rcuscale: Use call_rcu_flush() for async reader test
1d6e5e9 
rcuscale uses call_rcu() to queue async readers. With recent changes to
save power, the test will have fewer async readers in flight. Use the
call_rcu_flush() API instead to revert to the old behavior.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rcu/rcutorture: Use call_rcu_flush() where needed
9a1f034 
call_rcu() changes to save power will change the behavior of rcutorture
tests. Use the call_rcu_flush() API instead which reverts to the old
behavior.

Reported-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@urezki @chantra
scsi/scsi_error: Use call_rcu_flush() instead of call_rcu()
32e826f 
Slow boot time is seen on KVM running typical Linux distributions due to
SCSI layer calling call_rcu(). Recent changes to save power may be
causing this slowness. Using call_rcu_flush() fixes the issue and brings
the boot time back to what it originally was. Convert it.

Tested-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@urezki @chantra
workqueue: Make queue_rcu_work() use call_rcu_flush()
cbce0ac 
call_rcu() changes to save power will slow down RCU workqueue items
queued via queue_rcu_work(). This may not be an issue, however we cannot
assume that workqueue users are OK with long delays. Use
call_rcu_flush() API instead which reverts to the old behavior.

Signed-off-by: Uladzislau Rezki <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@joelagnel @chantra
rxrpc: Use call_rcu_flush() instead of call_rcu()
8c87f0f 
call_rcu() changes to save power may cause slowness. Use the
call_rcu_flush() API instead which reverts to the old behavior.

We find this via inspection that the RCU callback does a wakeup of a
thread. This usually indicates that something is waiting on it. To be
safe, let us use call_rcu_flush() here instead.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
@kernel-patches-bot
Copy link
Author

Master branch: fa70e60
series: https://patchwork.kernel.org/project/rcu/list/?series=686916
version: 1

Pull request is NOT updated. Failed to apply https://patchwork.kernel.org/project/rcu/list/?series=686916
error message:

Cmd('git') failed due to: exit code(128)
  cmdline: git am -3
  stdout: 'Applying: rcu: Simplify rcu_init_nohz() cpumask handling
Using index info to reconstruct a base tree...
M	kernel/rcu/tree_nocb.h
Falling back to patching base and 3-way merge...
Auto-merging kernel/rcu/tree_nocb.h
CONFLICT (content): Merge conflict in kernel/rcu/tree_nocb.h
Patch failed at 0001 rcu: Simplify rcu_init_nohz() cpumask handling
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".'
  stderr: 'error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch'

conflict:

diff --cc kernel/rcu/tree_nocb.h
index 9e1c8caec5ce,ce526cc2791c..000000000000
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@@ -1312,70 -1207,21 +1312,84 @@@ int rcu_nocb_cpu_offload(int cpu
  }
  EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload);
  
 +static unsigned long
 +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
 +{
 +	int cpu;
++<<<<<<< HEAD
 +	unsigned long count = 0;
 +
 +	/* Snapshot count of all CPUs */
 +	for_each_possible_cpu(cpu) {
 +		struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
 +
 +		count +=  READ_ONCE(rdp->lazy_len);
 +	}
 +
 +	return count ? count : SHRINK_EMPTY;
 +}
 +
 +static unsigned long
 +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 +{
 +	int cpu;
 +	unsigned long flags;
 +	unsigned long count = 0;
 +
 +	/* Snapshot count of all CPUs */
 +	for_each_possible_cpu(cpu) {
 +		struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
 +		int _count = READ_ONCE(rdp->lazy_len);
 +
 +		if (_count == 0)
 +			continue;
 +		rcu_nocb_lock_irqsave(rdp, flags);
 +		WRITE_ONCE(rdp->lazy_len, 0);
 +		rcu_nocb_unlock_irqrestore(rdp, flags);
 +		wake_nocb_gp(rdp, false);
 +		sc->nr_to_scan -= _count;
 +		count += _count;
 +		if (sc->nr_to_scan <= 0)
 +			break;
 +	}
 +	return count ? count : SHRINK_STOP;
 +}
 +
 +static struct shrinker lazy_rcu_shrinker = {
 +	.count_objects = lazy_rcu_shrink_count,
 +	.scan_objects = lazy_rcu_shrink_scan,
 +	.batch = 0,
 +	.seeks = DEFAULT_SEEKS,
 +};
 +
  void __init rcu_init_nohz(void)
  {
  	int cpu;
  	struct rcu_data *rdp;
  	const struct cpumask *cpumask = NULL;
  
++=======
++	struct rcu_data *rdp;
++	const struct cpumask *cpumask = NULL;
++
++>>>>>>> rcu: Simplify rcu_init_nohz() cpumask handling
  #if defined(CONFIG_NO_HZ_FULL)
  	if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))
  		cpumask = tick_nohz_full_mask;
  #endif
++<<<<<<< HEAD
+ 
+ 	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) &&
+ 	    !rcu_state.nocb_is_setup && !cpumask)
+ 		cpumask = cpu_possible_mask;
+ 
++=======
 +
 +	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) &&
 +	    !rcu_state.nocb_is_setup && !cpumask)
 +		cpumask = cpu_possible_mask;
 +
++>>>>>>> rcu: Simplify rcu_init_nohz() cpumask handling
  	if (cpumask) {
  		if (!cpumask_available(rcu_nocb_mask)) {
  			if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) {
@@@ -1391,9 -1237,6 +1405,12 @@@
  	if (!rcu_state.nocb_is_setup)
  		return;
  
++<<<<<<< HEAD
 +	if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy"))
 +		pr_err("Failed to register lazy_rcu shrinker!\n");
 +
++=======
++>>>>>>> rcu: Simplify rcu_init_nohz() cpumask handling
  	if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {
  		pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n");
  		cpumask_and(rcu_nocb_mask, cpu_possible_mask,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
merge-conflict new rcu-dev V1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.