crimson/os/seastore/epm: also do prepare_write for rbm

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

* refs/pull/50217/head:
	qa: enable the xfstests generic/317 test case
	qa: make the xfstests tmp directories to be readable and excutable

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

rgw/rados: remove sal handles from RGWRados::Object ops

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: don't dereference nullopt in DeleteMultiObj

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

copy and move operations were not deleted, so the compiler-generated
move operations were implemented as copies. the destructor of the
copied-from instance would destroy the handles that were still in use
by the copied-to instance, resulting in use-after-free or double-free

to prevent this, add noexcept move operations that leave the moved-from
instance empty. this also prevents the compiler from generating the
problematic copy operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

When emplacing objects into the trim notify handler of
BucketTrimWatcher, use a unique_ptr for the handler so that it is
destroyed if the emplace fails.

Though the destructor is already called, this behaviour cannot be relied
upon. std::map does not exhibit the same behaviour, and would have
leaked memory had it been used instead.

Fixes: https://tracker.ceph.com/issues/57938
Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

although torrent info is moving into RGW_ATTR_TORRENT, we'll still need
to support the rados backend's omap fallback.. forever. expose a new
Object API specifically for torrent info so we can remove all of the
omap-specific ones

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: catch manifest decode errors in GetObj

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

* refs/pull/49756/head:
	qa: add test for 'dump dir'
	mds: add a command to dump directory information

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

script: add 'reef' release to backport-create-issue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Break up the text of two similar admonitions into three paragraphs (in
each of the two instances). This makes the content of the admonition
much easier to read at a glance.

Signed-off-by: Zac Dover <zac.dover@proton.me>

…w-multisite-admonition-formatting

doc/radosgw: format admonitions

Refine zone-sync.svg (fix kerning and use fonts that render properly).

Signed-off-by: Zac Dover <zac.dover@proton.me>

RGW - Replace set_owner workaround with new fix

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

s3select submodule

integrate the limit-operator into RGW. current changes is for CSV format

integrate the limit-operator with CSV-flow, and JSON-flow. an update of s3select submodule

a fix for the input-serialization-type selection

debug functionality.

fix for error handling

adding the scan-range feature. i.e. enables the user to define the range of processing(text only)

remove the temp variable(the s3select-layer handles the sql-result setting upon JSON flow)

adding documentation. for JSON, SQL limit-operator, scan-range option.
a fix for JSON test framework in s3select module

investigate crash. replace len with it.length()

editorial changes; non-initialized variables

adding validation for offset&length. (it seems that the offset&length causes Invalid reads, in some cases)

changes related to trino integration.(1) syntax issues related to trino-engine statement conbersions(s3select parser handles that) (2).
(2). trino rejects(HIVE_CURSUR_ERROR) some of the s3select responses do to its size, the change is about control the size of the respond.

update JSON documentation

trino changes: response in paging (limitation on size), trino syntax issues.

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

…ite-master-zonegroup-image-refinement

doc/rgw: clean zone-sync.svg

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/cephadm: Redd up compatibility.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

drive_group: fix limit filter in drive_selection.selector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Zack Cerza <zack@redhat.com>

rgw/s3select: json integration: identifying new data-source and execute a JSON flow.

Reviewed-by: Albin Antony <aantony@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Full line-edit of doc/radosgw/keycloak.rst. Corrected syntax, grammar,
RST, and broken links.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/reshard: treat old RGWBucketInfo::num_shards=0 as 1 shard

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

…w-keycloak-rewrite

doc/rgw: refine keycloak.rst

Signed-off-by: Nizamudeen A <nia@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Rewrite the third paragraph of the caption to the figure depicting how a
zonegroup works.

Follows ceph#50316.

Signed-off-by: Zac Dover <zac.dover@proton.me>

See: 69def14

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

the Formatter constructors take a 'bool pretty' argument. the pointers
allocated by 'new' were being converted to bools and leaked by
make_unique

this was already fixed in 5302c18,
but reverted in a rebase by 200f71a

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: d3n: fix test failure when "inline_data": false

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Put an "only" in the correct place. Also, this is a very small change
that will allow me to see if Jenkins is working (it probably isn't).

Signed-off-by: Zac Dover <zac.dover@proton.me>

…w-s3-table

doc/rgw - fix grammar in table in s3.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

* refs/pull/48894/head:
	qa/tasks/cephfs: add "extra data pool" cephfs-data-scan tests
	qa/tasks/cephfs: use cephfs tags when recreating osd caps
	tools/cephfs-data-scan: make data pool command args optional
	tools/cephfs-data-scan: support for multi-datapool
	cls/cephfs: extend accumulate_inode_metadata client method to store pool id

Reviewed-by: Venky Shankar <vshankar@redhat.com>

* refs/pull/50310/head:
	client: remove unused method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

…de (srnode)

Signed-off-by: Venky Shankar <vshankar@redhat.com>

…prealm

Signed-off-by: Venky Shankar <vshankar@redhat.com>

…aprealm

Signed-off-by: Venky Shankar <vshankar@redhat.com>

…ode in snap trace reply

To handle older clients, introduce a cephfs feature bit. The MDS encodes a "new"
SNapRealmInfo structure for clients supporting this feature (which includes the
last modification timestamp and a version/change_attr for the snap realm). For
clients not advertising this feature, the MDS uses the existing SnapRealmInfo
structure for snapshot traces.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Signed-off-by: Venky Shankar <vshankar@redhat.com>

…cation time of a snap realm

For MDS that do not encode the "new" SnapReamInfo strucutre, fallback to
using SnapRealmInfo.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Fixes: http://tracker.ceph.com/issues/54501
Signed-off-by: Venky Shankar <vshankar@redhat.com>

…w-multisite-caption-3

doc/rgw: fix caption

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Fixes: https://tracker.ceph.com/issues/58822

This change disables reporting missing coverage for blocks that
contain copy and pasted code from other python versions and exist
to make those functions available to older python versions.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

The call function provides the ability to run subprocesses, log output,
and provides an optional timeout parameter. This timeout parameter does
not appear to function correctly today, so we make use of
pytest.param/pytest.mark.xfail to mark these cases as already known to
fail.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

build scripts: various cleanups and minor changes preparing for containerized build and test

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Add a full stop to a sentence in basic-workflow.rst. I looked at this
document and noticed that it wasn't there, and it was just bugging me.

Signed-off-by: Zac Dover <zac.dover@proton.me>

…ide-basic-workflow-full-stop

doc/dev: add full stop to sentence in basic-wo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Fixes: https://tracker.ceph.com/issues/58925
Signed-off-by: Laura Flores <lflores@redhat.com>

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

ceph@f967ac0
introduced an import of something in the cephadm module
in the prometheus module. This seems to break the prometheus
module in some non-cephadm setups. For example, the ceph-ansible
ci hit

failed: [mgr0 -> mon0] (item=prometheus) => changed=true
  ansible_loop_var: item
  cmd:
  - ceph
  - -n
  - client.admin
  - -k
  - /etc/ceph/ceph.client.admin.keyring
  - --cluster
  - ceph
  - mgr
  - module
  - enable
  - prometheus
  delta: '0:00:00.389965'
  end: '2023-03-03 15:30:07.631308'
  item: prometheus
  rc: 2
  start: '2023-03-03 15:30:07.241343'
  stderr: 'Error ENOENT: module ''prometheus'' reports that it cannot run on the active manager daemon: No module named ''cephadm'' (pass --force to force enablement)'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

so we need to be a bit more careful with this import and
make sure the prometheus module works fine without cephadm

Signed-off-by: Adam King <adking@redhat.com>

…50374

doc/radosgw: Redd up s3select.rst as followup to ceph#49411

Reviewed-by: Zac Dover <zac.dover@proton.me>

rgw/tests: Reset endpoint before creating additional zones

Reviewed-by: Casey Bodley <cbodley@redhat.com>

script: update backport scripts for reef

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>

…on replica

* assert internal_client_request is on primary since
  do_recover_missing is also called by internal requests.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

In the case of balanced read the op is not misdirected.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

* Move error checking to the beginning of do_process()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

…ed/localized

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Clear invalid obc from cache.

Fixes: https://tracker.ceph.com/issues/58089

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Wait on a PG to advance to the request's map epoch.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

discard op in the case where same_primary_since is later than
the MOSDOp's map epoch

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Before this patch, ObjectContext had a head member which was used
to get the head obc of a clone object.
This member caused the head object to being referenced while
attempting to 'clear_replica_obc' (Since we only evict un-referenced
obc from the obc_registery).
This mechanism, of obtaining the head, is no longer needed since
'with_clone_obc' loads the head object context first (using
oid.get_head).

In this commit, head is removed from ObjectContext class and users
are removed as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

In continuation to 7ca2690:
Now that the head ref is no longer a member of obc, we need a new
substitute way to get the head when needed.

When loading a clone object, the head object is loaded
first (See with_clone_obc). Therefore we can make use of this design
to move the loaded head forward to the relevant func (See with_head_and_clone_obc).
Usually, we wouldn't need to make use of both the head and the clone obc in the
same function. However, SnapTrimObjSubEvent::remove_or_update is an abnormal usage.

Note: We want to avoid holding any unneeded references to obcs
to allow the obc_registery to evict no longer valid obc.
Therefore, with_obc() which references only a single obc is the
preferred entry point for loading obcs.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Fixes: https://tracker.ceph.com/issues/58678
Signed-off-by: Milind Changire <mchangir@redhat.com>

Signed-off-by: Milind Changire <mchangir@redhat.com>

https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#c48-prefer-in-class-initializers-to-member-initializers-in-constructors-for-constant-initializers

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

common/intrusive_lru: Improve readability

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

qa: suppress memory leak in rocksdb

crimson/osd: Support balance reads

Reviewed-by: Samuel Just <sjust@redhat.com>

…-thread

rgw: don't start the bucket notification thread in radosgw-admin

reviewewd-by: cbodley

mgr/cephadm: try to avoid pull when getting container image info

Reviewed-by: Adam King <adking@redhat.com>

rgw: test kafka sasl scram

reviewed-by: anthonyeleven, cbodley

mgr/rgw: adding mgr rgw module to ceph image

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: don't add mgr into iscsi trusted_ip_list if it's already there

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add commands to set services to managed/unmanaged

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

any executable that links against the heap_profiler leaks 8 bytes for a
singleton:
> -----------------------------------------------------
> Suppressions used:
> count      bytes template
> 1          8 ^MallocExtension::Register
> -----------------------------------------------------

the python script bin/ceph has a lot of leaks:
> -----------------------------------------------------
> Suppressions used:
>   count      bytes template
>     205     294983 ^PyObject_Malloc
>      10       6216 ^_PyObject_Realloc
>       1        568 ^PyMem_Calloc
>       2          2 ^PyMem_Malloc
>      63      58935 ^PyUnicode_New
>      10       7294 ^_PyBytes_FromSize
>     166     149904 ^_PyObject_GC_Alloc
>       5       3360 ^_PyObject_GC_Resize
> -----------------------------------------------------

Signed-off-by: Casey Bodley <cbodley@redhat.com>