[DocDB] Upgrade failure in master from b250-> b332 due to regression

[kripasreenivasan]

Jira Link: DB-7266

Description

Master universe upgrade failed from b250-> b332
A regression perhaps caused by the commit- d6d0cf6

from the core file

 frame #1: 0x000055a955c97a78 yb-tserver`yb::tablet::TransactionStatusResolver::Impl::StatusReceived(yb::Status, yb::tserver::GetTransactionStatusResponsePB const&, int) [inlined] google::protobuf::RepeatedPtrField<yb::AppStatusPB>::Get(this=0x0000050c7505a4a0, index=0) const at repeated_field.h:1989:32
    frame #2: 0x000055a955c97a78 yb-tserver`yb::tablet::TransactionStatusResolver::Impl::StatusReceived(yb::Status, yb::tserver::GetTransactionStatusResponsePB const&, int) [inlined] yb::tserver::GetTransactionStatusResponsePB::deadlock_reason(this=0x0000050c7505a430, index=0) const at tserver_service.pb.h:7631:27
    frame #3: 0x000055a955c97a78 yb-tserver`yb::tablet::TransactionStatusResolver::Impl::StatusReceived(this=0x0000050c7c8a5300, status=<unavailable>, response=0x0000050c7505a430, request_size=<unavailable>) at transaction_status_resolver.cc:262:20

Code

if (response.deadlock_reason(i).code() != AppStatusPB::OK) {
        // response contains a deadlock specific error.
        status_info.expected_deadlock_status = StatusFromPB(response.deadlock_reason(i));
}

The code does not check the size of response.deadlock_reason and tries to access it by an index unconditionally. However the structure is empty for this case-

(lldb) print response
(const yb::tserver::GetTransactionStatusResponsePB) $1 = {
  _internal_metadata_ = {
    google::protobuf::internal::InternalMetadataWithArenaBase<google::protobuf::UnknownFieldSet, google::protobuf::internal::InternalMetadataWithArena> = (ptr_ = 0x0000000000000000)
  }
  _has_bits_ = {
    has_bits_ = ([0] = 2)
  }
  _cached_size_ = 0
  status_ = {
    current_size_ = 1
    total_size_ = 4
    rep_ = 0x0000050c74514be0
  }
  status_hybrid_time_ = {
    current_size_ = 1
    total_size_ = 4
    rep_ = 0x0000050c7ebfb0c0
  }
  num_replicated_batches_ = {
    current_size_ = 0
    total_size_ = 0
    rep_ = nullptr
  }
  coordinator_safe_time_ = {
    current_size_ = 1
    total_size_ = 4
    rep_ = 0x0000050c7eba9080
  }
  aborted_subtxn_set_ = {
    google::protobuf::internal::RepeatedPtrFieldBase = {
      arena_ = nullptr
      current_size_ = 1
      total_size_ = 4
      rep_ = 0x0000050c7e1ad4c0
    }
  }
  deadlock_reason_ = {
    google::protobuf::internal::RepeatedPtrFieldBase = {
      arena_ = nullptr
      current_size_ = 0
      total_size_ = 0
      rep_ = nullptr
    }
  }
  error_ = nullptr
  propagated_hybrid_time_ = 6920587650673926144
}

response.deadlock_reason can be empty when the response is received from a tablet server, which is not yet updated and that tablet server knows nothing about deadlock_reason.

Thanks, @arybochkin !

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.