CAS validation improvements (#10966)

* [CAS] Validate upstream CAS storage

When asked to validate the CAS, also validate the upstream CAS if
present. The entire upstream CAS is available as backup storage of the
currently active CAS. If the upstream CAS is corrupted, it can propagate
the error into downstream error, thus when upstream CAS is invalid, the
entire UnifiedCAS should be invalid.

* [CAS] Do not check if action cache value exists in the object store

Fix a false positive/crash when validating action cache from unified
data base by stop checking the cached value exists in the object store.
First of all, there is a bug that the check is always done against
primary object store, where the object can be missing. It is also
possible that the CAS is in a state where the cache keys are fault in,
but not the objects the cache pointed to. There is no good way to
enforce it, thus the check is too strict.

rdar://155231696
(cherry picked from commit 09bbceb)

Author: cachemeifyoucan

llvm/include/llvm/CAS/UnifiedOnDiskCache.h

@@ -155,6 +155,8 @@ class UnifiedOnDiskCache {
 
   Error validateActionCache();
 
+  OnDiskGraphDB *getUpstreamGraphDB() const { return UpstreamGraphDB; }
+
 private:
   UnifiedOnDiskCache();
 

llvm/lib/CAS/OnDiskCAS.cpp

@@ -90,13 +90,19 @@ class OnDiskCAS : public BuiltinCAS {
 
 void OnDiskCAS::print(raw_ostream &OS) const { DB->print(OS); }
 Error OnDiskCAS::validate(bool CheckHash) const {
-  return DB->validate(CheckHash, [](ArrayRef<ArrayRef<uint8_t>> Refs,
-                                    ArrayRef<char> Data,
-                                    SmallVectorImpl<uint8_t> &Result) {
+  auto Hasher = [](ArrayRef<ArrayRef<uint8_t>> Refs, ArrayRef<char> Data,
+                   SmallVectorImpl<uint8_t> &Result) {
     auto Hash = BuiltinObjectHasher<llvm::cas::builtin::HasherT>::hashObject(
         Refs, Data);
     Result.assign(Hash.begin(), Hash.end());
-  });
+  };
+
+  if (auto E = DB->validate(CheckHash, Hasher))
+    return E;
+  if (UniDB && UniDB->getUpstreamGraphDB())
+    return UniDB->getUpstreamGraphDB()->validate(CheckHash, Hasher);
+
+  return Error::success();
 }
 
 CASID OnDiskCAS::getID(ObjectRef Ref) const {

llvm/lib/CAS/UnifiedOnDiskCache.cpp

@@ -170,8 +170,6 @@ Error UnifiedOnDiskCache::validateActionCache() {
     };
     if (ID.getOpaqueData() == 0)
       return formatError("zero is not a valid ref");
-    if (!PrimaryGraphDB->containsObject(ID))
-      return formatError("cas does not contain ref");
     return Error::success();
   };
   if (Error E = PrimaryKVDB->validate(ValidateRef))