CKQueryOperation not returning records reliably

[rhunt222]

I have been able to successfully make multiple CKQueryOperations and CKQuery but have one that seems to not reliably return results but instead returns a CKErrorDomain Code = 0 error. Here is a test function I'm trying to execute:

`func test() {
let query = CKQuery(recordType: "WatchlistBeer", predicate: NSPredicate(value: true))
let operation = CKQueryOperation(query: query)

operation.recordFetchedBlock = { record in
    print("got record: \(record.recordID.recordName)")
}

operation.queryCompletionBlock = { cursor, error in
    if cursor != nil {
        print("we have a cursor: \(String(describing: cursor))")
    }
    if error != nil {
        print("we have an error: \(String(describing: error?.code)), \(String(describing: error?.localizedFailureReason)), \(String(describing: error?.localizedRecoverySuggestion)), |\(error.debugDescription)")
    }
}
database.add(operation)

}`

About 1 in 6 or 7 tries returns results, the rest of the time I get an error with code "0", failureReason = nil, localizedRecoverySuggestion = nil and debug description = Error Domain=CKErrorDomain Code=0 "(null)"

When I look at the live log on CloudKit dashboard the requests are showing successful with a response size that matches the size of the records when I get them back. I'm not sure where this is breaking as it seems as though CloudKit is getting the responses correctly and returning data. The response isn't huge, about 52kb and the error come back appropriately when I input the wrong record type or predicate. Not sure what I'm missing.

[malhal]

When the number of record is too large something different happens which could be the reason. It can either fail that the request was too large or it allows it but needs to batch the records by returning a cursor which is to be used in additional query operations. It sounds like you didn't get any records at all so it's more likely a flat out failure than a bug in the handling of the cursor situation. Since the error object is empty, which means that even no error information was returned, you could examine the lower level http error status code in CKWebRequest line 108 and look it up in this table:

https://developer.apple.com/library/content/documentation/DataManagement/Conceptual/CloudKitWebServicesReference/ErrorCodes.html#//apple_ref/doc/uid/TP40015240-CH4-SW1

[rhunt222]

Thanks @malhal , it seems as though the response is coming back ok (since the CloudKit logs show the responses coming in and the database providing data back) but I was able to uncover a couple of parsing Errors in CKQueryOperation file, performCKOperation(). The result from that is giving me two errors:

error(OpenCloudKit.CKError.parse(Error Domain=NSCocoaErrorDomain Code=3840 "Unterminated string around character 35373." UserInfo={NSDebugDescription=Unterminated string around character 35373.}))

and

error(OpenCloudKit.CKError.parse(Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set.}))

Looks like the JSON that comes back isn't in the right format?

[malhal]

Turn on CloudKit.verbose which will output the JSON and put it in a text editor and go to position 35373 and see what it looks like.

[rhunt222]

When I enable CloudKit.verbose it only prints the JSON response from CloudKit when it is successful. I was able to get one to go through successfully and that character (35406) everything looks alright (as it should since it parsed it ok). Any tip on getting the JSON printout in the event of an error? I'm wondering if the "JSON text did not start with array or object and option to allow fragments not set" is indicating that a cursor came back before the initial JSON array and the parser is not knowing how to handle it since the response did not start with array?

[malhal]

Ah yeh sorry, at line 200 in CKURLRequest.swift try something like:

let dataString = NSString(data: data, encoding: .utf8)
CloudKit.debugPrint(dataString as Any)

[rhunt222]

hmm, tried this and it still won't print anything:

let dataString = NSString(data: data, encoding: String.Encoding.utf8.rawValue)
CloudKit.debugPrint(dataString as Any)

[rhunt222]

Also tried changing the jsonObject at like 201 to:
let jsonObject = try JSONSerialization.jsonObject(with: data, options: [.allowFragments]) as! [String: Any]

but still have an error on the results. Prints twice:
Unterminated string around character 35406.
and
Invalid value around character 0

[rhunt222]

Also tried this before Serialization:
print("data received is \(data.count) bytes:\n\(data)")

And got this in the console:

data received is 35419 bytes: 35419 bytes caught error?: The data couldn’t be read because it isn’t in the correct format. result: error(OpenCloudKit.CKError.parse(Error Domain=NSCocoaErrorDomain Code=3840 "Unterminated string around character 35406." UserInfo={NSDebugDescription=Unterminated string around character 35406.})) case error parsing data received is 21855 bytes: 21855 bytes caught error?: The data couldn’t be read because it isn’t in the correct format. result: error(OpenCloudKit.CKError.parse(Error Domain=NSCocoaErrorDomain Code=3840 "Invalid value around character 0." UserInfo={NSDebugDescription=Invalid value around character 0.})) case error we have an error: Optional(0), nil, nil, |Optional(Error Domain=CKErrorDomain Code=0 "(null)"), 2018-01-07 03:29:52 +0000

This was only for one request but gave two prints of data received, both in the wrong format.

[rhunt222]

@malhal I think I got it.
After doing some research it looks like the didReceive data: function for URLSessionDataDelegate (in CKURLRequest) gets called when receiving anytime any data is received and not necessarily completed data. I suspected this was the case as I was getting two printouts from this func since it was being called twice and adding up the bytes received on each pass equaled the total size that CloudKit dashboard was showing for the response. What I did was implement another URLSessionDataDelegate function that gets called after all of the data is received and put the parsing functionality in there:

`func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {

    if let operationMetrics = metrics {
        metrics?.bytesDownloaded = UInt(data.count)
        metricsDelegate?.requestDidFinish(withMetrics: operationMetrics)
    }
    
    print("data received is \(data.count) bytes:\n\(data)")
}

func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, willCacheResponse proposedResponse: CachedURLResponse, completionHandler: @escaping (CachedURLResponse?) -> Void) {
    
    let data = proposedResponse.data
    print("finished data total: \(data.count)")
    
    // Parse JSON
    do {
        let jsonObject = try JSONSerialization.jsonObject(with: data, options: []) as! [String: Any]
        CloudKit.debugPrint(jsonObject)
        
        // Call completion block
        if let _ = CKErrorDictionary(dictionary: jsonObject) {
            completionBlock?(.error(CKError.server(jsonObject)))
        } else {
            let result = CKURLRequestResult.success(jsonObject)
            completionBlock?(result)
        }
        
    } catch let error as NSError {
        print("error on parse: \(error.localizedDescription)")
        completionBlock?(.error(.parse(error)))
    }
    
}`

I'll keep testing this but it is finally working with every request.

[malhal]

Very sad, didReceive data appending to a mutable receivedData and using it in the completion is an absolutely basic pattern. Well done spotting it but I guess you'll need to proceed with caution using this project.

[BennyKJohnson]

I believe this issue occurs as URL Session Data Task is limited to how much data can be downloaded. This is a bug and I believe implementing URL Session Download Task in its place will fix this issue.

[malhal]

Data task is correct it just needs the append to mutable data as mentioned. Download task is for retrieving a static file, I.e. something that may need to be resumed, and on iPhone something that can be set to continue in the background.