Skip to content

BatchJobUtilities

Jump to bottom
Anash P. Oommen edited this page Sep 6, 2018 · 2 revisions

Overview

If you use BatchJobService in the AdWords API, we recommend using the BatchJobUtilities utility in the client library. This utility handles the following interactions with Google Cloud Storage transparently:

  • Creating a resumable upload URL for a batch job.
  • Uploading operations for a given BatchJob, either incrementally or via a single method call.
  • Downloading the results of a BatchJob once it completes, and transforming those results into objects.

To learn more about batch processing, refer to our Batch Processing guide. Below we summarize the guide from the perspective of the .NET client library. For code examples, see AddCompleteCampaignsUsingBatchJob.cs and AddKeywordsUsingIncrementalBatchJob.cs.

Step 1: Create a job

To start using the BatchJobUtilities, you first need to create a BatchJob and retrieve its URL as follows:

// Get the BatchJobService instance.
BatchJobService batchJobService = (BatchJobService) user.GetService(
    AdWordsService.v201603.BatchJobService);

// Create a BatchJob.
BatchJobOperation addOp = new BatchJobOperation() {
  @operator = Operator.ADD,
  operand = new BatchJob()
};

BatchJob batchJob = batchJobService.mutate(
    new BatchJobOperation[] { addOp }).value[0];

// Get the upload URL from the new job.
string uploadUrl = batchJob.uploadUrl.url;

Step 2: Generate your operations

Next, create an array of operations you want to add to the batch. Then create a BatchJobUtilities instance as follows:

AdWordsUser user = new AdWordsUser();
...
BatchJobUtilities batchJobUploadHelper = new BatchJobUtilities(user);

By default, the BatchJobUtilities class uploads data as a single upload request. Under certain circumstances (e.g. there is a fixed time limit for individual requests, or if you need to provide some kind of upload progress indicator), it might make sense to go for a chunked upload approach. In such cases, you can use the overloaded constructor:

bool useChunking = true;
bool chunkSize = 32 * 1024 * 1024; // 32 MB chunks
BatchJobUtilities batchJobUploadHelper = new BatchJobUtilities(
    user, useChunking, chunkSize);

Keep in mind that chunking is not the preferred approach since there are performance costs associated with the additional requests, and it is generally not needed.

Step 3: Prepare the job for upload

When creating a new BatchJob, Google Ads servers create a job with a signed temporary URL. You need to exchange this URL for a resumable upload URL as follows:

String uploadUrl = batchJob.uploadUrl;
string resumableUploadUrl = batchJobUploadHelper.GetResumableUploadUrl(
    uploadUrl);

You need to keep track of the resumable URL, since this URL may be generated only once.

Step 4: Upload the operations

Finally, send all your operations to the server:

batchJobUploadHelper.Upload(resumableUploadUrl, operations.ToArray());

Normally, that’s all that you need to upload your operations. If your network connection breaks during upload, then you can retry the call as follows:

bool resumePreviousUpload = true;
batchJobUploadHelper.Upload(resumableUploadUrl, operations.ToArray(),
     resumePreviousUpload);

In this case, the BatchJobUtilities class attempts to figure out how much data was uploaded before it got interrupted and resumes the rest of the upload. This approach is especially useful if you are transferring a large number of operations and the likelihood of a network interruption or some other transmission failure is high, for example, when uploading from a mobile client app. It can also reduce your bandwidth usage in the event of network failures because you don't have to restart large file uploads from the beginning.

Step 5: Wait while the job is queued and executed

You need to wait while the job is pending. This may be done as follows:

bool isPending = batchJobUploadHelper.WaitForPendingJob(batchJob.id,
    TIME_TO_WAIT_FOR_COMPLETION,
    delegate(BatchJob waitBatchJob, long timeElapsed) {
      Console.WriteLine("[{0} seconds]: Batch job ID {1} has status '{2}'.",
          timeElapsed / 1000, waitBatchJob.id, waitBatchJob.status);

      // Optional: Save the job.
      batchJob = waitBatchJob;

	// Return true if you want to break the wait loop midway.
      return false;
    }
);

Step 6: Download your results

Once the job is no longer pending, you retrieve the job to examine its results.

Selector selector = new Selector() {
  fields = new string[] {
      BatchJob.Fields.Id, BatchJob.Fields.Status,
      BatchJob.Fields.DownloadUrl, BatchJob.Fields.ProcessingErrors,
      BatchJob.Fields.ProgressStats
  },
  predicates = new Predicate[] {
    Predicate.Equals(BatchJob.Fields.Id, batchJobId)
  }
};

BatchJob batchJob = batchJobService.get(selector).entries[0];

Note: These fields are already populated in the temporary object returned by the callback in WaitForPendingJob call.

Step 7: Examine the errors and results

You can examine the processing errors (if any) in the result as follows:

if (batchJob.processingErrors != null) {
  foreach (BatchJobProcessingError processingError in 
      batchJob.processingErrors) {
    Console.WriteLine("  Processing error: {0}, {1}, {2}, {3}, {4}",
        processingError.ApiErrorType, processingError.trigger,
        processingError.errorString, processingError.fieldPath,
        processingError.reason);
  }
}

Similarly, you can download the results as follows:

if (batchJob.downloadUrl != null && batchJob.downloadUrl.url != null) {
  BatchJobMutateResponse mutateResponse = batchJobUploadHelper.Download(
      batchJob.downloadUrl.url);
  Console.WriteLine("Downloaded results from {0}.", batchJob.downloadUrl.url);
  foreach (MutateResult mutateResult in mutateResponse.rval) {
    String outcome = mutateResult.errorList == null ? "SUCCESS" : "FAILURE";
    Console.WriteLine("  Operation [{0}] - {1}", mutateResult.index, outcome);
  }
}

(Optional) Cancel a job

Sometimes, you may want to cancel a job (e.g. the user of your platform accidently deleted 10K keywords in their account and hit the cancel button immediately upon realizing the mistake). In such cases, you can request a job to be cancelled. The server may cancel the job if it's still in a cancelable state, otherwise the call will return an error. The first step to cancel a job is to cancel the wait loop by returning false from the waitCallback method.

// A flag to determine if the job was requested to be cancelled. This
// typically comes from the user.
bool wasCancelRequested = false;
bool isPending = batchJobUploadHelper.WaitForPendingJob(batchJob.id,
     TIME_TO_WAIT_FOR_COMPLETION, 
     delegate(BatchJob waitBatchJob, long timeElapsed) {
       Console.WriteLine("[{0} seconds]: Batch job ID {1} has status '{2}'.",
           timeElapsed / 1000, waitBatchJob.id, waitBatchJob.status);
           batchJob = waitBatchJob;
           return wasCancelRequested;
     }
);

Since cancellation is also asynchronous, you need to wait for the cancellation to complete, as follows:

bool shouldWaitForCancellation = false;
if (isPending && wasCancelRequested) {
  BatchJobError cancellationError = batchJobUploadHelper.TryToCancelJob(
      batchJob.id);

  if (cancellationError == null) {
    Console.WriteLine("Successfully requested job cancellation.");
    shouldWaitForCancellation = true;
  } else {
    Console.WriteLine("Job cancellation failed. Error says: {0}.",
        cancellationError.reason);
  }

  if (shouldWaitForCancellation) {
    isPending = batchJobUploadHelper.WaitForPendingJob(batchJob.id,
      TIME_TO_WAIT_FOR_COMPLETION, 
      delegate(BatchJob waitBatchJob, long timeElapsed) {
        Console.WriteLine("[{0} seconds]: Batch job ID {1} has status '{2}'.",
            timeElapsed / 1000, waitBatchJob.id, waitBatchJob.status);
        batchJob = waitBatchJob;
        return false;
      }
    );
  }
}
Clone this wiki locally