Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery: timestamp parsing issue when fetching parameterized query jobs and their config #9221

Closed
hsvnsson opened this issue Jan 5, 2024 · 0 comments · Fixed by #9236
Closed
Assignees
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release.

Comments

@hsvnsson
Copy link

hsvnsson commented Jan 5, 2024

Client

BigQuery

Environment

MacOS

Go Environment

$ go version
go version go1.21.1 darwin/arm64

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/hsvnsson/Library/Caches/go-build'
GOENV='/Users/hsvnsson/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/hsvnsson/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/hsvnsson/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/6w/pc370vsn5c7dtm7gsxm1y9440000gn/T/go-build217012697=/tmp/go-build -gno-record-gcc-switches -fno-common'

Code

package main

import (
	"context"
	"errors"
	"flag"
	"log"
	"time"

	"cloud.google.com/go/bigquery"
	"google.golang.org/api/iterator"
	"google.golang.org/api/option"
)

func main() {
	ctx := context.Background()

	serviceAccountPath := flag.String("service-account", "", "Path to service account JSON file")
	project := flag.String("project", "", "Project")
	flag.Parse()

	client, err := bigquery.NewClient(ctx, *project, option.WithCredentialsFile(*serviceAccountPath))
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	defer client.Close()

	it := client.Jobs(ctx)
	it.AllUsers = true
	it.State = bigquery.Done
	it.MinCreationTime = time.Now().Add(-1 * time.Hour)
	it.MaxCreationTime = time.Now()

	for {
		job, err := it.Next()
		if errors.Is(err, iterator.Done) {
			break
		}
		if err != nil {
			log.Fatalf("Failed to get next job: %v", err)
		}

		jobConfig, err := job.Config()
		if err != nil {
			log.Fatalf("Failed to parse job: %v", err)
		}

		_ = jobConfig
	}
}

Expected behavior
Following the example in the docs to submit a BigQuery query with a timestamp parameter and then listing jobs using the golang sdk it should be possible to get the job configs without any errors.

Setup:

bq query \
   --use_legacy_sql=false \
   --parameter='ts_value:TIMESTAMP:2016-12-07 08:00:00' \
   'SELECT
     TIMESTAMP_ADD(@ts_value, INTERVAL 1 HOUR);'

Then using the code provided above, the call to job.Config() should return a properly parsed job config.

Actual behavior

Error while parsing timestamp:

$ go run main.go -project some-project -service-account ./creds.json
2024/01/04 17:47:40 Failed to parse job: parsing time "2016-12-07T08:00:00" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "Z07:00"
exit status 1

Additional context

if a timezone is included in the timestamp like --parameter='ts_value:TIMESTAMP:2016-12-07 08:00:00+00:00' then it works but since that's not required and also not included in the example then it would be good if the library handles that format as well.

@hsvnsson hsvnsson added the triage me I really want to be triaged. label Jan 5, 2024
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label Jan 5, 2024
@alvarowolfx alvarowolfx added priority: p2 Moderately-important priority. Fix may not be included in next release. and removed triage me I really want to be triaged. labels Jan 8, 2024
gcf-merge-on-green bot pushed a commit that referenced this issue Jan 24, 2024
BigQuery accepts timestamp parameters in many different formats, so when parsing the query parameters, we need to be able to support all of them. As reported on #9221, if a query is created using bq-cli (or any other system) with the `time.DateTime(2006-01-02 15:04:05)` format, our SDK was failing to parse the Job Config when retrieving it from the backend system. 

Resolves #9221
gcf-merge-on-green bot pushed a commit that referenced this issue Feb 6, 2024
🤖 I have created a release *beep* *boop*
---


## [1.59.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.58.0...bigquery/v1.59.0) (2024-02-06)


### Features

* **bigquery:** Add ExportDataStatstics to QueryStatistics ([#9371](https://togithub.com/googleapis/google-cloud-go/issues/9371)) ([261c8d9](https://togithub.com/googleapis/google-cloud-go/commit/261c8d944b53ac8953ea7d771c4bb50e4078d508))
* **bigquery:** Switch all timestamp representations to int64 usec ([#9368](https://togithub.com/googleapis/google-cloud-go/issues/9368)) ([8c1fb7d](https://togithub.com/googleapis/google-cloud-go/commit/8c1fb7d4728ebc3b21cb0d601952966dca9cd1e8))


### Bug Fixes

* **bigquery/storage/managedwriter:** Resolve data races ([#9360](https://togithub.com/googleapis/google-cloud-go/issues/9360)) ([fa31ec0](https://togithub.com/googleapis/google-cloud-go/commit/fa31ec0c0f04302a9713a9b1d3228bda2ba135c6))
* **bigquery:** Enable universe domain resolution options ([fd1d569](https://togithub.com/googleapis/google-cloud-go/commit/fd1d56930fa8a747be35a224611f4797b8aeb698))
* **bigquery:** Support more timestamp formats for query param ([#9236](https://togithub.com/googleapis/google-cloud-go/issues/9236)) ([cc98509](https://togithub.com/googleapis/google-cloud-go/commit/cc98509fc7961e3d3619b837d13e69f9621386e8)), refs [#9221](https://togithub.com/googleapis/google-cloud-go/issues/9221)

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants