rpc error: code = Unknown desc = COMMIT: pq: could not serialize access due to read/write dependencies among transactions

[invidian]

Expected Behaviour

Tink server should be capable of handling requests in paralell.

Current Behaviour

Currently, when one applies multiple requests to tink server in parallel, server sometimes returns the following error:

Error: removing workflow "08b64643-010d-4ade-a43f-6ffa7f0afce8": rpc error: code = Unknown desc = COMMIT: pq: could not serialize access due to read/write dependencies among transactions

Steps to Reproduce (for bugs)

Currently, this is sometimes reproducible when using https://github.com/kinvolk/terraform-provider-tinkerbell and when running terraform destroy on large number of Tinkerbell resources.

Context

Your Environment

Tested using Tinkerbell sandbox with tinkerbell server from image quay.io/tinkerbell/tink:sha-adb49da.

[invidian]

BTW, in some cases, like

tink/db/template.go

Lines 17 to 45 in 3427566

	tx, err := d.instance.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
	if err != nil {
	return errors.Wrap(err, "BEGIN transaction")
	}
	_, err = wflow.Parse([]byte(data))
	if err != nil {
	return err
	}
	_, err = tx.Exec(`
	INSERT INTO
	template (created_at, updated_at, name, data, id)
	VALUES
	($1, $1, $2, $3, $4)
	ON CONFLICT (id)
	DO
	UPDATE SET
	(updated_at, deleted_at, name, data) = ($1, NULL, $2, $3);
	`, time.Now(), name, data, id)
	if err != nil {
	return errors.Wrap(err, "INSERT")
	}
	err = tx.Commit()
	if err != nil {
	return errors.Wrap(err, "COMMIT")
	}
	return nil

, transaction is not even needed, as there is only a single query being made, which is already ensured to be atomic by the database itself, according to https://stackoverflow.com/a/1171807/2974814, unless I'm missing something.

[mmlb]

@kdeng3849 looks like what you're looking at right?

[invidian]

@parauliya this is mitigation I talked about: tinkerbell/terraform-provider-tinkerbell@d4e4080.

[kqdeng]

@mmlb yeah, that's exactly it

[thebsdbox]

@kdeng3849 what is the status of this?

[kqdeng]

It has been fixed with this PR: #346

[invidian]

Ah, BTW, this was also occurring for other resources, not only for workflows. I'll test if that's still the case.

[invidian]

This still occurs when workflows are removed:

       Error: removing workflow "b06356a8-3ed4-11eb-a31e-0242ac1b0004": rpc error: code = Unknown desc = Delete Workflow Error: pq: could not serialize access due to read/write dependencies among transactions



        Error: removing workflow "b05ef9be-3ed4-11eb-a31e-0242ac1b0004": rpc error: code = Unknown desc = COMMIT: pq: could not serialize access due to read/write dependencies among transactions



        Error: removing workflow "b046f1ae-3ed4-11eb-a31e-0242ac1b0004": rpc error: code = Unknown desc = COMMIT: pq: could not serialize access due to read/write dependencies among transactions

Running Tinkerbell 266637a. Can we please re-open?

[invidian]

Creating/deleting Hardware entries and Templates seems OK.

[gianarb]

@invidian asked me to re-open, so I think it is not a fixed issue yet

[jacobweinstock]

close by #654