Protocol 13 support

[abuiles]

Stellar Protocol 13 is coming soon, and we need your help bringing support for it to your SDK.

The most relevant changes for SDKs are:

• CAP-0015 - Fee-Bump Transactions. This is the big one. The new operation requires a fundamental change to txenvelopes and means there are now two kinds of transactions, not one.
• CAP-0027 - First-class multiplexed accounts. Although full support for such accounts is not needed, the XDR changes are significant.
• CAP-0018 - Fine-Grained Control of Authorization. A new trustline flag and corresponding change to the Allow Trust operation.

This issue includes all the necessary changes to support Protocol 13, and gives you a path to release a new version of the SDK which is backwards compatible with the current version of the protocol (12). We suggest you aim to release such a backwards compatible version to allow seamless operation through the protocol vote.

For reference, SDF expects to update the testnet to Protocol 13 on 7 May 2020, and pubnet approximately a month after that.

1 - Update XDR

The first step to support Protocol 13 is to recreate your XDR definitions. You can use xdrgen to do that.

You can find the latest XDR definitions in the stellar-core repo.

2 - Update code which depends on TransactionEnvelope

The low-level representation for TransactionEnvelope changed from an xdr.Struct to an xdr.Union. That means you'll need to get the discriminant before reading its value.

/* A TransactionEnvelope wraps a transaction with signatures. */
union TransactionEnvelope switch (EnvelopeType type)
{
case ENVELOPE_TYPE_TX_V0:
    TransactionV0Envelope v0;
case ENVELOPE_TYPE_TX:
    TransactionV1Envelope v1;
case ENVELOPE_TYPE_TX_FEE_BUMP:
    FeeBumpTransactionEnvelope feeBump;
};

As listed above, a transaction envelope can contain a TransactionV0Envelope, a TransactionV1Envelope, or a FeeBumpTransactionEnvelope.

TransactionV0Envelope and TransactionV1Envelope are very similar, but differ in one attribute: the source account.

Next, let's explore how to handle each discriminant. We'll use js-stellar-base as a reference; however, implementation might change depending on your programming language.

TransactionV0Envelope and TransactionV1Envelope

Both v0 and v1 can be handled in the same wrapper class; we extended the Transaction class in js-stellar-base to do so. https://github.com/stellar/js-stellar-base/blob/a15fc74c1f5d37e1a2a94bd610990ccea62292bb/src/transaction.js

In the constructor, we check for the discriminant and throw an error if the envelope doesn't contain a v0 or v1 transaction.

const envelopeType = envelope.switch();
if (
  !(
    envelopeType === xdr.EnvelopeType.envelopeTypeTxV0() ||
    envelopeType === xdr.EnvelopeType.envelopeTypeTx()
  )
) {
  throw new Error(
    `Invalid TransactionEnvelope: expected an envelopeTypeTxV0 or envelopeTypeTx but received an ${envelopeType.name}.`
  );
}

Something important to keep in mind here is that the source account is called sourceAccount of type MuxedAccount in V1, but it is called sourceAccountEd25519 of type Uint256 in V0.

The following code handles both scenarios to pull out their string key representation:

switch (this._envelopeType) {
   case xdr.EnvelopeType.envelopeTypeTxV0():
     this._source = StrKey.encodeEd25519PublicKey(
       this.tx.sourceAccountEd25519()
     );
     break;
   default:
     this._source = StrKey.encodeMuxedAccount(
       this.tx.sourceAccount().toXDR()
     );
     break;
}

Source

Another important thing to keep in mind is that you need to do some special handling when getting the signature base for a V0 transaction.

signatureBase() {
  let tx = this.tx;
  // Backwards Compatibility: Use ENVELOPE_TYPE_TX to sign ENVELOPE_TYPE_TX_V0
  // we need a Transaction to generate the signature base
  if (this._envelopeType === xdr.EnvelopeType.envelopeTypeTxV0()) {
    tx = xdr.Transaction.fromXDR(
      Buffer.concat([
        // TransactionV0 is a transaction with the AccountID discriminant
        // stripped off, we need to put it back to build a valid transaction
        // which we can use to build a TransactionSignaturePayloadTaggedTransaction
        xdr.PublicKeyType.publicKeyTypeEd25519().toXDR(),
        tx.toXDR()
      ])
    );
  }
  const taggedTransaction = new xdr.TransactionSignaturePayloadTaggedTransaction.envelopeTypeTx(
    tx
  );
  const txSignature = new xdr.TransactionSignaturePayload({
    networkId: xdr.Hash.fromXDR(hash(this.networkPassphrase)),
    taggedTransaction
  });
  return txSignature.toXDR();
}

Source

And finally, you also need to add special handling when converting the transaction back to an xdr.TransactionEnvelope (see the implementation of toEnvelope).

For your SDK to be compatible with Protocol 12 and Protocol 13, you need to create a TransactionEnvelope with TransactionV0. If you try to submit a v1 transaction or a fee bump transaction to an instance of Stellar running Protocol 12, then it will fail.

In js-stellar-base we have a builder class which generates v0 transactions by default. You can see the implementation here. You'll also notice that we added a feature flag which allows generation of v1 transactions.

Ideally, once Protocol 13 is released, you should update your SDK to generate V1 transactions by default.

FeeBumpTransactionEnvelope

To handle the scenario when the envelope contains a FeeBumpTransactionEnvelope, let's use js-stellar-base as a reference.

You can see that we added a new class called FeeBumpTransaction, which wraps a xdr.TransactionEnvelope with a Feebump transaction https://github.com/stellar/js-stellar-base/blob/a15fc74c1f5d37e1a2a94bd610990ccea62292bb/src/fee_bump_transaction.js

In the constructor for the class above, you can see the following code:

 const envelopeType = envelope.switch();
 if (envelopeType !== xdr.EnvelopeType.envelopeTypeTxFeeBump()) {
   throw new Error(
     `Invalid TransactionEnvelope: expected an envelopeTypeTxFeeBump but received an ${envelopeType.name}.`
   );
 }

It restricts the kind of transaction envelope which can be received by this class.

You can also see that it has a getter for the innerTransaction, which returns a Transaction class that allows consumers to query data about the transaction being wrapped by the fee bump transaction. This allows you to answer questions like:

• What was the initial fee that was going to be paid for this transaction? ftx.innerTransaction.fee
• What is the hash for the innerTransaction? ftx.innerTransaction.hash
• Who are the signers for the innerTransaction? ftx.innerTransaction.signatures

Besides that, it exposes other fee bump-specific attributes like feeSource, hash, signatureBase, etc.

We also extended the builder to make it easy for users to create fee bump transactions. The helper is called buildFeeBumpTransaction.

In addition, we added a fromXDR helper function to the builder to make it easier to read a Transaction or FeeBumpTransaction. See the implementation here.

3 - Handle muxed accounts

There are two major changes around muxed accounts. The first is updating fields which expect a MuxedAccount; the second is extending Strkey to generated M... accounts.

First, you need to update the following fields — which were previously an xdr.AccountID and are now an xdr.MuxedAccount:

• PaymentOp.destination
• PathPaymentStrictReceiveOp.destination
• PathPaymentStrictSendOp.destination
• Operation.sourceAccount
• Operation.destination (for ACCOUNT_MERGE)
• Transaction.sourceAccount
• FeeBumpTransaction.feeSource

The following pull requests show you the changes in the JS library:

• #325
• #337

Second, you need to implement SEP0023, which allows you to turn a MuxedAccount into a string. A reference implementation can be found in pull request #330. See the update below about M address support.

4 - Add support for Fine-Grained Control of Authorization

CAP 18 changed the data type for the attribute authorize in AllowTrustOp. It is now a uint32 instead of a boolean. It also includes a new TrustLineFlag which is authorizedToMaintainLiabilitiesFlag with value 2.

You can see the implementation in js-stellar-base here, which still accepts booleans but converts the value to an uint32.

5 - Update Horizon response for Transaction, Balance, Operation, and Effect.

In the Horizon 1.1.0 release we added new attributes to different resources:

• The following attributes are now included in the transaction resource:
  • fee_account (the account which paid the transaction fee)
  • fee_bump_transaction (only present in Protocol 13 fee bump transactions)
  • inner_transaction (only present in Protocol 13 fee bump transactions) (#2406).
• Add support for CAP0018: Fine-Grained Control of Authorization (Protocol 13) (#2423).
  • Add is_authorized_to_maintain_liabilities to Balance.

    "balances": [
      {
        "is_authorized": true,
        "is_authorized_to_maintain_liabilities": true,
        "balance": "27.1374422",
        "limit": "922337203685.4775807",
        "buying_liabilities": "0.0000000",
        "selling_liabilities": "0.0000000",
        "last_modified_ledger": 28893780,
        "asset_type": "credit_alphanum4",
        "asset_code": "USD",
        "asset_issuer": "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK"
      },
      {
        "balance": "1.5000000",
        "buying_liabilities": "0.0000000",
        "selling_liabilities": "0.0000000",
        "asset_type": "native"
      }
    ]
    

  • Add authorize_to_maintain_liabilities to AllowTrust operation.

    {
      "id": "124042211741474817",
      "paging_token": "124042211741474817",
      "transaction_successful": true,
      "source_account": "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK",
      "type": "allow_trust",
      "type_i": 7,
      "created_at": "2020-03-27T03:40:10Z",
      "transaction_hash": "a77d4ee5346d55fb8026cdcdad6e4b5e0c440c96b4627e3727f4ccfa6d199e94",
      "asset_type": "credit_alphanum4",
      "asset_code": "USD",
      "asset_issuer": "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK",
      "trustee": "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK",
      "trustor": "GA332TXN6BX2DYKGYB7FW5BWV2JLQKERNX4T7EUJT4MHWOW2TSGC2SPM",
      "authorize": true,
      "authorize_to_maintain_liabilities": true,
    }
    

  • Add effect trustline_authorized_to_maintain_liabilities.

    {
      "id": "0124042211741474817-0000000001",
      "paging_token": "124042211741474817-1",
      "account": "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK",
      "type": "trustline_authorized_to_maintain_liabilities",
      "type_i": 25,
      "created_at": "2020-03-27T03:40:10Z",
      "trustor": "GA332TXN6BX2DYKGYB7FW5BWV2JLQKERNX4T7EUJT4MHWOW2TSGC2SPM",
      "asset_type": "credit_alphanum4",
      "asset_code": "USD"
    }    
    

In the upcoming Horizon 1.2.0 release a base 64 encoding of the bytes in a transaction memo will be included in the Horizon transaction response as memo_bytes

Horizon 1.2.0 also schedule a couple of breaking changes which are required to support CAP-15.

• The type for the following attributes will be changed from int64 to string in 1.3.0:
  • fee_charged in Transaction resource.
  • max_fee in Transaction resource.

You SDK should be able to parse either a string or a number.

6 - Testing

Once you have your implementation ready, you can test it out by submitting a transaction to Horizon 1.1 or later. Remember that for now, your library should produce V0 transactions.

The Stellar Laboratory now has support for protocol 13, so you can also create V1 and fee bump transactions and test them out in the XDR viewer there.

References stellar/stellar-protocol#600.

[abuiles]

This has been implemented in stellar-sdk@5.0.0

[abuiles]

Hi! I forgot a couple of things.

If you support SEP0029 memo required check, then it needs to be updated to support fee bump transactions. You can check pull request #535 as a reference.

Also a couple of people have reached out asking for sample XDRs for V1 and FeeBump transactions, you can use the following XDRs:

• Fee Bump: https://laboratory.stellar.org/#xdr-viewer?input=AAAABQAAAADgSJG2GOUMy%2FH9lHyjYZOwyuyytH8y0wWaoc596L%2BbEgAAAAAAAADIAAAAAgAAAABzdv3ojkzWHMD7KUoXhrPx0GH18vHKV0ZfqpMiEblG1gAAAGQAAAAAAAAACAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAA9IYXBweSBiaXJ0aGRheSEAAAAAAQAAAAAAAAABAAAAAOBIkbYY5QzL8f2UfKNhk7DK7LK0fzLTBZqhzn3ov5sSAAAAAAAAAASoF8gAAAAAAAAAAAERuUbWAAAAQK933Dnt1pxXlsf1B5CYn81PLxeYsx%2BMiV9EGbMdUfEcdDWUySyIkdzJefjpR5ejdXVp%2FKXosGmNUQ%2BDrIBlzg0AAAAAAAAAAei%2FmxIAAABAijIIQpL6KlFefiL4FP8UWQktWEz4wFgGNSaXe7mZdVMuiREntehi1b7MRqZ1h%2BW%2BY0y%2BZ2HtMunsilT2yS5mAA%3D%3D&type=TransactionEnvelope&network=test
• V1 Transaction: https://laboratory.stellar.org/#xdr-viewer?input=AAAAAgAAAABzdv3ojkzWHMD7KUoXhrPx0GH18vHKV0ZfqpMiEblG1gAAAMgAAAAAAAAACAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAADgSJG2GOUMy%2FH9lHyjYZOwyuyytH8y0wWaoc596L%2BbEgAAAAAAAAAEqBfIAAAAAAAAAAAA&type=TransactionEnvelope&network=test

Thanks!

[abuiles]

As @overcat and @tamirms pointed out here, we shoud skip the SEP0029 check if the destination is an M.. address since M.. addresses encode a memo within them.

[abuiles]

Action needed: Revert support of M-strkeys from all SDKs

TL;DR: we need to remove support for M-strkeys (SEP23) from all the SDKs. If the SDK you are maintaining still doesn't have support for SEP23, you can ignore this message.

We have already reverted support for SEP23 in the Go SDK and the Javascript SDK. You can use those PRs as reference to do the same in your SDK.

Context

SEP23 is still a draft and may not be promoted to final. Adding support for it means that users may end up storing M-strkeys,
which can create a lot of problems if SEP23 ends up not being implemented. We are sorry we didn't identify this earlier.

We implemented support for SEP23 prematurely due as part of the Protocol 13 effort.
As part of the Protocol 13 effort we implemented initial support for CAP27.

Again, we are truly sorry we asked the community to implement support for it and later remove it. We apologize for the inconvenience.

What need to be done

The SDKs shouldn't generate or accept M-strkeys (e.g. "MCAAAAAAAAAAAAB7BQ2L7E5NBWMXDUCMZSIPOBKRDSBYVLMXGSSKF6YNPIB7Y77ITKNOG").

We encourage removing any code generating or parsing M-keys to avoid confusions (the code can later be recovered for VCS if need be).
Once that is done, the SDK may still need to deal with translations between strkey and MuxedAccount XDR representations without using M-strkeys.
We will drop the memo id of MuxedAccounts and use G-strkeys:

1. Whenever the XDR representation of a MuxedAccount needs to be transformed to a strkey, it will first be transformed to an AccountID (dropping the memo id if needed) and then it will transformated normally into a G-strkey.
2. Whenever a strkey needs to be transformed to the XDR representation of a MuxedAccount only G-strkeys will be accepted, the

cc @2opremio