Skip to content

Commit

Permalink
Merge pull request #4821 from mnaamani/update-bags-fix-test
Browse files Browse the repository at this point in the history
Update bags fix test
  • Loading branch information
mnaamani authored Jul 31, 2023
2 parents 29676f1 + 0d9c0c9 commit dac64ea
Show file tree
Hide file tree
Showing 8 changed files with 229 additions and 118 deletions.
4 changes: 4 additions & 0 deletions storage-node/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
### 3.7.0

- Updates `leader:update-bag` CLI command to `leader:update-bags` to accept multiple bag ids as input. This allows the command to be used to update storage buckets of multiple bags in a single batched transaction.

### 3.6.0

- Collosus can now store multiple keys in it's keyring.
Expand Down
73 changes: 38 additions & 35 deletions storage-node/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,8 @@ There is also an option to run Colossus as [Docker container](../colossus.Docker
* [`storage-node leader:remove-operator`](#storage-node-leaderremove-operator)
* [`storage-node leader:set-bucket-limits`](#storage-node-leaderset-bucket-limits)
* [`storage-node leader:set-global-uploading-status`](#storage-node-leaderset-global-uploading-status)
* [`storage-node leader:update-bag`](#storage-node-leaderupdate-bag)
* [`storage-node leader:update-bag-limit`](#storage-node-leaderupdate-bag-limit)
* [`storage-node leader:update-bags`](#storage-node-leaderupdate-bags)
* [`storage-node leader:update-blacklist`](#storage-node-leaderupdate-blacklist)
* [`storage-node leader:update-bucket-status`](#storage-node-leaderupdate-bucket-status)
* [`storage-node leader:update-data-fee`](#storage-node-leaderupdate-data-fee)
Expand Down Expand Up @@ -459,22 +459,49 @@ OPTIONS

_See code: [src/commands/leader/set-global-uploading-status.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/set-global-uploading-status.ts)_

## `storage-node leader:update-bag`
## `storage-node leader:update-bag-limit`

Update StorageBucketsPerBagLimit variable in the Joystream node storage.

```
USAGE
$ storage-node leader:update-bag-limit
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Path to key file to add to the keyring.
-l, --limit=limit (required) New StorageBucketsPerBagLimit value
-m, --dev Use development mode
-p, --password=password Password to unlock keyfiles. Multiple passwords can be passed, to try against all files.
If not specified a single password can be set in ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). If not specified a single key can be set in ACCOUNT_URI
environment variable.
Add/remove a storage bucket from a bag (adds by default).
--keyStore=keyStore Path to a folder with multiple key files to load into keystore.
```

_See code: [src/commands/leader/update-bag-limit.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/update-bag-limit.ts)_

## `storage-node leader:update-bags`

Add/remove a storage bucket/s from a bag/s. If multiple bags are provided, then the same input bucket ID/s would be added/removed from all bags.

```
USAGE
$ storage-node leader:update-bag
$ storage-node leader:update-bags
OPTIONS
-a, --add=add
[default: ] Comma separated list of bucket IDs to add to bag
[default: ] Comma separated list of bucket IDs to add to all bag/s
-h, --help
show CLI help
-i, --bagId=bagId
-i, --bagIds=bagIds
(required) Bag ID. Format: {bag_type}:{sub_type}:{id}.
- Bag types: 'static', 'dynamic'
- Sub types: 'static:council', 'static:wg', 'dynamic:member', 'dynamic:channel'
Expand All @@ -498,7 +525,10 @@ OPTIONS
password can be set in ACCOUNT_PWD environment variable.
-r, --remove=remove
[default: ] Comma separated list of bucket IDs to remove from bag
[default: ] Comma separated list of bucket IDs to remove from all bag/s
-s, --updateStrategy=(atomic|force)
[default: atomic] Update strategy to use. Either "atomic" or "force".
-u, --apiUrl=apiUrl
[default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
Expand All @@ -510,34 +540,7 @@ OPTIONS
Path to a folder with multiple key files to load into keystore.
```

_See code: [src/commands/leader/update-bag.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/update-bag.ts)_

## `storage-node leader:update-bag-limit`

Update StorageBucketsPerBagLimit variable in the Joystream node storage.

```
USAGE
$ storage-node leader:update-bag-limit
OPTIONS
-h, --help show CLI help
-k, --keyFile=keyFile Path to key file to add to the keyring.
-l, --limit=limit (required) New StorageBucketsPerBagLimit value
-m, --dev Use development mode
-p, --password=password Password to unlock keyfiles. Multiple passwords can be passed, to try against all files.
If not specified a single password can be set in ACCOUNT_PWD environment variable.
-u, --apiUrl=apiUrl [default: ws://localhost:9944] Runtime API URL. Mandatory in non-dev environment.
-y, --accountUri=accountUri Account URI (optional). If not specified a single key can be set in ACCOUNT_URI
environment variable.
--keyStore=keyStore Path to a folder with multiple key files to load into keystore.
```

_See code: [src/commands/leader/update-bag-limit.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/update-bag-limit.ts)_
_See code: [src/commands/leader/update-bags.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/update-bags.ts)_

## `storage-node leader:update-blacklist`

Expand Down
2 changes: 1 addition & 1 deletion storage-node/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "storage-node",
"description": "Joystream storage subsystem.",
"version": "3.6.0",
"version": "3.7.0",
"author": "Joystream contributors",
"bin": {
"storage-node": "./bin/run"
Expand Down
57 changes: 0 additions & 57 deletions storage-node/src/commands/leader/update-bag.ts

This file was deleted.

117 changes: 117 additions & 0 deletions storage-node/src/commands/leader/update-bags.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
import { flags } from '@oclif/command'
import _ from 'lodash'
import { customFlags } from '../../command-base/CustomFlags'
import ExitCodes from '../../command-base/ExitCodes'
import LeaderCommandBase from '../../command-base/LeaderCommandBase'
import logger from '../../services/logger'
import { updateStorageBucketsForBags } from '../../services/runtime/extrinsics'

/**
* CLI command:
* Updates bags-to-buckets relationships.
*
* @remarks
* Storage working group leader command. Requires storage WG leader priviliges.
* Shell command: "leader:update-bag"
*/
export default class LeaderUpdateBag extends LeaderCommandBase {
static description =
`Add/remove a storage bucket/s from a bag/s. If multiple bags are ` +
`provided, then the same input bucket ID/s would be added/removed from all bags.`

static flags = {
add: customFlags.integerArr({
char: 'a',
description: 'Comma separated list of bucket IDs to add to all bag/s',
default: [],
}),
remove: customFlags.integerArr({
char: 'r',
description: 'Comma separated list of bucket IDs to remove from all bag/s',
default: [],
}),
bagIds: customFlags.bagId({
char: 'i',
required: true,
multiple: true,
}),
updateStrategy: flags.enum<'atomic' | 'force'>({
char: 's',
options: ['atomic', 'force'],
description: 'Update strategy to use. Either "atomic" or "force".',
default: 'atomic',
}),

...LeaderCommandBase.flags,
}

async run(): Promise<void> {
const { flags } = this.parse(LeaderUpdateBag)

logger.info('Updating the bag...')
if (flags.dev) {
await this.ensureDevelopmentChain()
}

const uniqueBagIds = _.uniqBy(flags.bagIds, (b) => b.toString())
const uniqueAddBuckets = _.uniq(flags.add)
const uniqueRemoveBuckets = _.uniq(flags.remove)

if (_.isEmpty(uniqueAddBuckets) && _.isEmpty(uniqueRemoveBuckets)) {
logger.error('No bucket ID provided.')
this.exit(ExitCodes.InvalidParameters)
}

if (_.isEmpty(uniqueBagIds)) {
logger.error('No bag ID provided.')
this.exit(ExitCodes.InvalidParameters)
}

const account = this.getAccount()
const api = await this.getApi()

// Ensure that input bag ids exist
for (const bagId of uniqueBagIds) {
const bag = await api.query.storage.bags(bagId)
if (bag.isEmpty) {
logger.error(`Bag with ID ${bagId} does not exist`)
this.exit(ExitCodes.InvalidParameters)
}
}

// Ensure that input add bucket ids exist
for (const b of uniqueAddBuckets) {
const bucket = await api.query.storage.storageBucketById(b)
if (bucket.isEmpty) {
logger.error(`Add Bucket input with ID ${b} does not exist`)
this.exit(ExitCodes.InvalidParameters)
}
}

// Ensure that input remove bucket ids exist
for (const b of uniqueRemoveBuckets) {
const bucket = await api.query.storage.storageBucketById(b)
if (bucket.isEmpty) {
logger.error(`Remove Bucket input with ID ${b} does not exist`)
this.exit(ExitCodes.InvalidParameters)
}
}

const [success, failedCalls] = await updateStorageBucketsForBags(
api,
uniqueBagIds,
account,
flags.add,
flags.remove,
flags.updateStrategy
)

if (!_.isEmpty(failedCalls)) {
logger.error(`Following extrinsic calls in the batch Tx failed:\n ${JSON.stringify(failedCalls, null, 2)}}`)
} else {
logger.info('All extrinsic calls in the batch Tx succeeded!')
}

this.exitAfterRuntimeCall(success)
}
}
36 changes: 26 additions & 10 deletions storage-node/src/services/runtime/api.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
import { ApiPromise, WsProvider, SubmittableResult } from '@polkadot/api'
import type { Index } from '@polkadot/types/interfaces/runtime'
import { ISubmittableResult, IEvent } from '@polkadot/types/types'
import { TypeRegistry } from '@polkadot/types'
import { CLIError } from '@oclif/errors'
import { ApiPromise, SubmittableResult, WsProvider } from '@polkadot/api'
import { AugmentedEvent, SubmittableExtrinsic } from '@polkadot/api/types'
import { KeyringPair } from '@polkadot/keyring/types'
import { SubmittableExtrinsic, AugmentedEvent } from '@polkadot/api/types'
import { TypeRegistry } from '@polkadot/types'
import type { Index } from '@polkadot/types/interfaces/runtime'
import { DispatchError } from '@polkadot/types/interfaces/system'
import logger from '../../services/logger'
import ExitCodes from '../../command-base/ExitCodes'
import { CLIError } from '@oclif/errors'
import { IEvent, ISubmittableResult } from '@polkadot/types/types'
import { formatBalance } from '@polkadot/util'
import AwaitLock from 'await-lock'
import stringify from 'fast-safe-stringify'
import sleep from 'sleep-promise'
import AwaitLock from 'await-lock'
import ExitCodes from '../../command-base/ExitCodes'
import logger from '../../services/logger'

/**
* Dedicated error for the failed extrinsics.
Expand Down Expand Up @@ -171,7 +171,7 @@ async function lockAndGetNonce(api: ApiPromise, account: KeyringPair): Promise<I
* @param error - DispatchError instance
* @returns error string.
*/
function formatDispatchError(api: ApiPromise, error: DispatchError): string {
export function formatDispatchError(api: ApiPromise, error: DispatchError): string {
// Need to assert that registry is of TypeRegistry type, since Registry intefrace
// seems outdated and doesn't include DispatchErrorModule as possible argument for "findMetaError"
const typeRegistry = api.registry as TypeRegistry
Expand Down Expand Up @@ -230,3 +230,19 @@ export function getEvent<
}
return event as EventType
}

/**
* Helper function for parsing multiple events from the successful extrinsic result.
*
* @param result - extrinsic result
* @param section - pallet name
* @param eventNames - event name/s
* @returns void promise.
*/
export function getEvents<
S extends keyof ApiPromise['events'] & string,
M extends keyof ApiPromise['events'][S] & string,
EventType = ApiPromise['events'][S][M] extends AugmentedEvent<'promise', infer T> ? IEvent<T> : never
>(result: SubmittableResult, section: S, eventNames: M[]): EventType[] {
return result.filterRecords(section, eventNames).map((e) => e.event as unknown as EventType)
}
Loading

0 comments on commit dac64ea

Please sign in to comment.