Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Query Node] Adds StorageBag.objectsSize field #4818

Merged
merged 2 commits into from
Sep 23, 2023
Merged

[Query Node] Adds StorageBag.objectsSize field #4818

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9c565f
adds storageBag.objectsSize field to QN
zeeshanakram3 Jul 27, 2023
0f299f2
Merge branch 'master' into qn_add_storage_bag_size_field
mnaamani Sep 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions distributor-node/src/services/networking/query-node/schema.graphql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,6 +1047,9 @@ type StorageBag implements BaseGraphQLObject {
deletedById: ID
version: Int!
objects: [StorageDataObject!]!

"""Total size of data objects in Bag"""
objectsSize: BigInt!
storageBuckets: [StorageBucket!]!
distributionBuckets: [DistributionBucket!]!

Expand Down Expand Up @@ -12474,6 +12477,12 @@ input StorageBagWhereInput {
deletedAt_gte: DateTime
deletedById_eq: ID
deletedById_in: [ID!]
objectsSize_eq: BigInt
objectsSize_gt: BigInt
objectsSize_gte: BigInt
objectsSize_lt: BigInt
objectsSize_lte: BigInt
objectsSize_in: [BigInt!]
owner_json: JSONObject
objects_none: StorageDataObjectWhereInput
objects_some: StorageDataObjectWhereInput
Expand All @@ -12494,10 +12503,12 @@ input StorageBagWhereUniqueInput {
}

input StorageBagCreateInput {
objectsSize: String!
owner: JSONObject!
}

input StorageBagUpdateInput {
objectsSize: String
owner: JSONObject
}

Expand Down Expand Up @@ -31855,6 +31866,8 @@ enum StorageBagOrderByInput {
updatedAt_DESC
deletedAt_ASC
deletedAt_DESC
objectsSize_ASC
objectsSize_DESC
}

enum StorageBucketOperatorMetadataOrderByInput {
Expand Down
4 changes: 4 additions & 0 deletions query-node/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
### 1.3.0

- Adds `StorageBag.objectsSize` field to `StorageBag` entity schema. This enables to query the total size of all objects in a storage bag.

### 1.2.2

- Integrates OpenTelemetry API/SDK with Query Node's Graphql Server for exporting improved tracing logs & metrics to Elasticsearch. Adds `./start-elasticsearch-stack.sh` script to bootstrap elasticsearch services (Elasticsearch + Kibana + APM Server) with all the required configurations.
Expand Down
2 changes: 1 addition & 1 deletion query-node/mappings/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "query-node-mappings",
"version": "1.2.1",
"version": "1.3.0",
"description": "Mappings for hydra-processor",
"main": "lib/src/index.js",
"license": "MIT",
Expand Down
81 changes: 45 additions & 36 deletions query-node/mappings/src/storage/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,77 +2,77 @@
eslint-disable @typescript-eslint/naming-convention
*/
import { DatabaseManager, EventContext, StoreContext } from '@joystream/hydra-common'
import BN from 'bn.js'
import {
DistributionBucket,
DistributionBucketFamily,
DistributionBucketOperator,
DistributionBucketOperatorMetadata,
DistributionBucketOperatorStatus,
GeoCoordinates,
NodeLocationMetadata,
StorageBag,
StorageBucket,
StorageBucketOperatorStatusActive,
StorageBucketOperatorStatusInvited,
StorageBucketOperatorStatusMissing,
StorageDataObject,
} from 'query-node/dist/model'
import {
Storage_DataObjectsDeletedEvent_V1001 as DataObjectsDeletedEvent_V1001,
Storage_DataObjectsMovedEvent_V1001 as DataObjectsMovedEvent_V1001,
Storage_DataObjectsUpdatedEvent_V1001 as DataObjectsUpdatedEvent_V1001,
Storage_DataObjectsUploadedEvent_V1001 as DataObjectsUploadedEvent_V1001,
Storage_DistributionBucketCreatedEvent_V1001 as DistributionBucketCreatedEvent_V1001,
Storage_DistributionBucketDeletedEvent_V1001 as DistributionBucketDeletedEvent_V1001,
Storage_DistributionBucketsUpdatedForBagEvent_V1001 as DistributionBucketsUpdatedForBagEvent_V1001,
Storage_DistributionBucketFamilyCreatedEvent_V1001 as DistributionBucketFamilyCreatedEvent_V1001,
Storage_DistributionBucketFamilyDeletedEvent_V1001 as DistributionBucketFamilyDeletedEvent_V1001,
Storage_DistributionBucketFamilyMetadataSetEvent_V1001 as DistributionBucketFamilyMetadataSetEvent_V1001,
Storage_DistributionBucketMetadataSetEvent_V1001 as DistributionBucketMetadataSetEvent_V1001,
Storage_DistributionBucketInvitationAcceptedEvent_V1001 as DistributionBucketInvitationAcceptedEvent_V1001,
Storage_DistributionBucketInvitationCancelledEvent_V1001 as DistributionBucketInvitationCancelledEvent_V1001,
Storage_StorageOperatorMetadataSetEvent_V1001 as StorageOperatorMetadataSetEvent_V1001,
Storage_StorageBucketCreatedEvent_V1001 as StorageBucketCreatedEvent_V1001,
Storage_StorageBucketDeletedEvent_V1001 as StorageBucketDeletedEvent_V1001,
Storage_StorageBucketInvitationAcceptedEvent_V1001 as StorageBucketInvitationAcceptedEvent_V1001,
Storage_StorageBucketInvitationCancelledEvent_V1001 as StorageBucketInvitationCancelledEvent_V1001,
Storage_DistributionBucketMetadataSetEvent_V1001 as DistributionBucketMetadataSetEvent_V1001,
Storage_DistributionBucketModeUpdatedEvent_V1001 as DistributionBucketModeUpdatedEvent_V1001,
Storage_DynamicBagCreatedEvent_V1001 as DynamicBagCreatedEvent_V1001,
Storage_DynamicBagDeletedEvent_V1001 as DynamicBagDeletedEvent_V1001,
Storage_VoucherChangedEvent_V1001 as VoucherChangedEvent_V1001,
Storage_DistributionBucketOperatorInvitedEvent_V1001 as DistributionBucketOperatorInvitedEvent_V1001,
Storage_DistributionBucketOperatorRemovedEvent_V1001 as DistributionBucketOperatorRemovedEvent_V1001,
Storage_DistributionBucketStatusUpdatedEvent_V1001 as DistributionBucketStatusUpdatedEvent_V1001,
Storage_DistributionBucketsUpdatedForBagEvent_V1001 as DistributionBucketsUpdatedForBagEvent_V1001,
Storage_DynamicBagCreatedEvent_V1001 as DynamicBagCreatedEvent_V1001,
Storage_DynamicBagDeletedEvent_V1001 as DynamicBagDeletedEvent_V1001,
Storage_PendingDataObjectsAcceptedEvent_V1001 as PendingDataObjectsAcceptedEvent_V1001,
Storage_StorageBucketCreatedEvent_V1001 as StorageBucketCreatedEvent_V1001,
Storage_StorageBucketDeletedEvent_V1001 as StorageBucketDeletedEvent_V1001,
Storage_StorageBucketInvitationAcceptedEvent_V1001 as StorageBucketInvitationAcceptedEvent_V1001,
Storage_StorageBucketInvitationCancelledEvent_V1001 as StorageBucketInvitationCancelledEvent_V1001,
Storage_StorageBucketOperatorInvitedEvent_V1001 as StorageBucketOperatorInvitedEvent_V1001,
Storage_StorageBucketStatusUpdatedEvent_V1001 as StorageBucketStatusUpdatedEvent_V1001,
Storage_StorageBucketVoucherLimitsSetEvent_V1001 as StorageBucketVoucherLimitsSetEvent_V1001,
Storage_StorageBucketsUpdatedForBagEvent_V1001 as StorageBucketsUpdatedForBagEvent_V1001,
Storage_StorageOperatorMetadataSetEvent_V1001 as StorageOperatorMetadataSetEvent_V1001,
Storage_VoucherChangedEvent_V1001 as VoucherChangedEvent_V1001,
} from '../../generated/types'
import {
DistributionBucket,
DistributionBucketFamily,
DistributionBucketOperator,
DistributionBucketOperatorMetadata,
DistributionBucketOperatorStatus,
NodeLocationMetadata,
StorageBag,
StorageBucket,
StorageBucketOperatorStatusActive,
StorageBucketOperatorStatusInvited,
StorageBucketOperatorStatusMissing,
StorageDataObject,
GeoCoordinates,
} from 'query-node/dist/model'
import BN from 'bn.js'
import { getById, inconsistentState } from '../common'
import { videoRelationsForCounters, unsetAssetRelations } from '../content/utils'
import { unsetAssetRelations, videoRelationsForCounters } from '../content/utils'
import { getAllManagers } from '../derivedPropertiesManager/applications'
import {
processDistributionBucketFamilyMetadata,
processDistributionOperatorMetadata,
processStorageOperatorMetadata,
} from './metadata'
import {
createDataObjects,
getStorageBucketWithOperatorMetadata,
deleteDataObjects,
distributionBucketId,
distributionBucketIdByFamilyAndIndex,
distributionOperatorId,
getBag,
getDynamicBagId,
getDynamicBagOwner,
getDataObjectsInBag,
getDynamicBag,
getDistributionBucketFamilyWithMetadata,
getDistributionBucketOperatorWithMetadata,
distributionBucketId,
distributionOperatorId,
distributionBucketIdByFamilyAndIndex,
deleteDataObjects,
getDynamicBag,
getDynamicBagId,
getDynamicBagOwner,
getStorageBucketWithOperatorMetadata,
} from './utils'
import { getAllManagers } from '../derivedPropertiesManager/applications'

// STORAGE BUCKETS

Expand Down Expand Up @@ -233,6 +233,7 @@ export async function storage_DynamicBagCreated({ event, store }: EventContext &
const storageBag = new StorageBag({
id: getDynamicBagId(bagId),
owner: getDynamicBagOwner(bagId),
objectsSize: new BN(0),
storageBuckets: Array.from(storageBuckets).map((id) => new StorageBucket({ id: id.toString() })),
distributionBuckets: Array.from(distributionBuckets).map(
(id) => new DistributionBucket({ id: distributionBucketId(id) })
Expand Down Expand Up @@ -364,13 +365,21 @@ export async function storage_PendingDataObjectsAccepted({ event, store }: Event
export async function storage_DataObjectsMoved({ event, store }: EventContext & StoreContext): Promise<void> {
const [srcBagId, destBagId, dataObjectIds] = new DataObjectsMovedEvent_V1001(event).params
const dataObjects = await getDataObjectsInBag(store, srcBagId, dataObjectIds)
const srcBag = await getBag(store, srcBagId)
const destBag = await getBag(store, destBagId)
await Promise.all(
dataObjects.map(async (dataObject) => {
dataObject.storageBag = destBag
await store.save<StorageDataObject>(dataObject)
})
)

// Update source & destination bags size
const movedDataObjectsSize = dataObjects.reduce((acc, dataObject) => acc.add(dataObject.size), new BN(0))
srcBag.objectsSize = srcBag.objectsSize.sub(movedDataObjectsSize)
destBag.objectsSize = destBag.objectsSize.add(movedDataObjectsSize)
await store.save<StorageBag>(srcBag)
await store.save<StorageBag>(destBag)
}

export async function storage_DataObjectsDeleted({ event, store }: EventContext & StoreContext): Promise<void> {
Expand Down
34 changes: 23 additions & 11 deletions query-node/mappings/src/storage/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,39 @@
import { DatabaseManager } from '@joystream/hydra-common'
import {
PalletStorageBagIdType as BagId,
PalletStorageDataObjectCreationParameters as DataObjectCreationParameters,
PalletStorageDistributionBucketIdRecord as DistributionBucketId,
PalletStorageDynamicBagIdType as DynamicBagId,
PalletStorageStaticBagId as StaticBagId,
PalletStorageDistributionBucketIdRecord as DistributionBucketId,
PalletStorageDataObjectCreationParameters as DataObjectCreationParameters,
} from '@polkadot/types/lookup'
import BN from 'bn.js'
import {
DataObjectTypeUnknown,
DistributionBucketFamily,
DistributionBucketOperator,
StorageBag,
StorageDataObject,
StorageBagOwner,
StorageBagOwnerChannel,
StorageBagOwnerCouncil,
StorageBagOwnerMember,
StorageBagOwnerWorkingGroup,
StorageBucket,
DistributionBucketOperator,
DistributionBucketFamily,
StorageDataObject,
} from 'query-node/dist/model'
import BN from 'bn.js'
import { bytesToString, inconsistentState, getById, RelationsArr } from '../common'
import { In } from 'typeorm'
import { RelationsArr, bytesToString, getById, inconsistentState } from '../common'

import { BTreeSet } from '@polkadot/types'
import _ from 'lodash'
import {
WorkerId,
DataObjectId,
DistributionBucketFamilyId,
DistributionBucketIndex,
WorkerId,
} from '@joystream/types/primitives'
import { BTreeSet } from '@polkadot/types'
import { Balance } from '@polkadot/types/interfaces'
import _ from 'lodash'
import { unsetAssetRelations, videoRelationsForCounters } from '../content/utils'
import { getAllManagers } from '../derivedPropertiesManager/applications'
import { videoRelationsForCounters, unsetAssetRelations } from '../content/utils'

export type StorageDataObjectParams = {
storageBagOrId: StorageBag | BagId
Expand Down Expand Up @@ -155,6 +155,7 @@ export async function getStaticBag(
console.log(`Creating new static bag: ${id}`)
const newBag = new StorageBag({
id,
objectsSize: new BN(0),
owner: getStaticBagOwner(bagId),
})
await store.save<StorageBag>(newBag)
Expand Down Expand Up @@ -234,6 +235,11 @@ export async function createDataObjects(

await Promise.all(dataObjects.map((o) => store.save<StorageDataObject>(o)))

// Update bag size
const createdDataObjectsSize = dataObjects.reduce((acc, dataObject) => acc.add(dataObject.size), new BN(0))
storageBag.objectsSize = storageBag.objectsSize.add(createdDataObjectsSize)
await store.save<StorageBag>(storageBag)

return dataObjects
}

Expand All @@ -242,6 +248,7 @@ export async function deleteDataObjects(
bagId: BagId,
dataObjectIds: BTreeSet<DataObjectId>
): Promise<void> {
const storageBag = await getBag(store, bagId)
const dataObjects = await getDataObjectsInBag(store, bagId, dataObjectIds, [
'videoThumbnail',
...videoRelationsForCounters.map((item) => `videoThumbnail.${item}`),
Expand All @@ -255,6 +262,11 @@ export async function deleteDataObjects(

await unsetAssetRelations(store, dataObject)
}

// Update bag size
const deletedDataObjectsSize = dataObjects.reduce((acc, dataObject) => acc.add(dataObject.size), new BN(0))
storageBag.objectsSize = storageBag.objectsSize.sub(deletedDataObjectsSize)
await store.save<StorageBag>(storageBag)
}

export function distributionBucketId(runtimeBucketId: DistributionBucketId): string {
Expand Down
2 changes: 1 addition & 1 deletion query-node/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "query-node-root",
"version": "1.2.2",
"version": "1.3.0",
"description": "GraphQL server and mappings. Generated with ♥ by Hydra-CLI",
"scripts": {
"build": "./build.sh",
Expand Down
3 changes: 3 additions & 0 deletions query-node/schemas/storage.graphql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,9 @@ type StorageBag @entity {
"Data objects in the bag"
objects: [StorageDataObject!] @derivedFrom(field: "storageBag")

"Total size of data objects in Bag"
objectsSize: BigInt!

"Storage buckets assigned to the bag"
storageBuckets: [StorageBucket!]

Expand Down
13 changes: 13 additions & 0 deletions storage-node/src/services/queryNode/schema.graphql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,6 +1047,9 @@ type StorageBag implements BaseGraphQLObject {
deletedById: ID
version: Int!
objects: [StorageDataObject!]!

"""Total size of data objects in Bag"""
objectsSize: BigInt!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just as a general question, if an older version of storage node does a query for StorageBag and they are not aware of the new schema, how will it behave? Will it fail to parse the response or just not be aware of this new field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a new field to the graphql schema is generally future-proof (unless we make breaking changes to the schema), So if the older version of storage-node is unaware of the new schema and they don't query the new field, the query would work fine.

For example, the storage-node makes use of storageBagsConnection query to get all assigned bags. As you can see the only field being queried is bag id. So being unaware of the new schema does not create the issues.

storageBuckets: [StorageBucket!]!
distributionBuckets: [DistributionBucket!]!

Expand Down Expand Up @@ -12474,6 +12477,12 @@ input StorageBagWhereInput {
deletedAt_gte: DateTime
deletedById_eq: ID
deletedById_in: [ID!]
objectsSize_eq: BigInt
objectsSize_gt: BigInt
objectsSize_gte: BigInt
objectsSize_lt: BigInt
objectsSize_lte: BigInt
objectsSize_in: [BigInt!]
owner_json: JSONObject
objects_none: StorageDataObjectWhereInput
objects_some: StorageDataObjectWhereInput
Expand All @@ -12494,10 +12503,12 @@ input StorageBagWhereUniqueInput {
}

input StorageBagCreateInput {
objectsSize: String!
owner: JSONObject!
}

input StorageBagUpdateInput {
objectsSize: String
owner: JSONObject
}

Expand Down Expand Up @@ -31855,6 +31866,8 @@ enum StorageBagOrderByInput {
updatedAt_DESC
deletedAt_ASC
deletedAt_DESC
objectsSize_ASC
objectsSize_DESC
}

enum StorageBucketOperatorMetadataOrderByInput {
Expand Down