Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Accumulate Events #65

Merged
merged 41 commits into from
Jan 27, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ee49b2a
feat: editor config, sane!
whizzzkid Jan 19, 2023
d07079d
Add event accumulator class
whizzzkid Jan 19, 2023
ccece9b
feat: Update types and hookup accumulator
whizzzkid Jan 19, 2023
f1da631
fix: configs
whizzzkid Jan 20, 2023
06ef5a9
feat: moving types to set rootDir
whizzzkid Jan 20, 2023
525234c
adding test folder
whizzzkid Jan 20, 2023
b8cde72
adding relevant types
whizzzkid Jan 20, 2023
882075f
types migration
whizzzkid Jan 20, 2023
6a59727
fix: add_event
whizzzkid Jan 20, 2023
dd6a6cb
feat: adding accumulator test.
whizzzkid Jan 20, 2023
1802c4e
breaking down logic
whizzzkid Jan 20, 2023
9e62d8f
fix: import order
whizzzkid Jan 20, 2023
ded115c
fix: sinon stubs
whizzzkid Jan 20, 2023
719519c
feat: Adding impl using map
whizzzkid Jan 20, 2023
c459dcb
adding flushAll method
whizzzkid Jan 20, 2023
d23fcb6
Hooking up to the beforunload event
whizzzkid Jan 20, 2023
db97c66
lint
whizzzkid Jan 20, 2023
a28bc36
Merge branch 'main' into feat/accumulate-events
whizzzkid Jan 21, 2023
5791d84
fix: :twisted_rightwards_arrows: merged with upstream
whizzzkid Jan 21, 2023
98a7ab2
fix: :package: adding missing package after merge
whizzzkid Jan 21, 2023
90b37ba
fix: :test_tube: EventAccumulator
whizzzkid Jan 21, 2023
eb5ae38
fix: :adhesive_bandage: fix imports
whizzzkid Jan 21, 2023
f638eb8
fix: :truck: rename to spec.ts
whizzzkid Jan 21, 2023
1238a5f
fix: :recycle: cleanup
whizzzkid Jan 21, 2023
6bd68c6
fix: :package: package-lock.json
whizzzkid Jan 21, 2023
7c07a7d
fix: :pencil2: spec naming and scope
whizzzkid Jan 23, 2023
4fe767b
Merge branch 'main' into feat/accumulate-events
whizzzkid Jan 25, 2023
9eea9a5
fix: :wrench: don't disable rules globally
whizzzkid Jan 25, 2023
2722ec2
fix: :wrench: only apply eslint overrides to spec files.
whizzzkid Jan 25, 2023
8eb34e9
fix: :rotating_light: fixes lint.
whizzzkid Jan 26, 2023
d34928b
fix: :recycle: Moving tests to node folder
whizzzkid Jan 26, 2023
772018d
fix: :label: generics
whizzzkid Jan 26, 2023
2313806
fix: :necktie: instantiating at declaration
whizzzkid Jan 26, 2023
8c4f560
fix: :memo: adding missing documentation
whizzzkid Jan 26, 2023
c999b06
fix: :necktie: fixing unload event to handle both browser and node.
whizzzkid Jan 26, 2023
fbaa4f3
fix: :truck: digest -> accumulate
whizzzkid Jan 26, 2023
dfc72df
fix: generics
whizzzkid Jan 26, 2023
c2c4277
fix: :pencil2: dupe imports
whizzzkid Jan 26, 2023
a0242f9
fix: fixing appKey
whizzzkid Jan 26, 2023
ab218ab
feat: :zap: useFakeTimers instead of await
whizzzkid Jan 26, 2023
3549175
fix: :recycle: move call order
whizzzkid Jan 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
root=true

[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8
indent_style = space
indent_size = 2
99 changes: 99 additions & 0 deletions src/EventAccumulator.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
import type { CountlyWebSdk, IEventAccumulator, CountlyEvent, CountlyEventData } from 'countly-sdk-web'

const eventDefaults: CountlyEventData = {
key: '',
count: 1,
sum: 1,
dur: Date.now(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duration should be an amount of time, not a timestamp. This duration is currently going to be the full time since 1970.

We should probably default this to 0 or even leave it unset entirely.

see

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this gets reset before the event gets flushed. setting default to 0.

segmentation: {}
}

interface eventStore {
eventData: CountlyEventData
startTime: number
timeout: NodeJS.Timeout
}

/**
* EventAccumulator is a class that accumulates events and flushes them to the Countly server.
*/
export class EventAccumulator implements IEventAccumulator {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export class EventAccumulator implements IEventAccumulator {
export class EventAccumulator<T extends CountlyWebSdk | CountlyNodeSdk> implements IEventAccumulator {

private readonly metricsService: CountlyWebSdk
private readonly events: Record<string, eventStore>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private readonly events: Record<string, eventStore>
private readonly events: Map<string, eventStore>

private readonly flushInterval: number

/**
* Create a new EventAccumulator
*
* @param {CountlyWebSdk} metricsService - instance
* @param {number} flushInterval - in milliseconds
*/
constructor (metricsService: CountlyWebSdk, flushInterval: number = 5 * 60 * 1000) {
this.metricsService = metricsService
this.flushInterval = flushInterval
this.events = {}
whizzzkid marked this conversation as resolved.
Show resolved Hide resolved
}

/**
* Add an event to the accumulator
*
* @param {CountlyEvent} event - event to add
* @param {boolean} flush - optionally whether to flush the event immediately
*/
addEvent (event: CountlyEvent, flush: boolean = false): void {
// create a new event object with defaults.
const newEvent: CountlyEventData = { ...eventDefaults, ...event }
const { key, count } = newEvent

// validate event
if (key === '') {
throw new Error('Event key is required')
}

// if event is not in the store, add it.
if (!(key in this.events)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (!(key in this.events)) {
if (!this.events.has(key)) {

this.events[key] = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
this.events[key] = {
this.events.set(key, {

eventData: newEvent,
// set start time to now. This will be updated when the event is flushed.
startTime: Date.now(),
// set a timeout to flush the event after the flush interval.
timeout: setTimeout(() => {
this.flush(key)
}, this.flushInterval)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
})

} else {
// if event is in the store, update the event data.
const { eventData } = this.events[key]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const { eventData } = this.events[key]
const { eventData } = this.events.get(key) as eventStore

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

eventData.count += count
eventData.sum += 1
eventData.segmentation = { ...eventData.segmentation, ...newEvent.segmentation }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
eventData.segmentation = { ...eventData.segmentation, ...newEvent.segmentation }
eventData.segmentation = { ...eventData.segmentation, ...newEvent.segmentation }
this.events.set(key, eventData)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do I need to set it back? Map.get should get me reference to the object, modifying the object updates the referenced object.

}
whizzzkid marked this conversation as resolved.
Show resolved Hide resolved

// flush the event if flush is true.
if (flush) {
this.flush(key)
}
}

/**
* Flush an event from the accumulator
*
* @param {string} key - event key
*/
flush (key: string): void {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we have a flush_all?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added with tests

// if event is not in the store, return.
if (!(key in this.events)) {
return
}

const { eventData, startTime, timeout } = this.events[key]

// update duration to ms from start.
eventData.dur = Date.now() - startTime
whizzzkid marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove duration from the submitted event though, because countly doesn't really display them the way we think they should.. maybe we should chat about this while taking a peek at existing webui data in our countly server

// add event to the async queue.
this.metricsService.q.push(['add_event', eventData])
clearTimeout(timeout)
// eslint-disable-next-line @typescript-eslint/no-dynamic-delete
delete this.events[key]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we use a map for this.events we can delete it easier that way.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep

}
}
7 changes: 5 additions & 2 deletions src/MetricsProvider.ts
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
import { COUNTLY_SETUP_DEFAULTS } from './config.js'
import { EventAccumulator } from './EventAccumulator.js'

import type { metricFeatures, CountlyWebSdk } from 'countly-sdk-web'
import type { CountlyNodeSdk } from 'countly-sdk-nodejs'
import type { CountlyWebSdk, metricFeatures } from 'countly-sdk-web'
import type { consentTypes, consentTypesExceptAll } from './types/index.js'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should update eslint so it's organizing imports instead of depending on your editor for that! otherwise they'll keep getting out of order ;)

speaking of ordering imports. Can we group the imports? we've currently got two relative imports at the top, then 2 3p imports (countly-sdk-*), then another relative.


export interface MetricsProviderConstructorOptions<T> {
appKey: string
autoTrack?: boolean
interval?: number
max_events?: number
metricsService: T
queue_size?: number
session_update?: number
url?: string
metricsService: T
}

export default class MetricsProvider<T extends CountlyWebSdk & CountlyNodeSdk> {
public readonly accumulate: EventAccumulator
private readonly groupedFeatures: Record<consentTypes, metricFeatures[]> = this.mapAllEvents({
minimal: ['sessions', 'views', 'events'],
performance: ['crashes', 'apm'],
Expand All @@ -40,6 +42,7 @@ export default class MetricsProvider<T extends CountlyWebSdk & CountlyNodeSdk> {
url,
require_consent: true
})
this.accumulate = new EventAccumulator(metricsService)

this.metricsService.init(serviceConfig)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to revert this. appKey is not a valid argument for metricsService.init

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not typechecked as expected, I fixed it like:

    const { appKey, ...remainderConfig } = config
    const serviceConfig = {
      ...COUNTLY_SETUP_DEFAULTS,
      ...remainderConfig,
      app_key: appKey
    }

I should throw if appKey is missing?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea we should. we can handle in a separate PR though.

this.metricsService.group_features(this.groupedFeatures)
Expand Down
13 changes: 10 additions & 3 deletions src/types/countly.d.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
declare module 'countly-sdk-web' {
interface CountlyEventData {
export interface CountlyEventData {
key: string
count: number
sum: number
segmentation: Record<string, string | number>
dur: number
segmentation: Segments
}
interface CountlyEvent {
export interface CountlyEvent {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

// name or id of the event
key: string
// how many times did event occur
Expand All @@ -18,12 +19,18 @@ declare module 'countly-sdk-web' {
segmentation?: Segments
}

export interface IEventAccumulator {
addEvent: (event: CountlyEvent, flush: boolean) => void
flush: (key: string) => void
}

export type metricFeatures = 'apm' | 'attribution' | 'clicks' | 'crashes' | 'events' | 'feedback' | 'forms' |
'location' | 'scrolls' | 'sessions' | 'star-rating' | 'users' | 'views'
type Segments = Record<string, string>
type IgnoreList = Array<string | RegExp>
type CountlyEventQueueItem = [string, CountlyEventData] | [eventName: string, key: string] | [eventName: string]
export interface CountlyWebSdk {
accumulate: IEventAccumulator
group_features: (arg0: Record<import('./index.js').consentTypes, metricFeatures[]>) => unknown
check_consent: (consentFeature: metricFeatures | import('./index.js').consentTypes) => boolean
add_consent: (consentFeature: import('./index.js').consentTypes | Array<import('./index.js').consentTypes>) => void
Expand Down