CLOWDER and tests #25

Emily-ejag · 2024-08-26T21:45:13Z

The next steps are:

@Emily Arteaga and @Anya Ma will meet to add a fixed order method to jsCat
@Emily Arteaga will create a new typescript class to manage next item selection in ROAR apps. My strong recommendation is to call this class Clowder because part of it’s job will be to manage multiple Cat instances and “clowder” is the collective noun for cats.
The Clowder class will take input parameters that define multiple corpora, stimulus presentation rules, etc. and have a method called getNextStimulus or something like that.
@Emily Arteaga will pilot this new class in ROAR-Letter. The next target after that will be Palabra.

coveralls · 2024-08-26T21:52:53Z

Pull Request Test Coverage Report for Build 11807958523

Details

1319 of 1319 (100.0%) changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+2.4%) to 100.0%

Totals
Change from base Build 10530762939:	2.4%
Covered Lines:	1382
Relevant Lines:	1382

💛 - Coveralls

AnyaWMa · 2024-08-26T22:24:20Z

can you describe what's the purpose of this PR?

Co-authored-by: Adam Richie-Halford <richford@users.noreply.github.com>

src/__tests__/clowder.test.ts

src/clowder.ts

richford

Nevermind. I implemented my own comments since I was in there changing other things anyway.

richford · 2024-10-05T10:28:12Z

The test coverage indicates that there are some branches that are not tested in stopping.ts. But I don't think that's a blocker for this PR. I think it's ready for review by @AnyaWMa and to start implementing in ROAR-Letter.

AnyaWMa

hi I read through a few places, and added some questions. Many of them are probably due to my ignorance, but i just need to know more details how it works. Thank you!

AnyaWMa · 2024-10-11T17:45:16Z

src/corpus.ts

+  items: Stimulus[],
+  catNames: string[],
+  delimiter: '.' | string,
+  itemParameterFormat: 'symbolic' | 'semantic' = 'symbolic',


can it accept: "a, b, guessing, and d"

When using the prepareClowderCorpus function, the parameters need to be consistent—either all symbolic (a, b, c, d) or all semantic (discrimination, difficulty, guessing, slipping). We can't mix and match between symbolic and semantic formats in the same input. The function uses the itemParameterFormat option to convert everything to the desired format, but we need to pass one type for the input keys.

is this requirement documented somewhere?

i would suggest document it in README with sample code

AnyaWMa · 2024-10-11T17:48:19Z

src/stopping.ts

+}
+
+/**
+ * Class implementing early stopping based on a plateau in standard error of measurement.


how is this different from line 195?

I don't get it, do you mean we have repetitive classes?

This one stops if the SEMeasurement plateaus, meaning that is fails to decrease for a certain number of trials (called patience). E.g., if you have

trial number SEMeasurement

1 1.0

2 0.8

3 0.6

4 0.4

5 0.4

6 0.4

7 0.4

If the patience was set to 3, then the early stopping would have been triggered on trial 6 because the SE had failed to decrease for 3 consecutive trials.

got it! thank you!

AnyaWMa · 2024-10-11T17:51:49Z

src/stopping.ts

+/**
+ * Interface for input parameters to EarlyStopping classes.
+ */
+export interface EarlyStoppingInput {


i am confused how many combinations of early stopping criteria are available here? can you document this somewhere?

There are a few combinations we can use for early stopping:

Logical Operations: We can choose between and, or, and only for combining multiple stopping criteria:

and: All conditions need to be met to trigger stopping.

or: Any one condition being met will trigger stopping.

only: Only a specific condition is considered (requires you to specify the cat to evaluate).

Stopping Criteria Classes:

StopAfterNItems: Stops after a specified number of items.

StopOnSEMeasurementPlateau: Stops if the standard error (SE) of measurement remains stable (within a tolerance) for a specified number of items.

StopIfSEMeasurementBelowThreshold: Stops if the SE measurement drops below a set threshold.

I added these on the readme file :)

i think providing sample code in README would be helpful.

AnyaWMa · 2024-10-11T17:54:52Z

src/__tests__/clowder.test.ts

+  beforeEach(() => {
+    const clowderInput: ClowderInput = {
+      cats: {
+        cat1: { method: 'MLE', theta: 0.5 },


what is contained in the cat object? should cat be defined as a cat class?

In reality, it is a Cat instance. But for unit testing, it is enough to provide an object that "looks" like a Cat instance.

AnyaWMa · 2024-10-11T17:57:16Z

src/cat.ts

+
+const abilityPrior = normal();
+
+export interface CatInput {


what is the difference between CatInput and Cat

CatInput is just a configuration interface that defines the setup options for creating a Cat, like method or theta. On the other hand, Cat is the actual class that manages the behavior and state.

Basically, CatInput is for setup, and Cat does the work!

i am confused. does it mean the user need to create CatInput instead of Cat?

if yes, can you add documentation in the README to explain how to use jsClowder

And will that conflict the use if user only wants to use jsCAT instead of jsClowder?

I think we clarified this in our Slack huddle but I'll just document the conversation here. CatInput is an interface defining the input format that Cat expects when it is being instantiated. The user creates a new Cat instance by passing in parameters that conform to the CatInput interface. It's actually already defined in jsCat. Emily didn't add it in this PR. @AnyaWMa , you added it two years ago in this commit.

AnyaWMa · 2024-10-11T18:03:08Z

src/__tests__/clowder.test.ts

+      items: clowder.corpus[0],
+      answers: 1,
+    });
+    expect(nextItem).toBeDefined();


what does "toBeDefined()" mean?

This is a Jest "matcher" used for unit testing https://jestjs.io/docs/expect#tobedefined

okay, thanks for explaining.

AnyaWMa · 2024-10-11T18:07:06Z

src/__tests__/stopping.test.ts

+  StopIfSEMeasurementBelowThreshold,
+  StopIfSEMeasurementBelowThresholdInput,
+  StopOnSEMeasurementPlateau,
+  StopOnSEMeasurementPlateauInput,


what is StopOnSEMeasurementPlateau?

In simpler terms, if the SE doesn't significantly change for a set number of items, the process stops early because it indicates that the ability estimate is no longer improving. It's used to avoid unnecessary trials once the measurement has plateaued.

See my comment here: #25 (comment)

AnyaWMa · 2024-10-11T18:07:45Z

src/__tests__/stopping.test.ts

@@ -0,0 +1,739 @@
+import { Cat } from '..';


do we have tests the cat will stop when item bank is used up?

Yes, all lines are covered now

where is the test when item bank is used up?

line 303 is the test if you want a set number, if you don't add early stopping it will go trough all

And line 98 in src/__tests__/clowder.test.ts tests that Clowder will return undefined when an item bank for a specified catToSelect is used up.

src/__tests__/clowder.test.ts

src/clowder.ts

src/stopping.ts

src/__tests__/stopping.test.ts

AnyaWMa

Thank you for lots of improvement! I understand the code better now, but I will still appreciate more detailed documentation in the README. Specficially, I want to know will jsClowder will conflict any use if the user will want to use jsCAT?

AnyaWMa · 2024-10-29T03:38:56Z

README.md

+- Using **`or`** with `StopOnSEMeasurementPlateau` and `StopAfterNItems` allows early stopping if either condition is met.
+
+If you need more details or a specific example documented, feel free to ask!
+


thanks for the document. can you give sample lines of code here?

AnyaWMa · 2024-10-29T03:40:27Z

README.md

@@ -42,22 +42,56 @@ const stimuli = [{difficulty: -3, item: 'item1'}, {difficulty: -2,  item: 'item2
 const nextItem = cat.findNextItem(stimuli, 'MFI');
 ```



can you add documentation about the accepted stimuli types?

AnyaWMa · 2024-10-29T03:45:49Z

src/cat.ts

+
+const abilityPrior = normal();
+
+export interface CatInput {


i am confused. does it mean the user need to create CatInput instead of Cat?

if yes, can you add documentation in the README to explain how to use jsClowder

And will that conflict the use if user only wants to use jsCAT instead of jsClowder?

AnyaWMa · 2024-10-29T03:46:51Z

src/corpus.ts

+  items: Stimulus[],
+  catNames: string[],
+  delimiter: '.' | string,
+  itemParameterFormat: 'symbolic' | 'semantic' = 'symbolic',


i would suggest document it in README with sample code

AnyaWMa · 2024-10-29T03:47:38Z

src/stopping.ts

+/**
+ * Interface for input parameters to EarlyStopping classes.
+ */
+export interface EarlyStoppingInput {


i think providing sample code in README would be helpful.

AnyaWMa · 2024-10-29T03:56:49Z

README.md

@@ -42,22 +42,56 @@ const stimuli = [{difficulty: -3, item: 'item1'}, {difficulty: -2,  item: 'item2
 const nextItem = cat.findNextItem(stimuli, 'MFI');
 ```



more broadly, can we document and show sample code: how to set up a jsClowder, and some basic functions to run a clowder.

AnyaWMa

sorry for another round of clarification.

I am asking because I found in my current jsCAT: the desired zeta type is actually a mix of symbolic and semantic (a, difficulty, c, and d.). I am okay to just be all semantic or all symbolic, but I want to make sure all places that are hard-coded in the code will function as expected for this transition. Thanks!

AnyaWMa · 2024-10-30T06:40:21Z

src/cat.ts

+      // for mfi, we sort the arr by fisher information in the private function to select the best item,
+      // and then sort by difficulty to return the remainingStimuli
+      // for fixed, we want to keep the corpus order as input
+      arr.sort((a: Stimulus, b: Stimulus) => a.difficulty! - b.difficulty!);


will anything break here because it asks for difficulty not b?

Nope. Nine lines above, Emily ensures that the item array is in the semantic format using the fillZetaDefaults method.

AnyaWMa · 2024-10-30T06:53:10Z

src/__tests__/cat.test.ts

+
+    it.each`
+      deepCopy
+      ${true}
+      ${false}
+    `("correctly suggests the next item (closest method) with deepCopy='$deepCopy'", ({ deepCopy }) => {
+      const expected = { nextStimulus: s5, remainingStimuli: [s4, s1, s3, s2] };
+      const received = cat1.findNextItem(stimuli, 'closest', deepCopy);
+      expect(received).toEqual(expected);
+    });
+
+    it.each`
+      deepCopy
+      ${true}
+      ${false}
+    `("correctly suggests the next item (mfi method) with deepCopy='$deepCopy'", ({ deepCopy }) => {
+      const expected = { nextStimulus: s1, remainingStimuli: [s4, s5, s3, s2] };
+      const received = cat3.findNextItem(stimuli, 'MFI', deepCopy);
+      expect(received).toEqual(expected);
+    });
+
+    it.each`
+      deepCopy
+      ${true}
+      ${false}
+    `("correctly suggests the next item (middle method) with deepCopy='$deepCopy'", ({ deepCopy }) => {
+      const expected = { nextStimulus: s1, remainingStimuli: [s4, s5, s3, s2] };
+      const received = cat5.findNextItem(stimuli, undefined, deepCopy);
+      expect(received).toEqual(expected);
+    });
+
+    it.each`
+      deepCopy
+      ${true}
+      ${false}
+    `("correctly suggests the next item (fixed method) with deepCopy='$deepCopy'", ({ deepCopy }) => {
+      expect(cat8.itemSelect).toBe('fixed');
+      const expected = { nextStimulus: s1, remainingStimuli: [s2, s3, s4, s5] };
+      const received = cat8.findNextItem(stimuli, undefined, deepCopy);
+      expect(received).toEqual(expected);
+    });
+
+    it.each`
+      deepCopy
+      ${true}
+      ${false}
+    `("correctly suggests the next item (random method) with deepCopy='$deepCopy'", ({ deepCopy }) => {
+      let received;
+      const stimuliSorted = stimuli.sort((a: Stimulus, b: Stimulus) => a.difficulty! - b.difficulty!); // ask
+      let index = Math.floor(rng() * stimuliSorted.length);
+      received = cat4.findNextItem(stimuliSorted, undefined, deepCopy);
+      expect(received.nextStimulus).toEqual(stimuliSorted[index]);
+
+      for (let i = 0; i < 3; i++) {
+        const remainingStimuli = received.remainingStimuli;
+        index = Math.floor(rng() * remainingStimuli.length);
+        received = cat4.findNextItem(remainingStimuli, undefined, deepCopy);
+        expect(received.nextStimulus).toEqual(remainingStimuli[index]);
+      }
+    });


sorry for more questions.

I just realized i haven't reviewed these new tests before.

What does the "deepcopy" do here?

deepCopy is part of the original jsCat code. It dictates whether the input array should be deep copied before sorting in the findNextItem method. While the deepCopy parameter itself isn't new (it's been around since this commit, Emily noticed that it wasn't being tested, so now the unit tests test whether findNextItem works as expected with both deepCopy=true and deepCopy=false.

AnyaWMa · 2024-10-30T06:58:31Z

src/cat.ts

+  private selectorMFI(inputStimuli: Stimulus[]) {
+    const stimuli = inputStimuli.map((stim) => fillZetaDefaults(stim, 'semantic'));
+    const stimuliAddFisher = stimuli.map((element: Stimulus) => ({
+      fisherInformation: fisherInformation(this._theta, fillZetaDefaults(element, 'symbolic')),
+      ...element,
+    }));
+
+    stimuliAddFisher.sort((a, b) => b.fisherInformation - a.fisherInformation);
+    stimuliAddFisher.forEach((stimulus: Stimulus) => {
+      delete stimulus['fisherInformation'];
+    });
+    return {
+      nextStimulus: stimuliAddFisher[0],
+      remainingStimuli: stimuliAddFisher.slice(1).sort((a: Stimulus, b: Stimulus) => a.difficulty! - b.difficulty!),
+    };
+  }


I need a clarification here: the returned remainingStimuli will be semantic or symbolic, or it will be consistent with original zeta type?

The returned stimuli will be semantic, in keeping with the legacy behavior of jsCat.

AnyaWMa · 2024-10-30T16:26:32Z

i approved the PR, but I will mention this version will break current use of jsCAT in apps. The structure of taking zetas is completely different, and the README is not valid too. it is important to let the users know (swr, vocab, comp, and maybe levante). Thank you!

adding clowder class and tests

6a4a6f6

Emily-ejag self-assigned this Aug 26, 2024

changing Clowder for clowder

21b2f97

Emily-ejag requested review from richford and AnyaWMa August 26, 2024 21:49

Emily-ejag added the enhancement New feature or request label Aug 26, 2024

richford changed the title ~~CROWDER and tests~~ CLOWDER and tests Sep 4, 2024

Emily-ejag and others added 3 commits September 9, 2024 17:02

Adding updateCatAndGetNextItem function

759dae5

Co-authored-by: Adam Richie-Halford <richford@users.noreply.github.com>

eslint for unused -- used variables

3a0cdcb

clowder import

2b570a1

Emily-ejag requested review from richford and AnyaWMa and removed request for richford and AnyaWMa September 10, 2024 00:14

richford and others added 14 commits September 17, 2024 17:33

Add zetas for multiple cats to the corpus

a1f608a

Add TODO comments

423954f

Add util tests

22b85fe

Add documentation

242f02f

Document and test utils

9fb9436

Start adding clowder tests

fc72b04

Add more clowder tests

92a4d74

Add documentation and randomlySelectUnvalidated parameter

5082d5b

Add more tests and a random seed

f7cf2f7

Reorganize files

3d1a614

Don't export abilityPrior

8ac9f44

Update readme

668d68b

adding missing tests to clowder

3550d96

Import Cat, Clowder, and ClowderInput from index

a1d2bf9

Emily-ejag requested a review from richford October 2, 2024 18:43

richford requested changes Oct 5, 2024

View reviewed changes

Separate the stopping classes so that they don't share the same input

d641960

richford approved these changes Oct 5, 2024

View reviewed changes

updating cats for clowder

07e1f58

AnyaWMa requested changes Oct 11, 2024

View reviewed changes

Emily-ejag added 3 commits October 17, 2024 16:18

clowder changes based on letter implementation

a0ac266

addressing all lines of code for testing

3674a1f

adding documentation about early stopping

74bf577

Emily-ejag requested a review from AnyaWMa October 18, 2024 21:31

richford requested changes Oct 23, 2024

View reviewed changes

src/__tests__/clowder.test.ts Outdated Show resolved Hide resolved

src/clowder.ts Outdated Show resolved Hide resolved

src/stopping.ts Outdated Show resolved Hide resolved

src/__tests__/stopping.test.ts Show resolved Hide resolved

src/__tests__/stopping.test.ts Show resolved Hide resolved

Emily-ejag added 3 commits October 23, 2024 13:51

since we added only, we need to add catToSelect

508bc51

solving adams comments

efbd55b

deleting for loop

c26f57c

richford approved these changes Oct 23, 2024

View reviewed changes

AnyaWMa requested changes Oct 29, 2024

View reviewed changes

richford requested review from AnyaWMa and richford October 29, 2024 18:09

Emily-ejag added 2 commits October 29, 2024 14:02

adding stopping reason

7abca47

adding more stoppingReasons to the tests

fb5886c

richford previously approved these changes Oct 29, 2024

View reviewed changes

AnyaWMa requested changes Oct 30, 2024

View reviewed changes

richford mentioned this pull request Oct 30, 2024

WIP: Add documentation website #27

Draft

AnyaWMa previously approved these changes Oct 30, 2024

View reviewed changes

Update README.md

2f4285b

AnyaWMa dismissed stale reviews from richford and themself via 2f4285b October 30, 2024 16:46

filterin NA from overall corpus

e5a352e

		- Using `or` with `StopOnSEMeasurementPlateau` and `StopAfterNItems` allows early stopping if either condition is met.

		If you need more details or a specific example documented, feel free to ask!

		@@ -42,22 +42,56 @@ const stimuli = [{difficulty: -3, item: 'item1'}, {difficulty: -2, item: 'item2
		const nextItem = cat.findNextItem(stimuli, 'MFI');
		```

CLOWDER and tests #25

Are you sure you want to change the base?

CLOWDER and tests #25

Conversation

Emily-ejag commented Aug 26, 2024 • edited Loading

coveralls commented Aug 26, 2024 • edited Loading

Pull Request Test Coverage Report for Build 11807958523

Details

💛 - Coveralls

AnyaWMa commented Aug 26, 2024

richford left a comment

Choose a reason for hiding this comment

richford commented Oct 5, 2024

AnyaWMa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnyaWMa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnyaWMa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnyaWMa commented Oct 30, 2024

Emily-ejag commented Aug 26, 2024 •

edited

Loading

coveralls commented Aug 26, 2024 •

edited

Loading