Trinity EMS scheduling is now private; scraper needs updating #178

pfurbacher · 2021-03-24T15:22:02Z

This update accommodates the "private appointments only" change to the Trinity EMS scheduling page.

Adds extraData text from the alert on the page (for the moment, their text is "All appointment types are private, none are available for scheduling.")
Adds test and test data (HTML file) covering this new situation.
Adds to all tests the following check:

    expect(results).to.have.property("availability");
    expect(results).to.have.property("hasAvailability");

Refactoring the nested functions (moving out of "ScrapeWebsiteData()") requires adding the "private" results parameter to the moved function.

This reverts commit 357c504.

This reverts commit d4f9239.

…text"" This reverts commit ba4d77a.

…esponse text""" This reverts commit bebab9c.

…text"" This reverts commit ba4d77a.

…esponse text""" This reverts commit bebab9c.

…text"" This reverts commit ba4d77a.

…esponse text""" This reverts commit bebab9c.

Was yyyy-mm-dd; now m/d/yyyy.

…apers into Trinity-EMS

- added saving to s3 and slack messaging - implemented "service" to make the scraper more testable - fixed require("awk-sdk") -> s3 - made accumulating availability "more functional" - clarified confusing code comments - removed unused "days" var - fixed typo "writtent" -> "written" - removed guess as to how to parse slot availability text

Added unit testing for - no availability - availability across a number of days in several months Added a service interface for parsing slot count responses so that can be mocked. Added an additional "service" provider to the scraper to mock out s3 and slack messaging.

…apers into Trinity-EMS

Apparently, "test/bootstrap.js" supplies these in the automated test suite. But note that without require("./bootstrap.js") in this file, it's not possible for the developer to work with this code in VS Code: both "browser" and "expect" will be marked as undefined. I have found no way to insert bootstrap.js into the launch definition in VS Code.

- Previous version didn't allow testing of s3/slack messaging of the slot count fetch/response. Reworked the pageService interface to facilitate testing. - Added two expect statements relating to s3/slack messaging . - Removed extraneous logging.

…apers into Trinity-EMS

- Changes to handle the website's change to private appointments only. - Capture "private appointments only" text; assigned to "extraData" property of results object. - Added test and sample data (HTML file) to cover this new condition. Tests for presence of "extraData" property in results.

…ests expect(results).to.have.property("availability"); expect(results).to.have.property("hasAvailability");

pfurbacher · 2021-03-24T15:26:28Z

Oops! Did I do something wrong or are these conflicts expected?

livgust · 2021-03-25T00:24:35Z

@pfurbacher I don't know why you have conflicts in those files; could you perform a git merge from master to make sure everything's up-to-date? I don't think your branch is up to date.

…apers into Trinity-EMS

pfurbacher · 2021-03-25T08:24:57Z

Are the Fauna tests failing because of changes I made to TrinityEMS?

livgust · 2021-03-26T03:19:53Z

Says it's an authorization issue... @tkjones117 any insight here?

livgust

I tested and this looks good! Just waiting to hear back on auth problems with Fauna and possibly unintended changes to metrics.js.

livgust · 2021-03-26T03:21:24Z

lib/metrics.js

-            item?.availability &&
-            Object.keys(item.availability).length
-        ) {
+        if (item?.availability) {


do you know why this file changed?

I don't. Did I commit a change to metrics.js? I hope not because I haven't ever tinkered with it (I'm not even sure what it's supposed to do). Is it possible it was some artifact of me updating (merge/pull) from upstream, and then ... I have no idea. I'm not very experienced with git.

I can say that locally, when I ran rpm test it was only the Fauna tests which failed. I was surprised that 2 metrics tests failed in the pre-flight checks here. also, I figured that since the Fauna tests were using fake data, there couldn't be any connection to my code update.

I just checked and we definitely want the metrics.js file to have no diff here - the things that were reverted need to be put back. Once you can do that, @pfurbacher , I can merge!

livgust · 2021-03-30T02:51:32Z

lib/metrics.js

-            item?.availability &&
-            Object.keys(item.availability).length
-        ) {
+        if (item?.availability) {


I just checked and we definitely want the metrics.js file to have no diff here - the things that were reverted need to be put back. Once you can do that, @pfurbacher , I can merge!

…apers into Trinity-EMS

tkjones117 · 2021-03-30T20:30:58Z

@pfurbacher just make sure you have the most recent commits from master on this branch. It looks like you just merged some of Friday's commits, but you still need the commits that have been merged since Friday. Let me know if you want a hand with this.

Have no idea what happened here.

iann

This looks good to me. Trinity showing as private, however.

pfurbacher added 30 commits March 9, 2021 17:22

Initial commit

2c5cebb

Removed debug logging

2918b6a

Moved nested functions out of ScrapeWebsiteData(...)

de55381

Added tentative parsing of active day query response text

d4f9239

Added "results" parameter to "accumulateAvailabilityForMonth()"

357c504

Refactoring the nested functions (moving out of "ScrapeWebsiteData()") requires adding the "private" results parameter to the moved function.

Revert "Added "results" parameter to "accumulateAvailabilityForMonth()""

8996fde

This reverts commit 357c504.

Revert "Added tentative parsing of active day query response text"

ba4d77a

This reverts commit d4f9239.

Revert "Revert "Added tentative parsing of active day query response …

bebab9c

…text"" This reverts commit ba4d77a.

Revert "Revert "Revert "Added tentative parsing of active day query r…

d10deb5

…esponse text""" This reverts commit bebab9c.

Revert "Revert "Added tentative parsing of active day query response …

c2cea77

…text"" This reverts commit ba4d77a.

Revert "Revert "Revert "Added tentative parsing of active day query r…

3e2d7c8

…esponse text""" This reverts commit bebab9c.

Revert "Revert "Added tentative parsing of active day query response …

a59937b

…text"" This reverts commit ba4d77a.

Revert "Revert "Revert "Added tentative parsing of active day query r…

d83c418

…esponse text""" This reverts commit bebab9c.

Restored changes from reverts

96ac1e7

Updated confusing javadoc for "getDailyAvailabilityCountsForMonth()"

2167175

Corrected date format for results object keys

74e473a

Was yyyy-mm-dd; now m/d/yyyy.

Merge branch 'master' of https://github.com/livgust/covid-vaccine-scr…

2356116

…apers into Trinity-EMS

Removed sanity check code used in debugging

4a365d9

Added s3.savePageContent and slack messaging

69b1cb3

Added require slack

979c565

removed exploratory code

a03c6bc

Corrected the old aws require to lib/s3

3b7d024

Corrected CHROMEPATH issue

a654e90

Merge branch 'master' of https://github.com/livgust/covid-vaccine-scr…

23ea7c0

…apers into Trinity-EMS

Remove comment

08a79c6

Merge branch 'master' of https://github.com/livgust/covid-vaccine-scr…

8e5bab6

…apers into Trinity-EMS

pfurbacher added 2 commits March 24, 2021 11:09

Added checks for availability and hasAvailability properties to all t…

a99a1f3

…ests expect(results).to.have.property("availability"); expect(results).to.have.property("hasAvailability");

pfurbacher added 2 commits March 25, 2021 03:49

Merge branch 'master' of https://github.com/livgust/covid-vaccine-scr…

603506d

…apers into Trinity-EMS

Added signUpLink

1879fff

pfurbacher changed the title ~~Trinity EMS scheduling is now private; scraper needs updating #177~~ Trinity EMS scheduling is now private; scraper needs updating Mar 25, 2021

livgust requested changes Mar 26, 2021

View reviewed changes

livgust requested changes Mar 30, 2021

View reviewed changes

pfurbacher added 2 commits March 30, 2021 16:18

Merge remote-tracking branch 'origin' into Trinity-EMS

703ec49

Merge branch 'master' of https://github.com/livgust/covid-vaccine-scr…

fd67d30

…apers into Trinity-EMS

pfurbacher added 2 commits March 30, 2021 16:39

Fixing a weird commenting out of relevant code

22e9e9e

Have no idea what happened here.

Repairing bad commit

b2bebe6

livgust approved these changes Mar 31, 2021

View reviewed changes

livgust mentioned this pull request Apr 5, 2021

Trinity EMS scheduling is now private; scraper needs updating #177

Open

iann approved these changes Apr 9, 2021

View reviewed changes

livgust merged commit 2f91bf8 into livgust:master Apr 9, 2021

pfurbacher deleted the Trinity-EMS branch April 19, 2021 12:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trinity EMS scheduling is now private; scraper needs updating #178

Trinity EMS scheduling is now private; scraper needs updating #178

pfurbacher commented Mar 24, 2021

pfurbacher commented Mar 24, 2021

livgust commented Mar 25, 2021

pfurbacher commented Mar 25, 2021

livgust commented Mar 26, 2021

livgust left a comment

livgust Mar 26, 2021

pfurbacher Mar 26, 2021 •

edited

Loading

livgust Mar 30, 2021

livgust Mar 30, 2021

tkjones117 commented Mar 30, 2021

iann left a comment

Trinity EMS scheduling is now private; scraper needs updating #178

Trinity EMS scheduling is now private; scraper needs updating #178

Conversation

pfurbacher commented Mar 24, 2021

pfurbacher commented Mar 24, 2021

livgust commented Mar 25, 2021

pfurbacher commented Mar 25, 2021

livgust commented Mar 26, 2021

livgust left a comment

Choose a reason for hiding this comment

livgust Mar 26, 2021

Choose a reason for hiding this comment

pfurbacher Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

livgust Mar 30, 2021

Choose a reason for hiding this comment

livgust Mar 30, 2021

Choose a reason for hiding this comment

tkjones117 commented Mar 30, 2021

iann left a comment

Choose a reason for hiding this comment

pfurbacher Mar 26, 2021 •

edited

Loading