Merge Edge into Stable to upgrade PROD to latest #1871
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* update gtag usage * ignore gtag if GA value is blank
* docker local postgres * fix Constants capitalization bug in participationview * add perspective to example.env * add google apis to package.json * add perspective api key to config * config prettier * import google * prettier server.ts * fix typescript bluebird promise to enable async await * refactor post_comment route async, remove paths * perspective call * jigsaw toxicity threshold under flag. * text flag for toxic * add perspective to privacy policy * jigsaw TOS * Better type for getPca. Also removed promotion of error to the return value. Let the error propagate as an error. * Tell TypeScript to use a less ancient library target. * Trim trailing whitespace. * Add data export endpoint. This is a simple first pass which just reads the data from the database and delivers it. No caching, no fancy business. The endpoints are based on the report identifier and provide three separate .csv exports. For example: /api/v3/reportExport/r6ke7cdzte2jrsxctfyt9/summary.csv /api/v3/reportExport/r6ke7cdzte2jrsxctfyt9/comments.csv /api/v3/reportExport/r6ke7cdzte2jrsxctfyt9/votes.csv The format is made to match that of the old command line exporter as much as possible. * smaller font size * Use the correct column name here. * Set text/csv content type on responses. * export fonts from globals * data export info in report view * prettier app.js * move overview down * shorten toxic text --------- Co-authored-by: Colin Megill <colinmegill@gmail.com>
* prettier server, ts error 25 * ts errors 21 * ts errors 18 * type pid * ts errors 11 * modernize create xid function * ignore ts error on jigsaw * type uid * expand pca cache item * 8 errors. * modernize switchToUser function * void * PcaCacheItem types * missing args, null * ignore request.get error * tsignore .get on headers
* update github actions * typo * use docker compose in place of docker-compose * lint fix
* utilize docker compose --profile postgres, and POSTGRES_DOCKER var, to enable/disable using dockerized postgres * update configuration.md
* add comment in file-server/Dockerfile * switch back to url-health-check-action
…ailing Docker build in some environments.
* Switch to non-native Postgres client. And add a "streaming" API for making database queries, which streams the results from the database to Node as they are generated by Postgres. This allows Node to process the rows one by one (and garbage collect in between), which is much easier on the VM when we need to do big queries that summarize data (or just format it and incrementally spit it out an HTTP response). * Mostly refactoring. This moves the handle_GET_reportExport route into its own file, which necessitated refactoring some other things (zinvite and pca) out of server.ts as well. Chipping away at the monolith. This also converts the votes.csv report to use the streaming query from Postgres, which is mostly a smoke test. It seems to work, so next I'll convert it to stream the results incrementally to the HTTP response as well. * Split each report into separate function. * Count up comment votes in single pass over votes table. There was actually a bug in the old SQL that aggregated votes from _all_ conversations instead of just the conversation in question, which is why it took 30 seconds to run. With that bug fixed, even the super slow "do a full subquery for each comment row" was actually quite fast. But this is way cheaper/faster. * Add participant-votes.csv export. * Switch to non-native Postgres client. And add a "streaming" API for making database queries, which streams the results from the database to Node as they are generated by Postgres. This allows Node to process the rows one by one (and garbage collect in between), which is much easier on the VM when we need to do big queries that summarize data (or just format it and incrementally spit it out an HTTP response). * Mostly refactoring. This moves the handle_GET_reportExport route into its own file, which necessitated refactoring some other things (zinvite and pca) out of server.ts as well. Chipping away at the monolith. This also converts the votes.csv report to use the streaming query from Postgres, which is mostly a smoke test. It seems to work, so next I'll convert it to stream the results incrementally to the HTTP response as well. * Split each report into separate function. * Count up comment votes in single pass over votes table. There was actually a bug in the old SQL that aggregated votes from _all_ conversations instead of just the conversation in question, which is why it took 30 seconds to run. With that bug fixed, even the super slow "do a full subquery for each comment row" was actually quite fast. But this is way cheaper/faster. * Add participant-votes.csv export. * Flip vote polarity. In the raw votes table, -1 means agree and 1 means disagree, so we need to count things correctly. And when exporting votes in participant votes, we flip the sign so that 1 means agree and -1 means disagree. * Properly escape comment text. * add votes matrix, show data license preprod, logging. --------- Co-authored-by: Michael Bayne <mdb@samskivert.com>
This reverts commit 61d2940.
* Switch to non-native Postgres client. And add a "streaming" API for making database queries, which streams the results from the database to Node as they are generated by Postgres. This allows Node to process the rows one by one (and garbage collect in between), which is much easier on the VM when we need to do big queries that summarize data (or just format it and incrementally spit it out an HTTP response). * Mostly refactoring. This moves the handle_GET_reportExport route into its own file, which necessitated refactoring some other things (zinvite and pca) out of server.ts as well. Chipping away at the monolith. This also converts the votes.csv report to use the streaming query from Postgres, which is mostly a smoke test. It seems to work, so next I'll convert it to stream the results incrementally to the HTTP response as well. * Split each report into separate function. * Count up comment votes in single pass over votes table. There was actually a bug in the old SQL that aggregated votes from _all_ conversations instead of just the conversation in question, which is why it took 30 seconds to run. With that bug fixed, even the super slow "do a full subquery for each comment row" was actually quite fast. But this is way cheaper/faster. * Add participant-votes.csv export. * Switch to non-native Postgres client. And add a "streaming" API for making database queries, which streams the results from the database to Node as they are generated by Postgres. This allows Node to process the rows one by one (and garbage collect in between), which is much easier on the VM when we need to do big queries that summarize data (or just format it and incrementally spit it out an HTTP response). * Mostly refactoring. This moves the handle_GET_reportExport route into its own file, which necessitated refactoring some other things (zinvite and pca) out of server.ts as well. Chipping away at the monolith. This also converts the votes.csv report to use the streaming query from Postgres, which is mostly a smoke test. It seems to work, so next I'll convert it to stream the results incrementally to the HTTP response as well. * Split each report into separate function. * Count up comment votes in single pass over votes table. There was actually a bug in the old SQL that aggregated votes from _all_ conversations instead of just the conversation in question, which is why it took 30 seconds to run. With that bug fixed, even the super slow "do a full subquery for each comment row" was actually quite fast. But this is way cheaper/faster. * Add participant-votes.csv export. * Flip vote polarity. In the raw votes table, -1 means agree and 1 means disagree, so we need to count things correctly. And when exporting votes in participant votes, we flip the sign so that 1 means agree and -1 means disagree. * Properly escape comment text. * add votes matrix, show data license preprod, logging. * cleaned up pg-query; re-establish ssl db connection --------- Co-authored-by: Michael Bayne <mdb@samskivert.com> Co-authored-by: Colin Megill <colinmegill@gmail.com>
…tion (#1835) * remove maxmind and its references * remove unused geolocation_cache * remove/drop unused table and fields from db * clean up typings in config.ts * use APPLICATION_NAME as a flag for nonstandard db fields
* only use jigsaw API if key is provided; replace `console` with `logger` * npm run format
add port collision instructions to readme
add sign up link to sign in page
|
I'm unconcerned with bundlewatch, or docker image builds |
tevko
approved these changes
Jan 15, 2025
|
Cypress Test Suite passes locally on this merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.