-
Notifications
You must be signed in to change notification settings - Fork 3
Release Notes
jdcl32 edited this page May 30, 2017
·
97 revisions
Fixed long Weight list issue.
- New histograms implementation
- Reusable (Crunch) data visualization components
- Various Bug fixes
Fixed Notebooks link URL.
Now when searching the result's not found message is not visible for a second or two before showing results.
Now is possible to introduce branding colors and minimum base size when building a crunchbox.
Project list no longer overlaps the add button.
Allow decimal controls for variable cards even if counts is selected. Also, in datetime variables we were showing decimals on the counts, this has been fixed.
Fixed the way access time is transferred on our copy_from api call.
On the backend, our tasks are more easily tracked by dataset. This gives our developers even moar power.
Fixed an authentication bug.
Made some fixes to the way we send data to our long-term storage to reduce their throttling. (This should make datasets load faster system-wide)
When clicking in the 'What's new' notification, the link will now open in a new tab - taking you away from the app didn't seem very friendly.
Clicking the x next to a variable when creating a multitable template now removes the thing you clicked, not just the first variable in the template.
In array variable properties, we now always show unweighted counts and missing categories in subvariables.
Cleanup of editing variable categories.
Cleanup of the project members panel
If there are no slides in a deck, the button is now disabled.
All valid/missing/count values on variable cards are now correctly unweighted.
If a user has dashboard_url set, they will see the new dashboard icon at the top of the interface to access that dashboard.
Exporting a single multitable now correctly includes column totals.
Several changes and updates to get ready to start making feature announcements in the app.
Multitable dropzones now are the correct widths and opacity.
Multiple Response variables for which all categories are missing now show empty bars rather than full ones.
A boatload of work to allow Crunch to handle Multiple Response Variables with different base sizes for the subvariables.
In the project list, the + at the bottom is now in a button instead of just kind of floating around lonely.
Variable properties now use updated Personal/Shared terminology instead of Public/Private.
Tables were losing alignment when the number of decimal places were changed. Now they don't.
When row percentaging, exported tabbook bottom row is now labeled "All" instead of "N" since it contains both counts and percentages.
We're now showing the pointer cursor in browsers that don't support grab coughIE*cough.
Cleaned up the user settings panel.
Text variables now can create a "sentiment" analysis variable that automatically classifies responses into Positive/Negative/Neutral. Feature flagged for the time being.
Renaming a deck now immediately updates the deck dropdown with the new name.
Numerous improvements to internal error reporting. You may not see them, but it makes it easier for us to spot problems and fix them before they affect anyone.
On append some derived variables were getting treated as (and in fact converted to) not derived, meaning they weren't updating correctly. This behavior is now fixed.
Filter and weight builders now respect the decimal-places display setting.
Documented the updated Users/Permissions interfaces.
Tab books now available for all users.
When exporting a single multitable, weighted counts are now rounded to nearest integer.
The timeout for appends was sometimes being set too low. Since large appends can take a long time, we now make sure to wait an appropriate amount of time.
Fixed the layout on the account management interface.
Updated superadmin interface to fix deck copying and to improve the diagnose function.
As much as we'd like to live in a world where there are no errors, sometimes there are errors. Now we have a centralized place to report and track them.
Fixed an issue that could cause deck exports to be slooooow. Now they are so fast they've probably finished before you read this note.
Long category IDs were not being handled well. Now your categories ID can be as long as you like.
Added some tools to help us resolve data issues related to appends.
The import dataset interface is one of the oldest parts of the Crunch interface and was a bit behind the times in terms of styling. It's been given a makeover and put into a nice sliding panel to more closely fit our style.
When making an append in R, it is now supported to append only a subset of variables.
Once you could drag search results to the workspace to make analyses, filters, etc. Then that capability went away. We are pleased to announce it has returned.
When failing to append a dataset, we now show a more useful error message.
Documented that editing .R files restarts the Shiny app in the crunchy wiki.
The Export dropdown in Multitables is now caught up to other dropdowns in the filter bar, style-wise.
The Append UI got some much needed styling updates and now correctly shows progress and lets the user know when there has been a failure.
The recently added Users tab was showing up for viewers (but when they clicked it, they weren't actually able to open it).
If you export a tab book with significance testing on, the significance shading will now appear in the Excel spreadsheet as well. What you see is now more closely related to what you get (WYSINMCRTWYG).
When removing a filter, the top half of the 'x' was not clickable. You now have the full icon to click. Also, the icon was replaced with a fully formed 'x-in-circle' so things don't occasionally look one pixel off.
- Forking a dataset and changing public multitables and filters in the fork could result in a failure to merge, which it no longer does.
- We have changed the way we cache data on our processing servers so that we can keep more datasets "hot". This should reduce the number of times that we have to fetch data from our long term storage, which can cause long delays in displaying results.
- Exporting tabbooks with three or more variables will now display the column headers of multiple response variables correctly.
- Fixed a bug with "sig testing" on a categorical array crossed with another categorical array.
- A variety of internal changes to help reduce errors and provide more visibility in search indexing
- First time notebook users will no longer receive a 500 error and need to reload the page
- More ZZ9 instances are now available to help reduce errors during operations
- The profiles team will now be able to define custom icons for the deck.
- Tab books will now respect the columns you hid in the corresponding multitable. After all, you went to the trouble to hide them.
- Tab books now include an unweighted base size row, a frequently requested addition to this feature.
- Added API documentation for new public CrunchBox endpoints
- Variables created with a single value have learned to respect exclusion filters.
- Variable properties for some categorical arrays had a vast field of blank screen, which is now properly filled in with properties.
- Three-way tables with “tabs” containing missing categories followed by valid ones showed the wrong data for the latter tabs; now the correctly skip the missing one and show the right thing.
- The dataset search service in the web app was made more configurable thanks to a contribution from the YouGov Profiles team.
- The CrunchBox share icon has been beefed up to be easily spotted and more clickable than ever.
- Dataset search has been given some configuration options for those of you who like pushing all the buttons and turning all the knobs.
- Some weight variables were being rejected based on unexpected characters in the name. This has been fixed. Weight variables are now free to express their individuality as they please.
- The option "viewers can share" for dataset sharing has been relocated to the Permissions preferences view. All dataset sharing preferences are gathered together in one place for easy access.
- Added an admin tool to clone a dataset to our API
- Reverting a dataset no longer potentially causes duplicated data
CrunchBoxes now have a variety of new and improved features:
- Improved the CrunchBox builder to allow previews of the newly created CrunchBox
- New ways to easily share the data with a variety of social networks to allow more user interaction with the data
- Social media posts will now provide a rich preview of the content to increase user engagement
- Fixed: In the share modal, the "Share" button is enabled even if the user hasn't specified at least one user to share with.
- Fixed: Changing weight now properly updates numeric and date-time variable cards
- Fixed: Significance (p values) now correctly calculated after making modifications to variable cards.
- Fixed: When category names were long, the categories editor layout was broken.
- Fixed: Unregistered emails that contained number characters couldn't be entered in the share dialog box.
- Fixed: The "viewers can share" appeared as "NO" briefly even if the value was "Yes"
- Fixed: Dragging variables in IE now works as intended.
- Fixed: Locked 🔒 and large datasets became unusable — even for looking at! — for all users after editing even their own personal variables. Now, other users can still go about their business and even the user who edited something can work with the rest of the data
- Shared multitables will now continue to function even when you add new data
- Creating new personal derived variables could cause timeouts and or slowness. Process has been given a stern talking to and will now complete tasks faster.
- Added header row options for CSV export of datasets, no longer will you have to guess what a column refers to!
- CrunchBox data now includes creation timestamps for easier organizational manipulatory fiddling.
- Fixed: Variables with duplicate categories keys now throw appropriate 400 messages.
- Consistent defaults for margin and proptable: Fixed a potential issue with the frontend displaying the wrong percentages for weighted counts. A retrospective document is available, please contact support@crunch.io for details
- Unless the user is the current editor of the dataset, we no longer notify about editor changes
- Creating an array without values will no longer raise a validation error
- A new API was introduced to delete Crunchboxes for those users that like to create and iterate and had a long list
- Inviting an existing user would email them without providing an easy to click link to sign-in, users now get a speedy link back to Crunch
- the JSON tabbook export API now exports the same data used internally to generate Excel spreadsheets, allowing API consumers to easily get access to the same data and create their own exported data
- UI and UX fixes that hopefully provide a more unified and cleaner experience for Crunch users
- Hidden columns in multitable builder and editor now stay in their place. Hide and seek champions!
- Tabbook exports may now have a Table of Contents sheet that link directly to the tab with the relevant data. No longer do you have to hunt for the right Excel sheet, click and off you go! Also has fantastic support for single sheet exports!
- Canceling a change in the user settings panel will now do the correct thing: Cancel your changes.
- Force SSL for Google Analytics, saving a round-trip
- Bugfixes for handling of a user's dataset preferences
- Indexing for search was improved.
- More search ranking and indexing improvements (follow-ups from the 3.5.0 changes).
- Parts of the application where users need to fulfill multiple steps to continue — such as creating new filters or building weights or categorical variables — now prompt with more consistent (and colorful) messages.
- When exporting data to spss or csv, users can now specify a value to use for missing data.
- Tables and multitables in the web app now show unweighted base rows/columns as appropriate, even when displaying weighted percentages. (The option to go back and show weighted Ns will be implemented in the future.)
- It is now possible to set a scalar (single constant) value when creating a new variable.
- Stray user dataset preferences and user variable orders found malingering in the database have been cleaned up.
- Fixed: dataset viewers (those who cannot edit the dataset) no longer have an ineffective “account settings” option since they do not control their accounts.
- Fixed: on large datasets (like Profiles), users too often saw messages about the dataset having changed when it had not.
- Fixed: the ‘Viewers can share’ option for datasets now works as described.
- When sharing a crunchbox, it is now possible to include a persistent referrer.
- The service behind notebooks and dashboards in the app was down and is now up.
- The database is now more intelligently aware of free disk space in its environment.
- Recent performance work has eliminated most causes of timeouts when loading large datasets and some catalogs of datasets such as
joins/. - We’ve also squeezed major speed improvements out of the loading of datasets in the web app.
- Users switching between hosts (such as between app and profiles) could have datasets say they could not be opened; now they open as expected.
- The ‘tabs’ on three-way tables in the app received a design update.
- Editing multiple response variables sometimes claimed that changes “could not be saved” for a pretty silly reason; they can now be saved.
This release is a major upgrade to our search infrastructure and improves performance of searches, reliability of the search index contents, and quality of the results returned, specifically:
- performance and permissions improvements based on dataset-variable relationship addition
- better underscore matching
- better relevance for whole-word matching
- better phrase matching using quotes
- dataset name has higher relevance
- Stray dataset snapshots without correct version tags were removed.
- A number of small visual elements in the app got updates or details polished.
- Multitables may now be edited again
- Include/exclude work for multiple response variables
- Filtering a query on
variable == other variablehas been fixed - Weight variables, formerly a
catalog, are now aShoji:order
- the data store now has a separate cleaning process
- Display no longer “goes nuts” when a text variable with long responses is used as the columns in a crosstab.
- In tab books, row marginals are now being shown when using row percentages.
- Properly handling the original referrer when sharing a Crunchbox.
- Now including hostname in streaming error email subject, as requested by users who stream datasets to more than one host.
- Fixed: Some actions might not be registered in workers at the time a play workflow happens.
- Variable organizer has been fixed to re-align buttons as necessary. Button names have also been improved, making it easier to know which one you'd like to activate.
- A number of details that were wrong and quite distracting to our graphic designer have been made right and pleasing unto her.
- Fixed: if a user deleted a variable used in a multitable definition, or changed its type, the multitable would fail to load in a very opaque (as in blank white screen) sort of way. Now invalid definitions are removed.
- Tab books and deck exports have regained some formatting and ‘frozen panes’ that briefly they had lost. Like when your luggage is lost but then they bring it to your hotel the next day.
- Subtle stylistic refinements to a few higher-visibility interface elements for clarity and balance. These elements now pop without being "in your face".
- Migrated from actions to revisions, because moving data around is what we do!
- Loading very large datasets no longer times out in some more cases where it did before.
- New users who are not members of any ‘projects’ are now greeted by the application and their personal project, rather than a stream of console errors. We thought that would be a little more welcoming.
- Now when you are prompted for a name after saving a filter, you can just press enter after entering it.
- Users had asked for some way to save all applied filters into a single filter, so we’ve added an option to the filter menu to do that by clicking “Save Applied Filters…”
- Non-uniform multiple response variables did not display an ‘All’ margin in the app. Now they do.
- The app’s interface to share a dataset got a style update 💃
- Fixed: an issue with csv exports and hidden variables
- Fixed: it had become impossible to make many kinds of changes to multiple response and array variables in the app, due to an api that needed to be changed. So, we changed it, and you can now edit and change the order of those variables again!
- Tab book excel exports now contain the marginal count (base) in their direction of percentaging.
- When dataset definitions contain derivations, they were ignoring the descriptions. Now the description is correctly set at creation time.
- Multiple response variables with complex missingness patterns (non-uniform; varying per-item) can now be used in 3-way tables in the app.
- When exporting data to SPSS .sav format, our date/time variables apparently set a different formatting setting than SPSS users expected; we have brought it into line with their requests. 📆 📅 🗓
- We made the box where you type emails to share with a little friendlier. 📧
- Fixed: Streaming data automatic appends (batch streams) timed out more than it should, so we gave it a little more time to complete.
- Fixed:
copy_fromfor filters, analyses, and other dataset effects could time out on large datasets; it will now complete its work asynchronously. - Documentation about multitables has been improved.
- In the app sometimes the escape key didn’t dismiss a panel; now it more often does. ⌨️
- Multitables with long variable names sometimes behaved badly in the app, causing row labels not to line up with the table. We’ve applied some constraints and judicious use of a ruler. 📏
- Likewise, long row labels in multitables with hypothesis testing enabled caused some unfortunate side effects that have now gone away.
- Fixed: when organizing filters in the app, clicking ‘save’ did not save. Now they can be saved.
- Fixed: when changing the weight in the app, multitables did not refresh. Now they do.
- Fixed: certain toggles in the app thought it was opposite day and did the reverse of what they said they were set to. Since opposite day is not a thing, they now do what they say.
- Fixed: the display controls fade out again.
- SPSS .sav exports of categorical data by ids set their print format to have 0 decimal places.
- Likewise, SPSS exports of missing values for date/time and text variables should now better reflect what users expect.
- When creatin multitables in the app, the ‘hide columns’ transformation works better than it did.
- Certain appends timed out when they should not have done. Now they append!
- The footnote in tables with multiple response variables (that they “do not sum to 100%”) has returned!
- The tooltip about “save” in card view is less confusing, we hope. It’s sort of like a bookmark, restoring the view with that card in it.
- Fixed: variable cards for text variables could not be loaded because their ‘summary’ returned an error. Now it returns a summary.
- Fixed: when selecting variables to export or use in a tab book, groups that were all selected didn’t show a check. ☑️
- Fixed: combined variables’ properties had a stray word ‘items’ hanging out below their categories. No more.
- Another iteration of
copy_fromfixes adapting to users’ experience, well, copying_from. - Fixed: SPSS .sav export of numeric variables did not indicate missing values appropriately. Now it does.
- The legend for statistical significance now appears in multitables. Also, the alignment of base rows now respects hidden columns where it did not before.
- Fixed: “Use as weight” on numeric variables did not set the variable as a weight, which is required before using it as one.
- Fixed: manipulating tables and graphs more often works as expected — for example when you remove a column variable, it is removed.
- Fixed: When organizing variables and not showing hidden variables, the other ones didn’t show up right away because they were paginated beyond the hidden ones. The hidden ones no longer take up that space.
- Multitables in the app now contain a row reporting the N in the direction of percentaging, as do other tables in the app.
- Multiple response variables with varying-by-item missingness now perform on-axis margin and percentages correctly (that is: a row multiple-response by a column categorical will compute the percentage within each item across categories that sum to 100%).
- Sometimes, indexing could cause a Query Timeout when it really should not be able to. Now it cannot.
- A few improvements to search indexing; more to come. 🔍
- Making filters just got easier, by suggesting a name and prompting editing or confirmation when you click save.
- Fixed: a bug in extending datasets (material right join) regarding categorical data. Also, another failure on join keys that will never happen again.
- Fixed: selecting “is all of” in the filter builder did not set the option; in fact, it just sat there. Now it makes the filter you want.
- Fixed: Also in the filter builder, range selectors weren’t doing the right thing. Now the show the range selected.
- Fixed: an obscure append bug relating to combined categories in certain variables with few unique values, that gave a very cryptic message to users regarding “indices” and “integer or boolean type”. No more.
- The “Deck” to save analyses now appears in card view in the app. “Saving” a variable card will load card view with that variable card leftmost, and export a univariate table of it.
- A new ‘selections’ function allows a dimension to request to ‘collapse’ categories of a multiple response or array variable into 3: ‘Selected’, ‘Not selected’, and ‘Missing’ (with fixed ids
1,0, and-1respectively). Support for variables that require this function will develop over time throughout the application and other components. - Tab books can now produce a single page of output, rather than one row variable per sheet.
- Tab book data can also be retrieved as JSON rather than just the formatted xlsx, so that other clients than the crunch exporter might consume computed multitables.
- Deleting datasets and versions thereof could leaving some residue, which such actions cannot anymore. 🚮
- Streaming datasets can now create variables with metadata without simultaneously providing values.
- It is now possible to choose in SPSS .sav exports whether to use the variable’s
nameordescriptionas the SPSSvarlabelwith optionvar_label_field. - Various improvements to streaming should make it more reliable and stable.
- Fixed: the decimal place control in the web app sometimes would be visible but disabled, when clearly it would make sense to have more decimal places. It is now enabled/disabled in a much more intuitive manner.
- Fixed: when changing aliases, they appeared not to refresh in the app.
- Fixed: the target of a merge did not have its
last_modifiedfield updated so it appeared older than it truly was. - Fixed:
copy_fromcould set an invalid weight variable in the new dataset. It now maps the weight variable by alias, as it always claimed to do.
- Fixed: the order of saved analyses in a deck can be changed again.
- Fixed: Certain datasets with a very ancient history 📜 could not merge because their ancestors could not be located. Their ancestors weren’t needed for merging, so the system now leaves them in peace.
- Fixed: search results in card view could behave strangely. Now, clicking and dragging them works as expected.
- Working with multiple response variables could be confusing, with drop zones doing not quite what they should have done. They should now work better.
- It is no longer possible to change both the
subvariablesandsubreferencesof an array or multiple response variable, because the correct order of operations is not defined. - Due to @nealrichardson’s laserlike focus on a cURL bug in R, requests that were slow are now fast — for more see https://github.com/jeroenooms/curl/issues/79.
- A draft of a UI to create CrunchBoxes is now available for testing by select users.
- Slides having on-the-fly filters are now reloadable, as they should always have been.
- Tab books now export (1) only the variables selected in the UI; (2) in the correct order.
- fixed: a nasty iteration-mutation bug where some dataset or tab book exports with lots of variables selected would fail.
- fixed: searching for datasets in the datasets list produced strange indentation of some results. Oops! We were using a jagged ruler so we found a straight one.
This deployment includes a lot of bug fixes throughout the system.
- fixed: search index errors produced a lot of excess email. Now it only produces a lot; the excess has been scaled back.
- fixed: categorical arrays in tab books incorrectly set cell number formatting as if it were percentages. Now it only does that when they are actually percentages.
- fixed: decks could have extra slides, sometimes from the wrong dataset, mysteriously show up. We dumped everything out on the floor, sorted it into piles, then put it back in the deck drawers in correct drawer organizers.
- fixed: when creating new personal variables, such as by combining categories, they showed up in an unexpected place in the order. We found a better home for them.
- fixed: the multiple-conditions “new categorical variable” builder was not rendering the counts for categories correctly as you built it.
- fixed: some filters in some datasets would cause bad requests because they incorrectly had all-caps “AND” and “OR” in their definitions. They have been changed to lower case and now load.
- fixed: the
table/endpoint sometimes timed out because it was gathering far too much information before responding with only some of it. - fixed: in export and tab books, editors can now always include hidden variables if they would like to (they are shown greyed out in the app).
- fixed: when crunchboxes are very, very small, clicking anywhere will cause them to open in a full-screen embed window so users can actually interact with them.
- fixed: crosstabs were supposed to have fixed header rows and columns (labels), but sometimes they didn’t. They now sit and stay, like good dogs.
- fixed: crunchbox stacked bars with values >100% because of premature rounding.
- We thank the YouGov Profiles team for their work on customizing display settings in the web app and are pleased to accept their pull request.
- the parameters to a crunchbox now include display settings, to make sharing and embedding individual analyses work better.
- 🗺️ crunchbox geodata is now included without having to load the whole variables catalog
- Crunchboxes should load faster by prioritizing the analyses and doing some other stuff in the background.
- fixed: the
table/endpoint included some personal variables when it should not have. - fixed: combining categories and editing them later didn’t work right in the app. Now you can arrange and rearrange at will.
- fixed: a longstanding bug in the web app where showing the mean of a variable and then clicking the x to remove it would lead to a blank workspace.
betais dead, long liveapp!- To boost performace, some arguments are pre-evaluated some of the time.
- For extremely small displays, variable descriptions (usually question wording) is now suppressed in crunchboxes.
- In the web app, card view for categorical arrays now shows a range of both valid and missing, because each response (row) may be different.
- Documentation at support.crunch.io now includes the “Permissions” tab which lets editors set a default weight, as well as control whether viewers (clients) can change the weight, and whether they can export data.
- fixed: numeric variable cards included missing values in their valid count, which is contrary to the whole concept of missingness.
- fixed: the list of datasets managed to swap the “start date” and “description” columns.
- The database can now load alternate versions of data than
master__tip. It’s like the upside-down in Stranger Things, except without many-toothèd monsters (we hope). - fixed: some columns of
textdata could only hold tens of thousands of values. Now text columns support about a million unique values. - fixed: editing array variables could cause the names to disappear and then show “Missing”. They have been found.
- Now, when datasets already include a weight variable, the alias of the weight variabe can be specified in the create-dataset request. See docs.crunch.io for details.
- Slides in the deck now expose all of their analyses, and are capable of loading them in the web app.
- fixed: sometimes appends would fail without a reason. Now they either succeed or are kind enough to explain the problem better.
- fixed: the name of the file for tab books was made for robots. Now it is made for humans.
- link sharing in crunchbox
- better response time for streaming requests
- pagination for univariate barplots in crunchbox
- notes for crunchbox
- cr-select for maps in crunchbox
- fixed: ‘viewer’ members of projects couldn’t export csvs of their datasets. Now they can!
- fixed: derivations of some multiple-response and array variables could not be displayed. Now they can. This also allows new data to be appended to the same datasets.
- cleanup of some stale bits in preparation for versioning improvements
- merge changes from parent into a fork
- maps for census region now available
- spss export bug of subvars with different missing data
- spss export lowercasing
- some multiple items for dataset orders
- post to stream is now faster, streaming requests too
- decks could be saved to wrong dataset
- 409 conflict advises force true
- excluding missing in cat arrays could clobber names
- crunchbox crosstabs could cell percentage
- beta wouldn't load on ie
- precompute cubes for mr and filters
- refinements of crunchbox tables
- variablegeodata blocked savepoints
- r for managing ds viewer permissions (export, share), improvements to api also
- build for web app improvements
- better projects support help pages
- reporting to users about batchstreams
- export including hidden and personal variables
- csv export bugfix, list index out of range
- filters now support grouping
- localized number formatting!
- editing variable categories could 500
- deployment batchstreams
- improvements to streaming instruments
- savepoints via an oplog
- refinements to display of crunchbox graphs
- documentation now available for MR views api
- renaming of subvariables in a single request (validation relax)
- variable.delete could leave some things behind
- creating public widget fails at writing the variables catalog
- cursor drag thing when dragging datasets
- variable.bind is not replayable
- migrating while appending errors
- insert new subvar fails trying to pad existing rows
- you can relinquish ownership to a project even if not an editor of proj
- move dataset out of project
- 'duplicated' function
- appending bug in batches align extra base array
- timestamp of saved version
- copy things between datasets
- really fix broken arrays
- too many requests to feature flag from whaam
- force merge discards savepoints
- whaam import of new dataset should provide savepoint=false
- more deployment love
- pk set on creation
- creating ds with vars with geodata was broken
- sharable links could do the wrong thing in the app
- everyone can see them now
- While the api provides several equivalent ways to refer to missing values (specifically:
{function: "==", args: [{variable: id}, {value: 1}]}and{value: 1}), those (totally the same) misisng rules did not like to be combined and append together. Now they append. - In some cases a project could not be found when it was totally right there. Project invisibility cloaks have been confiscated.
- Likewise, we have improved the experience surrounding the “current editor” of a dataset when owned by a project, and made it smoother to import a dataset directly to a project.
- It is now possible to append a subset of variables from one dataset to another (via api; R functions and web app ui to come later).
- The “build multiple response” part of the web app now looks better, we think.
- Fixed: in the web app if an array or a third “tabs” variable was present, sometimes graphs did not update when a new tab was selected. They do now.
- Fixed: a migration produced a cryptic KeyError. No more.
- Fixed: a migration produced a cryptic ‘errno2 file not found’ error, also, for files that it either shouldn’t have been looking for or which were not actually missing.
- Removed: R package support for <3.0-style arrays. All datasets are expected to be valid datasets.
- New tools for debugging actions applied to specific variables.
- The
slide/analysisapi now supports multiple analyses per slide, and the ability to manipulate analyses on slides. - Project icons in unsupported file formats failed silently. Now they are rejected. Never fear, though — although .mov remains an invalid file format, animated gifs are supported. Windows paint .bmp files have never been and will never be valid files.
- Further refinements to the CrunchBox sharable results tool.
- Fixed: some oddities for editors working with datasets owned by projects.
- Fixed: Slides on shared decks with analyses referring to personal (non-public) filters failed to load. Now they load.
- Now, just like when merging,
appendoffers anautorollbackoptions which creates a version and restores it in the event of a failure. In general, users are advised to use the default (true). - It is now possible to specify an owner preëmptively when creating a dataset.
- When
extending a dataset, you can now filter the rows on the right-side dataset prior to matching on the key. - Appending the same dataset multiple times to a target works.
- A bug could fail to handle cube data in the web app because of a silly floating-point error. Even the most elegant of abstractions leaks sometimes.
- Downstream developers on the web app platform will now find Angular exposed in the module build time, so it can be
required at will. - Fixed: a bug when appending derived subvariables caused the whole append to fail. Now it derives and appends them.
- Fixed: failure to export a subset of a dataset when the subset does not include the variable on which a filter or exclusion was based. Now the dataset exports the rows, respecting the filter and not including its variables.
- Streaming datasets subject to ‘auto-batching’ are ‘batched’ (appended to target datasets) more often than they were before. More is better, right?
- New api features to generate summary tables, toplines, and crosstabs of entire datasets.
- New tools to provide multiple response ‘views’ (mentioned below).
- New tools to help us respond when users report search not working for them. Search should work.
- Fixed: some datasets could not export a tab book because they had subvariables with no valid data.
- Fixed: several types of errors about ‘missing rules’ and ‘missing reasons’ that were difficult for users to manage. Now, under most circumstances, missing rules are combined between datasets when appending.
- Fixed: reverting to a savepoint following an append left the reverted-out batch in an ambiguous and incorrect state.
- We added tools to debug how variables (especially array subvariables) are matched when appending.
- Previously, when loading a saved analysis containing just a subvariable as one of the dimensions, the app tried to load the whole array, which wasn’t the thing that was saved. It now loads the right thing.
- When deleting arrays in the Organizer of the web app, they ended up leaving bits of goo, er, subvariables, on the dataset. Now they are mopped up.
- Likewise, when deleting personal array variables could leave goo on the variable order that shouldn’t have been there. It is now squeegeed afterward.
- Some users had reported trouble editing subvariable names in the app. Subvariables have just as much right to a name change as the rest of us.
- When appending, some interaction of personal variables and exclusions caused it to not append. Now it appends.
- Fixed: the totals on text, numeric, and date/time variable cards included the missing values, as if they weren’t missing. The app has received remedial instruction about the meaning of ‘missing’.
- Some users who had been members of the (obsolete)
teamswere not added to the right (nonobsolete)projects. Now they are. - Further steps toward seamless deploys of the work described here, with no interruptions to users.
- When dragging a variable to the filter-bar, pages of 300 categories/responses at a time are now shown instead of less than that. Happy filtering!
- When appending variables, users must now resolve any ambiguity about whether variables belong in an array or not. Before, we were trying to be clever and infer intent where none was evident.
- When appending, some types of data could throw a cryptic
sparse categorical data_aserror. The error will not happen anymore and the data doesasit should. - Some troubleshooting tools for array variables and subvariables, following on the 3.0 release discussed below.
The major version 3.0.0 commemorates a breaking api change with respect to both Categorical Array and Multiple Response variable types. These types share the concept of subvariables, which have shared, common metadata. Previously, the array was a virtual type wrapped around first-class subvariables. With 3.0.0, the “parent” array is the first-class member of the dataset, and it defines categories and subreferences for its subvariables.
Subvariables — “Response #1”, array_sub_1 — had prior to 3.x been represented by //api/datasets/{dataset_id}/variables/{variable_id}/. Now, subvariables exist only as subvariables, that is, //api/datasets/{dataset_id}/variables/{array_id}/subvariables/{subvariable_id}/.
The change is less breaking than that might suggest. Already, subvariables had not been included on the variables catalog or variable orders. But they had first-class variable ids as if they could be. The 3.0 change formalizes their status as subvariables.
The promotion of parent variable definitions will enable “views” of subvariables that would not have been possible without consolidating attributes in the array definitions. For example, it will be possible to query a subset of subvariables, and to define dichotomous (multiple response) selections on different categories, because now the categories of all subvariables will be guaranteed to match in a way that wasn’t enforced before 3.0.
- Like variables, Filters now also support a
viewobject. - There is now better documentation at http://support.crunch.io for decks, user settings (showing aliases and notes), and multitable (tab book) export.
- A new setting lets editors permit or forbid viewers of a dataset from exporting it to csv or SPSS.
- After deleting a variable, the order of variable cards was getting confused as if it was looking for the deleted one. No more.
- The web app can now be built under windows. Progress?
- Some periodic maintenance was conducted on the
dropitcomponent of the web app.
- A new argument to the
mergeapi allows users to insist that they know what they are doing andforcea merge. (Some validation still happens, and you should almost always try to merge with force if a merge fails.) - One of the reasons merges failed was a cryptic error about getting unicode. But, we support unicode throughout, so we no longer have a cryptic error about it when we shouldn’t.
- Documentation for the variable geodata (discussed below) makes it easy to associate with variables at creation time or afterward. There are not yet R helper functions to post :map: things yet.
- Other parts of variables’
viewproperty are less strictly validated so you can store information beyond geodata there. - Fixed: appending with hidden or personal variables could fail. Those are not good reasons, and we no longer fail for them.
- Appending could in some circumstances resurrect references to variables that had been deleted, by rederiving other tones. No more zombies.
- 🌏 🌍 🌎 We are pleased to announce choropleth map support and an API for managing geographic data
- Variables for many regions can now be associated with geographic features (geojson/topojson) and used to plot colored maps in the web app.
- To associate a variable with a map, consult the Geodata API documentation.
- The
viewmember of variable entities is now more permissive in what it will accept and store, so clients may use it to store information about variables beyond what is usually included in a description or notes. - Joining columns from one dataset to another using a common key (exposed as
extendDatasetin R, and aliased tomergeby courtesy) is now asynchronous with the usualprogressreporting. - Fixed: When appending datasets after deleting a bunch of variables, there could be remnants of the deleted ones where there should not have been, and it could interfere with the append. No more.
- Further safeguards around forking and merging should prevent some types of grafting from preventing merges later on.
- We are pleased to announce a new edition of documentation of saved analyses.
- Fixed: further append difficulties with crytpic errors have not only been decoded, but no longer throw any error.
- Fixed: An overflow in reporting results when categorical variables included category ids of the highest id as well as system-missing data.
- Fixed: merging datasets could could fail before actually getting started, if saving a rollback version timed out. No longer the case.
- Fixed: appending no longer reports a cryptic error when one or the other dataset had a column that all had a single value.
- Fixed: multiple response variables that were all missing, when showing empty things, appeared full; we’ve put an end to this illusion.
- Fixed: shared decks with long names spread themselves out on their menu: they are now more tightly contained.
- Fixed: When deleting a user account, buttons appeared haphazardly. They, too, are back in their place.
- The datasets list is now retired. It hasn’t yet died, but its replacement — the project browser — has been installed in its place. A reception will be planned to celebrate.
- The display controls at the bottom of the screen look a little bit nicer.
- As we prepare to make ‘projects’ the default view for all users, we added a link to help next to ‘New project’ in the app. Like this❓
- Users within an organization with single-sign-on who have never used Crunch before previously had to do an extra, confusing step; now that is automatic. ✨
- Fixed: a validation bug when creating datasets.
- Fixed: a merge error for some actions that should have been playable now are.
- Display settings and filters are now respected for tab books — some users wanted row-percentaged tab books and multitables; now they can do that.
- SPSS import will now accept .sav files with a variable and ‘mrset’ having the same name (A unique alias is generated at import), because to crunch, they are both variables.
- Categorical arrays now display a range of valid values, because they often vary widely by row.
- Users can now toggle a deck to ‘public’ to share it with other users of the dataset.
- The ‘No data’ category can be reordered, so when new categories appear as the result of an append, it can still be presented, helpfully, at the end.
- In the event that importing a file fails, we now provide a slightly more helpful message in the app.
- A new materialized join feature allows users to join new columns onto a dataset by a common key. See the crunch R package documentation for more information.
- Much larger datasets can now be exported to csv/spss. A size limit remains in place for spss .sav downloads, but csv is unrestricted. SPSS exports may be quite slow with many ‘text’ variables.
- On the API, decks now have an
is_publicattribute; public decks can be viewed by any user who can view a dataset (like public filters). - When deleting variables, derived variables that depended on the deleted on (suche as combined personal variables) will be ‘integrated’ so they are still available.
- We documented notes on variables.
- Fixed: an empty update to a filter no longer throws an error.
- Fixed: “If you start editing a variable name in the organizer and then end up with the same name, it says it’s invalid”.
- Fixed: reverting to a version with a different owner could cause datasets in projects to become “lost”. 🗺
- Variables now all have a shiny new Notes field. Unclear whether it is a “notebook”(📓) or a “notebook with decorative cover”(📔).
- Users can now choose whether to show these Notes, as well as the variable Alias on variable cards in the app. (The setting is at the user level, so applies to all datasets you view.)
- As mentioned in .46 below, editors can now clash ⚔ among themselves about a bunch of properties. They can access and manipulate most of Dataset Properties without unlocking the dataset first. 🔏
- More information is recorded when merges do not successed, so next time they might.
- Fixed: appending datasets could fail because of a personal variable name. No more.
- Fixed: When editing variable names in the organizer, we wrote with invisible pixels.
- It is now possible to export (csv or sav) a filtered subset of a dataset — rows or columns.
- Other applications built on top of Whaam (the Crunch web app) can now use their own Google Analytics tokens in their own modules.
- Better monitoring of the pesky operations of
mergeandappendfor more successful merges and appends. - When appending, private variables in the incoming dataset will not be included.
- ⚔ Many attributes of a dataset can now be edited without requiring editors to “unlock” a dataset for editing. When editors disagree can resolve their disputes amongst each other.
- Fixed: several incorreclty positioned items in the app, from navigation arrows to the save-analysis button, have been put back in their places.
- Also, check marks in variable properties were too big. They have been resized.
- Fixed: groups full of hidden array and multiple response variables could cause an empty group to appear in the app.
- Fixed: Text variables full of all the same value (such as empty string) could not be summarized. Now that solitary value is dutifully reported.
- Fixed: a bug in handling an older format of missing rules for numeric variables that made them not append. Now they append.
- Datasets with lots of variables in any group (a very long list in the side bar) should be more pleasant to browse. In particular, more are shown at the outset, and when you click “Show more”, they are appended rather than just showing the next window.
- Deleting datasets is faster.
- Fixed: When importing a dataset to a project, it could not be dragged out of the project without refreshing. Now it can.
- You can now show discrete values of numeric variables in multitables.
- We updated the organization and first page of http://support.crunch.io.
- You can now search for variables by the name of their ‘group’ in the order, shown in the sidebar of the app.
- Fixed: large categorical arrays could appear to hang the browser when rendering on variable cards. Now they display, and load progressively as you scroll.
- Fixed: Actions that failed in a forked dataset are not attempted to be replayed when merging that fork.
- Fixed: Datasets that were in projects could not be shared. Now they can.
- Fixed: Categories that were obscured by a filter or exclusion filter could be removed; now they cannot.
- Sometimes when merging a fork, the replay of actions failed for obscure reasons. We have cast some light upon those reasons.
- R users now have more tools for comparing existing and incoming datasets they intend to append.
- The dataset ‘state’ endpoint should be faster, which is good because it is accessed a lot.
- Fixed: References to the catalogs of datasets now are actually the right catalog (personal or project).
- Fixed: When downloading the same tabbook a second or third time, the file could not be opened. Now it can again.
- Fixed: The weight builder in the web app now consistently shows and expects values as percentages, not proportions.
- The web app can show data using colors on a map. Ask us how! 🌐🌎🌍🌏
- When organizing variables, you can now hide/unhide a whole group of variables.
- Now when new users sign in, the app actually redirects to itself instead of only pretending to.
- Fixed: When importing data and canceling, the app no longer flashes glitchy error things — just cancels as requested.
- Fixed: The order of units is now correct when selecting a rollup for date/time variables 📆
- Developers of web app customizations can now specify their own Google Analytics keys in their app manifests.
- Users can now report a problem from the web app if that problem occurs when they’re trying to share the dataset with someone.
- When exporting a dataset, you can now choose how to represent categorical variables, by name or id.
- Back in 2.26.72, we strongly encouraged users to check
progressinstead of the-thing-being-updated. We now follow our own advice in the web app. - You can now rename and delete projects in the web app. Also, edit names of datasets that are part of projects.
- Fixed: Some decks appeared empty because they actually had entries pointing to slides that no longer existed. Now the ghost-slides are invisible again. 👻👻…Or are they?
- Fixed: You can now revert to a point, do stuff, and erase your stuff by reverting more than once to the same point.
- Fixed: Searching by project-folder should find datasets in that project/folder.
- Fixed: Organizing extremely large datasets (changing their
order) sometimes didn’t work because the payload was too large. Now it works. - Fixed: R users could not refresh a dataset after moving it to a project. We have provided datasets with refreshments.
- Fixed: sometimes appends could fail without any notification. Now both hands are used for clapping in a well populated forest. 👏🌲🌳
- The web app now uses AngularJS 1.5! 🎉
- Also, we turned off its default markup annotations, which makes things slightly faster.
- On the api, entities now contain a reference to their parent because some types of entities can have different ones that can’t be inferred from their IRI. In particular, datasets may appear on a user’s personal
/datasets/catalog, or on the catalog of a project. - The alias of a weight variable can now be specified at dataset creation time, if you’re uploading a dataset that already has a weight.
- The component in the web app for choosing whom to share a dataset with has been upgraded.
- Hark! We mentioned that there were multiple decks, and now we added documetation about them too.
- Exporting (to csv or spss) can now be filtered, and limited only to certain variables.
- Fixed: things could off the screen when creating benchmarks, or category-wise comparisons, in the app.
- Requesting team dataset catalogs could be slow because the server was rather frantically making lots more queries internally than it needed to.
- Progress is more smoothly reported and more informative when appending datasets (it won’t just jump from 0 or 2 to 100 anymore).
- Developers of custom web applications can now use fun modern javascript thanks to babeljs. (Crunch frontend developers rejoiced at this 🎉).
- The API now accepts all streams of data regardless of the current editor of the dataset. (Only the current editor can post the streamed rows as a
batch, however). - Now you can filter by range (less than, more than, between) with numeric variables, rather than selecting all the distinct values in a range.
- Fixed: Appending data sometimes didn’t work because the incoming data had very-large text variables. Now those appends work.
- Fixed: Another time appending data didn’t work because category ids did not match what the system expected.
- Project admins can now manage who else can view and edit things in the project (there is now support in the web app for managing project users).
- The web app now has tools for project admins to edit and manage projects (previously only available via api and R)
- Fixed: adding a subvariable to an array or multiple response variable would fail if the dataset had a primary key. But… those things… are unrelated.
- Fixed: the web app sometimes made it seem like a couple of properties could be changed when users could not in fact change them.
- You can now drag slides around in a deck to change their order.
- Now when adding users to a project it should be easier to find the right users to add.
- Fixed: Merging a fork of a dataset, then doing more stuff in the fork and merging again would not reflect the additional stuff. Now it does.
- Fixed: A truly bizarre bug where if you change the order or categories in a variable, then cancel changes, the “Filter by” dropzone shows up.
- Fixed: Derived variables in joined datasets didn’t work. Now they do.
- Fixed: Apparently when new users were invited to a project, they got an email with a silly image attachment rather than a pretty email with the image showing.
- The web application now lets you create, delete, and rename different decks for saving analyses.
- Users can no longer create invalid (undisplayable) variable orders: specifically, objects at the same level of hierarchy (variables or groups within a folder) cannot have the same name.
- The web app now persists the view of datasets so you don’t have to remember how you had it sorted when looking for a particular dataset.
- When sharing a dataset with an individual user, the email they’ll get now has the dataset name in the subject.
- Fixed: project permissions were not updated correctly when datasets were added to a project.
- All users (including those who primarily use OAuth or a single-sign-on provider) can now set a distinct Crunch password, that they can use with applications that don’t use OAuth, such as the R package. To set a password, just go to https://beta.crunch.io/password/reset.
- The pycrunch library now knows how to check progress via the progress api rather than checking status of other things. Users of pycrunch are strongly encouraged to upgrade it because checking things other than progress can have unintended side effects, especially on performance.
- Support now has some videos about making filters.
- The control for weights in the web app could sometimes blow up if the list of variables was not initialized just so. We’ve made it less fussy.
- Summaries for text variables now show the first few rows of data, rather than the most freqent values.
- The organization of variables in a dataset can now be customized for each user.
- Appending data could sometimes fail and still not be very forthcoming about why. We have helped its communication skills even more.
- Now appending (posting to
batches) will never time out; it will always return202and update about its progress. - Requests for dataset entities (a very common thing to want to do) are now faster.
- Project permissions are now included on the list of projects so users can know whether they can edit things or not.
- Fixed: adding data by posting to
batchessometimes claimed that it was not encodable. Everything is encodeable, somehow. - Fixed: changing values of rows (inserting by a defined unique primary key) could cause duplicate-aliased columns. As with other times we’ve opened parallel universes, we have sealed this one shut.
- Fixed: When removing a dataset from a project, the order sometimes still had a reference to the dataset that was no longer actually in the catalog. Now it is removed.
- Streamed updates to variables with certain properties could be slow; they are now faster.
- Changing variable types in the web app now works more better.
- Teams working on custom web applications can now also customize their feedback reporting through usersnap (the “Report a problem” link).
- Fixed: The icon to remove a filter sometimes looked wrong. Also the icon to remove a variable in a crosstab. Those are both styled better now.
- Fixed: if you searched for something while looking at a table or graph, the deck’s save control was on top of the box with search results. It has been put back where it goes.
- When datasets were deleted, some tasks still tried in vain to work with them. Those tasks are now deleted too.
- Reporting of errors is better, so we can respond better.
- We can now deploy updates to the web application without any possible interruption of service.
- When appending a dataset, some conditions in array or multiple response variables would prevent the incoming data from being appended. Now we tell you which variables are implicated so they can be fixed.
- The new batch compare endpoint can help prepare data to be appended to an existing dataset.
- Fixed: Reverting a dataset could result in a query timeout, leaving users uncertain about the status of their request.
- Fixed: Tab books made with a multiple-response column variable can, in fact, be exported.
- The web app now supports multiple decks, so you can save and delete groups of analyses together.
- You can now move a dataset to a project by dragging it to the project side bar in the web app.
- The python interface, pycrunch, now requests resources to be compressed (gzipped) by default.
- Updating datasets by posting to their
streamendpoint is better and faster. - Expressions such as those that define filters no longer need in most cases to include specification of
typeon many of their arguments that the system can be expected to infer. - Fixed: converting a variable to a categorical one could result in duplicate category definitions. The evil twin ones are now not created.
- Cursors when dragging, and the thing being dragged, should look better in the web app.
- 🔍 When there were large numbers of results, search could be very slow. We’e made it load faster, and also not reöpen if the search was cancelled.
- The application’s dataset properties view got some cosmetic improvements; and Export Dataset is now its own tab.
- Fixed: The labs feature of “comparisons” for categories of variables were not being displayed in tables. When we noticed, we restored them.
- Fixed: When viewing multitables with categorical array variables, changing tabs didn’t update the table. Now it does.
- Fixed: Single table export of multitables sometimes silently failed; now, it politely downloads.
- New members without any datasets were welcomed with a rather unwelcoming empty screen. We’ve added some helpful links about getting started.
- 🔍 Search results ordering is a hard problem, and one we strive to get right — they should make more sense now than they did before.
- Sometimes deleting a dataset could cause unintended consequences when the search service tried in vain to keep track of it. The process is smoother now.
- Certain types of errors when appending sources could inform users that they had errors, but they failed to notify our support staff. We have helped them call for help.
- Fixed: In the web app, it was not possible to edit category names in variable properties for variables created using “New Categorical Variable”. Now you can.
- When appending datasets, subvariables could be added to the wrong parent array or multiple response variable because they had identical
description. - When an asynchronous task encountered an error, it reported via progress that it was “100% complete” which is true in the sense that it isn’t running anymore and it won’t provide any more progress updates, but not true in the sense that it didn’t finish what it was supposed to do. Now it reports a more useful “–1”.
- 🔍 Search indexing has been tweaked yet again to deal with dataset and variable names and aliases with underscores and unusual capitalization.
- In addition to the context menu on a variable in a multitable, you can also drag a variable to use it as a categorical / binned numeric or take the mean of it across the multitable’s columns.
-
Numeric variables, or categorical ones with numeric values, can now display their mean when they are the row of a multitable.
-
Several aspects of decks were updated to behave as clients expect, learning some exciting new verbs like POST and PATCH.
-
Users are now informed if they say something is gzipped and it isn’t. (Before, it was a server error.)
-
Projects can contain datasets that are only visible to editors (we call these “drafts”). The easiest way to toggle this is via R (
is.published(ds) <- TRUE). -
A relatively arcane style issue was adjusted so that we believe text appears crisper and cleaner in the web app. As always, we’d love to hear any specific feedback. 🎨
-
Importing and appending data are now handled differently and more asynchronously in continued effort to fulfill requests in a timely manner.
-
Speaking of importing and appending, sometimes we overzealously saved lots of versions. We’ve calmed that down to just the one.
-
Items using a
progresswould be “100%” even if they were 100% failed, which was misleading. It is important to acknowledge failure and learn from it, so now progress can help with that (it returns–1and status failed). -
Fixed: The API represented some items as if they were usable HTTP verbs that weren’t. We have helped it with its grammar.
- Performance for making tab books, especially for very large datasets, is about 30% faster than before.
- Search results now consider the variable group from the variable order, so if you label something “demographics” and search “demographics” its variables will show up. Not sure why we didn’t think of that before!
- The application server is more independent of some of its underlying dependencies and can continue to function and retry if some services are temporarily unavailable.
- Icons for projects now always have a unique name so if you update it you won’t receive an outdated one from some cache in the cloud. After all, the cloud is just someone else’s computer. ⛅
- Array or multiple response derivations of combine-categories or shallow copy derivations in the past may not have aligned categories of subvariables correctly. How could we be so silly!
- SPSS dataset export was dropping string category names if they matched the numeric values (e.g., 1 labeled “1”). Although we think it a bit silly to do such a thing, users’ labels are now exported.
- Fixed: Closing a list of search results no longer annoyingly re-opens. Instead, it gets the hint to stop searching.
- Fixed: When the dataset catalog contained a large number of archived datasets, they could be counted as displayed even though they weren’t visible. Ghosts: busted! 🚫👻
- The web app is now much faster at loading the variable catalog and order, because it no longer fills up the console while doing so. That was helpful, but faster if it just does its work and doesn’t talk about it so much.
- A new view in the “projects” feature of the app allows project admins to create projects.
- Fixed: the web app will load even if modules from a manifest can’t be loaded. Features depending on the modules won’t work, obviously, but now the problem is confined to actual dependencies.
- Fixed: Forked datasets issued a mysterious error about a frame directory when attempting to save a version. They haven’t been framed, and now versions are saved correctly.
- Fixed: an issue when working with datetime and categorical variables together could report an internal server error instead of a more helpful error telling a user how to correct their request.
- Minor version bump because of a new way to manage (asynchronous, possibly parallel) ‘tasks’.
- When viewing a table with a numeric variable (or categorical with numeric values), users can now right-click the variable to use its mean as the cell value in the table — since it’s already there, you said, ¿why not? — so we made that happen.
- In the ‘Projects’ labs feature, organization of datasets into folders was improved (and documented); also, projects’ user membership 👤 and permissions should be clearer.
- Users can better manage sharing and permission of datasets that pertain to Projects.
- Fixed: Results in exported tab books exclude rows filtered by dataset-level exclusions.
- Fixed: Downloading the same tab book again now actually downloads it instead of silently failing.
- Fixed: Remember 2.25.49, where one icon (👤) looked strange? Then one of their peers felt out of place, so we helped them feel welcomed and lined up too.
- 🔍 Some things were being indexed for search that led to results showing up when humans didn’t expect them.
- 🔎 Some different things were not being indexed for search that led to results not showing up when humans reasonably expected them.
- 🔍 Datasets accessible via team sharing weren’t playing well as a team; we coached them a bit.
- A brief spike (about a week) in some response times over 200ms has been corrected.
- Large array or multiple-response variables could be slow retrieving their
subvariablescatalog, so we made that faster. - Restoring a version sometimes timed out, which was a frustrating experience, like a cat knocking over an hourglass 🐈 ⏳. Stretching this analogy, we have moved the hourglass away from the edge of the table (cat video).
- Converting text variables to datetimes is more likely to work (if the values are formatted in the one true and correct date format).
- Further work on deploying updates seamlessly, further diminishing the need to type the unpleasant word hiccough or even hiccup.
- Notifications in the web application relating to dataset state changes (such as someone else editing it) have been improved.
- Filters and some other analytic operations involving datetime variables were made faster. 📅
- Fixed: saved analyses with means of numeric or scale variables were exporting percentages rather than means.
- More validation (and better information about bad requests) is performed when creating datasets by posting metadata.
- (Potentially) long-running asynchronous tasks such as export and tab books are now handled better.
- Dataset sharing can now be revoked by email address (rather than just long cryptic user id).
- Deleting variables is now faster. 🚮
- The datasets catalog could be excessively slow when it was large. Now it is faster.
- Fixed: CSV exporting.
- Some old metadata items were collecting like dust bunnies 🐰. We have tidied them up.
- Long variable descriptions (often used for survey question wording) looked bad when editing variable properties. They now remain neatly in their box.
- Deployment was made faster, but may still produce momentary outages or hiccoughs. We are continuing to work on this so we never have to type ‘hiccough’ again.
- Moving between projects, datasets, and a particular dataset in the web application is smoother. Also, things like variable cards and the variables list in the side bar are refreshed when they should be.
- 🔍 Search results beyond what fit on a page were not being shown.
- 🔍 When performing a search, result sets could be far too large.
- 🔍 Now, results themselves are paginated, so return faster; and you can navigate pages of them.
- 🔍 Search_queries_containing_underscores actually work now!
- Loading more variable metadata is much faster due to an optimization that recently became obvious.
- When deploying updates, continue to shrink necessity and duration of downtime as services restart.
- When creating datasets with a metadata document, we now accept the documented Shoji:order.
- Also when creating datasets with metadata, we now handle
nullbetter and strive to provide better information about how we’re validating the input so users can produce correct payloads. - Fixed: The 👤 icon in the web app likes to move around sometimes, so we have to find new ways to make it stay put.
- Querying the dataset’s
statecould be slow sometimes, because it was doing some unnecessary things that it doesn’t do anymore. - Relatedly, authorization checks in the server are dramatically faster now.
- The web app’s datasets list and search entry bar are lined up better now through judicious use of 📏 and 📐.
- Querying a
cubeof categorical arrays, especially with large datasets and large arrays, is considerably faster. - Projects, a view that some users currently have but more will soon, received an update of its sidebar and fancy dynamic… ellipses when names don’t fit.
- Getting the variables catalog on large datasets took longer than it should. Now, because it is no longer an option to include subvariables in such a request, it is much faster. To get subvariable metadata, use the
tableendpoint with limit 0. - Fixed: When importing a file, storage optimizations were being delayed longer than they should.
- Derived variables in ‘My Variables’ can now be used to create multitables and filters.
- If an editor unweights or changes to an already-existing weight, other viewers no longer get a notification that the data has changed because it hasn’t.
- We now accept valid SPSS files with multiple
mrsetsdefined that contain the same base variables (because we believe this is ill advised, the base variables are copied for subsequent multiple response or array definitions). - In the variable organizer, now when you add a folder (➕📁), you can actually see the 📂New Folder you just added.
- When dragging variables on tables and graphs, the areas you could drag them to could be confusing, and you shouldn’t have to guess what will happen.
- We made it much faster to get the list of variables when there are lots of them. 🏇
- Fixed: a regression in the web app whereby viewers of datasets could not set or change weights in their datasets.
- CSV export now leaves system-missing cells empty (…having no data), rather than printing
No data. - R users can now work with hidden variables with greater ease.
- Many types of cube queries are now much faster than they were. 🚄
- Because 📅 is a useful thing to sort by, all users can now view and sort their datasets by start date, end date, created, accessed, and modified.
- Because sometimes exporting data is a useful thing to do, we now provide csv exports from dataset properties.
- The web application no longer makes several requests (especially of a
variable/{id}/) if it doesn’t need to. - Multiple response variables now can indicate that their items (subvariables) may have varying numbers of valid and missing rows.
- More performance improvements.
- The search box (for variables, datasets, all the things) is now more consistently available when it makes sense to search, and hidden when search wouldn’t be as useful.
- A new parameter
limit=0for thetableendpoint provides basic column metadata faster than any other resource. - The Notebooks feature — a hosted Jupyter application — has updated R and Python packages.
- Weight variables are easier to manage (select, apply, and unapply) in the web app.
- Better validation when creating array variables via the api or R.
- Better validation when creating a dataset with metadata; now only the system can defined system categories.
- A new
uniform_basisattribute of multiple response variables lets users indicate that subvariables may have different numbers of valid and missing rows. - The R package now provides tools to work with DatasetOrders for projects, and improved tools for working with dataset VariableOrders.
- Fixed: “When I create a new variable and return to my previous view, I should be seeing the new variable.”
- Fixed: Exclusions neither fork nor merge. Now they do both.
- Fixed: a critical bug whereby the web client couldn’t load some datasets at all due to an inconsistency that should not have stopped it. Now it powers through.
- Speed improvement: Looking up team datasets could be very slow, so we made it faster. 🏇
- Fixed: Viewers of a dataset cannot circumvent an Exclusion created by editors.
- Fixed: renaming categories of a variable used as the basis of derived variables could fail with a categorical error of questionable scalar categorical categorical grammaticality; now categories can be renamed.
- Fixed: dataset metadata definitions that specified
discarded: truefor categorical array variables didn’t heed that attribute; now arrays can be hidden at import time as documented and as users quite reasonably expected. - Fixed: when
binfunctions encountered ∞ it could inadvertently convert it to NaN. ∞ is a number, but a rather tedious one to explain to computers. 🤖 - Fixed: labs feature ‘Projects’ created new projects from teams but multiplied datasets by all their versions. Just one, please!
- Speed improvement: loading variable order on large new datasets.
- Speed improvement:
/datasets/was too slow, and is now faster. - Speed improvement: O hey! so is
/teams/{id}/datasets/! - ⌛ when reverting is no longer a timeout but now actually a revert.
- Likewise ⌛ when saving a version now saves in some cases where it didn’t before.
- Future speed improvements: finer logging of where slow requests spend their time. We will find you and make you faster.
- Fixed: For a small number of users, and for a very tedious set of reasons, it was possible that appending a dataset would say it worked but was lying. This is no longer possible.
- Fixed: Several containers tried to scroll horizontally in the datasets list and things got chaotic — not as bad as the Republican debates, but out of control.
- Fixed: Renaming categories in a categorical variable on which derived categorical variables were defined no longer gives a mysterious and highly uninformative error about
categorical categorical scalar categorical categorical. Instead, it renames the category. - Fixed: A small issue for non-webkit browsers that prevent correct interpretation of a regular expression.
- Fixed: An issue involving filter expressions with missing values of numeric variables.
- Fixed: Numeric summaries that actually had no bounds caused histogram (bin) functions to freak out; now they are reported as bounds for bins (though yes, of course ±∞ means unbounded)
- Fixed: Search indexing failed with ‘joined’ datasets.
- Fixed: Exporting the deck of a dataset with a dot in its name had no file type and could try to open with not-a-spreadsheet application. Now the file name always ends in
.xlsxand the type is set explicitly.
- 📣 A significant rearrangement of data storage promises much better performance for some common requests and actions.
- The tab book feature is now available for everyone, not just labs users.
- Fixed: doing nothing when clicking on ‘support’ in the app was decidedly unsupportive.
- If a dataset has an incorrect
order, the web client would simply say that things couldn’t be loaded. It now gives a more informative message on the off chance that you can fix the order by other means. - A new progress module consumes data from the progress endpoint, used in the context of tab books.
- A considerable efficiency improvement on some frequently accessed endpoints including
datasets,variables, andusers. - We optimized storage and culling of exported stuff like spss, csv, and tab books.
-
Labs users can now organize datasets by the time they last accessed it. Accessing a dataset is recorded by making an empty POST to
datasets/{id}/visit/(only the web client currently records access). - Fixed: The web application can no longer be hoisted by its own petard (query cache).
- Fixed: When combining responses, we now offer a more helpful message that all combinations must contain responses.
- Fixed: Date/time encoding could have an incorrect time zone.
- Fixed: When adding categories to the ‘new categorical variable’ builder, you could add more categories than the screen could hold; now you can scroll to actually see them.
- Fixed: Multiple response and categorical array variables in the same ‘new categorical variable’ builder no longer report incorrect summary numbers.
- Fixed: Excel export with categorical arrays didn’t respect the (implicit) row-based percentage direction setting. Now they are the same as what is shown in the application.
- Fixed: profiles.crunch.io no longer redirects elsewhere.
- A major-version upgrade to a piece of our architecture, and some configuration changes, bring greatly enhanced reliability.
- Tab book export is more accessible in multitable view: now just a menu under “Export” with no right-clicking.
- Fixed: Hidden variables are hidden from search now too, so will not appear as unclickable results in the list.
- Fixed: Once-confusing “percentage direction” settings with categorical arrays no longer invite confusion.
- Fixed: Restoring a version is much faster.
- Better streaming data error reporting is better.
- Fixed: Invalid data streams with duplicate aliases are now actually rejected as invalid.
- Fixed: sometimes search failures were reported to users, rather than to us. We can help!
- The
/table/endpoint not only respects limits and offsets, it requires them, so you don’t accidentally download GB of data. - Fixed: restoring a past version could leave a lingering multitable definition from the future. No more.
- Speed improvement: Variable entity requests (a very common request, as it happens) are considerably faster.
- Tab books no longer include hidden variables; instead, they just have the variables that you can see and probably expected them to contain.
- The
/table/api endpoint now looks more like its dataset-level peers with filters, and will respect offsets and limits. - Fixed: making a large change to a variable, such as reordering and renaming thousands of categories, failed with an ⌛. Now changes are applied.
- Fixed: Deleting a dataset entity caused a gateway timeout. Now it deletes a dataset. 🚮
- Fixed: It was possible to create a dataset with an invalid variable
order, but because the order was invalid the dataset couldn’t be displayed. That didn’t seem quite right, so we fixed it. - Fixed: Histograms or summaries of numeric variables containing infinite or undefined values freaked out and couldn’t even. Now they simply treat those values as missing when constructing bins.
- Much more server capacity!
- In addition to SPSS .sav format, the API now can now export a dataset as CSV.
- SPSS export respects the order of the variables in the dataset instead of tossing them in at random.
- You can now choose in a table to show discrete values of numeric variables instead of ranges (bins).
- Likewise, you can change the rollup of datetime variables in a table.
- A plethora of cross-browser issues were addressed by contributor @markrian. We are grateful.
- The variables catalog now contains the information you need to make a typical (‘rolled up’) query for datetime variables.
- Various improvements to icons and dropdown menus
- Search is now much more visible and obvious.
- Streaming clients now receive emails when contents contain errors, so they can put away their 🔮
- Fixed: if the order of subvariables was changed, appending data afterward resulted in a ‘conflict’. Now, it appends (and in the correct order)
- Fixed: making and reverting savepoints caused a gateway timeout. No more!
- Fixed: the variable organizer wouldn’t let you create the same-named subfolder in two folders. Now it does.
- Fixed: excel export of an analysis containing tabs (3-way tables) assumed that all tabs had the same length and interrupted them to place the following tab. It has been taught not to interrupt.
- Fixed: a bug that prevented the datasets list from scrolling and loading more than the initial 40 datasets.
- Fixed: a bug that prevent the application from loading at all in Internet Explorer.
- labs users can now create tab books by right-clicking “Export” in “multitable” view. A tab book is an excel workbook with one sheet per variable crossed by the columns defined in the given multitable.
- Streaming data can now be sent keyed by variable alias, not variable id.
- Better monitoring for the email service and for the multiple database servers.
- Speed gains on several very common database requests.
- Improved dataset load time with multiple database servers.
- Increase reliability of large import and append jobs.
- Fixed: left-clicking to export a single multitable sometimes didn’t do anything (failed silently). Now it exports.
- Fixed: refreshing variable cards when the exclusion filter was changed.
- Fixed: "too much data" error when attempting to merge a fork. The amount of data is now just right.
- Fixed: deck exporting didn't give any feedback that it was working and could time out if you clicked it over and over.
- Fixed: unbinding and rebinding multiple response variables could fail if multiple categories had been indicated as "selected"
- On-the-fly filtering!
- When exporting saved analyses that have ad-hoc filters, now we include a description of the filter.
- Performance improvement in dataset appending
- Performance gain in editing categories in array variables.
- Various issues to support third-party UIs
- Fixed: dataset list could look odd if the names of dataset owners were long. Now you can have as long of a name as you want.
- Fixed: when dragging a variable from the sidebar onto a variable card, if the card were short, the drop zone could get cut off.
- Fixed: odd case of SPSS dataset export failing with some combination of public and personal variables.
- Fixed: reordering array subvariables didn't persist the next time the dataset was loaded. Oops. Now it does.
- Fixed: API changes had broken the Pandas dataframe export in pycrunch.
- Fixed: another issue in merging forks
- Fixed: better validation and feedback when PATCHing the dataset catalog
- Fixed: an issue that caused delays in loading datasets when multiple database servers were deployed. Now they don't fight with each other.