Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read EAV -oneshot #138

Closed
wibeasley opened this issue Feb 26, 2017 · 1 comment
Closed

Read EAV -oneshot #138

wibeasley opened this issue Feb 26, 2017 · 1 comment
Assignees

Comments

@wibeasley
Copy link
Member

Read EAV (instead of wide/flat) from the REDCap API, and convert it to a flat (non-EAV) dataset. This will be more work for R on the client than before. But it should be faster overall, because the REDCap webserver doesn't have to concatenate each cell in the EAV to as long of a string.

The advantages will be even greater when the REDCap project is 'sparse', in the sense that many variables aren't assigned to every event. The string concatenation in PHP still has to add a comma (and thus the memory-performance hit) in the wide format. But in the EAV format, those columns/rows/cells are simply passed over.

Even if it took the client the same amount of time, it's still better, because the work is distributed to the clients, instead of clogging up the webserver on long 10k+ reads.

(Thanks to discussion and guidance from @taylorr4, @123andy, & @aarenson for a 1+ years)

@wibeasley wibeasley self-assigned this Feb 26, 2017
wibeasley added a commit that referenced this issue Feb 26, 2017
wibeasley added a commit that referenced this issue Feb 26, 2017
wibeasley added a commit that referenced this issue Feb 26, 2017
wibeasley added a commit that referenced this issue Feb 26, 2017
Proof-of-concept  -not generalized.  See #138
wibeasley added a commit that referenced this issue Feb 26, 2017
more robustly from during the creation of ds_eav_2.  Also, move the now-old snippets towards the bottom. See #138
@wibeasley wibeasley changed the title Read EAV Read EAV -oneshot Mar 21, 2017
@wibeasley
Copy link
Member Author

The playground for the oneshot is done. I'm moving on to experimenting w/ batching & EAV (#145).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant