Get-GSDriveFileList and memory usage

[indented-automation]

When querying large file drive structures the responses are grouped and held in memory by the Get-GSDriveFileList command.

This occurs because of the unnecessary use of array concatenation using the response variable in the block below.

$response = @()
[int]$i = 1
do {
    $result = $request.Execute()
    $response += $result.Files | Select-Object @{N = 'User';E = {$User}},*
    if ($result.NextPageToken) {
        $request.PageToken = $result.NextPageToken
    }
    [int]$retrieved = ($i + $result.Files.Count) - 1
    Write-Verbose "Retrieved $retrieved Files..."
    [int]$i = $i + $result.Files.Count
}
until (!$result.NextPageToken)
$response

This change removes the assignment, results will be immediately written to the output pipeline. A secondary benefit is that this version should perform better as it loses the expensive array creation / copy operation hidden behind +=.

[int]$i = 1
do {
    $result = $request.Execute()
    $result.Files | Select-Object @{N = 'User';E = {$User}},*
    if ($result.NextPageToken) {
        $request.PageToken = $result.NextPageToken
    }
    [int]$retrieved = ($i + $result.Files.Count) - 1
    Write-Verbose "Retrieved $retrieved Files..."
    [int]$i = $i + $result.Files.Count
}
until (!$result.NextPageToken)

Memory will only be consumed by this command for each page, or if the requestor performs their own assignment.

Are you happy with the change? Would you like a pull request?

[scrthq]

Hey Chris - Thanks for taking a crack at it, shoot over a PR! My original thought was to clean up Verbose output so there weren't large blocks of output in between the verbose status messages; that being said, if you're collecting the output in a variable when you run the command, it'll achieve the same purpose while also speeding up the function itself. It's worth noting that the other *List* functions mostly take the same approach as Get-GSDriveFileList (i.e. Get-GSUserListPrivate in the Private function folder of the repo, I believe), so those will need to be cleaned up down the line as well. We can work on it together =]

…

On Wed, May 2, 2018 at 4:03 AM Chris Dent ***@***.***> wrote: When querying large file drive structures the responses are grouped and held in memory by the Get-GSDriveFileList command. This occurs because of the unnecessary use of array concatenation using the response variable in the block below. $response = @()[int]$i = 1do { $result = $request.Execute() $response += $result.Files | Select-Object @{N = 'User';E = {$User}},* if ($result.NextPageToken) { $request.PageToken = $result.NextPageToken } [int]$retrieved = ($i + $result.Files.Count) - 1 Write-Verbose "Retrieved $retrieved Files..." [int]$i = $i + $result.Files.Count }until (!$result.NextPageToken)$response This change removes the assignment, results will be immediately written to the output pipeline. A secondary benefit is that this version should perform better as it loses the expensive array creation / copy operation hidden behind +=. [int]$i = 1do { $result = $request.Execute() $result.Files | Select-Object @{N = 'User';E = {$User}},* if ($result.NextPageToken) { $request.PageToken = $result.NextPageToken } [int]$retrieved = ($i + $result.Files.Count) - 1 Write-Verbose "Retrieved $retrieved Files..." [int]$i = $i + $result.Files.Count }until (!$result.NextPageToken) Memory will only be consumed by this command for each page, or if the requestor performs their own assignment. Are you happy with the change? Would you like a pull request? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#38>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMIo3a8sqhGMNAlmt7zSsiRQ-n2iV15eks5tuXZzgaJpZM4TvGWn> .

[scrthq]

@indented-automation added the changes and pushed it out in v2.5.3! Thanks for the suggestion!