Payload limit

[mintminttea]

Question

Is there a size limitation on the pode post data? if yes, is it possible to adjust in server setting or workaround to post the data?

I got error message when I try to post some json data and check the result in console.
If the json is small enough it will return via $Webevent.data, else my powershell return below.

Invoke-RestMethod: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

Server side

Add-PodeRoute -Method Post -Path "/submit"  -ContentType "application/json"  -ScriptBlock { 
        $webevent.Data | Out-Default
       Write-PodeJsonResponse -Value @{ 'test' = 'result'}
 }

Client side command
OK when select name and cpu only

Invoke-RestMethod  -method Post -Uri "http://localhost:8080/submit" -ContentType "application/json" -Body (get-process | select name,cpu -first 1 | ConvertTo-Json)

Error when contains all properties

Invoke-RestMethod  -method Post -Uri "http://localhost:8080/submit" -ContentType "application/json" -Body (get-process | select  -first 1 | ConvertTo-Json)

[Badgerati]

Hey!

There isn't a limit, but I might know what's happening. For me, Get-Process | Select -First 1 | ConvertTo-Json works; I had to do Get-Process | ConvertTo-Json to trip the error 😂 (even then, this actually sometimes did work!)

I'll investigate to see if my idea could be right though.

Thanks!

[Badgerati]

Hey @mintminttea,

My guess was right 😄, the body was so large at times the socket didn't receive all the data at once. The above commit made Get-Process | ConvertTo-Json always work for me, are you able to test on your end?

Thanks

[mintminttea]

Yes, I can confirm it is working as expected now, thanks.

[mintminttea]

Hi,
I got error again when try to run the same action under https, and the server response every slow too when posting data.
http is fine now.

$data = Get-Process
    Invoke-RestMethod -uri "https://xxx.xxx.com:8443/submit" -Method Post -Body ($data | convertto-json -Depth 8) -ContentType "application/json; charset=UTF-8" 

Invoke-RestMethod : 

    
        400
        
    

    

        
            
                ⚠️ 
                Bad Request
            
            
            
                400
            
            
                https://xxx.xxx.com:8443/submit
            

            
Conversion from JSON failed with error: Error parsing boolean value. Path 
'[3].Modules[26].FileVersionInfo.IsPatched', line 7437, position 67.
At C:\Users\xxx\Documents\PowerShell\Modules\Pode\2.0.3\Private\Helpers.ps1:1214 char:44
+ …             $Result.Data = ($Content | ConvertFrom-Json -AsHashtable)
+                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
at ConvertFrom-PodeRequestContent, C:\Users\xxx\Documents\PowerShell\Modules\Pode\2.0.3\Private\Helpers.ps1: 
line 1214
at <ScriptBlock>, C:\Users\xxx\Documents\PowerShell\Modules\Pode\2.0.3\Private\Middleware.ps1: line 286
at Invoke-PodeScriptBlock, C:\Users\xxx\Documents\PowerShell\Modules\Pode\2.0.3\Public\Utilities.ps1: line 652
at Invoke-PodeMiddleware, C:\Users\xxx\Documents\PowerShell\Modules\Pode\2.0.3\Private\Middleware.ps1: line 55
at <ScriptBlock>, <No file>: line 73
    + NotSpecified: (:) [ConvertFrom-Json], ArgumentException
                
        

        
            🧡 Powered by Pode
        

    


At line:1 char:5
+     Invoke-RestMethod -uri "https://xxx.xxx.com:8443/submit" -Meth ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExceptio 
   n
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

[Badgerati]

OK, so the commit above appears to solve sending via HTTPS - I'm still testing some new logic for reading in requests, so could change at some point.

I think it's worth saying that although Pode doesn't enforce any upper limits on the payload, the larger it gets, expect some instability or slowness. There's a chance using the Pode.Kestrel module could work better instead, though I am intrigued to see how far we can push the base Pode engine 🤔

You can shrink the size a bit by doing Get-Process | ConvertTo-Json -Compress, and for speed I'd recommend using PS6/7 which is way quicker when Pode does ConvertFrom-Json - that's the slowest part!.

[mintminttea]

I got below error when post the data now.

One or more errors occurred. (Unable to write data to the transport connection: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'..)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
   at Pode.PodeResponse.Send() in PodeResponse.cs:line 72

Did give Pode.Kestrel a try and it is running great, I think I can switch to this or use the tcp endpoint directly.

[Badgerati]

Hey @mintminttea, if Pode.Kestrel works, I'd recommend using that for larger payloads.

What I will do though, is move the above changes from this issue and open a new issue to look at improving Pode's listener to handle larger payloads. I'll keep the changes that were originally merged in for 2.1.0.

To help, would you be all right to answer the following questions? As they'll be useful for testing 😃 I've been able to run the above commit on HTTPS with an 85MB JSON payload without issue you see.

• I've been using a small server with the Route you supplied on opening the issue. Do you have an more full example of your server's script so I can test with it?
• Which version of PowerShell are you using?
• Is the server being run directly in a PowerShell window, or behind something like IIS/etc?
• Is the server/IRM being run from Windows, or Linux/MacOS/Container?
• Is the below the best IRM to use, and are you running this from the same box the server is running or from another machine?

Invoke-RestMethod -uri "https://localhost:8080/submit" -Method Post -Body (Get-Process | ConvertTo-Json -Depth 8) -ContentType "application/json"

Thanks!

[mintminttea]

Well actually I am trying to create a software inventory system in pure PowerShell as my own interest as well as using in my company. The client side is based on the inventory script in psgallery with some modification, then post the json data to pode server. The data is not huge which is usually under 100KB~200KB when I view the size from the database.
Server side is combination of pode, pode.web to read/writejson data to apache couchdb for reporting, so server script cannot test or run alone without couchdb.
But for posting data in server part, it is just read the json data and then put it into couchdb directly. One of the thing I am not sure is I need to convert $WebEvent.Data to json and back again to make it back to a pscustomobject, but server didn't process the scriptblock when having above issue, so it should be not related.

Add-PodeRoute -Method Post -Path "/submit" -ContentType "application/json; charset=UTF-8"  -ScriptBlock {
     Write-Output "Received client submit information" | out-default
    $data =($WebEvent.Data | Convertto-Json -Depth 8 | ConvertFrom-Json)
    return update-myCDBComputerItem -id (($data.BIOS.SerialNumber).replace(" ","_")) -Data $data
}

Which version of PowerShell are you using?

Using powershell 7.1.1

Is the server being run directly in a PowerShell window, or behind something like IIS/etc?

Directly in powershell and launch via cmd pwsh.exe -file server.ps1

Is the server/IRM being run from Windows, or Linux/MacOS/Container?

Running on Windows 10 on my PC and Windows server 2016 in other server, not inside container.

Is the below the best IRM to use, and are you running this from the same box the server is running or from another machine?

Tried both posting to my own pc and posting to the windows server.

[Badgerati]

Thank you, this should hopefully help!

[ili101]

That's a cool idea, I was thinking on doing something similar when Spiceworks development stopped
I tried the code on the Issue-655 branch and it worked for me
Server:

Import-Module -Name '.\src\Pode.psd1', '.\src\Pode.Web.psd1' -Force
Start-PodeServer {
    Add-PodeEndpoint -Address localhost -Protocol Https -SelfSigned

    Add-PodeRoute -Method Post -Path "/submit" -ContentType "application/json; charset=UTF-8"  -ScriptBlock {
        Write-Output "Received client submit information" | out-default
        $data = ($WebEvent.Data | Convertto-Json -Depth 8 | ConvertFrom-Json)

        # return update-myCDBComputerItem -id (($data.BIOS.SerialNumber).replace(" ", "_")) -Data $data
        $data | Out-File -FilePath .\test.json
        $null = Write-PodeJsonResponse -Value $data
    }
}

Run:

$TestSmall = [PSCustomObject]@{
    Name1 = 'foo'
    Name2 = 'bar'
} | ConvertTo-Json
$TestBig = (Get-Process | ConvertTo-Json -Depth 8)

$R = Invoke-RestMethod -uri "https://localhost:8443/submit" -Method Post -Body $TestBig -ContentType "application/json" -SkipCertificateCheck

[ili101]

I was having similar problems downloading excel files where Pode stopped responding sometimes.
Tried develop + Issue-669 + Issue-655 and it works now consistently 👍

[ili101]

Bug no the Issue-655 branch

Start-PodeServer {
    Add-PodeEndpoint -Address localhost -Protocol Http
    New-PodeLoggingMethod -File -Name 'Errors' | Enable-PodeErrorLogging
    New-PodeLoggingMethod -File -Name 'Requests' | Enable-PodeRequestLogging

    Set-PodeViewEngine -Type Pode

    Add-PodeRoute -Method Get -Path / -ScriptBlock {
        Write-PodeJsonResponse -Value @{ 'value' = 'Test Server!' }
    }
}

1. Run server and browse to it.
2. See in Requests log the connection (Route is executed).
3. Close the browser.
4. See in Requests log that the endpoint executed again unexpectedly when the browser was closed.
   How dose the Pode server even know that the browser was closed? Is some connection stay open until browser process closed, and then this somehow make Pode execute the last connection again?
   I'm intrigued 🤔

[Badgerati]

It's the way KeepAlive connections work, a connection is kept open between the browser and server to re-use - rather than re-creating a new connection for every request 😃

Closing the browser closes the connection, which is triggered as an empty read-event on the server, but I had the state update a line after it should have been! The commit above moves it and should work now.

[Badgerati]

I've actually just been testing this branch to see if it behaves via IIS, Docker, Linux, etc. So far, all seems good (bar the bug you raised! 🙈).

Thinking we might actually be able to push this with 2.1.0 🤔 the only setup I'm not able to properly test is when using an Azure App Proxy (@ericpritchett I believe you had this setup?). But from what the issue was there (SSL padding), and how this new logic works , it shouldn't cause an issue.

[ili101]

Fix LGTM 👍. Hope it get into 2.1.0 as on develop I get some download errors.
Don't know regarding Azure App Proxy but basic Nginx proxy looks ok.