sfctl does not support compression when uploading application to image stro

[talarari]

Our sfctl application upload command takes around 20 minutes, i see compression is supported in powershell but not in sfctl, why is that?

[talarari]

also trying pre-compressing the app with powershell and then using sfctl to upload and it seems like it uncompressed the package?

[13:31:35]	[Step 2/2] Starting: /home/soluto/BuildAgent/temp/agentTmp/custom_script4537417520140519763
[13:31:35]	[Step 2/2] in directory: /home/soluto/BuildAgent/work/fc4611665efdeb8d
[13:31:36]	[Step 2/2] Registering Service Fabric application...
[13:33:22]	[Step 2/2] [1/2059] files, ApplicationManifest.xml
[13:33:22]	[Step 2/2] [2/2059] files, _.dir
[13:33:22]	[Step 2/2] [3/2059] files, Felix.Feeder.GooglePubsubPkg/Config.zip
[13:33:22]	[Step 2/2] [4/2059] files, Felix.Feeder.GooglePubsubPkg/Code.zip
[13:33:22]	[Step 2/2] [5/2059] files, Felix.Feeder.GooglePubsubPkg/PubsubConfig.zip
[13:33:22]	[Step 2/2] [6/2059] files, Felix.Feeder.GooglePubsubPkg/ServiceManifest.xml
[13:33:22]	[Step 2/2] [7/2059] files, Felix.Feeder.GooglePubsubPkg/_.dir
[13:33:22]	[Step 2/2] [8/2059] files, Felix.Feeder.GooglePubsubPkg/PubsubConfig/homedev-26419.json
[13:33:22]	[Step 2/2] [9/2059] files, Felix.Feeder.GooglePubsubPkg/PubsubConfig/sizzling-heat-9792.json
[13:33:22]	[Step 2/2] [10/2059] files, Felix.Feeder.GooglePubsubPkg/PubsubConfig/_.dir
[13:33:22]	[Step 2/2] [11/2059] files, Felix.Feeder.GooglePubsubPkg/Config/Settings.xml
[13:33:22]	[Step 2/2] [12/2059] files, Felix.Feeder.GooglePubsubPkg/Config/_.dir
[13:33:22]	[Step 2/2] [13/2059] files, Felix.Feeder.GooglePubsubPkg/Code/Microsoft.Win32.Primitives.dll
[13:33:22]	[Step 2/2] [14/2059] files, Felix.Feeder.GooglePubsubPkg/Code/App.Metrics.Abstractions.dll
[13:33:22]	[Step 2/2] [15/2059] files, Felix.Feeder.GooglePubsubPkg/Code/Google.Apis.Auth.dll
[13:33:22]	[Step 2/2] [16/2059] files, Felix.Feeder.GooglePubsubPkg/Code/System.Threading.Tasks.Dataflow.dll
[13:33:22]	[Step 2/2] [17/2059] files, Felix.Feeder.GooglePubsubPkg/Code/api-ms-win-crt-process-l1-1-0.dll
[13:33:22]	[Step 2/2] [18/2059] files, Felix.Feeder.GooglePubsubPkg/Code/System.Security.Cryptography.ProtectedData.dll
[13:33:22]	[Step 2/2] [19/2059] files, Felix.Feeder.GooglePubsubPkg/Code/App.Metrics.Core.dll
[13:33:22]	[Step 2/2] [20/2059] files, Felix.Feeder.GooglePubsubPkg/Code/System.Linq.dll

as you can see Code.zip and Config.zip are there, but then there are all the dll files in Code folder, which i assume come from the command unpacking the Code.zip file.

is there anyway of uploading a compressed package with sfctl?

[talarari]

apparently i had old files cached in the CI machine, i am now getting a different error:
FABRIC_E_IMAGEBUILDER_UNEXPECTED_ERROR: The file 'D:\SvcFab\ImageBuilderProxy\AppType\230d4d35-6cbb-43c2-99a6-7fb65881e14f\Felix.ApiPkg\Config\Settings.xml' already exists.
although the file only exists once in the package

[Christina-Kang]

@talarari can you share which command in sfctl you are running to get the error FABRIC_E_IMAGEBUILDER_UNEXPECTED_ERROR? Is it sfctl application upload?

Can you share the output with the --debug option on? Are you uploading to the native image store, or are you providing a custom imagestore-string? What sfctl and cluster version are you using?

For a workaround, you can manually zip some files. It's not ideal, unfortunately, but would do the trick for now. If you go to the app package, and go down to the Code, Config, and Data folders under each service and use any zip utility for those, it should speed up the upload time. The zipped folders in Linux should end in .zip.

If you do have access to PowerShell on a different machine, you can also compress using that. It would be the same "Copy-ServiceFabricApplicationPackage" command, but with the "SkipCopy" command selected. Sorry for the inconvenience.

@suhuruli can answer the question on having compression in sfctl in the future, but I believe it is planned.

Thanks for the feedback!

Edit: Updated workaround

[suhuruli]

@talarari, plans to add compression is in the pipeline.

Currently, the compression is only available in the PowerShell module that we have available on Windows that ships with the SDK. This functionality along with a host of others will need to be extended to all the modules that are based on REST (SFCTL, new PowerShell module etc).

[jfalameda]

Is there still no way to upload a packaged app? I am working from a Mac and it takes a long time to upload an app file by file.

Thanks.

[Christina-Kang]

Hi @jfalameda you can upload a directory at a time, just uncompressed. It would recursively go through the sub-folders. Would that help your use case? Or do you mean that you are hitting a timeout issue?

Thanks!
Christina

[jfalameda]

Hi,

What, do you mean? To upload each service of the application at at time?
This is how my application build looks like. In total 6000+ files.

ApplicationManifest.xml
 + AddReaSagaPkg 
 + AreaPkg
 + EmailPkg
 ... more

I adapted the yeoman build script to build my app in mac before uploading it to service fabric (My application was originally generated on MS windows Visual Studio, so I don´t find a better way to compile to upload it to Service fabric). It looks like this:

DIR=`dirname $0`

dotnet restore $DIR/../DHCloud/AddAreaSaga/AddReaSaga.csproj -s https://api.nuget.org/v3/index.json
dotnet build $DIR/../DHCloud/AddAreaSaga/AddReaSaga.csproj -v normal

cd `dirname $DIR/../DHCloud/AddAreaSaga/AddReaSaga.csproj`
dotnet publish -o $DIR/../DHCloud/DHCloud/AddReaSagaPkg/Code
cp PackageRoot/* $DIR/../DHCloud/DHCloud/AddReaSagaPkg/

cd -

dotnet restore $DIR/../DHCloud/Api/Api.csproj -s https://api.nuget.org/v3/index.json
dotnet build $DIR/../DHCloud/Api/Api.csproj -v normal

cd `dirname $DIR/../DHCloud/Api/Api.csproj`
dotnet publish -o $DIR/../DHCloud/DHCloud/ApiPkg/Code
cp PackageRoot/* $DIR/../DHCloud/DHCloud/ApiPkg/

cd -

... more

And this is my upload script:

sfctl cluster select --endpoint http://localhost:19080
sfctl application upload --path DHCloud --show-progress
sfctl application provision --application-type-build-path DHCloud -t 1000000

The provision generates an error I documented here: #177

Kind Regards,
/ José

[Christina-Kang]

Thanks, @jfalameda for the clarification. We don't have compression yet, but I'll update this thread when we have a better estimate or when it's added. Thank you so much for sharing your feedback! We can look into the provision issue on the other thread.

[jfalameda]

@Christina-Kang Thanks, is this something planned on the roadmap? Thanks.

[Christina-Kang]

@jfalameda, yes, it is on the road map, though we do not yet have a specific date. A fix for the provision timeout issue is coming very soon though!

[jfalameda]

@Christina-Kang Thanks!