Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Linux Consumption] Support persistent storage using Azure file share #121

Merged
merged 1 commit into from
Jul 4, 2020
Merged

[Linux Consumption] Support persistent storage using Azure file share #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Common/Constants.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ public static class Constants
public const string NpmDebugLogFile = "npm-debug.log";

public const string DeploymentCachePath = "deployments";
public const string ArtifactsPath = "artifacts";
public const string SiteExtensionsCachePath = "siteextensions";
public const string DeploymentToolsPath = "tools";
public const string SiteFolder = @"site";
Expand Down Expand Up @@ -156,5 +157,10 @@ public static TimeSpan MaxAllowedExecutionTime
public const string LinuxLogEventStreamName = "MS_KUDU_LOGS";
public const string WebSiteHomeStampName = "WEBSITE_HOME_STAMPNAME";
public const string WebSiteStampDeploymentId = "WEBSITE_STAMP_DEPLOYMENT_ID";
public const string MeshInitURI = "MESH_INIT_URI";
public const string AzureWebJobsStorage = "AzureWebJobsStorage";
public const string KuduFileShareMountPath = "/kudu-mnt";
public const string KuduFileSharePrefix = "kudu-mnt";
public const string EnablePersistentStorage = "ENABLE_KUDU_PERSISTENT_STORAGE";
}
}
4 changes: 3 additions & 1 deletion Kudu.Console/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
using Kudu.Core.SourceControl.Git;
using Kudu.Core.Tracing;
using System.Reflection;
using Kudu.Core.LinuxConsumption;
using XmlSettings;
using log4net;
using log4net.Config;
Expand Down Expand Up @@ -283,7 +284,8 @@ private static IEnvironment GetEnvironment(string siteRoot, string requestId)
repositoryPath,
requestId,
Path.Combine(AppContext.BaseDirectory, "KuduConsole", "kudu.dll"),
null);
null,
new FileSystemPathProvider(new NullMeshPersistentFileSystem()));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does consumption functions support Local Git Deploys, this is called as a Git POST-Receive hook to stream the build output logs to the git client. In that case would the Oryx Build expected this path to be present ? (I'm not sure if ArtifactsPath could be present on /kudu-mnt)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local Git deploy is not supported in consumption. ArtifactsPath will never be under /kudu-mnt and I think Oryx doesn't access Artifacts folder directly. Oryx produces build output under /site/wwwroot and later as a post build step we use that build output to build a squashfs file under Artifacts folder and upload to storage. So nothing should change for Oryx. ( @Hazhzeng Please verify my comment :) )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Bala's right, we don't support Git deploy in Linux Consumption. For oryx build, it will build the package from /tmp/zipdeploy/extracted, and generated the built content into /site/wwwroot. After that, the SetupLinuxConsumptionFunctionAppDeployment step will zip update all content from /site/wwwroot into /site/deployment and upload the artifact to customer's storage account.

Build pipeline:
/tmp/zipdeploy/extracted => /site/wwwroot/ => /site/deployment => customer storage

Oryx Version      : 0.2.20200114.13, Commit: 204922f30f8e8d41f5241b8c218425ef89106d1d, ReleaseTagName: 20200114.13
Build Operation ID: |ETigkkgjbb4=.48f45d84_
Repository Commit : 09032b09e9a144f78a9cc005f97de358


Source directory     : /tmp/zipdeploy/extracted
Destination directory: /home/site/wwwroot

}
}
}
1 change: 1 addition & 0 deletions Kudu.Contracts/IEnvironment.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ public interface IEnvironment
string RepositoryPath { get; set; } // e.g. /site/repository
string WebRootPath { get; } // e.g. /site/wwwroot
string DeploymentsPath { get; } // e.g. /site/deployments
string ArtifactsPath { get; } // e.g /site/artifacts
string DeploymentToolsPath { get; } // e.g. /site/deployments/tools
string SiteExtensionSettingsPath { get; } // e.g. /site/siteextensions
string DiagnosticsPath { get; } // e.g. /site/diagnostics
Expand Down
21 changes: 20 additions & 1 deletion Kudu.Core/Environment.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,18 @@
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Extensions;
using Kudu.Core.Settings;
using Kudu.Core.LinuxConsumption;

namespace Kudu.Core
{
public class Environment : IEnvironment
{
private readonly IHttpContextAccessor _httpContextAccessor;
private readonly IFileSystemPathProvider _fileSystemPathsProvider;

private readonly string _webRootPath;
private readonly string _deploymentsPath;
private readonly string _artifactsPath;
private readonly string _deploymentToolsPath;
private readonly string _siteExtensionSettingsPath;
private readonly string _diagnosticsPath;
Expand Down Expand Up @@ -106,7 +109,8 @@ public Environment(
string repositoryPath,
string requestId,
string kuduConsoleFullPath,
IHttpContextAccessor httpContextAccessor)
IHttpContextAccessor httpContextAccessor,
IFileSystemPathProvider fileSystemPathsProvider)
{
RootPath = rootPath;

Expand All @@ -117,6 +121,7 @@ public Environment(
_zipTempPath = Path.Combine(_tempPath, Constants.ZipTempPath);
_webRootPath = Path.Combine(SiteRootPath, Constants.WebRoot);
_deploymentsPath = Path.Combine(SiteRootPath, Constants.DeploymentCachePath);
_artifactsPath = Path.Combine(SiteRootPath, Constants.ArtifactsPath);
_deploymentToolsPath = Path.Combine(_deploymentsPath, Constants.DeploymentToolsPath);
_siteExtensionSettingsPath = Path.Combine(SiteRootPath, Constants.SiteExtensionsCachePath);
_diagnosticsPath = Path.Combine(SiteRootPath, Constants.DiagnosticsPath);
Expand Down Expand Up @@ -162,6 +167,7 @@ public Environment(
RequestId = !string.IsNullOrEmpty(requestId) ? requestId : Guid.Empty.ToString();

_httpContextAccessor = httpContextAccessor;
_fileSystemPathsProvider = fileSystemPathsProvider ?? throw new ArgumentNullException(nameof(fileSystemPathsProvider));

KuduConsoleFullPath = kuduConsoleFullPath;
}
Expand Down Expand Up @@ -191,10 +197,23 @@ public string DeploymentsPath
{
get
{
if (_fileSystemPathsProvider.TryGetDeploymentsPath(out var path))
{
return FileSystemHelpers.EnsureDirectory(path); ;
}

return FileSystemHelpers.EnsureDirectory(_deploymentsPath);
}
}

public string ArtifactsPath
{
get
{
return FileSystemHelpers.EnsureDirectory(_artifactsPath);
}
}

public string DeploymentToolsPath
{
get
Expand Down
2 changes: 1 addition & 1 deletion Kudu.Core/Helpers/LinuxConsumptionDeploymentHelper.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ public static async Task SetupLinuxConsumptionFunctionAppDeployment(
string sas = settings.GetValue(Constants.ScmRunFromPackage) ?? System.Environment.GetEnvironmentVariable(Constants.ScmRunFromPackage);

string builtFolder = context.OutputPath;
string packageFolder = env.DeploymentsPath;
string packageFolder = env.ArtifactsPath;
string packageFileName = OryxBuildConstants.FunctionAppBuildSettings.LinuxConsumptionArtifactName;

// Package built content from oryx build artifact
Expand Down
16 changes: 16 additions & 0 deletions Kudu.Core/Infrastructure/OperationManager.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,5 +104,21 @@ public static T SafeExecute<T>(Func<T> action)
return default(T);
}
}

public static async Task<TResult> ExecuteWithTimeout<TResult>(Task<TResult> task, TimeSpan timeout)
{
var timeoutCancellationTokenSource = new CancellationTokenSource();

var completedTask = await Task.WhenAny(task, Task.Delay(timeout, timeoutCancellationTokenSource.Token));
if (completedTask == task)
{
timeoutCancellationTokenSource.Cancel();
return await task;
}
else
{
throw new TimeoutException("The operation has timed out.");
}
}
}
}
39 changes: 39 additions & 0 deletions Kudu.Core/LinuxConsumption/FileSystemPathProvider.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
using System;
using System.Diagnostics.Tracing;
using Kudu.Core.Infrastructure;
using Kudu.Core.Tracing;

namespace Kudu.Core.LinuxConsumption
{
public class FileSystemPathProvider : IFileSystemPathProvider
{
private readonly IMeshPersistentFileSystem _persistentFileSystem;

public FileSystemPathProvider(IMeshPersistentFileSystem persistentFileSystem)
{
_persistentFileSystem =
persistentFileSystem ?? throw new ArgumentNullException(nameof(persistentFileSystem));
}

public bool TryGetDeploymentsPath(out string path)
{
path = _persistentFileSystem.GetDeploymentsPath();
return !string.IsNullOrEmpty(path) && EnsureMountedDeploymentsPath(path);
}

private bool EnsureMountedDeploymentsPath(string path)
{
try
{
FileSystemHelpers.EnsureDirectory(path);
return true;
}
catch (Exception e)
{
KuduEventGenerator.Log().LogMessage(EventLevel.Informational, ServerConfiguration.GetApplicationName(),
$"{nameof(EnsureMountedDeploymentsPath)} Failed. Path = {path}", e.ToString());
return false;
}
}
}
}
7 changes: 7 additions & 0 deletions Kudu.Core/LinuxConsumption/IFileSystemPathProvider.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
namespace Kudu.Core.LinuxConsumption
{
public interface IFileSystemPathProvider
{
bool TryGetDeploymentsPath(out string path);
}
}
13 changes: 13 additions & 0 deletions Kudu.Core/LinuxConsumption/IMeshPersistentFileSystem.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
using System.Threading.Tasks;

namespace Kudu.Core.LinuxConsumption
{
public interface IMeshPersistentFileSystem
{
Task<bool> MountFileShare();

bool GetStatus(out string message);

string GetDeploymentsPath();
}
}
9 changes: 9 additions & 0 deletions Kudu.Core/LinuxConsumption/IMeshServiceClient.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
using System.Threading.Tasks;

namespace Kudu.Core.LinuxConsumption
{
public interface IMeshServiceClient
{
Task MountCifs(string connectionString, string contentShare, string targetPath);
}
}
9 changes: 9 additions & 0 deletions Kudu.Core/LinuxConsumption/IStorageClient.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
using System.Threading.Tasks;

namespace Kudu.Core.LinuxConsumption
{
public interface IStorageClient
{
Task CreateFileShare(string siteName, string connectionString, string fileShareName);
}
}
9 changes: 9 additions & 0 deletions Kudu.Core/LinuxConsumption/ISystemEnvironment.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
namespace Kudu.Core.LinuxConsumption
{
public interface ISystemEnvironment
{
string GetEnvironmentVariable(string name);

void SetEnvironmentVariable(string name, string value);
}
}
6 changes: 1 addition & 5 deletions Kudu.Core/LinuxConsumptionEnvironment.cs → ...onsumption/LinuxConsumptionEnvironment.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Kudu.Contracts;
using Kudu.Contracts.Settings;

namespace Kudu.Core
namespace Kudu.Core.LinuxConsumption
{
public class LinuxConsumptionEnvironment : ILinuxConsumptionEnvironment
{
Expand Down Expand Up @@ -59,8 +57,6 @@ public bool InStandbyMode
}
}

Task ILinuxConsumptionEnvironment.DelayCompletionTask => throw new NotImplementedException();

public void DelayRequests()
{
_delayLock.EnterUpgradeableReadLock();
Expand Down
Loading