Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

First round of i18n in acs-engine based on gettext. Subsequent change… #627

Merged
merged 22 commits into from
Jul 31, 2017
Merged

First round of i18n in acs-engine based on gettext. Subsequent change… #627

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ecdd9b9
First round of i18n in acs-engine based on gettext. Subsequent change…
JiangtianLi May 13, 2017
78be458
Update acs-engine go source to use translation functions. Generate tr…
JiangtianLi May 18, 2017
261ffdf
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi May 18, 2017
1867b0c
Vendor github.com/leonelquinteros/gotext package using glide
JiangtianLi May 18, 2017
638031b
Rebase and update translation for update and deploy command
JiangtianLi May 18, 2017
201c149
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi May 18, 2017
f9e7193
Move test translation files so that translations directory is used fo…
JiangtianLi May 18, 2017
150b711
Use go-bindata to add resource strings to acs-engine binary
JiangtianLi May 19, 2017
db742c9
Fix reading bindata and unit test
JiangtianLi May 19, 2017
1a54a26
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi May 19, 2017
bcbcac0
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jun 1, 2017
f16bb48
Update translation files
JiangtianLi Jun 1, 2017
3847870
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jul 7, 2017
9ea4527
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jul 20, 2017
67642cb
More fix/refactor after rebase and add README
JiangtianLi Jul 21, 2017
8337029
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jul 25, 2017
39fbb34
Update resource files
JiangtianLi Jul 25, 2017
7816133
Add LCG files converted from PO files.
JiangtianLi Jul 25, 2017
91ad8cd
Update translation bindata
JiangtianLi Jul 25, 2017
e7dc96a
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jul 28, 2017
4aa260e
Merge branch 'master' of https://github.com/Azure/acs-engine into jia…
JiangtianLi Jul 31, 2017
68e8ddb
Remove go generated translation bindata
JiangtianLi Jul 31, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ENV AZURE_CLI_VERSION 2.0.3

RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get -y install python-pip make build-essential curl openssl vim jq \
&& apt-get -y install python-pip make build-essential curl openssl vim jq gettext \
&& rm -rf /var/lib/apt/lists/*

RUN mkdir /tmp/godeb \
Expand All @@ -29,6 +29,10 @@ RUN git clone https://github.com/akesterson/cmdarg.git /tmp/cmdarg \
RUN git clone https://github.com/akesterson/shunit.git /tmp/shunit \
&& cd /tmp/shunit && make install && rm -rf /tmp/shunit

# Go tool for internationalization and localization
RUN go get github.com/JiangtianLi/gettext/... \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should ACS Engine be getting stuff from private repos?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried not to but I spent some time and didn't find a tool. That package is basically used for extract strings out of source file for translation so it will only run every time we update translation strings. I forked https://github.com/gosexy/gettext to address the parsing we need. I can try to merge into upstream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I submitted a PR to that tool: gosexy/gettext#16

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just echoing others and calling out we can't include this change as is, the repo has to be publically accessible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like https://github.com/JiangtianLi/gettext is public now (might have been from the beginning), so we're OK to use the fork, as long as we've registered it at https://ossmsft.visualstudio.com/_oss.

&& go install github.com/JiangtianLi/gettext/...

# Used by some CI jobs
ADD ./test/bootstrap/checkout-pr.sh /tmp/checkout-pr.sh

Expand Down
2 changes: 2 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ prereqs:
go get github.com/jteeuwen/go-bindata/...
go get github.com/Sirupsen/logrus
go get github.com/spf13/cobra
go get github.com/onsi/gomega
go get github.com/leonelquinteros/gotext

build: prereqs
go generate -v ./...
Expand Down
48 changes: 38 additions & 10 deletions cmd/generate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ import (

"github.com/Azure/acs-engine/pkg/acsengine"
"github.com/Azure/acs-engine/pkg/api"
"github.com/Azure/acs-engine/pkg/i18n"
"github.com/leonelquinteros/gotext"
)

const (
Expand All @@ -18,17 +20,19 @@ const (
)

type generateCmd struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about deploy and update commands?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reminding. I didn't notice that and will address it.

apimodelPath string
outputDirectory string // can be auto-determined from clusterDefinition
caCertificatePath string
caPrivateKeyPath string
classicMode bool
noPrettyPrint bool
parametersOnly bool
translationsDirectory string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caller should not need to pass translation dir to any command ever. This decreases usability of ACS Engine. This should happen automatically. In fact do we even care of localize ACS Engine commands? Localization is primarily needed for the RP.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, localization of ACS Engine command has lower priority. The primary translation happens in acsengine and api package, for the purpose of RP. I added here for the e2e test and it is nice to have for ACS Engine. For the translationsDirectory parameter, it will not be needed and default to the translation folder if acs-engine is run at the package root directory (I think that's the most used scenario?) It has the issue if go install is used without deploy translation file. Another approach is to used go-bindata to avoid such caveat. I'll look into it first.

apimodelPath string
outputDirectory string // can be auto-determined from clusterDefinition
caCertificatePath string
caPrivateKeyPath string
classicMode bool
noPrettyPrint bool
parametersOnly bool

// Parsed from inputs
containerService *api.ContainerService
apiVersion string
locale *gotext.Locale
}

func NewGenerateCmd() *cobra.Command {
Expand All @@ -44,6 +48,7 @@ func NewGenerateCmd() *cobra.Command {
}

f := generateCmd.Flags()
f.StringVar(&gc.translationsDirectory, "translations-directory", "", "translations directory (translations in the current directory if absent)")
f.StringVar(&gc.apimodelPath, "api-model", "", "")
f.StringVar(&gc.outputDirectory, "output-directory", "", "output directory (derived from FQDN if absent)")
f.StringVar(&gc.caCertificatePath, "ca-certificate-path", "", "path to the CA certificate to use for Kubernetes PKI assets")
Expand All @@ -59,6 +64,13 @@ func (gc *generateCmd) validate(cmd *cobra.Command, args []string) {
var caCertificateBytes []byte
var caKeyBytes []byte

locale, err := i18n.LoadTranslations(gc.translationsDirectory)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that a lot of setup is happening even before ACS Engine functions are called. Which means evey time someone needs to use the primary ACS Engine functions (in the commands or RP) these steps need to be followed. IMO this should be as simple as translation dir being passed to InitializeTemplateGenerator. And then everything is setup inside it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. The translation init routine includes set the locale, bind to translation directory, and select domain. I thought about the approach of have package level init function and have each exported function call it if it haven't called before. GNU gettext doc has the guidance of how to init in library. However, in our case, the ACS Engine package is called by RP, which serves requests with different language on multiple thread. For example, if InitializeTemplateGenerator is called by the same thread on different language, the locale needs to be updated. If InitializeTemplateGenerator is called by different thread, each thread needs to keep its own locale. If InitializeTemplateGenerator is called by the same thread with the same language, then we don't need to init again. I haven't found a simple way to handle that other than the caller needs to do some initialization and pass the locale in the context (adopting Go context pattern).

if err != nil {
log.Fatalf("error loading translation files: %s", err.Error())
}

i18n.Initialize(locale)

if gc.apimodelPath == "" {
if len(args) > 0 {
gc.apimodelPath = args[0]
Expand All @@ -75,7 +87,12 @@ func (gc *generateCmd) validate(cmd *cobra.Command, args []string) {
log.Fatalf("specified api model does not exist (%s)", gc.apimodelPath)
}

containerService, apiVersion, err := api.LoadContainerServiceFromFile(gc.apimodelPath)
apiloader := &api.Apiloader{
Translator: &i18n.Translator{
Locale: locale,
},
}
containerService, apiVersion, err := apiloader.LoadContainerServiceFromFile(gc.apimodelPath)
if err != nil {
log.Fatalf("error parsing the api model: %s", err.Error())
}
Expand All @@ -93,13 +110,19 @@ func (gc *generateCmd) validate(cmd *cobra.Command, args []string) {

gc.containerService = containerService
gc.apiVersion = apiVersion
gc.locale = locale
}

func (gc *generateCmd) run(cmd *cobra.Command, args []string) error {
gc.validate(cmd, args)
log.Infoln("Generating...")

templateGenerator, err := acsengine.InitializeTemplateGenerator(gc.classicMode)
ctx := acsengine.Context{
Translator: &i18n.Translator{
Locale: gc.locale,
},
}
templateGenerator, err := acsengine.InitializeTemplateGenerator(ctx, gc.classicMode)
if err != nil {
log.Fatalln("failed to initialize template generator: %s", err.Error())
}
Expand All @@ -120,7 +143,12 @@ func (gc *generateCmd) run(cmd *cobra.Command, args []string) error {
}
}

if err = acsengine.WriteArtifacts(gc.containerService, gc.apiVersion, template, parameters, gc.outputDirectory, certsGenerated, gc.parametersOnly); err != nil {
writer := &acsengine.ArtifactWriter{
Translator: &i18n.Translator{
Locale: gc.locale,
},
}
if err = writer.WriteArtifacts(gc.containerService, gc.apiVersion, template, parameters, gc.outputDirectory, certsGenerated, gc.parametersOnly); err != nil {
log.Fatalf("error writing artifacts: %s \n", err.Error())
}

Expand Down
21 changes: 12 additions & 9 deletions pkg/acsengine/engine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"text/template"

"github.com/Azure/acs-engine/pkg/api"
"github.com/Azure/acs-engine/pkg/i18n"
"github.com/ghodss/yaml"
)

Expand Down Expand Up @@ -165,7 +166,7 @@ func (t *TemplateGenerator) verifyFiles() error {
allFiles = append(allFiles, swarmTemplateFiles...)
for _, file := range allFiles {
if _, err := Asset(file); err != nil {
return fmt.Errorf("template file %s does not exist", file)
return t.Translator.Errorf("template file %s does not exist", file)
}
}
return nil
Expand All @@ -174,12 +175,14 @@ func (t *TemplateGenerator) verifyFiles() error {
// TemplateGenerator represents the object that performs the template generation.
type TemplateGenerator struct {
ClassicMode bool
Translator *i18n.Translator
}

// InitializeTemplateGenerator creates a new template generator object
func InitializeTemplateGenerator(classicMode bool) (*TemplateGenerator, error) {
func InitializeTemplateGenerator(ctx Context, classicMode bool) (*TemplateGenerator, error) {
t := &TemplateGenerator{
ClassicMode: classicMode,
Translator: ctx.Translator,
}

if err := t.verifyFiles(); err != nil {
Expand Down Expand Up @@ -207,15 +210,15 @@ func (t *TemplateGenerator) GenerateTemplate(containerService *api.ContainerServ

templ = template.New("acs template").Funcs(t.getTemplateFuncMap(containerService))

files, baseFile, e := prepareTemplateFiles(properties)
files, baseFile, e := t.prepareTemplateFiles(properties)
if e != nil {
return "", "", false, e
}

for _, file := range files {
bytes, e := Asset(file)
if e != nil {
err = fmt.Errorf("Error reading file %s, Error: %s", file, e.Error())
err = t.Translator.Errorf("Error reading file %s, Error: %s", file, e.Error())
return templateRaw, parametersRaw, certsGenerated, err
}
if _, err = templ.New(file).Parse(string(bytes)); err != nil {
Expand Down Expand Up @@ -280,7 +283,7 @@ func GenerateKubeConfig(properties *api.Properties, location string) (string, er
return kubeconfig, nil
}

func prepareTemplateFiles(properties *api.Properties) ([]string, string, error) {
func (t *TemplateGenerator) prepareTemplateFiles(properties *api.Properties) ([]string, string, error) {
var files []string
var baseFile string
if properties.OrchestratorProfile.OrchestratorType == api.DCOS {
Expand All @@ -296,7 +299,7 @@ func prepareTemplateFiles(properties *api.Properties) ([]string, string, error)
files = append(commonTemplateFiles, swarmModeTemplateFiles...)
baseFile = swarmBaseFile
} else {
return nil, "", fmt.Errorf("orchestrator '%s' is unsupported", properties.OrchestratorProfile.OrchestratorType)
return nil, "", t.Translator.Errorf("orchestrator '%s' is unsupported", properties.OrchestratorProfile.OrchestratorType)
}

return files, baseFile, nil
Expand Down Expand Up @@ -999,18 +1002,18 @@ func getSecurityRules(ports []int) string {
func (t *TemplateGenerator) getSingleLineForTemplate(textFilename string, cs *api.ContainerService, profile interface{}) (string, error) {
b, err := Asset(textFilename)
if err != nil {
return "", fmt.Errorf("yaml file %s does not exist", textFilename)
return "", t.Translator.Errorf("yaml file %s does not exist", textFilename)
}

// use go templates to process the text filename
templ := template.New("customdata template").Funcs(t.getTemplateFuncMap(cs))
if _, err = templ.New(textFilename).Parse(string(b)); err != nil {
return "", fmt.Errorf("error parsing file %s: %v", textFilename, err)
return "", t.Translator.Errorf("error parsing file %s: %v", textFilename, err)
}

var buffer bytes.Buffer
if err = templ.ExecuteTemplate(&buffer, textFilename, profile); err != nil {
return "", fmt.Errorf("error executing template for file %s: %v", textFilename, err)
return "", t.Translator.Errorf("error executing template for file %s: %v", textFilename, err)
}
expandedTemplate := buffer.String()

Expand Down
25 changes: 21 additions & 4 deletions pkg/acsengine/engine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,32 @@ import (
"bytes"
"fmt"
"io/ioutil"
"path"
"path/filepath"
"strings"
"testing"

"github.com/Azure/acs-engine/pkg/api"
"github.com/Azure/acs-engine/pkg/api/v20160330"
"github.com/Azure/acs-engine/pkg/api/vlabs"
"github.com/Azure/acs-engine/pkg/i18n"
"github.com/leonelquinteros/gotext"
)

const (
TestDataDir = "./testdata"
)

func TestExpected(t *testing.T) {
// Initialize locale for translation
locale := gotext.NewLocale(path.Join("..", "..", "translations"), "en_US")
i18n.Initialize(locale)

apiloader := &api.Apiloader{
Translator: &i18n.Translator{
Locale: locale,
},
}
// iterate the test data directory
apiModelTestFiles := &[]APIModelTestFile{}
if e := IterateTestFilesDirectory(TestDataDir, apiModelTestFiles); e != nil {
Expand All @@ -26,7 +38,7 @@ func TestExpected(t *testing.T) {
}

for _, tuple := range *apiModelTestFiles {
containerService, version, err := api.LoadContainerServiceFromFile(tuple.APIModelFilename)
containerService, version, err := apiloader.LoadContainerServiceFromFile(tuple.APIModelFilename)
if err != nil {
t.Errorf("Loading file %s got error: %s", tuple.APIModelFilename, err.Error())
continue
Expand All @@ -50,7 +62,12 @@ func TestExpected(t *testing.T) {
// 1. first time tests loaded containerService
// 2. second time tests generated containerService
// 3. third time tests the generated containerService from the generated containerService
templateGenerator, e3 := InitializeTemplateGenerator(isClassicMode)
ctx := Context{
Translator: &i18n.Translator{
Locale: locale,
},
}
templateGenerator, e3 := InitializeTemplateGenerator(ctx, isClassicMode)
if e3 != nil {
t.Error(e3.Error())
continue
Expand Down Expand Up @@ -118,11 +135,11 @@ func TestExpected(t *testing.T) {
t.Errorf("generated parameters different from expected for model %s: '%s'", tuple.APIModelFilename, diffstr)
}

b, err := api.SerializeContainerService(containerService, version)
b, err := apiloader.SerializeContainerService(containerService, version)
if err != nil {
t.Error(err)
}
containerService, version, err = api.DeserializeContainerService(b)
containerService, version, err = apiloader.DeserializeContainerService(b)
if err != nil {
t.Error(err)
}
Expand Down
Loading