Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: multi locale fallback #858

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions scripts/generateLocales.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { resolve } from 'node:path';
import type { Options } from 'prettier';
import { format } from 'prettier';
import options from '../.prettierrc.cjs';
import { faker } from '../src';
import type { LocaleDefinition } from '../src/definitions';
import { DEFINITIONS } from '../src/definitions';

Expand Down Expand Up @@ -81,19 +82,22 @@ function containsAll(checked?: string[], expected?: string[]): boolean {
}

function generateLocaleFile(locale: string): void {
faker.setLocale(locale);

const localeOrder = faker.localeOrder;

let content = `
${autoGeneratedCommentHeader}

import { Faker } from '../faker';
import ${locale} from '../locales/${locale}';
${locale !== 'en' ? "import en from '../locales/en';" : ''}
${localeOrder
.map((entry) => `import ${entry} from '../locales/${entry}';`)
.join('\n')}

const faker = new Faker({
locale: '${locale}',
localeFallback: 'en',
localeOrder: ${JSON.stringify(localeOrder)},
locales: {
${locale},
${locale !== 'en' ? 'en,' : ''}
${localeOrder.map((entry) => `${entry},`).join('\n')}
},
});

Expand Down
48 changes: 35 additions & 13 deletions src/faker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,12 @@ export type UsedLocales = Partial<Record<UsableLocale, LocaleDefinition>>;

export interface FakerOptions {
locales: UsedLocales;
locale?: UsableLocale;
localeFallback?: UsableLocale;
Comment on lines -41 to -42
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should somehow deprecate these with since v7 until v8 to make a smooth transition possible for users

localeOrder?: UsableLocale[];
}

export class Faker {
locales: UsedLocales;
locale: UsableLocale;
localeFallback: UsableLocale;
localeOrder: UsableLocale[];
Comment on lines 45 to +46
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we need to make these readonly, so we can control when setting it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs to be a way to set them, it may also be useful to get them.
Especially so, if you consider it in combination with seed.


readonly definitions: LocaleDefinition = this.initDefinitions();

Expand Down Expand Up @@ -95,25 +93,35 @@ export class Faker {
}

this.locales = opts.locales;
this.locale = opts.locale || 'en';
this.localeFallback = opts.localeFallback || 'en';
this.localeOrder = opts.localeOrder ?? ['en'];
}

/**
* Creates a Proxy based LocaleDefinition that virtually merges the locales.
*/
private initDefinitions(): LocaleDefinition {
// Returns the first resolved locale data that aren't undefined
const findFirst = <T>(
resolver: (data: LocaleDefinition) => T | undefined
): T | undefined => {
for (const locale of this.localeOrder) {
const baseData = resolver(this.locales[locale]);
if (baseData != null) {
return baseData;
}
}
return undefined;
};

// Returns the first LocaleDefinition[key] in any locale
const resolveBaseData = (key: keyof LocaleDefinition): unknown =>
this.locales[this.locale][key] ?? this.locales[this.localeFallback][key];
findFirst((data) => data[key]);

// Returns the first LocaleDefinition[module][entry] in any locale
const resolveModuleData = (
module: keyof LocaleDefinition,
entry: string
): unknown =>
this.locales[this.locale][module]?.[entry] ??
this.locales[this.localeFallback][module]?.[entry];
): unknown => findFirst((data) => data[module]?.[entry]);

// Returns a proxy that can return the entries for a module (if it exists)
const moduleLoader = (
Expand Down Expand Up @@ -159,11 +167,25 @@ export class Faker {
}

/**
* Set Faker's locale
* Set Faker's locale using the default fallback strategy.
*
* @param locale The locale to set (e.g. `en` or `en_AU`, `en_AU_ocker`).
* @param fallbacks The fixed fallbacks to use. Defaults to `['en']`.
*/
setLocale(locale: UsableLocale): void {
this.locale = locale;
setLocale(locale: UsableLocale, fallbacks: UsableLocale[] = ['en']): void {
const localeOrder = [];
const parts = locale.split('_');
for (let i = parts.length; i > 0; i--) {
const subLocale = parts.slice(0, i).join('_');
Comment on lines +177 to +179
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: could you explain me whats going on here? if you would like via voice chat 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'en_AU_ocker' -> ['en', 'AU', 'ocker'] // L177

-> ['en', 'AU', 'ocker'] -> 'en_AU_ocker' // L179
-> ['en', 'AU'] -> 'en_AU' // L179
-> ['en'] -> 'en' // L179

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh... I see, and it will not cause issues, if a in-between locale could not be found cause it's not defined?
Also maybe it should be documented that this exists / handle like this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note:

ES6 way:

'en_AU_ocker'.split('_').reduce((result, _, i, parts) => {
    return result.concat(parts.slice(0, parts.length - i).join('_'))
}, [])

No split:

'en_AU_ocker'.split('_').reduce((result, _, i, parts) => {
    parts.pop()
    return result.concat(parts.join('_'))
}, ['en_AU_ocker'])

Copy link
Member Author

@ST-DDT ST-DDT Apr 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh... I see, and it will not cause issues, if a in-between locale could not be found cause it's not defined?

if (this.locales[subLocale] != null) {

I probably have to add some tests explicitly for that case.

Also maybe it should be documented that this exists / handle like this

I will try to do that.

Side note:

IMO my "old school" approach is way easier to read for me.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO my "old school" approach is way easier to read for me.

Both fine with me. I just posted this for fun :) Possibly the no split version could be understood better then slice, not sure...

if (this.locales[subLocale] != null) {
localeOrder.push(subLocale);
}
}
for (const fallback of fallbacks) {
if (!localeOrder.includes(fallback) && this.locales[fallback] != null) {
localeOrder.push(fallback);
}
}
this.localeOrder = localeOrder;
}
}
3 changes: 1 addition & 2 deletions src/locale/af_ZA.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import af_ZA from '../locales/af_ZA';
import en from '../locales/en';

const faker = new Faker({
locale: 'af_ZA',
localeFallback: 'en',
localeOrder: ['af_ZA', 'en'],
ST-DDT marked this conversation as resolved.
Show resolved Hide resolved
locales: {
af_ZA,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/ar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import ar from '../locales/ar';
import en from '../locales/en';

const faker = new Faker({
locale: 'ar',
localeFallback: 'en',
localeOrder: ['ar', 'en'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As en is presented in every locale definition, I think we should skip it and add it as default on usage. Would make the config much simpler.

Copy link
Member

@Shinigami92 Shinigami92 Apr 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#642
With that it will someday be possible to create your own Faker instance without 'en'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So instead of configuring it explicitly you want to move it to a default / implicit value?
I explicitly didn't choose that approach to allow users to omit the en locale to avoid certain fallback mechanism issues.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be possible tbh. en is the only full locale and mostly maintained. If you would like to create a new feature which needs locale, we can't block it until its implemented on every locale, right?

The only interface we should offer for locales should be:

const faker = new Faker({ locales: ['de_AT', 'fr'] })

which would mean following fallback line: de_AT > de > fr > en

I can't imagine someone would prefer not working feature over getting a string in English when its not supported in a locale selected by him.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const faker = new Faker({ locales: ['de_AT', 'fr'] })

  1. locales is already in use and reserved for the locales itself that can be passed
  2. That would resolve to de_AT > de > fr not de_AT > de > fr > en, maybe you don't want en, if you want it, pass it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 locales is already in use and reserved for the locales itself that can be passed

I am not sure if I understand? This is exactly what we want, initialise faker with locales, right?

I think it should always add en no matter what. Imagine user calls faker.animal.dog() and there is no data for it in neither of the locales he specified. What should faker do then:

  1. return empty string?
  2. throw an error?
  3. or return something from en

I would vote for 3. Better to return English name, than nothing. But maybe we should throw?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 locales is already in use and reserved for the locales itself that can be passed

I am not sure if I understand? This is exactly what we want, initialise faker with locales, right?

locales -> available data; localeOrder -> active data (+ their order)

I think it should always add en no matter what. Imagine user calls faker.animal.dog() and there is no data for it in neither of the locales he specified. What should faker do then:

We add it by default, to exactly ensure this behavior.
However, to avoid an issue like #547, we have to avoid hardcoded fallbacks.
Sometimes having an undefined/error is easier to detect, then a random english word.

See also #823

Copy link
Member

@pkuczynski pkuczynski Apr 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

locales -> available data; localeOrder -> active data (+ their order)

This is not intuitive at all.

Sometimes having an undefined/error is easier to detect, then a random english word.

That's true, but since faker is meant to provide fake data for tests and maybe some demoes, I would be surprised if someone would prefer error to be thrown instead of seeing the value in English... If the error is thrown he would be a bit blocked vs when English is returned and he might be annoyed a bit and report the issue to us, which we would encourage him to fix by providing localisation...

I spent some time thinking about it and I wonder why would someone want to switch locales if he can simply have to separate instances of faker for every locale he needs? Can you come up with sensible example when switching locales on single version of faker is useful?

I would rather see something like this on the user side:

import { Faker, pl, de_AT } from '@faker-js/faker'

// or for someone who does not have tree-shaking bundler

import { Faker } from '@faker-js/faker'
import { pl } from '@faker-js/faker/locale/pl'
import { de_DE } from '@faker-js/faker/locale/de_DE'

const faker = {
  pl: new Faker({ locales: [pl] })
  de: new Faker({ locales: [de_AT] }) 
}

console.log(faker.pl.animal.bird())
console.log(faker.de.animal.bird())

That's clean.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you come up with sensible example when switching locales on single version of faker is useful?

Because its easier, otherwise we wouldn't do that in our tests.

locales -> available data; localeOrder -> active data (+ their order)

This is not intuitive at all.

For me it is, also I don't want to change the base logic too much with this PR.
(I intend to make this change backwards compatible, switching to ordered locale data instead of keys might make this impossible)

Please note, that this might be due to some thoughts I have regarding the locale loading.
I consider always adding all locale data to the faker instance, but load the actual contents only lazily.
See also #830
Since that's somewhat in the future, I would like to take one step at a time and fix some of our limitations, regarding the use of multiple locales.
In this PR, I would like to mostly stick to existing API. We can replace/improve the existing API in a separate PR to make reviews easier and focused on a single change.

I'm not really happy of how big of a change this fix actually is and think about ways to simplify it (e.g. don't replace fallbackLocale with localeOrder and just make locale a single key or array).

locales: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this would be an array and not object, you would not need the extra prop localeOrder right?

Copy link
Member Author

@ST-DDT ST-DDT Apr 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But, then you cannot easily switch between locales.
(strings are easier to replace, then full objects)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment above...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you switch between locales?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

faker.localeOrder = ['fr'];
faker.setLocale('fr_BE');

ar,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/az.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import az from '../locales/az';
import en from '../locales/en';

const faker = new Faker({
locale: 'az',
localeFallback: 'en',
localeOrder: ['az', 'en'],
locales: {
az,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/cz.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import cz from '../locales/cz';
import en from '../locales/en';

const faker = new Faker({
locale: 'cz',
localeFallback: 'en',
localeOrder: ['cz', 'en'],
locales: {
cz,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/de.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import de from '../locales/de';
import en from '../locales/en';

const faker = new Faker({
locale: 'de',
localeFallback: 'en',
localeOrder: ['de', 'en'],
locales: {
de,
en,
Expand Down
5 changes: 3 additions & 2 deletions src/locale/de_AT.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@
*/

import { Faker } from '../faker';
import de from '../locales/de';
import de_AT from '../locales/de_AT';
import en from '../locales/en';

const faker = new Faker({
locale: 'de_AT',
localeFallback: 'en',
localeOrder: ['de_AT', 'de', 'en'],
locales: {
de_AT,
de,
en,
},
});
Expand Down
5 changes: 3 additions & 2 deletions src/locale/de_CH.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@
*/

import { Faker } from '../faker';
import de from '../locales/de';
import de_CH from '../locales/de_CH';
import en from '../locales/en';

const faker = new Faker({
locale: 'de_CH',
localeFallback: 'en',
localeOrder: ['de_CH', 'de', 'en'],
locales: {
de_CH,
de,
en,
},
});
Expand Down
3 changes: 1 addition & 2 deletions src/locale/el.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import el from '../locales/el';
import en from '../locales/en';

const faker = new Faker({
locale: 'el',
localeFallback: 'en',
localeOrder: ['el', 'en'],
locales: {
el,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ import { Faker } from '../faker';
import en from '../locales/en';

const faker = new Faker({
locale: 'en',
localeFallback: 'en',
localeOrder: ['en'],
locales: {
en,
},
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_AU.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_AU from '../locales/en_AU';

const faker = new Faker({
locale: 'en_AU',
localeFallback: 'en',
localeOrder: ['en_AU', 'en'],
locales: {
en_AU,
en,
Expand Down
5 changes: 3 additions & 2 deletions src/locale/en_AU_ocker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@

import { Faker } from '../faker';
import en from '../locales/en';
import en_AU from '../locales/en_AU';
import en_AU_ocker from '../locales/en_AU_ocker';

const faker = new Faker({
locale: 'en_AU_ocker',
localeFallback: 'en',
localeOrder: ['en_AU_ocker', 'en_AU', 'en'],
locales: {
en_AU_ocker,
en_AU,
en,
},
});
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_BORK.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_BORK from '../locales/en_BORK';

const faker = new Faker({
locale: 'en_BORK',
localeFallback: 'en',
localeOrder: ['en_BORK', 'en'],
locales: {
en_BORK,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_CA.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_CA from '../locales/en_CA';

const faker = new Faker({
locale: 'en_CA',
localeFallback: 'en',
localeOrder: ['en_CA', 'en'],
locales: {
en_CA,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_GB.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_GB from '../locales/en_GB';

const faker = new Faker({
locale: 'en_GB',
localeFallback: 'en',
localeOrder: ['en_GB', 'en'],
locales: {
en_GB,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_GH.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_GH from '../locales/en_GH';

const faker = new Faker({
locale: 'en_GH',
localeFallback: 'en',
localeOrder: ['en_GH', 'en'],
locales: {
en_GH,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_IE.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_IE from '../locales/en_IE';

const faker = new Faker({
locale: 'en_IE',
localeFallback: 'en',
localeOrder: ['en_IE', 'en'],
locales: {
en_IE,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_IND.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_IND from '../locales/en_IND';

const faker = new Faker({
locale: 'en_IND',
localeFallback: 'en',
localeOrder: ['en_IND', 'en'],
locales: {
en_IND,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_NG.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_NG from '../locales/en_NG';

const faker = new Faker({
locale: 'en_NG',
localeFallback: 'en',
localeOrder: ['en_NG', 'en'],
locales: {
en_NG,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_US.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_US from '../locales/en_US';

const faker = new Faker({
locale: 'en_US',
localeFallback: 'en',
localeOrder: ['en_US', 'en'],
locales: {
en_US,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/en_ZA.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import en_ZA from '../locales/en_ZA';

const faker = new Faker({
locale: 'en_ZA',
localeFallback: 'en',
localeOrder: ['en_ZA', 'en'],
locales: {
en_ZA,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/es.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import es from '../locales/es';

const faker = new Faker({
locale: 'es',
localeFallback: 'en',
localeOrder: ['es', 'en'],
locales: {
es,
en,
Expand Down
5 changes: 3 additions & 2 deletions src/locale/es_MX.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@

import { Faker } from '../faker';
import en from '../locales/en';
import es from '../locales/es';
import es_MX from '../locales/es_MX';

const faker = new Faker({
locale: 'es_MX',
localeFallback: 'en',
localeOrder: ['es_MX', 'es', 'en'],
locales: {
es_MX,
es,
en,
},
});
Expand Down
3 changes: 1 addition & 2 deletions src/locale/fa.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import fa from '../locales/fa';

const faker = new Faker({
locale: 'fa',
localeFallback: 'en',
localeOrder: ['fa', 'en'],
locales: {
fa,
en,
Expand Down
3 changes: 1 addition & 2 deletions src/locale/fi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import en from '../locales/en';
import fi from '../locales/fi';

const faker = new Faker({
locale: 'fi',
localeFallback: 'en',
localeOrder: ['fi', 'en'],
locales: {
fi,
en,
Expand Down
Loading