Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HeanCMS Template and websites #397

Open
wants to merge 56 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
e826396
add HeanCMS decorator and update YugenMangaES
MikeZeDev Dec 11, 2023
2b679f9
small fixes and add tests
MikeZeDev Dec 11, 2023
e91776a
add PerfScan
MikeZeDev Dec 11, 2023
a7aeaf6
add OmegaScans
MikeZeDev Dec 11, 2023
237c012
add TempleScan
MikeZeDev Dec 11, 2023
eca3a49
Merge branch 'master' into HeanCMS
MikeZeDev Jan 5, 2024
460c123
Update _index.ts
MikeZeDev Jan 5, 2024
b12ee87
HeanCMS : fix pictures CDN
MikeZeDev Jan 6, 2024
af6ace5
Merge branch 'master' into HeanCMS
MikeZeDev Jan 16, 2024
10913f1
improve platform abstraction
MikeZeDev Jan 16, 2024
389cb58
remove YugenMangasES
MikeZeDev Feb 4, 2024
86260da
Merge branch 'master' into HeanCMS
MikeZeDev Feb 26, 2024
cb7e37b
Update _index.ts
MikeZeDev Feb 26, 2024
8fb0611
minor code changes
MikeZeDev Mar 9, 2024
446365f
Merge branch 'master' into HeanCMS
MikeZeDev Apr 12, 2024
7128544
Remove TempleScans
MikeZeDev Apr 12, 2024
a35928c
Revert "Remove TempleScans"
MikeZeDev Apr 12, 2024
5796316
handle v2 api
MikeZeDev Apr 13, 2024
0918056
better code
MikeZeDev Apr 14, 2024
4008c38
Modescanlor : use HeanCMS
MikeZeDev Apr 14, 2024
7281d22
Merge branch 'master' into HeanCMS
MikeZeDev Apr 26, 2024
d0aac35
Merge branch 'master' into HeanCMS
MikeZeDev Apr 27, 2024
e4c5aff
fix case
MikeZeDev Apr 27, 2024
2306b20
Merge branch 'master' into HeanCMS
MikeZeDev May 7, 2024
9ec054e
update e2e tests to use vitest
MikeZeDev May 7, 2024
0887b0d
Merge branch 'master' into HeanCMS
MikeZeDev May 18, 2024
3286dd1
Merge branch 'master' into HeanCMS
MikeZeDev Jun 15, 2024
d9fb7d5
Merge branch 'master' into HeanCMS
MikeZeDev Jul 11, 2024
c677b41
Merge branch 'master' into HeanCMS
MikeZeDev Jul 28, 2024
308cac8
Merge branch 'master' into HeanCMS
MikeZeDev Aug 14, 2024
e2effeb
Update ModeScanlator.ts
MikeZeDev Aug 14, 2024
6b5f32a
Update HeanCMS.ts
MikeZeDev Aug 14, 2024
33840ac
fix modescanlator tests
MikeZeDev Aug 14, 2024
4aec901
Merge branch 'master' into HeanCMS
MikeZeDev Aug 16, 2024
0679536
remove TempleScan
MikeZeDev Aug 21, 2024
e0fc2fe
Merge branch 'master' into HeanCMS
MikeZeDev Sep 9, 2024
f43ff8d
Merge branch 'master' into HeanCMS
MikeZeDev Sep 13, 2024
32b81f0
Update HeanCMS.ts
MikeZeDev Sep 23, 2024
3b70b4e
Merge branch 'master' into HeanCMS
MikeZeDev Sep 28, 2024
5d1a5a9
remove novel support & multiple fixes
MikeZeDev Sep 28, 2024
1f7c3b0
Merge branch 'master' into HeanCMS
MikeZeDev Oct 7, 2024
adb4d5e
Merge branch 'master' into HeanCMS
MikeZeDev Oct 13, 2024
1f6e066
turn HeanCMS into a template
MikeZeDev Oct 13, 2024
b794e4d
Update PerfScan_e2e.ts
MikeZeDev Oct 13, 2024
5f18aa5
Merge branch 'master' into HeanCMS
MikeZeDev Oct 13, 2024
7249c68
Merge branch 'master' into HeanCMS
MikeZeDev Oct 27, 2024
2613991
update tests
MikeZeDev Oct 27, 2024
7427e07
Merge branch 'master' into HeanCMS
MikeZeDev Nov 3, 2024
5ed3601
add base class tests
MikeZeDev Nov 3, 2024
9a87cb9
remove modescanlator
MikeZeDev Nov 11, 2024
7a9c892
Merge branch 'master' into HeanCMS
MikeZeDev Nov 11, 2024
4ea2e00
Merge branch 'master' into HeanCMS
MikeZeDev Nov 11, 2024
aabfe54
Update HeanCMS_e2e.ts
MikeZeDev Nov 11, 2024
e0c12d3
Merge branch 'master' into HeanCMS
MikeZeDev Nov 25, 2024
f85a000
Merge branch 'master' into HeanCMS
MikeZeDev Dec 15, 2024
a224b21
Update _index.ts
MikeZeDev Dec 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions web/src/engine/websites/OmegaScans.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { Tags } from '../Tags';
import icon from './OmegaScans.webp';
import { HeanCMS } from './templates/HeanCMS';

export default class extends HeanCMS {

public constructor() {
super('omegascans', 'OmegaScans', 'https://omegascans.org', Tags.Media.Manga, Tags.Media.Manhwa, Tags.Media.Manhua, Tags.Media.Novel, Tags.Language.English, Tags.Source.Scanlator, Tags.Rating.Pornographic);
}

public override get Icon() {
return icon;
}
}
Binary file added web/src/engine/websites/OmegaScans.webp
Binary file not shown.
24 changes: 24 additions & 0 deletions web/src/engine/websites/OmegaScans_e2e.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { TestFixture } from '../../../test/WebsitesFixture';

const ComicConfig = {
plugin: {
id: 'omegascans',
title: 'OmegaScans'
},
container: {
url: 'https://omegascans.org/series/trapped-in-the-academys-eroge',
id: JSON.stringify({ id: '8', slug: 'trapped-in-the-academys-eroge' }),
title: `Trapped in the Academy's Eroge`
},
child: {
id: JSON.stringify({ id: '3245', slug: 'chapter-76' }),
title: 'Chapter 76'
},
entry: {
index: 1,
size: 1_577_008,
type: 'image/jpeg'
}
};

new TestFixture(ComicConfig).AssertWebsite();
14 changes: 14 additions & 0 deletions web/src/engine/websites/PerfScan.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { Tags } from '../Tags';
import icon from './PerfScan.webp';
import { HeanCMS } from './templates/HeanCMS';

export default class extends HeanCMS {

public constructor() {
super('perfscan', 'Perf Scan', 'https://perf-scan.fr', Tags.Media.Manga, Tags.Media.Manhwa, Tags.Media.Manhua, Tags.Media.Novel, Tags.Language.French, Tags.Source.Scanlator);
}

public override get Icon() {
return icon;
}
}
Binary file added web/src/engine/websites/PerfScan.webp
Binary file not shown.
24 changes: 24 additions & 0 deletions web/src/engine/websites/PerfScan_e2e.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { TestFixture } from '../../../test/WebsitesFixture';

const ComicConfig = {
plugin: {
id: 'perfscan',
title: 'Perf Scan'
},
container: {
url: 'https://perf-scan.fr/series/martial-peak',
id: JSON.stringify({ id: '1', slug: 'martial-peak' }),
title: 'Martial Peak'
},
child: {
id: JSON.stringify({ id: '28768', slug: 'chapitre-1298' }),
title: 'Chapitre 1298'
},
entry: {
index: 2,
size: 939_807,
type: 'image/jpg'
}
};

new TestFixture(ComicConfig).AssertWebsite();
2 changes: 2 additions & 0 deletions web/src/engine/websites/_index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,7 @@ export { default as Noromax } from './Noromax';
export { default as NovelMic } from './NovelMic';
export { default as NoxScans } from './NoxScans';
export { default as OlympusScanlation } from './OlympusScanlation';
export { default as OmegaScans } from './OmegaScans';
export { default as Opiatoon } from './Opiatoon';
export { default as Oremanga } from './Oremanga';
export { default as OrigamiOrpheans } from './OrigamiOrpheans';
Expand All @@ -503,6 +504,7 @@ export { default as PCNet } from './PCNet';
export { default as PeanuToon } from './PeanuToon';
export { default as PelaTeam } from './PelaTeam';
export { default as Penlab } from './Penlab';
export { default as PerfScan } from './PerfScan';
export { default as PhenixScans } from './PhenixScans';
export { default as PhoenixScansIT } from './PhoenixScansIT';
export { default as Piccoma } from './Piccoma';
Expand Down
125 changes: 125 additions & 0 deletions web/src/engine/websites/templates/HeanCMS.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import { Exception } from '../../Error';
import { FetchJSON } from '../../platform/FetchProvider';
import {type MangaPlugin, Manga, Chapter, Page, DecoratableMangaScraper } from '../../providers/MangaPlugin';
import type { Priority } from '../../taskpool/TaskPool';
import * as Common from '../decorators/Common';
import { WebsiteResourceKey as R } from '../../../i18n/ILocale';

// TODO: Add Novel support

type APIManga = {
title: string
id: number,
series_type: 'Comic' | 'Novel',
series_slug: string,
}

type APIResult<T> = {
data: T
}

type APIChapter = {
index: string,
id: number,
chapter_name: string,
chapter_title: string,
chapter_slug: string,
}

type APIPages = {
chapter_type: 'Comic' | 'Novel',
paywall: boolean,
data: string[] | string
chapter: {
chapter_type: 'Comic' | 'Novel',
storage: string,
chapter_data?: {
images: string[]
}

}
}

type APIMediaID = {
id: string,
slug: string
}

type PageType = {
type: 'Comic' | 'Novel'
}

export class HeanCMS extends DecoratableMangaScraper {

protected apiUrl = this.URI.origin.replace('://', '://api.');

public override ValidateMangaURL(url: string): boolean {
return new RegExpSafe(`^${this.URI.origin}/series/[^/]+$`).test(url);
}

public override async FetchManga(provider: MangaPlugin, url: string): Promise<Manga> {
const slug = new URL(url).pathname.split('/').at(-1);
const { title, series_slug, id } = await FetchJSON<APIManga>(new Request(new URL(`${this.apiUrl}/series/${slug}`)));
return new Manga(this, provider, JSON.stringify({
id: id.toString(),
slug: series_slug
}), title);
}

public override async FetchMangas(provider: MangaPlugin): Promise<Manga[]> {
const mangaList: Manga[] = [];
for (const adult of [true, false]) { //adult flag mean ONLY adult, false mean ONLY all ages... Talk about stupid. Old api ignore that flag
for (let page = 1, run = true; run; page++) {
const mangas = await this.GetMangaFromPage(provider, page, adult);
mangas.length > 0 ? mangaList.push(...mangas) : run = false;
}
}
return mangaList.distinct();//filter in case of old api
}

private async GetMangaFromPage(provider: MangaPlugin, page: number, adult: boolean): Promise<Manga[]> {
const request = new Request(new URL(`${this.apiUrl}/query?perPage=100&page=${page}&adult=${adult}`));
const { data } = await FetchJSON<APIResult<APIManga[]>>(request);
return !data.length ? [] : data.map((manga) => new Manga(this, provider, JSON.stringify({ id: manga.id.toString(), slug: manga.series_slug }), manga.title));
}

public override async FetchChapters(manga: Manga): Promise<Chapter[]> {
const mangaid: APIMediaID = JSON.parse(manga.Identifier);
const { data } = await FetchJSON<APIResult<APIChapter[]>>(new Request(new URL(`${this.apiUrl}/chapter/query?series_id=${mangaid.id}&perPage=9999&page=1`)));
return data.map(chapter => new Chapter(this, manga, JSON.stringify({
id: chapter.id.toString(),
slug: chapter.chapter_slug,
}), `${chapter.chapter_name} ${chapter.chapter_title || ''}`.trim()));
}

public override async FetchPages(chapter: Chapter): Promise<Page<PageType>[]> {
const chapterid: APIMediaID = JSON.parse(chapter.Identifier);
const mangaid: APIMediaID = JSON.parse(chapter.Parent.Identifier);
const data = await FetchJSON<APIPages>(new Request(new URL(`${this.apiUrl}/chapter/${mangaid.slug}/${chapterid.slug}`)));

if (data.paywall) {
throw new Exception(R.Plugin_Common_Chapter_UnavailableError);
}

if (data.chapter.chapter_type.toLowerCase() === 'novel') {
throw new Exception(R.Plugin_HeanCMS_ErrorNovelsNotSupported);
}
//old API vs new
const listImages = data.data && Array.isArray(data.data) ? data.data as string[] : data.chapter.chapter_data.images;
return listImages.map(image => new Page<PageType>(this, chapter, this.ComputePageUrl(image, data.chapter.storage), { type: data.chapter.chapter_type }));
}

private ComputePageUrl(image: string, storage: string): URL {
switch (storage) {
case "s3": return new URL(image);
case "local": return new URL(`${this.apiUrl}/${image}`);
}
}

public override async FetchImage(page: Page<PageType>, priority: Priority, signal: AbortSignal, detectMimeType = true, deProxifyLink = true): Promise<Blob> {
if (page.Parameters?.type === 'Comic') {
return Common.FetchImageAjax.call(this, page, priority, signal, detectMimeType, deProxifyLink);
} else throw new Exception(R.Plugin_HeanCMS_ErrorNovelsNotSupported);
}

}
2 changes: 2 additions & 0 deletions web/src/engine/websites/templates/HeanCMS_e2e.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
import '../OmegaScans_e2e';
import '../PerfScan_e2e';
5 changes: 5 additions & 0 deletions web/src/i18n/ILocale.ts
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,11 @@ export enum WebsiteResourceKey {
Plugin_SheepScanlations_Settings_PasswordInfo = 'Plugin_SheepScanlations_Settings_PasswordInfo',
}

// [SECTION]: Template : HEANCMS
export enum WebsiteResourceKey {
Plugin_HeanCMS_ErrorNovelsNotSupported = 'Plugin_HeanCMS_ErrorNovelsNotSupported'
}

export const VariantResourceKey = {
...TagCategoryResourceKey,
...TagResourceKey,
Expand Down
2 changes: 2 additions & 0 deletions web/src/i18n/locales/en_US.ts
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,8 @@ const translations: VariantResource = {

Plugin_CuuTruyen_Error_NotProcessed: 'This chapter is still processing, please try again later.',

Plugin_HeanCMS_ErrorNovelsNotSupported: 'Novels are not (yet) supported in Hakuneko !',

Plugin_PocketComics_LanguageMismatchError: 'Unable to find manga {0} for selected language {1}',

Plugin_SheepScanlations_Settings_Username: 'Username',
Expand Down
Loading