Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: アバターをサーバーで圧縮して取得する #8337

Closed
wants to merge 46 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
4d4ac5d
wip
mei23 Jan 22, 2022
bb11a37
Update packages/client/src/os.ts
mei23 Jan 29, 2022
ed5805e
メニューをComposition API化、switchアイテム追加
tamaina Jan 29, 2022
0a788d1
メニュー型定義を分離 (TypeScriptの型支援が効かないので)
tamaina Jan 29, 2022
8da8329
disabled
tamaina Jan 29, 2022
bb67923
make keepOriginal to follow setting value
tamaina Jan 29, 2022
0ed6f4d
Merge branch 'v12-8173' into better-8176
tamaina Jan 29, 2022
323d020
:v:
tamaina Jan 29, 2022
0802306
fix
tamaina Jan 29, 2022
32dd3bf
fix
tamaina Jan 29, 2022
1ce9e92
:v:
tamaina Jan 29, 2022
8fcf86c
Merge branch 'menu-switch' into better-8176
tamaina Jan 29, 2022
3236eba
WEBP
tamaina Jan 29, 2022
35845bc
aaa
tamaina Jan 29, 2022
0f8f7de
:v:
tamaina Jan 29, 2022
8a32ced
webp
tamaina Jan 29, 2022
38f84a9
lazy load browser-image-resizer
tamaina Jan 29, 2022
4fdec40
rename
tamaina Jan 29, 2022
d42c985
rename 2
tamaina Jan 29, 2022
af42693
Fix
tamaina Jan 29, 2022
0133902
Merge branch 'menu-switch' into better-8176
tamaina Jan 29, 2022
8a5fdd1
clean up
tamaina Jan 29, 2022
7b4f5ac
add comment
tamaina Jan 29, 2022
2b7ec30
Merge branch 'menu-switch' into better-8176
tamaina Jan 29, 2022
4905299
Merge branch 'develop' into better-8176
tamaina Jan 30, 2022
39f0eb1
clean up
tamaina Jan 30, 2022
6a000b3
jpeg, pngにもどす
tamaina Jan 30, 2022
599510c
fix
tamaina Jan 30, 2022
ee6e163
fix name
tamaina Feb 4, 2022
e87e97f
Merge branch 'develop' into better-8176
tamaina Feb 4, 2022
e7d9221
webpでなくする ただしサムネやプレビューはwebpのまま (テスト)
tamaina Feb 6, 2022
fdb17e0
動画サムネイルはjpegに
tamaina Feb 6, 2022
7102e43
エラーハンドリング
tamaina Feb 6, 2022
c015001
Merge branch 'develop' into better-8176
tamaina Feb 16, 2022
8979dbc
:v:
tamaina Feb 16, 2022
211bdb0
v2.2.1-misskey-beta.2
tamaina Feb 16, 2022
40dfef9
browser-image-resizer#v2.2.1-misskey.1
tamaina Feb 16, 2022
60d7d03
:v:
tamaina Feb 16, 2022
673361f
fix alert
tamaina Feb 19, 2022
50b7ef7
Merge branch 'develop' into better-8176
tamaina Feb 19, 2022
05c011a
Merge branch 'develop' into better-8176
tamaina Feb 19, 2022
3401cf8
Merge branch 'develop' into better-8176
tamaina Feb 20, 2022
8f4c9ad
:v:
tamaina Feb 20, 2022
3f0b914
fix mention
tamaina Feb 20, 2022
db6acc9
静止画像の場合はstaticでないURLにリダイレクトしてキャッシュヒットさせる
tamaina Feb 20, 2022
105e6d0
acctのリクエストはuserIdへリダイレクトするように
tamaina Feb 20, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/backend/src/misc/populate-emojis.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ export async function populateEmoji(emojiName: string, noteUserHost: string | nu

const isLocal = emoji.host == null;
const emojiUrl = emoji.publicUrl || emoji.originalUrl; // || emoji.originalUrl してるのは後方互換性のため
const url = isLocal ? emojiUrl : `${config.url}/proxy/image.png?${query({ url: emojiUrl })}`;
const url = isLocal ? emojiUrl : `${config.url}/proxy/${encodeURIComponent((new URL(emojiUrl)).pathname)}?${query({ url: emojiUrl })}`;

return {
name: emojiName,
Expand Down
6 changes: 5 additions & 1 deletion packages/backend/src/models/repositories/user.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,10 +185,14 @@ export class UserRepository extends Repository<User> {
if (user.avatarUrl) {
return user.avatarUrl;
} else {
return `${config.url}/identicon/${user.id}`;
return this.getIdenticonUrl(user);
}
}

public getIdenticonUrl(user: User): string {
return `${config.url}/identicon/${user.id}`;
}

Comment on lines -188 to +195
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why moving the same code into a separate function is necessary? Since its only one line, at most I think a comment would make sense here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used at

return ctx.redirect(Users.getIdenticonUrl(user));

public async pack<ExpectsMe extends boolean | null = null, D extends boolean = false>(
src: User['id'] | User,
me?: { id: User['id'] } | null | undefined,
Expand Down
8 changes: 3 additions & 5 deletions packages/backend/src/server/file/send-drive-file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { DriveFiles } from '@/models/index';
import { InternalStorage } from '@/services/drive/internal-storage';
import { downloadUrl } from '@/misc/download-url';
import { detectType } from '@/misc/get-file-info';
import { convertToJpeg, convertToPng, convertToPngOrJpeg } from '@/services/drive/image-processor';
import { convertToWebp, convertToJpeg, convertToPng } from '@/services/drive/image-processor';
import { GenerateVideoThumbnail } from '@/services/drive/generate-video-thumbnail';
import { StatusError } from '@/misc/fetch';
import { FILE_TYPE_BROWSERSAFE } from '@/const';
Expand Down Expand Up @@ -65,10 +65,8 @@ export default async function(ctx: Koa.Context) {

const convertFile = async () => {
if (isThumbnail) {
if (['image/jpeg', 'image/webp'].includes(mime)) {
return await convertToJpeg(path, 498, 280);
} else if (['image/png', 'image/svg+xml'].includes(mime)) {
return await convertToPngOrJpeg(path, 498, 280);
if (['image/jpeg', 'image/webp', 'image/png', 'image/svg+xml'].includes(mime)) {
return await convertToWebp(path, 498, 280);
} else if (mime.startsWith('video/')) {
return await GenerateVideoThumbnail(path);
}
Expand Down
19 changes: 2 additions & 17 deletions packages/backend/src/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import { UserProfiles, Users } from '@/models/index';
import { genIdenticon } from '@/misc/gen-identicon';
import { createTemp } from '@/misc/create-temp';
import { publishMainStream } from '@/services/stream';
import * as Acct from 'misskey-js/built/acct';

export const serverLogger = new Logger('server', 'gray', false);

Expand Down Expand Up @@ -56,7 +55,8 @@ if (config.url.startsWith('https') && !config.disableHsts) {

app.use(mount('/api', apiServer));
app.use(mount('/files', require('./file')));
app.use(mount('/proxy', require('./proxy')));
app.use(mount('/proxy', require('./proxy/proxy-media')));
app.use(mount('/avatar.webp', require('./proxy/avatar')));

// Init router
const router = new Router();
Expand All @@ -66,21 +66,6 @@ router.use(activityPub.routes());
router.use(nodeinfo.routes());
router.use(wellKnown.routes());

router.get('/avatar/@:acct', async ctx => {
const { username, host } = Acct.parse(ctx.params.acct);
const user = await Users.findOne({
usernameLower: username.toLowerCase(),
host: host === config.host ? null : host,
isSuspended: false,
});

if (user) {
ctx.redirect(Users.getAvatarUrl(user));
} else {
ctx.redirect('/static-assets/user-unknown.png');
}
});

router.get('/identicon/:x', async ctx => {
const [temp] = await createTemp();
await genIdenticon(ctx.params.x, fs.createWriteStream(temp));
Expand Down
128 changes: 128 additions & 0 deletions packages/backend/src/server/proxy/avatar.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
import * as Koa from 'koa';
import { serverLogger } from '../index';
import { createTemp } from '@/misc/create-temp';
import { downloadUrl } from '@/misc/download-url';
import { detectType } from '@/misc/get-file-info';
import { StatusError } from '@/misc/fetch';
import { FILE_TYPE_BROWSERSAFE } from '@/const';
import * as Acct from 'misskey-js/built/acct';
import { Users } from '@/models/index';
import config from '@/config/index';
import * as sharp from 'sharp';
import { User } from '@/models/entities/user';
import * as cors from '@koa/cors';
import * as Router from '@koa/router';

const logger = serverLogger.createSubLogger('proxy-avatar', 'yellow');

// Init app
const app = new Koa();
app.use(cors());
app.use(async (ctx, next) => {
ctx.set('Content-Security-Policy', `default-src 'none'; img-src 'self'; media-src 'self'; style-src 'unsafe-inline'`);
await next();
});

// Init router
const router = new Router();

router.get('(.*)', proxyAvatar);

// Register router
app.use(router.routes());

module.exports = app;

export async function proxyAvatar(ctx: Koa.Context) {
let user: User | undefined;

if (ctx.query.acct) {
if (Array.isArray(ctx.query.acct)) {
logger.info(`acct is duplicated`);
return ctx.redirect('/static-assets/user-unknown.png');
};
const { username, host } = Acct.parse(ctx.query.acct);
user = await Users.findOne({
usernameLower: username.toLowerCase(),
host: host === config.host ? null : host,
isSuspended: false,
});
} else if (ctx.query.userId) {
if (Array.isArray(ctx.query.userId)) {
logger.info(`userId is duplicated`);
return ctx.redirect('/static-assets/user-unknown.png');
};
user = await Users.findOne(ctx.query.userId);
}

if (!user || user.isSuspended) {
logger.info(`user is not found or is suspended: ${ctx.query}`);
return ctx.redirect('/static-assets/user-unknown.png');
}
if (!user.avatarUrl) return ctx.redirect(Users.getIdenticonUrl(user));
if (ctx.query.acct || ctx.query.url !== user.avatarUrl) {
// 最新の、キャッシュすべきURLへリダイレクト
logger.info(`redirect`);
const url = new URL(ctx.URL);
url.searchParams.set('url', user.avatarUrl);
if (ctx.query.acct) {
url.searchParams.delete('acct');
url.searchParams.set('userId', user.id);
}
return ctx.redirect(url.toString());
}

// Create temp file
const [path, cleanup] = await createTemp();

try {
await downloadUrl(user.avatarUrl, path);

const { mime, ext } = await detectType(path);

if (!mime.startsWith('image/') || !FILE_TYPE_BROWSERSAFE.includes(mime)) {
cleanup();
return ctx.redirect(Users.getIdenticonUrl(user));
}

//#region If image is not animated, redirect to non static url
const metadata = await sharp(path).metadata();
const isAnimated = metadata.pages && metadata.pages > 1;
if (ctx.query.static && !isAnimated) {
logger.info(`redirect to non static url`);
cleanup();
const url = new URL(ctx.URL);
url.searchParams.delete('static');
ctx.status = 301;
ctx.redirect(url.toString());
return;
}
//#endregion

ctx.set('Content-Type', 'image/webp');
ctx.set('Cache-Control', 'max-age=31536000, immutable');
Copy link
Contributor

@Johann150 Johann150 Feb 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since avatars can change multiple times in the span of a year I think this caching is a bit too agressive. I think something between a week and a month might make more sense? Or remove immutable because that is not the case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ここに至るまでに、クエリ文字列にurlを持たない場合はリダイレクトされるはずなのですが、どうでしょうか。


ctx.body = await sharp(path, {
pages: 'static' in ctx.query ? 256 : 1,
})
.resize(256, 256, {
fit: 'cover',
withoutEnlargement: true,
})
.rotate()
.webp({
quality: 95,
})
.toBuffer();
} catch (e) {
serverLogger.error(`${e}`);

if (e instanceof StatusError && e.isClientError) {
ctx.status = e.statusCode;
} else {
ctx.status = 500;
}
} finally {
cleanup();
}
}
26 changes: 0 additions & 26 deletions packages/backend/src/server/proxy/index.ts

This file was deleted.

30 changes: 25 additions & 5 deletions packages/backend/src/server/proxy/proxy-media.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,34 @@
import * as fs from 'fs';
import * as Koa from 'koa';
import { serverLogger } from '../index';
import { IImage, convertToPng, convertToJpeg } from '@/services/drive/image-processor';
import { IImage, convertToWebp } from '@/services/drive/image-processor';
import { createTemp } from '@/misc/create-temp';
import { downloadUrl } from '@/misc/download-url';
import { detectType } from '@/misc/get-file-info';
import { StatusError } from '@/misc/fetch';
import { FILE_TYPE_BROWSERSAFE } from '@/const';
import * as cors from '@koa/cors';
import * as Router from '@koa/router';

export async function proxyMedia(ctx: Koa.Context) {
// Init app
const app = new Koa();
app.use(cors());
app.use(async (ctx, next) => {
ctx.set('Content-Security-Policy', `default-src 'none'; img-src 'self'; media-src 'self'; style-src 'unsafe-inline'`);
await next();
});

// Init router
const router = new Router();

router.get('/:url*', proxyMedia);

// Register router
app.use(router.routes());

module.exports = app;

async function proxyMedia(ctx: Koa.Context) {
const url = 'url' in ctx.query ? ctx.query.url : 'https://' + ctx.params.url;

if (typeof url !== 'string') {
Expand All @@ -27,11 +47,11 @@ export async function proxyMedia(ctx: Koa.Context) {
let image: IImage;

if ('static' in ctx.query && ['image/png', 'image/gif', 'image/apng', 'image/vnd.mozilla.apng', 'image/webp', 'image/svg+xml'].includes(mime)) {
image = await convertToPng(path, 498, 280);
image = await convertToWebp(path, 498, 280);
} else if ('preview' in ctx.query && ['image/jpeg', 'image/png', 'image/gif', 'image/apng', 'image/vnd.mozilla.apng', 'image/svg+xml'].includes(mime)) {
image = await convertToJpeg(path, 200, 200);
image = await convertToWebp(path, 200, 200);
} else if (['image/svg+xml'].includes(mime)) {
image = await convertToPng(path, 2048, 2048);
image = await convertToWebp(path, 2048, 2048, 1);
} else if (!mime.startsWith('image/') || !FILE_TYPE_BROWSERSAFE.includes(mime)) {
throw new StatusError('Rejected type', 403, 'Rejected type');
} else {
Expand Down
2 changes: 1 addition & 1 deletion packages/backend/src/server/web/url-preview.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ module.exports = async (ctx: Koa.Context) => {
function wrap(url?: string): string | null {
return url != null
? url.match(/^https?:\/\//)
? `${config.url}/proxy/preview.jpg?${query({
? `${config.url}/proxy/preview.webp?${query({
url,
preview: '1',
})}`
Expand Down
29 changes: 18 additions & 11 deletions packages/backend/src/services/drive/add-file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { deleteFile } from './delete-file';
import { fetchMeta } from '@/misc/fetch-meta';
import { GenerateVideoThumbnail } from './generate-video-thumbnail';
import { driveLogger } from './logger';
import { IImage, convertSharpToJpeg, convertSharpToWebp, convertSharpToPng, convertSharpToPngOrJpeg } from './image-processor';
import { IImage, convertSharpToJpeg, convertSharpToWebp, convertSharpToPng } from './image-processor';
import { contentDisposition } from '@/misc/content-disposition';
import { getFileInfo } from '@/misc/get-file-info';
import { DriveFiles, DriveFolders, Users, Instances, UserProfiles } from '@/models/index';
Expand Down Expand Up @@ -178,6 +178,7 @@ export async function generateAlts(path: string, type: string, generateWeb: bool
}

let img: sharp.Sharp | null = null;
let satisfyWebpublic: boolean;

try {
img = sharp(path);
Expand All @@ -191,6 +192,13 @@ export async function generateAlts(path: string, type: string, generateWeb: bool
thumbnail: null,
};
}

satisfyWebpublic = !!(
type !== 'image/svg+xml' && type !== 'image/webp' &&
!(metadata.exif || metadata.iptc || metadata.xmp || metadata.tifftagPhotoshop) &&
metadata.width && metadata.width <= 2048 &&
metadata.height && metadata.height <= 2048
);
} catch (err) {
logger.warn(`sharp failed: ${err}`);
return {
Expand All @@ -202,15 +210,15 @@ export async function generateAlts(path: string, type: string, generateWeb: bool
// #region webpublic
let webpublic: IImage | null = null;

if (generateWeb) {
if (generateWeb && !satisfyWebpublic) {
logger.info(`creating web image`);

try {
if (['image/jpeg'].includes(type)) {
if (['image/jpeg', 'image/webp'].includes(type)) {
webpublic = await convertSharpToJpeg(img, 2048, 2048);
} else if (['image/webp'].includes(type)) {
webpublic = await convertSharpToWebp(img, 2048, 2048);
} else if (['image/png', 'image/svg+xml'].includes(type)) {
} else if (['image/png'].includes(type)) {
webpublic = await convertSharpToPng(img, 2048, 2048);
} else if (['image/svg+xml'].includes(type)) {
webpublic = await convertSharpToPng(img, 2048, 2048);
} else {
logger.debug(`web image not created (not an required image)`);
Expand All @@ -219,18 +227,17 @@ export async function generateAlts(path: string, type: string, generateWeb: bool
logger.warn(`web image not created (an error occured)`, err as Error);
}
} else {
logger.info(`web image not created (from remote)`);
if (satisfyWebpublic) logger.info(`web image not created (original satisfies webpublic)`);
else logger.info(`web image not created (from remote)`);
}
// #endregion webpublic

// #region thumbnail
let thumbnail: IImage | null = null;

try {
if (['image/jpeg', 'image/webp'].includes(type)) {
thumbnail = await convertSharpToJpeg(img, 498, 280);
} else if (['image/png', 'image/svg+xml'].includes(type)) {
thumbnail = await convertSharpToPngOrJpeg(img, 498, 280);
if (['image/jpeg', 'image/webp', 'image/png', 'image/svg+xml'].includes(type)) {
thumbnail = await convertSharpToWebp(img, 498, 280);
} else {
logger.debug(`thumbnail not created (not an required file)`);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ export async function GenerateVideoThumbnail(path: string): Promise<IImage> {

const outPath = `${outDir}/output.png`;

// JPEGに変換 (Webpでもいいが、MastodonはWebpをサポートせず表示できなくなる)
const thumbnail = await convertToJpeg(outPath, 498, 280);

// cleanup
Expand Down
Loading