Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve file handling #1309

Merged
merged 25 commits into from
May 22, 2022
Merged

Improve file handling #1309

merged 25 commits into from
May 22, 2022

Conversation

nagmat84
Copy link
Collaborator

@nagmat84 nagmat84 commented May 7, 2022

Intro and Final Goal

This is the first of a series of four planned PRs which ultimately clean up the way how the MIME type of media files is handled.

  1. Concentrate all file handling in a bunch of "file" classes
    That is this PR.
  2. Use data streams everywhere not file paths
    This PR is already in the pipeline, but not yet ready, see https://github.com/LycheeOrg/Lychee/tree/use_filestreams if curious
  3. Clean up of a lot of source code files and Lychee directory structure
    This does not change any functionality but it is/will be necessary.
  4. Make the live photo the eights size variant and store filesize in database
    This was discussed on Gitter recently.

All these PRs together will also eventually and "accidentally" enable the option to host media files on a remote location such as AWS S3. 🎉 🎊 I decided to split the PRs into several ones to make reviews faster/easier.

This PR

This first PR basically aims at two aspects

  1. Stop passing around file paths as strings
  2. Concentrate checks whether a file type is supported in one place

Aspect 1: Stop passing around file paths as strings

This is crucial in order to get rid of file paths eventually and make the transition to data streams possible. To this end, I introduced a couple of "file classes" (FlysystemFile, NativeLocalFile, etc.) which provide a unified interface for writing to and reading from files. The interface is inspired by Flysystem. Currently, the file paths sill remain to be used under the hood, because our image handlers, ffmpeg extractor, etc., need them, but this will change with the 2nd PR. This first step makes the rest of Lychee using proper file objects as method parameters.

Aspect 2: Concentrate checks whether a file type is supported in one place

Previously, there has been an unfortunate trait Actions\Photo\Extensions\Constants which collects some of the supported MIME types and file extensions. Some classes used this trait, but in different and inconsistent ways. These constants have been moved to a new central point Image\MediaFile.php which is the new parent class for all file classes. The trait Constants has been nuked.

Moreover, a long standing asymmetry has been resolved. Previously, there were constant arrays for supported EXIF image types (but not for videos) and for supported MIME types for videos (but not for images). Now there are arrays for both. This also changes the way how a file is recognized as raw or as a photo. Previously, we replaced the MIME type of files which had been identified as raw by the special keyword 'raw' upon upload. Then a file was recognized as a photo, if it was neither a video nor raw. Now, as we have a list of MIME types for videos and for photos, videos and photos are identified as such, if there MIME type is included in the respective array. Everything else is raw. This means we don't use the special keyword 'raw' anymore and don't loose the information about the real MIME type.

Addon: Type-safe EXIF extractor

The previous Extractor used a non-type-safe array to transport EXIF data. This has been changed, too. The EXIF data are typed properties, now. Accidentally, this fixed a bug which seemingly nobody noticed so far. Matching the video and photo part of an Apple Live photos cannot have worked previously. The extractor used the attribute livePhotoContentID, but we tried to save live_photo_content_id' into the DB. This kind of bugs cannot happen anymore.

@nagmat84 nagmat84 requested review from kamil4 and qwerty287 May 7, 2022 16:59
@nagmat84 nagmat84 force-pushed the improve_file_handling branch from f7c2c3e to f87300e Compare May 7, 2022 17:19
Copy link
Contributor

@qwerty287 qwerty287 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm relatively new to PHP, so don't expect this to be really helpful. It will take some time until I can really help with reviews.

composer.json Outdated Show resolved Hide resolved
app/Image/MediaFile.php Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented May 9, 2022

Codecov Report

Merging #1309 (bffb452) into master (7a0c791) will decrease coverage by 1.44%.
The diff coverage is 63.16%.

@sonarqubecloud
Copy link

sonarqubecloud bot commented May 9, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 14 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

use App\Models\Album;
use App\Models\Configs;

class AddStrategyParameters
{
public ImportMode $importMode;

// Information about intended parent album
/** @var Album|null the intended parent album */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the /** @var Album|null ? Isn't the type in php enough ?

Copy link
Collaborator Author

@nagmat84 nagmat84 May 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the question is meant as "should it be theoretically sufficient to add the type hint to the variable in an optimal world?" Yes, I agree. 😀 Giving the type as a type hint in the code and as an type annotation in the comment introduces unnecessary redundancy.

Unfortunately, this is how PHPdoc expects it. I did not want to loose the original comment and attach it to the variable. So I changed to // into a /**. But in order to make the comment actually appear in the documentation, I had to add the @var keyword.

If one looks at https://docs.phpdoc.org/3.0/guide/references/phpdoc/tags/var.html#var how the syntax is defined, all three components of the @var tag - type, name, description - seem to be optional. But if you use the @var keyword and want to use the description you must add the type and the name, too. Otherwise, PHPdoc thinks that the first word of the description is the type and the second one the name.

Examples:

Does not work:

/** intended parent album */
public ?Album parent

Makes PHpdoc falsely assume that "intended" is the actual type, "parent" the name and "album" the description. 🙄

/** @var intended parent album */
public ?Album parent

Actually works, but occupies too many lines for my personal taste. This tiny, little variable gets more space in the source code than it deserves 😀

/**
 * Intended parent album
 * 
  *@var 
  */
public ?Album parent

This works, too, is a succinct one-liner but at the costs of redundancy

/** @var Album|null Intended parent album */
public ?Album parent

Hence, I decided to got with the last solution, especially as my IDE keeps the type hint and the type annotation in sync for me. 😋

Copy link
Member

@ildyria ildyria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks sane to me.
Need to test it.

Copy link
Contributor

@qwerty287 qwerty287 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested that some time ago already and it was working.

@nagmat84 nagmat84 merged commit eabec2d into master May 22, 2022
@qwerty287 qwerty287 deleted the improve_file_handling branch May 22, 2022 08:15
@nagmat84 nagmat84 mentioned this pull request May 26, 2022
@kamil4 kamil4 mentioned this pull request Jun 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants