Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement incremental compilation #79

Open
MysteryBlokHed opened this issue Jul 21, 2021 · 5 comments
Open

Implement incremental compilation #79

MysteryBlokHed opened this issue Jul 21, 2021 · 5 comments
Assignees
Labels
enhancement New feature or request source Affects the project's source
Milestone

Comments

@MysteryBlokHed
Copy link
Owner

MysteryBlokHed commented Jul 21, 2021

Right now, Databind will recompile every file in a project when built, even though it's unlikely that every file in a project has changed. In large projects, this can be a significant problem. I'm not certain how other incremental compilers make sure to only recompile changed files, but doing something like creating a .databind folder in a project's root with the SHA256 hashes of files could work.

In an example implementation, the following tree:

src
└──data
    └──namespace
        └──functions
               main.databind

would result in the following .databind folder:

.databind
└──src
    └──data
        └──namespace
            └──functions
                   main.databind.info

Before file's signatures are checked, the file modification date should be checked. There's no reason to check the hash of a file if it hasn't been modified. This could be stored in the same file as the hash, resulting in the following contents for main.databind.info:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 1626831659

Where e3b0c442... is the file's hash, and 1626831659 is the file's modification time in seconds.

@MysteryBlokHed MysteryBlokHed added this to the v0.6 milestone Jul 22, 2021
@MysteryBlokHed MysteryBlokHed added enhancement New feature or request source Affects the project's source labels Jul 23, 2021
@MysteryBlokHed

This comment has been minimized.

@MysteryBlokHed MysteryBlokHed removed this from the v0.6 milestone Jul 24, 2021
@MysteryBlokHed
Copy link
Owner Author

MysteryBlokHed commented Jul 27, 2021

With the addition of global macros in #78, it would also be necessary to store information on files with calls to global macros. The best way to do this would probably be to save which files depend on which other files during compilation.

For example, if there's a file !macros.databind with a definition for a macro called hello_world, and there's another file main.databind that calls that macro, then main.databind would depend on !macros.databind. This means that even if main.databind is unchanged, if !macros.databind changes, main.databind needs to be recompiled.

@MysteryBlokHed MysteryBlokHed added this to the Databind 0.7 milestone Aug 7, 2021
@MysteryBlokHed MysteryBlokHed modified the milestone: Databind 0.7 Aug 23, 2021
@MysteryBlokHed
Copy link
Owner Author

MysteryBlokHed commented Aug 23, 2021

A better solution than trying to create a new parser for something like this would be to just use TOML again. A file in the path .databind/files.toml in the root of the project could contain something like the following:

# This file is used by Databind and contains information about files
# and their modification to make compilation faster.
# It should not be modified.
[[file]]
path = "src/data/example/functions/!macros.databind"
hash = "92b772380a3f8e27a93e57e6deeca6c01da07f5aadce78bb2fbb20de10a66925"
modified = 2021-08-23T15:32:53Z
depends_on = []

[[file]]
path = "src/data/example/functions/main.databind"
hash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
modified = 2021-08-23T15:35:47Z
depends_on = ["src/data/example/functions/!macros.databind"]

Ideally, the filepaths will be relative and will all be the same across different platforms.

@MysteryBlokHed MysteryBlokHed self-assigned this Aug 23, 2021
@MysteryBlokHed
Copy link
Owner Author

MysteryBlokHed commented Aug 24, 2021

It's probably easier to store the Unix milliseconds instead of the RFC 3339 date for the modification time to avoid adding more dependencies than needed. The above example would then look like this:

# This file is used by Databind and contains information about files
# and their modification to make compilation faster.
# It should not be modified.
[[file]]
path = "src/data/example/functions/!macros.databind"
hash = "92b772380a3f8e27a93e57e6deeca6c01da07f5aadce78bb2fbb20de10a66925"
modified = 1629747173000
depends_on = []

[[file]]
path = "src/data/example/functions/main.databind"
hash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
modified = 1629747347000
depends_on = ["src/data/example/functions/!macros.databind"]

@MysteryBlokHed
Copy link
Owner Author

I'm probably going to move this up to the (not yet existent) v0.8 milestone so that I can get something released before working on something else that might take a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request source Affects the project's source
Projects
None yet
Development

No branches or pull requests

1 participant