This repository has been archived by the owner on Oct 1, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Basic setup License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Basic migrator functionality License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Strict mode License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Linting License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Working CLI License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Improving CLI usablity License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Dropping migration class interface License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Improving error handling and other tweaks of migrations License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Refactoring to drop ipfs-repo in favor of datastore License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Refactor migrator to pass only repo's path into migration and not datastore instance. License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * 'add' command for CLI License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Linting License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Writing proper README License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Revert tests License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Migrate tests License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * More tests License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Tests for lock.js and version.js License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Description of tests License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Update interface-datastore dependency License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Fixing package-lock with correct interface-datastore version License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Update wording of CLI reporter License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Correct doctype for optional parameters License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Exposing repo.getVersion License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Improving verification of reversibility License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Adding check if repo is initialized License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Renaming logging key License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Adding parameter ignoreLock License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Adding parameter for options License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Adding documentation for ignoreLock and options parameters License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Instructions on integration with js-ipfs License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Updating dependencies License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Implementing Error's codes License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Refactoring into options object License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Merging revert and migrate CLI commands License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Extanding README with usage and other details License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Integration test License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Removing error message assertions License: MIT Signed-off-by: Adam Uhlir <uhlir.a@gmail.com> * Removing package.json for migrations * Adding basic sharness tests * Removing 'reversible' flag from migration's API * Renaming progressCb parameter to onProgress * Removing package.json mentions and discussing migration's dependencies * Linting * Sharness tests * Non-throw error handling for getLatestMigrationVersion * Check if migration exist for migrate() * Fixing integration tests * Grammar check Co-Authored-By: dirkmc <dirkmdev@gmail.com> * Info about versioning * Grammar fixes Co-Authored-By: dirkmc <dirkmdev@gmail.com> * Messages tweak Co-Authored-By: dirkmc <dirkmdev@gmail.com> * Default paramaters Co-Authored-By: dirkmc <dirkmdev@gmail.com> * Tweaks * fix: update functions signature in templates * doc: CI/codecov/dependencies badges * style: lint fix * Message wordings Co-Authored-By: dirkmc <dirkmdev@gmail.com> * fix: tweaks * style: package json lint fix * fix: update add command * feat: utils for properly getting Datastore and its options * feat: when error occures save last succesfull migration version * style: lint * fix: apply review suggestions Co-Authored-By: David Dias <daviddias.p@gmail.com>
- Loading branch information
Showing
38 changed files
with
21,002 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| [*] | ||
| indent_style = space | ||
| indent_size = 2 | ||
| trim_trailing_whitespace = true | ||
| insert_final_newline = true | ||
| end_of_line = lf |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| * text=auto | ||
| test/test-repo/** text eol=lf |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| docs | ||
|
|
||
| .eslintrc | ||
|
|
||
| # Logs | ||
| logs | ||
| *.log | ||
|
|
||
| # Runtime data | ||
| pids | ||
| *.pid | ||
| *.seed | ||
|
|
||
| # Directory for instrumented libs generated by jscoverage/JSCover | ||
| lib-cov | ||
|
|
||
| # Coverage directory used by tools like istanbul | ||
| coverage | ||
|
|
||
| # node-waf configuration | ||
| .lock-wscript | ||
|
|
||
| build | ||
| dist | ||
|
|
||
| # Dependency directory | ||
| node_modules | ||
|
|
||
| # Tests | ||
| test/test-repo-for* | ||
| test/sharness/tmp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| language: node_js | ||
| cache: npm | ||
| stages: | ||
| - check | ||
| - test | ||
| - cov | ||
|
|
||
| node_js: | ||
| - '10' | ||
|
|
||
| os: | ||
| - linux | ||
| - osx | ||
| - windows | ||
|
|
||
| script: npx nyc -s npm run test:node -- --bail | ||
| after_success: npx nyc report --reporter=text-lcov > coverage.lcov && npx codecov | ||
|
|
||
| jobs: | ||
| include: | ||
| - stage: check | ||
| script: | ||
| - npx aegir commitlint --travis | ||
| - npx aegir dep-check | ||
| - npm run lint | ||
|
|
||
| - stage: test | ||
| name: chrome | ||
| addons: | ||
| chrome: stable | ||
| script: npx aegir test -t browser | ||
|
|
||
| - stage: test | ||
| name: firefox | ||
| addons: | ||
| firefox: latest | ||
| script: npx aegir test -t browser -- --browsers FirefoxHeadless | ||
|
|
||
| notifications: | ||
| email: false | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,277 @@ | ||
| # js-ipfs-migrator | ||
| # Migration tool for JS IPFS Repo | ||
|
|
||
| [](https://travis-ci.com/ipfs/js-ipfs-repo-migrations) | ||
| [](https://codecov.io/gh/ipfs/js-ipfs-repo-migrations) | ||
| [](https://david-dm.org/ipfs/js-ipfs-repo-migrations) | ||
| [](http://ipn.io) | ||
| [](http://ipfs.io/) | ||
| [](http://webchat.freenode.net/?channels=%23ipfs) | ||
| [](https://github.com/RichardLitt/standard-readme) | ||
| [](https://github.com/feross/standard) | ||
|  | ||
|  | ||
|
|
||
| > Migration framework for versioning of JS IPFS Repo | ||
| This package is inspired by the [go-ipfs repo migration tool](https://github.com/ipfs/fs-repo-migrations/) | ||
|
|
||
| ## Lead Maintainer | ||
|
|
||
| [Adam Uhlíř](https://github.com/auhau/) | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Background](#background) | ||
| - [Install](#install) | ||
| - [npm](#npm) | ||
| - [Use in Node.js](#use-in-nodejs) | ||
| - [Use in a browser with browserify, webpack or any other bundler](#use-in-a-browser-with-browserify-webpack-or-any-other-bundler) | ||
| - [Use in a browser Using a script tag](#use-in-a-browser-using-a-script-tag) | ||
| - [Usage](#usage) | ||
| - [Writing migration](#writing-migration) | ||
| - [Migrations matrix](#migrations-matrix) | ||
| - [API](#api) | ||
| - [CLI](#cli) | ||
| - [Versioning](#versioning) | ||
| - [Contribute](#contribute) | ||
| - [License](#license) | ||
|
|
||
| ## Background | ||
|
|
||
|
|
||
| As js-ipfs evolves and new technologies, algorithms and data structures are incorporated it is necessary to | ||
| enable users to transition between versions. Different versions of js-ipfs may expect a different IPFS repo structure or content (see: [IPFS repo spec](https://github.com/ipfs/specs/tree/master/repo), [JS implementation](https://github.com/ipfs/js-ipfs-repo) ). | ||
| So the IPFS repo is versioned, and this package provides a framework to create migrations to transition | ||
| from one version of IPFS repo to the next/previous version. | ||
|
|
||
| This framework: | ||
| * Handles locking/unlocking of repository | ||
| * Defines migrations API | ||
| * Executes and reports migrations in both directions: forward and backward | ||
| * Simplifies creation of new migrations | ||
| * Works on the browser too! | ||
|
|
||
| ## Install | ||
|
|
||
| ### npm | ||
|
|
||
| ```sh | ||
| > npm install ipfs-repo-migrations | ||
| ``` | ||
|
|
||
| ### Use in Node.js | ||
|
|
||
| ```js | ||
| const migrations = require('ipfs-repo-migrations') | ||
| ``` | ||
|
|
||
| ### Use in a browser with browserify, webpack or any other bundler | ||
|
|
||
| ```js | ||
| const migrations = require('ipfs-repo-migrations') | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| Example: | ||
|
|
||
| ```js | ||
| const migrations = require('ipfs-repo-migrations') | ||
| const getVersion = require('ipfs-repo-migrations/repo/version') | ||
|
|
||
| const repoPath = 'some/repo/path' | ||
| const repoVersion = await getVersion(repoPath) | ||
|
|
||
| if(repoVersion < migrations.getLatestMigrationVersion()){ | ||
| // Old repo! Lets migrate to latest version! | ||
| await migrations.migrate(repoPath) | ||
| } | ||
| ``` | ||
|
|
||
| To migrate your repository using the CLI, see the [how to run migrations](./run.md) tutorial. | ||
|
|
||
| ## API | ||
|
|
||
| ### `.migrate(path, {toVersion, ignoreLock, repoOptions, onProgress, isDryRun}) -> Promise<void>` | ||
|
|
||
| Executes a forward migration to a specific version, or to the latest version if a specific version is not specified. | ||
|
|
||
| **Arguments:** | ||
|
|
||
| * `path` (string, mandatory) - path to the repo to be migrated | ||
| * `options` (object, optional) - options for the migration | ||
| * `options.toVersion` (int, optional) - version to which the repo should be migrated. Defaults to the latest migration version. | ||
| * `options.ignoreLock` (bool, optional) - if true will not lock the repo when applying migrations. Use with caution. | ||
| * `options.repoOptions` (object, optional) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo). | ||
| * `options.onProgress` (function, optional) - callback that is called after finishing execution of each migration to report progress. | ||
| * `options.isDryRun` (bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes. | ||
|
|
||
| #### `onProgress(migration, counter, totalMigrations)` | ||
|
|
||
| Signature of the progress callback. | ||
|
|
||
| **Arguments:** | ||
| * `migration` (object) - object of migration that just successfully finished running. See [Architecture of migrations](#architecture-of-migrations) for details. | ||
| * `counter` (int) - index of current migration. | ||
| * `totalMigrations` (int) - total count of migrations that will be run. | ||
|
|
||
| ### `.revert(path, toVersion, {ignoreLock, options, onProgress, isDryRun}) -> Promise<void>` | ||
|
|
||
| Executes backward migration to a specific version. | ||
|
|
||
| **Arguments:** | ||
|
|
||
| * `path` (string, mandatory) - path to the repo to be reverted | ||
| * `toVersion` (int, mandatory) - version to which the repo should be reverted to. | ||
| * `options` (object, optional) - options for the reversion | ||
| * `options.ignoreLock` (bool, optional) - if true will not lock the repo when applying migrations. Use with caution. | ||
| * `options.options` (object, optional) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo). | ||
| * `options.onProgress` (function, optional) - callback that is called after finishing execution of each migration to report progress. | ||
| * `options.isDryRun` (bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes. | ||
|
|
||
| ### `getLatestMigrationVersion() -> int` | ||
|
|
||
| Return the version of the latest migration. | ||
|
|
||
| ## CLI | ||
|
|
||
| The CLI is a NodeJS binary named `jsipfs-repo-migrations`. | ||
| It has several commands: | ||
|
|
||
| * `migrate` - performs forward/backward migration to specific or latest version. | ||
| * `status` - check repo for migrations that should be run. | ||
| * `add` - bootstraps new migration. | ||
|
|
||
| For further details see the `--help` pages. | ||
|
|
||
| ## Creating a new migration | ||
|
|
||
| Migrations are one of those things that can be extremely painful on users. At the end of the day, we want users never to have to think about it. The process should be: | ||
|
|
||
| - SAFE. No data lost. Ever. | ||
| - Revertible. Tools must implement forward and backward (if possible) migrations. | ||
| - Tests. Migrations have to be well tested. | ||
| - To Spec. The tools must conform to the spec. | ||
|
|
||
| If your migration has several parts, it should be fail-proof enough that if one part of migration fails the previous changes | ||
| are reverted before propagating the error. If possible then the outcome should be consistent repo so it migration could | ||
| be run again. | ||
|
|
||
| ### Architecture of a migration | ||
|
|
||
| All migrations are placed in the `/migrations` folder. Each folder there represents one migration that follows the migration | ||
| API. | ||
|
|
||
| All migrations are collected in `/migrations/index.js`, which should not be edited manually. It is regenerated on | ||
| every run of `jsipfs-migrations add` (manual changes should follow the same style of modifications). | ||
| **The order of migrations is important and migrations must be sorted in ascending order**. | ||
|
|
||
| Each migration must follow this API. It must export an object in its `index.js` that has following properties: | ||
|
|
||
| * `version` (int) - Number that represents the version which the repo will migrate to (eg. `migration-8` will move the repo to version 8). | ||
| * `description` (string) - Brief description of what the migrations does. | ||
| * `migrate` (function) - Function that performs the migration (see signature of this function below) | ||
| * `revert` (function) - If defined then this function will revert the migration to the previous version. Otherwise it is assumed that it is not possible to revert this migration. | ||
|
|
||
| #### `.migrate(repoPath, isBrowser)` | ||
|
|
||
| _Do not confuse this function with the `require('ipfs-repo-migrations').migrate()` function that drives the whole migration process!_ | ||
|
|
||
| Arguments: | ||
| * `repoPath` (string) - absolute path to the root of the repo | ||
| * `options` (object, optional) - object containing `IPFSRepo` options, that should be used to construct a datastore instance. | ||
| * `isBrowser` (bool) - indicates if the migration is run in a browser environment (as opposed to NodeJS) | ||
|
|
||
| #### `.revert(repoPath, isBrowser)` | ||
|
|
||
| _Do not confuse this function with the `require('ipfs-repo-migrations').revert()` function that drives the whole backward migration process!_ | ||
|
|
||
| Arguments: | ||
| * `repoPath` (string) - path to the root of the repo | ||
| * `options` (object, optional) - object containing `IPFSRepo` options, that should be used to construct the datastore instance. | ||
| * `isBrowser` (bool) - indicates if the migration is run in a browser environment (as opposed to NodeJS) | ||
|
|
||
| ### Browser vs. NodeJS environments | ||
|
|
||
| The migration might need to distinguish in which environment it runs (browser vs. NodeJS). For this reason there is an argument | ||
| `isBrowser` passed to migrations functions. But with simple migrations it should not be necessary to distinguish between | ||
| these environments as the datastore implementation will handle the main differences. | ||
|
|
||
| There are currently two main datastore implementations: | ||
| 1. [`datastore-fs`](https://github.com/ipfs/js-datastore-fs) that is backed by file system and is used mainly in the NodeJS environment | ||
| 2. [`datastore-level`](https://github.com/ipfs/js-datastore-level) that is backed by LevelDB and is used mainly in the browser environment | ||
|
|
||
| Both implementations share the same API and hence are interchangeable. | ||
|
|
||
| When the migration is run in a browser environment, `datastore-fs` is automatically replaced with `datastore-level` even | ||
| when it is directly imported (`require('datastore-fs')` will return `datastore-level` in a browser). | ||
| So with simple migrations you shouldn't worry about the difference between `datastore-fs` and `datastore-level` | ||
| and by default use the `datastore-fs` package (as the replace mechanism does not work vice versa). | ||
|
|
||
| ### Guidelines | ||
|
|
||
| The recommended way to write a new migration is to first bootstrap a dummy migration using the CLI: | ||
|
|
||
| ```sh | ||
| > npm run new-migration | ||
| ``` | ||
|
|
||
| A new folder is created with the bootstrapped migration. You can then simply fill in the required fields and | ||
| write the rest of the migration! | ||
|
|
||
| ### Integration with js-ipfs | ||
|
|
||
| When a new migration is created, the repo version in [`js-ipfs-repo`](https://github.com/ipfs/js-ipfs-repo) should be updated with the new version, | ||
| together with updated version of this package. Then the updated version should be propagated to `js-ipfs`. | ||
|
|
||
| ### Tests | ||
|
|
||
| If a migration affects any of the following functionality, it must provide tests for the following functions | ||
| to work under the version of the repo that it migrates to: | ||
|
|
||
| * `/src/repo/version.js`:`getVersion()` - retrieving repository's version | ||
| * `/src/repo/lock.js`:`lock()` - locking repository that uses file system | ||
| * `/src/repo/lock-memory.js`:`lock()` - locking repository that uses memory | ||
|
|
||
| Every migration must have test coverage. Tests for migrations should be placed in the `/test/migrations/` folder. Most probably | ||
| you will have to plug the tests into `browser.js`/`node.js` if they require specific bootstrapping on each platform. | ||
|
|
||
| ### Empty migrations | ||
|
|
||
| For interop with go-ipfs it might be necessary just to bump a version of a repo without any actual | ||
| modification as there might not be any changes needed in the JS implementation. For that purpose you can create an "empty migration". | ||
|
|
||
| The easiest way to do so is with the CLI: | ||
|
|
||
| ```sh | ||
| > npm run new-migration -- --empty | ||
| ``` | ||
|
|
||
| This will create an empty migration with the next version. | ||
|
|
||
| ### Migrations matrix | ||
|
|
||
| | IPFS repo version | JS IPFS version | | ||
| | -----------------: |:----------------:| | ||
| | 7 | v0.0.0 - latest | | ||
|
|
||
| ## Developer | ||
|
|
||
| ### Module versioning notes | ||
|
|
||
| In order to have good overview of what version of package contains what kind of migrations, this package follows this versioning schema: `0.<versionOfLastMigration>.<patches>`. | ||
|
|
||
| ## Contribute | ||
|
|
||
| There are some ways you can make this module better: | ||
|
|
||
| - Consult our [open issues](https://github.com/ipfs/js-ipfs-repo/issues) and take on one of them | ||
| - Help our tests reach 100% coverage! | ||
|
|
||
| This repository falls under the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md). | ||
|
|
||
| [](https://github.com/ipfs/community/blob/master/contributing.md) | ||
|
|
||
| ## License | ||
|
|
||
| [MIT](LICENSE) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| 'use strict' | ||
|
|
||
| // Do not modify this file manually as it will be overridden when running 'add' CLI command. | ||
| // Modify migration-templates.js file | ||
|
|
||
| const emptyMigration = { | ||
| description: 'Empty migration.', | ||
| migrate: () => {}, | ||
| revert: () => {}, | ||
| empty: true, | ||
| } | ||
|
|
||
| module.exports = [ | ||
| ] |
Oops, something went wrong.