Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Release candidate] v10 prerelease #233

Closed
wants to merge 111 commits into from
Closed
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
c310af8
Initial commit
mafintosh Feb 6, 2019
0fc77d5
Simple reads/writes work
andrewosh Jan 25, 2019
79e282e
Forgot index file
andrewosh Jan 25, 2019
3c8a6a2
Added stat, dir, and checkout methods. Need to fix initialization for…
andrewosh Jan 26, 2019
f04561a
Stat + storage tests pass
andrewosh Jan 28, 2019
4a5a1bd
Use consts + remove all clear logic temporarily
andrewosh Jan 30, 2019
1859f52
Added watch
andrewosh Jan 31, 2019
48aecac
Added fuzzing using fuzzbuzz + fixed deprecation warnings
andrewosh Jan 31, 2019
27ae10a
Added createDirectoryStream + updated version handling + fixed deprec…
andrewosh Feb 1, 2019
2d90fdf
Added _contentFeedLength so mkdir needn't lock. this.trie -> this._db
andrewosh Feb 4, 2019
c532a2e
Added existingFileOverwrite to fuzz tests
andrewosh Feb 4, 2019
1689498
Added randomReadStream to fuzzer
andrewosh Feb 4, 2019
7e8a103
Update mkdir to use condition function
andrewosh Feb 5, 2019
bab5eae
Extract content/storage helpers into lib + more fuzzing
andrewosh Feb 5, 2019
401be48
Changed error message for ENOENT
andrewosh Feb 5, 2019
0fce431
Reuse ensureContent in initialize + more fuzzing
andrewosh Feb 5, 2019
6b87dca
Added missing lib/keys.js + updated docs + end handling in createRead…
andrewosh Feb 6, 2019
792185d
Add length to randomReadStream fuzz test
andrewosh Feb 6, 2019
483e72a
Update deps
andrewosh Feb 6, 2019
5427b7f
Update node versions in .travis.yml
andrewosh Feb 6, 2019
4bb6547
Fix bug in close + moved sodium-universal to dependencies
andrewosh Feb 7, 2019
7322926
file descriptor reads
mafintosh Feb 7, 2019
586b2be
Add BadFileDescriptor error
andrewosh Feb 7, 2019
c9122f1
martinheidegger's suggested changes + failing FD fuzz test
andrewosh Feb 12, 2019
cb5e19e
FD bugs + stateful FD fuzz tests
andrewosh Feb 13, 2019
397e94f
Initial writable FD support
andrewosh Feb 17, 2019
7db1178
Fix readable fd bug
andrewosh Feb 17, 2019
76caf69
Basic writable fd
andrewosh Feb 17, 2019
f9e9360
writable fd tests
andrewosh Feb 17, 2019
235b797
Add factory option
andrewosh Feb 18, 2019
a9141cf
Add updateMetadata
andrewosh Feb 18, 2019
5a1845d
Manual decoding
andrewosh Feb 21, 2019
10ebb9f
Better error handling + writable fd fuzz tests
andrewosh Feb 21, 2019
80d8cc8
Fixed bug in _update
andrewosh Feb 21, 2019
4beac45
Decode value in _statDirectory
andrewosh Feb 21, 2019
223bf00
createDirectoryStream decoding error
andrewosh Feb 21, 2019
6221c30
Fixed statDirectory + untested truncate
andrewosh Feb 21, 2019
d4b5128
Handful of untested, late-night changes
andrewosh Feb 21, 2019
8bc8ca3
Many bug fixes + create
andrewosh Feb 22, 2019
4191466
Temporarily add debugging
andrewosh Feb 22, 2019
c633824
Temporary buffer copy in writable fd (fuse workaround)
andrewosh Feb 24, 2019
0c5db68
Added batcher for fewer blocks
andrewosh Feb 24, 2019
7aeb124
Prefetch limit + readdir bug
andrewosh Feb 24, 2019
9fbe03d
Nested directory readdir test
andrewosh Feb 24, 2019
3b83a53
Added sparseMetadata test
andrewosh Feb 24, 2019
d1b6502
Prefetch 16 blocks in fd + add comment
andrewosh Feb 25, 2019
afbb687
Add createDiffStream (do not use until trie bug fix)
andrewosh Feb 25, 2019
81dbc09
Assorted changes for @mafintosh's review
andrewosh Mar 31, 2019
9bee543
Standardized
andrewosh Mar 31, 2019
930414e
Resolved merge conflict
andrewosh Mar 31, 2019
e1268af
Use filesystem-constants
andrewosh Apr 9, 2019
dd62a85
var -> let
andrewosh Apr 9, 2019
3877eec
10.0.0-rc0
andrewosh Apr 10, 2019
8d12c17
Update FD size during write (#237)
andrewosh Apr 17, 2019
0bb2210
10.0.0-rc1
andrewosh Apr 17, 2019
4700e26
fix create returning a buffer instead of a stat obj
mafintosh Apr 19, 2019
11c179b
10.0.0-rc2
mafintosh Apr 19, 2019
854ea78
Pass stat through FD createWriteStream + fix create when file exists
andrewosh Apr 23, 2019
b158ba8
Add file option to lstat
andrewosh Apr 23, 2019
e334190
1.0.0-rc3
andrewosh Apr 24, 2019
77ee7ae
bump to 7
mafintosh Apr 29, 2019
92b25ff
fix: allow content feed to be passed in constructor (#238)
Karissa May 2, 2019
baca0b6
Moving to corestore
andrewosh May 8, 2019
b9f967f
Add temporary git deps
andrewosh May 8, 2019
21436d6
Basic tests pass
andrewosh May 8, 2019
bde7cbe
Simple read/write from mount works
andrewosh May 12, 2019
49f2c28
Added mount tests
andrewosh May 12, 2019
f7cf4f3
Added symlinking + tests
andrewosh May 16, 2019
e6d4bf9
Update deps + remove console.logs
andrewosh May 24, 2019
41d52c0
Test refactoring + added support for mounting hypercores
andrewosh Jun 9, 2019
44b5875
Allow to add metadata to Stat entries (#234)
Frando Jun 12, 2019
b2c08a3
More changes for mounts
andrewosh Jun 14, 2019
ab2a961
Enabled more tests + bug fixes
andrewosh Jun 14, 2019
f498a0b
Temporary deps + update content feeds on creation
andrewosh Jun 15, 2019
229b4f9
Unskip fd tests and remove content update code
andrewosh Jun 15, 2019
d4dee91
Simple readdir with symlinks works
andrewosh Jun 17, 2019
be2b24a
Statting symlinks works for links-to-links
andrewosh Jun 17, 2019
8bb81c2
Added missing files
andrewosh Jun 17, 2019
2abcf7a
Removed debugging statements
andrewosh Jun 17, 2019
6feb8c4
Update deps
andrewosh Jun 17, 2019
c2c4ac8
Bump corestore version
andrewosh Jun 17, 2019
a1832e2
Re-enabled sparse-mode fuzz test
andrewosh Jun 17, 2019
e8b76eb
pipeline -> pump for node 8
andrewosh Jun 17, 2019
13d5227
Fix readdir bug
andrewosh Jun 17, 2019
1e87544
do less seeks (#239)
mafintosh Jun 18, 2019
9217df9
Resolved merge conflict
andrewosh Jun 18, 2019
faf6280
rmdir should use a gt iterator
andrewosh Jun 19, 2019
26a89d9
Re-add sparseMetadata
andrewosh Jun 20, 2019
2bda2cb
Another mount test + debugging
andrewosh Jun 20, 2019
79f4be5
Update deps + don't forward top-level options into corestore
andrewosh Jun 20, 2019
9d86d94
Remove debugging statements
andrewosh Jun 23, 2019
267198f
Updated tests to remove megastore and reuse single corestores
andrewosh Jun 30, 2019
30f38d4
Create new statOpts inside mount
andrewosh Jul 8, 2019
993a16f
Remove unused deps
andrewosh Jul 8, 2019
2330782
Update corestore dep
andrewosh Jul 8, 2019
c277459
Pass uid/gid into mount opts
andrewosh Jul 8, 2019
385bf7b
10.0.0-rc5
andrewosh Jul 9, 2019
5262c7d
Pass flags in _update + add unmount and unmount test
andrewosh Jul 9, 2019
ff8a96c
10.0.0-rc6
andrewosh Jul 9, 2019
065a540
Bump mountable-hypertrie dep
andrewosh Jul 9, 2019
e47b82a
10.0.0-rc7
andrewosh Jul 9, 2019
b848e78
Add listAllMounts
andrewosh Jul 11, 2019
3261a0b
10.0.0-rc8
andrewosh Jul 11, 2019
f4356eb
Added broken symlink readdir test + getAllMounts test
andrewosh Jul 18, 2019
0721730
10.0.0-rc9
andrewosh Jul 18, 2019
7ae2da8
Add secretKey to _getContent for mounts + add CORE_LOCK symbol
andrewosh Jul 18, 2019
59a8e73
10.0.0-rc10
andrewosh Jul 18, 2019
1ef6993
Update README.md
andrewosh Jul 30, 2019
e2d8896
Add isLocked to content state
andrewosh Jul 30, 2019
eeadea9
Merge branch 'v10' of github.com:mafintosh/hyperdrive into v10
andrewosh Jul 30, 2019
933e45e
10.0.0
andrewosh Jul 30, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 61 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,61 @@
node_modules
sandbox.js
sandbox
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# TypeScript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env

# next.js build output
.next
5 changes: 3 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
language: node_js
node_js:
- "node"
- "lts/*"
- "10"
- "8"
- "6"
- "4"
41 changes: 3 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Hyperdrive

#### *Note*: This is a prerelease version of Hyperdrive that's backed by [Hypertrie](https://github.com/mafintosh/hypertrie)
#### This version is not yet API-complete.

Hyperdrive is a secure, real time distributed file system

``` js
Expand Down Expand Up @@ -80,7 +83,6 @@ Options include:
sparseMetadata: true // only download data on metadata feed when requested
metadataStorageCacheSize: 65536 // how many entries to use in the metadata hypercore's LRU cache
contentStorageCacheSize: 65536 // how many entries to use in the content hypercore's LRU cache
treeCacheSize: 65536 // how many entries to use in the append-tree's LRU cache
}
```

Expand Down Expand Up @@ -145,10 +147,6 @@ You can use this with `.checkout(version)` to download a specific version of the
archive.checkout(version).download()
```

#### `var stream = archive.history([options])`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deleted methods here. Are they supposed to be re-implemented or be breaking changes?


Get a stream of all changes and their versions from this archive.

#### `var stream = archive.createReadStream(name, [options])`

Read a file out as a stream. Similar to fs.createReadStream.
Expand Down Expand Up @@ -180,20 +178,6 @@ or a string can be passed as options to simply set the encoding - similar to fs.

If `cached` is set to `true`, this function returns results only if they have already been downloaded.

#### `var stream = archive.createDiffStream(version, [options])`

Diff this archive this another version. `version` can both be a version number of a checkout instance of the archive. The `data` objects looks like this

``` js
{
type: 'put' | 'del',
name: '/some/path/name.txt',
value: {
// the stat object
}
}
```

#### `var stream = archive.createWriteStream(name, [options])`

Write a file as a stream. Similar to fs.createWriteStream.
Expand Down Expand Up @@ -263,13 +247,10 @@ stat.isFile()
Options include:
```js
{
cached: true|false // default: false,
wait: true|false // default: true
}
```

If `cached` is set to `true`, this function returns results only if they have already been downloaded.

If `wait` is set to `true`, this function will wait for data to be downloaded. If false, will return an error.

#### `archive.lstat(name, [options], callback)`
Expand All @@ -279,13 +260,10 @@ Stat an entry but do not follow symlinks. Similar to fs.lstat.
Options include:
```js
{
cached: true|false // default: false,
wait: true|false // default: true
}
```

If `cached` is set to `true`, this function returns results only if they have already been downloaded.

If `wait` is set to `true`, this function will wait for data to be downloaded. If false, will return an error.

#### `archive.access(name, [options], callback)`
Expand All @@ -295,25 +273,12 @@ Similar to fs.access.
Options include:
```js
{
cached: true|false // default: false,
wait: true|false // default: true
}
```

If `cached` is set to `true`, this function returns results only if they have already been downloaded.

If `wait` is set to `true`, this function will wait for data to be downloaded. If false, will return an error.

#### `archive.open(name, flags, [mode], callback)`

Open a file and get a file descriptor back. Similar to fs.open.

Note that currently only read mode is supported in this API.

#### `archive.read(fd, buf, offset, len, position, callback)`

Read from a file descriptor into a buffer. Similar to fs.read.

andrewosh marked this conversation as resolved.
Show resolved Hide resolved
#### `archive.close(fd, [callback])`

Close a file. Similar to fs.close.
Expand Down
Loading