Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support only compressed snapshots on stable urls #2893

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
6cddbfb
mark: 2873-modify-forest-and-forest-cli-to-prefer-compressed-files
aatifsyed May 16, 2023
2d180c7
feat: fetch and export compressed snapshots only
aatifsyed May 22, 2023
0f7bcb6
chore: remove unused deps
aatifsyed May 22, 2023
993c8bc
fix: missed an arg
aatifsyed May 22, 2023
b951e13
refactor: remove const_option macro
aatifsyed May 23, 2023
696d4e4
wip: replace retry macro
aatifsyed May 23, 2023
d0e94e4
refactor: remove retry macro
aatifsyed May 23, 2023
0a6042d
chore: remove unused dep
aatifsyed May 23, 2023
fb33dcb
docs: update changelog
aatifsyed May 23, 2023
cf89ff5
fix: tests should not use removed flag
aatifsyed May 23, 2023
65af7c7
chore: formatting
aatifsyed May 23, 2023
d4e0e4e
wip: refactor
aatifsyed May 24, 2023
97e7ee7
wip: progress bars
aatifsyed May 24, 2023
eff12c9
wip
aatifsyed May 25, 2023
d7d4683
wip
aatifsyed May 25, 2023
a9ecf9a
wip
aatifsyed May 25, 2023
407df7d
test: add snapshot tests
aatifsyed May 25, 2023
86cb6a2
wip: plumb fetch subcommand
aatifsyed May 30, 2023
1a8afe7
wip: refactor snapshot_cmd
aatifsyed May 30, 2023
7eb8c28
wip: tweaks
aatifsyed May 30, 2023
cb73a1c
refactor: slug not required
aatifsyed May 30, 2023
d5ba892
wip: daemon refactor
aatifsyed May 30, 2023
1dfb3dd
Revert "refactor: slug not required"
aatifsyed May 30, 2023
a03f22a
wip
aatifsyed May 30, 2023
349c81a
wip
aatifsyed May 30, 2023
a609b21
wip: refactor: store metadata alongside files
aatifsyed May 31, 2023
7fc0996
refactor: snapshot command
aatifsyed May 31, 2023
bf6a4f2
refactor: daemon can fetch snapshots
aatifsyed May 31, 2023
8782599
fix: catchup after rebase
aatifsyed May 31, 2023
7d81fe0
fix: remove udeps
aatifsyed May 31, 2023
b9a7477
test: snapshot fetching
aatifsyed May 31, 2023
3f04601
fix: ci lint
aatifsyed May 31, 2023
9f8a671
chore: document private items
aatifsyed May 31, 2023
a025f37
refactor: unnecessary result types in test
aatifsyed May 31, 2023
62e6339
fix: whoops, left a dead file
aatifsyed May 31, 2023
836ba11
refactor: cli tests
aatifsyed May 31, 2023
470c2f8
feat: intern subcommand
aatifsyed May 31, 2023
564f524
fix: revert to old way of cfg-gating method
aatifsyed May 31, 2023
b3bdf67
fix: document and lint private items
aatifsyed May 31, 2023
af0b65a
chore: remove unused dependencies`
aatifsyed May 31, 2023
09052a8
doc: document disallowed types
aatifsyed May 31, 2023
010156e
chore: remove unused deps
aatifsyed May 31, 2023
e1af05c
run: bundle install
aatifsyed May 31, 2023
a2a5b93
fix: recursion
aatifsyed May 31, 2023
18fef71
chore: add words to dictionary
aatifsyed May 31, 2023
c006e98
test: add integration test for interning
aatifsyed May 31, 2023
ff1da86
tweak: more machine readable output for list
aatifsyed Jun 1, 2023
1cedde7
chore: forest_utils::net should not exist
aatifsyed Jun 1, 2023
21ddef3
test: integration tests
aatifsyed Jun 1, 2023
1799607
fix: do not require openssl (for ci build)
aatifsyed Jun 2, 2023
fb94e0a
fix: snapshot test script
aatifsyed Jun 2, 2023
0e6eb3a
fix: ci
aatifsyed Jun 2, 2023
ff05a73
fix: fucking spellcheck
aatifsyed Jun 2, 2023
fa3a0b1
review: spellcheck stuff
aatifsyed Jun 5, 2023
b30d91d
chore: prettier
aatifsyed Jun 5, 2023
1d49fee
review: point to issue
aatifsyed Jun 5, 2023
237d8ac
chore: document spellchecking
aatifsyed Jun 5, 2023
35586e5
wip
aatifsyed Jun 6, 2023
88277be
review: markups
aatifsyed Jun 6, 2023
8ac358c
fix: no freeze, aria2c works
aatifsyed Jun 7, 2023
254ddd1
fix: spellcheck config, build config, spellcheck, todos
aatifsyed Jun 7, 2023
6ea198e
chore: remove unused deps
aatifsyed Jun 7, 2023
20f6fdf
wip: fix pipelines
aatifsyed Jun 7, 2023
8d922fd
refactor: password prompt. This also fixes interrupting
aatifsyed Jun 7, 2023
b07379e
typo
aatifsyed Jun 7, 2023
45252ce
Catch up with main, improve asyncify
aatifsyed Jun 7, 2023
c22bf3e
catch up with main
aatifsyed Jun 8, 2023
836fb7d
chore: spellcheck
aatifsyed Jun 8, 2023
a18764f
review: remove superfluous commands
aatifsyed Jun 9, 2023
e4b9abb
review: markups
aatifsyed Jun 9, 2023
cb20275
doc: changelog
aatifsyed Jun 9, 2023
08624aa
fix: tests
aatifsyed Jun 9, 2023
aaef338
Merge main into aatifsyed/2873-modify-forest-and-forest-cli-to-prefer…
LeshyBot Jun 9, 2023
3f5af9a
fix: do not silently ignore spellcheck failures
aatifsyed Jun 12, 2023
929bed6
review: markups
aatifsyed Jun 12, 2023
6811c43
refactor: do not loop and sleep
aatifsyed Jun 12, 2023
8cb1a5c
review: markups
aatifsyed Jun 12, 2023
1a7939b
Merge remote-tracking branch 'origin/main' into aatifsyed/2873-modify…
aatifsyed Jun 12, 2023
476d8ea
Merge main into aatifsyed/2873-modify-forest-and-forest-cli-to-prefer…
LeshyBot Jun 12, 2023
3b0f1fe
fix: .cargo/config.toml::env is causing bug in tests
aatifsyed Jun 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
[alias]
# Permits `cargo cli --chain calibnet ...`
cli = "run --bin forest-cli --"
daemon = "run --bin forest --"

[build]
incremental = true

# TODO(aatifsyed): remove - this can be pushed out to readme
# In all cases, pass --cfg=tokio_unstable for tokio console integration
# See (https://github.com/ChainSafe/forest/pull/2245)
# Note that this may be overriden by user configuration at ~/.cargo/config.toml
rustflags = ["--cfg=tokio_unstable"]

[net]
Expand Down
28 changes: 21 additions & 7 deletions .clippy.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,21 @@
# https://rust-lang.github.io/rust-clippy/master/index.html#disallowed_types
disallowed-types = [
{ path = "std::collections::HashMap", reason = "use ahash::HashMap instead" },
{ path = "std::collections::HashSet", reason = "use ahash::HashSet instead" },
{ path = "std::sync::RwLock", reason = "use parking_lot::RwLock instead" },
{ path = "std::sync::Mutex", reason = "use parking_lot::Mutex instead" },
]
# Banned in #2454, trading security/being idiomatic for speed
[[disallowed-types]]
path = "std::collections::HashMap"
reason = """the standard library hasher is secure by default, but not very fast.
use ahash::HashMap instead."""

[[disallowed-types]]
path = "std::collections::HashSet"
reason = """the standard library hasher is secure by default, but not very fast.
use ahash::HashSet instead."""

# Banned in #2600, presumably so that poisoning won't need to be user-handled
[[disallowed-types]]
path = "std::sync::RwLock"
reason = """the standard library synchronization primitives are poisoned when aquiring threads panic.
use parking_lot::RwLock instead to silently ignore panics."""

[[disallowed-types]]
path = "std::sync::Mutex"
reason = """the standard library synchronization primitives are poisoned when aquiring threads panic.
use parking_lot::Mutex instead to silently ignore panics."""
4 changes: 4 additions & 0 deletions .config/forest.dic
aatifsyed marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Ethereum
exa
FIL
Filecoin/M
Filops
FVM
GC
GiB
Expand Down Expand Up @@ -58,7 +59,10 @@ serializable
serializer/SM
statediff
stateful
stderr
stdout
synchronizer
syscall/S
TCP
tipset/SM
tipsetkey/S
Expand Down
72 changes: 72 additions & 0 deletions .config/spellcheck.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Spellcheck

We run spellchecks using
[`cargo-spellcheck`](https://crates.io/crates/cargo-spellcheck).

It delegates to a program called
[`Hunspell`](https://github.com/hunspell/hunspell).

Hunspell accepts uses `dictionary` files for words and `affix` files to define
acceptable modifications to those words.

Note that cargo-spellcheck comes with
[default dictionary and affix files](https://github.com/drahnr/cargo-spellcheck/blob/dff48db8ca954fce14a0cd5aea127ce59a929624/src/checker/hunspell.rs#L32).
Our vendored `en_US.dic` is larger than theirs.

## How it works

See `forest.dic` in this directory:

```dic
Filecoin/M
```

`Filecoin` is the word, and `/M` applies the `M` flag in the
[affix file](https://github.com/drahnr/cargo-spellcheck/blob/dff48db8ca954fce14a0cd5aea127ce59a929624/hunspell-data/en_US.aff#L103):

```aff
SFX M 0 's .
```

In this case, `'s` and `s` are acceptable suffixes, so the following are
allowed:

- `Filecoin`
- `Filecoins`
- `Filecoin's`

As another example, take the following entry:

```dic
syscall/S
```

Where the `S` flag is
[as follows](https://github.com/drahnr/cargo-spellcheck/blob/dff48db8ca954fce14a0cd5aea127ce59a929624/hunspell-data/en_US.aff#L91-L95):

<!-- Use a block instead of inline code to stop the spacing being murdered by 'pretter' -->

```c
// Define a suffix, called S, allow mixing prefixes and suffixes, with 4 rules.
SFX S Y 4
// Remove a trailing `y`, replace it with `ies`, if the word ends in a `y` not preceded by a vowel.
SFX S y ies [^aeiou]y
// Don't remove any trailing characters, add an s, if the word ends in a `y` preceded by a vowel.
SFX S 0 s [aeiou]y
SFX S 0 es [sxzh]
SFX S 0 s [^sxzhy]
```

So the following would be allowed:

- `syscall`
- `syscalls`

Flags may be combined - you will often see `/SM`, for example.

For more information see
[the `Hunspell` manual](https://manpages.ubuntu.com/manpages/bionic/man5/hunspell.5.html)

## Tips

- Include symbols in `backticks` - they won't have to be added to the dictionary
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@

## Forest unreleased

### Breaking

- [#2873](https://github.com/ChainSafe/forest/issues/2873)
- remove `--compressed` from the CLI. Snapshots are now always compressed.
- Remove snapshot ops - snapshots fetched to the current directory by default.

### Added

- [#2706](https://github.com/ChainSafe/forest/issues/2706): implement
Expand Down
Loading