Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: built-in content blocking based on IPIP-383 #10161

Merged
merged 22 commits into from
Oct 28, 2023
Merged

feat: built-in content blocking based on IPIP-383 #10161

merged 22 commits into from
Oct 28, 2023

Commits on Oct 3, 2023

  1. plugin: Add support for content blocking in Kubo

    Fixes #8492.
    
    This introduces "nopfs" as a preloaded plugin into Kubo.
    
    It automatically make Kubo watch *.deny files found in:
    
    - /etc/ipfs/denylists
    - $XDG_CONFIG_HOME/ipfs/denylists
    - $IPFS_PATH/denylists
    
    (files need to be present before boot in order to be watched).
    
    Debug logging can be enabled with `GOLOG_LOG_LEVEL="nopfs=debug"`.
    
    All blocks are logged to "nopfs-blocks", so logging requests for blocked
    content can be achieved with
    `GOLOG_LOG_LEVEL="nopfs-blocks=warn"`:
    
    ```
    WARN (...) QmRFniDxwxoG2n4AcnGhRdjqDjCM5YeUcBE75K8WXmioH3: blocked (test.deny:9)
    ```
    
    Interactive/gateway users will also receive errors as responses but with less details:
    
    ```
    Error: /ipfs/QmQvjk82hPkSaZsyJ8vNER5cmzKW7HyGX5XVusK7EAenCN is blocked and cannot be provided
    ```
    
    One particularity to keep in mind is that GetMany() will silently drop blocked
    blocks from the response (a warnings are logged). AddMany() will act
    similarly and avoid adding blocked blocks.
    
    The code implementing all this is actually in nopfs:
    
    - https://github.com/ipfs-shipyard/nopfs (main library)
    - https://github.com/ipfs-shipyard/nopfs/tree/master/ipfs (wrappers)
    
    The interpretation of the list rules and block detection is well tested, but a
    general review might be in order.
    hsanjuan committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    d09c1f9 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Configuration menu
    Copy the full SHA
    bcae8a4 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. docs: content blocking

    lidel committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    51acfb9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    da3cf31 View commit details
    Browse the repository at this point in the history
  3. test: content_blocking_test.go

    Includes dependency update from ipfs-shipyard/nopfs#25
    lidel committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    0a8f651 View commit details
    Browse the repository at this point in the history
  4. chore: fix go lint

    lidel committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    e689080 View commit details
    Browse the repository at this point in the history
  5. test: /ipfs/cid/* rule on CLI and gateway

    CLI works as expected, gateway does not respect the rule
    (needs investigation)
    lidel committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    511b670 View commit details
    Browse the repository at this point in the history
  6. fix: accept paths in dag export

    lidel committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    0c47250 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Configuration menu
    Copy the full SHA
    fd842c2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2eed42d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    04c1f68 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e1689b5 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f7483a6 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f516fc5 View commit details
    Browse the repository at this point in the history
  7. test: fix sharness of dag export

    cosmetic change to error message, this should be more robust
    lidel committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    0129417 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Latest nopfs

    hsanjuan committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    9f4fefe View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b4b7ca7 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. test: double-hash Path block (blake3-multihash)

    this ensures we test something other than default sha256
    
    Ref. ipfs-shipyard/nopfs#28
    lidel committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    c3bdfff View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. test: Gateway.NoFetch and GatewayOverLibp2p

    adds missing tests for "no fetch" gateways one can expose,
    in both cases the offline mode is done by passing custom
    blockservice/exchange into path resolver, which means
    global path resolver that has nopfs intercept is not used,
    and the content blocking does not happen on these gateways.
    
    needs to be fixed, but at least now we have tests that
    fail until it is fixed.
    lidel committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    c99068e View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2023

  1. fix: use offline path resolvers where appropriate

    this fixes the problem described in
    #10161 (comment)
    by adding explicit offline path resolvers that are backed
    by offline exchange, and using them in NoFetch gateways
    instead of the default online ones
    lidel committed Oct 28, 2023
    Configuration menu
    Copy the full SHA
    da2dc47 View commit details
    Browse the repository at this point in the history
  2. fix(gw): blocked content produces http error 410

    requires ipfs/boxo#497
    which is based on top of the boxo already used in kubo master
    to avoid issues caused by ilater commits in boxo main
    lidel committed Oct 28, 2023
    Configuration menu
    Copy the full SHA
    3d8feac View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d4766b6 View commit details
    Browse the repository at this point in the history