Skip to content

Commit

Permalink
updated to point to statcan fork instead of original
Browse files Browse the repository at this point in the history
  • Loading branch information
Mathis Marcotte authored and voyvodov committed Sep 18, 2024
1 parent 1bda057 commit b54d0c7
Show file tree
Hide file tree
Showing 25 changed files with 57 additions and 57 deletions.
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ install:
script: travis_retry ./test/run-tests.sh GoofysTest
go:
- 1.16
go_import_path: github.com/kahing/goofys
go_import_path: github.com/StatCan/goofys
matrix:
include:
- name: "S3Proxy"
Expand Down
2 changes: 1 addition & 1 deletion README-azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ $ $GOPATH/bin/goofys adl://servicename.azuredatalakestore.net:prefix <mountpoint

# Azure Data Lake Storage Gen2

Configure your credentials the same way as [Azure Blob Storage](https://github.com/kahing/goofys/blob/master/README-azure.md#azure-blob-storage) above, and then:
Configure your credentials the same way as [Azure Blob Storage](https://github.com/StatCan/goofys/blob/master/README-azure.md#azure-blob-storage) above, and then:

```ShellSession
$ $GOPATH/bin/goofys abfs://container <mountpoint>
Expand Down
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<img src="doc/goofys.png" height="32" width="32" align="middle" /> Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go

[![Build Status](https://travis-ci.org/kahing/goofys.svg?branch=master)](https://travis-ci.org/kahing/goofys)
[![Github All Releases](https://img.shields.io/github/downloads/kahing/goofys/total.svg)](https://github.com/kahing/goofys/releases/)
[![Build Status](https://travis-ci.org/StatCan/goofys.svg?branch=master)](https://travis-ci.org/StatCan/goofys)
[![Github All Releases](https://img.shields.io/github/downloads/StatCan/goofys/total.svg)](https://github.com/StatCan/goofys/releases/)
[![Twitter Follow](https://img.shields.io/twitter/follow/s3goofys.svg?style=social&label=Follow)](https://twitter.com/s3goofys)
[![Stack Overflow Questions](https://img.shields.io/stackexchange/stackoverflow/t/goofys?label=Stack%20Overflow%20questions)](https://stackoverflow.com/search?q=%5Bgoofys%5D+is%3Aquestion)

Expand All @@ -14,12 +14,12 @@ for performance first and POSIX second. Particularly things that are
difficult to support on S3 or would translate into more than one
round-trip would either fail (random writes) or faked (no per-file
permission). Goofys does not have an on disk data cache (checkout
[catfs](https://github.com/kahing/catfs)), and consistency model is
[catfs](https://github.com/StatCan/catfs)), and consistency model is
close-to-open.

# Installation

* On Linux, install via [pre-built binaries](https://github.com/kahing/goofys/releases/latest/download/goofys).
* On Linux, install via [pre-built binaries](https://github.com/StatCan/goofys/releases/latest/download/goofys).
You may also need to install fuse too if you want to mount it on startup.

* On macOS, install via [Homebrew](https://brew.sh/):
Expand All @@ -33,8 +33,8 @@ $ brew install goofys

```ShellSession
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
$ go get github.com/StatCan/goofys
$ go install github.com/StatCan/goofys
```

# Usage
Expand All @@ -59,28 +59,28 @@ configured for `root`, and can add this to `/etc/fstab`:
goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666,--dir-mode=0777 0 0
```

See also: [Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2](https://github.com/kahing/goofys/blob/master/README-azure.md).
See also: [Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2](https://github.com/StatCan/goofys/blob/master/README-azure.md).

Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20)
Got more questions? Check out [questions other people asked](https://github.com/StatCan/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20)

# Benchmark

Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys
`-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the
benchmark can be found in
[bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/)
[bench.sh](https://github.com/StatCan/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/StatCan/goofys/blob/master/bench/)
is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a
connected to a bucket in us-west-2. Units are seconds.

![Benchmark result](/bench/bench.png?raw=true "Benchmark")

To run the benchmark, configure EC2's instance role to be able to write to `$TESTBUCKET`, and then do:
```ShellSession
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache StatCan/goofys-bench
# result will be written to $TESTBUCKET
```

See also: [cached benchmark result](https://github.com/kahing/goofys/blob/master/bench/cache/README.md) and [result on Azure](https://github.com/kahing/goofys/blob/master/bench/azure/README.md).
See also: [cached benchmark result](https://github.com/StatCan/goofys/blob/master/bench/cache/README.md) and [result on Azure](https://github.com/StatCan/goofys/blob/master/bench/azure/README.md).

# License

Expand Down Expand Up @@ -128,7 +128,7 @@ Additionally, goofys also works with the following non-S3 object stores:
* Data is stored on [Amazon S3](https://aws.amazon.com/s3/)
* [Amazon SDK for Go](https://github.com/aws/aws-sdk-go)
* Other related fuse filesystems
* [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys
* [catfs](https://github.com/StatCan/catfs): caching layer that can be used with goofys
* [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3
* [gcsfuse](https://github.com/googlecloudplatform/gcsfuse):
filesystem for
Expand Down
4 changes: 2 additions & 2 deletions api/api.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
package goofys

import (
. "github.com/kahing/goofys/api/common"
"github.com/kahing/goofys/internal"
. "github.com/StatCan/goofys/api/common"
"github.com/StatCan/goofys/internal"

"context"
"fmt"
Expand Down
8 changes: 4 additions & 4 deletions bench/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ RUN git clone --depth 1 https://github.com/s3fs-fuse/s3fs-fuse.git && \
cd s3fs-fuse && ./autogen.sh && ./configure && make -j8 > /dev/null && make install && \
cd .. && rm -Rf s3fs-fuse

RUN curl -L -O https://github.com/kahing/catfs/releases/download/v0.8.0/catfs && \
RUN curl -L -O https://github.com/StatCan/catfs/releases/download/v0.8.0/catfs && \
mv catfs /usr/bin && chmod 0755 /usr/bin/catfs

# ideally I want to clear out all the go deps too but there's no
# way to do that with ADD
ENV PATH=$PATH:/root/go/bin
ADD . /root/go/src/github.com/kahing/goofys
WORKDIR /root/go/src/github.com/kahing/goofys
ADD . /root/go/src/github.com/StatCan/goofys
WORKDIR /root/go/src/github.com/StatCan/goofys
RUN go get . && make install

ENTRYPOINT ["/root/go/src/github.com/kahing/goofys/bench/run_bench.sh"]
ENTRYPOINT ["/root/go/src/github.com/StatCan/goofys/bench/run_bench.sh"]
8 changes: 4 additions & 4 deletions bench/Dockerfile.azure
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ RUN git clone --depth 1 https://github.com/Azure/azure-storage-fuse.git && \
cd azure-storage-fuse && bash ./build.sh > /dev/null && make -C build install && \
cd .. && rm -Rf azure-storage-fuse

RUN curl -L -O https://github.com/kahing/catfs/releases/download/v0.8.0/catfs && \
RUN curl -L -O https://github.com/StatCan/catfs/releases/download/v0.8.0/catfs && \
mv catfs /usr/bin && chmod 0755 /usr/bin/catfs

# ideally I want to clear out all the go deps too but there's no
# way to do that with ADD
ENV PATH=$PATH:/root/go/bin
ADD . /root/go/src/github.com/kahing/goofys
WORKDIR /root/go/src/github.com/kahing/goofys
ADD . /root/go/src/github.com/StatCan/goofys
WORKDIR /root/go/src/github.com/StatCan/goofys
RUN go get . && make install

ENTRYPOINT ["/root/go/src/github.com/kahing/goofys/bench/run_bench.sh"]
ENTRYPOINT ["/root/go/src/github.com/StatCan/goofys/bench/run_bench.sh"]
10 changes: 5 additions & 5 deletions bench/Dockerfile.gcs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM golang:1.14 AS goofys-builder

# install goofys
WORKDIR $GOPATH/src/github.com/kahing/goofys
WORKDIR $GOPATH/src/github.com/StatCan/goofys

COPY . .

Expand All @@ -24,7 +24,7 @@ RUN apt-get update && \
&& apt-get clean

# install catfs, required to run goofys with cache
RUN curl -L -O https://github.com/kahing/catfs/releases/download/v0.8.0/catfs && \
RUN curl -L -O https://github.com/StatCan/catfs/releases/download/v0.8.0/catfs && \
mv catfs /usr/bin && chmod 0755 /usr/bin/catfs

# goofys graph generation
Expand All @@ -33,12 +33,12 @@ RUN pip install numpy
ENV PATH=$PATH:/root/go/bin

# copy go binaries
COPY --from=goofys-builder /go/src/github.com/kahing/goofys/goofys /root/go/bin/goofys
COPY --from=goofys-builder /go/src/github.com/StatCan/goofys/goofys /root/go/bin/goofys
COPY --from=goofys-builder /go/bin/gcsfuse /root/go/bin/gcsfuse

WORKDIR /root/go/src/github.com/kahing/goofys
WORKDIR /root/go/src/github.com/StatCan/goofys

# copy bench scripts
COPY bench bench

ENTRYPOINT ["/root/go/src/github.com/kahing/goofys/bench/run_bench.sh"]
ENTRYPOINT ["/root/go/src/github.com/StatCan/goofys/bench/run_bench.sh"]
2 changes: 1 addition & 1 deletion bench/azure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ To run the benchmark, do:
```ShellSession
$ export AZURE_STORAGE_ACCOUNT=myaccount
$ export AZURE_STORAGE_KEY=STORAGE-ACCESS-KEY
$ sudo docker run -e BUCKET=$TESTBUCKET -e AZURE_STORAGE_ACCOUNT=$AZURE_STORAGE_ACCOUNT -e AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY --rm --privileged --net=host -v /mnt/cache:/tmp/cache kahing/goofys-bench:azure-latest
$ sudo docker run -e BUCKET=$TESTBUCKET -e AZURE_STORAGE_ACCOUNT=$AZURE_STORAGE_ACCOUNT -e AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY --rm --privileged --net=host -v /mnt/cache:/tmp/cache StatCan/goofys-bench:azure-latest
# result will be written to $TESTBUCKET
```
2 changes: 1 addition & 1 deletion bench/cache/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ To run the benchmark, configure EC2's instance role to be able to write to `$TES
```ShellSession
$ export AWS_ACCESS_KEY_ID=AKID1234567890
$ export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=true --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=true --rm --privileged --net=host -v /tmp/cache:/tmp/cache StatCan/goofys-bench
# result will be written to $TESTBUCKET
```
6 changes: 3 additions & 3 deletions example/test_api.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package main

import (
goofys "github.com/kahing/goofys/api"
common "github.com/kahing/goofys/api/common"
goofys "github.com/StatCan/goofys/api"
common "github.com/StatCan/goofys/api/common"

"fmt"
"context"
"fmt"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/kahing/goofys
module github.com/StatCan/goofys

go 1.14

Expand Down
4 changes: 2 additions & 2 deletions internal/aws_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"
. "gopkg.in/check.v1"

"fmt"
Expand All @@ -39,7 +39,7 @@ func (s *AwsTest) SetUpSuite(t *C) {
}

func (s *AwsTest) TestRegionDetection(t *C) {
s.s3.bucket = "goofys-eu-west-1.kahing.xyz"
s.s3.bucket = "goofys-eu-west-1.StatCan.xyz"

err, isAws := s.s3.detectBucketLocationByHEAD()
t.Assert(err, IsNil)
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_adlv1.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"bytes"
"context"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_adlv2.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"context"
"encoding/base64"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_azblob.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"bytes"
"context"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_gcs.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package internal

import (
"github.com/kahing/goofys/api/common"
"github.com/StatCan/goofys/api/common"

"bytes"
"context"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_gcs3.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"fmt"
"io"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_gcs_test.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package internal

import (
"github.com/kahing/goofys/api/common"
"github.com/StatCan/goofys/api/common"

"bytes"
"context"
Expand Down
2 changes: 1 addition & 1 deletion internal/backend_s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"fmt"
"net/http"
Expand Down
2 changes: 1 addition & 1 deletion internal/buffer_pool.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"io"
"runtime"
Expand Down
2 changes: 1 addition & 1 deletion internal/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"fmt"
"io"
Expand Down
2 changes: 1 addition & 1 deletion internal/goofys.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"context"
"fmt"
Expand Down
12 changes: 6 additions & 6 deletions internal/goofys_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"

"bufio"
"bytes"
Expand Down Expand Up @@ -166,11 +166,11 @@ func (t *GoofysTest) deleteBlobsParallelly(cloud StorageBackend, blobs []string)
// groupByDecresingDepths takes a slice of path strings and returns the paths as
// groups where each group has the same `depth` - depth(a/b/c)=2, depth(a/b/)=1
// The groups are returned in decreasing order of depths.
// - Inp: [] Out: []
// - Inp: ["a/b1/", "a/b/c1", "a/b2", "a/b/c2"]
// Out: [["a/b/c1", "a/b/c2"], ["a/b1/", "a/b2"]]
// - Inp: ["a/b1/", "z/a/b/c1", "a/b2", "z/a/b/c2"]
// Out: [["z/a/b/c1", "z/a/b/c2"], ["a/b1/", "a/b2"]
// - Inp: [] Out: []
// - Inp: ["a/b1/", "a/b/c1", "a/b2", "a/b/c2"]
// Out: [["a/b/c1", "a/b/c2"], ["a/b1/", "a/b2"]]
// - Inp: ["a/b1/", "z/a/b/c1", "a/b2", "z/a/b/c2"]
// Out: [["z/a/b/c1", "z/a/b/c2"], ["a/b1/", "a/b2"]
func groupByDecresingDepths(items []string) [][]string {
depthToGroup := map[int][]string{}
for _, item := range items {
Expand Down
2 changes: 1 addition & 1 deletion internal/minio_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
package internal

import (
. "github.com/kahing/goofys/api/common"
. "github.com/StatCan/goofys/api/common"
. "gopkg.in/check.v1"

"context"
Expand Down
6 changes: 3 additions & 3 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ import (
"io/ioutil"
"strconv"

goofys "github.com/kahing/goofys/api"
. "github.com/kahing/goofys/api/common"
. "github.com/kahing/goofys/internal"
goofys "github.com/StatCan/goofys/api"
. "github.com/StatCan/goofys/api/common"
. "github.com/StatCan/goofys/internal"

"fmt"
"os"
Expand Down

0 comments on commit b54d0c7

Please sign in to comment.