Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permission denied for redis:import #248

Open
chapterjason opened this issue Jul 17, 2024 · 11 comments
Open

Permission denied for redis:import #248

chapterjason opened this issue Jul 17, 2024 · 11 comments

Comments

@chapterjason
Copy link

Description of problem

Running redis:import to import previously exported data causes permission error.

How reproducible

I've just created the service and exported the data to a file, after that just import it with the commands in streps to reproduce.

Steps to Reproduce

  1. dokku redis:create rtest
  2. dokku redis:export rtest > redis.dump
  3. dokku redis:import rtest < redis.dump

Actual Results

/var/lib/dokku/plugins/enabled/redis/functions: line 187: /var/lib/dokku/services/redis/rtest/data/dump.rdb: Permission denied
 !     Failed to execute dokku command over ssh: exit code 0
 !     If there was no output from Dokku, ensure your configured SSH Key can connect to the remote server

Expected Results

Imports the data without error.

Environment Information

dokku report APP_NAME output

The redis service isn't linked to any application yet.

This is required! Issues missing this information may be closed.

For problems affecting all applications, the report output for a broken application is useful for our debugging.
In these cases, you may run dokku report without any arguments to display the top-level reporting information.

Here is the output of the command with an app:

-----> uname: Linux [...REDACTED HOSTNAME...] 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux
-----> memory: 
                      total        used        free      shared  buff/cache   available
       Mem:           64224        1573       53745          81        9695       62650
       Swap:          32734           0       32734
-----> disk utilization: 
       Filesystem      Size  Used Avail Use% Mounted on
       /dev/md2        436G  9.0G  405G   3% /
       /dev/md1        989M  117M  822M  13% /boot
-----> disk inode utilization: 
       Filesystem     Inodes IUsed IFree IUse% Mounted on
       /dev/md2          28M  342K   28M    2% /
       /dev/md1          64K   638   64K    1% /boot
-----> docker version: 
       Client: Docker Engine - Community
        Version:           27.0.3
        API version:       1.46
        Go version:        go1.21.11
        Git commit:        7d4bcd8
        Built:             Sat Jun 29 00:02:50 2024
        OS/Arch:           linux/amd64
        Context:           default
       
       Server: Docker Engine - Community
        Engine:
         Version:          27.0.3
         API version:      1.46 (minimum version 1.24)
         Go version:       go1.21.11
         Git commit:       662f78c
         Built:            Sat Jun 29 00:02:50 2024
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.7.18
         GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
        runc:
         Version:          1.7.18
         GitCommit:        v1.1.13-0-g58aa920
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client: Docker Engine - Community
        Version:    27.0.3
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.15.1
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.28.1
           Path:     /usr/libexec/docker/cli-plugins/docker-compose
       
       Server:
        Containers: 9
         Running: 9
         Paused: 0
         Stopped: 0
        Images: 47
        Server Version: 27.0.3
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
        runc version: v1.1.13-0-g58aa920
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 6.1.0-22-amd64
        Operating System: Debian GNU/Linux 12 (bookworm)
        OSType: linux
        Architecture: x86_64
        CPUs: 12
        Total Memory: 62.72GiB
        Name: [...REDACTED HOSTNAME...]
        ID: f86c594c-3228-42dc-ae51-e793fbace1fa
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 80
         Goroutines: 84
         System Time: 2024-07-17T17:27:09.57197101+02:00
         EventsListeners: 1
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> herokuish version: 
 !     Herokuish image gliderlabs/herokuish:latest-22 is not available
-----> dokku version: dokku version 0.34.6
 !     dokku-event-listener binary is not available
 !     dokku-update binary is not available
-----> docker-container-healthchecker version: v0.11.0
-----> docker-image-labeler version: v0.8.0
-----> git version: git version 2.39.2
-----> lambda-builder version:        v0.8.0                                                                          
-----> netrc version: v0.10.0
 !     pack binary is not available
-----> plugn version: plugn: v0.16.0
-----> sigil version: v0.11.0
-----> sshcommand version: sshcommand v0.18.0
-----> dokku plugins: 
         00_dokku-standard    0.34.6 enabled    dokku core standard plugin
         20_events            0.34.6 enabled    dokku core events logging plugin
         app-json             0.34.6 enabled    dokku core app-json plugin
         apps                 0.34.6 enabled    dokku core apps plugin
         builder              0.34.6 enabled    dokku core builder plugin
         builder-dockerfile   0.34.6 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.34.6 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.34.6 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.34.6 enabled    dokku core builder-nixpacks plugin
         builder-null         0.34.6 enabled    dokku core builder-null plugin
         builder-pack         0.34.6 enabled    dokku core builder-pack plugin
         buildpacks           0.34.6 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.34.6 enabled    dokku core caddy-vhosts plugin
         certs                0.34.6 enabled    dokku core certificate management plugin
         checks               0.34.6 enabled    dokku core checks plugin
         common               0.34.6 enabled    dokku core common plugin
         config               0.34.6 enabled    dokku core config plugin
         cron                 0.34.6 enabled    dokku core cron plugin
         docker-options       0.34.6 enabled    dokku core docker-options plugin
         domains              0.34.6 enabled    dokku core domains plugin
         enter                0.34.6 enabled    dokku core enter plugin
         git                  0.34.6 enabled    dokku core git plugin
         haproxy-vhosts       0.34.6 enabled    dokku core haproxy-vhosts plugin
         logs                 0.34.6 enabled    dokku core logs plugin
         network              0.34.6 enabled    dokku core network plugin
         nginx-vhosts         0.34.6 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.34.6 enabled    dokku core openresty-vhosts plugin
         plugin               0.34.6 enabled    dokku core plugin plugin
         ports                0.34.6 enabled    dokku core ports plugin
         postgres             1.37.0 enabled    dokku postgres service plugin
         proxy                0.34.6 enabled    dokku core proxy plugin
         ps                   0.34.6 enabled    dokku core ps plugin
         redis                1.37.5 enabled    dokku redis service plugin
         registry             0.34.6 enabled    dokku core registry plugin
         repo                 0.34.6 enabled    dokku core repo plugin
         resource             0.34.6 enabled    dokku core resource plugin
         run                  0.34.6 enabled    dokku core run plugin
         scheduler            0.34.6 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.34.6 enabled    dokku core scheduler-docker-local plugin
         scheduler-k3s        0.34.6 enabled    dokku core scheduler-k3s plugin
         scheduler-null       0.34.6 enabled    dokku core scheduler-null plugin
         shell                0.34.6 enabled    dokku core shell plugin
         ssh-keys             0.34.6 enabled    dokku core ssh-keys plugin
         storage              0.34.6 enabled    dokku core storage plugin
         trace                0.34.6 enabled    dokku core trace plugin
         traefik-vhosts       0.34.6 enabled    dokku core traefik-vhosts plugin
=====> [...REDACTED APPNAME...] app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> [...REDACTED APPNAME...] app information
       App created at:                1720774555
       App deploy source:             git-push
       App deploy source metadata:    [...REDACTED GIT-HASH...]
       App dir:                       /home/dokku/[...REDACTED APPNAME...]
       App locked:                    false
=====> [...REDACTED APPNAME...] builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> [...REDACTED APPNAME...] builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> [...REDACTED APPNAME...] builder-herokuish information
       Builder herokuish computed allowed: true                     
       Builder herokuish global allowed: true                     
       Builder herokuish allowed:                              
=====> [...REDACTED APPNAME...] builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml               
       Builder lambda global lambdayml path: lambda.yml               
       Builder lambda lambdayml path:                          
=====> [...REDACTED APPNAME...] builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml            
       Builder nixpacks global nixpackstoml path: nixpacks.toml            
       Builder nixpacks nixpackstoml path:                          
       Builder nixpacks computed no cache: false                    
       Builder nixpacks global no cache: false                    
       Builder nixpacks no cache:                              
=====> [...REDACTED APPNAME...] builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> [...REDACTED APPNAME...] buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-22
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> [...REDACTED APPNAME...] caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.9
       Caddy letsencrypt email:                                
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR                    
       Caddy polling interval:        5s                       
       Caddy tls internal:            false                    
=====> [...REDACTED APPNAME...] ssl information
       Ssl dir:                       /home/dokku/[...REDACTED APPNAME...]/tls
       Ssl enabled:                   false                    
       Ssl hostnames:                                          
       Ssl expires at:                                         
       Ssl issuer:                                             
       Ssl starts at:                                          
       Ssl subject:                                            
       Ssl verified:                                           
=====> [...REDACTED APPNAME...] checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
       Checks computed wait to retire: 60                       
       Checks global wait to retire:  60                       
       Checks wait to retire:                                  
=====> [...REDACTED APPNAME...] docker options information
       Docker options build:          --link dokku.postgres.[...REDACTED APPNAME...]:dokku-postgres-[...REDACTED APPNAME...] --link dokku.postgres.[...REDACTED APPNAME...]-postgres:dokku-postgres-[...REDACTED APPNAME...]-postgres --link dokku.redis.[...REDACTED APPNAME...]:dokku-redis-[...REDACTED APPNAME...] --link dokku.redis.[...REDACTED APPNAME...]-redis:dokku-redis-[...REDACTED APPNAME...]-redis 
       Docker options deploy:         --link dokku.postgres.[...REDACTED APPNAME...]:dokku-postgres-[...REDACTED APPNAME...] --link dokku.postgres.[...REDACTED APPNAME...]-postgres:dokku-postgres-[...REDACTED APPNAME...]-postgres --link dokku.redis.[...REDACTED APPNAME...]:dokku-redis-[...REDACTED APPNAME...] --link dokku.redis.[...REDACTED APPNAME...]-redis:dokku-redis-[...REDACTED APPNAME...]-redis --restart=unless-stopped 
       Docker options run:            --link dokku.postgres.[...REDACTED APPNAME...]:dokku-postgres-[...REDACTED APPNAME...] --link dokku.postgres.[...REDACTED APPNAME...]-postgres:dokku-postgres-[...REDACTED APPNAME...]-postgres --link dokku.redis.[...REDACTED APPNAME...]:dokku-redis-[...REDACTED APPNAME...] --link dokku.redis.[...REDACTED APPNAME...]-redis:dokku-redis-[...REDACTED APPNAME...]-redis 
=====> [...REDACTED APPNAME...] domains information
       Domains app enabled:           true                     
       Domains app vhosts:            [...REDACTED APPNAME...].[...REDACTED HOSTNAME...]  
       Domains global enabled:        true                     
       Domains global vhosts:         [...REDACTED HOSTNAME...]              
=====> [...REDACTED APPNAME...] git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       [...REDACTED GIT-HASH...]
       Git source image:                                       
       Git last updated at:           1720775426               
=====> [...REDACTED APPNAME...] haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.4.0  
       Haproxy letsencrypt email:                              
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR                    
=====> [...REDACTED APPNAME...] logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector global image:      timberio/vector:0.39.0-debian
       Logs vector sink:              
=====> [...REDACTED APPNAME...] network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.7:5000
=====> [...REDACTED APPNAME...] nginx information
       Nginx access log format:                                
       Nginx computed access log format:                          
       Nginx global access log format:                          
       Nginx access log path:                                  
       Nginx computed access log path: /var/log/nginx/[...REDACTED APPNAME...]-access.log
       Nginx global access log path:  /var/log/nginx/[...REDACTED APPNAME...]-access.log
       Nginx bind address ipv4:                                
       Nginx computed bind address ipv4:                          
       Nginx global bind address ipv4:                          
       Nginx bind address ipv6:                                
       Nginx computed bind address ipv6: ::                       
       Nginx global bind address ipv6: ::                       
       Nginx client max body size:                             
       Nginx computed client max body size: 1m                       
       Nginx global client max body size: 1m                       
       Nginx disable custom config:                            
       Nginx computed disable custom config: false                    
       Nginx global disable custom config: false                    
       Nginx error log path:                                   
       Nginx computed error log path: /var/log/nginx/[...REDACTED APPNAME...]-error.log
       Nginx global error log path:   /var/log/nginx/[...REDACTED APPNAME...]-error.log
       Nginx hsts include subdomains:                          
       Nginx computed hsts include subdomains: true                     
       Nginx global hsts include subdomains: true                     
       Nginx hsts max age:                                     
       Nginx computed hsts max age:   15724800                 
       Nginx global hsts max age:     15724800                 
       Nginx hsts preload:                                     
       Nginx computed hsts preload:   false                    
       Nginx global hsts preload:     false                    
       Nginx hsts:                                             
       Nginx computed hsts:           true                     
       Nginx global hsts:             true                     
       Nginx last visited at:                                  
       Nginx nginx conf sigil path:                            
       Nginx computed nginx conf sigil path: nginx.conf.sigil         
       Nginx global nginx conf sigil path: nginx.conf.sigil         
       Nginx proxy buffer size:                                
       Nginx computed proxy buffer size: 4k                       
       Nginx global proxy buffer size: 4k                       
       Nginx proxy buffering:                                  
       Nginx computed proxy buffering: on                       
       Nginx global proxy buffering:  on                       
       Nginx proxy buffers:                                    
       Nginx computed proxy buffers:  8 4k                     
       Nginx global proxy buffers:    8 4k                     
       Nginx proxy busy buffers size:                          
       Nginx computed proxy busy buffers size: 8k                       
       Nginx global proxy busy buffers size: 8k                       
       Nginx proxy read timeout:                               
       Nginx computed proxy read timeout: 60s                      
       Nginx global proxy read timeout: 60s                      
       Nginx underscore in headers:                            
       Nginx computed underscore in headers: off                      
       Nginx global underscore in headers: off                      
       Nginx x forwarded for value:                            
       Nginx computed x forwarded for value: $remote_addr             
       Nginx global x forwarded for value: $remote_addr             
       Nginx x forwarded port value:                           
       Nginx computed x forwarded port value: $server_port             
       Nginx global x forwarded port value: $server_port             
       Nginx x forwarded proto value:                          
       Nginx computed x forwarded proto value: $scheme                  
       Nginx global x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
       Nginx computed x forwarded ssl:                          
       Nginx global x forwarded ssl:                           
=====> [...REDACTED APPNAME...] openresty information
       Openresty access log format:                            
       Openresty access log path:     /var/log/nginx/[...REDACTED APPNAME...]-access.log
       Openresty allowed letsencrypt domains func base64: cmV0dXJuIHRydWUK         
       Openresty bind address ipv4:                            
       Openresty bind address ipv6:   ::                       
       Openresty client max body size:                          
       Openresty error log path:      /var/log/nginx/[...REDACTED APPNAME...]-error.log
       Openresty global hsts:         true                     
       Openresty computed hsts:       true                     
       Openresty hsts:                                         
       Openresty hsts include subdomains: true                     
       Openresty hsts max age:        15724800                 
       Openresty hsts preload:        false                    
       Openresty image:               dokku/openresty-docker-proxy:0.8.0
       Openresty letsencrypt email:                            
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4k                       
       Openresty proxy buffering:     on                       
       Openresty proxy buffers:       8 4k                     
       Openresty proxy busy buffers size: 8k                       
       Openresty proxy read timeout:  60s                      
       Openresty underscore in headers: off                      
       Openresty x forwarded for value: $remote_addr             
       Openresty x forwarded port value: $server_port             
       Openresty x forwarded proto value: $scheme                  
       Openresty x forwarded ssl:                              
=====> [...REDACTED APPNAME...] ports information
       Ports map:                     
       Ports map detected:            http:80:5000 https:443:5000
=====> [...REDACTED APPNAME...] proxy information
       Proxy computed type:           traefik
       Proxy enabled:                 true
       Proxy global type:             traefik
       Proxy type:                    
=====> [...REDACTED APPNAME...] ps information
       Deployed:                      true
       Processes:                     5
       Ps can scale:                  false
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             unless-stopped
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 79841b50862)
       Status worker 1:               running (CID: 7dda1f0ce30)
       Status worker 2:               running (CID: 2610de456bf)
       Status worker 3:               running (CID: 561a6051db2)
       Status worker 4:               running (CID: d0a26ac5407)
=====> [...REDACTED APPNAME...] registry information
       Registry computed image repo:        dokku/[...REDACTED APPNAME...]
       Registry computed push on release:   false
       Registry computed server:            
       Registry global image repo template: 
       Registry global push on release:     
       Registry global server:              
       Registry image repo:                 
       Registry push extra tags:            
       Registry push on release:            
       Registry server:                     
       Registry tag version:                
=====> [...REDACTED APPNAME...] resource information
=====> [...REDACTED APPNAME...] scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> [...REDACTED APPNAME...] scheduler-docker-local information
       Scheduler docker local init process: true                     
       Scheduler docker local parallel schedule count:                          
=====> [...REDACTED APPNAME...] scheduler-k3s information
       Scheduler k3s computed deploy timeout:       300s
       Scheduler k3s computed image pull secrets:   
       Scheduler k3s computed letsencrypt server:   prod
       Scheduler k3s computed namespace:            default
       Scheduler k3s computed rollback on failure:  false
       Scheduler k3s deploy timeout:                
       Scheduler k3s global deploy timeout:         300s
       Scheduler k3s global image pull secrets:     
       Scheduler k3s global ingress class:          nginx
       Scheduler k3s global kube context:           
       Scheduler k3s global kubeconfig path:        /etc/rancher/k3s/k3s.yaml
       Scheduler k3s global letsencrypt email prod: 
       Scheduler k3s global letsencrypt email stag: 
       Scheduler k3s global letsencrypt server:     prod
       Scheduler k3s global namespace:              default
       Scheduler k3s global network interface:      eth0
       Scheduler k3s global rollback on failure:    false
       Scheduler k3s image pull secrets:            
       Scheduler k3s letsencrypt server:            
       Scheduler k3s namespace:                     
       Scheduler k3s rollback on failure:           
=====> [...REDACTED APPNAME...] storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:                                     
=====> [...REDACTED APPNAME...] traefik information
       Traefik api enabled:           false                    
       Traefik api vhost:             traefik.dokku.me         
       Traefik basic auth password:                            
       Traefik basic auth username:                            
       Traefik dashboard enabled:     false                    
       Traefik image:                 traefik:v3.0.4           
       Traefik letsencrypt email:     [...REDACTED E-MAIL...]        
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR                    
       Traefik http entry point:      http                     
       Traefik https entry point:     https

How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Using the install script (deb) on a physical server.

Additional information

Latest lines of the command with trace:on:

+ rm -rf /tmp/dokku-3085019-suppress_output.XXMYRx
+ return 0
+ docker container run --rm -v /var/lib/dokku/services/redis/rtest/data:/data redis:7.2.5 bash -c 'rm -f /data/dump.rdb'
+ cat
/var/lib/dokku/plugins/enabled/redis/functions: line 187: /var/lib/dokku/services/redis/rtest/data/dump.rdb: Permission denied
 !     Failed to execute dokku command over ssh: exit code 0
 !     If there was no output from Dokku, ensure your configured SSH Key can connect to the remote server

Permissions of the folder:

$ ls -lar /var/lib/dokku/services/redis/rtest
total 44
-rw-r----- 1 dokku dokku   65 Jul 17 17:22 PASSWORD
-rw-r--r-- 1 dokku dokku    0 Jul 17 17:22 LINKS
-rw-r--r-- 1 dokku dokku    6 Jul 17 17:22 IMAGE_VERSION
-rw-r--r-- 1 dokku dokku    6 Jul 17 17:22 IMAGE
-rw-r--r-- 1 dokku dokku   64 Jul 17 17:22 ID
-rw-r--r-- 1 dokku dokku    1 Jul 17 17:22 ENV
-rw-r--r-- 1 dokku dokku    6 Jul 17 17:22 DATABASE_NAME
drwxr-xr-x 2   999 dokku 4096 Jul 17 17:23 data
-rw-r--r-- 1 dokku dokku    1 Jul 17 17:22 CONFIG_OPTIONS
drwxr-xr-x 2 dokku dokku 4096 Jul 17 17:22 config
drwxr-xr-x 5 dokku dokku 4096 Jul 17 17:22 ..
drwxr-xr-x 4 dokku dokku 4096 Jul 17 17:22 .

The data folder is empty.

After changing the permission with chown dokku:dokku /var/lib/dokku/services/redis/rtest/data it continues but fails at start of redis; after that the permission will be set back as listed above.

       1:M 17 Jul 2024 15:39:04.719 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:04.719 # Fatal error loading the DB, check server logs. Exiting.

Full output

$ dokku redis:import rtest < redis.dump
=====> Starting container
       Waiting for container to be ready
Waiting for dokku-redis-rtest:6379  .nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.nc: getaddrinfo: Name does not resolve
.  ERROR: unable to connect
=====> Start of rtest container output
       1:C 17 Jul 2024 15:39:04.140 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:04.140 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:04.140 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:04.140 * Configuration loaded
       1:M 17 Jul 2024 15:39:04.140 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:04.140 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:04.140 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:04.140 * Server initialized
       1:M 17 Jul 2024 15:39:04.140 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:04.140 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:04.718 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:04.718 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:04.718 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:04.718 * Configuration loaded
       1:M 17 Jul 2024 15:39:04.718 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:04.719 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:04.719 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:04.719 * Server initialized
       1:M 17 Jul 2024 15:39:04.719 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:04.719 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:05.178 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:05.178 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:05.178 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:05.178 * Configuration loaded
       1:M 17 Jul 2024 15:39:05.179 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:05.179 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:05.179 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:05.179 * Server initialized
       1:M 17 Jul 2024 15:39:05.179 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:05.179 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:05.878 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:05.878 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:05.878 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:05.878 * Configuration loaded
       1:M 17 Jul 2024 15:39:05.878 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:05.879 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:05.879 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:05.879 * Server initialized
       1:M 17 Jul 2024 15:39:05.879 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:05.879 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:06.977 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:06.977 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:06.977 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:06.977 * Configuration loaded
       1:M 17 Jul 2024 15:39:06.977 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:06.978 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:06.978 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:06.978 * Server initialized
       1:M 17 Jul 2024 15:39:06.978 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:06.978 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:08.895 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:08.895 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:08.895 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:08.895 * Configuration loaded
       1:M 17 Jul 2024 15:39:08.895 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:08.895 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:08.895 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:08.895 * Server initialized
       1:M 17 Jul 2024 15:39:08.895 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:08.895 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:12.458 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:12.458 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:12.458 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:12.458 * Configuration loaded
       1:M 17 Jul 2024 15:39:12.458 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:12.459 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:12.459 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:12.459 * Server initialized
       1:M 17 Jul 2024 15:39:12.459 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:12.459 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:19.170 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:19.170 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:19.170 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:19.170 * Configuration loaded
       1:M 17 Jul 2024 15:39:19.170 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:19.170 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:19.170 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:19.171 * Server initialized
       1:M 17 Jul 2024 15:39:19.171 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:19.171 # Fatal error loading the DB, check server logs. Exiting.
       1:C 17 Jul 2024 15:39:32.303 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can a
lso cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
       1:C 17 Jul 2024 15:39:32.303 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
       1:C 17 Jul 2024 15:39:32.303 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
       1:C 17 Jul 2024 15:39:32.303 * Configuration loaded
       1:M 17 Jul 2024 15:39:32.304 * monotonic clock: POSIX clock_gettime
       1:M 17 Jul 2024 15:39:32.304 # Failed to write PID file: Permission denied
       1:M 17 Jul 2024 15:39:32.304 * Running mode=standalone, port=6379.
       1:M 17 Jul 2024 15:39:32.304 * Server initialized
       1:M 17 Jul 2024 15:39:32.304 # Wrong signature trying to load DB from file
       1:M 17 Jul 2024 15:39:32.304 # Fatal error loading the DB, check server logs. Exiting.
=====> End of rtest container output
 !     Failed to execute dokku command over ssh: exit code 0
 !     If there was no output from Dokku, ensure your configured SSH Key can connect to the remote server
@josegonzalez
Copy link
Member

Are you running the import/export over ssh?

@chapterjason
Copy link
Author

@josegonzalez Yes, they are running with the ~/.dokku/contrib/dokku_client.sh-Client.

@chapterjason
Copy link
Author

Same result running as root on server:

root@server/home/user # dokku redis:create rtest
       Waiting for container to be ready
=====> Redis container created: rtest
=====> rtest redis service information
       Config dir:          /var/lib/dokku/services/redis/rtest/config
       Config options:
       Data dir:            /var/lib/dokku/services/redis/rtest/data
       Dsn:                 redis://...@dokku-redis-rtest:6379
       Exposed ports:       -
       Id:                  f638fcf6513216ce0cecd667847ba6f854a21f2ac8fb8726c70a633ac69911fe
       Internal ip:         172.17.0.2
       Initial network:
       Links:               -
       Post create network:
       Post start network:
       Service root:        /var/lib/dokku/services/redis/rtest
       Status:              running
       Version:             redis:7.2.5
root@server/home/user # dokku redis:export rtest > redis.dump
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
root@server/home/user # dokku redis:import rtest < redis.dump
/var/lib/dokku/plugins/enabled/redis/functions: line 187: /var/lib/dokku/services/redis/rtest/data/dump.rdb: Permission denied
root@server/home/user #

@josegonzalez
Copy link
Member

Other than that warning - which we should fix - seems fine here:

root@dokku:~# dokku redis:create rtest
       Waiting for container to be ready
=====> Redis container created: rtest
=====> rtest redis service information
       Config dir:          /var/lib/dokku/services/redis/rtest/config
       Config options:
       Data dir:            /var/lib/dokku/services/redis/rtest/data
       Dsn:                 redis://:269c828580fa125b1acffe1a68a18e437d38d8a9709c3ba531148611e8a82ee7@dokku-redis-rtest:6379
       Exposed ports:       -
       Id:                  ccd4de703c0fe5fe6da322a8a18d1fe18c85a90d745aa59488a92d50a7c9c1f9
       Internal ip:         172.17.0.12
       Initial network:
       Links:               -
       Post create network:
       Post start network:
       Service root:        /var/lib/dokku/services/redis/rtest
       Status:              running
       Version:             redis:7.2.5
root@dokku:~# dokku redis:export rtest > redis.dump
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
                                                                                                 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

root@dokku:~# ls redis.dump
redis.dump
root@dokku:~# ls -lah redis.dump
-rw-r--r-- 1 root root 88 Jul 19 00:19 redis.dump
root@dokku:~# dokku redis:import rtest < redis.dump

Can you run the following and post the output?

ls -lah /var/lib/dokku/services/redis/rtest/data

The dump.rdb in there might be owned by root for some reason and therefore you don't have access to write to it.

@chapterjason
Copy link
Author

chapterjason commented Jul 19, 2024 •
edited
Loading 

root@server~ # dokku redis:logs rtest
1:C 19 Jul 2024 04:53:14.474 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:C 19 Jul 2024 04:53:14.474 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19 Jul 2024 04:53:14.474 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 19 Jul 2024 04:53:14.474 * Configuration loaded
1:M 19 Jul 2024 04:53:14.474 * monotonic clock: POSIX clock_gettime
1:M 19 Jul 2024 04:53:14.475 # Failed to write PID file: Permission denied
1:M 19 Jul 2024 04:53:14.475 * Running mode=standalone, port=6379.
1:M 19 Jul 2024 04:53:14.475 * Server initialized
1:M 19 Jul 2024 04:53:14.475 * Ready to accept connections tcp
root@server~ # ls -lah redis.dump
-rw-r--r-- 1 root root 0 Jul 19 06:53 redis.dump
root@server~ # dokku redis:import rtest < redis.dump
/var/lib/dokku/plugins/enabled/redis/functions: line 187: /var/lib/dokku/services/redis/rtest/data/dump.rdb: Permission denied
root@server~ # dokku redis:logs rtest
 !     Service container does not exist

root@server~ # ls -lah /var/lib/dokku/services/redis/rtest/data
total 8.0K
drwxr-xr-x 2   999 dokku 4.0K Jul 19 06:53 .
drwxr-xr-x 4 dokku dokku 4.0K Jul 19 06:53 ..

The folder is empty.

@chapterjason
Copy link
Author

chapterjason commented Jul 19, 2024 •
edited
Loading

Had the case that there is a file:

drwxr-xr-x 2   999 dokku           4.0K Jul 19 21:51 .
drwxr-xr-x 4 dokku dokku           4.0K Jul 19 06:56 ..
-rw------- 1   999 systemd-journal   88 Jul 19 21:51 dump.rdb

Really confusing... 😄

systemd-journal also resolved in 999

@chapterjason
Copy link
Author

I think I've fixed it like this:

  suppress_output service_container_rm "$SERVICE"
  "$DOCKER_BIN" container run --rm -v "$SERVICE_HOST_ROOT/data:/data" "$PLUGIN_IMAGE:$PLUGIN_IMAGE_VERSION" bash -c "rm -f /data/dump.rdb"
+ "$DOCKER_BIN" container run --rm -i -v "$SERVICE_HOST_ROOT/data:/data" "$PLUGIN_IMAGE:$PLUGIN_IMAGE_VERSION" bash -c "cat > /data/dump.rdb" < <(cat)
- cat >"$SERVICE_HOST_ROOT/data/dump.rdb"
  "$DOCKER_BIN" container run --rm -v "$SERVICE_HOST_ROOT/data:/data" "$PLUGIN_IMAGE:$PLUGIN_IMAGE_VERSION" bash -c "chown redis: /data/dump.rdb"
  suppress_output service_start "$SERVICE"

It also makes sense, as we are trying to manipulate a data fs-store outside from the actual used container.

@josegonzalez
Copy link
Member

Wait so was there a file inside the container or outside the container?

@chapterjason
Copy link
Author

I've never checked inside the container. I just realized that we always interact with the dump file from within temporary or the service container itself, except for the stdin redirect to the file (write operation) is running on the host system.

@josegonzalez
Copy link
Member

Sorry I'm asking where this is from: #248 (comment)

You mentioned previously there was nothing at that directory but now there is? Just trying to figure out whats going on since your comments aren't super clear and I'm unable to replicate this otherwise.

@chapterjason
Copy link
Author

Here the state after every command, just splitted for better overview.

Cleanup

root@server ~ # dokku redis:destroy rtest --force
=====> Deleting rtest
=====> Pausing container
       Container paused
       Removing container
       Removing data
=====> Redis container deleted: rtest

root@server ~ # ls -lar /var/lib/dokku/services/redis/rtest/data
ls: cannot access '/var/lib/dokku/services/redis/rtest/data': No such file or directory

Create

root@server ~ # dokku redis:create rtest
       Waiting for container to be ready
=====> Redis container created: rtest
=====> rtest redis service information
       Config dir:          /var/lib/dokku/services/redis/rtest/config
       Config options:
       Data dir:            /var/lib/dokku/services/redis/rtest/data
       Dsn:                 redis://:[...]@dokku-redis-rtest:6379
       Exposed ports:       -
       Id:                  fc057791ef92f0bacfceabf9e9bc11cbae1541de583768349c4b01cda21a0160
       Internal ip:         172.17.0.7
       Initial network:
       Links:               -
       Post create network:
       Post start network:
       Service root:        /var/lib/dokku/services/redis/rtest
       Status:              running
       Version:             redis:7.2.5

root@server ~ # ls -lar /var/lib/dokku/services/redis/rtest/data
total 8
drwxr-xr-x 4 dokku dokku 4096 Jul 29 19:58 ..
drwxr-xr-x 2   999 dokku 4096 Jul 29 19:58 .

Export

root@server ~ # dokku redis:export rtest > redis.dump
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

root@server ~ # ls -lar /var/lib/dokku/services/redis/rtest/data
total 12
-rw------- 1   999 systemd-journal   88 Jul 29 19:59 dump.rdb
drwxr-xr-x 4 dokku dokku           4096 Jul 29 19:58 ..
drwxr-xr-x 2   999 dokku           4096 Jul 29 19:59 .

root@server ~ # ls -lar redis.dump
-rw-r--r-- 1 root root 88 Jul 29 19:59 redis.dump

Import

root@server ~ # dokku redis:import rtest < redis.dump
/var/lib/dokku/plugins/enabled/redis/functions: line 190: /var/lib/dokku/services/redis/rtest/data/dump.rdb: Permission denied

root@server ~ # ls -lar /var/lib/dokku/services/redis/rtest/data
total 8
drwxr-xr-x 4 dokku dokku 4096 Jul 29 19:59 ..
drwxr-xr-x 2   999 dokku 4096 Jul 29 20:00 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@josegonzalez @chapterjason