[FR] Allow expiration and deletion of unpublished shares

[Tolriq]

Currently using a public alias for users to upload private files (like logs) so I can access them securely.

This is a follow up to #1275 (comment)

I'd like to have a way for those shares to expire and be deleted too, without having to publish them as no one should access them.

From your proposal I think a setting (per alias or global) to also delete unpublished share would be perfect, automatic share publish does not work for privacy and security reason.

[eikek]

Also somewhat related: #573

[Tolriq]

Small bump to know if there's short term plans or not on this? I stupidly stopped manually purging so should do it again if not planned as the list grows fast :p

[eikek]

Hi @Tolriq I'm sorry, but there is currently no ETA. I'd like to address this for the next version. But not sure when this will be.

[Tolriq]

Ok no problem back to deleting, this list grows :)

BTW don't know if it's normal but the /app/uploads only list 100 entries with no way to navigate the rest (But show the proper size)

Your Shares #100/215.79M

While the limit is not that a big deal, maybe still showing the actual number of files too could be nice.

[Tolriq]

Hi there, any chance to have this feature implemented ?

Do you take donations or anything? Some users are uploading multiple times things and manual cleanup really have became a pain :(

[eikek]

hi @Tolriq I'm currently too busy to tackle all that :/. how do you do the manual cleanup now? Perhaps we can have not-so-great-but-automatic solution anyways?

Edit: I haven't had a closer look, I'll do that and see if there maybe is a quick way.

[Tolriq]

Currently doing 4 clicks per uploads, click the id, then details, then delete then yes.

Any kind of script that can run in docker would be nice yes.
I currently store the files in a file folder so I can easily script the file part deletion, and a script that delete the entries if the files are missing could work too.

I just don't really know how the app work to easily try to figure out something.

Thanks.

[eikek]

ok, so that means you only need to delete shares you own, right?

[Tolriq]

Yes, I have an alias where people can upload things to. They no more have access to those files after and I can see them via the your share page.

They do have a validity time, but it's not actually applied so the files are currently never removed.

All I need is a way to either automatically remove the files when the validity time is expired, or eventually a quick delete way from the interface. But 4 clicks + delays and different positions when there's dozens of uploads per day is no more working.

[eikek]

ok, so this is a bit easier, since we can use the api to write a simple shell script. It would be more difficult if you had to delete things from other users. So this is something I typed together very quick:

#!/usr/bin/env bash
# you must have curl, jq and coreutils
set -euo pipefail

username="${SHARRY_USER:-}"
password="${SHARRY_PASSWORD:-}"
sharry_url="${SHARRY_URL:-}"
created_before="${SHARRY_REMOVE_BEFORE:-}"

temp_dir=$(mktemp -d "sharry-delete.XXXXXX")
#temp_dir="sharry-delete.A2zWWn"

json_header='Content-Type: application/json'
auth_tpl='{
  "account":$username,
  "password":$password
}'

trap cleanup 1 2 3 6 ERR

debug() {
    printf "%s\n" "$*" >&2
}

cleanup() {
    if ! [ "$temp_dir" == "." ]; then
        debug "Cleanup temporary files: $temp_dir"
        rm -rf "$temp_dir"
    fi
    exit
}

if [ -z "sharry_url" ]; then
    debug "The url to sharry is required"
    debug "  > Set env var SHARRY_URL"
    exit 1
fi

if [ -z "$username" ] || [ -z "$password" ]; then
    debug "sharry login data is needed."
    debug "  > set env vars SHARRY_USER and SHARRY_PASSWORD"
    exit 1
fi

if [ -z "$created_before" ]; then
    debug "No SHARRY_REMOVE_BEFORE date set"
    debug "  > Set this to a date, where any share created before that you want to be deleted"
    debug "  > Use RFC-3339 format, like 2006-08-14 02:34:56-06:00"
    exit 1
else
    before_ms=$(date "+%s" -d "$created_before")
    before_ms=$(( $before_ms * 1000 ))
fi

# login
auth_file="${temp_dir}/auth.json"
if ! [ -e "$auth_file" ]; then
    jq --null-input --arg username "$username" --arg password "$password" "$auth_tpl" | \
        curl -sfSL -H $json_header "${sharry_url}/api/v2/open/auth/login" -d @- > "$auth_file"
fi


# get list of shares
token=$(jq -r '.token' "$auth_file")
shares_file="${temp_dir}/shares.json"
if ! [ -e "$shares_file" ]; then
    curl -sfSL -H $json_header -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/search" > "$shares_file"
fi

# filter out those to delete
delete_file="${temp_dir}/delete.json"
if ! [ -e "$delete_file" ]; then
    cat "$shares_file" | jq ".items[] |. +{until:(.validity + .created)}  | select(.until < $before_ms)" > "$delete_file"
fi

# present and ask
count=$(cat "$delete_file" | jq -r .id | wc -l)
cat "$delete_file" | jq
read -p "Delete these $count shares? (y/n)" confirm

if [ "$confirm" == "y" ]; then
    while read id; do
        echo "Deleting share $id"
        curl -sfSL -X DELETE -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/$id"
    done < <(cat "$delete_file" | jq -r .id)
fi

of course, before running, read and verify :-) The idea is that you could run this script periodically, the input is your username and password and a timestamp. Every share whose created + validity is below that time will be deleted. (you need to adopt a bit to be non-interacitve and probably replace the fixed timstamp with the current date-time…)

[Tolriq]

OMG, I'm so sorry, I absolutely did not thought about checking what the API provided ...

Thanks so much for the script, it makes me gain a lot of time.

[eikek]

You're welome and no worries! Perhaps it is useful later as well (might be even myself 😄)

[Tolriq]

Ok so it kinda works but the API have the same limitation as the GUI it only returns the last 100 shares.

And in my case what I need to delete is often after the first 100.

Edit: Yes seems you have a couple of take(100) in the code :(

[eikek]

Ah yes, you are right, I always forget about it. It is left from the very first days of the project 😞. So, I'd then suggest to make it a bit uglier and search directly on the database. There you have all the power…

#!/usr/bin/env bash
# you must have curl, jq and coreutils
# you must set PGPASSWORD to the postgres password
set -euo pipefail

sharry_username="${SHARRY_USER:-}"
sharry_password="${SHARRY_PASSWORD:-}"
sharry_url="${SHARRY_URL:-}"
created_before="${SHARRY_REMOVE_BEFORE:-}"
pg_user="${PGUSER:-}"
pg_host="${PGHOST:-}"
pg_db="${PGDB:-}"

temp_dir=$(mktemp -d "sharry-delete.XXXXXX")
#temp_dir="sharry-delete.VCiw1f"

json_header='Content-Type: application/json'
auth_tpl='{
  "account":$sharry_username,
  "password":$sharry_password
}'

trap cleanup 1 2 3 6 ERR

debug() {
    printf "%s\n" "$*" >&2
}

cleanup() {
    if ! [ "$temp_dir" == "." ]; then
        debug "Cleanup temporary files: $temp_dir"
        rm -rf "$temp_dir"
    fi
    exit
}

if [ -z "sharry_url" ]; then
    debug "The url to sharry is required"
    debug "  > Set env var SHARRY_URL"
    exit 1
fi

if [ -z "$sharry_username" ] || [ -z "$sharry_password" ]; then
    debug "sharry login data is needed."
    debug "  > set env vars SHARRY_USER and SHARRY_PASSWORD"
    exit 1
fi

if [ -z "$pg_user" ]; then
    debug "Set PGUSER env var to the postgres user"
    debug "And PGPASSWORD to the corresponding password"
    exit 1
fi
if [ -z "$pg_host" ]; then
    debug "Set PGHOST env var to the postgres host"
    exit 1
fi
if [ -z "$pg_db" ]; then
    debug "Set PGDB env var to the postgres database for sharry"
    exit 1
fi


if [ -z "$created_before" ]; then
    debug "No SHARRY_REMOVE_BEFORE date set"
    debug "  > Set this to a date, where any share created before that you want to be deleted"
    debug "  > Use RFC-3339 format, like 2006-08-14 02:34:56-06:00"
    exit 1
else
    before_ms=$(date "+%s" -d "$created_before")
    before_ms=$(( $before_ms * 1000 ))
fi

# get list of shares
shares_file="${temp_dir}/shares.json"
if ! [ -e "$shares_file" ]; then
    # curl -sfSL -H $json_header -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/search" > "$shares_file"
    psql -t -h "$pg_host" -U "$pg_user" "$pg_db" -c "
select json_object('id': s.id, 'name': s.name_)
from share s
inner join account_ a on a.id = s.account_id
where a.login = '$sharry_username' AND
   (extract(epoch from s.created) * 1000 + s.validity) < $before_ms" > "$shares_file"
fi

# present and ask
count=$(cat "$shares_file" | jq -r .id | wc -l)

if [ $count -eq 0 ]; then
    echo "Nothing to delete"
    exit
else
    cat "$shares_file" | jq
    read -p "Delete these $count shares? (y/n)" confirm

    if [ "$confirm" == "y" ]; then
        # login and delete each
        auth_file="${temp_dir}/auth.json"
        if ! [ -e "$auth_file" ]; then
            jq --null-input --arg sharry_username "$sharry_username" \
               --arg sharry_password "$sharry_password" "$auth_tpl" | \
                curl -sfSL -H "$json_header" "${sharry_url}/api/v2/open/auth/login" -d @- > "$auth_file"
        fi
        token=$(jq -r '.token' "$auth_file")

        while read id; do
            echo "Deleting share $id"
            curl -sfSL -X DELETE -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/$id"
        done < <(cat "$shares_file" | jq -r .id)
    fi
fi
cleanup

Again, needs to change as you see fit. It is probably nicer to remove the created_before argument and use postgres to refer to the current date.

[Tolriq]

Thanks, I must admit I prefer the full API version as I can run it anywhere in my normal scheduled tasks.
In docker with the DB not exposed this is a little more complicated.

Do you think you can increase the value or make it configurable?

[eikek]

Yes, I think as a preliminary solution we can make a config setting

[Tolriq]

So fully back from holidays and a millions uploads to clean :)

Can you give me some pointer about how you want the option to be defined in the settings and how I can pass the settings down to the functions? So I can try to make a PR.

[eikek]

Hi @Tolriq sorry to hear, hope you had good holidays! I just took some time and pushed some code to add a max size setting and also enable paging requests for searching shares.

[Tolriq]

Thanks a lot for this.

[Tolriq]

And tested it works (But the value is not used by default, you need to pass it via the API calls and so it's not applied to the GUI).
Could be nice to have more than 100 in the web UI but it's a details, I can now automate the cleanup.

Delete these 1714 shares? (y/n)

That would have been a lot of clicks :)

Do you have a ko-fi page or something?

[eikek]

Oh yes that would have been a lot of clicks 😆 I know it's not yet in the ui, but I figured the api is a good start and lets you continue to do the cleanup without the greatest possible hassle :)