rclone support in netrocks (via FTP) #1816

unxed · 2023-08-29T10:01:58Z

Users continue to suffer without cloud services support in far2l's netrocks. And suddenly I saw this:
https://rclone.org/commands/rclone_serve_ftp/

It turns out that rclone itself can work as FTP server, giving "proxy" access to a remote host.

It would be great to add to netrocks the ability to automatically launch rclone with the necessary parameters, and only then connect via FTP. So that everything is configured in one place — in netrocks itself.

phanex · 2023-08-29T10:08:36Z

It could be super!
I see it in the following way:
The user opens netrocks and adds a remote.
In the drop-down, he chooses rclone.
In the connection address, he enters the existing remote name + optionally a path.
...
Magic!

elfmz · 2023-08-30T12:12:35Z

There is similar option for 'file' protocol. It can be extended to all protocols, is it looks ok?

…-connection time adjustement setting

elfmz · 2023-08-30T19:17:56Z

May try out on branch nr-improvements. In connection settings - press Extra settings and you may setup arbitrary command to be executed on connection opening

phanex · 2023-08-30T19:51:14Z

So did I understand right? It will be possible to do following:

I choose ftp.
I have an option to execute command in the protocol settings.
I write my connection data such as remote name and if needed working directory.
In the protocol settings, I am using variables like $HOST and so on.
And then everything works.

phanex · 2023-08-30T20:16:22Z

It would be also great if there were another command in protocol options, like "execute on closing a connection". To stop rclone server afterwards.

elfmz · 2023-08-30T20:25:45Z

Note that there can be multiple far2l having same connection opened, also even single far2l may have multiple times opened same connection (background copying or browsing same connection on both panels) so connect/disconnect scripts must then have some kind of reference counting.. (that ideally would be good to be resilient to abnormal process termination..)

atolismesh · 2023-08-30T22:01:54Z

May try out on branch nr-improvements.

Can't compile this branch -

far2l/NetRocks/src/Protocol/ProtocolInitCommand.cpp:47:37: error: ‘time’ was not declared in this scope

  47 |                 time_t time_limit = time(NULL) + std::max(3, protocol_options.GetInt("CommandTimeLimit", 30));
     |                                     ^~~~

elfmz · 2023-08-31T06:47:15Z

Can't compile this branch -

fixed, try now

atolismesh · 2023-08-31T16:52:21Z

Now it compiles successfully.
Exploring new features
Try to integrate with
rclone serve sftp remote: --user $USER --pass $PASSWORD &

elfmz · 2023-08-31T16:55:54Z

i think i will also make 'deinit' command with possibility to distinguish first init / last deinit so it will be possible to properly start/shutdown rclone

atolismesh · 2023-08-31T16:58:43Z

It'll be great!
As I know, rclone serve can be deinited only by kill

atolismesh · 2023-08-31T17:10:40Z

~~rclone serve sftp remote: --user $USER --pass $PASS &~~
~~works great, as expected~~

atolismesh · 2023-08-31T18:16:00Z

Stable errors:
╔═══════════════════════════════════════ Operation failed ════════════════════════════════════════╗
║ Timeout - <5>NOTICE: Loaded 1 authorized keys from "/home/user/.ssh/authorized_keys"◙<5>NOTICE ║
║ SFTP server listening on 127.0.0.1:2022◙ (110) ║
╟─────────────────────────────────────────────────────────────────────────────────────────────────╢
║ { Ok } ║
╚═════════════════════════════════════════════════════════════════════════════════════════════════╝

rclone correctly started sftp server.
But netrocks do not try to connect it.

atolismesh · 2023-08-31T19:04:35Z

May be it is waiting for the rclone process exit?

elfmz · 2023-08-31T19:07:49Z

Yes, command is expected to (quickly) exit.

atolismesh · 2023-08-31T19:40:02Z

Run a script, not rclone directly - it was my first idea.
But it didn't work.
Adding redirects to the script made a trick:

#!/bin/bash
rclone -vv serve sftp gdrive: --user $USER --pass $PASSWORD  >/dev/null 2>&1 &

Now it seems to work as expected.

elfmz · 2023-08-31T19:57:34Z

Its cuz other end of stdout is being closed just on exit of command, but rclone wants to write it smthng and gets SIGPIPE then..

elfmz · 2023-08-31T20:36:13Z

Added support for 'deinit' command and added way to do shared init/deinit across different NetRocks connection instances: just do if [ "$SINGULAR" = 1 ]; then ... in your init/deinit scripts

atolismesh · 2023-08-31T20:47:37Z

Do you have any idea how to 'deinit' right rclone process?

elfmz · 2023-08-31T20:49:55Z

if you can identify it by command line, then
ps aux | awk ' { if (...) { system("kill "$2); }; }'

atolismesh · 2023-08-31T21:12:44Z

Может лучше сохранять в NetRocks PID запущенного rclone?
А сам PID получать как exitcode запущенного 'init' скрипта?
Тогда точно не промахнемся с kill.
Т.е. в скрипте использовать следующую конструкцию:

#!/bin/bash
rclone  serve sftp gdrive: --user $USER --pass $PASSWORD  >/dev/null 2>&1 &
exit $!

atolismesh · 2023-08-31T21:24:43Z

Или так - отлавливать в NetRocks последнюю выведенную в скрипте строку и сохранять ее в $NRPID, а затем в deinit скрипте делать kill $NRPID:

#!/bin/bash
rclone  serve sftp gdrive: --user $USER --pass $PASSWORD  >/dev/null 2>&1 &
echo $!
exit

unxed · 2023-08-31T21:29:48Z

Я не понял, можно, наконец, через фар в облака ходить, не выкачивая содержимое целиком?

atolismesh · 2023-08-31T21:31:08Z

Да, конечно! rclone не выкачивает ничего - только отображает в виде виртуальной файловой системы

unxed · 2023-08-31T22:27:28Z

А как там какой-нибудь Яндекс.Диск цепануть теперь? Вместо gdrive что-то другое писать?

elfmz · 2023-09-01T07:07:46Z

Добавил переменную $STORAGE которая указывает на уникальный для данного конекшена путь в который можно положить PID а потом в deinit вычитать его. И удалить, чтож мусор не оставялть.
И вмержил все в мастер

unxed · 2023-09-01T07:46:01Z

А можно, пожалуйста, сделать какой-нибудь Wizard, куда просто логин и пароль от Яндекса/ДругихСервисов говоришь, и он всё настраивает и прописывает? Это ж не сложно должно быть, давайте пожалеем пользователей :)

elfmz · 2023-09-01T07:47:44Z

Так rclone сам по себе такой визард. Просто консольный

unxed · 2023-09-01T07:48:21Z

| Extra environment variables available for command process: |
| $HOST $PORT $USER $PASSWORD $DIRECTORY |

— забыта $STORAGE

unxed · 2023-09-01T07:49:05Z

Так rclone сам по себе такой визард. Просто консольный

ну блин, требовать от пользователя писать скрипт запуска rclone (ещё и несколько) просто чтоб стянуть файл из облака? то есть конечно оч классно что оно появилось для тех, кто в состоянии разобраться)) теперь для всех остальных бы

unxed · 2023-09-01T07:55:52Z

может, можно сделать штатные шаблоны этих скриптов хотя бы для яндекса и гугла? там же только логин и пароль меняется, я правильно понимаю?

atolismesh · 2023-09-01T09:46:41Z

Добавил переменную $STORAGE

So, init script for sftp + rclone now looks like:

#!/bin/bash
rclone  serve sftp remote: --user $USER --pass $PASSWORD  >/dev/null 2>&1 &
echo $! >$STORAGE

Simple deinit script looks like:

#!/bin/bash
if [ "$SINGULAR" = 1 ] && [ -r $STORAGE ]; then kill $(< $STORAGE) ; fi

Right?

elfmz · 2023-09-01T10:20:50Z

1 need to check for if [ "$SINGULAR" = 1 ] also in init otherwise second subsequent opens of connection will launch another rclone
2 also to unlink $STORAGE in deinit to not leave uneeded files

unxed · 2023-09-01T10:43:25Z

Why sftp? Is encryption really needed within localhost? Also FTP is built in far2l without any additional dependencies.

atolismesh · 2023-09-01T11:07:49Z

Why sftp?

It just an example
ubuntu 23.04 repository has rclone 1.6.0-dev package missing ftp serve support
current rclone 1.6.3 from rclone.org has bug in ftp serve server - ftp dir/ls command do not output year in dates

elfmz · 2023-09-01T11:10:31Z

I'd also add that ftp has poor symlinks support, so if some cloud supports them, this support would be lost in such translation

atolismesh · 2023-09-01T11:14:02Z

1 need to check for if [ "$SINGULAR" = 1 ] a

init script for sftp + rclone now looks like:

#!/bin/bash
if [ "$SINGULAR" = 1 ] ; then 
	rclone  serve sftp remote: --user "$USER" --pass "$PASSWORD"  >/dev/null 2>&1 &
	echo $! >"$STORAGE"
fi

Simple deinit script looks like:

#!/bin/bash
if [ "$SINGULAR" = 1 ] && [ -r "$STORAGE" ]; then 
	kill $(< "$STORAGE") 
	unlink "$STORAGE"
fi

elfmz · 2023-09-01T11:16:34Z

almost ok, but better to take $STORAGE into quotes everywhere as theoretically home directory (where $STORAGE usually located in) may contain spaces
Same for $USER $PASSWORD

atolismesh · 2023-09-01T11:27:28Z

better to take $STORAGE into quotes

Thanks!
II remembered this but forgot to fix it
Fixed in script above

elfmz · 2023-09-01T11:34:49Z

BTW does it really needs /bin/bash ? from first look should work with /bin/sh

atolismesh · 2023-09-01T12:29:15Z

BTW does it really needs /bin/bash

As I remember, sh do not support $(<
So:

Init script for sftp + rclone for sh looks like:

#!/bin/sh
if [ "$SINGULAR" = 1 ] ; then 
	rclone  serve sftp remote: --user "$USER" --pass "$PASSWORD"  >/dev/null 2>&1 &
	echo $! >"$STORAGE"
fi

Simple deinit script for sh looks like:

#!/bin/sh
if [ "$SINGULAR" = 1 ] && [ -r "$STORAGE" ]; then 
	kill $(cat "$STORAGE") 
	unlink "$STORAGE"
fi

unxed · 2023-09-01T18:16:38Z

Has it been tested?

unxed · 2023-09-01T18:16:54Z

Can we close an issue now?

spvkgn · 2023-09-02T06:27:00Z

Еще можно просто к systemd прикрутить:
~/.config/systemd/user/rclone-serve@.service

[Unit]
Description=rclone: serve the remote %i over SFTP
After=network-online.target
Wants=network-online.target

[Service]
ExecStart= \
  %h/.local/bin/rclone serve sftp %i: \
    --no-auth \
    --config %h/.config/rclone/rclone.conf \
    --log-level INFO \
    --log-file /tmp/rclone-serve-%i.log \
    --umask 022

[Install]
WantedBy=default.target

Cтарт/стоп systemctl --user enable/disable --now rclone-serve@gdrive

Раньше так вручную запускал, вроде проблем не было.

phanex · 2023-09-10T12:02:36Z

atolismesh, может лучше так:

rclone serve sftp $HOST: --user $USER --pass $PASSWORD >/dev/null 2>&1 &
Ну, чтобы делать строку универсальной, а название ремоты писать прямо в параметрах соединения? Или я сильно туплю?

atolismesh · 2023-09-10T14:48:00Z

@phanex
$HOST здесь не подходит, так как это адрес хоста, на котором запускается rclone serve. В большинстве случаев это localhost, и во всех случаях это локальный адрес хоста, никак не связанный с облаком, к которому соединяется rclone.

Я для универсальности использую $EXTRA

rclone serve sftp "$EXTRA": --user "$USER" --pass "$PASSWORD" --addr :"$PORT" >/dev/null 2>&1 &

unxed · 2023-09-26T12:50:44Z

native implementation to be discussed in #1861

LihacheFF · 2024-04-29T19:24:30Z

Привет.
Как подключить облачное хранилище Terabox (terabox.com) в far2l?
Спасибо

unxed · 2024-04-29T19:54:15Z

Как подключить облачное хранилище Terabox (terabox.com) в far2l?

Надо сначала чтоб в rclone поддержку этого хранилища добавили. Если чаще лайкать и писать что нужна такая поддержка по ссылочкам, вероятность что сделают может стать выше (а может и не стать, но и лайкнуть же не трудно!).

We first need to add support for this storage provider to rclone. If you like more often and write that such support is needed via links, the likelihood that they will do so may become higher (or it may not be, but it’s not difficult to like!).

rclone/rclone#7496

https://forum.rclone.org/t/degoo-terabox-jiocloud-mediafire-support-rclone/35974/2
https://forum.rclone.org/t/can-rclone-add-support-for-terabox/29807/4

unxed · 2024-08-03T17:34:37Z

Пошаговая инструкция по использованию в NetRocks примерно любого протокола (Amazon S3, etc):

Устанавливаем rclone:

sudo apt install rclone

Настраиваем подключение, запоминаем его имя (name):

rclone config

Создаем в домашней папке два скрипта:

nr_rclone_up.sh

#!/bin/sh
if [ "$SINGULAR" = 1 ] ; then 
	rclone serve sftp "$EXTRA": --user "$USER" --pass "$PASSWORD" --addr :"$PORT" >/dev/null 2>&1 &
	sleep 3
	echo $! >"$STORAGE"
fi

nr_rclone_down.sh

#!/bin/sh
if [ "$SINGULAR" = 1 ] && [ -r "$STORAGE" ]; then 
	kill $(cat "$STORAGE") 
	unlink "$STORAGE"
fi

Ставим им право на выполнение:

chmod +x nr_rclone*

Создаем новое соединение в NetRocks.
Протокол: sftp
Хост: localhost
Порт: 2022
Режим входа: сохраненный пароль (saved password)
Имя пользователя: 1111 (любое не пустое, на самом деле)
Пароль: 1111 (любой не пустой, на самом деле)
Идём в Extra options
Command to execute on connect: ~/nr_rclone_up.sh
Command to execute on disconnect: ~/nr_rclone_down.sh
Extra string passed to command: имя (name) вашего соединения rclone с шага 2

Сохраняем всё. Подключаемся. Пользуемся.

Для каждого дополнительного соединения rclone указываем другой свободный порт: 2022, 2023 и т.д. Это позволяет запускать несколько облачных источников одновременно. И копировать между ними, если открыты в разных панелях.

PS: Для быстрых соединений sleep 3 можно заменить на 2 или 1. Для медленных — увеличить.

peterazmanov · 2024-09-30T00:07:02Z

Are there any advantages to this approach over rclone mount which mounts cloud storage as a FUSE file system?

phanex · 2024-09-30T07:13:57Z

Are there any advantages to this approach over rclone mount which mounts cloud storage as a FUSE file system?

For me, this is a no-brainer. There is no need to have FUSE installed to do mount and unmount a remote. It's very useful to be able to access a remote from time to time without any strings attached.

elfmz added a commit that referenced this issue Aug 30, 2023

NetRocks: init-connection command for any protocol (touch #1816); per…

ac56f94

…-connection time adjustement setting

This was referenced Sep 1, 2023

[multiarc] (macos ventura 13.0.1) редактирование форматов из custom.ini приводит к крэшу #1446

Open

netrocks: freezed upload to yandex.disk (webdav) #606

Closed

elfmz closed this as completed Sep 2, 2023

unxed changed the title ~~rclone support in netrocks~~ rclone support in netrocks (via FTP) Sep 13, 2023

rclone support in netrocks (via FTP) #1816

rclone support in netrocks (via FTP) #1816

Comments

unxed commented Aug 29, 2023 • edited Loading

phanex commented Aug 29, 2023

elfmz commented Aug 30, 2023

elfmz commented Aug 30, 2023 • edited Loading

phanex commented Aug 30, 2023 • edited Loading

phanex commented Aug 30, 2023

elfmz commented Aug 30, 2023 • edited Loading

atolismesh commented Aug 30, 2023

elfmz commented Aug 31, 2023

atolismesh commented Aug 31, 2023 • edited Loading

elfmz commented Aug 31, 2023

atolismesh commented Aug 31, 2023 • edited Loading

atolismesh commented Aug 31, 2023 • edited Loading

atolismesh commented Aug 31, 2023

atolismesh commented Aug 31, 2023

elfmz commented Aug 31, 2023 • edited Loading

atolismesh commented Aug 31, 2023 • edited Loading

elfmz commented Aug 31, 2023

elfmz commented Aug 31, 2023

atolismesh commented Aug 31, 2023

elfmz commented Aug 31, 2023

atolismesh commented Aug 31, 2023 • edited Loading

atolismesh commented Aug 31, 2023

unxed commented Aug 31, 2023

atolismesh commented Aug 31, 2023

unxed commented Aug 31, 2023

elfmz commented Sep 1, 2023

unxed commented Sep 1, 2023

elfmz commented Sep 1, 2023

unxed commented Sep 1, 2023

unxed commented Sep 1, 2023 • edited Loading

unxed commented Sep 1, 2023

atolismesh commented Sep 1, 2023 • edited Loading

elfmz commented Sep 1, 2023

unxed commented Sep 1, 2023

atolismesh commented Sep 1, 2023

elfmz commented Sep 1, 2023

atolismesh commented Sep 1, 2023 • edited Loading

elfmz commented Sep 1, 2023 • edited Loading

atolismesh commented Sep 1, 2023

elfmz commented Sep 1, 2023

atolismesh commented Sep 1, 2023

unxed commented Sep 1, 2023

unxed commented Sep 1, 2023

spvkgn commented Sep 2, 2023 • edited Loading

phanex commented Sep 10, 2023

atolismesh commented Sep 10, 2023 • edited Loading

unxed commented Sep 26, 2023

LihacheFF commented Apr 29, 2024

unxed commented Apr 29, 2024 • edited Loading

unxed commented Aug 3, 2024 • edited Loading

peterazmanov commented Sep 30, 2024

phanex commented Sep 30, 2024 • edited Loading

unxed commented Aug 29, 2023 •

edited

Loading

elfmz commented Aug 30, 2023 •

edited

Loading

phanex commented Aug 30, 2023 •

edited

Loading

elfmz commented Aug 30, 2023 •

edited

Loading

atolismesh commented Aug 31, 2023 •

edited

Loading

atolismesh commented Aug 31, 2023 •

edited

Loading

atolismesh commented Aug 31, 2023 •

edited

Loading

elfmz commented Aug 31, 2023 •

edited

Loading

atolismesh commented Aug 31, 2023 •

edited

Loading

atolismesh commented Aug 31, 2023 •

edited

Loading

unxed commented Sep 1, 2023 •

edited

Loading

atolismesh commented Sep 1, 2023 •

edited

Loading

atolismesh commented Sep 1, 2023 •

edited

Loading

elfmz commented Sep 1, 2023 •

edited

Loading

spvkgn commented Sep 2, 2023 •

edited

Loading

atolismesh commented Sep 10, 2023 •

edited

Loading

unxed commented Apr 29, 2024 •

edited

Loading

unxed commented Aug 3, 2024 •

edited

Loading

phanex commented Sep 30, 2024 •

edited

Loading