-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create ftpserver.lua #2345
Create ftpserver.lua #2345
Conversation
lua_modules/ftp/server.lua
Outdated
@@ -0,0 +1,157 @@ | |||
-- a simple ftp server | |||
USER = "test" | |||
PASS = "12345" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keep these in locals, and they will be accesses upvals in nested closures. See my FAQ for moy explanation:
local USER, PASS = "test", "12345"
lua_modules/ftp/server.lua
Outdated
USER = "test" | ||
PASS = "12345" | ||
local file,net,pairs,print,string,table = file,net,pairs,print,string,table | ||
data_fnc = nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local data_fnc = function() end
and then the listener becomes ftp_data:listen(20, function (s) data_fnc(s) end)
You still need the extra closure because you overwrite data_fnc/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and owerwrite with blank function after STOR,RETR is comlete?
lua_modules/ftp/server.lua
Outdated
local buf = "" | ||
local t = 0 | ||
socket:on("receive", function(c, d) | ||
a = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be a local.
lua_modules/ftp/server.lua
Outdated
for i in string.gmatch(d, "([^ \r\n]+)") do | ||
table.insert(a,i) | ||
end | ||
if(a[1] == nil or a[1] == "")then return end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local a1,a2=unpack(a)
and use a1
and a2
below instead of a[1]
and a[2]
; faster and generates less code.
lua_modules/ftp/server.lua
Outdated
table.insert(a,i) | ||
end | ||
if(a[1] == nil or a[1] == "")then return end | ||
if(s == 0 and a[1] == "USER")then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You know that the Lua syntax is if cond then ...
. This isn't C or Javascript 😃. IMO if a2 ~= USER then
is clearer, but even if you want to keep the () or even ((())) then add the spaces to make it readable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - wery more JS and Cpp make me use If()
automaticly
lua_modules/ftp/server.lua
Outdated
for k,v in pairs(l) do | ||
if(a[1] == "NLST")then | ||
sd:send(k.."\r\n") | ||
else |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I said previously multiple sends in a single CB is a no-no. So you should do something like
local resp={}
for k,v in pairs(file.list()) do
resp[#resp+1] = (a[1] == "NLST") and
(k.."\r\n") or
("+r,s"..v..",\t"..k.."\r\n")
end
multi_send(resp)
Where multi_send(resp)
is a separate routine declared above to do the multi-send. I'll discuss this separately. Note the sd
used here is an upval which wll create a resource leak.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just simplify code:
sd:send((a[1] == "NLST") and (k.."\r\n") or ("+r,s"..v..",\t"..k.."\r\n"))
@NeiroNx, Read my FAQ and particularly how to use Re the This non-blocking I/O can be a real PITA sometimes. 😄 |
lua_modules/ftp/server.lua
Outdated
sd:on("receive", function(cd, dd) | ||
f:write(dd) | ||
end) | ||
socket:on("disconnection", function(c) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be sd:on
instead of socket:on
?
turn it in module like "http"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make changes and make module like "http"
lua_modules/ftp/server.lua
Outdated
for k,v in pairs(l) do | ||
if(a[1] == "NLST")then | ||
sd:send(k.."\r\n") | ||
else |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just simplify code:
sd:send((a[1] == "NLST") and (k.."\r\n") or ("+r,s"..v..",\t"..k.."\r\n"))
lua_modules/ftp/server.lua
Outdated
USER = "test" | ||
PASS = "12345" | ||
local file,net,pairs,print,string,table = file,net,pairs,print,string,table | ||
data_fnc = nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and owerwrite with blank function after STOR,RETR is comlete?
lua_modules/ftp/server.lua
Outdated
table.insert(a,i) | ||
end | ||
if(a[1] == nil or a[1] == "")then return end | ||
if(s == 0 and a[1] == "USER")then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - wery more JS and Cpp make me use If()
automaticly
lua_modules/ftp/server.lua
Outdated
if b then | ||
sd:send(b) | ||
b=f:read(1024) | ||
else |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great module. This was on my wish list for some time.
Reading the next chunk of the file right before sending it and setting the buffer to nil right afterwards would save some memory.
b=f:read(1024)
sd:send(b)
b=nil
The logic would have to be adapted also.
Update file reading algoritm - reading chunk before sending and nil it after send called.
May I ask why ftp and not tftp? |
Your |
TFTP:
|
On some OSes you can mount FTP servers and many file managers can access FTP semi-transparently without mounting, so this will be a nice way to handle code on your favourite ESP-pets. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some hints on locals and memory leaking upvals.
lua_modules/ftp/ftpserver.lua
Outdated
return c:send("200 S OK\r\n") | ||
end | ||
if a1 == "PASV" then | ||
local _,ip = socket:getaddr() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
socket
-> c
lua_modules/ftp/ftpserver.lua
Outdated
return | ||
end | ||
if a1 == "RETR" then | ||
f = file.open(a2:gsub("%/",""),"r") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local f
end | ||
c:send("150 Accepted data connection\r\n") | ||
data_fnc = function(sd) | ||
sd:on("sent", function(cd) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if b
should be declared local before the sent callback. Just to avoid that b
in the lines below refers to a global variable.
lua_modules/ftp/ftpserver.lua
Outdated
sd:on("sent", function(cd) | ||
b=f:read(1024) | ||
if b then | ||
sd:send(b) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sd
-> cd
lua_modules/ftp/ftpserver.lua
Outdated
sd:send(b) | ||
b=nil | ||
else | ||
sd:close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sd
-> cd
lua_modules/ftp/ftpserver.lua
Outdated
return | ||
end | ||
if a1 == "STOR" then | ||
f = file.open(a2:gsub("%/",""),"w") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local f
lua_modules/ftp/README.md
Outdated
FTP server uses only one specified user and password to authenticate clients. | ||
All clients need authentication - anonymous access is not supported yet. | ||
All files have RW access. | ||
Directory creation do not supported because SPIFFS do not have that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While this is technically correct you might want to mention that you can mimic directories by prefixing file names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It took me some time to understand why this module doesn't work with the standard ftp client on my linux machine. E.g. the ls
command would hang after the server responded with
150 Accepted data connection
Reason is the module's assumption that a client will not connect to the data port before the actual LIST
command (or RETR
, STOR
). It seems that my ftp client already connects right after the server's response to PASV
but data_func
is still nil
at that time. As a consequence, the listen event is lost and the data transfer never starts.
I suggest to extend the logic in this module to handle the "early connect" scenario as laid out in the inline comments below. These changes work fine for me but I can't test whether they break the server for a client which connects late.
lua_modules/ftp/ftpserver.lua
Outdated
local createServer = function (user, pass) | ||
local data_fnc = nil | ||
ftp_data = net.createServer(net.TCP, 180) | ||
ftp_data:listen(20, function (s) if data_fnc then data_fnc(s) end end) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Proposal to handle "early connect" scenario:
if data_fnc then data_fnc(s) else data_sock = s end
With a local data_sock = nil
above.
lua_modules/ftp/ftpserver.lua
Outdated
data_fnc = nil | ||
c:send("226 Transfer complete.\r\n") | ||
end | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Proposal for "early connect" scenario:
if data_sock then
node.task.post(function() data_fnc(data_sock); data_sock = nil end)
end
return
This starts data_fnc
manually after the return.
lua_modules/ftp/ftpserver.lua
Outdated
sd:send(b) | ||
b=nil | ||
end | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also post data_fnc
here for "early connect".
lua_modules/ftp/ftpserver.lua
Outdated
end) | ||
c:send("226 Transfer complete.\r\n") | ||
end | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also post data_fnc
here for "early connect".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test with Filezilla - it do not connect before transfer. It send PASV every time
Fix "early connect" logic
Works very well in my environment after the latest changes - thanks a lot @NeiroNx for your contribution and dedication! |
The current implementation does not work in my environment (Windows and FileZilla). if a1 == "PASV" then
local _,ip = c:getaddr()
local _,_,i1,i2,i3,i4 = string.find(ip,"(%d+).(%d+).(%d+).(%d+)")
return c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..",0,20).\r\n")
end Changed into: if(a[1] == "PASV")then
local port, _ = c:getpeer()
local _,_,i1,i2,i3,i4 = string.find(IP,"(%d+).(%d+).(%d+).(%d+)")
c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..","..(port / 256)..","..(port % 256)..").\r\n")
if ftp_data ~= nil then
ftp_data:close()
end
ftp_data = net.createServer(net.TCP, 180)
ftp_data:listen(port, function (s)
if data_fnc then
data_fnc(s)
end
end)
return
end Then my connection with FileZilla in (passive mode) works correctly (on an integer build). |
@FrankX0 The only missing server constraint I can deduce from the linked information is that the server's data port should be >1023 in passive mode. This is achieved for sure by your dynamic port switching code. |
@devsaurus Yep! Then also FileZilla works!
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With these suggested changes, it will be possible to use ftp functionality from a browser (Chrome on Windows in my case).
end | ||
end) | ||
local b=f:read(1024) | ||
sd:send(b) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check for b not being nil, to prevent crashing reading an empty file.
if a1 == "QUIT" then | ||
return c:send("221 Goodbye\r\n", function (s) s:close() end) | ||
end | ||
c:send("500 Unknown error\r\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add support for SIZE command: this is generally used by web-browsers.
if a1 == "SIZE" then
local st,size
st = file.stat(string.sub(a2, 2))
if st then
return c:send("213 "..st.size.."\r\n")
else
return c:send("550 Could not get file size.\r\n")
end
end
lua_modules/ftp/ftpserver.lua
Outdated
return c:send("257 \""..cwd.."\" is your current directory\r\n") | ||
end | ||
cwd = a2 | ||
return c:send("250 OK. Current directory is "..cwd.."\r\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest to change to:
if a2 == "/" then
cwd = a2
return c:send("250 OK. Current directory is "..cwd.."\r\n")
end
return c:send("550 Failed to change directory.\r\n")
Needed to support web-browsers to retrieve the index of the root and prevent a CWD into a file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"cwd" in file is not main problem.
Some clients send a "LIST -a /test.txt" commands and i am do not know how-to process them.
Add SIZE and SYST commands, Fix "cwd" into file. Next stage will be check with file.stat() Test with "Chrome" and "ES Explorer"
need more tests, may be needs MLST,MLSD - but it needs more and more code and more memory for it |
remove debug print
@NeiroNx, you are still breaking the SDK rules in the LIST processing. You can't do for k,v in pairs(l) do
sd:send( a1 == "NLST" and k.."\r\n" or "-rwxrwxrwx 1 esp esp "..v.." Jan 1 2018 "..k.."\r\n")
end Also I've done a rework your code in the latest of my gists. I don't have the time or desire to create an alternative to your implementation, as you have done all of the hard work and have the energy to see this to completion. But copy this example and consider some of the Lua coding techniques in your own style 😄. For example:
Anyway have a look at this coding example and pick / choose anything that you feel might enhance your coding and implementation. Best regards Terry |
Hi @NeiroNx first of all thx for the great work. |
@HHHartmann ... please publish your changes. |
I have been a little concerned about @NeiroNx silence. I've got to get the LFS patch out and then I will be on a Greek island in May / early June. If he hasn't progressed it then I will finish this off then. |
if data_sock then | ||
node.task.post(function() data_fnc(data_sock);data_sock=nil end) | ||
end | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To make the list commands work use this data_fnc. tested with 70+ files
data_fnc = function(sd)
sd:on("sent", function(cd)
if #files > 0 then
local file = table.remove(files,1)
local size = l and l[file]
cd:send( a1 == "NLST" and file.."\r\n" or "-rwxrwxrwx 1 esp esp "..size.." Jan 1 2018 "..file.."\r\n")
else
l = nil
files = nil
cd:close()
data_fnc = nil
c:send("226 Transfer complete.\r\n")
collectgarbage()
end
end)
l = file.list();
files = {}
for file in pairs(l) do
table.insert(files, file)
end
if a1 == "NLST" then l = nil end
local file = table.remove(files,1)
local size = l and l[file]
sd:send( a1 == "NLST" and file.."\r\n" or "-rwxrwxrwx 1 esp esp "..size.." Jan 1 2018 "..file.."\r\n")
end
f:close() | ||
data_fnc = nil | ||
end) | ||
c:send("226 Transfer complete.\r\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The c:send("226 Transfer complete.\r\n")
has to be executed after disconnect only. Else it stops sending the file after a couple of blocks which results in cut off files.
The on disconnection
should look loke this: (also note the rename of the function param from c
to cd
)
sd:on("disconnection", function(cd)
f:close()
data_fnc = nil
c:send("226 Transfer complete.\r\n")
end)
@TerryE If you would not finish it, is there a way/best practice to hand over a PR to someone else (without commit rights)? |
@HHHartmann Gregor, The committers don't commit their own contributions anyway. Anyone is free to fork nodemcu-firmware or a fork or it and raise a PR. Let's give NeiroNx another week or so to do the work before anyone takes over. But if you do want to, then have a look at my gist ftpserver.lua for some suggested coding improvements. |
I think that @NeiroNx has timed out. I will close this PR and open another based on my re-implementation. No withstanding this, the impetus and original work has been done by NeiroNx so he or she deserves our thanks and full acknowledgment. 👍 |
Fixes #2343 .
dev
branch rather than formaster
.docs/en/*
.Some times need upload files withuot rs232 connection. FTP is best solution.