1. Problem with FTP
- Posted by SergioGelli Jan 08, 2010
- 1694 views
I am developing a program with built-in sockets FTP and need some help.
When the program is performing a communication with the FTP server and the connection drops, the program would stay waiting for an answer, for a very long time, and all program elements (buttons, windows, menus) are inaccessible.
Trying to solve this, I installed a timer to perform a procedure that performs an ABORT command after 30 seconds but the timer is also locked. Please, anyone have any idea how to solve this problem?
Thank in advance.
Sérgio Gelli
2. Re: Problem with FTP
- Posted by mattlewis (admin) Jan 08, 2010
- 1656 views
I am developing a program with built-in sockets FTP and need some help.
Could you elaborate a little on this? Does this mean using the v4 std library?
Matt
3. Re: Problem with FTP
- Posted by m_sabal Jan 08, 2010
- 1778 views
Here are a few suggestions (note: examples are based on eunet, from which Eu4 sockets are derived. Some of the procedure names may be a little different): 1) Poll the socket before each blocking send/recv call. Ex: status = eunet_poll(,100) 2) Use a non-blocking call instead. Ex: recvdata = eunet_recvfrom(socket,0) 3) If possible, figure out why the connection keeps dropping. The FTP program may be the lesser of several problem. 4) Modify the BLOCK_SIZE in sockets.e to be a smaller value. This will require more calls to recv for a given file, but you'll have more opportunities to check for a failed connection.
HTH, Michael J. Sabal
4. Re: Problem with FTP
- Posted by SergioGelli Jan 08, 2010
- 1638 views
Hi Matt and Michael
I am using "eunet.e" and "FTP stub" examples and V3.1.1.
The problem occurs when a line (as below) is executed and the connection to the Internt is very slow or dropped for some reason that does not matter here
eucode
status = eunet_send12(socketT,"NOOP "&13&10,0)
data = eunet_recv12(socketT,0)
end eucode
This is just one example, but the problem also occurs in all other commands (PASS, RETR, STOR and others)
The problem I have is that the program stops responding while waiting for response from the FTP, so I can not do any work on this program's window until it comes to server response (which eventually takes a long time)
5. Re: Problem with FTP
- Posted by m_sabal Jan 09, 2010
- 1569 views
The forum didn't respect the newlines in my last reply, making it somewhat hard to read.
The main issue is that you are using blocking calls. Review the suggestions I made in my last post, and make sure you are using the non-blocking versions (recvfrom, sendto).
6. Re: Problem with FTP
- Posted by sergelli Jan 11, 2010
- 1505 views
o.k The "eunet_recvfrom (socketR, MSG_DONTWAIT)" works well. However, it is not very different from the "eunet_recv (socketR, 0)" In both cases, I can not do anything until the file is downloaded I want a function to stop receiving a file (or do anything else) without closing the program
Currently, the ways to stop the collection are: 1-Close the program 2-Burning PID 3-Wait Internet fall 4-Turn Off Computer
So, I still unresolved for my problem.
7. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 12, 2010
- 1428 views
o.k The "eunet_recvfrom (socketR, MSG_DONTWAIT)" works well. However, it is not very different from the "eunet_recv (socketR, 0)" In both cases, I can not do anything until the file is downloaded I want a function to stop receiving a file (or do anything else) without closing the program
Currently, the ways to stop the collection are: 1-Close the program 2-Burning PID 3-Wait Internet fall 4-Turn Off Computer
So, I still unresolved for my problem.
If I understand correctly, the original problem (the GUI was frozen when the FTP connection dropped) is fixed, and now the GUI is responsive.
But now we have a new problem, in that although the GUI works, the program is still trying to finish downloading the file. And you want to change it so that when the FTP connection drops, the program immediately aborts the download?
8. Re: Problem with FTP
- Posted by m_sabal Jan 12, 2010
- 1458 views
The default block size is 4K. This means, in order to receive a file larger than 4K, you have to be using a loop. If this is a command-line program, use get_key to check for a keyboard interrupt. If using win32lib, check the documentation for doEvents() and add a global flag that can be used to terminate the loop early. I'm afraid I can't help with wxWindows, but I'm sure there's a similar way to check or process the event queue from inside a loop. If you are trying to interrupt a file transfer of 4K or less, you will have to modify the BLOCK_SIZE constant in eunet.e to a smaller value.
9. Re: Problem with FTP
- Posted by SergioGelli Jan 12, 2010
- 1439 views
1-Yes jimcbrown, I want to change it so that when the FTP connection drops, the program immediately aborts the download, AND WHEN I USE THE PROGRAM TO OTHER THINGS WHILE I'M WAITING FOR AN END COMMUNICATION
2-Sabal Yes, I use a "while" for large files, because only then can receive several blocks of a file that is larger than my bockSize.
I understand that I can use (and am using) a "onEvent", between calls to a "eunet_recvfrom" or "eunet_recv", yet when the connection is too slow, I have to wait long for any part of the program return to work.
If I am not mistaken, some servers are configured to disconnect a connection that does not respond only after 900 seconds. Nobody wants to wait that long to use a program.
As I wrote earlier, I tried to use a "timer()" to have a chance to abort the operation from time to time, but the "timer()" is also frozen, while the program tries to communicate.
It seems that the ftp server is the owner of my computer :)
10. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 12, 2010
- 1417 views
1-Yes jimcbrown, I want to change it so that when the FTP connection drops, the program immediately aborts the download, AND WHEN I USE THE PROGRAM TO OTHER THINGS WHILE I'M WAITING FOR AN END COMMUNICATION
I am confused as to what the problem is. Your wording suggests that you want toe FTP download to be aborted whenever you use the program to do anything else, which seems nonsensical to me.
I'm going to assume that
a) the FTP connection dropping continues to hang your GUI
b) the FTP download is not being aborted when the FTP connection fails
2-Sabal Yes, I use a "while" for large files, because only then can receive several blocks of a file that is larger than my bockSize.
I understand that I can use (and am using) a "onEvent", between calls to a "eunet_recvfrom" or "eunet_recv", yet when the connection is too slow, I have to wait long for any part of the program return to work.
If I am not mistaken, some servers are configured to disconnect a connection that does not respond only after 900 seconds. Nobody wants to wait that long to use a program.
As I wrote earlier, I tried to use a "timer()" to have a chance to abort the operation from time to time, but the "timer()" is also frozen, while the program tries to communicate.
It seems that the ftp server is the owner of my computer :)
I think you need to do the following:
a) use eunet_recvfrom() with MSG_DONTWAIT
b) change the BLOCK_SIZE from 4K to a much smaller value
c) chuck out your while loop. if you are simply looping using eunet_recv() or eunet_recvfrom() until you've got the entire file, even if the socket is not blocking, the while loop will block your program until its finished. You need to rewrite this so that the data is only recv'd on an event call
Alternatively, you can stop using eunet for FTP and instead just use this:
system("start /b ftp "&ftp_url_to_download&", 2)
11. Re: Problem with FTP
- Posted by SergioGelli Jan 12, 2010
- 1406 views
a) the FTP connection dropping continues to hang your GUI b) the FTP download is not being aborted when the FTP connection fails
The connection need not fall, to hang the GUI.
Just the connection to be slow to hang the GUI.
Eg: See what happens when a server is unavailable:
1-The connection does not fall immediately. As I wrote earlier, some of them take 900 seconds to abort.
2-How the connection can be very slow or stop the looping "while" a very long time waiting for a new cycle. So I have to find another way to make the program work with other things, including the choice to abort a file download that was locked in "while"
I think you need to do the following: a) use eunet_recvfrom() with MSG_DONTWAIT b) change the BLOCK_SIZE from 4K to a much smaller value c) chuck out your while loop. if you are simply looping using eunet_recv() or eunet_recvfrom() until you've got the entire file, even if the socket is not blocking, the while loop will block your program until its finished. You need to rewrite this so that the data is only recv'd on an event call
Alternatively, you can stop using eunet for FTP and instead just use this:
system("start /b ftp "&ftp_url_to_download&", 2)
1-I am using eunet_recvfrom() with MSG_DONTWAIT.
2-If I change the BLOCK_SIZE from 4K to a much smaller value, like 256, and que server stop, my loop stop too..
3-system("start /b ftp "&ftp_url_to_download&", 2) This solution will stop responding exactly as described above, if the server stops responding or slow down, the advantage is the you cant type ctrl-c to abort...
12. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 12, 2010
- 1441 views
3-system("start /b ftp "&ftp_url_to_download&", 2) This solution will stop responding exactly as described above, if the server stops responding or slow down, the advantage is the you cant type ctrl-c to abort...
That was not step three. It was an alternative to all three steps, and an alternative to using eunet.
Anyways, I do not understand why it has hung your GUI. start /b is *suppose* to run the command in the background - and return immediately to the calling program. So what is suppose to happen is that system() calls start, start runs ftp in the background, ftp.exe starts downloading the file while your program continues to operate (GUI fully functional and responsive).
a) the FTP connection dropping continues to hang your GUI b) the FTP download is not being aborted when the FTP connection fails
The connection need not fall, to hang the GUI.
Just the connection to be slow to hang the GUI.
Ok, so then you do have both problems as I describe below:
a) the FTP connection dropping *or being slow* continues to hang your GUI b) the FTP download is not being aborted when the FTP connection fails
I think you need to do the following: a) use eunet_recvfrom() with MSG_DONTWAIT b) change the BLOCK_SIZE from 4K to a much smaller value c) chuck out your while loop. if you are simply looping using eunet_recv() or eunet_recvfrom() until you've got the entire file, even if the socket is not blocking, the while loop will block your program until its finished. You need to rewrite this so that the data is only recv'd on an event call
1-I am using eunet_recvfrom() with MSG_DONTWAIT.
2-If I change the BLOCK_SIZE from 4K to a much smaller value, like 256, and que server stop, my loop stop too..
So it seems the problem is with step 3, then....
13. Re: Problem with FTP
- Posted by SergioGelli Jan 12, 2010
- 1422 views
system("start /b ftp "&ftp_url_to_download&", 2) The program can find out if the operation is over?
14. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 12, 2010
- 1394 views
system("start /b ftp "&ftp_url_to_download&", 2) The program can find out if the operation is over?
No, not directly. If you know the size of the file being downloaded though then you can simply do a dir() on the file to find out if it is done yet or not.
Otherwise, the following batch script provides a work-around:
@echo off rem Lets assume this batch script is named ftpbatch.bat rem Usage: ftpbatch.bat ftp://site.tld/file/to/get c:\temp\ok_download_is_done.txt ftp "%1" echo %ERRORLEVEL% > "%2"
and then you would do
system("start /b ftpbatch.bat "&ftp_url_to_download&" "&completion_file, 2)
Then you could do a dir() on completion_file, and when it is created you know the ftp download is finished. (And if you need to see the exit code of ftp.exe to know if the download was successful or aborted, you can read the exit code in the file.)
15. Re: Problem with FTP
- Posted by sergelli Jan 12, 2010
- 1411 views
Thank you, your code looks very simple.
It is as the "egg of Columbus"
I do not know why I did not have this idea before! :)
But I have the program 98% done with "eunet.e" library So, I'll do some testing with your suggestions, but I will continue looking for a way to run the functions of "eunet.e" in the background. with the ability to intervene in the progress of tasks.
Any suggestions are still welcome
16. Re: Problem with FTP
- Posted by m_sabal Jan 13, 2010
- 1382 views
In order to run any socket-based routine as a background operation, you need to have true threading. Euphoria does not currently support true threading (which with today's multi-core CPUs, isn't the best way to code anymore anyway), so the only way to do a parallel process is to spawn a separate program. Jim's suggestion is one way to do that, and is probably the simplest way of accomplishing your goals. This is one reason a full FTP implementation hasn't been included in eunet yet.
If you wanted a pure Euphoria solution, you could have your ftp routines built as a separate program, probably as a Windows service, and include either a file-based or socket-based inter-process communication API to allow your GUI to give instructions to the FTP process and query its status. Obviously that is a bit more complicated, but parallel programming usually is.
17. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 13, 2010
- 1375 views
you need to have true threading. Euphoria does not currently support true threading (which with today's multi-core CPUs, isn't the best way to code anymore anyway)
This is not true, and has not been true since the release of 4.0 beta3.
In order to run any socket-based routine as a background operation, you need to have true threading. Euphoria does not currently support true threading (which with today's multi-core CPUs, isn't the best way to code anymore anyway), so the only way to do a parallel process is to spawn a separate program. This is one reason a full FTP implementation hasn't been included in eunet yet.
If you wanted a pure Euphoria solution, you could have your ftp routines built as a separate program, probably as a Windows service, and include either a file-based or socket-based inter-process communication API to allow your GUI to give instructions to the FTP process and query its status. Obviously that is a bit more complicated, but parallel programming usually is.
I'm not sure if it is possible to build a service out of an Euphoria program. (You could always write a standard Eu program and install it through something like winserv, of course.)
18. Re: Problem with FTP
- Posted by SergioGelli Jan 14, 2010
- 1382 views
I think a small program Euphoria, using "eunet.e", for example, STOR.EXE, could be executed within the main program with "system("stor.exe parameter 1 parameter 2 parameter 3 ...", 2)" , this is like the suggestion made by JimcBrown.
This operation would be in the background, o.k.?
Meanwhile, the main program could monitor the completion of tasks and enable the User to choose an abortion, if it happens one undesired event.
But I do not know how to kill the execution of STOR.EXE, how can I do that? Does this will work?
19. Re: Problem with FTP
- Posted by m_sabal Jan 14, 2010
- 1364 views
you need to have true threading. Euphoria does not currently support true threading (which with today's multi-core CPUs, isn't the best way to code anymore anyway)
This is not true, and has not been true since the release of 4.0 beta3.
Sorry, I haven't been keeping up with the latest beta. So you've added full pthread support to the core, as opposed to the time-share algorithm we've had? If so, that's great. I still stand by my analysis, though, that threads are much less important and less useful today than they were a few years ago.
I'm not sure if it is possible to build a service out of an Euphoria program. (You could always write a standard Eu program and install it through something like winserv, of course.)
I've been doing SaaS in Euphoria for a number of years now. See this contribution.
20. Re: Problem with FTP
- Posted by jimcbrown (admin) Jan 14, 2010
- 1402 views
you need to have true threading. Euphoria does not currently support true threading (which with today's multi-core CPUs, isn't the best way to code anymore anyway)
I still stand by my analysis, though, that threads are much less important and less useful today than they were a few years ago.
Huh? What analysis? Are you saying that pre-emptive multithreading is less useful than being able to take advantage of multi-core? Or are you saying that true multiprocess parallel programs are becoming more important these days?
I've been doing SaaS in Euphoria for a number of years now. See this contribution.
Cool. I'm impressed.