1. Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 06, 2012
- 1531 views
I use Win7 and Eu3.1 with Win32Lib. I have a data logging program that saves data on networked drive; it opens, appends, and closes. That is fine. If I read that file, all is fine. But if I read it again I get the cached version and not the real data which is on disk.
Is it possible to force a disk open or disk read to access the actual data and not the cached data? I want to have a program accessing the data on a regular basis to see how the data logger is doing, but all I get is the same old data over and over. I often want to access this file from home; I use NetDrive which accesses the data by FTP; I'm sure that is getting the data but I'm not. When I'm at work I go straight to the file on our server with on the internal network but the result is the same.
Does anyone have any ideas? I could build in the FTP access to my program but that does seem like hard work. And I'm not completely sure that is right anyway.
Andy
2. Re: Clearing file cache to read what's really on disk
- Posted by BRyan Feb 06, 2012
- 1511 views
I use Win7 and Eu3.1 with Win32Lib. I have a data logging program that saves data on networked drive; it opens, appends, and closes. That is fine. If I read that file, all is fine. But if I read it again I get the cached version and not the real data which is on disk.
Is it possible to force a disk open or disk read to access the actual data and not the cached data? I want to have a program accessing the data on a regular basis to see how the data logger is doing, but all I get is the same old data over and over. I often want to access this file from home; I use NetDrive which accesses the data by FTP; I'm sure that is getting the data but I'm not. When I'm at work I go straight to the file on our server with on the internal network but the result is the same.
Does anyone have any ideas? I could build in the FTP access to my program but that does seem like hard work. And I'm not completely sure that is right anyway.
Andy
Andy: What about if you do a rewind.
seek(fn, 0)
Bernie
3. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 06, 2012
- 1488 views
Andy: What about if you do a rewind.
seek(fn, 0)
Bernie
No. I use a seek anayway to the last read point I accessed. I tried a rewind but nothing changed. Nice try, though, Bernie.
4. Re: Clearing file cache to read what's really on disk
- Posted by jimcbrown (admin) Feb 06, 2012
- 1574 views
I use Win7 and Eu3.1 with Win32Lib. I have a data logging program that saves data on networked drive; it opens, appends, and closes. That is fine. If I read that file, all is fine. But if I read it again I get the cached version and not the real data which is on disk.
I'm curious, does this only happen with NetDrive network drive? Do you have the same problem when using regular mapped network drives to a share on another computer?
Is it possible to force a disk open or disk read to access the actual data and not the cached data? I want to have a program accessing the data on a regular basis to see how the data logger is doing, but all I get is the same old data over and over. I often want to access this file from home; I use NetDrive which accesses the data by FTP; I'm sure that is getting the data but I'm not. When I'm at work I go straight to the file on our server with on the internal network but the result is the same.
Actually, it's probably not going there. At least not right away. A few articles on NetDrive imply that you'll need to manually flush the cache yourself to ensure that all the data is really transferred over.
http://www.freewaregenius.com/2007/09/18/netdrive/
http://www.muhlenberg.edu/main/aboutus/oit/faculty_computing/remote_access/net_drive.html
Of course, they explain how to flush the cache, but only through the shell extension. I don't know how to do it programatically (aside from the obviously problematic method of programmically taking control of the mouse and keyboard to mimic a user performing the actions to trigger a manual flush).
Does anyone have any ideas? I could build in the FTP access to my program but that does seem like hard work. And I'm not completely sure that is right anyway.
Andy
Building in FTP access directly is probably the right way to do it. You can go full out making a new FTP library (and even submitting it to the dev team to get it included in the standard library) or probably go the easier route of reading and writing to temp files and then manipulating ftp.exe (using pipeio:exec() and friends) to do the heavy lifting for you.
I can think of other ways to do achieve this - set up a Linux/GNU or FreeBSD or other *nix system with FUSE supported, install an FTP-FS like curlftpfs, mount the ftp server using the FTP-FS on the *nix system, share it as a regular CIFS share via Samba, then map the Samba share on your Windoze 7 system as a regular mapped drive. For example.
Yeah... building in FTP access is probably the right way to do it. Just saying.
5. Re: Clearing file cache to read what's really on disk
- Posted by DerekParnell (admin) Feb 06, 2012
- 1460 views
Is it possible to force a disk open or disk read to access the actual data and not the cached data? ... I use NetDrive which accesses the data by FTP ...
This sounds like an issue with NetDrive and not Windows or Euphoria.
6. Re: Clearing file cache to read what's really on disk
- Posted by jimcbrown (admin) Feb 06, 2012
- 1491 views
Is it possible to force a disk open or disk read to access the actual data and not the cached data? ... I use NetDrive which accesses the data by FTP ...
This sounds like an issue with NetDrive and not Windows or Euphoria.
My first impression of their support forums suggest that they're particularly competent either. http://www.netdrive.net/phpBB3/viewtopic.php?f=5&t=650
Good luck.
7. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 06, 2012
- 1535 views
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network. I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
I'll scan the two links above but I am pretty sure it's a Windows feature; they cache the file data when it has been read, and just give it back next time you ask for it. Same stuff as before. I should be able to flush the read cache, IMHO.
A
8. Re: Clearing file cache to read what's really on disk
- Posted by jimcbrown (admin) Feb 06, 2012
- 1535 views
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network.
Right, because your program has written it via NetDrive, but NetDrive hasn't flushed it out from cache to FTP yet, so when you look at it directly you see the older version of the file.
Likewise, if you are writing the file locally but then looking at it from a different computer through NetDrive, it's possible that NetDrive's caching prevents you from seeing the latest copy on that other computer.
I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
Well, for test purposes it would be nice to try to have the logger write to a normal file on C: drive, and see if you can see the updates right away.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
A test that completely removes NetDrive from the equation is necessary to demonstrate that this is in fact not a NetDrive problem at all.
I'll scan the two links above but I am pretty sure it's a Windows feature; they cache the file data when it has been read, and just give it back next time you ask for it. Same stuff as before. I should be able to flush the read cache, IMHO.
A
Well, do you have any links documenting this Windoze read cache feature?
9. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 06, 2012
- 1536 views
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network.
Right, because your program has written it via NetDrive, but NetDrive hasn't flushed it out from cache to FTP yet, so when you look at it directly you see the older version of the file.
No, I'm writing directly to file via the internal network; the data is being written perfectly. I'm reading via NetDrive so I can do it from home. I can read it fine if I turn off my laptop and power it up again - the first read reads all the data written, and that never changes till I power off & on again.
I can read it through the internal network with notepad - and same thing. The version I read the first time is the only one I read until I power off & on again. [quotejcb]
I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
Well, for test purposes it would be nice to try to have the logger write to a normal file on C: drive, and see if you can see the updates right away.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
A test that completely removes NetDrive from the equation is necessary to demonstrate that this is in fact not a NetDrive problem at all.
I'll scan the two links above but I am pretty sure it's a Windows feature; they cache the file data when it has been read, and just give it back next time you ask for it. Same stuff as before. I should be able to flush the read cache, IMHO.
A
Well, do you have any links documenting this Windoze read cache feature?
No. I will do some more tests since it is clear that there is no simple answer to my initial query. I will write a program to just save ever-increasing text to a file like the logger does, and try the same test on the same machine. And so on. And I will let you know what the result is.
flush() flushes out a write cache; there is no equivalent to clear a read cache, I guess?
A
10. Re: Clearing file cache to read what's really on disk
- Posted by mattlewis (admin) Feb 06, 2012
- 1513 views
I found a post in the NetDrive forum that says that there's no way to flush the cache without disconnecting and reconnecting. The post was made in November 2010. No versions are mentioned, so who knows?
Question and reply:
I have two systems that read and write to the same file, anywhere from 5 minutes apart to 45 minutes apart. Somehow, when one updates the file, the other still has the old file in the cache and doesn't see the changes.
I close the file and reload it each time.
Any ideas?
Unfortunately you need to reload or reconnect in order to empty cache at this time.
We'll add flushing cache function in the near future.
Matt
11. Re: Clearing file cache to read what's really on disk
- Posted by jimcbrown (admin) Feb 06, 2012
- 1669 views
I know I am asking a lot of questions, but this problem is really complicated, and a lot of detailed information is required to accurately diagnose this issue.
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network.
Right, because your program has written it via NetDrive, but NetDrive hasn't flushed it out from cache to FTP yet, so when you look at it directly you see the older version of the file.
No, I'm writing directly to file via the internal network; the data is being written perfectly.
How are you writing it? If it's via FTP you'd still need to use NetDrive or a similiar utility to map it to a drive, so I'm guessing that you can just map the directory as a CIFS share (windows mapped drives) when you are on the internal network. And doing so allows you to write to it without using NetDrive at all - and then check that it's been written to by using NotePad on the server where the file is stored itself (again without using NetDrive at all). That's the only thing that makes sense, so I'm going to go with that assumption.
If you are directly using NetDrive on the internal network to do either reading or writing, then NetDrive hasn't been eliminated as a suspect.
I'm reading via NetDrive so I can do it from home. I can read it fine if I turn off my laptop and power it up again - the first read reads all the data written, and that never changes till I power off & on again.
I can read it through the internal network with notepad - and same thing. The version I read the first time is the only one I read until I power off & on again.
Ok, just to be perfectly clear - you write to it either using the internal network, by mapping the drive to a share (e.g. CIFS or SMB) or directly on the server (so it's a local write) but either way without using NetDrive at all, and then you read it (on a different computer) through the internal network with notepad, and you get the old version?
I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
Well, for test purposes it would be nice to try to have the logger write to a normal file on C: drive, and see if you can see the updates right away.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
If that doesn't work, you probably need to call flush() more often.
Beyond that, you could test the logger with a regular mapped drive.
A test that completely removes NetDrive from the equation is necessary to demonstrate that this is in fact not a NetDrive problem at all.
Well, do you have any links documenting this Windoze read cache feature?
No. I will do some more tests since it is clear that there is no simple answer to my initial query. I will write a program to just save ever-increasing text to a file like the logger does, and try the same test on the same machine. And so on. And I will let you know what the result is.
flush() flushes out a write cache; there is no equivalent to clear a read cache, I guess?
A
The tests are a good idea, they'll help narrow this issue down. Still, even if it is a Windows issue, we might not be able to help too much. At least, it sounds like it's not directly caused by an issue or problem in Euphoria itself.
A read cache is not really typical. The write cache points to a memory buffer that's used to speed up disk I/O (you can "write" data to a file and the go do toher stuff while the data is committed to disk in the background) but the with network filesystems like SMBFS, CIFS, NFS, etc there's another layer of caching (in this case it's suppose to be transparent, hence no need for the application to call to do a read cache flush).
It sounds like NetDrive's stuff is either not meant to be transparent (you're suppose to explicitly flush the FTP/webDAV cache) or if it is suppose to be transparent then it is implemented in a really poor way.
12. Re: Clearing file cache to read what's really on disk
- Posted by useless_ Feb 07, 2012
- 1487 views
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network. I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
I'll scan the two links above but I am pretty sure it's a Windows feature; they cache the file data when it has been read, and just give it back next time you ask for it. Same stuff as before. I should be able to flush the read cache, IMHO.
A
I have used logging files on winxpee to keep code sync'd across 8 computers on a dhcp lan, and other than the usual idiocy of winders, i have had no problem getting up-to-date data from the logging files, no matter which computer wrote to which file on any other computer, and at speeds up to 10 hits per second. I have more of a problem with winders suddenly and completely ignoring another computer on the lan, which isn't an Euphoria problem.
Reading more posts on this, i suggest you kill NetDrive and see if you can read the file's new data. I think it's an ownership issue in winders, and until NetDrive explicitly tells winders that it no longer owns the file, winders won't touch it with a ten foot pole to get updates to the file. Killing NetDrive won't be guaranteed to tell windows to allow other access, but it might.
useless
13. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 07, 2012
- 1481 views
Sorry, did I say it only happened with NetDrive? It also happens when I am at my desk looking at the file on the internal network. I can't make the logger write directly to a file actually located on my computer - it's not at a constant IP address.
I'll scan the two links above but I am pretty sure it's a Windows feature; they cache the file data when it has been read, and just give it back next time you ask for it. Same stuff as before. I should be able to flush the read cache, IMHO.
A
I have used logging files on winxpee to keep code sync'd across 8 computers on a dhcp lan, and other than the usual idiocy of winders, i have had no problem getting up-to-date data from the logging files, no matter which computer wrote to which file on any other computer, and at speeds up to 10 hits per second. I have more of a problem with winders suddenly and completely ignoring another computer on the lan, which isn't an Euphoria problem.
Reading more posts on this, i suggest you kill NetDrive and see if you can read the file's new data. I think it's an ownership issue in winders, and until NetDrive explicitly tells winders that it no longer owns the file, winders won't touch it with a ten foot pole to get updates to the file. Killing NetDrive won't be guaranteed to tell windows to allow other access, but it might.
useless
OK, now that IS interesting. I killed NetDrive and the problem seems to go away. That is, my log file reader keeps up with new logged data. I am now at the office and using the direct mapped drive. When I had NetDrive running, which I do to make access to our server easy at home, the thing didn't keep up. You might be right that if dear old Windoze is being held off by NetDrive then I would get the total update failure I was seeing. Hmmm. I think your username maybe rather inappropriate, Sir.
More tests called for, but two steps forward and one back is progress ...
A
14. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 07, 2012
- 1442 views
I killed NetDrive, and ran the viewer looking at a conventional mapped drive file and it ran cheerfully. Then I started NetDrive again and the viewer carried on working. But if I pointed it to the NetDrive mapped file, it did NOT update as the data changed. I suspect that doing it this way Windoze has first call on the server data so it carries on working. But if I can't get current data from NetDrive then it is not what I want anyway.
I think it would be best if I found an alternative to NetDrive to map my server via FTP to a local drive letter. In the meantime I am at least a lot further on than I was. Thanks for all your help, guys, and unusual apologies to Microsoft....
Andy
15. Re: Clearing file cache to read what's really on disk
- Posted by AndyDrummond Feb 07, 2012
- 1442 views
Sorry, Gentlemen, I found the fault. NetDrive was accessing the drive via WebDAV and not FTP. Changing it to FTP made it work. I never knew I could access the server with WebDAV, nor did I know that it would result in its inability to see new data. But I guess WebDAV does have that kind of "I'm using it, keep your fingers off it!" design. So - darn it - all my fault after all.
Andy
