OpenEuphoria

2. Re: Adding/Removing byes from the beginning of a file

• Posted by BRyan Feb 21, 2015
• 1834 views

How about reading the whole file into a sequence.

Chop the front off the sequence and write the sequence back into the file.

4. Re: Adding/Removing byes from the beginning of a file

• Posted by ryanj Feb 21, 2015
• 1830 views

I haven't tried this, but you could probably use file_num = open("my_file", "u") to open text file for update (reading and writing), then copy data within the file a section at a time to limit memory useage, using seek, getc, putc, or other functions in http://openeuphoria.org/docs/std_io.html. I'm not sure how to reduce the size of the file, but i believe it grows if you write past the end of the file.

6. Re: Adding/Removing byes from the beginning of a file

• Posted by GreenEuphorian Feb 21, 2015
• 1820 views

I am venturing here something that might be impractical or even impossible: what about gaining low-level access to the file allocation table and modifying it in such a way that the beginning point of a file is moved a few bytes after its current position? This way, no re-writing of the file would be needed. Does this make sense?

8. Re: Adding/Removing byes from the beginning of a file

• Posted by ryanj Feb 21, 2015
• 1813 views

I think large binary files are typically managed by creating a sort of file system inside the file. Part of the file has an allocation table or bitmap that maps out the data inside the file. Parts of the file would contain valid data and parts would be "empty". To delete data from anywhere in the file, mark that area as "empty". A cleanup or compact function would be called periodically to rebuild the file by defragmenting valid areas and removing excess empty areas, probably by copying all the data to a fresh new file and building a new allocation table.

10. Re: Adding/Removing byes from the beginning of a file

• Posted by petelomax Feb 21, 2015
• 1852 views

I would have said much the same, but in blocks of anywhere between 8 and 256K. Actually, Eu hides much of this from you, so getc/putc is not as bad as it could be, and in fact quite reasonable performance-wise.

An equivalent imaginary "SetStartOfFile" function (from MSDN) would only work (without moving 100s GB) when the part you wanted to remove happened to be a whole file sector, or similar, so basically, no.

The suggestion I would make is rewrite the first few bytes with a special sequence meaning "ignore N bytes at start of file". Obviously that depends on who/what has to read it and whether you can modify that to obey said special sequence, however I would hazard that is a non-starter as if it were that simple you'd just modify the exact same things to skip whatever it is you want them to skip. Sorry.

Pete

12. Re: Adding/Removing byes from the beginning of a file

• Posted by DerekParnell (admin) Feb 21, 2015
• 1770 views

You have to create a new file based on copying the original file, adding bytes or not copying bytes as required.

Sorry but there is no easy way out of this. Doing low-level filesystem manipulation is bound to be messy and dangerous; definitely not worth the effort.

If running on Windows, you could open the file in update mode, copying bytes from later locations to earlier locations (means remembering and setting current file position) and then when you have finished that, calling the API routine SetEndOfFile() in the kernel.dll.

14. Re: Adding/Removing byes from the beginning of a file

• Posted by petelomax Feb 21, 2015
• 1783 views

Erm, fairly straightforward I should think:

fn = open(filename,"ub") 
if seek(fn,0)!=SEEK_OK then ?9/0 end if -- (not strictly neccessary; and SEEK_OK is automatically defined as 0 in Phix) 
puts(fn,some_bytes) 
close(fn) 

Obviously, caveat emptor, untested, expect problems if you clobber more bytes than you should, as Derek said it may make the file unusable by other apps, etc.

Pete

PS: Another (random) thought occurs to me that if you don't have enough bytes (ignore this if you do), and you really don't want to move 100s of GB, a "get first 7 bytes from file 5" type scheme might help.
PPS: It also might be sensible to blat the first few bytes, do your thing, then restore those overwritten bytes.

16. Re: Adding/Removing byes from the beginning of a file

• Posted by Ekhnat0n Feb 22, 2015
• 1739 views

Simply declare a struct called New_File consisting of 2 structures eg
File_Data and New_Data.
Then write your new data to the struct New_Data and the original file to File_Data.

Now write the structure New_File to disk and that's all.

Not that hard to come up with and really a piece of cake.

18. Re: Adding/Removing byes from the beginning of a file

• Posted by GreenEuphorian Feb 22, 2015
• 1710 views

What about this?

20. Re: Adding/Removing bytes from the beginning of a file

• Posted by GreenEuphorian Feb 22, 2015
• 1702 views

Well, the initial idea when I started this thread was prepending bytes and removing bytes from a file, but from the answers I got I realised that this is not convenient at all, and may also lead to troubles.

So my current task is: overwriting the first few bytes of a file (containing the file signature, a.k.a. magic number, or simply header) and then, later on, overwriting them again, back to their original state. I already got the info about doing this, thanks to Pete.

Now, the only question unanswered is: will this simple byte-overwriting process (without any truncation or prepending) entail also a complete re-writing of the whole file on the disk when the file is updated? Or will those few bytes alone be modified on the disk (which indeed is my desired outcome)? There are significant performance issues involved, especially in the case of huge files. That's why I am asking.

Thanks again to all.

22. Re: Adding/Removing bytes from the beginning of a file

• Posted by GreenEuphorian Feb 22, 2015
• 1660 views

putc() ?!? I could not find it in the manual. Or do you mean puts()?

24. Re: Adding/Removing bytes from the beginning of a file

• Posted by SDPringle Feb 25, 2015
• 1594 views

True, they could have been called put2 and put4 respectively. It is hard to get people to change the names for things though, even before release I lobbied to name the regex functions match rather than find, but nobody would go along with that. Write your own library to wrap the function names that are too long.

26. Re: Adding/Removing bytes from the beginning of a file

• Posted by ghaberek (admin) Feb 25, 2015
• 1588 views

It's interesting that you bring this up because I only recently discovered that there even was a "find" function in the regex library. I always use has_match() or is_match() and then matches() or all_matches() to process my regex queries. I'm generally more concerned with using patterns to extract one bit of text from a larger string, for which matches() works much better. YMMV, of course.

regex re_pattern = regex:new( `(.*?)` ) 
 
if regex:has_match( re_pattern, my_text ) then 
 
    sequence matches = regex:matches( re_pattern, my_text ) 
    -- play with matches ;) 
 
end if 

-Greg