OpenEuphoria

2. Re: How to get list of output of find command of Linux

• Posted by _tom (admin) Jul 14, 2017
• 2118 views

• try walkdir; this is recursive

include std/filesys.e  
 
walk_dir(sequence path_name, object your_function, 
integer scan_subdirs = types :FALSE, 
object dir_source = types :NO_ROUTINE_ID) 

• try writing to a file

system( find --help > foo.txt ) 
 
--> load "foo.txt" 
--> parse and use the output 

_tom

4. Re: How to get list of output of find command of Linux

• Posted by ghaberek (admin) Jul 17, 2017
• 2003 views

I would not recommend using system() to make an external call to find. That method is not portable and should be considered bad practice.

Here is the method I for recursively finding files in a directory without using walk_dir() and without making recursive calls to the same function.

Essentially, we just keep a running queue of the directories we'd like to scan while adding the files found to a list for output.

include std/filesys.e 
include std/wildcard.e 
 
public function get_files( sequence path = current_dir(), sequence name = "*", integer maxdepth = 0 ) 
-- 
-- path     : directory path to scan 
-- name     : wildcard pattern to look for 
-- maxdepth : maximum folder depth to scan within path 
-- 
    integer depth 
    sequence files, queue, item, full_path 
    object items 
 
    depth = 1 
    files = {} 
    path = canonical_path( path ) 
 
    -- push initial path into queue 
    queue = { {path,depth} } 
 
    while length( queue ) do 
 
        -- pop an item off the queue 
        item = queue[1] 
        queue = queue[2..$] 
 
        -- get the queue item values 
        path = item[1] 
        depth = item[2] 
 
        -- get the directory items 
        items = dir( path ) 
 
        if atom( items ) then 
            -- error 
            continue 
        end if 
 
        for i = 1 to length( items ) do 
            item = items[i] 
 
            if find( item[D_NAME], {".",".."} ) then 
                -- skip these entries 
                continue 
            end if 
 
            full_path = join_path({ path, item[D_NAME] }) 
 
            if find( 'd', item[D_ATTRIBUTES] ) then 
                -- is a directory 
 
                if maxdepth = 0 or depth < maxdepth then 
                    -- queue this directory 
                    queue = append( queue, {full_path,depth+1} ) 
                end if 
 
            else 
                -- is a file 
 
                if wildcard:is_match( name, item[D_NAME] ) then 
                    -- add this to the output 
                    files = append( files, full_path ) 
                end if 
 
            end if 
 
        end for 
 
    end while 
 
    return files 
end function 

-Greg

6. Re: How to get list of output of find command of Linux

• Posted by ghaberek (admin) Jul 18, 2017
• 1973 views

I'm not against recursion per se, but it's not really a good approach for scanning through a file system. With any programming language, but especially interpreted ones like Euphoria, deeply recursive functions create additional overhead, and file systems tend to be deeply recursive, so there's a potential for lots of overhead.

Alternatively, if it applies to your project, what I'd recommend is using a queued approach like I've shown, but combine that with a visitor function like walk_dir() uses, so that you don't collect all the file names you've found in memory. I can provide an example on how to do that if you need.

In the end, the "best" method for this is probably an iterator that holds its position in the queue and only returns the values as they're requested. Other languages have this feature, like Python, and it's a pretty neat, albeit complicated, way to run through a list of items very efficiently.

-Greg

8. Re: How to get list of output of find command of Linux

• Posted by petelomax Jul 18, 2017
• 1966 views

Maybe something like this? http://rosettacode.org/wiki/Generator/Exponential#Phix (but with only one task)

Pete

10. Re: How to get list of output of find command of Linux

• Posted by petelomax Jul 18, 2017
• 1943 views

It's pretty simple

sequence files = {} 
function look_at(sequence path, sequence d) 
    files = append(files,{join_path({path,d[D_NAME]}),d[D_SIZE]}) 
    return 0 -- keep going 
end function 
if walk_dir("C:\\Downloads\\Software", routine_id("look_at"), TRUE)!=0 then 
    ?"oops" 
end if 
for i=1 to length(files) do 
    printf(1, "%s: %d\n",files[i]) 
end for 

Output:

<snip> 
C:\Downloads\Software\SKU322505 inpa\inpa\InpaCANinstall.doc: 4561920 
C:\Downloads\Software\SKU322505 inpa\inpa\InpaCANinstall.pdf: 300349 
C:\Downloads\Software\SKU322505 inpa\inpa\autorun.inf: 60 
C:\Downloads\Software\SKU322505 inpa\inpa\cemtl.ico: 4286 
C:\Downloads\Software\tdm-gcc-5.1.0-3.exe: 36802006 
C:\Downloads\Software\tdm64-gcc-5.1.0-2.exe: 48071122 

For a fuller example, see http://openeuphoria.org/docs/std_filesys.html#_1282_walk_dir

Pete

PS: you'll probably also need include std/filesys.e at the top

12. Re: How to get list of output of find command of Linux

• Posted by ghaberek (admin) Jul 24, 2017
• 1922 views
• Last edited Jul 25, 2017

I had some time to sit down and knock out an example of what I was describing. Basically, my first version above was an eager scanner (it gets all of the files at once), this is a lazy scanner (it only gets more files when it needs to).

To use it, simple call dir_open() to start scanning the directory, then loop until dir_next() returns an empty sequence, and call dir_close() to clean up when you're done.

This is the most efficient method in terms of both speed -- you don't have to wait for it to find all the files before starting the loop, and memory -- it only keeps a subset of directories and file paths in memory at any given time.

Edit: actually, this is probably only faster as a matter of perception since you can get started on the body of the loop sooner; so it feels faster. Just running the numbers quickly on my machine, the eager method (see above) may be about 20-30% faster, at least it is on my system. So if memory is of no concern, then scanning the entire directory first is probably faster, but real-world cases will vary wildly. My tests are doing nothing but printing the path to the console. YMMV

Also, the find command is like, a bazillion times faster than any of this. Not sure what it's doing but there's got to be some lower-level operating system or file system calls going on.

-- dir_scan.e 
include std/filesys.e 
include std/wildcard.e 
 
sequence d = {} 
 
enum PATTERN,DIRS,FILES 
 
-- 
-- prepare a new directory scanner instance 
-- 
public function dir_open( sequence path = ".", sequence pattern = "*" ) 
 
    if path[$] != SLASH then path &= SLASH end if 
    path = canonical_path( path ) 
 
    integer id = find( {}, d ) 
 
    if id = 0 then 
        d = append( d, {} ) 
        id = length( d ) 
    end if 
 
    d[id] = {pattern,{path},{}} -- PATTERN,DIRS,FILES 
 
    return id 
end function 
 
-- 
-- clean up an unused directory scanner 
-- 
public procedure dir_close( integer id ) 
    d[id] = {} 
end procedure 
 
-- 
-- get the next available file in the queue 
-- (returns "" when queues are exhausted) 
-- 
public function dir_next( integer id ) 
 
    sequence pattern = d[id][PATTERN] 
 
    if length( d[id][FILES] ) != 0 then 
        -- we have files, return the next one 
 
        -- pop a file off the queue 
        sequence next_file = d[id][FILES][1] 
        d[id][FILES] = d[id][FILES][2..$] 
 
        return next_file 
 
    end if 
 
    while length( d[id][DIRS] ) != 0 do 
        -- we need to find more files 
 
        -- pop a directory off the queue 
        sequence next_dir = d[id][DIRS][1] 
        d[id][DIRS] = d[id][DIRS][2..$] 
 
        -- get the directory items 
        object items = dir( next_dir ) 
        if atom( items ) then 
            continue 
        end if 
 
        for i = 1 to length( items ) do 
 
            -- get the item details 
            sequence item_name = items[i][D_NAME] 
            sequence item_attr = items[i][D_ATTRIBUTES] 
            sequence full_path = next_dir & item_name 
 
            if find( item_name, {".",".."} ) then 
                -- skip these items 
                continue 
 
            elsif find( 'd', item_attr ) then 
                -- add this to the directory queue 
                d[id][DIRS] = append( d[id][DIRS], full_path & SLASH ) 
 
            elsif wildcard:is_match( pattern, item_name ) then 
                -- add this to the files queue 
                d[id][FILES] = append( d[id][FILES], full_path ) 
 
            end if 
 
        end for 
 
        if length( d[id][FILES] ) != 0 then 
            -- we have files now, return the next one 
 
            -- pop a file off the queue 
            sequence next_file = d[id][FILES][1] 
            d[id][FILES] = d[id][FILES][2..$] 
 
            return next_file 
        end if 
 
    end while 
 
    return "" 
end function 

-- find.ex 
include dir_scan.e 
 
procedure main() 
 
    integer id = dir_open( "/home/greg", "*.e" ) 
 
    sequence path 
 
    while length( path ) with entry do 
        printf( 1, "%s\n", {path} ) 
    entry 
        path = dir_next( id ) 
    end while 
 
    dir_close( id ) 
 
end procedure 
 
main() 

-Greg

14. Re: How to get list of output of find command of Linux

• Posted by ghaberek (admin) Jul 25, 2017
• 1902 views

I dug through the code for find and discovered that it's using a Linux API called fts which, ironically, uses a similar method to the one I've implemented above. So I guess I was on the right track!

I can work on writing a wrapper for this in Euphoria but that presents a challenge since the API relies heavily on a structure and structure support in Euphoria is all over the place right now.

Edit: my searching also found this conversation, which I forgot about from last year: dir() on linux via c_func, anyone?

-Greg