OpenEuphoria

Hi All,

I am trying to speed up how I scan a audio folder and add it to a Playlist.
It works fast if there is only a few hundred files, but if there is 500
or more files it can get to be slow. I scan the folder using walk_dir
and add each file into the database. Do any of you guys think I can do
this without using walk_dir? I have pasted a bit of the code below.

function ScanDir(sequence path_name, sequence ItemEntry)
sequence addaudio,streamtime,filecount,filetitle
atom len,tempstream

streamtime={}
addaudio={}
filecount={}
doEvents(0)


if equal(ItemEntry[D_ATTRIBUTES], "d") then  ----If directory do nothing

  else
if match("aac", get_file_ext(lower(ItemEntry[D_NAME]))) or   ---- If
         supported file add to the Playlist
         match("mp4", get_file_ext(lower(ItemEntry[D_NAME]))) or
         match("mp3", get_file_ext(lower(ItemEntry[D_NAME]))) or
         match("mp2", get_file_ext(lower(ItemEntry[D_NAME]))) or
         match("mp1", get_file_ext(lower(ItemEntry[D_NAME]))) or
         match("ogg", get_file_ext(lower(ItemEntry[D_NAME]))) or
         match("wav", get_file_ext(lower(ItemEntry[D_NAME])))
  then

tempstream = BASS_StreamCreateFile(0, path_name & "\\" & ItemEntry[D_NAME],
    0, 0, BASS_STREAM_AUTOFREE) ---Create a temp file to get the file length
len = BASS_ChannelBytes2Seconds(tempstream, 
      BASS_ChannelGetLength(tempstream))
streamtime = sprintf("               %d:%02d", {len / 60, remainder(len,
       60)})
        filetitle = get_file_title(ItemEntry[D_NAME])

if mdb_insert(appdir&"\\Playlist.dat","Playlist",convertCase(replace
         (filetitle,"_"," ")),{path_name & "\\" & ItemEntry[D_NAME],streamtime}) = -1
            then
               else
addaudio &= addLVItem(Playlist,listicon2,{" 
                "&convertCase(replace (filetitle,"_"," "))," "&streamtime})
                end if
               setIndex(Playlist, 1)
              getCount()
            end if
          end if
        doEvents(0)
     return 0
end function

VOID = walk_dir(audiofolder, routine_id("ScanDir"), 1)

2. Re: Could this be faster?

• Posted by cklester <cklester at yahoo.com> Dec 13, 2005
• 557 views
• Last edited Dec 14, 2005

C Bouzy wrote:
> 
> if equal(ItemEntry[D_ATTRIBUTES], "d") then  ----If directory do nothing
> 
>   else
>          if match("aac", get_file_ext(lower(ItemEntry[D_NAME]))) or   ---- If
>          supported file add to the Playlist
>          match("mp4", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp3", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp2", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp1", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("ogg", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("wav", get_file_ext(lower(ItemEntry[D_NAME])))
>   then

It might be faster to do this:

ext_list = { "aac", "mp4", "mp3", "mp2", "mp1", "ogg", "wav" }

if find(get_file_ext(lower(ItemEntry[D_NAME])),ext_list) <> 0 then...

That way you only have one call to lower(), one subscripting of D_NAME, one
call to get_file_ext(), etc., though I can't imagine it would be tons faster.
Let me know! :)

-=ck
"Programming in a state of Euphoria."
http://www.cklester.com/euphoria/

cklester wrote:

> It might be faster to do this:
> 
> ext_list = { "aac", "mp4", "mp3", "mp2", "mp1", "ogg", "wav" }
> 

Earlier I ran a test by removing the file checking and the temp
file for getting the length of each file, and it was still the same speed.
It seems the slowdown is with walk_dir. I need an alternative to using
walk_dir.

----If you continue to do what you have always done,
you will get what you have always gotten.----

4. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 13, 2005
• 578 views
• Last edited Dec 14, 2005

cklester wrote:
> 
> if find(get_file_ext(lower(ItemEntry[D_NAME])),ext_list) <> 0 then...

I almost forgot..you do not need <> 0
if find(get_file_ext(lower(ItemEntry[D_NAME])),ext_list)then

should work just fine

----If you continue to do what you have always done,
you will get what you have always gotten.----

C Bouzy wrote:
> 
> Hi All,
> 
> I am trying to speed up how I scan a audio folder and add it to a Playlist.
> It works fast if there is only a few hundred files, but if there is 500
> or more files it can get to be slow. I scan the folder using walk_dir
> and add each file into the database. Do any of you guys think I can do
> this without using walk_dir? I have pasted a bit of the code below.

I wrote an MP3 playing system for the company I work for (so I can't release 
it--in it's current form :) and I found using Greg's win_walkdir to be much
faster than walk_dir.  But it still took about 10 minutes to create the
database from scratch (for about 9000 tracks).  After that though, it 
periodically scans the folders for changes which takes all of 1.6s.  I think
the built-in walk_dir took about 30-40s on my test system and I never tried
it on the production system (as it would probably take about an hour?).

I can post more on this if you're interested but have to go to work now 

Gary

6. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 13, 2005
• 556 views
• Last edited Dec 14, 2005

ags wrote:
> I found using Greg's win_walkdir to be much faster than walk_dir.

I tried using Greg's win_walkdir and I could not get it to work at all.
It just returned the directory name and thats it.

 ----If you continue to do what you have always done,
you will get what you have always gotten.----

If you do this:

system(dir \\*.aac > MDIR.txt,2)
system(dir \\*.mp4 >> MDIR.txt,2)
system(dir \\*.mp3 >> MDIR.txt,2)
system(dir \\*.mp2 >> MDIR.txt,2)
system(dir \\*.mp1 >> MDIR.txt,2)
system(dir \\*.ogg >> MDIR.txt,2)
system(dir \\*.wav >> MDIR.txt,2)

Then you would have a list of your files
on your system.

By using some certain options after dir command
you can control some parameters ( do a dir /? for options )

Also you might consider piping the output
through a custom filter file.

I think this would make thing faster.   

Bernie

My files in archive:
w32engin.ew mixedlib.e eu_engin.e win32eru.exw

Can be downloaded here:
http://www.rapideuphoria.com/cgi-bin/asearch.exu?dos=on&win=on&lnx=on&gen=on&keywords=bernie+ryan

8. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 14, 2005
• 556 views

Bernie Ryan wrote:
> 
> 
> If you do this:
> }}}
<eucode>
> system(dir \\*.aac > MDIR.txt,2)
> system(dir \\*.mp4 >> MDIR.txt,2)
> system(dir \\*.mp3 >> MDIR.txt,2)
> system(dir \\*.mp2 >> MDIR.txt,2)
> system(dir \\*.mp1 >> MDIR.txt,2)
> system(dir \\*.ogg >> MDIR.txt,2)
> system(dir \\*.wav >> MDIR.txt,2)
> </eucode>
{{{



Bernie,

Thanks for the suggestion, but that will not work for what I am doing.
That would only work for the parent directory. If there is more than
one directory in the parent directory, it will not scan the child directories.


----If you continue to do what you have always done,
you will get what you have always gotten.----

C Bouzy wrote:
> 
> Bernie Ryan wrote:
> > 
> > 
> > If you do this:
> > }}}
<eucode>
> > system(dir \\*.aac > MDIR.txt,2)
> > system(dir \\*.mp4 >> MDIR.txt,2)
> > system(dir \\*.mp3 >> MDIR.txt,2)
> > system(dir \\*.mp2 >> MDIR.txt,2)
> > system(dir \\*.mp1 >> MDIR.txt,2)
> > system(dir \\*.ogg >> MDIR.txt,2)
> > system(dir \\*.wav >> MDIR.txt,2)
> > </eucode>
{{{

> 
> 
> Bernie,
> 
> Thanks for the suggestion, but that will not work for what I am doing.
> That would only work for the parent directory. If there is more than
> one directory in the parent directory, it will not scan the child directories.
> 
> 
> ----If you continue to do what you have always done,
> you will get what you have always gotten.----

  TYPE DIR /? on your system and you will see AN OPTION called /S
  which will do sub-directory.

  Remember if you use this technique and do it right
  you can get all the desired files on your system with their sizes
  and paths all in one swoop.
   

Bernie

My files in archive:
w32engin.ew mixedlib.e eu_engin.e win32eru.exw

Can be downloaded here:
http://www.rapideuphoria.com/cgi-bin/asearch.exu?dos=on&win=on&lnx=on&gen=on&keywords=bernie+ryan

10. Re: Could this be faster?

• Posted by ags <eu at 531pi.co.nz> Dec 14, 2005
• 591 views

C Bouzy wrote:
> 
> ags wrote:
> > I found using Greg's win_walkdir to be much faster than walk_dir.
> 
> I tried using Greg's win_walkdir and I could not get it to work at all.
> It just returned the directory name and thats it.
> 
>  ----If you continue to do what you have always done,
> you will get what you have always gotten.----

I had to modify it I think for the way I was using it.  Here's the code 
(thanks again Greg :) followed by a short context of how I use it:

Oh and if it finds a new file, it has to open it (using BASS) and get the
length, which makes the initial database creation very slow, hence the
10 mins mentioned in my previous post.

--	win_dir() function
--	by Greg Haberek <ghaberek at gmail.com>
--
--      walk_win_dir()
--      modified RDS Euphoria walk_dir in file.e to deal with trailing slash
quirks
--      Windows only function, naturally.
--      by Gary Shingles <eu at 531pi.co.nz>
--
--	Works just like dir() but specifically for Windows. Uses Unicode and supports
--	any and all available characters, and up to 32,767 character path names.
--
--	The idea for this goes to Euman who devised a similar function in his
--	Euphoria Free File Manager (EFFM), in the Archive.
--
--	Note:
--	  I know my structure access is a little crude, but I didn't
--	  want this library to be dependant on anything like Win32Lib.

include dll.e
include get.e
include file.e
include machine.e
without warning

global constant
	kernel32_dll			= open_dll( "kernel32.dll" ),
	xlstrlenW				= define_c_func( kernel32_dll, "lstrlenW", {C_POINTER}, C_LONG ),
xFindFirstFile			= define_c_func( kernel32_dll, "FindFirstFileW", {C_POINTER,
C_POINTER}, C_LONG ),
xFindNextFile			= define_c_func( kernel32_dll, "FindNextFileW", {C_LONG,
C_POINTER}, C_INT ),
	xFindClose				= define_c_proc( kernel32_dll, "FindClose", {C_LONG} ),
xFileTimeToSystemTime	= define_c_func( kernel32_dll, "FileTimeToSystemTime",
{C_POINTER, C_POINTER}, C_INT )

global function allocate_unicode( sequence string )
-- allocates a null-terminated Unicode string into memory

	integer len, byte0, byte1
	atom mem
	
	len = length( string )
	mem = allocate( (len*2) + 2 )
	
	for i = 1 to len do
		byte0 = and_bits( string[i], #FF )
		byte1 = floor( string[i] / #100 )
		poke( mem + ((i-1)*2)+0, byte0 )
		poke( mem + ((i-1)*2)+1, byte1 )
	end for

	-- double null terminator
	poke( mem + (len*2)+0, 0 )
	poke( mem + (len*2)+1, 0 )

	return mem
end function

global function peek_unicode( atom mem )
-- reads a Unicode string from memory

	integer i, byte0, byte1
	sequence string

	i = 1
	string = ""
	while length(string) <= 32767 do
		byte0 = peek( mem + ((i-1)*2)+0 )
		byte1 = peek( mem + ((i-1)*2)+1 )
		if byte0 = 0 and byte1 = 0 then
			-- null terminator
			string = string[1..i-1]
			exit
		else
			string &= (byte1 * #100) + byte0
			i += 1
		end if
	end while

	return string
end function

function peek2( atom mem )
-- read a 2-byte value (Word) from memory
	return (peek(mem+1) * #100) + peek(mem+0)
end function

procedure poke2( atom mem, atom word )
	poke( mem+0, and_bits( word, #FF ) )
	poke( mem+1, floor( word / #100 ) )
end procedure

--global constant
--	-- FILETIME Structure
--	ft_dwLowDateTime		= w32allot( DWord ),	-- The FILETIME structure is a 64-bit
value representing the
--	ft_dwHighDateTime		= w32allot( DWord ),	-- number of 100-nanosecond intervals
since January 1, 1601.
--	SIZEOF_FILETIME			= w32allotted_size()

constant FILETIME_SIZE = 8
function FILETIME( object ft )

	atom ptr, dwLowDateTime, dwHighDateTime

	if atom( ft ) then
		-- read structure
		ptr = ft
		
		dwLowDateTime	= peek4s( ptr + 0 )
		dwHighDateTime	= peek4s( ptr + 4 )
		return {dwLowDateTime, dwHighDateTime}

	else
		-- write structure
		ptr = allocate( FILETIME_SIZE )

		poke4( ptr + 0, ft[1] )
		poke4( ptr + 4, ft[2] )
		return ptr

	end if

end function

--global constant
--	-- SYSTEMTIME Structure
--	st_wYear				= w32allot( Word ),
--	st_wMonth				= w32allot( Word ),
--	st_wDayOfWeek			= w32allot( Word ),
--	st_wDay					= w32allot( Word ),
--	st_wHour				= w32allot( Word ),
--	st_wMinute				= w32allot( Word ),
--	st_wSecond				= w32allot( Word ),
--	st_wMilliseconds		= w32allot( Word ),
--	SIZEOF_SYSTEMTIME		= w32allotted_size()

constant SYSTEMTIME_SIZE = 16

function SYSTEMTIME( object st )

	atom ptr, wYear, wMonth, wDayOfWeek, wDay,
		wHour, wMinute, wSecond, wMilliseconds
	
	if atom( st ) then
		-- read structure
		ptr = st
		
		wYear			= peek2( ptr +  0 )
		wMonth			= peek2( ptr +  2 )
		wDayOfWeek		= peek2( ptr +  4 )
		wDay			= peek2( ptr +  6 )
		wHour			= peek2( ptr +  8 )
		wMinute			= peek2( ptr + 10 )
		wSecond			= peek2( ptr + 12 )
		wMilliseconds	= peek2( ptr + 14 )
		return {wYear, wMonth, wDayOfWeek, wDay,
			wHour, wMinute, wSecond, wMilliseconds}
			
	else
		-- write structure
		ptr = allocate( SYSTEMTIME_SIZE )
		
		poke2( ptr +  0, st[1] )
		poke2( ptr +  2, st[2] )
		poke2( ptr +  4, st[3] )
		poke2( ptr +  6, st[4] )
		poke2( ptr +  8, st[5] )
		poke2( ptr + 10, st[6] )
		poke2( ptr + 12, st[7] )
		poke2( ptr + 14, st[8] )
		return ptr
	
	end if

end function

--global constant
--	-- WIN32_FIND_DATA Structure
--	wfd_dwFileAttributes	= w32allot( DWord ),
--	wfd_ftCreationTime		= w32allot( SIZEOF_FILETIME ),
--	wfd_ftLastAccessTime	= w32allot( SIZEOF_FILETIME ),
--	wfd_ftLastWriteTime		= w32allot( SIZEOF_FILETIME ),
--	wfd_nFileSizeHigh		= w32allot( DWord ),
--	wfd_nFileSizeLow		= w32allot( DWord ),
--	wfd_dwReserved0			= w32allot( DWord ),
--	wfd_dwReserved1			= w32allot( DWord ),
--	wfd_cFileName			= w32allot({ Lpsz, MAX_SIZE }),
--	wfd_cAlternateFileName	= w32allot({ Lpsz, 14 }),
--	SIZEOF_WIN32_FIND_DATA	= allotted_size()

constant WIN32_FIND_DATA_SIZE = 40 + 32767 + 12

function WIN32_FIND_DATA( object wfd )

	integer len
atom ptr, dwFileAttributes, nFileSizeHigh, nFileSizeLow, dwReserved0,
dwReserved1
sequence ftCreationTime, ftLastAccessTime, ftLastWriteTime, cFileName,
cAlternateFileName
	
	if atom( wfd ) then
		-- read structure
		ptr = wfd
		
		dwFileAttributes		= peek4s( ptr + 0 )
		ftCreationTime			= FILETIME( ptr +  4 )
		ftLastAccessTime		= FILETIME( ptr + 12 )
		ftLastWriteTime			= FILETIME( ptr + 20 )
		nFileSizeHigh			= peek4s( ptr + 28 )
		nFileSizeLow			= peek4s( ptr + 32 )
		dwReserved0				= peek4s( ptr + 36 )
		dwReserved1				= peek4s( ptr + 40 )

		len = c_func( xlstrlenW, {ptr+44} )
		cFileName				= peek_unicode( ptr+44 )
		cAlternateFileName		= peek_unicode( ptr+44+len )

		return {dwFileAttributes, ftCreationTime, ftLastAccessTime, ftLastWriteTime,
nFileSizeHigh, nFileSizeLow, dwReserved0, dwReserved1, cFileName,
cAlternateFileName}
		
	else
		-- write structure
		ptr = allocate( WIN32_FIND_DATA_SIZE
						+ (length(wfd[9])*2)+2
						+ (length(wfd[10])*2)+2 )
		
		poke4( ptr +  0, wfd[1] )
		poke4( ptr +  4, wfd[2] )
		poke4( ptr + 12, wfd[3] )
		poke4( ptr + 20, wfd[4] )
		poke4( ptr + 24, wfd[5] )
		poke4( ptr + 28, wfd[6] )
		poke4( ptr + 32, wfd[7] )
		poke4( ptr + 36, wfd[8] )
		
		len = length( wfd[9] )
		poke4( ptr + 40, wfd[9] )
		poke4( ptr + 40 + len, wfd[10] )
		return ptr
		
	end if

end function

constant
	-- File Attributes
    zFILE_ATTRIBUTE_READONLY             = #00000001,
    zFILE_ATTRIBUTE_HIDDEN               = #00000002,
    zFILE_ATTRIBUTE_SYSTEM               = #00000004,
    zFILE_ATTRIBUTE_DIRECTORY            = #00000010,
    zFILE_ATTRIBUTE_ARCHIVE              = #00000020,
    zFILE_ATTRIBUTE_DEVICE               = #00000040,
    zFILE_ATTRIBUTE_NORMAL               = #00000080,
    zFILE_ATTRIBUTE_TEMPORARY            = #00000100,
    zFILE_ATTRIBUTE_SPARSE_FILE          = #00000200,
    zFILE_ATTRIBUTE_REPARSE_POINT        = #00000400,
    zFILE_ATTRIBUTE_COMPRESSED           = #00000800,
    zFILE_ATTRIBUTE_OFFLINE              = #00001000,
    zFILE_ATTRIBUTE_NOT_CONTENT_INDEXED  = #00002000,
    zFILE_ATTRIBUTE_ENCRYPTED            = #00004000

constant
	INVALID_HANDLE_VALUE	= -1

constant POSSIBLE_ATTRIBUTES = {
	{ zFILE_ATTRIBUTE_DIRECTORY,	'd' },
	{ zFILE_ATTRIBUTE_READONLY,		'r' },
	{ zFILE_ATTRIBUTE_HIDDEN,		'h' },
	{ zFILE_ATTRIBUTE_SYSTEM,		's' },
	{ zFILE_ATTRIBUTE_DEVICE,		'v' },	-- just guessing here
	{ zFILE_ATTRIBUTE_ARCHIVE,		'a' }}

-- must include file.e for these constants:
--global constant 
--	D_NAME = 1,
--	D_ATTRIBUTES = 2,
--	D_SIZE = 3,
--
--	D_YEAR = 4,
--	D_MONTH = 5,
--	D_DAY = 6,
--
--	D_HOUR = 7,
--	D_MINUTE = 8,
--	D_SECOND = 9

function check_path( sequence path )

	if length(path) then
		if path[$] = '\\' then
			path &= '*'
		end if

		path = "\\\\?\\" & path
		return path
	end if

end function

function append_file( sequence wfd )

	sequence entry
	object filetime, systemtime

	entry = repeat(0,9)
	entry[D_NAME] = wfd[9]
	entry[D_SIZE] = (wfd[5] * #10000) + wfd[6]

	entry[D_ATTRIBUTES] = ""
	for i = 1 to length(POSSIBLE_ATTRIBUTES) do
		if and_bits( wfd[1], POSSIBLE_ATTRIBUTES[i][1] ) then
			entry[D_ATTRIBUTES] &= POSSIBLE_ATTRIBUTES[i][2]
		end if
	end for

	filetime = FILETIME( wfd[4] )
	systemtime = allocate( SYSTEMTIME_SIZE )
	if c_func( xFileTimeToSystemTime, {filetime, systemtime} ) then
		systemtime = SYSTEMTIME( systemtime )
			entry[D_YEAR] = systemtime[1] - 1900
		entry[D_MONTH] = systemtime[2]
		entry[D_DAY] = systemtime[4] + 1
		entry[D_HOUR] = systemtime[5]
		entry[D_MINUTE] = systemtime[6]
		entry[D_SECOND] = systemtime[7]
	end if

	return {entry}
end function


global function win_dir( sequence name )
-- returns directory information, given the name
-- of a file or directory. Format returned is:
-- {
--  {"name1", attributes, size, year, month, day, hour, minute, second},
--  {"name2", ...                                                     },
-- }

	atom lpName, lpWfd, handle
	sequence wfd, files

	name = check_path( name )
	files = {}

	lpName = allocate_unicode( name )
	lpWfd = allocate( WIN32_FIND_DATA_SIZE )
	
	handle = c_func( xFindFirstFile, {lpName, lpWfd} )
	if handle = INVALID_HANDLE_VALUE then
		-- failed!
		free( lpName )
		free( lpWfd )
		return 0
	end if
	
	wfd = WIN32_FIND_DATA( lpWfd )
	files &= append_file( wfd )

	while c_func( xFindNextFile, {handle, lpWfd} ) do
		wfd = WIN32_FIND_DATA( lpWfd )
		files &= append_file( wfd )
	end while

	c_proc( xFindClose, {handle} )
	
	free( lpName )
	free( lpWfd )
	return files
end function

-- ags -- win_dir() not compatible with walk_dir, see new walk_win_dir
--global function walk_win_dir(sequence path_name, integer your_function, 
--                               integer scan_subdirs)
--    object nothing
--    trace(1)
--    my_dir = routine_id("win_dir") -- use win_dir() please
--                                   -- my_dir is global atom in file.e
--    nothing = walk_dir(path_name & "\\", your_function, scan_subdirs)
--    my_dir = -2 	               -- constant DEFAULT=-2 in file.e
--    return nothing
--end function

-- check_slash(): appends a '\' to the path if needed so that win_dir can append
'*' 
-- NOTE: since this is windows specific, slash is only '\'
-- won't cope with "\\*" though
function check_slash(sequence path)
	if not equal (path[$], "\\")  then
		path = path & "\\" -- to be compatible with win_dir()
	end if
	return path
end function

global function walk_win_dir(sequence path_name, integer your_function, 
			 integer scan_subdirs)
-- Generalized Directory Walker
--  modified by Gary Shingles <eu at 531pi.co.nz> for win_dir()
-- Walk through a directory and (optionally) its subdirectories,
-- "visiting" each file and subdirectory. Your function will be called
-- via its routine id. The visits will occur in alphabetical order.
-- Your function should accept the path name and dir() entry for
-- each file and subdirectory. It should return 0 to keep going,
-- or an error code (greater than 0) to quit, or it can return
-- any sequence or atom other than 0 as a useful diagnostic value.
    object d, abort_now, path
    
    -- get the full directory information
    path = check_slash(path_name)
    
    d = win_dir(path)
    if atom(d) then
	return W_BAD_PATH
    end if
    
    -- trim any trailing blanks or slashes from the path
    while length(path) > 0 and 
	  find(path[$], {' ', '\\'}) do
	path = path[1..$-1]
    end while
    
    for i = 1 to length(d) do
	if find('d', d[i][D_ATTRIBUTES]) then
	    -- a directory
	    if not find(d[i][D_NAME], {".", ".."}) then
		abort_now = call_func(your_function, {path, d[i]})
		if not equal(abort_now, 0) then
		    return abort_now
		end if
		if scan_subdirs then
		    abort_now = walk_win_dir(path & '\\' & d[i][D_NAME],
					 your_function, scan_subdirs)
		    
		    if not equal(abort_now, 0) and 
		       not equal(abort_now, W_BAD_PATH) then
			-- allow BAD PATH, user might delete a file or directory 
			return abort_now
		    end if
		end if
	    end if
	else
	    -- a file
abort_now = call_func(your_function, {path_name, d[i]}) -- note: path_name
without slash
	    if not equal(abort_now, 0) then
		return abort_now
	    end if
	end if
    end for
    return 0
end function

Example use (essential bits only):

constant MODE_UPDATE = 1,
         MODE_INSERT = 2
function real_indexer(sequence pathname, sequence entry)
 
     real_pathname = pathname & "\\" & entry[D_NAME]
     rec = blank_record

str = lower(reverse(entry[D_NAME])) FOLDERS if equal(str,".") or equal(str, "..") then return 0

SHORT too short elsif length(str) < 5 then return 0

LNK FILE .lnk file elsif equal(str[1..4], "knl.") then real_pathname = deref_lnk(pathname & "
" & entry[D_NAME])

if length(real_pathname) = 0 then msg(sprintf("Broken Link %s", {entry[D_NAME]})) return 0 broken link end if str = reverse(real_pathname) update str

if equal(str[1..4], "3pm.") then files = {real_pathname} elsif equal(str[1..4], "trc.") then files = get_cart_files(real_pathname) else return 0 not interested end if elsif equal(str[1..4], "3pm.") then MP3 files = {real_pathname} elsif equal(str[1..4], "trc.") then CRT files = get_cart_files(real_pathname) else return 0 end if

rec[TRK_KEY] = pathname & "
" & entry[D_NAME] always rec[TRK_PATH] = pathname rec[TRK_FILE] = files

idx = db_find_key(rec[TRK_KEY]) if idx >= 0 then orig_rec = db_record_data(idx) orig_ts = orig_rec[TRK_TS] comp = compare(entry[3..9], orig_ts) size and/or time changed if comp = 0 then return 0 no change else cart_seq = orig_rec[TRK_CART] mode = MODE_UPDATE end if else mode = MODE_INSERT end if

parent eg: C:\MUSIC, parent = C: C:\MUSIC\MODERN parent = C:\MUSIC bits = split(pathname, '
') if length(bits) = 0 then

weird msg("Weirdness in real_indexer") return 0 elsif length(bits) = 1 then str = bits[1] cats = {} else str = {} cats = {} for i = 1 to length(bits) - 1 do str &= bits[i] if i > 3 then cats = append(cats, bits[i]) never a "C:" category or "MUSIC" category of MUSIC end if

if i != length(bits) - 1 then str &= '
' end if end for cats = append(cats, bits[length(bits)]) end if pdep = length(bits) - 1 TRK_PDEP path depth rec[TRK_PDEP] = pdep TRK_PARENT if find(pathname, rootfols) then this is a root folder, parent is {} rec[TRK_PARENT] = {} else rec[TRK_PARENT] = str end if

TRK_CATS rec[TRK_CATS] = cats

TRK_NAME rec[TRK_NAME] = get_track_name(real_pathname) TRK_TS rec[TRK_TS] = entry[3..9] han = BASS_StreamCreateFile(BASS_FALSE, rec[TRK_FILE][cart_seq], 0, 0, 0) len = BASS_StreamGetLength(han) sec = BASS_ChannelBytes2Seconds(han, len) BASS_StreamFree(han)

TRK_LEN rec[TRK_LEN] = {len} TRK_SEC rec[TRK_SEC] = {s2hms(sec)}

TRK_CART rec[TRK_CART] = cart_seq COMMIT

if mode = MODE_UPDATE then msg(sprintf("Updated: %s", {rec[TRK_KEY]})) db_replace_data(idx, rec) else msg(sprintf("Inserting: %s", {rec[TRK_KEY]})) junk = db_insert(rec[TRK_KEY], rec) if junk != DB_OK then msg(sprintf("Insert failed! %s (err: %d)", {entry[D_NAME], junk})) end if end if return 0 doEvents(0) end function

procedure index_folder(sequence folname) junk = walk_win_dir(folname, routine_id("real_indexer"),1) end procedure

gary }}}

Not Categorized, Please Help

new topic     » goto parent     » topic index » view message » categorize

C Bouzy wrote:
> 
> Hi All,
> 
> I am trying to speed up how I scan a audio folder and add it to a Playlist.
> It works fast if there is only a few hundred files, but if there is 500
> or more files it can get to be slow. I scan the folder using walk_dir
> and add each file into the database. Do any of you guys think I can do
> this without using walk_dir? I have pasted a bit of the code below.
> 
> 
> }}}
<eucode>
> function ScanDir(sequence path_name, sequence ItemEntry)
> sequence addaudio,streamtime,filecount,filetitle
> atom len,tempstream
> 
> streamtime={}
> addaudio={}
> filecount={}
> doEvents(0)
> 
> 
> if equal(ItemEntry[D_ATTRIBUTES], "d") then  ----If directory do nothing
> 
>   else
>          if match("aac", get_file_ext(lower(ItemEntry[D_NAME]))) or   ---- If
>          supported file add to the Playlist
>          match("mp4", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp3", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp2", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("mp1", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("ogg", get_file_ext(lower(ItemEntry[D_NAME]))) or
>          match("wav", get_file_ext(lower(ItemEntry[D_NAME])))
>   then
> 
>     tempstream = BASS_StreamCreateFile(0, path_name & "\\" &
>     ItemEntry[D_NAME], 0, 0, BASS_STREAM_AUTOFREE) ---Create a temp file to get the
>     file length
>       len = BASS_ChannelBytes2Seconds(tempstream, 
>       BASS_ChannelGetLength(tempstream))
>        streamtime = sprintf("               %d:%02d", {len / 60,
>        remainder(len, 60)})
>         filetitle = get_file_title(ItemEntry[D_NAME])
> 
>          if mdb_insert(appdir&"\\Playlist.dat","Playlist",convertCase(replace
>          (filetitle,"_"," ")),{path_name & "\\" & ItemEntry[D_NAME],streamtime})
> = -1
>             then
>                else
>                 addaudio &= addLVItem(Playlist,listicon2,{" 
>                 "&convertCase(replace (filetitle,"_"," "))," "&streamtime})
>                 end if
>                setIndex(Playlist, 1)
>               getCount()
>             end if
>           end if
>         doEvents(0)
>      return 0
> end function
> </eucode>
{{{

> 
> 
> VOID = walk_dir(audiofolder, routine_id("ScanDir"), 1)

I don't think it is a problem with walk_dir.  I can scan through 85,000 files
and print the results to the screen (very costly) in about 72 sec. on an old
900MHz Intel PIII (should be a similar cost to inserting a record in your DB).
Without printing the information on the screen, the files are scanned in about
2 sec.

Your costs are most likely a bit higher because you open up a stream 
extract the play time and insert data into your database.

I notice that you never close your temporary stream. I see that you have the 
"BASS_STREAM_AUTOFREE" flag set but I don't see a call to either
BASS_ChannelStop()
or BASS_Stop(). It appears to me that you are loading up your available memory
with
temporary streams and bogging down your system.

John

12. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 14, 2005
• 587 views

John Sas wrote:

> I notice that you never close your temporary stream. I see that you have the
> 
> "BASS_STREAM_AUTOFREE" flag set but I don't see a call to either
> BASS_ChannelStop()
> or BASS_Stop(). It appears to me that you are loading up your available memory
> with
> temporary streams and bogging down your system.
John,

I am not playing a file so I do not need to call Bass_Stop. 
The stream is being created only to get the length of each file.
Once I open a new stream the previous stream is automatically freed.
As I stated before I tried this without using the Bass functions and it
was still slow. It is not Bass that is slowing it down, it is walk_dir.


----If you continue to do what you have always done,
you will get what you have always gotten.----

ags wrote:

> I had to modify it I think for the way I was using it.  Here's the code 
> (thanks again Greg :) followed by a short context of how I use it:
> 
Hi ags,

Thank you for the post. I began tweaking Greg's walk_win_dir function
right after your first post. I figured out why it was not working and
tried it with my code. It did speed up finding files, but still not as
fast as I want. I took a look at your code, try using BASS_STREAM_AUTOFREE,
instead of calling BASS_StreamFree(han). You will need the latest Bass.dll
to use BASS_STREAM_AUTOFREE.

If you are interested, I am using the Playlist creation function for the
following app:
http://www.insight-concepts.com/nexusradio/beta/nexusradio_beta1.exe



----If you continue to do what you have always done,
you will get what you have always gotten.----

14. Re: Could this be faster?

• Posted by petelomax at fastmail.fm Dec 15, 2005
• 566 views

I though I'd try walk_dir using the windows API.
This appears to be about 3 times faster, YMMV.
It uses arwen, though it should be relatively trivial to change
the constants to conform to win32lib.

include arwen.ew
constant 
    -- Define FINDDATA Structure
    FINDDATA                = new_struct(),
    FINDDATA_dwAttr         = struc(C_LONG),
    FINDDATA_Times3         = struc(anySize(24)),
    FINDDATA_nFileSizeHigh  = struc(C_LONG),
    FINDDATA_nFileSizeLow   = struc(C_LONG),
    FINDDATA_res2           = struc(anySize(8)),
    FINDDATA_cFileName      = struc(anySize(260)),
    FINDDATA_cAltFileName   = struc(anySize(14)),

    fd = allocate(sizeof(FINDDATA)),

    xFindFirstFile  = link_c_func(kernel32, "FindFirstFileA",
        {C_POINTER, --  LPCTSTR  lpFileName,    // address of name of
        file to search for  
         C_POINTER},--  LPWIN32_FIND_DATA  lpFindFileData   // address
         of returned information 
        C_LONG),    -- HANDLE

    xFindNextFile   = link_c_func(kernel32, "FindNextFileA",
        {C_POINTER, --  HANDLE  hFindFile,  // handle of search  
         C_POINTER},--  LPWIN32_FIND_DATA  lpFindFileData   // address
         of structure for data on found file  
        C_INT),     -- BOOL

    xFindClose      = link_c_func(kernel32, "FindClose",
        {C_POINTER},--  HANDLE  hFindFile   // file search handle 
        C_INT),     -- BOOL

    INVALID_HANDLE_VALUE=-1

function get_file_ext(sequence filename)
integer c
    for i=length(filename) to 1 by -1 do
        c = filename[i]
        if c = '.' then
            return filename[i+1..length(filename)]
        elsif c>='A' and c<='Z' then
            filename[i] = c+32
        end if
    end for
    return ""
end function

procedure wd(sequence path)
-- walk_dir coded using winAPI direct.
-- results not in any particular order.
integer d
atom fH, attr
sequence longname
sequence ipath
sequence dirstack

    ipath = path
    dirstack={}

    if platform()!=2 then ?9/0 end if   --Win32 only supported

    while 1 do
        --
        -- Force consistent use of '\\' vs '/'.
        --
        d=find('/',path)
        if d=0 then exit end if
        path[d]='\\'
    end while

    fH=INVALID_HANDLE_VALUE
    if length(path) and path[length(path)]='\\' then
        path&="*.*"
    elsif length(path) and path[length(path)]=':' then
        ipath = path&'\\'
        path&="\\*.*"
    --NB I doubt this will work with wildcards!
    elsif not find('*',path) and not find('?',path) then
        --
        -- Check if the passed path is a directory;
        -- if it is, list it's contents, not just it.
        --
        fH = c_func(xFindFirstFile,{allocate_StringZ(path),fd})
        if fH!=INVALID_HANDLE_VALUE then
            attr = peek4u(fd + FINDDATA_dwAttr)
            if and_bits(attr,#10) then  -- directory bit
                ipath&='\\'
                path&="\\*.*"
                if c_func(xFindClose,{fH}) then end if
                fH=INVALID_HANDLE_VALUE
            end if
        end if
    end if
    if fH=INVALID_HANDLE_VALUE then
        fH = c_func(xFindFirstFile,{allocate_StringZ(path),fd})
    end if
    if fH!=INVALID_HANDLE_VALUE then
        while 1 do
            attr = peek4u(fd + FINDDATA_dwAttr)
            longname = peek_string(fd + FINDDATA_cFileName)
            if and_bits(attr,#10) then
                if not find(longname,{".",".."}) then
                    dirstack = append(dirstack,ipath&longname)
                end if
            else
                if find(get_file_ext(longname),{"aac","mp4", "mp3",
                "mp2", "mp1", "ogg", "wav"}) then
-- DEV: call external routine/append to result stack
                    puts(1,longname&'\n')
                end if
            end if
            if not c_func(xFindNextFile,{fH,fd}) then exit end if
        end while
        if c_func(xFindClose,{fH}) then end if
        for i=1 to length(dirstack) do
            wd(dirstack[i])
        end for
    end if
end procedure
wd("C:\\")

Regards,
Pete

-- 
  
  petelomax at fastmail.fm

-- 
http://www.fastmail.fm - Access your email from home and the web

petelomax wrote:

> I though I'd try walk_dir using the windows API.
> This appears to be about 3 times faster, YMMV.
> It uses arwen, though it should be relatively trivial to change
> the constants to conform to win32lib.

[code snipped]

Cool. 

Mike, maybe you'd like to include that code "officially" into ARWEN?
Just an idea.

Regards,
   Juergen

16. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 15, 2005
• 575 views

petelomax wrote:
> I though I'd try walk_dir using the windows API.
> This appears to be about 3 times faster, YMMV.
> It uses arwen, though it should be relatively trivial to change
> the constants to conform to win32lib.

Hello Pete,

I tried your exmaple and it is indeed 3x faster or more than EU native
walk_dir. Good job and thanks.....

Chris

----If you continue to do what you have always done,
you will get what you have always gotten.----

Here is the Win32Lib friendly version to Pete's walk_dir if anyone
is interested.

include win32lib.ew
without warning

sequence ListOfStructures
ListOfStructures ={}

integer CurrentStructure, LatestSize
atom LabelPointerStart, LabelBlockSize, LabelPointer, LabelPointerEnd

function new_struct()
	ListOfStructures &= 0
	CurrentStructure = length(ListOfStructures)
	LatestSize = 0
	return CurrentStructure
end function

function sizeof(integer id)
	return ListOfStructures[id]
end function

function struc(integer sType)
	integer disp, size
	disp = LatestSize
	if sType > #00FFFFFF then -- this type is defined in dll.e
		size = and_bits(sType, #00FFFFFF)
	else
		size = ListOfStructures[sType]
	end if
	LatestSize += size
	ListOfStructures[CurrentStructure] = LatestSize
	return disp
end function

function anySize(integer size) -- allow arbitrary sized elements
	return or_bits(#01000000, size)
end function

LabelBlockSize = 100000
LabelPointerStart = allocate(LabelBlockSize)
LabelPointer = LabelPointerStart
LabelPointerEnd = LabelPointerStart + LabelBlockSize - 2

function allocate_StringZ(sequence string)
-- allocate, poke & null-terminate a text string

	atom len, temp

	len = length(string)

        -- advance the pointer
	temp = LabelPointer
	LabelPointer += len + 1
	if LabelPointer > LabelPointerEnd then
		LabelPointer = LabelPointerStart + len + 1
		temp = LabelPointerStart
	end if

	-- poke the string into memory
	poke(temp, string)
	poke(temp + len, 0)

	-- exit
	return temp
end function

procedure FatalErr(sequence err_msg)
	integer void
	VOID = message_box("Fatal Error", "Error", {MB_OK, MB_ICONERROR})
	abort(1)
end procedure

procedure link_error(sequence name)
	FatalErr("Couldn't link function:\n\n " & name & "()")
end procedure

function link_c_func(atom dll, sequence name, sequence args, atom result)
	integer handle
	handle = define_c_func(dll, name, args, result)
	if handle = -1 then
                link_error(name)
	else
		return handle
	end if
end function

-- dynamically link a C routine as a Euphoria function
function link_c_proc(atom dll, sequence name, sequence args)
	integer handle
	handle = define_c_proc(dll, name, args)
	if handle = -1 then
                link_error(name)
	else
		return handle
	end if
end function

global function assign_dll(sequence dll_file)
	atom handle
	handle = open_dll(dll_file)
	if handle = NULL then
                link_error(dll_file)
	end if
	return handle
end function

constant
    Kernel32 = assign_dll("kernel32.dll"),
    FINDDATA                = new_struct(),
    FINDDATA_dwAttr         = struc(C_LONG),
    FINDDATA_Times3         = struc(anySize(24)),
    FINDDATA_nFileSizeHigh  = struc(C_LONG),
    FINDDATA_nFileSizeLow   = struc(C_LONG),
    FINDDATA_res2           = struc(anySize(8)),
    FINDDATA_cFileName      = struc(anySize(260)),
    FINDDATA_cAltFileName   = struc(anySize(14)),
    fd = allocate(sizeof(FINDDATA)),
xFindFirstFile  = link_c_func(Kernel32, "FindFirstFileA", {C_POINTER,
    C_POINTER},  C_LONG),
xFindNextFile   = link_c_func(Kernel32, "FindNextFileA",{C_POINTER,
    C_POINTER}, C_INT),
    xFindClose      = link_c_func(Kernel32, "FindClose", {C_POINTER},C_INT),

    INVALID_HANDLE_VALUE=-1

function get_file_ext(sequence filename)
integer c
    for i=length(filename) to 1 by -1 do
        c = filename[i]
        if c = '.' then
            return filename[i+1..length(filename)]
        elsif c>='A' and c<='Z' then
            filename[i] = c+32
        end if
    end for
    return ""
end function

procedure wd(sequence path)
-- walk_dir coded using winAPI direct.
-- results not in any particular order.
integer d
atom fH, attr
sequence longname
sequence ipath
sequence dirstack

    ipath = path
    dirstack={}

    if platform()!=2 then ?9/0 end if   --Win32 only supported

    while 1 do
        --
        -- Force consistent use of '\\' vs '/'.
        --
        d=find('/',path)
        if d=0 then exit end if
        path[d]='\\'
    end while

    fH=INVALID_HANDLE_VALUE
    if length(path) and path[length(path)]='\\' then
        path&="*.*"
    elsif length(path) and path[length(path)]=':' then
        ipath = path&'\\'
        path&="\\*.*"
    --NB I doubt this will work with wildcards!
    elsif not find('*',path) and not find('?',path) then
        --
        -- Check if the passed path is a directory;
        -- if it is, list it's contents, not just it.
        --
        fH = c_func(xFindFirstFile,{allocate_StringZ(path),fd})
        if fH!=INVALID_HANDLE_VALUE then
            attr = peek4u(fd + FINDDATA_dwAttr)
            if and_bits(attr,#10) then  -- directory bit
                ipath&='\\'
                path&="\\*.*"
                if c_func(xFindClose,{fH}) then end if
                fH=INVALID_HANDLE_VALUE
            end if
        end if
    end if
    if fH=INVALID_HANDLE_VALUE then
        fH = c_func(xFindFirstFile,{allocate_StringZ(path),fd})
    end if
    if fH!=INVALID_HANDLE_VALUE then
        while 1 do
            attr = peek4u(fd + FINDDATA_dwAttr)
            longname = peek_string(fd + FINDDATA_cFileName)
            if and_bits(attr,#10) then
                if not find(longname,{".",".."}) then
                    dirstack = append(dirstack,ipath&longname)
                end if
            else
                if find(get_file_ext(longname),{"aac","mp4", "mp3",
                "mp2", "mp1", "ogg", "wav"}) then
-- DEV: call external routine/append to result stack
                    puts(1,longname&'\n')
                end if
            end if
            if not c_func(xFindNextFile,{fH,fd}) then exit end if
        end while
        if c_func(xFindClose,{fH}) then end if
        for i=1 to length(dirstack) do
            wd(dirstack[i])
        end for
    end if
end procedure
wd("C:\\")

----If you continue to do what you have always done,
you will get what you have always gotten.----

18. Re: Could this be faster?

• Posted by C Bouzy <eucoder at hotmail.com> Dec 15, 2005
• 565 views
• Last edited Dec 16, 2005

Pete,

I added lower() to the following:

if find(get_file_ext(lower(longname)),{"aac","mp4", "mp3","mp2", "mp1", "ogg",
"wav"}) then

Without the lower() some files will be skipped do to the fact the file
extension is not the same case.

Chris

----If you continue to do what you have always done,
you will get what you have always gotten.----

C Bouzy wrote:

> Pete,
> 
> I added lower() to the following:
> }}}
<eucode> 
> if find(get_file_ext(lower(longname)),{"aac","mp4", "mp3","mp2", "mp1", "ogg",
> "wav"}) then
> </eucode>
{{{

> 
> Without the lower() some files will be skipped do to the fact the file
> extension is not the same case.


If Pete's procedure wd() should be suited for general use, that code
shouldn't be inside the procedure anyway ... 

Regards,
   Juergen

20. Re: Could this be faster?

• Posted by petelomax at fastmail.fm Dec 16, 2005
• 601 views

On Thu, 15 Dec 2005 20:14:40 -0800, "C Bouzy" <guest at RapidEuphoria.com>
said:
> I added lower() to the following:
> }}}
<eucode> 
> if find(get_file_ext(lower(longname)),{"aac","mp4", "mp3","mp2", "mp1",
> "ogg", "wav"}) then
> </eucode>
{{{

> 
> Without the lower() some files will be skipped do to the fact the 
> file extension is not the same case.

This should be handled by get_file_ext:

if not equal(get_file_ext("a.AAC"),"aac") then ?9/0 end if
if not equal(get_file_ext("a.Mp4"),"mp4") then ?9/0 end if
if not equal(get_file_ext("a.MP4"),"mp4") then ?9/0 end if
if not equal(get_file_ext("a.ogg"),"ogg") then ?9/0 end if
if not equal(get_file_ext("a.ogG"),"ogg") then ?9/0 end if
if not equal(get_file_ext("a.Wav"),"wav") then ?9/0 end if
if not equal(get_file_ext("a.WAV"),"wav") then ?9/0 end if

The above works fine here. Did you change get_file_ext?

Regards,
Pete
PS Since I was looking at it, I spotted an unnecessary memory alloc.
This should work exactly the same but be a fraction faster:

function get_file_ext(sequence filename)
integer c
sequence res
    for i=length(filename) to 1 by -1 do
        c = filename[i]
        if c = '.' then
            res = filename[i+1..length(filename)]
            for j=1 to length(res) do
                c = res[j]
                if c>='A' and c<='Z' then
                    res[j] = c+32
                end if
            end for
            return res
        end if
    end for
    return ""
end function

PPS JL is right: the extension set should really be a parameter or
similar.
-- 
  
  petelomax at fastmail.fm

-- 
http://www.fastmail.fm - Access all of your messages and folders
                          wherever you are

petelomax wrote:

> The above works fine here. Did you change get_file_ext?
> 

I already had a function called get_file_ext, so I used mines instead.
I assumed your version was the same as mines, I assumed wrong. Here is the
version I am using:

function get_file_ext(sequence fname)
        for i = length (fname) to 1 by -1 do
                if fname [i] = '.' then
                        return fname [i + 1 .. length (fname)]
                elsif fname [i] = '\\' or fname [i] = '/' then
                        return ""
                end if
        end for
        return ""
end function

Then all I do is check for the file extensions

if find(get_file_ext(lower(fname)),{"aac","mp4", "mp3","mp2", "mp1", "ogg",
"wav"}) then

----If you continue to do what you have always done,
you will get what you have always gotten.----