OpenEuphoria

2. Re: Downloading messages

• Posted by Mario Steele <eumario at trilake.net> May 18, 2005
• 421 views

Robert Craig wrote:
> 
> Some people have Internet connections where they pay by the minute, 
> or have a monthly time limit. Those people don't like to read 
> EUforum while connected, but in some cases they can't use 
> Topica either, due to spam filtering or some other silly problem.
> 
> It might be helpful for these people to know that all
> messages for the current month are stored in:
> 
>   <a
>   href="http://www.listfilter.com/mlist/current.txt">http://www.listfilter.com/mlist/current.txt</a>
> 
> Right now it's around 300K. It gets updated immediately
> when a new message is posted.
> 
> Previous months are stored as:
>    YYYYMM.TXT
> 
> e.g. April 2005 is stored as
>    <a
>    href="http://www.listfilter.com/mlist/200504.TXT">http://www.listfilter.com/mlist/200504.TXT</a>

Thanks Rob, this should make things easier for people.  And to add another level
of simplistic views to it, I'm currently working on a Front End GUI, with
wxEuphoria,
to download the Messages, and display them in a GUI Format.  I've got all the
base
parsing, and ideals down for it.  It will, as many may want, have Threading
abilities
in it.

Meaning, that messages such as this, will come up looking like this in a Tree
View:
[-] Downloading Messages
    Robert Craig (May 17 4:47)
    Mario Steele (May 18 6:02)

Etc Etc.

Any suggestions, ideas, and what nots, are much appreciated.  Also, Rob, would
it be possible to setup a CGI Script that'll take the User's Name, Password,
and post information all at the same time?  Cause I might also incorperate a
posting ability through the Web Forum into the program itself.

Mario Steele
http://enchantedblade.trilake.net
Attaining World Dominiation, one byte at a time...

4. Re: Downloading messages

• Posted by Robert Craig <rds at RapidEuphoria.com> May 19, 2005
• 417 views

Mario Steele wrote:
> Robert Craig wrote:
> > 
> > Some people have Internet connections where they pay by the minute, 
> > or have a monthly time limit. Those people don't like to read 
> > EUforum while connected, but in some cases they can't use 
> > Topica either, due to spam filtering or some other silly problem.
> > 
> > It might be helpful for these people to know that all
> > messages for the current month are stored in:
> > 
> >   <a
> >   href="http://www.listfilter.com/mlist/current.txt">http://www.listfilter.com/mlist/current.txt</a>
> > 
> > Right now it's around 300K. It gets updated immediately
> > when a new message is posted.
> > 
> > Previous months are stored as:
> >    YYYYMM.TXT
> > 
> > e.g. April 2005 is stored as
> >    <a
> >    href="http://www.listfilter.com/mlist/200504.TXT">http://www.listfilter.com/mlist/200504.TXT</a>
> 
> Thanks Rob, this should make things easier for people.  And to add another
> level
> of simplistic views to it, I'm currently working on a Front End GUI, with
> wxEuphoria,
> to download the Messages, and display them in a GUI Format.  I've got all the
> base
> parsing, and ideals down for it.  It will, as many may want, have Threading
> abilities
> in it.
> 
> Meaning, that messages such as this, will come up looking like this in a Tree
> View:
> [-] Downloading Messages
>     Robert Craig (May 17 4:47)
>     Mario Steele (May 18 6:02)
> 
> Etc Etc.
> 
> Any suggestions, ideas, and what nots, are much appreciated.  Also, Rob, would
> it be possible to setup a CGI Script that'll take the User's Name, Password,
> and post information all at the same time?  Cause I might also incorperate a
> posting ability through the Web Forum into the program itself.

I'm not sure how much work that would be.
I'd have to discuss it with Junko.
I guess if a lot of people wanted to use your system,
I could spare some time.

Thanks,
   Rob Craig
   Rapid Deployment Software
   http://www.RapidEuphoria.com

6. Re: Downloading messages

• Posted by Derek Parnell <ddparnell at bigpond.com> May 20, 2005
• 426 views

Robert Craig wrote:
[snip]

> all messages for the current month are stored in ...

I wrote a tool to convert EUForum archive files into the .MBX format.
Many news readers and email clients can import messages stored
in that format.

procedure Usage(sequence pName)
puts  (1, "(c) 2005 Derek Parnell. Convert a EUforum archive to MBX
    format\n")
    printf(1, "Usage:- %s filename ...\n", {pName})
    puts  (1, "  For each input file, a new file of type '.mbx' is created.\n")
end procedure

function begins(sequence pString, sequence pSubStr)
    if length(pString) < length(pSubStr) then
        return 0
    end if

    if length(pSubStr) = 0 then
        return 0
    end if

    if equal(pString[1..length(pSubStr)], pSubStr) then
        return length(pSubStr) + 1
    else
        return 0
    end if

end function

function format_date(sequence pText)
    -- Returns a sequence of YYYY,MMM,dd,hh,mm,ss
    -- Input is in the form YYYY MMM dd hh:mm

    sequence lRetValue
    integer lPos
    integer lWS

    lRetValue = repeat("", 6)
    lPos = 1
    for i = 1 to length(pText) do
        if pText[i] != ' ' and pText[i] != ':' then
            lRetValue[lPos] &= pText[i]
            lWS = 1
        else
            if lWS = 1 then
                lPos += 1
                lWS = 0
            end if
        end if
    end for

    for i = 4 to 6 do
        if length(lRetValue[i]) = 1 then
            lRetValue[i] = '0' & lRetValue[i]
        end if
    end for
    return lRetValue
end function

constant
    kSpecHdrs = {"X-EUFORUM", "Date", "From"},
    kMessageId = 1,
    kDate = 2,
    kFrom = 3

procedure process_file(sequence pFileName)
    integer lInputFH
    integer lOutputFH
    object lLine
    integer lLineCnt
    sequence lOutName
    integer lPos
    sequence lHeaders
    integer lDoingHdrs
    object lTemp
    sequence lDate

    printf(1, "Processing '%s' ... ", {pFileName})

    lInputFH = open(pFileName, "r")
    if lInputFH = -1 then
        printf(2, "** Unable to open input file.\n", {})
        abort(1)
    end if

    lPos = length(pFileName)
    lOutName = pFileName
    while lPos > 0 do
        if pFileName[lPos] = '.' then
            lOutName = pFileName[1 .. lPos-1]
            exit
        end if
        lPos -= 1
    end while

    lOutName &= ".mbx"
    lOutputFH = open(lOutName, "w")
    if lOutputFH = -1 then
        printf(2, "** Unable to open output file '%s'.\n", {lOutName})
        abort(1)
    end if

    lLineCnt = 0
    lDoingHdrs = 1
    lLine = gets(lInputFH)
    while sequence(lLine) do
        lLineCnt += 1
        if length(lLine) > 0 then
            if lLine[$] = 10 then
                lLine = lLine[1..$-1]
            end if
        end if

        if equal(lLine, "-= B E G I N  =-") then
            lHeaders = repeat("", length(kSpecHdrs))
            lDoingHdrs = 1
        elsif length(lLine) = 0 then
            if lDoingHdrs = 1 then
                lDoingHdrs = 0
                -- Process the headers. --
                lDate = format_date(lHeaders[kDate])

                -- The 'MBX message start line'
                lPos = find('<', lHeaders[kFrom])
                lTemp = lHeaders[kFrom][lPos+1 .. $-1]
                for j = 1 to length(lTemp) do
                    if lTemp[j] = ' ' then
                        lTemp[j] = '.'
                    end if
                end for
                printf(lOutputFH, "From %s %s %s %s:%s:00 %s\n", {lTemp,
                                lDate[2],
                                lDate[3],
                                lDate[4],
                                lDate[5],
                                lDate[1]
                                    })

                -- The message id
                printf(lOutputFH, "Message-ID: <%s>\n", {lHeaders[kMessageId]})

                printf(lOutputFH, "From: %s\n", {lHeaders[kFrom]})
                printf(lOutputFH, "Date: Mon, %s %s %s %s:%s:00 +0000\n",
                            {
                                lDate[3],
                                lDate[2],
                                lDate[1],
                                lDate[4],
                                lDate[5]
                                })

                for j = length(kSpecHdrs)+1 to length(lHeaders) do
                    printf(lOutputFH, "%s\n", {lHeaders[j]})
                end for

                puts(lOutputFH, "\n")

                lHeaders = repeat("", length(kSpecHdrs))
            else
                printf(lOutputFH, "%s\n", {lLine})
            end if
        else
            if lDoingHdrs = 1 then
                for j = 1 to length(kSpecHdrs) do
                    lPos = begins(lLine, kSpecHdrs[j] & ": ")
                    if lPos > 0 then
                        lHeaders[j] = lLine[lPos .. $]
                        exit
                    end if
                    if j = length(kSpecHdrs) then
                        -- Not a special header.
                        lHeaders = append(lHeaders, lLine)
                    end if
                end for


            else
                printf(lOutputFH, "%s\n", {lLine})
            end if
        end if

        lLine = gets(lInputFH)
    end while

    printf(1, "( %d lines)\n", lLineCnt)
    close(lInputFH)
    close(lOutputFH)
end procedure

procedure main(sequence pArgs)
    if length(pArgs) < 3 then
        if equal(pArgs[1], pArgs[2]) then
            Usage(pArgs[1])
        else
            Usage("ex " & pArgs[2])
        end if
        abort(0)
    end if

    for i = 3 to length(pArgs) do
        process_file(pArgs[i])
    end for

end procedure

main(command_line())

-- 
Derek Parnell
Melbourne, Australia
irc://irc.sorcery.net:9000/euphoria

8. Re: Downloading messages

• Posted by Derek Parnell <ddparnell at bigpond.com> May 20, 2005
• 425 views
• Last edited May 21, 2005

cklester wrote:
> 
> Derek Parnell wrote:
> > 
> > I wrote a tool to convert EUForum archive files into the .MBX format.
> > Many news readers and email clients can import messages stored
> > in that format.
> 
> Derek, what does that same program look like in D? (just curious; send via
> email if you want, but i think comparisons are always on topic...). :)

Note: D's arrays start at zero and not 1.

// --------CODE STARTS -------------
import std.stdio;
import std.stream;
import std.string;

alias char[] string;

void Usage(string pName)
{
    writefln("(c) 2005 Derek Parnell. "
             "Convert a EUforum archive to MBX format");
    writefln("Usage:- %s filename ...", pName);
    writefln("  For each input file, "
             "a new file of type '.mbx' is created.");
}

int begins(string pString, string pSubStr)
{
    if (pString.length < pSubStr.length)
        return -1;

    if (pSubStr.length == 0)
        return -1;

    if (pString[0..pSubStr.length] == pSubStr)
        return pSubStr.length;
    else
        return -1;

}

string[] format_date(string pText)
{
    // Returns an array of strings of YYYY,MMM,dd,hh,mm,ss
    // Input is in the form YYYY MMM dd hh:mm

    string[] lRetValue;
    int lPos;
    bool lWS;

    lRetValue.length = 6;
    lPos = 0;
    foreach(char c; pText)
    {
        if (c != ' ' && c != ':')
        {
            lRetValue[lPos] ~= c;
            lWS = true;
        }
        else
        {
            if (lWS)
            {
                lPos += 1;
                lWS = false;
            }
        }
    }

    for (int i = 3; i < 6; i++)
    {
        if (lRetValue[i].length == 1)
            lRetValue[i] = "0" ~ lRetValue[i];

    }
    return lRetValue;
}

const string[]  kSpecHdrs = ["X-EUFORUM", "Date", "From"];
const int kMessageId = 0;
const int kDate = 1;
const int kFrom = 2;

void process_file(string pFileName)
{

    BufferedFile lInputFH = new BufferedFile;
    BufferedFile lOutputFH = new BufferedFile;
    string lLine;
    int lLineCnt;
    string lOutName;
    int lPos;
    string[] lHeaders;
    bool lDoingHdrs;
    string lTemp;
    string[] lDate;
    
    writef("Processing '%s' ... ", pFileName);

    lInputFH.open(pFileName, FileMode.In);
    
    lPos = pFileName.length-1;
    // Copy the input file name rather than just reference it.
    lOutName = pFileName.dup; 
    while (lPos >= 0)
    {
        if (lOutName[lPos] == '.')
        {   // Trim off everything from the last dot.
            lOutName.length = lPos-1;
            break;
        }
        lPos--;
    }

    lOutName ~= ".mbx";
    lOutputFH.create(lOutName);

    lLineCnt = 0;
    lDoingHdrs = true;
    while (lInputFH.eof() == false)
    {
        lLine = lInputFH.readLine();
        toASCII(lLine); // Remove any non-ASCII chars (bad UTF-8)
        lLineCnt++;

        if (lLine == "-= B E G I N  =-")
        {
            lHeaders.length = kSpecHdrs.length;
            lDoingHdrs = true;
        }
        else if (lLine.length == 0)
        {
            if (lDoingHdrs)
            {
                lDoingHdrs = false;
                // Process the headers. --
                lDate = format_date(lHeaders[kDate]);

                // The 'MBX message start line'
                lPos = find(lHeaders[kFrom],"<");
                lTemp = lHeaders[kFrom][lPos+1 .. $-1];
                foreach (inout char c; lTemp)
                {
                    if (c == ' ')
                        c = '.';
                }
                lOutputFH.writefln("From %s %s %s %s:%s:00 %s", lTemp,
                                lDate[1],
                                lDate[2],
                                lDate[3],
                                lDate[4],
                                lDate[0]
                                    );

                // The message id
                lOutputFH.writefln("Message-ID: <%s>", lHeaders[kMessageId]);

                lOutputFH.writefln("From: %s", lHeaders[kFrom]);
                lOutputFH.writefln("Date: Mon, %s %s %s %s:%s:00 +0000",
                                lDate[2],
                                lDate[1],
                                lDate[0],
                                lDate[3],
                                lDate[4]
                                );

                for (int j = kSpecHdrs.length; j < lHeaders.length; j++)
                {
                    lOutputFH.writeLine(lHeaders[j]);
                }

                lOutputFH.writeLine("");

                lHeaders.length = kSpecHdrs.length;
            }
            else
            {
                lOutputFH.writeLine(lLine);
            }
        }
        else
        {
            if (lDoingHdrs)
            {
                foreach( int idx, string lHdr; kSpecHdrs)
                {
                    lPos = begins(lLine, lHdr ~ ": ");
                    if (lPos >= 0)
                    {
                        lHeaders[idx] = lLine[lPos .. $];
                        break;
                    }
                    if (idx == kSpecHdrs.length-1)
                    {
                        // Not a special header.
                        lHeaders ~= lLine;
                    }
                }
            }
            else
            {
                lOutputFH.writeLine(lLine);
            }
        }
    }

    writefln("( %d lines)", lLineCnt);
    lInputFH.close();
    lOutputFH.close();
}

int main(string[] pArgs)
{
    if (pArgs.length  < 2)
    {
        Usage(pArgs[0]);
        return 0;
    }

    foreach (int idx, string lArg; pArgs)
    {        
        if (idx > 0)
            process_file(pArgs[idx]);
    }
    
    return 0;
}

void toASCII(string pText)
{
    foreach(inout char c; pText)
    {
        if (c > 127)
            c = '?';
    }
}
// --------CODE ENDS -------------

-- 
Derek Parnell
Melbourne, Australia
irc://irc.sorcery.net:9000/euphoria