1. Downloading messages
- Posted by Robert Craig <rds at RapidEuphoria.com> May 17, 2005
- 448 views
Some people have Internet connections where they pay by the minute, or have a monthly time limit. Those people don't like to read EUforum while connected, but in some cases they can't use Topica either, due to spam filtering or some other silly problem. It might be helpful for these people to know that all messages for the current month are stored in: http://www.listfilter.com/mlist/current.txt Right now it's around 300K. It gets updated immediately when a new message is posted. Previous months are stored as: YYYYMM.TXT e.g. April 2005 is stored as http://www.listfilter.com/mlist/200504.TXT Regards, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
2. Re: Downloading messages
- Posted by Mario Steele <eumario at trilake.net> May 18, 2005
- 420 views
Robert Craig wrote: > > Some people have Internet connections where they pay by the minute, > or have a monthly time limit. Those people don't like to read > EUforum while connected, but in some cases they can't use > Topica either, due to spam filtering or some other silly problem. > > It might be helpful for these people to know that all > messages for the current month are stored in: > > <a > href="http://www.listfilter.com/mlist/current.txt">http://www.listfilter.com/mlist/current.txt</a> > > Right now it's around 300K. It gets updated immediately > when a new message is posted. > > Previous months are stored as: > YYYYMM.TXT > > e.g. April 2005 is stored as > <a > href="http://www.listfilter.com/mlist/200504.TXT">http://www.listfilter.com/mlist/200504.TXT</a> Thanks Rob, this should make things easier for people. And to add another level of simplistic views to it, I'm currently working on a Front End GUI, with wxEuphoria, to download the Messages, and display them in a GUI Format. I've got all the base parsing, and ideals down for it. It will, as many may want, have Threading abilities in it. Meaning, that messages such as this, will come up looking like this in a Tree View: [-] Downloading Messages Robert Craig (May 17 4:47) Mario Steele (May 18 6:02) Etc Etc. Any suggestions, ideas, and what nots, are much appreciated. Also, Rob, would it be possible to setup a CGI Script that'll take the User's Name, Password, and post information all at the same time? Cause I might also incorperate a posting ability through the Web Forum into the program itself. Mario Steele http://enchantedblade.trilake.net Attaining World Dominiation, one byte at a time...
3. Re: Downloading messages
- Posted by "Juergen Luethje" <j.lue at gmx.de> May 18, 2005
- 534 views
Robert Craig wrote: > Some people have Internet connections where they pay by the minute, > or have a monthly time limit. Those people don't like to read > EUforum while connected, but in some cases they can't use > Topica either, due to spam filtering or some other silly problem. > > It might be helpful for these people to know that all > messages for the current month are stored in: > > http://www.listfilter.com/mlist/current.txt > > Right now it's around 300K. It gets updated immediately > when a new message is posted. > > Previous months are stored as: > YYYYMM.TXT > > e.g. April 2005 is stored as > http://www.listfilter.com/mlist/200504.TXT Thanks for providing this opportunity! As I expected the format of the text file is nearly mbox format <http://www.qmail.org/man/man5/mbox.html>, which is a quasi-standard. The following program converts EUforum text files to mbox files.
function rextract (sequence source, object x) for i = length(source) to 1 by -1 do if equal(source[i], x) then return source[1..i-1] end if end for return source end function constant FROM_LINE = "From support@localhost Sat Jan 01 12:00:00 2000\n", -- regarding the From_ line see note below(*) GENERAL = 0, MOZILLA = 1 object line sequence infile, outfile integer ifn, ofn, target --------------------- -- example infile = "200504.txt" target = GENERAL --------------------- outfile = rextract(infile, '.') if target = GENERAL then outfile &= ".mbx" end if ifn = open(infile, "r") ofn = open(outfile, "w") line = gets(ifn) while sequence(line) do if equal(line, "-= B E G I N =-\n") then line = FROM_LINE end if puts(ofn, line) line = gets(ifn) end while close(ifn) close(ofn)
(*) Note: I strongly recommend to use a From_ line of the form used above. Some programs use e.g. "From - Fri Jan 21 15:53:40 2005" (Mozilla Mail, Thunderbird) or just "From " or whatever. Most of these special 'ingenious' From_ lines cause problems. This is for instance the reason why mbox files generated by Mozilla Mail or Thunderbird can't be imported by The Bat! <shameless plug> People who want to migrate from Thunderbird to The Bat use my program 'mbx2eml' <http://home.arcor.de/luethje/prog/> to split the Thunderbird mbox files into separate mail files, and then they use my program 'eml2mbx' to create "good" mbox files, that can be read by The Bat. I was rather surprised when I noticed that. The "secret" of 'eml2mbx' simply is, that it uses a *clean* From_ line to denote the beginning of a new message! </shameless plug> Many e-mail clients can import mbox files, e.g. Mozilla Mail, Thunderbird, Opera M2, Pegasus Mail, Becky, The Bat, ... It's also easy to split an EUforum text file into separate mail files (popular extensions are .eml and .msg). Many e-mail clients can import these mail files, e.g. Outlook Express, Foxmail, Phoenix Mail, Becky, The Bat, ... Besides the points mentioned above by Rob, there are additional advantages of a local message database. For instance, good mail clients offer a "threaded view". Also it will provide other and often better searching capabilities than EUforum offers. E.g. using Thunderbird, we can save search results in so called "virtual folders". Using a local mail and news server such as Hamser <http://tglsoft.de/> (free!) you can do more, even convert thousands of .msg files to local newsgroups, and then read them with your favourite newsreader. We can also write our own searching program, that uses any algorthm that we want, e.g. 'soundex'. Regards, Juergen -- A: Because it considerably reduces the readability of the text. Q: Why? A: Top posting. Q: What is annoying in e-mail and news?
4. Re: Downloading messages
- Posted by Robert Craig <rds at RapidEuphoria.com> May 19, 2005
- 417 views
Mario Steele wrote: > Robert Craig wrote: > > > > Some people have Internet connections where they pay by the minute, > > or have a monthly time limit. Those people don't like to read > > EUforum while connected, but in some cases they can't use > > Topica either, due to spam filtering or some other silly problem. > > > > It might be helpful for these people to know that all > > messages for the current month are stored in: > > > > <a > > href="http://www.listfilter.com/mlist/current.txt">http://www.listfilter.com/mlist/current.txt</a> > > > > Right now it's around 300K. It gets updated immediately > > when a new message is posted. > > > > Previous months are stored as: > > YYYYMM.TXT > > > > e.g. April 2005 is stored as > > <a > > href="http://www.listfilter.com/mlist/200504.TXT">http://www.listfilter.com/mlist/200504.TXT</a> > > Thanks Rob, this should make things easier for people. And to add another > level > of simplistic views to it, I'm currently working on a Front End GUI, with > wxEuphoria, > to download the Messages, and display them in a GUI Format. I've got all the > base > parsing, and ideals down for it. It will, as many may want, have Threading > abilities > in it. > > Meaning, that messages such as this, will come up looking like this in a Tree > View: > [-] Downloading Messages > Robert Craig (May 17 4:47) > Mario Steele (May 18 6:02) > > Etc Etc. > > Any suggestions, ideas, and what nots, are much appreciated. Also, Rob, would > it be possible to setup a CGI Script that'll take the User's Name, Password, > and post information all at the same time? Cause I might also incorperate a > posting ability through the Web Forum into the program itself. I'm not sure how much work that would be. I'd have to discuss it with Junko. I guess if a lot of people wanted to use your system, I could spare some time. Thanks, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
5. Re: Downloading messages
- Posted by "Juergen Luethje" <j.lue at gmx.de> May 20, 2005
- 423 views
Mario Steele wrote: <snip> > And to add another level > of simplistic views to it, I'm currently working on a Front End GUI, with > wxEuphoria, > to download the Messages, and display them in a GUI Format. I've got all the > base > parsing, and ideals down for it. It will, as many may want, have Threading > abilities > in it. > > Meaning, that messages such as this, will come up looking like this in a Tree > View: > [-] Downloading Messages > Robert Craig (May 17 4:47) > Mario Steele (May 18 6:02) > > Etc Etc. > > Any suggestions, ideas, and what nots, are much appreciated. Also, Rob, would > it be possible to setup a CGI Script that'll take the User's Name, Password, > and post information all at the same time? Cause I might also incorperate a > posting ability through the Web Forum into the program itself. Sounds interesting. A posting ability directly from the program to the Web Forum would make the program outstanding. It looks as if you are going to invent a new technology ... Some suggestions and ideas: - Yes, threading is very nice. - The subjects of new/unread messages (in the Treeview) should be displayed in a different way than the subjects of read messages. - Right click on a subject in the Treeview could open a context menu, which offers e.g. "Reply", "Mark as read/unread, important/normal" etc. - Use a fixed width font (such as Courier New) to display the messages. Otherwise tables and e.g. the position of ^ in posted Eu error messages will not be shown correctly. - Posibility to save the currently viewed message as plain text file. - Searching the message database, maybe with the possibility to use AND and OR, and to save the search results. I'm curious about the program. Regards, Juergen
6. Re: Downloading messages
- Posted by Derek Parnell <ddparnell at bigpond.com> May 20, 2005
- 425 views
Robert Craig wrote: [snip] > all messages for the current month are stored in ... I wrote a tool to convert EUForum archive files into the .MBX format. Many news readers and email clients can import messages stored in that format.
procedure Usage(sequence pName) puts (1, "(c) 2005 Derek Parnell. Convert a EUforum archive to MBX format\n") printf(1, "Usage:- %s filename ...\n", {pName}) puts (1, " For each input file, a new file of type '.mbx' is created.\n") end procedure function begins(sequence pString, sequence pSubStr) if length(pString) < length(pSubStr) then return 0 end if if length(pSubStr) = 0 then return 0 end if if equal(pString[1..length(pSubStr)], pSubStr) then return length(pSubStr) + 1 else return 0 end if end function function format_date(sequence pText) -- Returns a sequence of YYYY,MMM,dd,hh,mm,ss -- Input is in the form YYYY MMM dd hh:mm sequence lRetValue integer lPos integer lWS lRetValue = repeat("", 6) lPos = 1 for i = 1 to length(pText) do if pText[i] != ' ' and pText[i] != ':' then lRetValue[lPos] &= pText[i] lWS = 1 else if lWS = 1 then lPos += 1 lWS = 0 end if end if end for for i = 4 to 6 do if length(lRetValue[i]) = 1 then lRetValue[i] = '0' & lRetValue[i] end if end for return lRetValue end function constant kSpecHdrs = {"X-EUFORUM", "Date", "From"}, kMessageId = 1, kDate = 2, kFrom = 3 procedure process_file(sequence pFileName) integer lInputFH integer lOutputFH object lLine integer lLineCnt sequence lOutName integer lPos sequence lHeaders integer lDoingHdrs object lTemp sequence lDate printf(1, "Processing '%s' ... ", {pFileName}) lInputFH = open(pFileName, "r") if lInputFH = -1 then printf(2, "** Unable to open input file.\n", {}) abort(1) end if lPos = length(pFileName) lOutName = pFileName while lPos > 0 do if pFileName[lPos] = '.' then lOutName = pFileName[1 .. lPos-1] exit end if lPos -= 1 end while lOutName &= ".mbx" lOutputFH = open(lOutName, "w") if lOutputFH = -1 then printf(2, "** Unable to open output file '%s'.\n", {lOutName}) abort(1) end if lLineCnt = 0 lDoingHdrs = 1 lLine = gets(lInputFH) while sequence(lLine) do lLineCnt += 1 if length(lLine) > 0 then if lLine[$] = 10 then lLine = lLine[1..$-1] end if end if if equal(lLine, "-= B E G I N =-") then lHeaders = repeat("", length(kSpecHdrs)) lDoingHdrs = 1 elsif length(lLine) = 0 then if lDoingHdrs = 1 then lDoingHdrs = 0 -- Process the headers. -- lDate = format_date(lHeaders[kDate]) -- The 'MBX message start line' lPos = find('<', lHeaders[kFrom]) lTemp = lHeaders[kFrom][lPos+1 .. $-1] for j = 1 to length(lTemp) do if lTemp[j] = ' ' then lTemp[j] = '.' end if end for printf(lOutputFH, "From %s %s %s %s:%s:00 %s\n", {lTemp, lDate[2], lDate[3], lDate[4], lDate[5], lDate[1] }) -- The message id printf(lOutputFH, "Message-ID: <%s>\n", {lHeaders[kMessageId]}) printf(lOutputFH, "From: %s\n", {lHeaders[kFrom]}) printf(lOutputFH, "Date: Mon, %s %s %s %s:%s:00 +0000\n", { lDate[3], lDate[2], lDate[1], lDate[4], lDate[5] }) for j = length(kSpecHdrs)+1 to length(lHeaders) do printf(lOutputFH, "%s\n", {lHeaders[j]}) end for puts(lOutputFH, "\n") lHeaders = repeat("", length(kSpecHdrs)) else printf(lOutputFH, "%s\n", {lLine}) end if else if lDoingHdrs = 1 then for j = 1 to length(kSpecHdrs) do lPos = begins(lLine, kSpecHdrs[j] & ": ") if lPos > 0 then lHeaders[j] = lLine[lPos .. $] exit end if if j = length(kSpecHdrs) then -- Not a special header. lHeaders = append(lHeaders, lLine) end if end for else printf(lOutputFH, "%s\n", {lLine}) end if end if lLine = gets(lInputFH) end while printf(1, "( %d lines)\n", lLineCnt) close(lInputFH) close(lOutputFH) end procedure procedure main(sequence pArgs) if length(pArgs) < 3 then if equal(pArgs[1], pArgs[2]) then Usage(pArgs[1]) else Usage("ex " & pArgs[2]) end if abort(0) end if for i = 3 to length(pArgs) do process_file(pArgs[i]) end for end procedure main(command_line())
-- Derek Parnell Melbourne, Australia irc://irc.sorcery.net:9000/euphoria
7. Re: Downloading messages
- Posted by cklester <cklester at yahoo.com> May 20, 2005
- 425 views
Derek Parnell wrote: > > I wrote a tool to convert EUForum archive files into the .MBX format. > Many news readers and email clients can import messages stored > in that format. Derek, what does that same program look like in D? (just curious; send via email if you want, but i think comparisons are always on topic...). :) -=ck "Programming in a state of EUPHORIA." http://www.cklester.com/euphoria/
8. Re: Downloading messages
- Posted by Derek Parnell <ddparnell at bigpond.com> May 20, 2005
- 424 views
- Last edited May 21, 2005
cklester wrote: > > Derek Parnell wrote: > > > > I wrote a tool to convert EUForum archive files into the .MBX format. > > Many news readers and email clients can import messages stored > > in that format. > > Derek, what does that same program look like in D? (just curious; send via > email if you want, but i think comparisons are always on topic...). :) Note: D's arrays start at zero and not 1. // --------CODE STARTS ------------- import std.stdio; import std.stream; import std.string; alias char[] string; void Usage(string pName) { writefln("(c) 2005 Derek Parnell. " "Convert a EUforum archive to MBX format"); writefln("Usage:- %s filename ...", pName); writefln(" For each input file, " "a new file of type '.mbx' is created."); } int begins(string pString, string pSubStr) { if (pString.length < pSubStr.length) return -1; if (pSubStr.length == 0) return -1; if (pString[0..pSubStr.length] == pSubStr) return pSubStr.length; else return -1; } string[] format_date(string pText) { // Returns an array of strings of YYYY,MMM,dd,hh,mm,ss // Input is in the form YYYY MMM dd hh:mm string[] lRetValue; int lPos; bool lWS; lRetValue.length = 6; lPos = 0; foreach(char c; pText) { if (c != ' ' && c != ':') { lRetValue[lPos] ~= c; lWS = true; } else { if (lWS) { lPos += 1; lWS = false; } } } for (int i = 3; i < 6; i++) { if (lRetValue[i].length == 1) lRetValue[i] = "0" ~ lRetValue[i]; } return lRetValue; } const string[] kSpecHdrs = ["X-EUFORUM", "Date", "From"]; const int kMessageId = 0; const int kDate = 1; const int kFrom = 2; void process_file(string pFileName) { BufferedFile lInputFH = new BufferedFile; BufferedFile lOutputFH = new BufferedFile; string lLine; int lLineCnt; string lOutName; int lPos; string[] lHeaders; bool lDoingHdrs; string lTemp; string[] lDate; writef("Processing '%s' ... ", pFileName); lInputFH.open(pFileName, FileMode.In); lPos = pFileName.length-1; // Copy the input file name rather than just reference it. lOutName = pFileName.dup; while (lPos >= 0) { if (lOutName[lPos] == '.') { // Trim off everything from the last dot. lOutName.length = lPos-1; break; } lPos--; } lOutName ~= ".mbx"; lOutputFH.create(lOutName); lLineCnt = 0; lDoingHdrs = true; while (lInputFH.eof() == false) { lLine = lInputFH.readLine(); toASCII(lLine); // Remove any non-ASCII chars (bad UTF-8) lLineCnt++; if (lLine == "-= B E G I N =-") { lHeaders.length = kSpecHdrs.length; lDoingHdrs = true; } else if (lLine.length == 0) { if (lDoingHdrs) { lDoingHdrs = false; // Process the headers. -- lDate = format_date(lHeaders[kDate]); // The 'MBX message start line' lPos = find(lHeaders[kFrom],"<"); lTemp = lHeaders[kFrom][lPos+1 .. $-1]; foreach (inout char c; lTemp) { if (c == ' ') c = '.'; } lOutputFH.writefln("From %s %s %s %s:%s:00 %s", lTemp, lDate[1], lDate[2], lDate[3], lDate[4], lDate[0] ); // The message id lOutputFH.writefln("Message-ID: <%s>", lHeaders[kMessageId]); lOutputFH.writefln("From: %s", lHeaders[kFrom]); lOutputFH.writefln("Date: Mon, %s %s %s %s:%s:00 +0000", lDate[2], lDate[1], lDate[0], lDate[3], lDate[4] ); for (int j = kSpecHdrs.length; j < lHeaders.length; j++) { lOutputFH.writeLine(lHeaders[j]); } lOutputFH.writeLine(""); lHeaders.length = kSpecHdrs.length; } else { lOutputFH.writeLine(lLine); } } else { if (lDoingHdrs) { foreach( int idx, string lHdr; kSpecHdrs) { lPos = begins(lLine, lHdr ~ ": "); if (lPos >= 0) { lHeaders[idx] = lLine[lPos .. $]; break; } if (idx == kSpecHdrs.length-1) { // Not a special header. lHeaders ~= lLine; } } } else { lOutputFH.writeLine(lLine); } } } writefln("( %d lines)", lLineCnt); lInputFH.close(); lOutputFH.close(); } int main(string[] pArgs) { if (pArgs.length < 2) { Usage(pArgs[0]); return 0; } foreach (int idx, string lArg; pArgs) { if (idx > 0) process_file(pArgs[idx]); } return 0; } void toASCII(string pText) { foreach(inout char c; pText) { if (c > 127) c = '?'; } } // --------CODE ENDS ------------- -- Derek Parnell Melbourne, Australia irc://irc.sorcery.net:9000/euphoria