1. Experimental "peer to peer" modular programming concept

Working on large GUI applications over the years, i have always been frustrated with how difficult it can be to organize all the parts of an application, especially since there is usually special data processing mixed with user interface code and various states or views the application may switch to. Usually, i end up with a small main file and 2 or 3 huge files, with GUI and data processing code mixed together, and a few lower-level libraries that perform specific generic functions. It is difficult to define clear "modules" or some sort of hierarchy when different parts of the program need to communicate with other parts, so i end up with a make-shift message queue system that is difficult to maintain across different parts of 2 or 3 large files. Part of the problem is the way include files work assumes that an included file is somehow lower in the hierarchy, and there is no clear way to make different parts of the program "peers" or independent "modules" without some other library to include them and control them.

Does anyone else have this problem?

Possible solution: I have created an experimental "sync.e" library that allows any include file in a program to send messages without being aware of where they get received. Each file has to include "sync.e and subscribe to a "sync group", specifying a routine_id that will process sync messages. If any subscriber of the group calls sync:send(), all other subscribers' handler routines will be called by the sync.e library. Each library can then process incoming messages without a queue or tasks required.

No processing queue or controlling hierarchy is required. A main file can include many small .e files and not have to do anything with them once everything is initialized. Each .e file can be independent of the others, handling it's little part of the whole program. For example, if i create a file browser application, separate .e files could handle their own parts of a GUI: a folder tree, a location bar, a file list, etc. If any part of the gui is changed by the user, a message is sent out, and the other files process what they need to and update their part of the gui. Another file that is responsible for manipulating files could receive a message to do something with a specified file (open, delete, copy, move, rename, etc.) This makes the program very modular and easier to add or remove features later.

This is still experimental and theoretical, but i am currently rewriting all the source code of RedyCode to test this concept.

Here's the experimental sync.e library. Any thoughts? Let's discuss.

sequence 
gNames = {}, 
gSubscriberNames = {}, 
gSubscriberRids = {} 
 
atom debugRid = 0 
 
 
public procedure send(sequence groupname, sequence subscribername, sequence msgname, object msgdata) 
--sync data with other subscribers in group 
    atom idx = find(groupname, gNames) 
    if idx > 0 then 
        for s = 1 to length(gSubscriberNames[idx]) do 
            if not equal(gSubscriberNames[idx][s], subscribername) then 
                if gSubscriberRids[idx][s] > 0 then 
                    call_proc(gSubscriberRids[idx][s], {groupname, subscribername, msgname, msgdata}) 
                end if 
            end if  
        end for 
    end if 
    if debugRid > 0 then 
        call_proc(debugRid, {groupname, subscribername, msgname, msgdata}) 
    end if 
end procedure 
 
 
public procedure subscribe(sequence groupname, sequence subscribername, atom msghandlerid) 
--subscribe to a sync group to receive sync updates 
    atom idx = find(groupname, gNames) 
    if idx > 0 then 
        gSubscriberNames[idx] &= {subscribername} 
        gSubscriberRids[idx] &= {msghandlerid} 
    else 
        gNames &= {groupname} 
        gSubscriberNames &= {{subscribername}} 
        gSubscriberRids &= {{msghandlerid}} 
    end if 
end procedure 
 
 
public procedure unsubscribe(sequence groupname, sequence subscribername) 
--unsubscribe from a sync group 
    atom idx = find(groupname, gNames) 
    if idx > 0 then 
        gNames = remove(gNames, idx) 
        gSubscriberNames = remove(gSubscriberNames, idx) 
        gSubscriberRids = remove(gSubscriberRids, idx) 
    end if 
end procedure 
 
 
public function list_groups() 
--list all sync groups 
    return gNames 
end function 
 
 
public function list_subscribers(sequence groupname) 
--list all subscribers in a specified sync group 
    atom idx = find(groupname, gNames) 
    if idx > 0 then 
        return gSubscriberNames[idx] 
    else 
        return 0 
    end if 
end function 
 
 
public procedure debug(atom debughandlerid) 
--register a routine to be called for debug messages 
    debugRid = debughandlerid 
end procedure 
new topic     » topic index » view message » categorize

2. Re: Experimental "peer to peer" modular programming concept

What you're describing is a publish-subscribe pattern, not a peer-to-peer model, mostly because this is a one-to-many relationship, and peer-to-peer is by definition one-to-one.

I suggest renaming your "send" function to "publish" or "post", because that is the conceptual opposite of "subscribe".

I've played around with this concept in ZeroMQ, which is pretty nifty. It provides a "PUB/SUB" (publish/subscribe) pattern, among others.

-Greg

new topic     » goto parent     » topic index » view message » categorize

3. Re: Experimental "peer to peer" modular programming concept

ghaberek said...

What you're describing is a publish-subscribe pattern, not a peer-to-peer model, mostly because this is a one-to-many relationship, and peer-to-peer is by definition one-to-one.

Ah, i see. Interesting.

ghaberek said...

I suggest renaming your "send" function to "publish" or "post", because that is the conceptual opposite of "subscribe".

I've played around with this concept in ZeroMQ, which is pretty nifty. It provides a "PUB/SUB" (publish/subscribe) pattern, among others.

-Greg

I like the "publish" terminology better. Thanks.

new topic     » goto parent     » topic index » view message » categorize

4. Re: Experimental "peer to peer" modular programming concept

ryanj said...

Part of the problem is the way include files work assumes that an included file is somehow lower in the hierarchy, and there is no clear way to make different parts of the program "peers" or independent "modules" without some other library to include them and control them.

Does anyone else have this problem?

Certainly.

Whenever too much code is in the same level in the hierarchy, the program is hard to maintain or understand.

A simple way to deal with it is by splitting the code into many include files and placing groups of files in different directories, and by adding documentation that explains what is going on; but it's fake and confusing.

It's hard to avoid some kind of pattern in this case, since the code must remain maintainable.

new topic     » goto parent     » topic index » view message » categorize

5. Re: Experimental "peer to peer" modular programming concept

This is a situation where a global variable of the struct-type will solve your problems.
By sending the result of any code producing results or part of them
to that struct and any code needing part of that result to retrieve it from.

new topic     » goto parent     » topic index » view message » categorize

6. Re: Experimental "peer to peer" modular programming concept

Ekhnat0n said...

This is a situation where a global variable of the struct-type will solve your problems.
By sending the result of any code producing results or part of them
to that struct and any code needing part of that result to retrieve it from.

I thought about using this method. In fact, i do use it on Propeller micro-controllers. But it requires each part of the program to actively monitor the global variables for changes. The subscribe/publish method is automatic. Publishing data causes all subscribers' handler routines to be called immediately. Of course, i have to be careful not to do anything that can block or take a long time to process (treat them the same way as GUI event handlers, basically.) If necessary, a handler routine could store the message data in a local variable, which can then be processed by a separate task.

new topic     » goto parent     » topic index » view message » categorize

7. Re: Experimental "peer to peer" modular programming concept

I don't know RedyCode specifically, but in general, it's better to do the last heroic effort to move significant part of the code into generic library. Many times there is a way, although it's not obvious at all.

I wouldn't use a pattern (or tasks) unless I'm totally convinced that there is absolutely no way to create generic libraries for processing data, etc. In case that there is a way - a pattern might be not suitable.

There is nothing more modular then generic code, so it's worth a true effort.

new topic     » goto parent     » topic index » view message » categorize

8. Re: Experimental "peer to peer" modular programming concept

ryanj said...
Ekhnat0n said...

This is a situation where a global variable of the struct-type will solve your problems.
By sending the result of any code producing results or part of them
to that struct and any code needing part of that result to retrieve it from.

I thought about using this method. In fact, i do use it on Propeller micro-controllers. But it requires each part of the program to actively monitor the global variables for changes. The subscribe/publish method is automatic. Publishing data causes all subscribers' handler routines to be called immediately. Of course, i have to be careful not to do anything that can block or take a long time to process (treat them the same way as GUI event handlers, basically.) If necessary, a handler routine could store the message data in a local variable, which can then be processed by a separate task.

Is the resulting published message the same format as standard message passing in the GUI? (I guess I could've checked the code myself.. just lazy.. ). If it were the same would it be possible/desirable to merge the functionality? ie, inject the messages into the usual message stream. Just a thought.

Spock

new topic     » goto parent     » topic index » view message » categorize

9. Re: Experimental "peer to peer" modular programming concept

Spock said...

Is the resulting published message the same format as standard message passing in the GUI? (I guess I could've checked the code myself.. just lazy.. ). If it were the same would it be possible/desirable to merge the functionality? ie, inject the messages into the usual message stream. Just a thought.

Spock

It is similar to GUI event handling, but more generic. Any part of the program can publish or subscribe to a topic. It is up to the programmer to define topics, message names, and message data formats, which should be listed in comments somewhere. I suppose this encourages good documentation and organization.

public procedure publish(sequence topicname, sequence subscribername, sequence msgname, object msgdata)  
--send data to other subscribers  to a topic by calling each subscriber's message handler proc 
    atom idx = find(topicname, gNames)  
    if idx > 0 then  
        for s = 1 to length(gSubscriberNames[idx]) do  
            if not equal(gSubscriberNames[idx][s], subscribername) then  
                if gSubscriberRids[idx][s] > 0 then  
                    call_proc(gSubscriberRids[idx][s], {topicname, subscribername, msgname, msgdata})  
                end if  
            end if   
        end for  
    end if  
    if debugRid > 0 then  
        call_proc(debugRid, {topicname, subscribername, msgname, msgdata})  
    end if  
end procedure 
new topic     » goto parent     » topic index » view message » categorize

10. Re: Experimental "peer to peer" modular programming concept

Shian_Lee said...

I don't know RedyCode specifically, but in general, it's better to do the last heroic effort to move significant part of the code into generic library. Many times there is a way, although it's not obvious at all.

I wouldn't use a pattern (or tasks) unless I'm totally convinced that there is absolutely no way to create generic libraries for processing data, etc. In case that there is a way - a pattern might be not suitable.

There is nothing more modular then generic code, so it's worth a true effort.

I find that large programs have lots of GUI code mixed with data processing, and i have a difficult time making it modular, because there is always a higher layer that has to control all the modules by calling public/export routines. That higher layer grows and grows until it is 1000-2000 lines of code in a single file and i get so lost, i can't even finish adding all my planned features. This publish-subscribe design doesn't require a higher level to control modules, but instead a lower-level library that all modules include and control. Then, many small include files can handle specific parts of the GUI and/or data processing of a large program.

new topic     » goto parent     » topic index » view message » categorize

11. Re: Experimental "peer to peer" modular programming concept

ryanj said...

I find that large programs have lots of GUI code mixed with data processing, and i have a difficult time making it modular, because there is always a higher layer that has to control all the modules by calling public/export routines. That higher layer grows and grows until it is 1000-2000 lines of code in a single file and i get so lost, i can't even finish adding all my planned features. This publish-subscribe design doesn't require a higher level to control modules, but instead a lower-level library that all modules include and control. Then, many small include files can handle specific parts of the GUI and/or data processing of a large program.

Your situation (work + GUI) is probably similar to mine. Yet what I end up with in my programs is lots of include files each handling some low-level aspect. The problem is then compounded by having *additional* include files to manage the behaviour of groups of those files. The reason for this difference could be that I tend to refactor progams in order to compartmentalise unique/logical functionalities, eg, all vessels are listed in just one module, all customers are managed similarly in another module, all invoices are handled in another, etc.. the job of these modules is just to respond appropriately when interrogated by others. On top of that there are specific modules to link the GUI + behaviour aspects of the program.

I had started doing this refactoring with the GUI but got a bit sidetracked. The result would (will) have been having a separate module manage each control class. Does Redy have this structure already? I ain't looked recently.. One prime candidate for GUI refactoring is the list of constants. ATM I have a single list of many constants but very often a routine might only need access to a very specific group. I'd prefer to locate those constants inside the calling routine. But if those constants are ever needed elsewhere I have a problem. The solution could be a change to the language that might allow this:

function myfunc() 
   include constants.ew : mysmallgroup 
   .. 
end function 

In this theoretical case only the constants declared in mysmallgroup (which is a just a small section inside constants.ew) would be visible to the routine. Admittedly it is a bit revolutionary for Euphoria but it's the direction I would like to take a language.

The uni-directional behaviour of include could also be improved but mutually including modules are already possible in Euphoria (to my knowledge). What makes this a bit dangerous is how module initialising is done when there are dependencies. These days I tend to have a specific init() routine in each module that is called from a higher level after the GUI entry point.

Your idea of using publish-subscribe would certainly be useful in my programs, eg, when an event affecting many modules only needs to be transmitted as a message once rather than many times. Other than that I'm not sure it would necessarily improve other things. In general, I structure each module so that all handler routines are clustered at the end of the file and their job is solely to interpret the Windows messages as behaviours, ie, there is little or no actual work done by the message handlers - they simply pass that on to the main program.

Spock

new topic     » goto parent     » topic index » view message » categorize

Search



Quick Links

User menu

Not signed in.

Misc Menu