1. Help me wrap PahoMQTT

Hi, I have some personal IOT projects, like a "smart" fridge to make homebrew beer, and decided to do an upgrade on my code, had the idea to add mqtt as the communication protocol.

I have limited knoledge of C, but decided to try to wrap pahomqtt library, documented here: https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/index.html

I'm stuck, trying to get my head around this C struct https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/struct_m_q_t_t_client__connect_options.html

C lib download at https://www.eclipse.org/downloads/download.php?file=/paho/1.4/eclipse-paho-mqtt-c-win32-1.3.0.zip or https://www.eclipse.org/downloads/download.php?file=/paho/1.4/Eclipse-Paho-MQTT-C-1.3.0-Linux.tar.gz

Can someone help me get this struct on EU code?

My wrapper so far:

include std/dll.e 
include std/machine.e 
include std/console.e 
 
atom paho_c_dll = open_dll("paho-mqtt3c.dll") 
if paho_c_dll <= 0 then 
	abort(1) 
end if 
 
atom xMQTTClient_getVersionInfo = define_c_func(paho_c_dll, "+MQTTClient_getVersionInfo",{}, C_POINTER) 
atom xMQTTClient_create = define_c_func(paho_c_dll, "+MQTTClient_create", {C_HANDLE, C_POINTER ,C_POINTER, C_INT, C_INT}, C_INT) 
atom xMQTTClient_connect = define_c_func(paho_c_dll, "+MQTTClient_connect", {C_HANDLE, C_POINTER}, C_INT) 
 
function MQTTClient_getVersionInfo() 
	atom ret = c_func(xMQTTClient_getVersionInfo, {}) 
 
	sequence ptrs = peek4u({ret,2}) 
	 
	return {peek_string(ptrs[1]),peek_string(ptrs[2])} 
end function 
 
function MQTTClient_create(sequence server_uri, sequence client_id, atom persistence_type, atom persistence_context) 
	atom hndl = allocate(4) 
	atom ptr_server_uri = allocate_string(server_uri) 
	atom ptr_client_id = allocate_string(client_id) 
 
	atom ret = c_func(xMQTTClient_create, {hndl, ptr_server_uri ,ptr_client_id ,persistence_type, persistence_context}) 
 
	free(ptr_server_uri) 
	free(ptr_client_id) 
 
	if ret = 0 then 
		return hndl 
	else 
		return ret 
	end if 
end function 
 
function MQTTClient_connect(atom hndl) 
	 
	--where madness hit me 
	atom MQTTClient_connectOptions 
	MQTTClient_connectOptions = 0 
 
	return c_func(xMQTTClient_connect, {hndl, MQTTClient_connectOptions}) 
end function 
 
 
--versioninfo works ;) 
sequence ver =  MQTTClient_getVersionInfo() 
display(ver) 

sorry about my bad english

new topic     » topic index » view message » categorize

2. Re: Help me wrap PahoMQTT

<double dip>

new topic     » goto parent     » topic index » view message » categorize

3. Re: Help me wrap PahoMQTT

xfox26 said...

Can someone help me get this struct on EU code?

First try using a persistent_type of MQTTCLIENT_PERSISTENCE_NONE(=1) and a persistent_context of NULL on the create. The hndl use is also wrong:

--atom xMQTTClient_create = define_c_func(paho_c_dll, "+MQTTClient_create", {C_HANDLE, C_POINTER ,C_POINTER, C_INT, C_POINTER}, C_INT) 
--(^changing the 5th arg to ptr is technically more accurate, but won't make much difference, ditto persistent_type of int below) 
constant MQTTCLIENT_PERSISTENCE_NONE = 1, 
         MQTTCLIENT_SUCCESS = 0 
 
function MQTTClient_create(sequence server_uri, sequence client_id) --, integer persistence_type, atom persistence_context) 
	atom hndl = allocate(4) 
	atom ptr_server_uri = allocate_string(server_uri) 
	atom ptr_client_id = allocate_string(client_id) 
 
	atom ret = c_func(xMQTTClient_create, {hndl, ptr_server_uri, ptr_client_id, MQTTCLIENT_PERSISTENCE_NONE, NULL}) 
        if ret = MQTTCLIENT_SUCCESS then 
           ret = peek4u(hndl) 
        else 
           ret = NULL 
        end if 
	free(ptr_server_uri) 
	free(ptr_client_id) 
        free(hndl) 
	return ret 
end function 

Now, for connect, in https://www.eclipse.org/paho/files/mqttdoc/MQTTClient/html/_m_q_t_t_client_8h_source.html I find

typedef struct 
 { 
         char struct_id[4]; 
         int struct_version; 
         int keepAliveInterval; 
         int cleansession; 
         int reliable; 
         MQTTClient_willOptions* will; 
         const char* username; 
         const char* password; 
         int connectTimeout; 
         int retryInterval; 
         MQTTClient_SSLOptions* ssl; 
         int serverURIcount; 
         char* const* serverURIs; 
         int MQTTVersion; 
         struct 
         { 
                 const char* serverURI;      
                 int MQTTVersion;      
                 int sessionPresent;   
         } returned; 
         struct 
         { 
                 int len;            
                 const void* data;   
         } binarypwd; 
         int maxInflightMessages; 
         /* 
          * MQTT V5 clean start flag.  Only clears state at the beginning of the session. 
          */ 
         int cleanstart; 
 } MQTTClient_connectOptions; 
  
 #define MQTTClient_connectOptions_initializer { {'M', 'Q', 'T', 'C'}, 6, 60, 1, 1, NULL, NULL, NULL, 30, 0, NULL, 0, NULL, MQTTVERSION_DEFAULT, {NULL, 0, 0}, {0, NULL}, -1, 0} 
  
 #define MQTTClient_connectOptions_initializer5 { {'M', 'Q', 'T', 'C'}, 6, 60, 0, 1, NULL, NULL, NULL, 30, 0, NULL, 0, NULL, MQTTVERSION_5, {NULL, 0, 0}, {0, NULL}, -1, 1} 
which is what you should really be looking at. It'll be different if you're runing 64 bit, but I'm going to assume you are using 32 bit.
I count 21 4-byte fields, which you need to set as per one of the last two lines, so it goes something like this (untested!):

atom pConnectOptions = allocate(4*21) 
poke(pConnectOptions+0,"MQTC") 
poke4(pConnectOptions+4,6) 
poke4(pConnectOptions+8,60) 
... 
poke4(pConnectOptions+4*19,-1) 
poke4(pConnectOptions+4*20,1) 

Good luck!

PS: using Phix's cffi might make things somewhat easier, but it would not surprise me if it baulked at the nested "returned" and "binarypwd" structs, if so I'd just delete the containers leaving just the inner fields.

new topic     » goto parent     » topic index » view message » categorize

4. Re: Help me wrap PahoMQTT

A ZeroMQ wrapper already exists: https://archive.usingeuphoria.com/libzmq.zip

Maybe this could be a good base.

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

5. Re: Help me wrap PahoMQTT

Thank you very much With your help got the connect function to work. I will wrap some more basic functions.

When it's usable where is the best place to upload? RapidEuphoria? Github? Bitbucket?

new topic     » goto parent     » topic index » view message » categorize

6. Re: Help me wrap PahoMQTT

Hi

IMHO

http://phix.x10.mx/pmwiki/pmwiki.php?n=Main.HomePage

You have to treat like a wiki entry, and follow the instructions / manual to do it, but it's doable.

Cheers

Chris

new topic     » goto parent     » topic index » view message » categorize

7. Re: Help me wrap PahoMQTT

xfox26 said...

When it's usable where is the best place to upload? RapidEuphoria? Github? Bitbucket?

The archive uploads on the RapidEuphoria site has been broken for a while now. That being said, I think we (the Internet) have moved past "upload your zip file here" toward better community participation in projects. This is what GitHub, GitLab, BitBucket etc. offer: a place to post your living code and its release files for those interested to see, download, and participate in. If we want Euphoria to be relevant in the modern world, we need to go where the people are, and I believe that's GitHub. The more active Euphoria projects others see on GitHub, the more likely we are to catch the attention of more developers. The more developers we attract to the language, the more successful we'll be in the long-term.

Thank you for coming to my TED talk.

-Greg

new topic     » goto parent     » topic index » view message » categorize

8. Re: Help me wrap PahoMQTT

Uploaded code to GitHub at https://github.com/xfox26/paho.mqtt.euphoria

Fell free criticize my work.
Any help is appreciated.

new topic     » goto parent     » topic index » view message » categorize

9. Re: Help me wrap PahoMQTT

Agreed about GitHub. I just wish GitHub was easier to use and more versatile.

I went there this morning and set up a EuGTK page: https://github.com/irvm/EuGTK

Took quite a while to figure out the arcane markup tricks, and when done, it looks rather plain.

new topic     » goto parent     » topic index » view message » categorize

10. Re: Help me wrap PahoMQTT

irv said...

I went there this morning and set up a EuGTK page: https://github.com/irvm/EuGTK

Took quite a while to figure out the arcane markup tricks, and when done, it looks rather plain.

Nah! Looks great, irv! Welcome to the 21st century. grin

GitHub is just a code repo. It's not meant to be fancy. You can always link back to the EuGTK home page from there to show off your HTML skillz.

new topic     » goto parent     » topic index » view message » categorize

11. Re: Help me wrap PahoMQTT

xfox26 said...

Uploaded code to GitHub at https://github.com/xfox26/paho.mqtt.euphoria

Fell free criticize my work.
Any help is appreciated.

Couple of quick suggestions, change

public function MQTTClient_create(sequence server_uri, sequence client_id, atom persistence_type, atom persistence_context) 
... 
	atom ret = c_func(xMQTTClient_create, {hndl, ptr_server_uri, ptr_client_id, MQTTCLIENT_PERSISTENCE_NONE, NULL})  

to

public function MQTTClient_create(sequence server_uri, sequence client_id, atom persistence_type = MQTTCLIENT_PERSISTENCE_NONE, atom persistence_context = NULL) 
... 
	atom ret = c_func(xMQTTClient_create, {hndl, ptr_server_uri, ptr_client_id, persistence_type, persistence_context})  

and

poke4(MQTTClient_connectOptions+4*1, options[1])--sctruct version 
... 
poke4(MQTTClient_connectOptions+4*20, options[20])--cleanstart 

to

global constant CO_STRUCT_VERSION = 1, 
... 
                CO_CLEANSTART = 20 
poke4(MQTTClient_connectOptions+4, options) -- remaining 20 in one poke 

You might also want to suggest something like the following usage pattern in the docs:

  sequence options = default_connectOptions 
  options[CO_RETRY_INTERVAL] = 120 
  atom res = MQTTClient_connect(hndl, options) 

Pete

new topic     » goto parent     » topic index » view message » categorize

12. Re: Help me wrap PahoMQTT

petelomax said...

Couple of quick suggestions, change

Updated code with your suggestions, thanks!

new topic     » goto parent     » topic index » view message » categorize

13. Re: Help me wrap PahoMQTT

I have added some more functions, the wrapper can send and receive messages.

But got some questions, is this implementation correct?

C code:

int messageArrived(void *context, char *topicName, int topicLen, MQTTAsync_message *message) 
{ 
	//do things... 
 
	MQTTAsync_freeMessage(&message); 
	MQTTAsync_free(topicName); 
	return 1; 
} 

EU code:

function default_messageArrived_dispacher(atom ptr_context, atom ptr_topicName, atom topicLen, atom ptr_client_message) 
	 
	--do things... 
 
	MQTTClient_freeMessage(ptr_client_message) 
	MQTTClient_free(ptr_topicName) 
 
	return call_func(user_messageArrived_callback, {topic, message, context, raw_message}) 
end function 
 
public procedure MQTTClient_freeMessage(atom ptr_msg) 
	atom ptr_struct = allocate(4) 
	poke4(ptr_struct, ptr_msg) 
	c_proc(xMQTTClient_freeMessage, {ptr_struct}) 
	free(ptr_struct) 
end procedure 

I'm asking about the MQTTClient_freeMessage part, where I do allocate, poke the pointer and call the function passing a pointer to the pointer... feels so wrong...

And, there is some simple way to test if there are a memory leak somewhere?

new topic     » goto parent     » topic index » view message » categorize

14. Re: Help me wrap PahoMQTT

xfox26 said...

I'm asking about the MQTTClient_freeMessage part, where I do allocate, poke the pointer and call the function passing a pointer to the pointer... feels so wrong...

And, there is some simple way to test if there are a memory leak somewhere?

Given that (in C) void MQTTClient_freeMessage(MQTTClient_message** msg) means it is asking for a pointer to a pointer to msg, looks about right to me.
Apart from running taskmanager and seeing whether memory use just keeps on growing, if MQTTClinet_freeMessage(NULL) crashes then it is pretty likely anything non-crashing has got it right.

new topic     » goto parent     » topic index » view message » categorize

15. Re: Help me wrap PahoMQTT

I'm close to call it a "release version", got almost everything working on win32, linux32, and ARM (tested on my Raspberry Pi).

But now I'm lost at making it x64 compatble. Can someone give me some directions on making it x64 compatible?

Code on https://github.com/xfox26/paho.mqtt.euphoria

new topic     » goto parent     » topic index » view message » categorize

16. Re: Help me wrap PahoMQTT

xfox26 said...

Can someone give me some directions on making it x64 compatible?

Don't do this:

atom hndl = allocate(4) 
atom ret = peek4u(hndl) 

Do this instead:

atom hndl = allocate_data( sizeof(C_POINTER) ) 
atom ret = peek_pointer( hndl ) 

Define structures like this:

ifdef BITS64 then 
 
constant 
    MQTTClient_willOptions__struct_id       =  0, -- char[4] 
    MQTTClient_willOptions__struct_version  =  4, -- int 
    MQTTClient_willOptions__topicName       =  8, -- const char* 
    MQTTClient_willOptions__message         = 16, -- const char* 
    MQTTClient_willOptions__retained        = 24, -- int 
    MQTTClient_willOptions__qos             = 28, -- int 
    MQTTClient_willOptions__payload_len     = 32, -- int 
    MQTTClient_willOptions__payload_data    = 40, -- const void* 
    SIZEOF_MQTTClient_willOptions           = 48, 
$ 
 
elsedef 
 
constant 
    MQTTClient_willOptions__struct_id       =  0, -- char[4] 
    MQTTClient_willOptions__struct_version  =  4, -- int 
    MQTTClient_willOptions__topicName       =  8, -- const char* 
    MQTTClient_willOptions__message         = 12, -- const char* 
    MQTTClient_willOptions__retained        = 16, -- int 
    MQTTClient_willOptions__qos             = 20, -- int 
    MQTTClient_willOptions__payload_len     = 24, -- int 
    MQTTClient_willOptions__payload_data    = 28, -- const void* 
    SIZEOF_MQTTClient_willOptions           = 32, 
$ 
 
end ifdef 

And if you need to use them frequently, create wrapper functions to peek/poke your structure members safely:

-- peek a string pointer and return the string or "" if the ptr is NULL 
function peek_str( atom ptr ) 
     
    ptr = peek_pointer( ptr ) 
     
    if ptr = NULL then 
        return "" 
    end if 
     
    return peek_string( ptr ) 
end function 
 
function peek_MQTTClient_willOptions( atom ptr ) 
     
    sequence struct_id  =    peek({ ptr + MQTTClient_willOptions__struct_id, 4 }) 
    atom struct_version =   peek4s( ptr + MQTTClient_willOptions__struct_version ) 
    sequence topicName  = peek_str( ptr + MQTTClient_willOptions__topicName ) 
    sequence message    = peek_str( ptr + MQTTClient_willOptions__message ) 
    atom retained       =   peek4s( ptr + MQTTClient_willOptions__retained ) 
    atom qos            =   peek4s( ptr + MQTTClient_willOptions__qos ) 
    atom payload_len    =   peek4s( ptr + MQTTClient_willOptions__payload_len ) 
    object payload_data =   peek4s( ptr + MQTTClient_willOptions__payload_data ) 
     
    if payload_len != 0 and payload_data != NULL then 
        -- actually peek the data if it's present 
        payload_data = peek({ payload_data, payload_len }) 
    end if 
     
    return { 
        struct_id, 
        struct_version, 
        topicName, 
        message, 
        retained, 
        qos, 
        payload_len, 
        payload_data 
    } 
end function 

Tips and tricks:

  • Invert your current ifdef's; use ifdef BITS64 and then else for 32-bit.
  • Use sizeof() to determine the size of C types.
  • Use allocate_data() unless you explicitly need executable memory.
  • Ensure the correct peek/poke variants for your structures.
  • Define separate 32/64 bit structures where necessary.

-Greg

new topic     » goto parent     » topic index » view message » categorize

17. Re: Help me wrap PahoMQTT

What I do to get structure alignments reliably, is construct simple C programs to output everything for me.

Makefile

CFLAGS = -Ieclipse/paho.mqtt.c/src 
 
all : test64.exe test32.exe 
    @test64.exe 
    @echo. 
    @test32.exe 
 
test64.exe : test.c 
    gcc -m64 -DBITS=64 -o $@ $< $(CFLAGS) 
 
test32.exe : test.c 
    gcc -m32 -DBITS=32 -o $@ $< $(CFLAGS) 

test.c

#include <stdio.h> 
#include <stddef.h> 
 
#include "MQTTClient.h" 
 
#if ( BITS == 64 ) 
#define ARCH_TYPE() printf( "-- 64-bit ------------------------------------\n" ) 
#else 
#define ARCH_TYPE() printf( "-- 32-bit ------------------------------------\n" ) 
#endif 
 
#define OFFSETOF( st, mb ) printf( "%-40s = %2d,\n", #st "__" #mb, offsetof(st,mb) ) 
#define SIZEOF( st ) printf( "SIZEOF_%-33s = %2d,\n", #st, sizeof(st) ) 
 
int main( int argc, char* argv[] ) 
{ 
    ARCH_TYPE(); 
     
    OFFSETOF( MQTTClient_willOptions, struct_id ); 
    OFFSETOF( MQTTClient_willOptions, struct_version ); 
    OFFSETOF( MQTTClient_willOptions, topicName ); 
    OFFSETOF( MQTTClient_willOptions, message ); 
    OFFSETOF( MQTTClient_willOptions, retained ); 
    OFFSETOF( MQTTClient_willOptions, qos ); 
    OFFSETOF( MQTTClient_willOptions, payload.len ); 
    OFFSETOF( MQTTClient_willOptions, payload.data ); 
    SIZEOF( MQTTClient_willOptions ); 
     
    return 0; 
} 

Output

C:\>mingw32-make 
-- 64-bit ------------------------------------ 
MQTTClient_willOptions__struct_id        =  0, 
MQTTClient_willOptions__struct_version   =  4, 
MQTTClient_willOptions__topicName        =  8, 
MQTTClient_willOptions__message          = 16, 
MQTTClient_willOptions__retained         = 24, 
MQTTClient_willOptions__qos              = 28, 
MQTTClient_willOptions__payload.len      = 32, 
MQTTClient_willOptions__payload.data     = 40, 
SIZEOF_MQTTClient_willOptions            = 48, 
 
-- 32-bit ------------------------------------ 
MQTTClient_willOptions__struct_id        =  0, 
MQTTClient_willOptions__struct_version   =  4, 
MQTTClient_willOptions__topicName        =  8, 
MQTTClient_willOptions__message          = 12, 
MQTTClient_willOptions__retained         = 16, 
MQTTClient_willOptions__qos              = 20, 
MQTTClient_willOptions__payload.len      = 24, 
MQTTClient_willOptions__payload.data     = 28, 
SIZEOF_MQTTClient_willOptions            = 32, 

Now I can copy/paste that output and format in my wrapper as I showed above.

-Greg

new topic     » goto parent     » topic index » view message » categorize

18. Re: Help me wrap PahoMQTT

ghaberek said...

What I do to get structure alignments reliably, is construct simple C programs to output everything for me.

Helped A LOT! Thank you very much!

I've just uploaded to Github the x64 compatible code. (a bit ugly)

Now I will focus on documentation.

new topic     » goto parent     » topic index » view message » categorize

Search



Quick Links

User menu

Not signed in.

Misc Menu