OpenEuphoria

1. libcurl - help needed

• Posted by jmduro Oct 23, 2016
• 3677 views

Forked from Re: HTTPS with OpenSSL

I wrote a libcurl 7.50 wrapper for Eu3, mostly from C source code with some routines borrowed form Ray's eulibcurl02. Source code is here: http://jean-marc.duro.pagesperso-orange.fr/libcurl_v0.0.3.zip

I can access https sites but there are some functionnalities I can't get to work:

• cookies (there is no error but the list of cookies remains empty)
• recover HTML body (as with Ray's curl_easy_perform_ex)

Help needed

Jean-Marc

3. Re: libcurl - help needed

• Posted by jmduro Oct 23, 2016
• 3835 views

Additional libraries needed are available here: http://jean-marc.duro.pagesperso-orange.fr/myLibs3_v1.5.22_2016-10-20.zip

Jean-Marc

5. Re: libcurl - help needed

• Posted by ghaberek (admin) Oct 24, 2016
• 3641 views

To recover the HTML body from the call to curl_easy_perform(), you need to establish a write function via curl_easy_setopt() that collects the data.

This example is untested, but it's how I've done this in C code that uses cURL.

include std/eumem.e 
include std/machine.e 
 
function curl_write_callback( atom ptr, atom size, atom nmemb, atom userdata ) 
 
    atom realsize = size * nmemb 
    sequence data = peek({ ptr, realsize }) 
 
    eumem:ram_space[userdata] = append( eumem:ram_space[userdata], data ) 
 
    return realsize 
end function 
 
public function curl_easy_perform_ex( atom handle ) 
 
    atom write_cb = call_back( routine_id("curl_write_callback") ) 
    curl_easy_setopt( handle, CURLOPT_WRITEFUNCTION, write_cb ) 
 
    atom userdata = eumem:malloc( "" ) 
    curl_easy_setopt( handle, CURLOPT_WRITEDATA, userdata ) 
 
    atom result = curl_easy_perform( handle ) 
    sequence content = eumem:ram_space[userdata] 
 
    eumem:free( userdata ) 
 
    return {result,content} 
end function 

-Greg

7. Re: libcurl - help needed

• Posted by jmduro Oct 24, 2016
• 3619 views

Hi Greg,

I ported your code to Eu 3

include machine.e  
  
sequence ram_space 
integer ram_free_list, free_rid 
 
ram_space = {} 
ram_free_list = 0 
 
function malloc(object mem_struct_p) 
  integer temp_ 
 
  if atom(mem_struct_p) then 
    mem_struct_p = repeat(0, mem_struct_p) 
  end if 
  if ram_free_list = 0 then 
    ram_space = append(ram_space, mem_struct_p) 
    return length(ram_space) 
  end if 
   
  temp_ = ram_free_list 
  ram_free_list = ram_space[temp_] 
  ram_space[temp_] = mem_struct_p 
 
  return temp_ 
end function 
 
procedure free(atom mem_p) 
  if mem_p < 1 then return end if 
  if mem_p > length(ram_space) then return end if 
 
  ram_space[mem_p] = ram_free_list 
  ram_free_list = floor(mem_p) 
end procedure 
free_rid = routine_id("free") 
 
function curl_write_callback( atom ptr, atom size, atom nmemb, atom userdata )  
  atom realsize 
  sequence data 
 
  realsize = size * nmemb  
  data = peek({ ptr, realsize })  
 
  ram_space[userdata] = append( ram_space[userdata], data )  
 
  return realsize  
end function  
  
global function curl_easy_perform_ex( atom handle )  
  atom write_cb, userdata, result 
  sequence content 
   
  write_cb = call_back( routine_id("curl_write_callback") )  
  void = curl_easy_setopt( handle, CURLOPT_WRITEFUNCTION, write_cb )  
  
  userdata = malloc( "" )  
  void = curl_easy_setopt( handle, CURLOPT_WRITEDATA, userdata )  
  
  result = curl_easy_perform( handle )  
  content = ram_space[userdata]  
  
  free( userdata )  
  
  return {result,content}  
end function  

I got the same crash. Then I tried to replace

write_cb = call_back( routine_id("curl_write_callback") )

by

write_cb = call_back( { '+', routine_id("curl_write_callback")} )

and it doesn't crash anymore!

Thanks

Jean-Marc

9. Re: libcurl - help needed

• Posted by CraigWelch Oct 24, 2016
• 3558 views

Back, when I was using Ray's libcurl on a daily basis, this is the code I used regarding cookies. It worked as expected.

else -- Direct 
	void = curl_easy_setopt(curl_handle, CURLOPT_PROXYTYPE, CURLPROXY_HTTP) 
	void = curl_easy_setopt(curl_handle, CURLOPT_PROXY, "") 
	void = curl_easy_setopt(curl_handle, CURLOPT_COOKIEFILE, "C:\\EUPHORIA\\greys\\Direct_cookies") 
	void = curl_easy_setopt(curl_handle, CURLOPT_COOKIEJAR, "C:\\EUPHORIA\\greys\\Direct_cookies") 
	void = curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Opera/9.21 (Windows NT 5.1; U; en)") 
end if 
void = curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYPEER, 0) 
-- CURLOPT_CAINFO and CURLOPT_CAPATH would be used to point to certificates, 
-- but the line above says not to verify the SSL certificate 

I'll be re-writing that code for re-use in the next couple of months, I'll report back.

11. Re: libcurl - help needed

• Posted by jmduro Oct 26, 2016
• 3488 views

The last example I ported from C does write the HTML page to a file but crashes on curl_easy_perform.

C:\Data\Euphoria\v3\Personnel\libcurl\examples\../curl.e:2417 in function curl_easy_perform()  
A machine-level exception occurred during execution of this statement  
    curl = 35925736 
 
... called from C:\Data\Euphoria\v3\Personnel\libcurl\examples\url2file.exw:82  

I can't understand why it crashes. Output file page.out is well written and complete.

Here is the code. Whole library is on RDS Site and here: http://jean-marc.duro.pagesperso-orange.fr/libcurl_v0.0.7.zip

-- <DESC> 
-- Download a given URL into a local file named page.out. 
-- </DESC> 
 
 
include file.e 
include get.e 
include ../curl.e 
include myLibs/myDebug.e 
include myLibs/myFile.e 
 
function curl_write_callback( atom ptr, atom size, atom nmemb, atom stream )  
  atom realsize 
  sequence data 
 
  realsize = size * nmemb  
  data = peek({ ptr, realsize })  
  puts(stream, data) 
  return realsize 
end function  
 
  sequence cmd, lNames 
  atom curl, write_cb 
  sequence res 
  integer pagefile 
 
  cmd = command_line() 
  lNames = splitFilename(cmd[2]) 
  void = chdir(lNames[1]) 
  InitialDir = current_dir() 
  f_debug = open(InitialDir & SLASH & "debug.log", "w") 
 
  void = curl_global_init(CURL_GLOBAL_ALL) 
 
  -- init the curl session 
  curl = curl_easy_init() 
  if curl then 
    void = curl_easy_setopt(curl, CURLOPT_PROXYTYPE, CURLPROXY_HTTP) 
    void = curl_easy_setopt(curl, CURLOPT_PROXY, "") 
    void = curl_easy_setopt(curl, CURLOPT_RESOLVE, "example.com") 
    void = curl_easy_setopt(curl, CURLOPT_URL, "http://example.com") 
 
    -- Switch on full protocol/debug output while testing 
    void = curl_easy_setopt(curl, CURLOPT_VERBOSE, 1) 
 
    -- disable progress meter, set to 0L to enable and disable debug output 
    void = curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1) 
 
    write_cb = call_back( routine_id("curl_write_callback") )  
    -- send all data to this function  
    void = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_cb )  
 
    -- open the file 
    pagefile = open(InitialDir & SLASH & "page.out", "wb") 
    if  (pagefile) then 
 
      -- write the page body to this file handle 
      void = curl_easy_setopt(curl, CURLOPT_WRITEDATA, pagefile) 
 
      -- get it! 
      res = curl_easy_perform(curl) 
       
      -- Check for errors 
      if (res[1] != CURLE_OK) then 
        printf(2, "curl_easy_perform() failed: %s\n", 
               {curl_easy_strerror(res[1])}) 
      else 
        analyzeObject(split(res[2][1], "\n", 0), "data", 1, 0) 
        puts(1, "curl_easy_perform() succeeded\n") 
      end if 
 
      -- close the header file 
      close(pagefile) 
    end if 
 
    -- cleanup curl stuff 
    curl_easy_cleanup(curl) 
 
  end if 
 
  curl_global_cleanup() 
  close(f_debug) 
  puts(1, "Press any key to continue.\n") 
  void = wait_key() 

Jean-Marc

13. Re: libcurl - help needed

• Posted by jmduro Oct 27, 2016
• 3438 views

Thank you Greg,

Not only mice fall in the same trap many times, aging frenchies do also! To avoid having the same problem many times more, I will use following routine in Eu3 version of std/machine.e (updated):

-- return a CDECL call_back handle for a routine 
global function cdecl_callback(integer rtn) 
  return call_back({ '+', rtn }) 
end function 

Here is the updated code which works.

-- <DESC> 
-- Download a given URL into a local file named page.out. 
-- </DESC> 
 
 
include file.e 
include get.e 
include ../curl.e 
include myLibs/myDebug.e 
include myLibs/myFile.e 
 
function curl_write_callback( atom ptr, atom size, atom nmemb, atom stream )  
  atom realsize 
  sequence data 
 
  realsize = size * nmemb  
  data = peek({ ptr, realsize })  
  puts(stream, data) 
  return realsize 
end function  
 
  sequence cmd, lNames 
  atom curl, write_cb 
  integer res 
  integer pagefile 
 
  cmd = command_line() 
  lNames = splitFilename(cmd[2]) 
  void = chdir(lNames[1]) 
  InitialDir = current_dir() 
  f_debug = open(InitialDir & SLASH & "debug.log", "w") 
 
  void = curl_global_init(CURL_GLOBAL_ALL) 
 
  -- init the curl session 
  curl = curl_easy_init() 
  if curl then 
    void = curl_easy_setopt(curl, CURLOPT_PROXYTYPE, CURLPROXY_HTTP) 
    void = curl_easy_setopt(curl, CURLOPT_PROXY, "http://proxy.rd.francetelecom.fr:8080") 
    void = curl_easy_setopt(curl, CURLOPT_RESOLVE, "example.com") 
    void = curl_easy_setopt(curl, CURLOPT_URL, "http://example.com") 
 
    -- Switch on full protocol/debug output while testing 
    void = curl_easy_setopt(curl, CURLOPT_VERBOSE, 1) 
 
    -- disable progress meter, set to 0L to enable and disable debug output 
    void = curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1) 
 
    write_cb = cdecl_callback(routine_id("curl_write_callback")) 
    -- send all data to this function  
    void = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_cb )  
 
    -- open the file 
    pagefile = open(InitialDir & SLASH & "page.out", "wb") 
    if  (pagefile) then 
 
      -- write the page body to this file handle 
      void = curl_easy_setopt(curl, CURLOPT_WRITEDATA, pagefile) 
 
      -- get it! 
      res = curl_easy_perform(curl) 
       
      -- Check for errors 
      if (res != CURLE_OK) then 
        printf(2, "curl_easy_perform() failed: %s\n", 
               {curl_easy_strerror(res)}) 
      else 
        puts(1, "curl_easy_perform() succeeded\n") 
      end if 
 
      -- close the header file 
      close(pagefile) 
    end if 
 
    -- cleanup curl stuff 
    curl_easy_cleanup(curl) 
 
  end if 
 
  curl_global_cleanup() 
  close(f_debug) 
  puts(1, "Press any key to continue.\n") 
  void = wait_key() 

I hope libcurl will be chosen as a replacement of std/socket.e as it covers a broader range of functionalities. Static libraries are also available.

Regards

Jean-Marc

15. Re: libcurl - help needed

• Posted by ghaberek (admin) Oct 27, 2016
• 3410 views

I couldn't agree more. One issue with bringing cURL into Euphoria via static library is exposing all of the C functions from the backend. I had proposed this idea to Jim a while ago:

Re: Euphoria JSON parser

I would imagine that exposing cURL would work the same way.

-Greg

17. Re: libcurl - help needed

• Posted by _tom (admin) Oct 27, 2016
• 3450 views

Googledrive files for downloading...

"curl.e" ( edited include file )

https://drive.google.com/open?id=0B4wfRhusHmUbam9yRG05eDBjWnc

"curl.zip" ( pdf documentation for include file curl.e )

https://drive.google.com/open?id=0B4wfRhusHmUbUzBNTHh3cHIzck0

I made some cosmetic changes to curl.e so that I can generate documentation that matches the style used for oE.

If you like the output, we can work to refine the documentation.

( warning: not tested so the curl.e file may be damaged.)

_tom

19. Re: libcurl - help needed

• Posted by jmduro Oct 28, 2016
• 3373 views

I get the idea for the syntax to adopt, but something sounds bizarre:

curl_version function has curl_easy_escape documentation!

Jean-Marc

21. Re: libcurl - help needed

• Posted by _tom (admin) Oct 28, 2016
• 3399 views

Added another page http://openeuphoria.org/wiki/view/standard%20doc.wc showing how a standard documented routine can be parsed into a new "look."

The idea is that more information can be parsed out of the signature and less typing is needed when creating documentation.

_tom

23. Re: libcurl - help needed

• Posted by jimcbrown (admin) Oct 29, 2016
• 3298 views

Fixed.

25. Re: libcurl - help needed

• Posted by jmduro Oct 31, 2016
• 3228 views

Thank you Greg,

I don't have the problem anymore. I can read cookies yet.

Regards

Jean-Marc

27. Re: libcurl - help needed

• Posted by jmduro Nov 10, 2016
• 3119 views

The libcurl wrapper has been uploaded: http://www.RapidEuphoria.com/uploads/libcurl2.zip

I will port it to EU4 later if someone needs it.

Jean-Marc

29. Re: libcurl - help needed

• Posted by euphoric (admin) Jan 04, 2017
• 2710 views

Has this been ported to EU4 yet?