Download HTTP file to memory buffer

[Quin]

A version of DOWNLOAD that takes a data pointer and a size in order to write the data to a particular location in memory as opposed to the hard drive would be incredibly useful. Removes the dependency on the disk for calling out to REST APIs, for example.

[MrBcx]

Quin,

Here's an earlier thread on the same subject. 

https://bcxbasiccoders.com/smf/index.php?topic=225.msg4009#msg4009

Armando's work is small enough to be kept in your snippets or stored in a .bi file
but I'm certainly open to adding a DownloadToString$() function to the BCX vocabulary,
if people will actually use it.

[Quin]

Nifty little function there!

I can't speak for anyone else, but I personally would use it if it was in the standard vocabulary.

[MrBcx]

Nifty little function there!

I can't speak for anyone else, but I personally would use it if it was in the standard vocabulary.

After much messing about, I've decided my original DownloadToStr() function is hard to beat,
for reliability, speed, and being easy to understand.

Here is my new demo that uses the "War and Peace" url posted by Armando.

Code Select Expand



MACRO WarAndPeace = "https://raw.githubusercontent.com/mmcky/nyu-econ-370/master/notebooks/data/book-war-and-peace.txt"

DIM AS DWORD RemoteFileSize
RemoteFileSize = GetHttpFileSize(WarAndPeace)   

IF RemoteFileSize THEN
    DIM TheBook$ * RemoteFileSize
    PRINT "Size of 'War and Peace' on server = ", USING$("#", RemoteFileSize), " bytes"    ' ~3.2MB File
    PRINT "Downloading into string variable now ... please wait ..."
    TheBook$ = DownloadToStr(WarAndPeace)
    BCX_MDIALOG (Demo, "'War And Peace'", NULL, 0, 0, 300, 300, 0, 0, "Arial", 12)
    END
ELSE
    PRINT "WarAndPeace URL was not found."
    PAUSE
    END
END IF




BEGIN MODAL DIALOG AS Demo
    SELECT CASE CBMSG
        '******************
    CASE WM_INITDIALOG
        '******************
        STATIC hRichEdit AS HWND
        hRichEdit = BCX_RICHEDIT (0, CBHWND, 12345, 10, 10, 280, 280)
        SETWINDOWRTFTEXT(hRichEdit, TheBook$)
        CENTER (CBHWND)
        FUNCTION = TRUE
        '******************
    CASE WM_COMMAND
        '******************
        SELECT CASE CBCTL
            '******************
        CASE IDOK
            '******************
            CLOSEDIALOG
            '******************
        CASE IDCANCEL
            '******************
            CLOSEDIALOG
        END SELECT
    END SELECT
END DIALOG



FUNCTION DownloadToStr$ (Url$)            ' New updated function by MrBcx - July 24, 2024
    LOCAL Fname$
    Fname$ = TEMPFILENAME$(TEMPDIR$, "any")
    DOWNLOAD (Url$, Fname$)
    DIM Buffer$ * LOF(Fname$) + 1
    IF EXIST(Fname$) THEN
        Buffer$ = LOADFILE$(Fname$)
        KILL Fname$
        FUNCTION = Buffer$
    ELSE
        FUNCTION = ""
    END IF
END FUNCTION



FUNCTION GetHttpFileSize(url$) AS DWORD   ' New updated function by MrBcx - July 24, 2024
    DIM AS HINTERNET hInternet
    DIM AS HINTERNET hConnect
    DIM AS DWORD fileSize
    DIM AS DWORD bufferSize
    DIM AS DWORD statusCode
    DIM AS DWORD statusSize

    hInternet = InternetOpen("xcb", INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, 0)
    IF hInternet = NULL THEN
        FUNCTION = 0
    END IF

    hConnect = InternetOpenUrl(hInternet, url$, NULL, 0, INTERNET_FLAG_RELOAD, 0)
    IF hConnect = NULL THEN
        InternetCloseHandle(hInternet)
        FUNCTION = 0
    END IF

    statusSize = SIZEOF(DWORD)    ' Check the HTTP status code
    IF NOT HttpQueryInfo(hConnect, HTTP_QUERY_STATUS_CODE BOR HTTP_QUERY_FLAG_NUMBER, &statusCode, &statusSize, NULL) THEN
        InternetCloseHandle(hConnect)
        InternetCloseHandle(hInternet)
        FUNCTION = 0
    END IF

    IF statusCode <> 200 THEN     ' If the status code is not 200 (OK), treat it as file not found
        InternetCloseHandle(hConnect)
        InternetCloseHandle(hInternet)
        FUNCTION = 0
    END IF

    bufferSize = SIZEOF(DWORD)

    IF NOT HttpQueryInfo(hConnect, HTTP_QUERY_CONTENT_LENGTH BOR HTTP_QUERY_FLAG_NUMBER, &fileSize, &bufferSize, NULL) THEN
        fileSize = 0
    END IF

    InternetCloseHandle(hConnect)
    InternetCloseHandle(hInternet)

    FUNCTION = fileSize
END FUNCTION




[MrBcx]

The code in my previous post has been updated.

1)  GetHttpFileSize() now returns 0 when the HttpQueryInfo status <> 200  "OK"
      Important for when your URL's are misspelled or no longer exist (i.e. error 404)

2)  DownloadToStr() now verifies that web content was actually received locally.

3)  Improved the layout of the demonstration

[jbk]

thanks MrBcx 
this looks very useful 👍