Author Topic: Large File Support  (Read 94 times)

Robert

  • Sr. Member
  • ****
  • Posts: 383
    • View Profile
Large File Support
« on: July 30, 2020, 09:30:34 PM »
Any interest in Large File Support for BCX's LOF and any other file functions that should be promoted to 64 bit?

MrBcx

  • Administrator
  • Sr. Member
  • *****
  • Posts: 434
    • View Profile
Re: Large File Support
« Reply #1 on: July 30, 2020, 11:01:29 PM »
Robert -- I'm looking forward to the discussion. 


I shared these earlier this year ... somewhat related to the topic.

https://bcxbasiccoders.com/smf/index.php?topic=97.0


   
« Last Edit: July 30, 2020, 11:18:04 PM by MrBcx »

Robert

  • Sr. Member
  • ****
  • Posts: 383
    • View Profile
Re: Large File Support
« Reply #2 on: July 31, 2020, 03:49:16 AM »
Robert -- I'm looking forward to the discussion. 


I shared these earlier this year ... somewhat related to the topic.

https://bcxbasiccoders.com/smf/index.php?topic=97.0


Hi Mr BCX:

Thank you for the reminder about the disk capacity example.

I have been on a size_t kick lately and that example gave me another venue to try out size_t by using it to replace the __int64 datatypes.

32 bit compiled but didn't have enough fingers to make the full count.

Here's a 64 bit LOF for Microsoft and Nuwen.

Clang should work too, but at the moment I do not have access.

Code: [Select]

DIM InFile$
DIM FileLength AS size_t
InFile$ = COMMAND$(1)
FileLength = Loaf(InFile$)
' PRINT FileLength
! printf("%zd\n", FileLength);

$CCODE funcsub
size_t Loaf(LPCTSTR FileName)
{
  int FileNumber;
  size_t FileLength;

  FileNumber = _open(FileName, _O_BINARY );

  FileLength = _lseeki64(FileNumber, 0, SEEK_END);

  _close(FileNumber);

 return FileLength;
}
$CCODE




The DeRanged.exe random file builder was used
to build a test file named deranged.txt.

I:\t>deranged 50 100 90000000 deranged.txt

The test file build took 273.375 seconds.

I:\t>dir deranged.txt
 Volume in drive I is ST1000
 Volume Serial Number is 6023-B423

 Directory of I:\t

2020-07-30  23:45     7,020,180,951 deranged.txt
               1 File(s)  7,020,180,951 bytes

The above BCX program was translated with BCX 7.5.1
and compiled as snipm.exe with the latest Visual Studio Preview tools.

snipm was then run with deranged.txt as input.

I:\t>snipm deranged.txt
7020180951

The length of file was correctly returned.

Execution also was successful with a Nuwen compilation.

Pelles C does not contain _lseeki64 but has _fseek64 which
has different input parameters and would need special attention.

Regarding a 32 bit compile ... Shirley, you must be joking !!!

**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.6.5
** Copyright (c) 2020 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'
Compiling "snipm.cpp"..........
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28806 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

snipm.cpp
Linking ................
Microsoft (R) Incremental Linker Version 14.26.28806.0
Copyright (C) Microsoft Corporation.  All rights reserved.

Finished!

C:\dev\BCX\BCXCode\LOF>snipm I:\t\deranged.txt
-1569753641

Unicode, with and without the BOM ...

Maybe this was not such a good idea ...

Just stick it in the snippets tool bin and forget about integration.


MrBcx

  • Administrator
  • Sr. Member
  • *****
  • Posts: 434
    • View Profile
Re: Large File Support
« Reply #3 on: July 31, 2020, 09:19:19 AM »
Robert,

It's too early in the discussion to throw in the towel.

I'm going to do my own research and hopefully tumble into some alternative approaches.

Perhaps a first step could involve a revised LOF function with a conditional compilation frame
that tests for _WIN64, so at least those of us running on and compiling for 64-bit can benefit
from the extra fingers that are afforded to us. 

Implementing large file support on a 32-bit machine seems unlikely to happen in any universe.

MrBcx

  • Administrator
  • Sr. Member
  • *****
  • Posts: 434
    • View Profile
Re: Large File Support
« Reply #4 on: July 31, 2020, 12:53:27 PM »
Robert,

Not a big deal but in your example, you use

 
Code: [Select]
printf("%zd\n", FileLength);

I found a test case on LLVM which I tested on MSVC, Mingw, Clang and Pelles that demonstrates that "%zd" will overflow sooner and that "%zu" might serve you better.

The test below must be compiled for 64-bit on all systems, in order to function correctly.

Code: [Select]
#include <stdio.h>

int main() {
  size_t s = 1024 * 1024 * 1024 + 3;
  s *= 1024 * 1024 * 1024;
  s *= 8;
 
  printf("size: %d\n", sizeof(s));

  printf("s %%d: %d\n", s);
  printf("s %%lu: %lu\n", s);
  printf("s %%zd: %zd\n", s);
  printf("s %%zu: %zu\n", s);
  return 0;
}

Robert

  • Sr. Member
  • ****
  • Posts: 383
    • View Profile
Re: Large File Support
« Reply #5 on: July 31, 2020, 03:54:35 PM »
Code below compiles 32 and 64 bit Pelles and Microsoft. Also compiles with Nuwen which is 64 bit only.
Both _lseeki64 and _filelengthi64 work as expected.
Pelle has _lseeki64 and _filelengthi64 defined as _MSC_EXTENSIONS macros, defining the Microsoft keywords as his home brewed _lseek64 and _filelength64.
As expected, 32 bit compiles do not handle large files.

Code: [Select]

DIM InFile$
DIM FileLength AS size_t
InFile$ = COMMAND$(1)
FileLength = Loaf(InFile$)
' PRINT FileLength
! printf("%zu\n", FileLength);

$CCODE funcsub
size_t Loaf(LPCTSTR FileName)
{
  int FileNumber;
  size_t FileLength;

  FileNumber = _open(FileName, _O_BINARY );

//  FileLength = _lseeki64(FileNumber, 0, SEEK_END);
  FileLength = _filelengthi64(FileNumber);

  _close(FileNumber);

 return FileLength;
}
$CCODE


MrBcx

  • Administrator
  • Sr. Member
  • *****
  • Posts: 434
    • View Profile
Re: Large File Support
« Reply #6 on: July 31, 2020, 07:27:51 PM »
New LOF function

I've replaced the LOF runtime in BCX with the code below.  BCX re-compiles & runs correctly
in 32-bit & 64-bit versions using MSVC, Mingw, Pelles, LccWin32, and Clang.   BCX make copious
use of the LOF function, so I consider this to be a very good litmus test.

I created a 16gb test file using Robert's DeRanged program and the new LOF function
reported the size correctly using all the compilers mentioned above in both 32 and 64 bit modes.

Barring any surprises, this new LOF function will be included in BCX 7.5.2



Code: [Select]
LONGLONG lof(LPCTSTR FileName)
{
  HANDLE hFile={0};
  LARGE_INTEGER Fsize={0};
  LONGLONG lSize={0};
  if (FileName[0] == 0) return 0;
  hFile = CreateFile(FileName,GENERIC_READ,FILE_SHARE_READ,NULL,OPEN_EXISTING,0,NULL);
  if (hFile != INVALID_HANDLE_VALUE)
    {
      GetFileSizeEx(hFile, &Fsize);
      lSize = Fsize.QuadPart;
      CloseHandle(hFile);
      return lSize;
     }
  return 0;
}

« Last Edit: July 31, 2020, 11:47:32 PM by MrBcx »

Robert

  • Sr. Member
  • ****
  • Posts: 383
    • View Profile
Re: Large File Support
« Reply #7 on: July 31, 2020, 09:24:04 PM »
New LOF function

I've replaced the LOF runtime in BCX with the code below.  BCX re-compiles & runs correctly
in 32-bit & 64-bit versions using MSVC, Mingw, Pelles, LccWin32, and Clang.   BCX make copious
use of the LOF function, so I consider this to be a very good litmus test.

I created a 16gb test file using Robert's DeRanged program and the new LOF function
reported the size correctly using all the compilers mentioned above in both 32 and 64 bit modes.

Barring any surprises, this new LOF function will be included in BCX 7.5.2



Code: [Select]
LONGLONG lof(LPCTSTR FileName)
{
  HANDLE hFile;
  LARGE_INTEGER Fsize;
  LONGLONG lSize;
  if (FileName[0] == 0) return 0;
  hFile = CreateFile(FileName, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
  if (hFile != INVALID_HANDLE_VALUE)
    {
      GetFileSizeEx(hFile, &Fsize);
      CloseHandle(hFile);
      lSize = Fsize.QuadPart;
      return lSize;
     }
  return 0;
}


Hi MrBCX:

In BCX 7.5.1, the EditLoadFile procedure begining at line 17707 is going to need tweaking. I've been playing with it using the ColorSplitterDemo and fiddling with the .cpp output. I can get it to compile and run but it doesn't load the file into the edit box. Here's ColorSplitterDemo.bas. Be sure to name it ColorSplitterDemo.bas because that is what is loaded into the edit control.

Code: [Select]

 $BCXVERSION "5.08"
 
 GUI "ColorSplitterDemo", PIXELS
 
 ENUM
   IDC_Edit1 = 50
   IDC_Edit2
   IDC_List1
   IDC_Split1
   IDC_Split2
   IDC_Button
   IDC_ComboFG
   IDC_ComboBG
 END ENUM
 
 CONST ComboStyle = WS_BORDER | WS_TABSTOP |WS_VISIBLE | WS_CHILD | CBS_DROPDOWNLIST
 
 GLOBAL Form1  AS HWND
 GLOBAL Edit1  AS CONTROL
 GLOBAL Edit2  AS CONTROL
 GLOBAL List1  AS CONTROL
 GLOBAL Split1 AS CONTROL
 GLOBAL Split2 AS CONTROL
 GLOBAL Button AS CONTROL
 GLOBAL ComboExFG AS CONTROL
 GLOBAL ComboExBG AS CONTROL
 
 
 SUB FORMLOAD
 
   Form1   = BCX_FORM("BCX Splitter Demo", 0, 0, 640, 480)
 
   Split1  = BCX_SPLITTER(Form1,IDC_Split1, HS_VERTICAL,1)
   List1   = BCX_LISTVIEW("Listview1", Split1, IDC_List1, 0, 0, 10, 10,0,WS_EX_CLIENTEDGE)
   Split2  = BCX_SPLITTER(Split1, IDC_Split2, HS_HORIZONTAL, TRUE)
   Edit1   = BCX_RICHEDIT("", Split2, IDC_Edit1, 0, 0, 10, 10)
   Edit2   = BCX_INPUT("", Split2,IDC_Edit2, 0, 0, 200, 240)
 
   Button   = BCX_BUTTON("EXIT DEMO", Form1, IDC_Button, 5, 5)
   ComboExFG = BCX_CONTROL(WC_COMBOBOXEX, Form1,"", IDC_ComboFG, 180, 5 , 175, 180, ComboStyle, 0)
   ComboExBG = BCX_CONTROL(WC_COMBOBOXEX, Form1,"", IDC_ComboBG, 360, 5 , 175, 180, ComboStyle, 0)
   MakeColorCombobox(ComboExFG)
   MakeColorCombobox(ComboExBG)
   SNDMSG(ComboExFG, CB_SETCURSEL,  9, 0)
   SNDMSG(ComboExBG, CB_SETCURSEL, 23, 0)
 
   MODSTYLE(List1,LVS_SHOWSELALWAYS ,LVS_EDITLABELS)
   BCX_SETSPLITPOS(Split2,80)
   BCX_SETSPLITPOS(Split1,25)
 
   '-----------------------------------------------------------------
   '  This should be called if the splitter will follow the form size
   '-----------------------------------------------------------------
   MoveWindow(Form1, 0, 0, 640, 480, TRUE)
 
   '-----------------------------------------------------
   '  Removes CS_VREDRAW and CS_HREDRAW to reduce flicker
   '-----------------------------------------------------
   SetClassLong(Form1,GCL_STYLE,CS_OWNDC)
 
 
   CENTER(Form1)
   SHOW(Form1)
 
   LOCAL lvItem AS LV_ITEM
 
   lvItem.mask = LVIF_TEXT
 
   lvItem.pszText =(LPSTR)"splitter movement"
   ListView_InsertItem(List1, &lvItem)
   lvItem.pszText =(LPSTR)"traditional style"
   ListView_InsertItem(List1, &lvItem)
   lvItem.pszText =(LPSTR)"splitter uses the"
   ListView_InsertItem(List1, &lvItem)
   lvItem.pszText =(LPSTR)"This      --->"
   ListView_InsertItem(List1, &lvItem)
   ListView_SetColumnWidth(List1,0,125)
 
   BCX_SET_TEXT Edit2,"The HORIZONTAL SPLITTER above uses the new splitter bar"
   EDITLOADFILE(Edit1, APPEXEPATH$ + "ColorSplitterDemo.bas")
 
 END SUB
 
 BEGIN EVENTS
   SELECT CASE CBMSG
 
     '---------------------------
    CASE WM_COMMAND
     '---------------------------
    IF HIWORD(wParam) = CBN_SELCHANGE THEN
       DIM RAW cursel = SNDMSG((HWND)lParam,CB_GETCURSEL,0,0)
       IF cursel <> CB_ERR AND cursel < 32 THEN
         IF CBCTL = IDC_ComboFG THEN
           SplitBarFG = QBCOLOR(cursel)
         ELSEIF CBCTL = IDC_ComboBG THEN
           SplitBarBG = QBCOLOR(cursel)
         END IF
         SetFocus(Edit1)
       END IF
     END IF
     IF CBCTL=IDC_Button THEN DestroyWindow(Form1)
 
     '---------------------------
    CASE WM_EXITSIZEMOVE
     '---------------------------
    REFRESH(hWnd)
 
     '---------------------------
    CASE WM_SIZE
     '---------------------------
    DIM RAW rc AS RECT
     DIM STATIC Zoomed
 
     IF wParam <> SIZE_MINIMIZED THEN
       GetClientRect(hWnd, &rc)
       MoveWindow(Split1, 0, 36, rc.right-rc.left,(rc.bottom-rc.top)-36, FALSE)
       IF Zoomed THEN REFRESH(hWnd) : Zoomed = 0
     END IF
     IF wParam = SIZE_MAXIMIZED THEN Zoomed = TRUE
 
   END SELECT
 END EVENTS
 
 SUB MakeColorCombobox(hWnd AS HWND)
   DIM RAW cbei   AS COMBOBOXEXITEM
   DIM RAW g_himl AS HIMAGELIST
   DIM RAW iCnt
 
   SHAREDSET pszText[] AS LPSTR
     " 0 Black",
     " 1 Blue",
     " 2 Green",
     " 3 Cyan",
     " 4 Red",
     " 5 Magenta",
     " 6 Brown",
     " 7 Light Gray",
     " 8 Gray",
     " 9 Light Blue",
     "10 Light Green",
     "11 Light Cyan",
     "12 Light Red",
     "13 Light Magenta",
     "14 Light Yellow",
     "15 Bright White",
     "16 Soft Black",
     "17 Soft Blue",
     "18 Soft Green",
     "19 Soft Cyan",
     "20 Soft Red",
     "21 Soft Magenta",
     "22 Soft Yellow",
     "23 Soft White",
     "24 Soft Gray",
     "25 Soft Light Blue",
     "26 Soft Light Green",
     "27 Soft Light Cyan",
     "28 Soft Light Red",
     "29 Soft Light Magenta",
     "30 Soft Light Yellow",
     "31 Soft Bright White"
     '"User Defined"
  END SET
   g_himl = ImageList_Create(24, 16, ILC_COLOR24|ILC_MASK, 0, 33)
   ' Assign the image list to the ComboBoxEx control
   SNDMSG(hWnd,CBEM_SETIMAGELIST,0,(LPARAM)g_himl)
 
   'Initialize the COMBOBOXEXITEM struct.
   'Set the mask common to all items.
   CLEAR(cbei)
   cbei.mask = CBEIF_TEXT | CBEIF_IMAGE | CBEIF_SELECTEDIMAGE
   FOR iCnt = 0 TO 31
     ImageList_AddMasked(g_himl, RoundRectBmp(QBCOLOR(iCnt)), RGB(254,0,254))
 
     cbei.iItem          = iCnt
     cbei.pszText        = pszText[iCnt]
     cbei.iImage         = iCnt
     cbei.iSelectedImage = iCnt
 
     IF SNDMSG(hWnd,CBEM_INSERTITEM,0,&cbei) = -1 THEN EXIT FOR
   NEXT
   SNDMSG(hWnd,CB_SETCURSEL,0,0)    'Set Default
END SUB
 
 
 FUNCTION RoundRectBmp(cl AS COLORREF) AS HBITMAP
   DIM STATIC custbmp AS HBITMAP
   DIM RAW    oldobj  AS HGDIOBJ
   DIM RAW    hdc     AS HDC
   DIM AUTO   bi      AS BITMAPINFO
 
   IF custbmp THEN DeleteObject(custbmp)
 
   bi.bmiHeader.biSize        = SIZEOF(BITMAPINFOHEADER)
   bi.bmiHeader.biWidth       = 24
   bi.bmiHeader.biHeight      = 16
   bi.bmiHeader.biPlanes      = 1
   bi.bmiHeader.biBitCount    = 24
   bi.bmiHeader.biCompression = BI_RGB
 
   hdc     = CreateCompatibleDC(NULL)
   custbmp = CreateDIBSection(hdc, &bi,DIB_RGB_COLORS, NULL, NULL, 0)
   oldobj  = SelectObject(hdc,custbmp)
   'fill with mask color
   BCX_RECTANGLE(0, 0, 0, 24, 16, RGB(254, 0, 254), TRUE, hdc)
 
   'Draw black outline and fill with color
   BCX_ROUNDRECT(0, 0, 2, 24, 14, 4, 2, cl, TRUE, hdc)
   BCX_ROUNDRECT(0, 0, 1, 24, 15, 4, 2, RGB(0,0,0), FALSE, hdc)
 
   SelectObject(hdc,oldobj)
   DeleteDC(hdc)
   FUNCTION = custbmp
 END FUNCTION


« Last Edit: August 01, 2020, 12:14:00 AM by MrBcx »

MrBcx

  • Administrator
  • Sr. Member
  • *****
  • Posts: 434
    • View Profile
Re: Large File Support
« Reply #8 on: July 31, 2020, 11:49:17 PM »
Robert -- it was an issue with the new LOF function.

I updated the LOF code (above)

This is the changed line

  hFile = CreateFile(FileName,GENERIC_READ,FILE_SHARE_READ,NULL,OPEN_EXISTING,0,NULL);
« Last Edit: August 01, 2020, 12:13:21 AM by MrBcx »

Robert

  • Sr. Member
  • ****
  • Posts: 383
    • View Profile
Re: Large File Support
« Reply #9 on: August 01, 2020, 12:23:32 AM »
Robert -- it was an issue with the new LOF function.

I updated the LOF code (above)

This is the changed line

  hFile = CreateFile(FileName,GENERIC_READ,FILE_SHARE_READ,NULL,OPEN_EXISTING,0,NULL);

Hi MrBCX:

Very good. I was wondering about that flag when I was looking at the docs for GetFileSizeEx function where it said

Quote
The handle must have been created with the FILE_READ_ATTRIBUTES access right or equivalent