Problems with UTF-8

[MrBcx]

I updated my Nvidia drivers to the latest (with vulkan support), and it's much faster now.

Hey AIR,

List the 50 U.S. states  ... takes 1 minute on my Dell 10th gen i7 with no dedicated graphics card.

I'm curious to learn how quickly it finishes on yours.

[airr]

I aborted after 5 minutes of no results.

HOWEVER, when I queried like this: ""What are the names of all of the States in the United States of America?"

It took 13secs.....

AIR.

[MrBcx]

I aborted after 5 minutes of no results.

HOWEVER, when I queried like this: ""What are the names of all of the States in the United States of America?"

It took 13secs.....

AIR.

Thanks for that. 

I've tweaked my UI a bit -- see attached screenshot.

List the 50 U.S. states is my default prompt ( useful for testing )

Strange that List the 50 U.S. states tanked on you.

I suppose it doesn't matter ... we're just experimenting.

[tinyBigGAMES]

I updated my Nvidia drivers to the latest (with vulkan support), and it's much faster now.

Hey AIR,

List the 50 U.S. states  ... takes 1 minute on my Dell 10th gen i7 with no dedicated graphics card.

I'm curious to learn how quickly it finishes on yours.

I get ~37 tokens/sec on my RTX 3060.

Code: (Delphi) [Select]

procedure Test05();
var
  LResponse: PAnsiChar;
  LTokenInputSpeed: Single;
  LTokenOutputSpeed: Single;
  LInputTokens: Integer;
  LOutputTokens: Integer;
  LTotalTokens: Integer;
begin
  Dllama_InitConfig(CModelPath, -1, False, 27);
  Dllama_LoadModelDb('models.json');

  Dllama_AddMessage('system', 'you are Dllama, a helpful AI assistant.');
  Dllama_AddMessage('user', 'list all states in the United States.');

  // do inference
  if Dllama_Inference('phi3:4B:Q4', @LResponse, 1024, 0.5, 1111) then
    begin
      // display usage
      Dllama_Console_PrintLn(#10#13, 4 or 2);
      Dllama_GetInferenceUsage(@LTokenInputSpeed, @LTokenOutputSpeed, @LInputTokens, @LOutputTokens, @LTotalTokens);
      Dllama_Console_PrintLn(PUTF8Char(UTF8String(Format('Tokens :: Input: %d, Output: %d, Total: %d, Speed: %3.1f t/s', [LInputTokens, LOutputTokens, LTotalTokens, LTokenOutputSpeed]))), 4 or 2 or 8);
    end
  else
    begin
      // display errors
      writeln('Error: %s', Dllama_GetError());
    end;

  // unload model
  Dllama_UnloadModel();
end;

https://www.youtube.com/watch?v=pLZAfriWDsI

[airr]

Hi, tBG, and welcome!

We've been using Dllama_Simple_Inference while testing;  would using the API as you demonstrated result in a boost in performance?

I could probably translate most of it, being an old Delphi user (from back in the v4 days! D8 killed it for me, though) and was wondering if you thought it would be worth doing?

Thanks,

AIR.

[tinyBigGAMES]

Hi, tBG, and welcome!

We've been using Dllama_Simple_Inference while testing;  would using the API as you demonstrated result in a boost in performance?

I could probably translate most of it, being an old Delphi user (from back in the v4 days! D8 killed it for me, though) and was wondering if you thought it would be worth doing?

Thanks,

AIR.

Hi, and thanks. In Dllama_Simple_Inference, you have to wait until the model loads, then you have to wait until inference has completed, the perceived delay stacks up and becomes somewhat annoying. What you see in the video really takes the same amount of time, just that you are getting visual feedback. There are callbacks for model load progress and for inference. You can display status updates in those callbacks, which you see the model loading percentage and it displays the tokens directly after being generated (streaming). It just gives a better overall experience as you see in the video.

In the distro, you find Dllama.pas and Dllama.h/lib in [lib] folder, which should make it easier to build bindings. I think I saw in the docs that BCX can consume C code? I'm trying to keep the API small yet powerful, doing just the things that are needed without complications. At present there are about 40 routines. I would start with the routines in the demo, which are the basic ones to get you started. You would also need Dllama_AddModel so that you can define your models. See UTestbed.pas in [examples] folder for more examples of use.

Let me know if/when you get BCX bindings. I will add to the bindings list and help promote it. Feel free to contact me anytime if you have got questions, need help, etc. You are welcome to join my discord also.

[MrBcx]

Hi Jarrod and welcome to our forum.

I've had a useful albeit casual experience with various LLM's since they burst on the scene a year ago. 
Your project has revived my interest in building a BCX project with a local LLM.  Thanks for sharing
Dllama and I genuinely hope the development advances.

Many thanks,
~MrBcx~

[tinyBigGAMES]

Hi Jarrod and welcome to our forum.

I've had a useful albeit casual experience with various LLM's since they burst on the scene a year ago. 
Your project has revived my interest in building a BCX project with a local LLM.  Thanks for sharing
Dllama and I genuinely hope the development advances.

Many thanks,
~MrBcx~

Thank you!

[airr]

Quick Demo.  Change CModelPath to point to the models.json file

I had to regenerate the import lib, due to alignment issue with the one from the github when using PellesC.

Also had to create a new header file.  The one in git is for creating the DLL.

Both files are in the attached zip file.

Code: [Select]

#include "DllamaC.h"

$pragma comment(lib,"Dllama.lib")

dim as const pchar CModelPath = "LLM\models.json"
dim as const char* LProject

Dllama_InitConfig(CModelPath, -1, False, VK_ESCAPE)

Dllama_GetVersionInfo(null, null, null, null, null, null, &LProject)
Dllama_Console_PrintLn(LProject, CYAN)

Dllama_Console_Pause(true,YELLOW,"")

Dllama_UnloadModel()



[MrBcx]

Quick Demo.  Change CModelPath to point to the models.json file

I had to regenerate the import lib, due to alignment issue with the one from the github when using PellesC.

Also had to create a new header file.  The one in git is for creating the DLL.

Both files are in the attached zip file.

Time-saving and useful.  Gracias!

[tinyBigGAMES]

Quick Demo.  Change CModelPath to point to the models.json file

I had to regenerate the import lib, due to alignment issue with the one from the github when using PellesC.

Also had to create a new header file.  The one in git is for creating the DLL.

Both files are in the attached zip file.

Code: [Select]

#include "DllamaC.h"

$pragma comment(lib,"Dllama.lib")

dim as const pchar CModelPath = "LLM\models.json"
dim as const char* LProject

Dllama_InitConfig(CModelPath, -1, False, VK_ESCAPE)

Dllama_GetVersionInfo(null, null, null, null, null, null, &LProject)
Dllama_Console_PrintLn(LProject, CYAN)

Dllama_Console_Pause(true,YELLOW,"")

Dllama_UnloadModel()



- Dllama.h - in [lib],  it defines #define DLLAMA_API extern "C" __declspec(dllimport), which is for importing. It "should" work both in C and C++ compilers. Only tested with Visual Studio 2022 and C++Builder 12.1 so far. I will have to check PellesC.
- Does  PellesC support win64 (Dllama.dll is a win64 dll). Tell me more about the alignment problem you had.
- Dllama_InitConfig(CModelPath, -1, False, VK_ESCAPE), the first params sets the path to where your LLM files are located, you use AddModel("filename_of_model.gguf"...) to define the actual model. When you call inference, it will use CModelPath  + the model filename to load it into memory. The -1 means use GPU, all layers, if you use 0, it will use the CPU only.

Code: (Delphi) [Select]




procedure AddModels();
begin
  Dllama_AddModel(
    // model filename
    'Meta-Llama-3-8B-Instruct-Q6_K.gguf',


    // model reference name
    'llama3:8B:Q6',


    // model max context size
    1000*8, {8K}


    // model prompt format
    '<|begin_of_text|><|start_header_id|>%s<|end_header_id|>\n %s<|eot_id|>',


    // model prompt format ending
    '<|start_header_id|>assistant<|end_header_id|>',


    // model stop sequences
    ['<|eot_id|>', '<|start_header_id|>', '<|end_header_id|>', 'assistant']
  );


  Dllama_AddModel(
    // model filename
    'Phi-3-mini-4k-instruct-q4.gguf',


    // model reference name
    'phi3:4B:Q4',


    // model max context size
    1000*4, {4K}


    // model prompt format
    '<|%s|>\n %s<|end|>\n',


    // model prompt format ending
    '<|assistant|>\n',


    // model stop sequences
    ['<|user|>', '<|assistant|>', '<|system|>', '<|end|>', '<|endoftext|>']
    );


  Dllama_SaveModelDb('models.json');
end;


procedure Test01();
var
  LResponse: string;
  LTokenInputSpeed: Single;
  LTokenOutputSpeed: Single;
  LInputTokens: Integer;
  LOutputTokens: Integer;
  LTotalTokens: Integer;
begin
  // init config
  Dllama_InitConfig('C:\LLM\gguf', -1, False, VK_ESCAPE);


  // add models
  AddModels();


  // add messages
  Dllama_AddMessage(ROLE_SYSTEM, 'You are Dllama, a helpful AI assistant, created in 2024 by tinyBigGAMES LLC.');



  Dllama_AddMessage(ROLE_USER, 'List the 50 U.S. states');




  // display user prompt
  Dllama_Console_PrintLn(Dllama_GetLastUserMessage(), [], DARKGREEN);


  // do inference
  if Dllama_Inference('phi3:4B:Q4', LResponse) then
  //if Dllama_Inference('llama3:8B:Q6', LResponse) then
    begin
      // display usage
      Dllama_Console_PrintLn(CRLF, [], WHITE);
      Dllama_GetInferenceUsage(@LTokenInputSpeed, @LTokenOutputSpeed, @LInputTokens, @LOutputTokens, @LTotalTokens);
      Dllama_Console_PrintLn('Tokens :: Input: %d, Output: %d, Total: %d, Speed: %3.1f t/s', [LInputTokens, LOutputTokens, LTotalTokens, LTokenOutputSpeed], BRIGHTYELLOW);
    end
  else
    begin
      // display errors
      Dllama_Console_PrintLn('Error: %s', [Dllama_GetError()], RED);
    end;


  // unload model
  Dllama_UnloadModel();
end;
See MODELS.txt for the models that I've fully tested and their download locations.

[tinyBigGAMES]

I updated Dllama.h to properly work for both C and C++ compiler and put together a project that compiles in PellesC (all updated files in zip). I set to use MS extensions and now the only thing I see are the warnings about data alignment. I searched PellesC forums and saw similar. The general consensus is that it's safe to ignore? If anyone has more insight on this, please feel free to share your thoughts or suggestions.

[MrBcx]

... and now the only thing I see are the warnings about data alignment. I searched PellesC forums and saw similar.
The general consensus is that it's safe to ignore?
If anyone has more insight on this, please feel free to share your thoughts or suggestions.

Common consequences of data boundary alignment issues are inefficient data access by the CPU ( slower performance )
and depending on the compiler, data padding which can waste memory.  Whether either of those has any measurable or
practical consequence is mostly a function of what the application is doing.  Actually, there is a third consequence,
undefined behavior.  Think, needle in a haystack.  My philosophy is, it's not a problem until it's a problem.

If you really want to take a dip, you can look into _Alignas  ( for plain C ) or alignas ( for C++ )
I've never felt compelled to use either one and therefore have no experience.

https://en.cppreference.com/w/c/language/_Alignas

https://en.cppreference.com/w/cpp/language/alignas

[airr]

I updated Dllama.h to properly work for both C and C++ compiler and put together a project that compiles in PellesC (all updated files in zip). I set to use MS extensions and now the only thing I see are the warnings about data alignment. I searched PellesC forums and saw similar. The general consensus is that it's safe to ignore? If anyone has more insight on this, please feel free to share your thoughts or suggestions.

Okay, in BCX your code would look like this:

Code: [Select]

#include "Dllama.h"

$pragma comment(lib,"Dllama.lib")

dim as const pchar CModelPath = "C:/LLM/gguf"

Dllama_InitConfig(CModelPath, -1, False, VK_ESCAPE)


if not Dllama_LoadModelDb("models.json") then
    print "Load Model Error: ", Dllama_GetError()
    end = 1
end if
   
dim as pchar question = "List all the states in the United State?";
dim as pchar* response   
Dllama_AddMessage(ROLE_USER, &question[0])

if Dllama_Inference("phi3:4B:Q4", response, 1024, 0.5, 1111) then
else
    print "Inference Error: ", Dllama_GetError()
    end = 1
end if

Dllama_UnloadModel()

Dllama_Console_Pause(true, WHITE, "")

Changed the path to the models to use forward slashes.  I always forget the double back slashes when interfacing with DLLs that expect that.
Changed the response var to a pchar pointer.  Less typing.   

In the header file, I had to add this:

Code: [Select]

// Virtual Keys
#if !defined(__POCC__)
    #define VK_ESCAPE 27
#endif

The reason being that one of the files that BCX includes already defines VK_ESCAPE.

Other than that (and the alignment warning when I use your lib) it works as advertised.

AIR.

[airr]

Here's a slightly cleaned up version, which allows you to enter the query at the command prompt (easier to test queries):

Code: [Select]

#include "Dllama.h"

$Pragma comment(lib, "Dllama.lib")

Dim As Const Pchar CModelPath = "C:/LLM/gguf"

Dim As Pchar response

If ARGC <> 2 Then
    Print "Missing Query. Command Usage: ", Command$(0), " <query>"
    Pause
    End = 1
End If

Dllama_InitConfig(CModelPath, -1, False, VK_ESCAPE)

If Not Dllama_LoadModelDb("models.json") Then
    Print "Load Model Error: ", Dllama_GetError$()
    End = 1
End If

Dllama_AddMessage(ROLE_USER, Command$(1))

If Not Dllama_Inference("phi3:4B:Q4", &response, 1024, 0.5, 1111) Then
    Print "Inference Error: ", Dllama_GetError$()
    End = 1
End If

Dllama_UnloadModel()

Pause

AIR.