I love qwen 2.5 coder 7b instruct too. I haven't tried it bcx skills yet though. But it can generate code for simple VisualBasic 6.0 applications. However, i use more interesting tool than jan.ai.
Q4 quant has acceptable quality but it's not enough for best results in code generation. I found pure c llm engine made from scratch by talented coder/scientist.
Here's it's github repo: https://github.com/pierrel55/llama_st
This engine allows to run qwen 2.5 coder 7b instruct in fp12 mode which makes code generation much better (qwen makes less mistakes). I don't have gpu and run qwen on cpu (ryzen 7 4700u) and 16gb ram. It is slow but brings me happiness when after long waiting i finally get working code. llama_st also supports fp8 quantization which is worse than fp12 but still much more clever than q4. It is obvious that qwen in fp8 mode requires less ram and runs faster than in fp12 mode.
Jan.ai can't work without llama.cpp/ggml backend, it also can't work in fp12 mode. llama_st doesn't require any backend like llama.cpp/ggml. it can be compiled in w64devkit 1.21.0 or in Microsoft Visual Studio Express 2012. It has only one disadvantage: it loads unquantatized weights of qwen which requires much more free space on your ssd.
I was so fascinated with coolness of llama_st.exe that used Deepseek (https://chat.deepseek.com/) to modify llama_st source code in such way that it can be compiled and used as dll which has single-shot question-answer function.
I called this function from bcx code, sent random question to llama_st.dll and qwen answered my question. Unfortunately, my bcx and c skills are too low for porting chatting functionality from llama_st.exe to llama_st.dll. Deepseek probably could do that but forcing Deepseek completing such task requires decent English skills (my native language is different) and sufficient coding experience because Deepseek needs mindful guidance from it's human master otherwise it generates non-working code.
I attached llama_st.zip to this post. It contains minimal bcx bas file needed to run dll, llama_st.dll, modified main.c source file of llama_st.exe (modifications were done to make it possible to compile it to dll instead of exe), llama_st.exe (compiled minimal bcx bas), build_dll.bat, modified json file containing qwen parameters.
First of all you need to download llama_st zipped source files archive (https://github.com/pierrel55/llama_st/archive/refs/heads/main.zip), unpack it to some folder, replace original main.c with modified main.c, place build_dll.bat to folder where your lama_st source files are.
Then you need to download and install w64devkit 1.21.0 (https://github.com/skeeto/w64devkit/releases/download/v1.21.0/w64devkit-1.21.0.zip). After installing it you should launch w64devkit.exe.
Inside console window of w64devkit.exe you should type and execute something like cd C:\\llama_st (assuming that you unpacked source of llama_st there). Then you should type and execute build_dll.bat. W64devkit.exe will use commands from build_dll.bat for llama_st.dll compilation.
You will need to download unquantatized weights of qwen - https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/tree/main All downloaded weights files should be put into some folder. You should edit modified_qwen_json.json accordingly. By default weights folder path specified in modified_qwen_json.json is C:/mdl2. Original llama_st.exe uses unmodified json file which has ability to load three different qwen models. And weights folder path is specified for each qwen model. But in our case we use only one qwen. However, we should specify weights folder path three times in modified_qwen_json.json.
So what we have for the moment?
1) compiled llama_st.dll
2) downloaded qwen weights files in C:\mdl2 (json file requires different slash in path: /)
Now we should do several more things:
1) create run_json subfolder This subfolder should be created in the same folder where llama_st.dll is.
2) move modified_qwen_json.json to run_json subfolder
3) launch BED
4) open llama_st.bas in BED (llama_st.bas can be found in zip archive attached to this post)
5) compile llama_st.bas 6) launch llama_st.exe
Here's code of llama_st.bas:
DECLARE FUNCTION ai$ LIB "llama_st.dll" ALIAS "get_answer_from_ai"(sent_question$, command_line$)
ai$("Why cats can't bark?", "run_json/modified_qwen_json.json")
As far as i know none of Python/Rust/C++ llm engines can run qwen 7b model in fp12 mode on 16gb ram. Only C-coded llama_st can. And the most amazing thing: conversion from fp16 (original format of qwen weights) to fp12 is made in few seconds and in ram without any Python scripts and writing quantatized weights to ssd. It's pure magic. Glory to standard C (and BCX, of course)!
P.S: my attempt to convert code of exe to code of dll using Deepseek isn't perfect. It has some weak places due to my lack of understanding coding etc. For example, qwen doesn't stop text generation in some cases. So you should just close llama_st.exe window. If you're experienced C coder it will not be a problem for you to fix main.c. Also maybe i broke something in json file unwillingly, by accident.
So if you will encounter non-stoping qwen text generation you should probably try original qwen_json.json from llama_st source archive. However, in any case you will need to modify 3 pathes to weights files folder. Original llama_st.exe hasn't problem with unstopping qwen text generation.
Q4 quant has acceptable quality but it's not enough for best results in code generation. I found pure c llm engine made from scratch by talented coder/scientist.
Here's it's github repo: https://github.com/pierrel55/llama_st
This engine allows to run qwen 2.5 coder 7b instruct in fp12 mode which makes code generation much better (qwen makes less mistakes). I don't have gpu and run qwen on cpu (ryzen 7 4700u) and 16gb ram. It is slow but brings me happiness when after long waiting i finally get working code. llama_st also supports fp8 quantization which is worse than fp12 but still much more clever than q4. It is obvious that qwen in fp8 mode requires less ram and runs faster than in fp12 mode.
Jan.ai can't work without llama.cpp/ggml backend, it also can't work in fp12 mode. llama_st doesn't require any backend like llama.cpp/ggml. it can be compiled in w64devkit 1.21.0 or in Microsoft Visual Studio Express 2012. It has only one disadvantage: it loads unquantatized weights of qwen which requires much more free space on your ssd.
I was so fascinated with coolness of llama_st.exe that used Deepseek (https://chat.deepseek.com/) to modify llama_st source code in such way that it can be compiled and used as dll which has single-shot question-answer function.
I called this function from bcx code, sent random question to llama_st.dll and qwen answered my question. Unfortunately, my bcx and c skills are too low for porting chatting functionality from llama_st.exe to llama_st.dll. Deepseek probably could do that but forcing Deepseek completing such task requires decent English skills (my native language is different) and sufficient coding experience because Deepseek needs mindful guidance from it's human master otherwise it generates non-working code.
I attached llama_st.zip to this post. It contains minimal bcx bas file needed to run dll, llama_st.dll, modified main.c source file of llama_st.exe (modifications were done to make it possible to compile it to dll instead of exe), llama_st.exe (compiled minimal bcx bas), build_dll.bat, modified json file containing qwen parameters.
First of all you need to download llama_st zipped source files archive (https://github.com/pierrel55/llama_st/archive/refs/heads/main.zip), unpack it to some folder, replace original main.c with modified main.c, place build_dll.bat to folder where your lama_st source files are.
Then you need to download and install w64devkit 1.21.0 (https://github.com/skeeto/w64devkit/releases/download/v1.21.0/w64devkit-1.21.0.zip). After installing it you should launch w64devkit.exe.
Inside console window of w64devkit.exe you should type and execute something like cd C:\\llama_st (assuming that you unpacked source of llama_st there). Then you should type and execute build_dll.bat. W64devkit.exe will use commands from build_dll.bat for llama_st.dll compilation.
You will need to download unquantatized weights of qwen - https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/tree/main All downloaded weights files should be put into some folder. You should edit modified_qwen_json.json accordingly. By default weights folder path specified in modified_qwen_json.json is C:/mdl2. Original llama_st.exe uses unmodified json file which has ability to load three different qwen models. And weights folder path is specified for each qwen model. But in our case we use only one qwen. However, we should specify weights folder path three times in modified_qwen_json.json.
So what we have for the moment?
1) compiled llama_st.dll
2) downloaded qwen weights files in C:\mdl2 (json file requires different slash in path: /)
Now we should do several more things:
1) create run_json subfolder This subfolder should be created in the same folder where llama_st.dll is.
2) move modified_qwen_json.json to run_json subfolder
3) launch BED
4) open llama_st.bas in BED (llama_st.bas can be found in zip archive attached to this post)
5) compile llama_st.bas 6) launch llama_st.exe
Here's code of llama_st.bas:
DECLARE FUNCTION ai$ LIB "llama_st.dll" ALIAS "get_answer_from_ai"(sent_question$, command_line$)
ai$("Why cats can't bark?", "run_json/modified_qwen_json.json")
As far as i know none of Python/Rust/C++ llm engines can run qwen 7b model in fp12 mode on 16gb ram. Only C-coded llama_st can. And the most amazing thing: conversion from fp16 (original format of qwen weights) to fp12 is made in few seconds and in ram without any Python scripts and writing quantatized weights to ssd. It's pure magic. Glory to standard C (and BCX, of course)!
P.S: my attempt to convert code of exe to code of dll using Deepseek isn't perfect. It has some weak places due to my lack of understanding coding etc. For example, qwen doesn't stop text generation in some cases. So you should just close llama_st.exe window. If you're experienced C coder it will not be a problem for you to fix main.c. Also maybe i broke something in json file unwillingly, by accident.
So if you will encounter non-stoping qwen text generation you should probably try original qwen_json.json from llama_st source archive. However, in any case you will need to modify 3 pathes to weights files folder. Original llama_st.exe hasn't problem with unstopping qwen text generation.
