Nim : how to run external binary ? #3
Comments
|
btw. i'm planning to build a Nim GUI with Owlkettle. |
|
I think actually I never did it myself, and it is not mentioned in my book. But I read about it in the past. See https://stackoverflow.com/questions/75712315/how-to-run-an-external-program https://glenngillen.com/learning-nim/executing-external-commands/ https://stackoverflow.com/questions/71432607/how-to-get-what-is-on-the-terminal-in-nim-lang |
|
And this is the answer from GPT-4. Please try yourself, it should be helpful, but GPT-4 is not free from errors: https://chat.openai.com/share/3be72d94-e0f8-4017-abe9-0e9452bc9c54 |
that link suits me the most : the last code part on that page should be able to feed the "uci" command to the import osproc, strutils, streams
let process = startProcess("stockfish", args = [], options = {poInteractive, poUsePath})
let (FROMp, TOp) = (process.outputStream, process.inputStream)
TOp.write "uci\n"
TOp.flush
echo FROMp.readLine
discard process.waitForExit
process.closei think i'm missing something obvious .. indeed i expect such in/out functionality can be done by just a few lines of Nim code, but it fails .. i tried several minor tweaks in such code, compared other documents & forum entries, too no avail .. i guess you could easily reproduce this with your stockfish binary ? btw. it's strange to me : shouldn't |
|
i'm getting there ..! hold on.. |
|
newest working (!) code : import osproc, streams
let process = startProcess( "stockfish", args = [], options = { poInteractive, poUsePath } )
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = true
# echo the first Stockfish "welcome line"
echo fromP.readLine
fromP.flush()
toP.write("uci\n")
toP.flush
#echo all uci output lines
s = fromP.readLine
while TF:
echo s
fromP.flush()
try:
# after the last 'uciok' line, the process waits here forever : no error is catched
s = fromP.readLine
except CatchableError:
TF = false
# the script doesn't ever reach these lines
discard process.waitForExit
process.closeoutput : |
|
i must be almost there ..? |
process.waitForExit is Nim's method call syntax for waitForExit(process). You can also write process.waitForExit() but the braces can be left out. write "uci\n" is the syntax with omitted braces which is used typically for the echo statement. I am using typically braces, and only leave them out for echo. All explained in much detail in my book. |
do you see / know why ? |
|
that's why i'm wondering if threads are neede here : i imagine a threaded process runs "on its own" and its state can be read, so the main script can act upon this : read new output or feed new input .. am i wrong ? |
|
I have never done this, but maybe I should add it to the book. GPT-4 just suggested to use poParentStreams, see https://chat.openai.com/share/3be72d94-e0f8-4017-abe9-0e9452bc9c54 Generally, you could search how this communication is done in C langauge. For Nim it is then very similar. Generally, I have some doubts that such a communication gives enough freedom for user interaction, like forcing a move, give the user a hint, and all that. My impression was that such communication is typically done by network protocols like TCP. |
|
Sorry, new GTP link does not work. Here is the text User Yes, you can use the input stream of the process to send messages or data to it in Nim. This is particularly useful when you need to interact with an external process that reads from its standard input. Here's how you can do it: Here’s an example demonstrating this: nim import osproc, os let binaryPath = "/path/to/external/binary" Start the process with input streaming enabledlet proc = startProcess(binaryPath, options = {poParentStreams, poStdErrToStdOut}) Write to the process's input streamproc.inputStream.write("Message to send\n") Close the input streamproc.inputStream.close() Wait for the process to completelet exitCode = proc.waitForExit() Read the outputlet output = proc.outputStream.readAll() echo "Process output: ", output Check exit codeif exitCode == 0: Close the processproc.close() In this code, write is used to send a string to the process's standard input. The \n at the end of the string is often necessary to signal the end of an input line, especially for console-based programs that read input line by line. Remember to replace the message and the binary path with your specific requirements. Also, depending on the external program, you may need to format the input data differently. |
i only want to communicate with the engine by its normal UCI commands : "go .." will end with a best move .. before that, we can set some UCI options, the starting position etc. with isready (:readyOK) in between .. however, it would be nice to also be able to send a "stop" command which the engine will "hear" .. no need to force a move or give a hint yet .. |
yes, i saw and studied many of those scripts, also non-Nim .. i do python and a little bash, but not C .. i really like a working Nim example .. |
|
One more: https://www.codeproject.com/Articles/1159746/StockChess Sorry, I have no time for further investigations. |
|
THANKS !! |
|
If you don't get it working yourself, my advice would be to find a working Python, Rust, or C solution, and then ask in the Nim forum or at StackOverflow. When there is already a solution for some language, then people are typically motivated to do it in Nim as well. Or, mention that you will use Owekettle, that may motivate someone to help you. Bye. |
|
[ i leave this Issue Open for now - i might post a final working solution here, soon ] |
|
This Nim forum thread may be relevant. But reading delayed multi-line data seems to be difficult: |
|
I think i solved the problem .. i will write about it tomorrow :) |
|
It is difficult: https://stackoverflow.com/questions/32301415/interfacing-with-stockfish-shell-using-c Working with timeouts as in the Nim forum thread may work, but is not really nice. For input, stockfish seems to wait for the quit command, but it is difficult to detect the end of stockfish's output. |
|
Wait, I may be wrong: From https://chess.stackexchange.com/questions/28755/communication-with-stockfish-engine-via-python stockfish seems to terminate its output with message 'readyok'. |
|
basically (and probably all i need) only 3 last output lines are possible and relevant :
so, it's possible to intercept those (last) output lines and our script must respond accordingly. it's late :) |
|
Have installed stockfish now, and run your code, and compared to the terminal output. Seems to be identical, which is fine. So stockfish seems to wait for new input: |
|
i knew you'd be intrigued by this :) |
|
OK, this is my latest script .. it works : import std/[streams, osproc, strutils]
let process = startProcess( "stockfish", args = [], options = { poInteractive, poUsePath } )
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = true
toP.write("uci\n") # returns "uciok"
echo "<=- uci"
toP.flush
while TF:
s = fromP.readLine
echo "-=> " & s
if s == "uciok":
toP.write("ucinewgame\n") # returns nothing
toP.flush
echo "<=- ucinewgame"
toP.write("isready\n") # returns "readyok"
toP.flush
echo "<=- isready"
s = fromP.readLine
echo "-=> " & s # should be "readyok"
toP.write("position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1\n") # returns nothing
toP.flush
echo "<=- position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1"
toP.write("isready\n") # returns "readyok"
toP.flush
echo "<=- isready"
s = fromP.readLine
echo "-=> " & s # should be "readyok"
toP.write("go depth 12\n") # returns a lot of info lines, ending with "bestmove [...]"
toP.flush
echo "<=- go depth 12"
elif s == "readyok":
echo "<=- readyok"
elif s.startsWith("bestmove"):
TF = false # this will be the last echoed line
# this example script ends here,
# but we could input other UCI commands now (keeping TF true)
toP.write("quit\n")
toP.flush
discard process.waitForExit
process.close |
|
editted .. tomorrow :) |
|
Fine that it works for you. I can not really continue testing, because I do not know all the stockfish commands. |
|
I hope you will have success with stockfish and owlkettle. The https://github.com/elcritch/figuro package seems to be still in an too early state to use it for a serious project. I think I will convert my chess game to Rust this weekend, should be not that hard, as the code is short and only uses Nim concepts that are supported by Rust as well, that is no exceptions, no inheritance. I just tried a plain Gintro to GTK-rs translation, see https://discourse.gnome.org/t/fixing-rust-gtk4-code-with-gpt-4-works-really-well/18143. With some help of GPT-4 it was easy. But of course the Rust code looks not that clean as Nim or Python code, and is a bit longer. |
my script works now .. it was easy, at last .. i adjusted the previous example code to my latest version today .. it works with any (?) UCI chess engine binary AFAIK. Also read my comments in the code. The page https://nim-lang.org/docs/osproc.html gives all info. Owlkettle seems one of few good tools to create a GUI with Nim .. i follow the PRs of this GitHub project, still lots of changes and new things. btw. you should consider studying the UCI protocol, for various reasons .. it's not hard and well documented .. i also propose your newest engine should be UCI driven, eg. let it play in CuteChess against other engines. Good luck with the Rust adventure, i'm looking forward to the result ! |
|
looking for answers i found some .. and it's working now, more or less .. my latest discovery and question is at https://forum.nim-lang.org/t/10719 do you know how to set PRIORITY_HIGH_IDLE ? FYI: here is my latest script, "unstable" though, as i wrote at the forum .. import std/[streams, os, osproc, strutils, strformat]
import owlkettle
viewable App:
buffer: TextBuffer
monospace: bool = false
cursorVisible: bool = true
editable: bool = true
acceptsTab: bool = true
indent: int = 0
sensitive: bool = true
sizeRequest: tuple[x, y: int] = (-1, -1)
tooltip: string = ""
var MS = 1 # sleep [#] MilliSeconds
var DP = 18 # eval DePth
var TFecho = true
#var TFecho = false
when compileOption("threads"):
var thread: Thread[AppState]
proc chgBuffer(app: AppState, s: string) =
var abt = ""
var s2 = ""
if TFecho:
echo s
try:
abt = app.buffer.text
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 1"
try:
s2 = abt & s & "\n"
app.buffer.text = s2
#app.buffer.text = s & "\n"
#app.buffer.text = app.buffer.text & s & "\n"
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 2"
sleep(500)
try:
app.redrawFromThread()
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 3"
sleep(500)
#discard
proc threadProc(app: AppState) {.thread.} =
#let process = startProcess( "./examples/widgets/", args = [], options = { poInteractive, poUsePath } )
let process = startProcess( "./examples/widgets/Clubfoot_1.0.8e3b4da_compiled_HP", args = [], options = { poInteractive, poUsePath } )
#let process = startProcess( "./examples/widgets/amoeba_v3.4_compiled_fast", args = [], options = { poInteractive, poUsePath } )
#let process = startProcess( "./examples/widgets/camel_v1.2.0-PR90-compiled_HP", args = [], options = { poInteractive, poUsePath } ) # ???
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = false
proc mySleep() =
sleep(MS)
#discard
toP.write("uci\n") # returns "uciok"
toP.flush
s = fromP.readLine # should return "uciok"
mySleep()
chgBuffer(app,s)
TF = true
while TF:
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE A"
continue
chgBuffer(app, "-=> " & s)
if s == "uciok":
toP.write("ucinewgame\n") # returns nothing
toP.flush
mySleep()
chgBuffer(app,"<=- ucinewgame")
toP.write("isready\n") # returns "readyok"
toP.flush
mySleep()
chgBuffer(app,"<=- isready")
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE B"
continue
# should be "readyok"
chgBuffer(app,"-=> " & s)
toP.write("position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1\n") # returns nothing
toP.flush
mySleep()
chgBuffer(app,"<=- position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1")
toP.write("isready\n") # returns "readyok"
toP.flush
mySleep()
chgBuffer(app,"<=- isready")
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE C"
continue
# should be "readyok"
chgBuffer(app,"-=> " & s)
# returns a lot of info lines, ending with "bestmove [...]"
toP.write(&"go depth {DP}\n") # NOTE: using fmt"..." does NOT work here !? : use the '&' syntax..
toP.flush
mySleep()
chgBuffer(app, fmt"<=- go depth {DP}")
elif s == "readyok":
try:
chgBuffer(app,"<=- readyok")
except Exception as e:
echo "FAILED READYOK ***********************"
elif s.startsWith("bestmove"):
TF = false # this will be the last echoed line
# this example script ends here,
# but we could input other UCI commands now (keeping TF true)
#sleep(1000)
toP.write("quit\n")
toP.flush
mySleep()
echo "<=- quit"
discard process.waitForExit
process.close
echo "MSG : process closed"
method view(app: AppState): Widget =
result = gui:
Window:
title = "*** TEST ***"
defaultSize = (1100, 600)
HeaderBar {.addTitlebar.}:
Button {.addLeft.}:
style = [ButtonFlat]
text = "GO"
proc clicked() =
when compileOption("threads"):
createThread(thread, threadProc, app)
ScrolledWindow:
TextView:
margin = 12
buffer = app.buffer
monospace = app.monospace
cursorVisible = app.cursorVisible
editable = app.editable
acceptsTab = app.acceptsTab
indent = app.indent
sensitive = app.sensitive
tooltip = app.tooltip
sizeRequest = app.sizeRequest
proc changed() = discard
#when compileOption("threads") and (defined(gcOrc) or defined(gcArc)):
#when compileOption("threads") and defined(mmArc):
when compileOption("threads"):
let buffer = newTextBuffer()
buffer.text = "first text line\n"
brew(gui(App(buffer = buffer)))
else:
#quit "Compile with --threads:on and --gc:orc to enable threads" # ORG
quit "Compile with --threads:on and --mm:arc to enable threads" |
|
btw. as you see, i used |
|
Sorry, for multi-threading with Owlkettle I can not help you. Multi-threading with GTK is generally not that easy. My GTK4 book may have some examples, or there may be at least some examples in the Gintro package. I think one was drawing a sine wave. But for Owlkettle all may be different. And while for Gintro typical program layout was close to C layout, so we always could ask at Gnome forum, for Owlkettle it may be very different. For threading and channel communication, I gave some examples in my Nim book. I concentrate on learning Rust myself now, which is some fun. I guess that chances that I may ever return to Nim are not that big. Some days ago I found a Rust chess book, see https://rustic-chess.org/front_matter/about_book.html. Looks nice. Have you at least been able to draw a chessboard with Owlkettle? Should be not difficult, when Owlkettle has Cairo or other drawing support -- maybe I could add your board to my salewski-chess. |
today i learned about Channels in my forum post https://forum.nim-lang.org/t/10719 .. user @Isofruit delved into the matter, also according to @Araq, and posted a basic working Owlkettle example, which seems well made .. i'm beginning to understand these techniques .. now i must replace the fixed message (sent by button click) by output of the chess engine, which should probably run in another thread .. it would be great if you could also compile the current Isofruit script and give some feedback .. these days i will study this tech and then post an update .. i did GUI programming before, and rather extensively, but only with TKinter and Python3, which is well documented and complete .. threads are new to me, they can be "locked" by an event, as i understand correctly .. while creating the Nim script as i imagine it, many new terms arrive .. i'm delighted some programmers are willing to help .. running an external process in the GUI and managing its in/output must be rather common, but it was not easy for me to find a solution / examples .. i think my final script can be a good example for your book ? - i will certainly study the Chapters you mentioned |
i have some ideas to create a chess GUI .. presenting a board with pieces should not be too difficult indeed .. first i want to tackle the big bottleneck we're discussing : threads in Nim and how to interact with the binary : set another chess position, take back one move, let it think for x seconds or upto a certain "depth", etc. the salewski-chess engine is GTK4 based, but did you create it with Owlkettle ? Could i have known ? I think the piece font could be better .. i know a lot of nice chess piece fonts exist, i used them, this will not be a problem. |
|
ahhh .. Gintro |
|
learning Rust seems a good idea .. many modern chess engines are built with it .. it also has complete modules with functions to create chess programs .. so, lots of example code for me !? I looked into Julia, because it should run fast, but i discovered not much faster than using PYPY with Python scripts .. and Julia can not be compiled, so to run it the user should have Julia installed .. also GO seems good, in many aspects, but it's all connected to Google, which i try to avoid, just like Microsoft stuff like .NET .. ahwell, i'm a Linux adapt .. |
I never really looked into Owlkettle, and think I newer will. I admire all the energy and work Can Lehman put into it. But all what is possible with Owlkettle, was possible with Gintro for years. It is just that Gintro uses the classical syntax, while Owlkettle rewrote it to a declarative notation. So one more layer to learn. IsoFruit, alias Phil, alias Philip M. Doehner, is someone who recently came to Owlkettle, and he seems to drive it development. But I think he already discovered that GTK is not that easy -- he is the one posting all the questions to StackOverflow. For your mention of Julia and Go: Both interesting languages, but both use a GC and are no true systems programming languages. Rust has some restrictions: No inheritance, no method name overload, no exceptions, no subrange types, no default values for function parameters. Each function parameter needs a type annotation, so max(a, b: int) has to be written max(a: int, b: int). And global variables are nearly unsupported. And of course curly brackets and semicolons everywhere. Coming from Nim, it is hard to accept all these restrictions, but I guess I will manage it. |
- this is not about your engine, but programming Nim in general -
i already wrote some words about it in #2 : how to create code in Nim which runs an external binary (on Linux), esp. a UCI chess engine (like SF), and feed UCI command strings and catch its output ? I'm stuck on this, can you help or point me to a working example ?
see also my recent Issue mdoege/nimTUROCHAMP#4 and its links.
i guess threads are involved, but this subject is difficult for me .. i found some Nim modules / code for "async" and so, but i have no clue ..
The text was updated successfully, but these errors were encountered: