This person or this collective have been transpiling C projects into Pure Go projects, like sqlite[1], libx11[2], xcb[3], and much more[4]… They use their own homegrown transpiler from C99 to Go.[5]
I can only have respect and admiration for whoever is behind this effort :)
Based upon info in an AUTHORS file [0] for a different project by the user cznic, I think there may indeed be some kind of connection with nic.cz ...?
But I believe, based upon my membership on the GoNuts group / mailing list, that this is mostly the work of one individual, Jan Mercl (he is quite active in GoNuts) — as also stated in the previously mentioned AUTHORS file, and in the AUTHORS file for the port of Tcl that is the subject of the OP [1].
I have used some of his non-transpiled code/projects in my own Go projects in the past. He seems to be a very solid coder, often happy to share his views in GoNuts, and also frequently reviews others' code too.
I agree - scrolling through the codebase, I see a lot of unsafe.Pointer usage, which is rarely used in idiomatic Go, and throws a good part of static analysis through the window. So all the bugs in C code would also be in Go code - or might even be worse because of different runtime.
> This sort of code isn't readable, idiomatic nor maintainable
isn't really fair. The original code is hopefully readable, idiomatic and maintainable, and if the transpiler does a correct job, who cares about the output any more than I care about the ugliness of the ASM in my object code when I compile it from source.
It seems fair to me. They only said that it should be seen as something separate than a port, not that it was bad. Honestly I think you're both right.
Yes, it's true that it doesn't matter if you only want to get the code running in a Go environment for whatever reason. But anyone wanting to learn about TCL or idiomatic Go by reading the source code for this project will be disappointed, and that struck me as the main appeal of this when I saw the headline.
If a program was classically built for x86 and was ported to ARM, the compiled output wouldn’t be any more readable. I’m not sure transpiled, which is just compiled by another name, is significant.
The point of GP is to differentiate porting from compiling, whether you call that compiled or transpiled is immaterial, the goal is to differentiate between a codebase which is idiomatic, understandable, and maintainable, and one which is a compiler artefact.
A port would imply and independent maintenance.
Here you wouldn't maintain that codebase, instead you'd fix the upstream or the compiler, and recompile.
Quite the opposite. Port refers to targeting a different environment using the same codebase. The aforementioned ARM port wouldn't require a new codebase to maintain, but rather a new/modified compiler to target the ARM environment. Similarly, a macOS port of Linux software wouldn't require maintaining another codebase, it would require changing the compiler to support a macOS target.
> Here you wouldn't maintain that codebase, instead you'd fix the upstream or the compiler, and recompile.
Exactly. In this case a different compiler that targets the Go environment has been introduced. That does not imply that the compiled output is what should be maintained going forward. You wouldn't maintain the product of an ARM port or macOS port either. It is a port, not a fork. Fork is the word we use when there is implication of independent maintenance.
These are rather unusual definitions. Where I come from, porting to a new platform would mean rewriting platform-specific parts of the app, not coming up with a compiler that magically supports both targets. The choice between forking and refactoring the app into shared and platform-specific modules depends on many factors. Though I'd really like to see a compiler, not wrapper library, that supports both Apple's Core Whatever and your choice of Linux libraries.
On extreme end, porting something from let's say Commodore 64 to Spectrum ZX would be more a complete rewrite.
I also don't really understand the point of this project. Since it's compiled from C, it doesn't get any of possible benefits of using a language that is not C.
> porting to a new platform would mean rewriting platform-specific parts of the app
That may be part of the process, sure. The formal definition isn't clear on exactly what process is necessary to target a new environment, just that some action is necessary. Introducing a new compiler is one possible action to achieve the result.
However, when you do have to modify the code like you suggest, typically you are adding the environment specific aspects in #ifdef-type directives, so you're still not introducing a new codebase to maintain. It's still the same codebase that will be maintained just as it always way.
And even if you want to split hairs, no matter how you slice it there is no implication that the product of the port is what you will maintain. I still have no idea where that is coming from.
> I also don't really understand the point of this project.
To avoid cgo when using the library, I'm sure. Isn't that the point of a lot of cznic/modernc.org's projects?
> The formal definition isn't clear on exactly what process is necessary to target a new environment
I'm interested in the source of this "formal definition", because it doesn't really match anything I have come across over the years.
> typically you are adding the environment specific aspects in #ifdef-type directives
Not really, more like writing platform-specific parts as separate modules. Of course you can write Java for Android and Objective C for iOS in the same file if you really want, but it doesn't make much sense. Or even write a compiler that takes a source file with target-specific paths and transpiles to Java and Objective C if you want to go all in. I've never seen anyone do that.
> I'm interested in the source of this "formal definition", because it doesn't really match anything I have come across over the years.
"In software engineering, porting is the process of adapting software for the purpose of achieving some form of execution in a computing environment that is different from the one that a given program (meant for such execution) was originally designed for (e.g., different CPU, operating system, or third party library)."
Here we have software (Tcl) that was adapted to work in a computing environment (Go) that is different than the given program was originally designed for. This doesn't seem very controversial.
> Of course you can write Java for Android and Objective C for iOS in the same file if you really want, but it doesn't make much sense.
If you are changing the language software is written in, it is a stretch to call that porting. That isn't even a fork. That is a new project.
Compiling can also be done once and the object files can be published on Github if you for some reason want to do that. As far as I understand, this "port" is transpiled from C source and the Go source files are compiler artifacts just like object files.
Nor is transpiling - or compiling as it is more commonly known - to another CPU architecture or operating system, but that’s not what we are talking about here. The topic of the day is porting. Porting and compiling are quite different.
What I am to see? This Tcl port is clearly not source-to-source compiled. Everyone in this thread, including yourself as far as I can tell, agree that the product of this port is not usable as the source. The original Tcl codebase written in C remains the source and must be compiled each time it is modified to keep this Go package in sync. Hell, this entire thread started because someone pointed out that the Go target output isn't usable as the source...
The Go project's gc compiler was source-to-source compiled from its original C codebase into a Go codebase, which became the new source as the original C code was abandoned. Is this the origin of your confusion? That is a strange place to arrive just because Go is in the title, though.
Or is it that you also don't understand what the word source means?
I don't understand the logic of the argument. Let's go through the claims. First,
> Compiling is repeatable, however. Porting is done once.
Second,
> The original Tcl codebase written in C remains the source and must be compiled each time it is modified to keep this Go package in sync.
which implies that it is done more than once. If you are talking about something else than the Go "port" of tcl, it is not really obvious to me what it is.
I'm not sure what the struggle is here. The terms are well defined and well understood. Porting is the process of adapting software to work in an environment it was not originally designed for. Compiling is the process of transforming code written in one language into another language.
In the case in question, Tcl was both ported and compiled to Go. There was a process required to allow the software to run in an environment it was not originally designed for, and there was a process required to transform the software into another language.
Said compiler is one of the things that was produced to allow porting to the new environment. The compiler is not the port. Porting is not compiling. We've been over that many times now.
Ok, the confusing parts are first the computing environment: So, Tcl was ported from Unix to Unix. To me they sound the some environment, but I guess you could just claim otherwise. Second: The "port" was achieved by source-to-source compilation and is unmaintainable by itself. Yet somehow "Porting is not compiling." Where exactly is the port?
Cool, this is excellent. AOL embedded TCL interpreters attached to a listener on the network for all the services, so you could telnet in and see in memory what was going on and change config and what not. A tcl command is just a CS 1 program with argv and argc, so even pretty junior people could safely add commands. The framework was event based, no threads, so while the tcl was running nothing else was being done, so you could rearrange memory and it was cool as long as you finished before the end of the command. If would be pretty simple to add in a control port thing to Go. The advantage of TCL with its simple syntax is most commands are just cmd arg1 arg2 arg3 but if you have some general logic, you can express that as.
This is horrifying. Thanks for the nightmare fuel.
> while the tcl was running nothing else was being done, so you could rearrange memory and it was cool as long as you finished before the end of the command
My mind immediately springs to all the ways this can go horrifically wrong.
Tcl uses a memory model internally where each piece of memory belongs strictly to the thread that allocated it, with this enforced in lots of places. There are a few loopholes past it (for process-wide concepts such as the current working directory cache, or for inter-thread messaging) but by and large you write what appears to be single-threaded code.
There's support for deep coroutines and non-blocking I/O so it isn't limiting. You only really need multiple threads when dealing with compute-heavy code or a particularly crufty API.
What server are you talking about? I distinctly remember naviserver/aolserver being multithreaded and being able to share information between threads using a specific set of nsv_* commands.
SAPI - the Server API C stack. AOL server was a web surfer purchased from some small innovative company. It did also have TCL but was threaded. Way harder to extend then the No Threads Kernel that back-ended all the classic AOL clients and the AIM clients. I wrote the ns_flap.so shared library for the threaded AOLserver and ai maintained the non-threaded flap library ( a simple framing layer for TCP connections https://handwiki.org/wiki/OSCAR_protocol#FLAP_header ).
It was desgin3d to be easily horizontally scalable, and to be easy to write code for that was stable. We would put one process per CPU, and leave one for the OS. Also would put in one Ethernet card per process, if it was one that talked to the internet, and then used a user-land TCP stack on top of DLPI (?).
The services there were all high volume but not super latency sensitive, so. A poll/select loop that spun a thousand times a second or so was fine. You could add or reload all kinds of config without restarting and hence without losing any traffic. It was designed for zero downtime maintenance.
They weren’t, mostly, web servers, but handled various proprietary network formats from the proprietary clients and amongst themselves. Point to point mesh connect for AIM, hub and spoke for AOL, smart clients that could route to the appropriate endpoint based on the message contents. AIM had a “reconnect here” host to client message so you could dynamically move load around for maintenance. It was a type of select or pool loop like libuv, but before we had more than a few hundred hosts to interconnect. He entire internet and all client traffic came thru one file descriptor for the dedicated Ethernet for that process. It was all straight C, timers, Various layers of network protocols (tcp, ssl, flap, etc. the AOL client servers had lots of international localizations, which could be reloaded via a TCL command. You could see histograms of al the network latencies, at various levels, in a TCL command. I think at some point we added times to the prefix of allocated memory and had histograms on memory lifetimes. (Mostly we only allocated large chunks of memory, then at steady state would shuffle stuff of of a free list.
There were a few HTTP servers written for things like AOLTV and as an RPC gateway from third parties into the “host complex” e.g. one called ewoks was used for user registration. It was headless, only gave out 302 replies or 4xx replies.
One of my first jobs was in Tcl and I still wake up with night sweats about it. Some languages were just meant to die. I have no idea why anyone would be interested in using it in 2022.
I had to use Tcl/Tk for some computational chemistry circa 2005. I'm sure it seemed amazing in the early 90s for quick GUI apps, but 10 years later it was just an absolute nightmare compared to powerful languages like Python.
Some of my graduate level work was in Tcl, and my first job used a lot of Tcl. I have fond memories of the time, and Tcl is one of my favorite languages to this day. It's power and flexibility are fantastic.
My first job too! So true. Everyone knows how easy it is to make serious quoting mistakes in a Bourne shell script, but Tcl takes the risk to a whole new danger level.
But any harm done by Tcl is more than redeemed by its creator being the author of one of my favorite of all programming books, A Philosophy of Software Design.
It is the de facto scripting language in the EDA world unfortunately. I agree though. Terrible language. I initially thought this was going the other way round so you could write your EDA scripts in Go. No idea why you would want to use TCL when you can use Go.
I briefly worked on a WASM to TCL transpiler so I could write EDA scripts in Rust. Got it kind of working but WASM is actually quite large and TCL integers in the version I was forced to use (thanks PowerArtist) are completely broken so it ended up being not worth it.
many years ago, when my ex-girlfriend was in college, for whatever reason she managed to get herself into a course that involve programming and for some unknown reasons, they were asked to do it in tcl.
obviously I was the go to expert on everything related to computers and I still remember the shock when she told me that I need to spend my weekends on helping her to understand tcl.
My first long term job was in tcl and I have nothing but love for that language. Extremely simple syntax wise with the ability to extend the language whenever you wanted. Still makes other languages look bad imo. Only experience that came close was clojure. There is so much power in tcls expressiveness.
Is "ported" the right term here? It know the repo's README says "CGo-free port", but this is the C version of TCL transpiled from C to Go (see the ~13MB .go files per platform in the "lib" directory). Which is a very cool idea, and the author has done the same thing with SQLite, to avoid CGo (https://gitlab.com/cznic/sqlite).
This is of course generated code (output from the C to Go converter), so to "easily maintain" it, you'd edit the Tcl C source files and re-generate these. The generated Go code is committed to the repo so the person using the library doesn't need the C-to-Go converter tools.
But seriously, I would expect most editors to handle large files relatively gracefully, even if they aren’t as fast as vim for example. Browser based code viewers on the other hand, I grant you, aren’t generally up to the task.
I know TCL has an interesting story about how the syntax is basically all embedded strings, or something. Does anyone know if it has a recursive structure, as in lisp? Maybe it's more of an "everything is an X" structure, as in forth where everything is a "word".
The entire grammar is spelled out in the tcl(n) man page[0]. It’s amazingly concise and easy to learn from just this. To answer your question, it’s “as if” everything was a string, but the implementation is reasonably efficient and doesn’t convert all the time.
Yes, in Tcl everything is basically a string, but has Lisp-like recursive structures: An expression like `{foo {bar baz}}` evaluates to the string "foo {bar baz}" at first, but then that might be split into two strings "foo" and "bar baz" by the interpreter.
I recall Tcl being described as a Lisp without the parenthesis many years ago, which I still think is somewhat apt.
Antirez' article "Tcl the Misunderstood" [0] is well worth a read if you want to know more.
He also has an interpreter, Picol [1], in 500 lines of C code that is very easy to read and to understand.
Would love to find some kind of write up by CZNIC about why they're doing this.
Tcl can still be found in a few places powering some pretty critical infrastructure, I know a few networks running with significant automation being driven with Tcl
So we know this is transpiled, wondering what is the usual approach to determine the quality of the transpliation output? Any systematic work into this aspect?
I was wondering how this was done entirely without CGO since hooking up the transpiled code to a native Go X client like xgb sounds really hard. Apparently it uses a transpiled version of libX11 and freetype. Which seems pretty crazy but I guess there is no reason it couldn't work once you have a robust translator.
Tcl examples? I assume it's just a standard Tcl interpreter, so should run standard Tcl code. So you should be able to find examples anywhere online... e.g. https://www.tcl.tk/about/language.html
Haha, indeed - I also greatly prefer Tk to Qt or Electron.
Some things I love about Tk:
- adds a cool 1990s retro aesthetic
- usable in X11 tunneled over an SSH connection
- doesn't consume all available CPU and memory resources
- the Tk canvas is a beautiful thing
This person or this collective have been transpiling C projects into Pure Go projects, like sqlite[1], libx11[2], xcb[3], and much more[4]… They use their own homegrown transpiler from C99 to Go.[5]
I can only have respect and admiration for whoever is behind this effort :)
[1] https://modernc.org/sqlite
[2] https://gitlab.com/cznic/x11
[3] https://modernc.org/xcb
[4] https://gitlab.com/users/cznic/projects
[5] https://gitlab.com/cznic/ccgo