X Macros are a classic C++ pattern. Less known than some of the more popular techniques like curiously recurring template pattern, but still quite common in large codebases.
(Although... it is a neat trick, but... It is kind of mostly useful because C++ macros are not very powerful. If they were more powerful, most uses of X Macros could be replaced by just having a single macro do all of the magic.)
I recently saw something I hadn't seen before in a similar vein. There's a million different bin2c generators, and most of them will generate a declaration for you. However, I just saw a setup where the bin2c conversion just generates the actual byte array text, e.g. "0x00, 0x01, ..." so that you could #include it into whatever declaration you want. Of course, this is basically a poor man's #embed, but I found it intriguing nonetheless. (It's nice that programming languages have been adding file embedding as a first-class feature. This really simplifies a lot of stuff and removes the need to rely on more complex solutions for problems that don't really warrant them.)
https://en.wikipedia.org/wiki/X_macro claims they go back to the 01960s, but unfortunately the reference is to a DDJ article. Is there a chance we could like run a Kickstarter to buy DDJ's archives and fix their website?
You could try, I contacted the new owners awhile ago. They don't/won't even sell new copies of the archive dvd. I tried. https://news.ycombinator.com/item?id=14345682 but a kind soul responded to that request :)
Sad part is that the DVD doesn't have issues going back to when it started as "Dr. Dobb's Journal of COMPUTER Calisthenics & Orthodontia Running Light Without Overbyte"
I have a like 81-84 that I should just send to the IA.
The Living Computer Museum had a small library that had the complete archive of Dr Dobbs, but who knows where it went after Paul Allen's sister closed it and sold everything off.
Y'know, I've never actually seen this before, but that makes a lot of sense. They're also (of course) a common pattern in C, and probably basically any language that has a C-style preprocessor.
This is really more a C pattern than a C++ pattern, isn't it?
Frustrated by C's limitations relative to Golang, last year I sketched out an approach to using X-macros to define VNC protocol message types as structs with automatically-generated serialization and deserialization code. So for example you would define the VNC KeyEvent message as follows:
And that would generate a KeyEvent typedef (to an anonymous struct type) and two functions named read_KeyEvent_big_endian and write_KeyEvent_big_endian. (It wouldn't be difficult to add debug_print_KeyEvent.) Since KeyEvent is a typedef, you can use it as a field type in other, larger structs just like u8 and u32.
Note that here there are two Xes, and they are passed as parameters to the KeyEvent_fields macro rather than being globally defined and undefined over time. To me this feels cleaner than the traditional way.
What I love are macros that take parameters that are the names of other macros to call, and parameters that are fragments of code without balanced brackets to insert.
The term of art for that and the X pattern kind of approach is "macrology".
An infamous example is Microsoft's original C preprocessor macros for generating COM components for C code.
MFC infamously used a similar approach for its COM/OLE bindings to C++ too. You could include the same interface definition .h files in either C or C++ code with the proper meta-macros defined to declare the same interfaces in either C or C++ code (which was kind of the whole point of COM).
That was long before the (in contrast) much more elegant ATL (ActiveX Template Library) implemented with C++ templates.
Microsoft isn't the only offender -- Sun did it to!
Check out the "arctochain.c" code in the original NeWS server by Vaughan Pratt and James Gosling for a great example of macros that take the names of other macros and code fragments containing gotos, to make a state machine with a twisty maze of hopping gotos, plus some manual exception handling gotology like "goto this_curve_is_trash;" and "goto straight;" thrown in for fun:
/*-
Convert a conic arc to a curve
arctochain.c, Tue Jun 11 14:23:17 1985
"Elegance and truth are inversely related." -- Becker's Razor
Vaughan Pratt
Sun Microsystems
This code is a version of the conix package that has been
heavily massaged by James Gosling.
*/
>[...] Two pieces of work were done at SUN which provide other key components of the solution to the imaging problems. One is Vaughan Pratt's Conix [53], a package for quickly manipulating curve bounded regions, and the other is Craig Taylor's Pixscene [63], a package for performing graphics operations in overlapped layers of bitmaps. [...]
>[...] Pixscene is based on a shape algebra package. The ability, provided by Conix, to do algebra very rapidly on curves should make non-rectangular windows perform well. [...]
>The main loop of the NeWS PostScript interpreter itself is the realization of Duff's dire warning "Actually, I have another revolting way to use switches to implement interrupt driven state machines but it's too horrid to go into."
#define case4(tag) case tag: case tag+1: case tag+2: case tag+3
#define case16(tag) case4(tag):case4(tag+4):case4(tag+8):case4(tag+12)
#define case32(tag) case16(tag):case16(tag+16)
I hate X-Macros, but they're very useful in some situations. We use them to generate an enum of error values and also a string table of those names. (Unlike two separate tables, the X macro version is guaranteed to be the exact correct length / values align with the corresponding string.)
That's been my primary usage of them. Another is to create a list of config file options (or command line arguments) like
X(foo, int, "set the number of foos")
X(filename, std::string, "input filename")
X(verbose, bool, "verbose logging")
which can then be used to (a) generate the fields of a config struct, define the mapping from string to field (using the stringifying macro operators), define what functions to use for parsing each field, create the help message, etc. Basically like `argparse` or `clap` but much hackier.
As gross as they are, the ability to define one table of data that's used multiple ways in multiple places is handy.
I understand the use case of this, but when I see it I always wonder if, and think I would prefer, some external code generation step instead rather than falling back on macros in the preprocessor. Like an external script or something.
Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with.
Compare with a solution that's 100% standard C++, integrates into your build with zero work, can be immediately understood by anyone reasonably skilled in the language, and puts the "generated" code right where it belongs.
CMake makes this pretty painless. My codegen targets have only two additional instructions to handle the generation itself and dependencies: add_custom_command to call the codegen exec, and then add_custom_target to wrap my outputs in a "virtual" target I can then make the rest of my program depend on, but this is just for tidying up.
And I'll dispute the fact that any complex C prepro task "can be immediately understood by anyone reasonably skilled in the language". Besides, code should ideally be understood by "anyone reasonably likely to look at this code to work in it", not "reasonably skilled".
This isn't complex. It's a bit unusual, but not hard to understand if you understand the basics of how #include and #define work.
If you're working on the sort of C++ codebase that would benefit from this sort of code generation, and you're not reasonably skilled in C++, then god help you.
Are you talking about the X macro itself, or more generally?
I may be the obtuse one here, but for a more complex example, it took me a few hours to manage to make nested loops using Boost PP (for explicit instantiations). Even so, I avoid having to write a new one that's not a quick copy-paste because it's quite different from usual C++ programming, so my painfully acquired understanding quickly evaporated... as I suspect is the case of anyone who doesn't particularly focus on the C prepro.
In the end, it's just simpler to get some Python script or C++ program to write a string and dump that to a file than to write something illegible with the C preprocessor, if doing something at all complicated (in my opinion).
I'm talking about X-macros. There's a wide range of preprocessor shenanigans, from "everybody needs to know this" to "oh my god why." Each construct needs to be evaluated on its merits. IMO X-macros are closer to the simpler side of that spectrum. Consider writing things out by hand if you just have a few, but if you have a lot of things repeating like this, they're a fine tool to use. Boost PP is a whole different level of ridiculousness and I don't see ever using that sort of thing for anything serious.
> Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with.
The preprocessor is already a bespoke language embedded in your C++, and code written in it is generally harder to maintain than, like, Python.
The cost of doing something non-standard is real, but not infinite; at some point the benefit in code maintainability and sanity is worth it.
I agree that you can go too far with it and it becomes better to do it a different way, but the X-macros technique is straightforward and easy to understand.
I've done this in C with the C preprocessor and Java with m4[0].
The upside of doing it natively is that it keeps the build simpler. And everybody at least knows about the existence of the C preprocessor, even if they don't know it well. And it's fairly limited, which prevents you from getting too clever.
The big downside of doing it with the C preprocessor is that the resulting code looks like vomit if it's more than a line or two because of the lack of line breaks in the generated code. Debugging it is unenjoyable. I'd recommend against doing anything super clever.
The upside of doing it out of band is that your generated source files look decent. m4 tends to introduce a little extra whitespace, but it's nothing objectionable. Plus you get more power if you really need it.
The downside is that almost nobody knows m4[1]. If you choose something else, it becomes a question of what, does anyone else know it, and is it available everywhere you need to build.
Honestly, integrating m4 into the build in ant really wasn't too bad. We were building on one OS on two different architectures. For anything truly cross-platform, you'll likely run into all the usual issues.
ETA: Getting an IDE to understand the out of band generation might be a hassle, as other folks have mentioned. I'm a vim kinda guy for most coding, and doing it either way was pretty frictionless. The generated java code was read-only and trivial, so there wasn't a lot of reason to ever look at it. By the time you get to debugging, it would entirely transparent because you're just looking at another set of java files.
[0] This was so long ago, I no longer remember why it seemed like a good idea. I think there was an interface, a trivial implementation, and some other thing? Maybe something JNI-related? At least at first, things were changing often enough that I didn't want to have to keep three things in sync by hand.
[1] Including me. I re-learn just enough to get done with the job at hand every time I need it.
This is what I do, these days. Whenever I would previously have reached for X-macros or some other macro hack, I tend to use Cog [1] now instead.
It's quite a clever design; you write Python to generate your C++ code and put it inside a comment. Then when you run the Cog tool on your source file, it writes the generated code directly into your C++ file right after your comment (and before a matching "end" comment).
This is great because you don't need Cog itself to build your project, and your IDE still understands your C++ code. I've also got used to being able to see the results of my code generation, and going back to normal macros feels a bit like fiddling around in the dark now.
IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. IDEs also can expand the macro invocations. So, I prefer the macros when possible :-).
The C# "source generator" approach is a good compromise; it runs within the build chain so has the ease-of-use of macros in that respect, but they don't need to be written in a weird macro language (they are C# or can call external tool) and when you debug your program, you debug through the generated source and can see it, more accessible than macros. Not sure if there is something similar in C/C++ integrated with the common toolchains.
But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.
> But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.
Me to, and that's even in Lisp!
Preprocessor macros are hard and bugprone because they share the failings of Unix philosophy of "text as universal interface" - you're playing with unstructured (or semi-structured) pieces of text, devoid of all semantics. And this is also what makes them occasionally useful - some code transformations are much, much easier to do when you can manipulate the text form directly, ignoring syntax and grammar and everything.
Only the final value must be correct code - starting point and intermediary values can be anything, and you don't need to make sure you can get from here to there through valid data transformations. This is a really powerful capability to have.
(I also explicitly compared preprocessor macros to "plaintext everything" experience that's seen as divine wisdom, to say: y'all are slinging unstructured text down the pipes way too much, and using preprocessor way too little.)
Using the C preprocessor is standard, available, compatible and the major usage patterns are "known". For a lot of cases, they're way easier to reason about rather than learning how an external generation tool is used to generate the code. In order to understand these macros all I need is to read the source code where they're used.
Nothing C++ related in the pattern though. This C preprocessor trickery is practically so classic you couldn't necessarily even call it a "trick".
After trying to wrangle Boost PP and other advertised compile-time libraries such as Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values), I've finally converged to simply writing C++ files that write other C++ files. Could be Python, but I rather keep the build simple in my C++ project. Code generation is painless with CMake, no idea with other build configuration utilities.
CMake has a particularly irritating flaw here, though, in that it makes no distinction between host and target which cross-compiling, which makes it really difficult to do this kind of code generation when supporting this use-case (which is becoming more and more commoon).
Right, I hadn't thought of that, to be honest. If I understand correctly, you're saying the codegen targets will be compiled to the target arch, and then can't be run on the machine doing the compiling?
I think one solution might be to use target_compile_options() which lets you specify flags per target (instead of globally), assuming you're passing flags to specify the target architecture.
That only works if it's mostly the same compiler, unfortunately. They could be completely different executables, calling conventions, etc. I don't know why CMake still has such a huge hole in its feature set, but it's quite unfortunate.
One case I benchmarked was Bernstein/Bézier and Lagrange element evaluation. This is: given a degree d triangle or tetrahedron, given some barycentric coordinates, get the physical coordinate and the Jacobian matrix of the mapping.
"Runtime" here means everything is done using runtime loops, "Hana" using Boost Hana to make loops compile-time and use some constexpr ordering arrays, "hardcoded" is a very Fortran-looking function with all hardcoded indices and operations all unrolled.
As you see, using Boost Hana does bring about some improvement, but there is still a factor 2x between that and hardcoded. This is all compiled with Release optimization flags. Technically, the Hana implementation is doing the same operations in the same order as the hardcoded version, all indices known at compile time, which is why I say there must be some runtime overhead to using hana::while.
In the case of Bernstein elements, the best solution is to use de Casteljau's recursive algorithm using templates (10x to 15x speedup to runtime recursive depending on degree). But not everything recasts itself nicely as a recursive algorithm, or I didn't find the way for Lagrange anyways. I did enable flto as, from my understanding (looking at call stacks), hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating)
Similar results to compute Bernstein coefficients of the Jacobian matrix determinant of a Q2 tetrahedron, factor 5x from "runtime" to "hana" (only difference is for loops become hana::whiles), factor 3x from "hana" to "hardcoded" (the loops are unrolled). So a factor 15x between naive C++ and code generated files. In the case of this function in particular, we have 4 nested loops, it's branching hell where continues are hit very often.
That would be fairly interesting to look at the actual code you've used, and have a look at the codegen. By a chance, is it viable for you to open-source it? I'd guess it should bear lots of interest for Hana author/s.
What compiler/version did you use? For example, MSVC isn't (at least wasn't) good at always evaluating `constexpr` in compile-time...
> hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating)
Hmm, I'd say it (LTO) shouldn't influence, as these lambdas are already fully visible to a compiler.
I never thought to contact them, but I might do that, thanks for the suggestion. This is something I tested almost two years ago, I have these benchmarks written down but I've since deleted the code I've used, save for the optimal implementations (though it wouldn't take too long to rewrite it).
I tested with clang on my Mac laptop and gcc on a Linux workstation. Version, not sure. If I test this again to contact the Hana people, I'll try and give all this information. I did test the constexpr ordering arrays by making sure I can pass, say, arr[0] as a template parameter. This is only possible if the value is known at compile time. Though it's also possible the compiler could be lazy in other contexts, as in not actually evaluating at compile time if it figures out the result is not necessary to be known at compile time.
Oh yeah, you're right, I was confusing translation unit and function scope.
Yeah, it's all done automatically when you build, and dependencies are properly taken into account: if you modify one of the code generating sources, its outputs are regenerated, and everything that depends on them is correctly recompiled. This doesn't take much CMake logic at all to make work.
In my case, no, it's dumb old code writing strings and dumping that to files. You could do whatever you want in there, it's just a program that writes source files.
I do use some template metaprogramming where it's practical versus code generation, and Boost Hana provides some algorithmic facilities at compile time but those incur some runtime cost. For instance, you can write a while loop with bounds evaluated at compile time, that lets you use its index as a template parameter or evaluate constexpr functions on. But sometimes the best solution has been (for me, performance/complexity wise) to just write dumb files that hardcode things for different cases.
External codegen introduces a lot of friction in random places. Like how your editor can no longer understand the file before you start building. Or how it can go out of date with respect to the rest of your code until you build. If you can do it with a macro it tends to work better than codegen in some ways.
"X" macros are great until you need two of them visible in the same translation unit. It is much better to pass a list macro as an argument to a uniquely named X macro and avoid the need to ever undef anything.
Basically, instead of faffing around with undefing values and including different files, you define your list like this:
#define AN_X_LIST(X) \
X(foo, bar) \
X(bar, baz)
And then you use it like so:
#define AN_ASSIGNMENT_STATEMENT(a,b) a = STRINGIFY(b);
And so
AN_X_LIST(AN_ASSIGNMENT_STATEMENT)
Will expand to
foo = "bar";
bar = "baz";
The nice thing about this approach is you can define multiple lists and macros, and make higher order macros which use them. I have a system like this which allows me to define reflective structs in C++ easily, i.e. I define a struct like:
(where DECLARE_REFLECTIVE_STRUCT basically just does the same dance as above with passing different per-element structs into the list that it is passed for the struct definition, and other utility functions associated with it)
which then makes a struct Foo with members bar and baz with the right types and default values, but also I can do 'foo_instance.fetch_variant("baz")' and other such operations.
The biggest pain with this approach is it's basically dealing with a bunch of multi-line macros, so it can get messy if there's a typo somewhere (and I strongly recommend an auto-formatter if you don't like having a ragged line of backslashes to the right of all the code that uses it).
Some minor tweaks to what the author shows to make it even better.
Give the macro a more descriptive name. For their example, call it GLOBAL_STRING instead of X. I think this helps make things clearer.
#undef the macro at the end of the header. That removes one line of boilerplate from every use of the header.
Use #ifndef at the top of the header and emit a nice error if the macro isn't defined. This will make it easier to understand what's wrong if you forget the #define or misspell the macro.
despite the extra boilerplate, I feel like it's still better to undef the macros in the same scope they were defined in, so that they clearly delimit the code in which the macro is active.
I can't believe this actually has a name (is this an attempt to make fetch happen?) and it's not considered an anti-pattern. This sort of preprocessor/code mixing is impossible to debug and maintain; most senior C++ programmers have advised people to avoid doing this since the 1990's.
There is a role for the preprocessor to automate simple tables in C and such, but anything complex should be avoided like the plague if you don't like tech debt. And there's not much excuse for using it in C++ to this extent.
The vast majority of the use of X macros (and to be clear, they are not that common) I have seen in the wild have been very simple uses that are quite straightforward to debug.
Should we have a better solution in 2025? Yes. Lacking that, are X macros better than other workarounds? Also yes.
It's not particularly difficult to deal with, as far as C++ goes (I'd take it over most template shenanigans, the rules are a little simpler). It would be easier if C++ had ever tried to improve the preprocessor instead of just saying "don't use it" and trying to get templates to do a subset of the same kind of things you might want the preprocessor for.
(Also, there's a much better version of this I decribed above: mucking about with re-including the same file multiple times is unnecessary and constraints things too much. You can make a much nicer interface for users of the macro with only a little extra effort)
It is a clever trick. Very useful in C also, maybe more than in C++.
It can be overused, though.
Kind of works like Rust declarative macros (`macro_rules!`) in that it is often used to repeat and expand something common across an axis of things that vary.
It's funny that the simple name X Macros has stuck and is a de facto standard. E.g. https://en.wikipedia.org/wiki/X_macro suggests they date to the 1960s.
Famous in IBM 360 assembly language, in particular these were used for building code based on data schemas in CICS [1] applications which were remarkably similar to 2000-era cgi-bin applications in that you drew forms on 3270 terminals [2] and instead of sending a character for each keystroke like the typical minicomputer terminal, users would hit a button to submit the form, like an HTML form, and the application mostly shuttled data between forms and the database.
At first I though this was just normal macros, not X-macros. But you're right, and at first I just wasn't understanding it. In linux/include/trace/stages/stage1_struct_define.h, linux/include/trace/stages/stage2_data_offsets.h, etc., we see the different definitions of __field and __array.
Also, in linux/include/linux/genl_magic_struct.h we see multiple different definitions of __field and __array, each followed by #include GENL_MAGIC_INCLUDE_FILE, and I had no idea you could even source the name of a file to #include from a preprocessor macro! But that's a different __array with a different number of arguments.
One the one hand it is great and probably usable to solve actual problems.
On the other hand it seems fishy that you need all these hacks to do it. There must be a way simpler language than C++ where you could do the same easier.
>These are some highlights from a prescient talk by Bill Joy in February of 1991.
>“It’s vintage wnj. When assessing wnj-speak, remember Eric Schmidt’s comment that Bill is almost always qualitatively right, but the time scale is sometimes wrong.” -David Hough
>C++++-=
>“C++++-= is the new language that is a little more than C++ and a lot less.”
-Bill Joy
>In this talk from 1991, Bill Joy predicts a new hypothetical language that he calls “C++++-=”, which adds some things to C++, and takes away some other things.
>Oak
>It’s no co-incidence that in 1991, James Gosling started developing a programming language called Oak, which later evolved into Java.
>“Java is C++ without the guns, knives, and clubs.”
-James Gosling
>Fortunately James had the sense to name his language after the tree growing outside his office window, instead of calling it “C++++-=”. (Bill and James also have very different tastes in text editors, too!)
i hate this "trick" with a passion. i am the one who winds up having to debug this shit, and theres nothing i love doing more than trawling through hundreds of thousands of preprocessed C++ that gets defecated by `gcc -E ...`. even better when the macro expansion is 10 levels deep.
you need tables? use python (or your scripting language of choice) as a pre-compile step that generates the .cpp/.c/.h files, and then build that.
I don't quite understand this comment. Why would adding another preprocessing step using another tool make it easier? I always found it nice that you can use 'gcc -E' and inspect the output. Of yours, 10 levels of macro expansion are a problem, but simple X macros would not usually do this.
X Macros are a classic C++ pattern. Less known than some of the more popular techniques like curiously recurring template pattern, but still quite common in large codebases.
(Although... it is a neat trick, but... It is kind of mostly useful because C++ macros are not very powerful. If they were more powerful, most uses of X Macros could be replaced by just having a single macro do all of the magic.)
I recently saw something I hadn't seen before in a similar vein. There's a million different bin2c generators, and most of them will generate a declaration for you. However, I just saw a setup where the bin2c conversion just generates the actual byte array text, e.g. "0x00, 0x01, ..." so that you could #include it into whatever declaration you want. Of course, this is basically a poor man's #embed, but I found it intriguing nonetheless. (It's nice that programming languages have been adding file embedding as a first-class feature. This really simplifies a lot of stuff and removes the need to rely on more complex solutions for problems that don't really warrant them.)
> X Macros are a classic C++ pattern
I've seen them in 1970s assembler code.
https://en.wikipedia.org/wiki/X_macro claims they go back to the 01960s, but unfortunately the reference is to a DDJ article. Is there a chance we could like run a Kickstarter to buy DDJ's archives and fix their website?
You could try, I contacted the new owners awhile ago. They don't/won't even sell new copies of the archive dvd. I tried. https://news.ycombinator.com/item?id=14345682 but a kind soul responded to that request :)
https://archive.org/search?query=dr+dobbs
https://archive.org/details/DrDobbsDVD5
https://archive.org/details/DDJDVD6
Sad part is that the DVD doesn't have issues going back to when it started as "Dr. Dobb's Journal of COMPUTER Calisthenics & Orthodontia Running Light Without Overbyte"
https://archive.org/details/dr_dobbs_journal_vol_01_201710
I have a like 81-84 that I should just send to the IA.
The Living Computer Museum had a small library that had the complete archive of Dr Dobbs, but who knows where it went after Paul Allen's sister closed it and sold everything off.
I regret to inform you that 01960 is an invalid octal literal.
Not in K&R!
Y'know, I've never actually seen this before, but that makes a lot of sense. They're also (of course) a common pattern in C, and probably basically any language that has a C-style preprocessor.
> If they were more powerful, most uses of X Macros could be replaced by just having a single macro do all of the magic
You can do exactly that, it is just very tedious without a support preprocessor library like boost.PP.
> If they were more powerful, most uses of X Macros could be replaced by just having a single macro do all of the magic
While this is probably true, I once used this pattern in Rust, which has way more powerful macro system (and also a powerful generic system).
This is really more a C pattern than a C++ pattern, isn't it?
Frustrated by C's limitations relative to Golang, last year I sketched out an approach to using X-macros to define VNC protocol message types as structs with automatically-generated serialization and deserialization code. So for example you would define the VNC KeyEvent message as follows:
And that would generate a KeyEvent typedef (to an anonymous struct type) and two functions named read_KeyEvent_big_endian and write_KeyEvent_big_endian. (It wouldn't be difficult to add debug_print_KeyEvent.) Since KeyEvent is a typedef, you can use it as a field type in other, larger structs just like u8 and u32.Note that here there are two Xes, and they are passed as parameters to the KeyEvent_fields macro rather than being globally defined and undefined over time. To me this feels cleaner than the traditional way.
The usage above is in http://canonical.org/~kragen/sw/dev3/binmsg_cpp.c, MESSAGE_TYPE and its ilk are in http://canonical.org/~kragen/sw/dev3/binmsg_cpp.h, and an alternative approach using Python instead of the C preprocessor to generate the required C is in http://canonical.org/~kragen/sw/dev3/binmsg.py.
> This is really more a C pattern than a C++ pattern, isn't it?
You're right, although it feels a bit safer in C++ with the extra type safety. And that in my mind makes it more powerful in C++.
What I love are macros that take parameters that are the names of other macros to call, and parameters that are fragments of code without balanced brackets to insert.
The term of art for that and the X pattern kind of approach is "macrology".
An infamous example is Microsoft's original C preprocessor macros for generating COM components for C code.
MFC infamously used a similar approach for its COM/OLE bindings to C++ too. You could include the same interface definition .h files in either C or C++ code with the proper meta-macros defined to declare the same interfaces in either C or C++ code (which was kind of the whole point of COM).
That was long before the (in contrast) much more elegant ATL (ActiveX Template Library) implemented with C++ templates.
Microsoft isn't the only offender -- Sun did it to!
Check out the "arctochain.c" code in the original NeWS server by Vaughan Pratt and James Gosling for a great example of macros that take the names of other macros and code fragments containing gotos, to make a state machine with a twisty maze of hopping gotos, plus some manual exception handling gotology like "goto this_curve_is_trash;" and "goto straight;" thrown in for fun:
https://donhopkins.com/home/NeWS/SUN/src/server/graphics/arc...
More NeWS macrology:https://news.ycombinator.com/item?id=15680904
>[...] Two pieces of work were done at SUN which provide other key components of the solution to the imaging problems. One is Vaughan Pratt's Conix [53], a package for quickly manipulating curve bounded regions, and the other is Craig Taylor's Pixscene [63], a package for performing graphics operations in overlapped layers of bitmaps. [...]
>[...] Pixscene is based on a shape algebra package. The ability, provided by Conix, to do algebra very rapidly on curves should make non-rectangular windows perform well. [...]
>[53] Pratt, V., Techniques for Conic Splines, Computer Graphics 19(3), pp.151159 (1985).
>[63] Taylor, C. and Pratt, V., A Technique for Representing Layered Images. SUN Microsystems Inc., 2550 Garcia Avenue, Mountain View, CA94043.
Techniques for Conic Splines:
https://dl.acm.org/doi/pdf/10.1145/325334.325225
>The main loop of the NeWS PostScript interpreter itself is the realization of Duff's dire warning "Actually, I have another revolting way to use switches to implement interrupt driven state machines but it's too horrid to go into."
https://news.ycombinator.com/item?id=4360306
https://www.lysator.liu.se/c/duffs-device.html
https://donhopkins.com/home/code/NeWS/SUN/src/server/nucleus...
The blocking PostScript parser also has some charming macrology, by the same guy who brought you Java, written in the morning of New Years Eve 1984:
https://donhopkins.com/home/code/NeWS/SUN/src/server/nucleus...
These are awesome! I didn't realize NeWS had been released in source form, or that Pratt had worked on it. Does it build today?
`goto straight;` sounds appalling. Can't escape heteronormativity even in NeWS!
Alas, it's quite dependent on old compilers and libraries, and only really useful for "educational and research purposes".
With all due respect, in that code, "goto straight;" is only for boring degenerate conics, as it should be.
There's a funny commented out line with a call to an undefined function that I love:
I hate X-Macros, but they're very useful in some situations. We use them to generate an enum of error values and also a string table of those names. (Unlike two separate tables, the X macro version is guaranteed to be the exact correct length / values align with the corresponding string.)
That's been my primary usage of them. Another is to create a list of config file options (or command line arguments) like
which can then be used to (a) generate the fields of a config struct, define the mapping from string to field (using the stringifying macro operators), define what functions to use for parsing each field, create the help message, etc. Basically like `argparse` or `clap` but much hackier.As gross as they are, the ability to define one table of data that's used multiple ways in multiple places is handy.
I agree. Terrible hack, but if you're in terrible hack land they're quite useful. LLVM uses them extensively.
I understand the use case of this, but when I see it I always wonder if, and think I would prefer, some external code generation step instead rather than falling back on macros in the preprocessor. Like an external script or something.
Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with.
Compare with a solution that's 100% standard C++, integrates into your build with zero work, can be immediately understood by anyone reasonably skilled in the language, and puts the "generated" code right where it belongs.
CMake makes this pretty painless. My codegen targets have only two additional instructions to handle the generation itself and dependencies: add_custom_command to call the codegen exec, and then add_custom_target to wrap my outputs in a "virtual" target I can then make the rest of my program depend on, but this is just for tidying up.
And I'll dispute the fact that any complex C prepro task "can be immediately understood by anyone reasonably skilled in the language". Besides, code should ideally be understood by "anyone reasonably likely to look at this code to work in it", not "reasonably skilled".
This isn't complex. It's a bit unusual, but not hard to understand if you understand the basics of how #include and #define work.
If you're working on the sort of C++ codebase that would benefit from this sort of code generation, and you're not reasonably skilled in C++, then god help you.
Are you talking about the X macro itself, or more generally?
I may be the obtuse one here, but for a more complex example, it took me a few hours to manage to make nested loops using Boost PP (for explicit instantiations). Even so, I avoid having to write a new one that's not a quick copy-paste because it's quite different from usual C++ programming, so my painfully acquired understanding quickly evaporated... as I suspect is the case of anyone who doesn't particularly focus on the C prepro.
In the end, it's just simpler to get some Python script or C++ program to write a string and dump that to a file than to write something illegible with the C preprocessor, if doing something at all complicated (in my opinion).
I'm talking about X-macros. There's a wide range of preprocessor shenanigans, from "everybody needs to know this" to "oh my god why." Each construct needs to be evaluated on its merits. IMO X-macros are closer to the simpler side of that spectrum. Consider writing things out by hand if you just have a few, but if you have a lot of things repeating like this, they're a fine tool to use. Boost PP is a whole different level of ridiculousness and I don't see ever using that sort of thing for anything serious.
> Now you have a additional stage in your build, a bunch of new code to maintain, and either a bespoke language embedded in your standard C++ or a bunch of code emitting C++ separately from the code it logically belongs with.
The preprocessor is already a bespoke language embedded in your C++, and code written in it is generally harder to maintain than, like, Python.
The cost of doing something non-standard is real, but not infinite; at some point the benefit in code maintainability and sanity is worth it.
The preprocessor is part of C++.
I agree that you can go too far with it and it becomes better to do it a different way, but the X-macros technique is straightforward and easy to understand.
I've done this in C with the C preprocessor and Java with m4[0].
The upside of doing it natively is that it keeps the build simpler. And everybody at least knows about the existence of the C preprocessor, even if they don't know it well. And it's fairly limited, which prevents you from getting too clever.
The big downside of doing it with the C preprocessor is that the resulting code looks like vomit if it's more than a line or two because of the lack of line breaks in the generated code. Debugging it is unenjoyable. I'd recommend against doing anything super clever.
The upside of doing it out of band is that your generated source files look decent. m4 tends to introduce a little extra whitespace, but it's nothing objectionable. Plus you get more power if you really need it.
The downside is that almost nobody knows m4[1]. If you choose something else, it becomes a question of what, does anyone else know it, and is it available everywhere you need to build.
Honestly, integrating m4 into the build in ant really wasn't too bad. We were building on one OS on two different architectures. For anything truly cross-platform, you'll likely run into all the usual issues.
ETA: Getting an IDE to understand the out of band generation might be a hassle, as other folks have mentioned. I'm a vim kinda guy for most coding, and doing it either way was pretty frictionless. The generated java code was read-only and trivial, so there wasn't a lot of reason to ever look at it. By the time you get to debugging, it would entirely transparent because you're just looking at another set of java files.
[0] This was so long ago, I no longer remember why it seemed like a good idea. I think there was an interface, a trivial implementation, and some other thing? Maybe something JNI-related? At least at first, things were changing often enough that I didn't want to have to keep three things in sync by hand.
[1] Including me. I re-learn just enough to get done with the job at hand every time I need it.
This is what I do, these days. Whenever I would previously have reached for X-macros or some other macro hack, I tend to use Cog [1] now instead.
It's quite a clever design; you write Python to generate your C++ code and put it inside a comment. Then when you run the Cog tool on your source file, it writes the generated code directly into your C++ file right after your comment (and before a matching "end" comment).
This is great because you don't need Cog itself to build your project, and your IDE still understands your C++ code. I've also got used to being able to see the results of my code generation, and going back to normal macros feels a bit like fiddling around in the dark now.
[1] https://nedbatchelder.com/code/cog/
IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this. IDEs also can expand the macro invocations. So, I prefer the macros when possible :-).
> IDEs understand preprocessor macros, so IDE features (jump2def, etc) work with this.
Do they? X macros often are used with token pasting (https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html), as for example in (FTA)
Are modern IDEs/compiler toolchains smart enough to tell you that START_foo was created by an expansion of that macro?Yes. I use this with VS2019 for generating enum names, and they interact fine with auto complete and go to definition.
any non-toy IDE can do that. Most IDEs use clang directly for parsing nowadays.
The C# "source generator" approach is a good compromise; it runs within the build chain so has the ease-of-use of macros in that respect, but they don't need to be written in a weird macro language (they are C# or can call external tool) and when you debug your program, you debug through the generated source and can see it, more accessible than macros. Not sure if there is something similar in C/C++ integrated with the common toolchains.
But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.
> But when working outside C/C++ I've found myself missing the flexibility of macros more times than I can count.
Me to, and that's even in Lisp!
Preprocessor macros are hard and bugprone because they share the failings of Unix philosophy of "text as universal interface" - you're playing with unstructured (or semi-structured) pieces of text, devoid of all semantics. And this is also what makes them occasionally useful - some code transformations are much, much easier to do when you can manipulate the text form directly, ignoring syntax and grammar and everything.
Only the final value must be correct code - starting point and intermediary values can be anything, and you don't need to make sure you can get from here to there through valid data transformations. This is a really powerful capability to have.
(I also explicitly compared preprocessor macros to "plaintext everything" experience that's seen as divine wisdom, to say: y'all are slinging unstructured text down the pipes way too much, and using preprocessor way too little.)
Using the C preprocessor is standard, available, compatible and the major usage patterns are "known". For a lot of cases, they're way easier to reason about rather than learning how an external generation tool is used to generate the code. In order to understand these macros all I need is to read the source code where they're used.
Nothing C++ related in the pattern though. This C preprocessor trickery is practically so classic you couldn't necessarily even call it a "trick".
After trying to wrangle Boost PP and other advertised compile-time libraries such as Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values), I've finally converged to simply writing C++ files that write other C++ files. Could be Python, but I rather keep the build simple in my C++ project. Code generation is painless with CMake, no idea with other build configuration utilities.
CMake has a particularly irritating flaw here, though, in that it makes no distinction between host and target which cross-compiling, which makes it really difficult to do this kind of code generation when supporting this use-case (which is becoming more and more commoon).
Right, I hadn't thought of that, to be honest. If I understand correctly, you're saying the codegen targets will be compiled to the target arch, and then can't be run on the machine doing the compiling?
I think one solution might be to use target_compile_options() which lets you specify flags per target (instead of globally), assuming you're passing flags to specify the target architecture.
That only works if it's mostly the same compiler, unfortunately. They could be completely different executables, calling conventions, etc. I don't know why CMake still has such a huge hole in its feature set, but it's quite unfortunate.
> Boost Hana (which still has some runtime overhead compared to the same logic with hardcoded values)
Can you elaborate on that? What was your use-case for which this was true?
One case I benchmarked was Bernstein/Bézier and Lagrange element evaluation. This is: given a degree d triangle or tetrahedron, given some barycentric coordinates, get the physical coordinate and the Jacobian matrix of the mapping.
Degree 2, Lagrange:
- runtime: 3.6M/s - Hana: 16.2M/s - Hardcoded: 37.7M/s
Degree 3, Lagrange: 2.6M/s, 6.4M/s, 13.4M/s (same order).
"Runtime" here means everything is done using runtime loops, "Hana" using Boost Hana to make loops compile-time and use some constexpr ordering arrays, "hardcoded" is a very Fortran-looking function with all hardcoded indices and operations all unrolled.
As you see, using Boost Hana does bring about some improvement, but there is still a factor 2x between that and hardcoded. This is all compiled with Release optimization flags. Technically, the Hana implementation is doing the same operations in the same order as the hardcoded version, all indices known at compile time, which is why I say there must be some runtime overhead to using hana::while.
In the case of Bernstein elements, the best solution is to use de Casteljau's recursive algorithm using templates (10x to 15x speedup to runtime recursive depending on degree). But not everything recasts itself nicely as a recursive algorithm, or I didn't find the way for Lagrange anyways. I did enable flto as, from my understanding (looking at call stacks), hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating)
Similar results to compute Bernstein coefficients of the Jacobian matrix determinant of a Q2 tetrahedron, factor 5x from "runtime" to "hana" (only difference is for loops become hana::whiles), factor 3x from "hana" to "hardcoded" (the loops are unrolled). So a factor 15x between naive C++ and code generated files. In the case of this function in particular, we have 4 nested loops, it's branching hell where continues are hit very often.
Hhhmmm, interesting, thanks for reply!
That would be fairly interesting to look at the actual code you've used, and have a look at the codegen. By a chance, is it viable for you to open-source it? I'd guess it should bear lots of interest for Hana author/s.
What compiler/version did you use? For example, MSVC isn't (at least wasn't) good at always evaluating `constexpr` in compile-time...
> hana::while creates lambda functions, so perhaps a simple function optimization becomes a cross-unit affair if it calls hana::while. (speculating)
Hmm, I'd say it (LTO) shouldn't influence, as these lambdas are already fully visible to a compiler.
I never thought to contact them, but I might do that, thanks for the suggestion. This is something I tested almost two years ago, I have these benchmarks written down but I've since deleted the code I've used, save for the optimal implementations (though it wouldn't take too long to rewrite it).
I tested with clang on my Mac laptop and gcc on a Linux workstation. Version, not sure. If I test this again to contact the Hana people, I'll try and give all this information. I did test the constexpr ordering arrays by making sure I can pass, say, arr[0] as a template parameter. This is only possible if the value is known at compile time. Though it's also possible the compiler could be lazy in other contexts, as in not actually evaluating at compile time if it figures out the result is not necessary to be known at compile time.
Oh yeah, you're right, I was confusing translation unit and function scope.
This sounds pragmatic, but are you writing C++ executables that when run create the generated code? Are there templating libraries involved?
Yeah, it's all done automatically when you build, and dependencies are properly taken into account: if you modify one of the code generating sources, its outputs are regenerated, and everything that depends on them is correctly recompiled. This doesn't take much CMake logic at all to make work.
In my case, no, it's dumb old code writing strings and dumping that to files. You could do whatever you want in there, it's just a program that writes source files.
I do use some template metaprogramming where it's practical versus code generation, and Boost Hana provides some algorithmic facilities at compile time but those incur some runtime cost. For instance, you can write a while loop with bounds evaluated at compile time, that lets you use its index as a template parameter or evaluate constexpr functions on. But sometimes the best solution has been (for me, performance/complexity wise) to just write dumb files that hardcode things for different cases.
External codegen introduces a lot of friction in random places. Like how your editor can no longer understand the file before you start building. Or how it can go out of date with respect to the rest of your code until you build. If you can do it with a macro it tends to work better than codegen in some ways.
Spoken like a Go programmer :D
That introduces some other tool (with its own syntax), an extra build step, possible non-composability with other features.
A preprocessor (for good or bad) IS a code generation step.
I wonder if c++26 reflection can manage to replace the need for this pattern with something more robust.
Another issue is cross-compiling.
External code generation requires (cross) execution of the (cross) compiled binary program.
Or to build the generating code for the host instead. Most build systems that support cross-compilation can do this, except CMake.
"X" macros are great until you need two of them visible in the same translation unit. It is much better to pass a list macro as an argument to a uniquely named X macro and avoid the need to ever undef anything.
I don’t think I follow, do you mind giving a concrete example?
Basically, instead of faffing around with undefing values and including different files, you define your list like this:
And then you use it like so: And so Will expand to The nice thing about this approach is you can define multiple lists and macros, and make higher order macros which use them. I have a system like this which allows me to define reflective structs in C++ easily, i.e. I define a struct like: (where DECLARE_REFLECTIVE_STRUCT basically just does the same dance as above with passing different per-element structs into the list that it is passed for the struct definition, and other utility functions associated with it)which then makes a struct Foo with members bar and baz with the right types and default values, but also I can do 'foo_instance.fetch_variant("baz")' and other such operations.
The biggest pain with this approach is it's basically dealing with a bunch of multi-line macros, so it can get messy if there's a typo somewhere (and I strongly recommend an auto-formatter if you don't like having a ragged line of backslashes to the right of all the code that uses it).
_This_ is the best pattern for X macros, without any of that noise of undef'ing anything.
My approach is to wrap the list elements with two macros: an inner transformation one and an outer delimiter, like so:
Then you can compose different pieces of code for different contexts by just swapping out the delimiter. A very contrived example:Some minor tweaks to what the author shows to make it even better.
Give the macro a more descriptive name. For their example, call it GLOBAL_STRING instead of X. I think this helps make things clearer.
#undef the macro at the end of the header. That removes one line of boilerplate from every use of the header.
Use #ifndef at the top of the header and emit a nice error if the macro isn't defined. This will make it easier to understand what's wrong if you forget the #define or misspell the macro.
despite the extra boilerplate, I feel like it's still better to undef the macros in the same scope they were defined in, so that they clearly delimit the code in which the macro is active.
I think this pattern is obvious enough that it's clear what's going on, but I can see where you're coming from.
I've learned about them, staring endlessly at the luajit source code - for example (not the best example I can remember, but still) - https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_lex.h#L15
there it defines a TOKEN(_,__) macro generator the luajit token/keywords and later generates enums with them.
I've used it recently to wrap a "C" api with lots of functions, such that I can redirect calls.
I can't believe this actually has a name (is this an attempt to make fetch happen?) and it's not considered an anti-pattern. This sort of preprocessor/code mixing is impossible to debug and maintain; most senior C++ programmers have advised people to avoid doing this since the 1990's.
There is a role for the preprocessor to automate simple tables in C and such, but anything complex should be avoided like the plague if you don't like tech debt. And there's not much excuse for using it in C++ to this extent.
The vast majority of the use of X macros (and to be clear, they are not that common) I have seen in the wild have been very simple uses that are quite straightforward to debug.
Should we have a better solution in 2025? Yes. Lacking that, are X macros better than other workarounds? Also yes.
It's not particularly difficult to deal with, as far as C++ goes (I'd take it over most template shenanigans, the rules are a little simpler). It would be easier if C++ had ever tried to improve the preprocessor instead of just saying "don't use it" and trying to get templates to do a subset of the same kind of things you might want the preprocessor for.
(Also, there's a much better version of this I decribed above: mucking about with re-including the same file multiple times is unnecessary and constraints things too much. You can make a much nicer interface for users of the macro with only a little extra effort)
It is a clever trick. Very useful in C also, maybe more than in C++.
It can be overused, though.
Kind of works like Rust declarative macros (`macro_rules!`) in that it is often used to repeat and expand something common across an axis of things that vary.
It's funny that the simple name X Macros has stuck and is a de facto standard. E.g. https://en.wikipedia.org/wiki/X_macro suggests they date to the 1960s.
Famous in IBM 360 assembly language, in particular these were used for building code based on data schemas in CICS [1] applications which were remarkably similar to 2000-era cgi-bin applications in that you drew forms on 3270 terminals [2] and instead of sending a character for each keystroke like the typical minicomputer terminal, users would hit a button to submit the form, like an HTML form, and the application mostly shuttled data between forms and the database.
[1] https://en.wikipedia.org/wiki/CICS
[2] https://en.wikipedia.org/wiki/IBM_3270
That's sort of how Linux kernel trace-events are implemented [1]. E.g.
[1] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds...At first I though this was just normal macros, not X-macros. But you're right, and at first I just wasn't understanding it. In linux/include/trace/stages/stage1_struct_define.h, linux/include/trace/stages/stage2_data_offsets.h, etc., we see the different definitions of __field and __array.
Also, in linux/include/linux/genl_magic_struct.h we see multiple different definitions of __field and __array, each followed by #include GENL_MAGIC_INCLUDE_FILE, and I had no idea you could even source the name of a file to #include from a preprocessor macro! But that's a different __array with a different number of arguments.
Generally I find that most uses of x macros are obviated by an increasingly powerful constexpr.
No need for any of this if you use D!
X Macros are great, but always happy when I can replace them with something else.
One the one hand it is great and probably usable to solve actual problems.
On the other hand it seems fishy that you need all these hacks to do it. There must be a way simpler language than C++ where you could do the same easier.
This is a C pattern, that people also happen to use in C++ and Objective-C.
We called these “fruit lists”.
How is this C++? This is cpp (c preprocessor)
Lucky for us C++ is backwards compatible ;)
Too bad about C++++-= (aka Live Oak aka Java)!
https://donhopkins.medium.com/bill-joys-law-2-year-1984-mill...
>Introduction
>These are some highlights from a prescient talk by Bill Joy in February of 1991.
>“It’s vintage wnj. When assessing wnj-speak, remember Eric Schmidt’s comment that Bill is almost always qualitatively right, but the time scale is sometimes wrong.” -David Hough
>C++++-=
>“C++++-= is the new language that is a little more than C++ and a lot less.” -Bill Joy
>In this talk from 1991, Bill Joy predicts a new hypothetical language that he calls “C++++-=”, which adds some things to C++, and takes away some other things.
>Oak
>It’s no co-incidence that in 1991, James Gosling started developing a programming language called Oak, which later evolved into Java.
>“Java is C++ without the guns, knives, and clubs.” -James Gosling
>Fortunately James had the sense to name his language after the tree growing outside his office window, instead of calling it “C++++-=”. (Bill and James also have very different tastes in text editors, too!)
i hate this "trick" with a passion. i am the one who winds up having to debug this shit, and theres nothing i love doing more than trawling through hundreds of thousands of preprocessed C++ that gets defecated by `gcc -E ...`. even better when the macro expansion is 10 levels deep.
you need tables? use python (or your scripting language of choice) as a pre-compile step that generates the .cpp/.c/.h files, and then build that.
I don't quite understand this comment. Why would adding another preprocessing step using another tool make it easier? I always found it nice that you can use 'gcc -E' and inspect the output. Of yours, 10 levels of macro expansion are a problem, but simple X macros would not usually do this.