Code as data in LISP and Curse of LISP

I heard about this "code as data" in LISP is so cool recently, and I was wondering how come I never realize it when I learned it, so I want back and took the Programming Languages B course again. It is taught in Racket (a dialect of LISP). And Dan only briefly mentioned this "code as data" idea in an optional lecture.

The idea is that LISP can take a program at runtime, modify it, (and then evaluate/run it). Actually, this is not something unique to LISP. A lot of "LISP inspired languages", such as Python, can take a string at runtime, modify the string, and then evaluate it. (Python also have ast.parse to parse the string into AST to make it easier to modify.) The reason it is especially cool in LISP is that: Data is represented as a list, and a program is also represented as a list (you need a quote if you really want to interpret it as data before evaluation).

Another advantage of this same representation of data and code in LISP is that: when you write macro's, what you write is what will be generated. Unlike in Julia, for example, you'd better check the generated code is what you want.

I found this video showcasing this feature in Python, Scheme, and Julia.

I feel this idea sounds very attractive to maybe people how study mathematics or other hard core science, because it has some physiological beauty. To me, maybe some other software engineers as well, it is only something sounds beautiful, but not really usefuly in most cases. As Dan mention, often times when people think they need it, there is actually better way to do it instead of doing eval.

Different people may have different understandings of this. My understanding is more like a "Curse of Macro", so it applies to other languages that has similar macro as LISP, such as Julia. When you compose functions, all you need to care about is the input and output of the function. Inside each function, it can go as complicated as it wants, while from the outside, you, as a composer, don't need to peak into the mess inside the function. If you work with a strongly typed language, the language even gives you the input and output type of a function. It is like a quick start manual, which gives you a jump start at composing.

On the other hand, if you want to compose macros in a fancier way (chain a few macros.), you mostly want to modify the code inside. Otherwise, why not just do function composition. In this case, since it is a direct peak into the mess, and there is no quick start manual for you, so you need to do a lot of the check yourself, and may face some surprises caused by previous macros. In the end, each package/module just builds their only macros. If not designed carefully, they may not work well with others' macros. The difficulty of composition is, IMO, the curse of LISP.

In other words, a function is a good way to abstract (with the help of input and output type), while a macro is not good at abstract. If components are not abstracted well, they will become hard to compose eventually.