Macros

Macros, and the metaprogramming they enable, are one of the characteristic features of Lisp, and one of the main advantages Hy offers over vanilla Python. Much of the material covered in this chapter will be familiar to veterans of other Lisps, but there are also a lot of Hyly specific details.

What are macros for?

The gist of metaprogramming is that it allows you to program the programming language itself (hence the word). You can create new control structures, like do-while, or other kinds of new syntax, like a concise literal notation for your favorite data structure. You can also modify how existing syntax is understood within a region of code, as by making identifiers that start with a capital letter implicitly imported from a certain module. Finally, metaprogramming can improve performance in some cases by effectively inlining functions, or by computing something once at compile-time rather than several times at run-time. With a Lisp-like macro system, you can metaprogram in a slicker and less error-prone way than generating code as text with conventional string formatting, or with lexer-level macros like those provided by the C preprocessor.

Types of macros

Hy offers two types of macros: regular macros and reader macros.

Regular macros, typically defined with defmacro, are the kind Lispers usually mean when they talk about "macros". Regular macros are called like a function, with an expression whose head is the macro name: for example, (foo a b) could call a macro named foo. A regular macro is called at compile-time, after the entire top-level form in which it appears is parsed, and receives parsed models as arguments. Regular macros come in three varieties, which vary in scope.

Reader macros, typically defined with defreader, are lower-level than regular macros. They're called with the hash sign #; for example, #foo calls a reader macro named foo. A reader macro is called at parse-time. It doesn't receive conventional arguments. Instead, it uses an implicitly available parser object to parse the subsequent source text. When it returns, the standard Hy parser picks up where it left off.

Related constructs

There are three other constructs that perform compile-time processing much like macros, and hence are worth mentioning here.

• do-mac is essentially shorthand for defining and then immediately calling a regular macro with no arguments.

• eval-when-compile evaluates some code at compile-time, but contributes no code to the final program, like a macro that returns None in a context where the None doesn't do anything.

• eval-and-compile evaluates some code at compile-time, like eval-when-compile, but also leaves the same code to be re-evaluated at run-time.

When to use what

The variety of options can be intimidating. In addition to all of Hy's features listed above, Python is a dynamic programming language that allows you to do a lot of things at run-time that other languages would blanch at. For example, you can dynamically define a new class by calling type. So, watch out for cases where your first thought is to use a macro, but you don't actually need one.

When deciding what to use, a good rule of thumb is to use the least powerful option that suffices for the syntax, semantics, and performance that you want. So first, see if Python's dynamic features are enough. If they aren't, try a macro-like construct or a regular macro. If even those aren't enough, try a reader macro. Using the least powerful applicable option will help you avoid the macro pitfalls described below, as well as other headaches such as wanting to use a macro where a Python API needs a function. But for the sake of providing simpler examples, much of the below discussion will ignore this advice and consider example macros that could easily be written as functions.

The basics

A regular macro can be defined with defmacro using a syntax similar to that of defn. Here's how you could define and call a trivial macro that takes no arguments and returns a constant:

(defmacro seventeen []
  17)

(print (seventeen))

To see that seventeen is expanded at compile-time, run hy2py on this script and notice that it ends with print(17) rather than print(seventeen()). If you insert a print call inside the macro definition, you'll also see that the print happens when the file is compiled, but not when it's rerun (so long as an up-to-date bytecode file exists).

A more useful macro returns code. You can construct a model the long way, like this:

(defmacro addition []
  (hy.models.Expression [
    (hy.models.Symbol "+")
    (hy.models.Integer 1)
    (hy.models.Integer 1)]))

or more concisely with quote, like this:

(defmacro addition []
  '(+ 1 1))

You don't need to always return a model because the compiler calls hy.as-model on everything before trying to compile it. Thus, the 17 above works fine in place of (hy.models.Integer 17). But trying to compile something that hy.as-model chokes on, like a function object, is an error.

Arguments are always passed in as models. You can use quasiquotation (see quasiquote) to concisely define a model with partly literal and partly evaluated components:

(defmacro set-to-2 [variable]
 `(setv ~variable 2))
(set-to-2 foobar)
(print foobar)

Macros don't understand keyword arguments like functions do. Rather, the keyword objects themselves are passed in literally. This gives you flexibility in how to handle them. Thus, #** kwargs and * aren't allowed in the parameter list of a macro, although #* args and / are. See hyrule.defmacro-kwargs if you want to handle keyword arguments like a function.

On the inside, macros are functions, and obey the usual Python semantics for functions. For example, setv inside a macro will define or modify a variable local to the current macro call, and return ends macro execution and uses its argument as the expansion.

Macros from other modules can be brought into the current scope with require.

Pitfalls

Macros are powerful, but with great power comes great potential for anguish. There are a few characteristic issues you need to guard against to write macros well, and, to a lesser extent, even to use macros well.

Name games

A lot of these issues are variations on the theme of names not referring to what you intend them to, or in other words, surprise shadowing. For example, the macro below was intended to define a new variable named x, but it ends up modifying a preexisting variable.

(defmacro upper-twice [arg]
  `(do
     (setv x (.upper ~arg))
     (+ x x)))
(setv x "Okay guys, ")
(setv salutation (upper-twice "bye"))
(print (+ x salutation))
  ; Intended result: "Okay guys, BYEBYE"
  ; Actual result: "BYEBYEBYE"

If you avoid the assignment entirely, by using an argument more than once, you can cause a different problem: surprise multiple evaluation.

(defmacro upper-twice [arg]
  `(+ (.upper ~arg) (.upper ~arg)))
(setv items ["a" "b" "c"])
(print (upper-twice (.pop items)))
  ; Intended result: "CC"
  ; Actual result: "CB"

A better approach is to use hy.gensym to choose your variable name:

(defmacro upper-twice [arg]
  (setv g (hy.gensym))
  `(do
     (setv ~g (.upper ~arg))
     (+ ~g ~g)))

Hyrule provides some macros that make using gensyms more convenient, like defmacro! and with-gensyms.

On the other hand, you can write a macro that advertises a specific name (or set of names) as part of its interface. For example, Hyrule's anaphoric macro ap-if assigns the result of a test form to it, and allows the caller to include forms that refer to it:

(import os)
(ap-if (.get os.environ "PYTHONPATH")
   (print "Your PYTHONPATH is" it))

Macro subroutines

A case where you could want something to be in the scope of a macro's expansion, and then it turns out not to be, is when you want to call a function or another macro in the expansion:

(defmacro hypotenuse [a b]
  (import math)
  `(math.sqrt (+ (** ~a 2) (** ~b 2))))
(print (hypotenuse 3 4))
  ; NameError: name 'math' is not defined

The form (import math) here appears in the wrong context, in the macro call itself rather than the expansion. You could use import or require to bind the module name or one of its members to a gensym, but an often more convenient option is to use the one-shot import syntax hy.I or the one-shot require syntax hy.R:

(defmacro hypotenuse [a b]
  `(hy.I.math.sqrt (+ (** ~a 2) (** ~b 2))))
(print (hypotenuse 3 4))

A related but distinct issue is when you want to use a function (or other ordinary Python object) in a macro's code, but it isn't available soon enough:

(defn subroutine [x]
  (hy.models.Symbol (.upper x)))
(defmacro uppercase-symbol [x]
  (subroutine x))
(setv (uppercase-symbol foo) 1)
  ; NameError: name 'subroutine' is not defined

Here, subroutine is only defined at run-time, so uppercase-symbol can't see it when it's expanding (unless you happen to be calling uppercase-symbol from a different module). This is easily worked around by wrapping (defn subroutine …) in eval-and-compile (or eval-when-compile if you want subroutine to be invisible at run-time).

By the way, despite the need for eval-and-compile, extracting a lot of complex logic out of a macro into a function is often a good idea. Functions are typically easier to debug and to make use of in other macros.

The important take-home big fat WARNING

A typical macro should use only four kinds of names in its expansion: gensyms, core macros, objects that Python puts in scope by default (like its built-in functions), and hy and its attributes. It's possible to rebind nearly all these names, so surprise shadowing is still theoretically possible. Unfortunately, the only way to prevent these pathological rebindings from coming about is… don't do that. Don't make a new macro named setv or name a function argument type unless you're ready for every macro you call to break, the same way you wouldn't monkey-patch a built-in Python module without thinking carefully. This kind of thing is the responsibility of the macro caller; the macro writer can't do much to defend against it. There is at least a pragma warn-on-core-shadow, enabled by default, that causes defmacro and require to warn you if you give your new macro the same name as a core macro.

Reader macros

Reader macros allow you to hook directly into Hy's parser to customize how text is parsed into models. They're defined with defreader, or, like regular macros, brought in from other modules with require. Rather than receiving function arguments, a reader macro has access to a hy.HyReader object named &reader, which provides all the text-parsing logic that Hy uses to parse itself (see hy.HyReader and its base class hy.Reader for the available methods). A reader macro is called with the hash sign #, and like a regular macro, it should return a model or something convertible to a model.

The simplest kind of reader macro doesn't read anything:

(defreader hi
  `(print "Hello."))

#hi #hi #hi

A less trivial, and more common, usage of reader macros is to call hy.HyReader.parse_one_form() to get a single form from the following source text. Such a reader macro is like a unary regular macro that's called with # instead of parentheses.

(defreader do-twice
  (setv x (.parse-one-form &reader))
  `(do ~x ~x))

#do-twice (print "This line prints twice.")

Here's a moderately complex example of a reader macro that couldn't be implemented as a regular macro. It reads in a list of lists in which the inner lists are newline-separated, but newlines are allowed inside elements.

(defreader matrix
  (.slurp-space &reader)
  (setv start (.getc &reader))
  (assert (= start "["))
  (.slurp-space &reader)
  (setv out [[]])
  (while (not (.peek-and-getc &reader "]"))
    (cond
      (any (gfor  c " \t"  (.peek-and-getc &reader c)))
        None
      (.peek-and-getc &reader "\n")
        (.append out [])
      True
        (.append (get out -1) (.parse-one-form &reader))))
  (lfor  line out  :if line  line))

(print (hy.repr #matrix [
    1 (+ 1 1) 3
    4 ["element" "containing"
          "a" "newline"]        6
    7 8 9]))
  ; => [[1 2 3] [4 ["element" "containing" "a" "newline"] 6] [7 8 9]]

Note that because reader macros are evaluated at parse-time, and top-level forms are completely parsed before any further compile-time execution occurs, you can't use a reader macro in the same top-level form that defines it:

(do
  (defreader up
    (.slurp-space &reader)
    (.upper (.read-one-form &reader)))
  (print #up "hello?"))
    ; LexException: reader macro '#up' is not defined

Of the potential problems discussed above that apply to regular macros, such as surprise shadowing, most also apply to reader macros.

Macro namespaces and operations on macros

Macros don't share namespaces with ordinary Python objects. That's why something like (defmacro m []) (print m) fails with a NameError, and how hy.pyops can provide a function named + without hiding the core macro +.

There are three scoped varieties of regular macro. First are core macros, which are built into Hy; the set of core macros is fixed. They're available by default. You can inspect them in the dictionary bulitins._hy_macros, which is attached to Python's usual builtins module. The keys are strings giving mangled names and the values are the function objects implementing the macros.

Global macros are associated with modules, like Python global variables. They're defined when you call defmacro or require in a global scope. You can see them in the global variable _hy_macros associated with the same module. You can manipulate _hy_macros to list, add, delete, or get help on macros, but be sure to use eval-and-compile or eval-when-compile when you need the effect to happen at compile-time, which is often. (Modifying bulitins._hy_macros is of course a risky proposition.) Here's an example, which also demonstrates the core macro get-macro. get-macro provides syntactic sugar for getting all sorts of macros as objects.

(defmacro m []
  "This is a docstring."
  `(print "Hello, world."))
(print (in "m" _hy_macros))   ; => True
(help (get-macro m))
(m)                           ; => "Hello, world."
(eval-and-compile
  (del (get _hy_macros "m")))
(m)                           ; => NameError
(eval-and-compile
  (setv (get _hy_macros (hy.mangle "new-mac")) (fn []
    '(print "Goodbye, world."))))
(new-mac)                     ; => "Goodbye, world."

Local macros are associated with function, class, or comprehension scopes, like Python local variables. They come about when you call defmacro or require in an appropriate scope. You can call local-macros to view local macros, but adding or deleting elements is ineffective. Beware that local macro definitions apply to the results of expanding other macros in the given context, and hence may not be as local as you expect:

(defmacro number []
  1)

(defmacro uses-number []
  '(number))

(defn f []
  (defmacro number []
    2)
  (uses-number))

(print (uses-number))  ; => 1
(print (f))            ; => 2
(print (uses-number))  ; => 1

For this reason, shadowing a core macro is risky even with a local macro.

Finally, _hy_reader_macros is a per-module dictionary like _hy_macros for reader macros, but here, the keys aren't mangled. There are no local reader macros, and there's no official way to introspect on Hy's handful of core reader macros. So, of the three scoped varieties of regular macro, reader macros most resemble global macros.