syntax-case takes a syntax object as an argument and destructures it via pattern matching into pattern variables.

What are pattern variables composed of?

syntax takes a template, and the pattern variables in it, to produce a syntax object that has the same lexical scoping as that of the pattern variable, so where does that lexical scope get stored?

and then Ryan explained:

The short answer is that syntax objects store scoping information and
that pattern variables have nothing to do with it.

Here’s a longer answer:

A pattern variable is essentially bound to an ellipsis-depth n and syntax object(s). If the pattern variable doesn’t have ellipses after it, then n is 0 and the pattern variable refers to a single syntax object. If it has one ellipsis after it, then n is 1 and there is a list of syntax objects. And so forth.

The ‘syntax’ form is like ‘quasiquote’, except that there is no explicit ‘unquote’; rather, pattern variables are automatically substituted, and the rest of the template becomes syntax object literals (via ‘quote-syntax’) that remember the bindings in scope where they occurred.

So pattern variables, while convenient, are not necessary for writing macros. You could do just the same with ‘quote-syntax’, ‘syntax-e’, ‘datum->syntax’, ‘car’, etc.

Lexical scoping is managed by the syntax objects. The macro arguments already have their lexical contexts embedded in them, and that is preserved as the macro takes them apart and puts them together with new syntax. This new syntax mostly comes from syntax literals.

The essence of the algorithm is explained in “Syntactic Abstraction in Scheme” by Dybvig et al.

;; the working if-it & when-it
 
(define-syntax (if-it stx)
  (syntax-case stx ()
    ((if-it test? then else)
     (with-syntax ((it (datum->syntax #'if-it 'it)))
       #'(let ((it test?))
           (if it then else)))))) 
 
(define-syntax (when-it stx)
  (syntax-case stx ()
    ((~ test? exp exp2 ...)
     (with-syntax ((it (datum->syntax #'~ 'it)))
       #'(let ((it test?)) (when it exp exp2 ...))))))
 
;; the non-working cond-it 
(define-syntax (cond-it stx)
  (syntax-case stx (else)
    ((cond-it (else exp exp2 ...))
     #'(begin exp exp2 ...))
    ((cond-it (test? exp exp2 ...))
     #'(when-it test? exp exp2 ...))
    ((cond-it (test? exp exp2 ...) cond1 cond2 ...)
     #'(if-it test? (begin exp exp2 ...)
              (cond-it cond1 cond2 ...)))))

When ‘cond-it’ expands and produces an ‘if-it’ expression, the ‘if-it’ is marked by the macro expander as coming from a macro. That means its lexical context is different from the ‘it’ variables in the branches. That means that the ‘it’ variable binding produced by ‘if-it’ does not capture the ‘it’ references in the branches.

— Ryan

Only superstars on their highest alert level [while writing macros] get things right all the time.

–Matthias

That said, hygiene alone isn’t a cure-all for writing predictable macros. If you don’t understand what you are doing, thoroughly; you will screw up eventually. “Thar be dragons here” as someone once warned on the PLT list.

(via PLT)

Over my holiday I revisited the paper and found, for lack of a better way to say it, that it “made sense”. In the past year I’ve tried to: read a lot, write a lot of code, read other people’s code, and most importantly do so with some discipline and attention to detail that I had not applied before. That seems to have paid off; as it felt like I was reading the paper with brand new eyes.

This time around, I also listened to a recording of the presentation itself and watched the PowerPoint that went along with it. This was especially valuable because it drew attention to various tidbits that appeared in the paper but were much easier to delineate by virtue of Shriram pointing them out in context of the presentation. It was those comments, along with what I had learned in the past year that really got me thinking about things again.

In a conversation with Aaron Hsu, he once said something to the effect that “Scheme is hard because it is subtle.”, and that sort of rang true with my intuition, but I had a difficult time verbalizing exactly why I thought that to be true. Outwardly Scheme appears to be “simple”, but the more you learn about it and the deeper you dig, you start to see that it is Deceptively Simple.

Scheme seems to take only a few days to learn, but a few years to master; but why? I don’t know yet, but what I do know is that some of the ideas and topics presented in this paper and presentation are surely some of the aforementioned “subtleties” that serve as landmarks on this multi-year path towards understanding Scheme. For my own reference, and for others who may be studying Scheme and wondering exactly why “Subtlety is hard”, perhaps this paper will serve to shed some light on things.

My advice: read LAMBDA: The Ultimate Imperative and LAMBDA: The Ultimate GOTO first.

Abstract

Lisp programmers have long used macros to extend their language. Indeed, their success has inspired macro notations for a variety of other languages, such as C and Java. There is, however, a paucity of effective pedagogic examples of macro use. This paper presents a short, non-trivial example that implements a construct not already found in mainstream languages. Furthermore, it motivates the need for tail-calls, as opposed to mere tail-recursion, and illustrates how support for tail-call optimization is crucial to support a natural style of macro-based language extension.

Key notes from the audio

These are mostly my notes and summaries for some key points that rang true with me. The ideas that they represent may or may not be contained within my summary as it would be too far outside the scope of this blog post to address them. Such work is better done by other papers and presentations.

• 09:00: Without the hidden gems, syntax, macros, and tail calls, it all falls apart.
• 11:16: What you generate must fit well into the language.
• 18:00: It is really tail-transfer (tail-calls) that matter, not tail-recursion.
• 21:00: Scheme is really an indentation based language.
• 23:00: Macro languages belong in modules; you can use your data (DSL) where you wish.
• 25:30: If you claim to be smart, leverage the work of others.
• 26:10: Scheme is subtle. Takes years to understand what is beautiful. You can look at a book but you won’t notice. The language hides it jewels.
• 26:45 If you don’t get tail calls, you don’t get it.

Key notes from the paper

• P2P3: “A macro gives the programmer a consistent, representation-independent means of describing the datum
  while still resolving the representation before compilation.”
• P7P4: Macros are likely to be used to optimize the abstraction that the data represents.
• P8P3: Scheme’s macro system could be called “a lightweight compiler API”.
• P9P2: Tail-calls ought to be called tail-transfer per Steele. I wonder why they didn’t push that definition more.
• P9P3: Here is a good explanation of tail calls.
• P9P5: “languages represent the ultimate form of reuse, because we get to reuse everything from the mathematical (semantics) to the practical (libraries), as well as decades of research and toil in compiler construction (Hudak, 1998).”
• P13P2: “it shows how features that would otherwise seem orthogonal, such as macros and tail-calls, are in fact intimately wedded together; in particular, the absence of the latter would greatly complicate use of the former.”

The automaton language

; An automaton that recognizes the language c(ad)*r
(define m 
  (automaton init
             (init : 
                   (c → more))
             (more : 
                   (a → more)
                   (d → more)
                   (r → end))
             (end : accept)))

While the post itself is a troll, the author:

1. Uses a very accessible analogy.
2. Quite clearly communicates a number of differences between Scheme and Common Lisp when it comes not only to hygiene but also language design.

The interesting thing about this post is that the only difference between this post being a troll and an educational article is how the content was delivered. If you remove the cruddy name-calling, whining, and complaining; it would have been a really good post on the difference between the languages. That is wild!

(reference mysophobic)

This chapter will show that variable capture can also be used constructively. There are some useful macros which couldn’t be written without it.

My evaluation of that claim is that while the former is true, anaphoric macros are not evidence of such a case as they only save you a variable binding. The latter claim is interesting because it begs the question of whether or not they should be written as macros at all. It made me wonder how anaphoric macros might look in Scheme, how they might look as functions, and whether one is clearly superior to the other.

I wrote two versions of aif, the first is a macro (m-aif) and the second is a function (f-aif). Both provide the same functionality albeit with a different mechanism for implementation and syntax for application:

m-aif

(define-syntax m-aif
  (λ (stx)
    (syntax-case stx ()
      ((_ test-form then-form)
       (syntax (_ test-form then-form void)))
      ((_ test-form then-form else-form)
       (with-syntax ((it (syntax-local-introduce (syntax it))))
         (syntax
          (let ((it test-form))
            (if it
                then-form
                else-form))))))))

f-aif

(define f-aif
  (λ (test-form then-form . else-form)
    (let ((it (test-form)))
      (if it
          (then-form it)
          (unless (null? else-form)
            ((car else-form)))))))

The difference between utilizing the two versions becomes a matter of syntactic preference:

m-aif

(m-aif (identity 10)
       (display (* it 23)))

f-aif

(f-aif (λ () (identity 10))
       (λ (it) (display (* it 23))))

Which do you like better?

It should be an easy decision; with f-aif you get the desired functionality without breaking hygiene, albeit at the “cost” forcing the user to utilize lambda expressions.

In practice, requiring the use of lambda expressions is not an egregious demand of the user; instead, it is a pleasure. lambda is a special thing. The functionality that it provides is often more than enough to implement functionality that may initially appear to require macros (hygienic or not).

Macros are often times clearly the only, and the best, solution. This time isn’t one of them.

ADDENDUM: 03/04/11:

Jim asked:

If you are concerned about variable capture then why don’t you write an anaphoric if macro that takes the variable name to introduce as an argument?

Answer: When I wrote this I still didn’t grok hygiene.

Here is what I mean (and think Jim means):

#!r6rs
 
(import (rnrs))
 
(define-syntax aif*
  (syntax-rules ()
    ((_ it test con . alt)
     (let ((it test))
       (if it con . alt)))))
 
(define-syntax stomp?
  (syntax-rules ()
    ((_ body)
     (let ((foo 0))
       (begin
         body
         (display "Stomp's foo: ")
         (display foo)(newline))))))
 
(aif* foo (+ 10) 
      (stomp? 
       (begin (display "My foo: ")
              (display foo)(newline))))

What do you think would get printed out here? At first glance you might think that stomp? will clobber foo and the user will not get the value they had expected, something like this:

My foo: 0
Stomp's foo: 0

Here is the output though:

My foo: 10
Stomp's foo: 0

Surprised? syntax-rules retains the correct lexical binding so the user’s code and the macro’s code both referred to the correct binding of foo (foo_user==10 vs. foo_{stomp macro}==0). This is the benefit of “hygeine”. Looking back, I’m not sure why hygiene didn’t jump out as an extremely simple concept: macro bodies retain their lexical bindings.

Great question. Thanks for posting even after 3 years!

Perhaps a better way is to answer that question is to look at how companies are actually using them today, rather then to simply guess!

Have a look at the “case studies” section in this presentation on ContractML to see how companies are using DSLs today.

(via cufp)

Pass a message to the integer start index which includes both the inclusive stop index and a single argument function that is to be called with the current index in the iteration. The implementation of this feature, which can be found here, is written nearly as you would imagine, albeit in C. Here is how the code looks when you call this function:

1.upto(3) {|x| printf "#{x}\n"}
1
2
3

With my programmer’s glasses on this, I would say that this syntax feels nice. In fact, it feels nice enough to think about adding a feature like this to Scheme. Here is what I came up with:

(define-syntax from
  (syntax-rules (upto)
    ((from x upto y fn)
     (let ([finish (add1 y)])
       (let loop ([cur x])
         (if (not (= cur finish))
             (begin
               (fn cur)
               (loop (add1 cur)))))))))

Here is how the Ruby sample from above looks in Scheme.

(from 1 upto 3 (λ (x) (printf "~a~n" x)))
1
2
3

What do you think? The two samples really don’t look all that different.

Grow-able languages are truly exciting. If you want to add a feature to the language, you add it.

Programming with grow-able languages takes you from “power-user” to empowered-user!

Which would you rather be?

When it comes to learning about new ideas, there is simply nothing more helpful than both great books and talking to folks knowledgeable on the topic. So, TSPL, SICP, TLS, along with the wonderful folks in both comp.lang.scheme and the PLT discussion list, have really made this effort a lot of fun.

To capture the fruits of my labor, both for myself and for any folks interested in the same topic, I prepared a presentation on code generation and DSLs in Scheme. The presentation itself is driven by a popular Martin Fowler article on the topic of programming language tools entitled “Language Workbenches: The Killer-App for Domain Specific Languages?”. The presentation is written for folks familiar with object oriented programming, my hope being that familiar territory would ease the introduction of the topic. While *numerous* code samples in Scheme are provided both in the presentation itself and in the supplementary archive, you should feel free to skip them; they are included only for folks who wish to delve further into learning Scheme. If you would like to run them, please use DrScheme v371 and set the language level to “Pretty Big”.

Code Generation and DSLs in Scheme: The presentation

Code Generation and DSLs in Scheme: The code

Enjoy!

Addendum: 8/2/8

When I wrote this presentation, I was just beginning to scratch the surface of macros in Scheme. There is way, way too much code in it. Reflecting back on this, I see that teaching is a lot different than shoving a lot of information down someone’s throat. I will keep this here, though, for historical reference, and for anyone else who may benefit from this content.