Ruby Hacking Guide

Chapter 17: Dynamic evaluation

Overview

I have already finished to describe about the mechanism of the evaluator by the previous chapter. In this chapter, by including the parser in addition to it, let’s examine the big picture as “the evaluator in a broad sense”. There are three targets: eval, Module#module_eval and Object#instance_eval.

`eval`

I’ve already described about eval, but I’ll introduce more tiny things about it here.

By using eval, you can compile and evaluate a string at runtime in the place. Its return value is the value of the last expression of the program.

p eval("1 + 1")   # 2

You can also refer to a variable in its scope from inside of a string to eval.

lvar = 5
@ivar = 6
p eval("lvar + @ivar")   # 11

Readers who have been reading until here cannot simply read and pass over the word “its scope”. For instance, you are curious about how is its “scope” of constants, aren’t you? I am. To put the bottom line first, basically you can think it directly inherits the environment of outside of eval.

And you can also define methods and define classes.

def a
  eval('class C;  def test() puts("ok") end   end')
end

a()          # define class C and C#test
C.new.test   # shows ok

Moreover, as mentioned a little in the previous chapter, when you pass a Proc as the second argument, the string can be evaluated in its environment.

def new_env
  n = 5
  Proc.new { nil }   # turn the environment of this method into an object and return it
end

p eval('n * 3', new_env())   # 15

`module_eval` and `instance_eval`

When a Proc is passed as the second argument of eval, the evaluations can be done in its environment. module_eval and instance_eval is its limited (or shortcut) version. With module_eval, you can evaluate in an environment that is as if in a module statement or a class statement.

lvar = "toplevel lvar"   # a local variable to confirm this scope

module M
end
M.module_eval(<<'EOS')   # a suitable situation to use here-document
    p lvar   # referable
    p self   # shows M
    def ok   # define M#ok
      puts 'ok'
    end
EOS

With instance_eval, you can evaluate in an environment whose self of the singleton class statement is the object.

lvar = "toplevel lvar"   # a local variable to confirm this scope

obj = Object.new
obj.instance_eval(<<'EOS')
    p lvar   # referable
    p self   # shows #<Object:0x40274f5c>
    def ok   # define obj.ok
      puts 'ok'
    end
EOS

Additionally, these module_eval and instance_eval can also be used as iterators, a block is evaluated in each environment in that case. For instance,

obj = Object.new
p obj                 # #<Object:0x40274fac>
obj.instance_eval {
    p self            # #<Object:0x40274fac>
}

Like this.

However, between the case when using a string and the case when using a block, the behavior around local variables is different each other. For example, when creating a block in the a method then doing instance_eval it in the b method, the block would refer to the local variables of a. When creating a string in the a method then doing instance_eval it in the b method, from inside of the string, it would refer to the local variables of b. The scope of local variables is decided “at compile time”, the consequence differs because a string is compiled every time but a block is compiled when loading files.

`eval`

`eval()`

The eval of Ruby branches many times based on the presence and absence of the parameters. Let’s assume the form of call is limited to the below:

eval(prog_string, some_block)

Then, since this makes the actual interface function rb_f_eval() almost meaningless, we’ll start with the function eval() which is one step lower. The function prototype of eval() is:

static VALUE
eval(VALUE self, VALUE src, VALUE scope, char *file, int line);

scope is the Proc of the second parameter. file and line is the file name and line number of where a string to eval is supposed to be located. Then, let’s see the content:

▼ `eval()` (simplified)

static VALUE
eval(self, src, scope, file, line)
    VALUE self, src, scope;
    char *file;
    int line;
{
    struct BLOCK *data = NULL;
    volatile VALUE result = Qnil;
    struct SCOPE * volatile old_scope;
    struct BLOCK * volatile old_block;
    struct RVarmap * volatile old_dyna_vars;
    VALUE volatile old_cref;
    int volatile old_vmode;
    volatile VALUE old_wrapper;
    struct FRAME frame;
    NODE *nodesave = ruby_current_node;
    volatile int iter = ruby_frame->iter;
    int state;
5002
    if (!NIL_P(scope)) {  /* always true now */
        Data_Get_Struct(scope, struct BLOCK, data);
        /* push BLOCK from data */
        frame = data->frame;
        frame.tmp = ruby_frame; /* to prevent from GC */
        ruby_frame = &(frame);
        old_scope = ruby_scope;
        ruby_scope = data->scope;
        old_block = ruby_block;
        ruby_block = data->prev;
        old_dyna_vars = ruby_dyna_vars;
        ruby_dyna_vars = data->dyna_vars;
        old_vmode = scope_vmode;
        scope_vmode = data->vmode;
        old_cref = (VALUE)ruby_cref;
        ruby_cref = (NODE*)ruby_frame->cbase;
        old_wrapper = ruby_wrapper;
        ruby_wrapper = data->wrapper;
        self = data->self;
        ruby_frame->iter = data->iter;
    }
    PUSH_CLASS();
    ruby_class = ruby_cbase;  /* == ruby_frame->cbase */
5047
    ruby_in_eval++;
    if (TYPE(ruby_class) == T_ICLASS) {
        ruby_class = RBASIC(ruby_class)->klass;
    }
    PUSH_TAG(PROT_NONE);
    if ((state = EXEC_TAG()) == 0) {
        NODE *node;
5055
        result = ruby_errinfo;
        ruby_errinfo = Qnil;
        node = compile(src, file, line);
        if (ruby_nerrs > 0) {
            compile_error(0);
        }
        if (!NIL_P(result)) ruby_errinfo = result;
        result = eval_node(self, node);
    }
    POP_TAG();
    POP_CLASS();
    ruby_in_eval--;
    if (!NIL_P(scope)) {  /* always true now */
        int dont_recycle = ruby_scope->flags & SCOPE_DONT_RECYCLE;
5070
        ruby_wrapper = old_wrapper;
        ruby_cref  = (NODE*)old_cref;
        ruby_frame = frame.tmp;
        ruby_scope = old_scope;
        ruby_block = old_block;
        ruby_dyna_vars = old_dyna_vars;
        data->vmode = scope_vmode; /* save the modification of the visibility scope */
        scope_vmode = old_vmode;
        if (dont_recycle) {
                  /* ……copy SCOPE BLOCK VARS…… */
        }
    }
    if (state) {
        if (state == TAG_RAISE) {
                  /* ……prepare an exception object…… */
            rb_exc_raise(ruby_errinfo);
        }
        JUMP_TAG(state);
    }
5125
    return result;
}

(eval.c)

If this function is shown without any preamble, you probably feel “oww!”. But we’ve defeated many functions of eval.c until here, so this is not enough to be an enemy of us. This function is just continuously saving/restoring the stacks. The points we need to care about are only the below three:

unusually FRAME is also replaced (not copied and pushed)
ruby_cref is substituted (?) by ruby_frame->cbase
only scope_vmode is not simply restored but influences data.

And the main parts are the compile() and eval_node() located around the middle. Though it’s possible that eval_node() has already been forgotten, it is the function to start the evaluation of the parameter node. It was also used in ruby_run().

Here is compile().

▼ `compile()`

static NODE*
compile(src, file, line)
    VALUE src;
    char *file;
    int line;
{
    NODE *node;
4975
    ruby_nerrs = 0;
    Check_Type(src, T_STRING);
    node = rb_compile_string(file, src, line);
4979
    if (ruby_nerrs == 0) return node;
    return 0;
}

(eval.c)

ruby_nerrs is the variable incremented in yyerror(). In other words, if this variable is non-zero, it indicates more than one parse error happened. And, rb_compile_string() was already discussed in Part 2. It was a function to compile a Ruby string into a syntax tree.

One thing becomes a problem here is local variable. As we’ve seen in Chapter 12: Syntax tree construction, local variables are managed by using lvtbl. However, since a SCOPE (and possibly also VARS) already exists, we need to parse in the way of writing over and adding to it. This is in fact the heart of eval(), and is the worst difficult part. Let’s go back to parse.y again and complete this investigation.

`top_local`

I’ve mentioned that the functions named local_push() local_pop() are used when pushing struct local_vars, which is the management table of local variables, but actually there’s one more pair of functions to push the management table. It is the pair of top_local_init() and top_local_setup(). They are called in this sort of way.

▼ How `top_local_init()` is called

program :   { top_local_init(); }
          compstmt
            { top_local_setup(); }

Of course, in actuality various other things are also done, but all of them are cut here because it’s not important. And this is the content of it:

▼ `top_local_init()`

static void
top_local_init()
{
    local_push(1);
    lvtbl->cnt = ruby_scope->local_tbl?ruby_scope->local_tbl[0]:0;
    if (lvtbl->cnt > 0) {
        lvtbl->tbl = ALLOC_N(ID, lvtbl->cnt+3);
        MEMCPY(lvtbl->tbl, ruby_scope->local_tbl, ID, lvtbl->cnt+1);
    }
    else {
        lvtbl->tbl = 0;
    }
    if (ruby_dyna_vars)
        lvtbl->dlev = 1;
    else
        lvtbl->dlev = 0;
}

(parse.y)

This means that local_tbl is copied from ruby_scope to lvtbl. As for block local variables, since it’s better to see them all at once later, we’ll focus on ordinary local variables for the time being. Next, here is top_local_setup().

▼ `top_local_setup()`

static void
top_local_setup()
{
    int len = lvtbl->cnt;  /* the number of local variables after parsing */
    int i;                 /* the number of local varaibles before parsing */
5296
    if (len > 0) {
        i = ruby_scope->local_tbl ? ruby_scope->local_tbl[0] : 0;
5299
        if (i < len) {
            if (i == 0 || (ruby_scope->flags & SCOPE_MALLOC) == 0) {
                VALUE *vars = ALLOC_N(VALUE, len+1);
                if (ruby_scope->local_vars) {
                    *vars++ = ruby_scope->local_vars[-1];
                    MEMCPY(vars, ruby_scope->local_vars, VALUE, i);
                    rb_mem_clear(vars+i, len-i);
                }
                else {
                    *vars++ = 0;
                    rb_mem_clear(vars, len);
                }
                ruby_scope->local_vars = vars;
                ruby_scope->flags |= SCOPE_MALLOC;
            }
            else {
                VALUE *vars = ruby_scope->local_vars-1;
                REALLOC_N(vars, VALUE, len+1);
                ruby_scope->local_vars = vars+1;
                rb_mem_clear(ruby_scope->local_vars+i, len-i);
            }
            if (ruby_scope->local_tbl &&
                      ruby_scope->local_vars[-1] == 0) {
                free(ruby_scope->local_tbl);
            }
            ruby_scope->local_vars[-1] = 0;  /* NODE is not necessary anymore */
            ruby_scope->local_tbl = local_tbl();
        }
    }
    local_pop();
}

(parse.y)

Since local_vars can be either in the stack or in the heap, it makes the code complex to some extent. However, this is just updating local_tbl and local_vars of ruby_scope. (When SCOPE_MALLOC was set, local_vars was allocated by malloc()). And here, because there’s no meaning of using alloca(), it is forced to change its allocation method to malloc.

Block Local Variable

By the way, how about block local variables? To think about this, we have to go back to the entry point of the parser first, it is yycompile().

▼ setting `ruby_dyna_vars` aside

static NODE*
yycompile(f, line)
{
    struct RVarmap *vars = ruby_dyna_vars;
         :
    n = yyparse();
         :
    ruby_dyna_vars = vars;
}

This looks like a mere save-restore, but the point is that this does not clear the ruby_dyna_vars. This means that also in the parser it directly adds elements to the link of RVarmap created in the evaluator.

However, according to the previous description, the structure of ruby_dyna_vars differs between the parser and the evalutor. How does it deal with the difference in the way of attaching the header (RVarmap whose id=0)?

What is helpful here is the “1” of local_push(1) in top_local_init(). When the argument of local_push() becomes true, it does not attach the first header of ruby_dyna_vars. It means, it would look like Figure 1. Now, it is assured that we can refer to the block local variables of the outside scope from inside of a string to eval.

figure 1: `ruby_dyna_vars` inside `eval`

Well, it’s sure we can refer to, but didn’t you say that ruby_dyna_vars is entirely freed in the parser? What can we do if the link created at the evaluator will be freed? … I’d like the readers who noticed this to be relieved by reading the next part.

▼ `yycompile()` − freeing `ruby_dyna_vars`

    vp = ruby_dyna_vars;
    ruby_dyna_vars = vars;
    lex_strterm = 0;
    while (vp && vp != vars) {
        struct RVarmap *tmp = vp;
        vp = vp->next;
        rb_gc_force_recycle((VALUE)tmp);
    }

(parse.y)

It is designed so that the loop would stop when it reaches the link created at the evaluator (vars).

`instance_eval`

The Whole Picture

The substance of Module#module_eval is rb_mod_module_eval(), and the substance of Object#instance_eval is rb_obj_instance_eval().

▼ `rb_mod_module_eval() rb_obj_instance_eval()`

VALUE
rb_mod_module_eval(argc, argv, mod)
    int argc;
    VALUE *argv;
    VALUE mod;
{
    return specific_eval(argc, argv, mod, mod);
}

VALUE
rb_obj_instance_eval(argc, argv, self)
    int argc;
    VALUE *argv;
    VALUE self;
{
    VALUE klass;
5305
    if (rb_special_const_p(self)) {
        klass = Qnil;
    }
    else {
        klass = rb_singleton_class(self);
    }
5312
    return specific_eval(argc, argv, klass, self);
}

(eval.c)

These two methods have a common part as “a method to replace self with class”, that part is defined as specific_eval(). Figure 2 shows it and also what will be described. What with parentheses are calls by function pointers.

Whichever instance_eval or module_eval, it can accept both a block and a string, thus it branches for each particular process to yield and eval respectively. However, most of them are also common again, this part is extracted as exec_under().

But for those who reading, one have to simultaneously face at 2 times 2 = 4 ways, it is not a good plan. Therefore, here we assume only the case when

it is an instance_eval
which takes a string as its argument

. And extracting all functions under rb_obj_instance_eval() in-line, folding constants, we’ll read the result.

After Absorbed

After all, it becomes very comprehensible in comparison to the one before being absorbed.

▼specific_eval()−instance_eval, eval, string

static VALUE
instance_eval_string(self, src, file, line)
    VALUE self, src;
    const char *file;
    int line;
{
    VALUE sclass;
    VALUE result;
    int state;
    int mode;

    sclass = rb_singleton_class(self);

    PUSH_CLASS();
    ruby_class = sclass;
    PUSH_FRAME();
    ruby_frame->self       = ruby_frame->prev->self;
    ruby_frame->last_func  = ruby_frame->prev->last_func;
    ruby_frame->last_class = ruby_frame->prev->last_class;
    ruby_frame->argc       = ruby_frame->prev->argc;
    ruby_frame->argv       = ruby_frame->prev->argv;
    if (ruby_frame->cbase != sclass) {
        ruby_frame->cbase = rb_node_newnode(NODE_CREF, sclass, 0,
                                            ruby_frame->cbase);
    }
    PUSH_CREF(sclass);

    mode = scope_vmode;
    SCOPE_SET(SCOPE_PUBLIC);
    PUSH_TAG(PROT_NONE);
    if ((state = EXEC_TAG()) == 0) {
        result = eval(self, src, Qnil, file, line);
    }
    POP_TAG();
    SCOPE_SET(mode);

    POP_CREF();
    POP_FRAME();
    POP_CLASS();
    if (state) JUMP_TAG(state);

    return result;
}

It seems that this pushes the singleton class of the object to CLASS and CREF and ruby_frame->cbase. The main process is one-shot of eval(). It is unusual that things such as initializing FRAME by a struct-copy are missing, but this is also not create so much difference.

Before being absorbed

Though the author said it becomes more friendly to read, it’s possible it has been already simple since it was not absorbed, let’s check where is simplified in comparison to the before-absorbed one.

The first one is specific_eval(). Since this function is to share the code of the interface to Ruby, almost all parts of it is to parse the parameters. Here is the result of cutting them all.

▼ `specific_eval()` (simplified)

static VALUE
specific_eval(argc, argv, klass, self)
    int argc;
    VALUE *argv;
    VALUE klass, self;
{
    if (rb_block_given_p()) {

        return yield_under(klass, self);
    }
    else {

        return eval_under(klass, self, argv[0], file, line);
    }
}

(eval.c)

As you can see, this is perfectly branches in two ways based on whether there’s a block or not, and each route would never influence the other. Therefore, when reading, we should read one by one. To begin with, the absorbed version is enhanced in this point.

And file and line are irrelevant when reading yield_under(), thus in the case when the route of yield is absorbed by the main body, it might become obvious that we don’t have to think about the parse of these parameters at all.

Next, we’ll look at eval_under() and eval_under_i().

▼ `eval_under()`

static VALUE
eval_under(under, self, src, file, line)
    VALUE under, self, src;
    const char *file;
    int line;
{
    VALUE args[4];
5229
    if (ruby_safe_level >= 4) {
        StringValue(src);
    }
    else {
        SafeStringValue(src);
    }
    args[0] = self;
    args[1] = src;
    args[2] = (VALUE)file;
    args[3] = (VALUE)line;
    return exec_under(eval_under_i, under, under, args);
}

static VALUE
eval_under_i(args)
    VALUE *args;
{
    return eval(args[0], args[1], Qnil, (char*)args[2], (int)args[3]);
}

(eval.c)

In this function, in order to make its arguments single, it stores them into the args array and passes it. We can imagine that this args exists as a temporary container to pass from eval_under() to eval_under_i(), but not sure that it is truly so. It’s possible that args is modified inside evec_under().

As a way to share a code, this is a very right way to do. But for those who read it, this kind of indirect passing is incomprehensible. Particularly, because there are extra castings for file and line to fool the compiler, it is hard to imagine what were their actual types. The parts around this entirely disappeared in the absorbed version, so you don’t have to worry about getting lost.

However, it’s too much to say that absorbing and extracting always makes things easier to understand. For example, when calling exec_under(), under is passed as both the second and third arguments, but is it all right if the exec_under() side extracts the both parameter variables into under? That is to say, the second and third arguments of exec_under() are, in fact, indicating CLASS and CREF that should be pushed. CLASS and CREF are “different things”, it might be better to use different variables. Also in the previous absorbed version, for only this point,

VALUE sclass = .....;
VALUE cbase = sclass;

I thought that I would write this way, but also thought it could give the strange impression if abruptly only these variables are left, thus it was extracted as sclass. It means that this is only because of the flow of the texts.

By now, so many times, I’ve extracted arguments and functions, and for each time I repeatedly explained the reason to extract. They are

there are only a few possible patterns
the behavior can slightly change

Definitely, I’m not saying “In whatever ways extracting various things always makes things simpler”.

In whatever case, what of the first priority is the comprehensibility for ourself and not keep complying the methodology. When extracting makes things simpler, extract it. When we feel that not extracting or conversely bundling as a procedure makes things easier to understand, let us do it. As for ruby, I often extracted them because the original is written properly, but if a source code was written by a poor programmer, aggressively bundling to functions should often become a good choice.

Chapter 17: Dynamic evaluation

Overview

eval

module_eval and instance_eval

eval

eval()

top_local

Block Local Variable

instance_eval

The Whole Picture

After Absorbed

Before being absorbed

`eval`

`module_eval` and `instance_eval`

`eval`

`eval()`

`top_local`

`instance_eval`