Skip to content

__default_token__ and __default__ not called for Lark(transformer=transformer). #1582

@fyellin

Description

@fyellin

If I generate a parse tree, and then call myTransformer.transform(parse_tree), then __default_token__ and __default__ are called as documented. However if I use parser = Lark(...., transformer=myTransformer), then these methods are not called when I parse the tree.

GRAMMAR = """
    %import common.WS
    %import common.INT
    
    start     : number*
    number    : INT
    
    %ignore WS
"""

class SimpleTransformer(lark.Transformer):
    def __default__(self, data, children, meta):
        print("VISITING", data)
        return children

    def __default_token__(self, token):
        print("VISITING", token.type)
        return token

def run():
    print("Separate transformer")
    parser = lark.Lark(GRAMMAR, parser='lalr')
    parser.parse("1 2 3")
    SimpleTransformer().transform(parser.parse("1 2 3"))

    print("Combined transformer")
    parser = lark.Lark(GRAMMAR, parser='lalr', transformer=SimpleTransformer())
    parser.parse("4 5 6")

Result is

Separate transformer
VISITING INT
VISITING number
VISITING INT
VISITING number
VISITING INT
VISITING number
VISITING start
Combined transformer

Running lark version 0.12.0

Not sure if this is a bug, or if the documentation for those two methods needs to be updated to say they can't be used in inlined transformers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions