Opened 4 months ago

Closed 4 months ago

#33002 closed Cleanup/optimization (fixed)

Make DebugLexer.tokenize() more similar to Lexer.tokenize()

Reported by: Chris Jerdonek Owned by: Chris Jerdonek
Component: Template system Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Currently, DebugLexer.tokenize() looks somewhat dissimilar to Lexer.tokenize(), even though they do more or less the same thing. For example, the Lexer class's implementation contains just one call to self.create_token(), whereas DebugLexer contains three calls to it.

This ticket is to make DebugLexer.tokenize() look visibly much more the same as Lexer.tokenize(), and in particular simpler. One advantage is that it will be more obvious on inspection that DebugLexer acts the same as Lexer. Another advantage is that it will be easier to keep the two implementations in step when applying optimizations to Lexer. It will be more maintainable and less likely to introduce bugs if the implementations for the two lexers don't diverge too much.

The idea will become more clear once the PR is posted.

Change History (10)

comment:1 Changed 4 months ago by Chris Jerdonek

Has patch: set

comment:2 Changed 4 months ago by Chris Jerdonek

Patch needs improvement: set

comment:3 Changed 4 months ago by Mariusz Felisiak

Triage Stage: UnreviewedAccepted

comment:4 Changed 4 months ago by Chris Jerdonek

Patch needs improvement: unset

comment:5 Changed 4 months ago by Mariusz Felisiak

Triage Stage: AcceptedReady for checkin

comment:6 Changed 4 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In b2be7e1:

Refs #33002 -- Made template_tests.tests.py's tests test both Lexer and DebugLexer.

comment:7 Changed 4 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 65ed96fa:

Refs #33002 -- Added DebugLexer/Lexer.tokenize() test for trailing text token.

comment:8 Changed 4 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 6242c22:

Refs #33002 -- Optimized Lexer.tokenize() by skipping computing lineno when not needed.

comment:9 Changed 4 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 6fedd868:

Refs #33002 -- Renamed variable from bit to token_string in Lexer.tokenize().

comment:10 Changed 4 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In f0776a55:

Fixed #33002 -- Made DebugLexer.tokenize() more closely parallel Lexer.tokenize().

Note: See TracTickets for help on using tickets.
Back to Top