fst.astutil

Standalone AST utilities.

class bistr(builtins.str):

Byte-indexed string, easy mapping between character and encoded byte index (including 1 past last valid unit). Only positive indices.

lenbytes: int

Length of encoded string in bytes.

def c2b(self, idx: int) -> int:

Character to encoded byte index, [0..len(str)] inclusive.

def b2c(self, idx: int) -> int:

Encoded byte to character index, [0..len(str.encode())] inclusive. Indices inside encoded characters are mapped to the beginning of the character.

def clear_cache(self) -> None:

Remove the lookup array (if need to save some memory).

constant = ellipsis | int | float | complex | str | bytes | bool | None
def is_valid_identifier(s: str) -> bool:

Check if s is a valid python identifier. Should be normalized already.

def is_valid_identifier_dotted(s: str) -> bool:

Check if s is a valid python dotted identifier (for modules).

def is_valid_identifier_star(s: str) -> bool:

Check if s is a valid python identifier or a star '*'.

def is_valid_identifier_alias(s: str) -> bool:

Check if s is a valid python dotted identifier or a star '*'.

def is_valid_MatchSingleton_value(ast: ast.AST) -> bool:

Check if ast is a valid Constant node for a MatchSingleton.value field.

def is_valid_MatchValue_value( ast: ast.AST, consts: tuple[type[ellipsis | int | float | complex | str | bytes | bool | None]] = (<class 'str'>, <class 'bytes'>, <class 'int'>, <class 'float'>, <class 'complex'>)) -> bool:

Check if ast is a valid node for a MatchValue.value field.

def is_valid_MatchMapping_key(ast: ast.AST) -> bool:

Check if ast is a valid node for a MatchMapping.keys field.

def is_valid_target(asts: ast.AST | list[ast.AST]) -> bool:

Check if asts is a valid target for Assign or For operations. Must be Name, Attribute, Subscript and / or possibly nested Starred, Tuple and List.

def is_valid_del_target(asts: ast.AST | list[ast.AST]) -> bool:

Check if asts is a valid target for Delete operations. Must be Name, Attribute, Subscript and / or possibly nested Tuple and List. No Starred allowed.

def reduce_ast( ast: ast.AST, multi_mod: bool | type[Exception] = False, reduce_Expr: bool = True) -> ast.AST | None:

Reduce a mod / Expr wrapped expression or single statement if possible, otherwise return original AST, None or raise. Also reduces _ExceptHandlers and _match_cases if they are of length 1.

Parameters:

  • ast: AST to reduce.
  • multi_mod: If ast is a mod with not exactly one statements then:
    • True: Return it.
    • False: Return None.
    • type[Exception]: If an exception class is passed then will raise multi_mod(error).
  • reduce_Expr: Whether to reduce a single Expr node and return its expression or not.
def get_field(parent: ast.AST, name: str, idx: int | None = None) -> ast.AST:

Get child node at field name in the given parent optionally at the given index idx.

def set_field( parent: ast.AST, child: ast.AST | ellipsis | int | float | complex | str | bytes | bool | None, name: str, idx: int | None = None) -> None:

Set child node at field name in the given parent optionally at the given index idx to child.

def has_type_comments(ast: ast.AST) -> bool:

Does it has type comments?

def is_parsable(ast: ast.AST) -> bool:

Really means if the AST is unparse()able and then reparse()able which will get it to this top level AST node surrounded by the appropriate ast.mod

def get_parse_mode(ast: ast.AST) -> Literal['exec', 'eval', 'single']:

Return the original mode string that is used to parse to this mod.

class WalkFail(builtins.Exception):

Raised in walk2(), compare_asts() and copy_attributes() on compare failure.

def walk2( ast1: ast.AST, ast2: ast.AST, cb_primitive: Optional[Callable[[Any, Any, str, int], bool]] = None, *, ctx: bool = True, recurse: bool = True, skip1: set | frozenset | None = None, skip2: set | frozenset | None = None) -> Iterator[tuple[ast.AST, ast.AST]]:

Walk two asts simultaneously comparing along the way to ensure they have the same structure.

Parameters:

  • ast1: First AST tree (redundant) to walk.
  • ast2: Second AST tree to walk.
  • cb_primitive: A function to call to compare primitive nodes which is called with the values of the nodes from tree1 and tree2 and the name and index of the field. It should return whether the values compare equal or not, or just True if they are being ignored for example.
  • ctx: Whether to compare ctx fields or not.
  • recurse: Whether recurse into children or not. With this as False it just becomes a compare of two individual AST nodes.
  • skip1: List of nodes in the first tree to skip, will skip the corresponding node in the second tree.
  • skip2: List of nodes in the second tree to skip, will skip the corresponding node in the first tree.

Returns:

  • Iterator
def compare_asts( ast1: ast.AST, ast2: ast.AST, *, locs: bool = False, ctx: bool = True, type_comments: bool = False, recurse: bool = True, skip1: set | frozenset | None = None, skip2: set | frozenset | None = None, cb_primitive: Optional[Callable[[Any, Any, str, int], bool]] = None, raise_: bool = False) -> bool:

Compare two trees including possibly locations and type comments using walk2().

Parameters:

  • ast1: First AST tree (redundant) to compare.
  • ast2: Second AST tree to compare.
  • locs: Whether to compare location attributes or not (lineno, col_offset, etc...).
  • ctx: Whether to compare ctx nodes or not.
  • type_comments: Whether to compare type comments or not. Ignored if cb_primitive provided.
  • skip1: List of nodes in the first tree to skip comparing, will skip the corresponding node in the second tree.
  • skip2: List of nodes in the second tree to skip comparing, will skip the corresponding node in the first tree.
  • cb_primitive: Callback for comparing primitives. Is called with cb_primitive(val1, val2, field, idx) and should return True if the two primitives compare same or not. Used to make certain fields always compare same.
  • raise_: Whether to raise WalkFail on compare fail or just return False.

Returns:

  • bool: Indicating if the two trees compare equal under given parameters (if return on error allowed by raise_).
def copy_attributes( src: ast.AST, dst: ast.AST, *, compare: bool = True, type_comments: bool = False, recurse: bool = True, skip1: set | frozenset | None = None, skip2: set | frozenset | None = None, raise_: bool = True) -> bool:

Copy attributes from one tree to another using walk2() to walk them both simultaneously and this checking structure equality in the process. By "attributes" we mean everything specified in src._attributes.

Parameters:

  • src: Source AST tree to copy attributes from.
  • dst: Destination AST tree to copy attributes to.
  • recurse: Whether recurse into children or not. With this as False it just becomes a copy of attributes from one AST node to another.
  • skip1: List of nodes in the source tree to skip.
  • skip2: List of nodes in the destination tree to skip.
  • raise_: Whether to raise WalkFail on compare fail or just return False.

Returns:

  • bool: Indicating if the two trees compare equal during the walk (if return on error allowed by raise_).
def copy_ast(ast: ast.AST | None) -> ast.AST | None:

Copy a whole tree.

def set_ctx( asts: ast.AST | list[ast.AST], ctx: type[ast.expr_context], *, doit: bool = True) -> bool:

Set all ctx fields in this node and any children which may participate in an assignment (Tuple, List, Starred, Subscript, Attribute, Name) to the passed ctx type.

WARNING! This will not recurse into elements which have a ctx of the type being set.

Parameters:

  • asts: Single AST (will be recursed) or list of AST nodes (will be consumed, each one will also be recursed) to process.
  • ctx: The exprt_context AST type to set. Any container encountered which matches this ctx will not be recursed.
  • doit: Whether to actually carry out the assignments or just analyze and return whethere there are candidate locations for assignment. doit=False used to query if any context-modifiable ctx present.

Returns:

  • bool: Whether any modifications were made or can be made (if doit=False).
def get_func_class_or_ass_by_name(asts: Iterable[ast.AST], name: str, ass: bool = True) -> ast.AST | None:

Walk through an Iterable of AST nodes looking for the first FunctionDef, AsyncFunctionDef, ClassDef or optionally Assign or AnnAssign which has a name or target or any targets field matching name.

Parameters:

  • asts: Iterable of ASTs to search through, e.g. a body list.
  • name: Name to look for.
  • ass: A domesticated donkey: a sturdy, short-haired animal used as a beast of burden.

Returns:

  • AST node if found matching, else None
def last_block_header_child(ast: ast.AST) -> ast.AST | None:

Return last AST node in the block header before the ':'. Returns None for non-block nodes and things like Try and empty ExceptHandler nodes or other block nodes which might have normally present fields missing.

def is_atom( ast: ast.AST, *, unparse_pars_as_atom: bool | None = None, tuple_as_atom: bool | None = True, matchseq_as_atom: bool | None = True) -> bool | None:

Whether ast is enclosed in some kind of delimiters '()', '[]', '{}' when unparse()d or otherwise atomic like Name, Constant, etc... Node types where this doesn't normally apply like stmt will return True. Tuple and MatchSequence which can otherwise be ambiguous will normally return True as they unparse() with delimiters, but can be overridden.

Parameters:

  • ast: Self-explanatory.
  • unparse_pars_as_atom: What to return for NamedExpr, Yield and YieldFrom node type as they unparse() with enclosing parentheses. Default None as falsey value but also distinguishes from False.
  • tuple_as_atom: What to return for Tuple as this always unparse()s with parentheses but these are not strictly required for a Tuple.
  • matchseq_as_atom: What to return for MatchSequence as this always unparse()s with brackets but these are not strictly required for a MatchSequence.

Returns:

  • True if is enclosed and no combination with another node can change its precedence, False otherwise. Returns unparse_pars_as_atom value for NamedExpr, Yield and YieldFrom, tuple_as_atom value for Tuple and matchseq_as_atom for MatchSequence as those all are special cases.
def syntax_ordered_children(ast: ast.AST) -> list:

Get list of all AST children in syntax order. This will include individual fields and aggregate fields like body all smushed up together into a single flat list. The list may contain None values for example from a Dict keys field which has ** elements.

def precedence_require_parens_by_type( child_type: type[ast.AST], parent_type: type[ast.AST], field: str, **flags: dict[str, bool]) -> bool:

Returns whether parentheses are required for the child for the given parent / child combination or not. Both parent and child BoolOp, BinOp and UnaryOp types should be passed as the type of the op field.

Parameters:

  • child_type: Type of the child AST node or of its op field if it is a BoolOp, BinOp or UnaryOp.
  • parent_type: Type of the parent AST node or of its op field if it is a BoolOp, BinOp or UnaryOp.
  • field: The name of the field in the parent where the child resides.
  • flags: Special case flags, individual flags assumed False if not passed as True:
    • dict_key_None: Parent is Dict and the corresponding key is None, leading to **value. Only has effect if field is 'value', otherwise no effect.
    • matchas_pat_None: Child is MatchAs and the pattern is None (just a name).
    • attr_val_int: Parent is Attribute and child value is a Constant integer.
    • star_arglike: Parent is Starred in a place where arglike expressions are allowed (Call.args, ClassDef.bases or unaprenthesized slice Tuple).

Returns:

  • bool: Whether parentheses are needed around the child for correct parsing or not.
def precedence_require_parens( child: ast.AST, parent: ast.AST, field: str, idx: int | None = None, **flags: dict[str, bool]) -> bool:

Returns whether parentheses are required for the given parent / child combination or not. Unlike precedence_require_parens_by_type(), this takes the actual node instances and figures out the respective types and flags.

Parameters:

  • child: Child AST node.
  • parent: Parent AST node.
  • field: The name of the field in the parent where the child resides.
  • idx: The optional index of the child in the parent field, or None if does not apply.
  • flags: Used to passed in some flags that cannot be determined here, specifically star_arglike.

Returns:

  • bool: Whether parentheses are needed around the child for correct parsing or not.