Backends

Parser backends convert C/C++ source code into headerkit's IR. autopxd2 does not define backends of its own; it selects among the ones headerkit registers.

Backend Registry

backends

Parser backends for headerkit.

This package contains parser backend implementations that convert C/C++ source code into the headerkit IR (Intermediate Representation).

Available Backends

libclang LLVM clang-based parser with full C++ support. Requires system libclang library. Uses vendored clang Python bindings.

Example

::

from headerkit.backends import get_backend, list_backends

# Get the default backend
backend = get_backend()

# Get a specific backend
backend = get_backend("libclang")

# List available backends
for name in list_backends():
    print(name)

get_backend(name=None)

Get a parser backend instance.

Returns a new instance of the requested backend. If no name is provided, returns the default backend (libclang).

::

from headerkit.backends import get_backend

# Get default backend
backend = get_backend()

# Get libclang backend
clang = get_backend("libclang")

# Parse a header
header = backend.parse(code, "myheader.h")

Parameters:

Name	Type	Description	Default
name	str | None	Backend name (e.g., "libclang"), or None for the default backend.	None

Returns:

Type	Description
ParserBackend	New instance of the requested backend.

Raises:

Type	Description
ValueError	If the requested backend is not available. Example

list_backends()

List names of all registered backends.

::

from headerkit.backends import list_backends

for name in list_backends():
    print(f"Available: {name}")

Returns:

Type	Description
list[str]	List of backend names that can be passed to :func:get_backend. Example

register_backend(name, backend_class, is_default=False, priority=Priority.STANDARD)

Register a parser backend.

Called by backend modules during import to add themselves to the registry. The first registered backend becomes the default unless is_default is explicitly set on a later registration.

Parameters:

Name	Type	Description	Default
name	str	Unique name for the backend (e.g., "libclang").	required
backend_class	type[ParserBackend]	Class implementing the :class:~headerkit.ir.ParserBackend protocol.	required
is_default	bool	If True, this becomes the default backend for :func:get_backend.	False
priority	int	Execution priority for hook dispatch.	STANDARD

libclang Backend

libclang

libclang-based parser backend.

This backend uses libclang (LLVM's C/C++ parser) to parse header files. It provides full C/C++ support including templates, namespaces, and classes.

Requirements

• System libclang library must be installed
• headerkit includes vendored clang Python bindings that are version-matched automatically to the system LLVM version

The backend class is always registered. If system libclang is not available, the error surfaces when the backend is actually used (e.g. when parse() is called), not at import time.

Advantages

• Full C++ support (classes, templates, namespaces)
• Handles complex preprocessor constructs
• Uses the same parser as production compilers
• Better error messages with source locations

Limitations

• Macro extraction is limited due to Python bindings constraints
• Requires system libclang installation

Example

::

from headerkit.backends.libclang import LibclangBackend

backend = LibclangBackend()
header = backend.parse(code, "myheader.hpp", extra_args=["-std=c++17"])

LibclangBackend

Parser backend using libclang.

Uses LLVM's libclang to parse C and C++ code. This backend supports the full C++ language including templates, classes, and namespaces.

Properties

name : str Returns "libclang". supports_macros : bool Returns False - macro extraction is limited in Python bindings. supports_cpp : bool Returns True - full C++ support.

Example

::

from headerkit.backends.libclang import LibclangBackend

backend = LibclangBackend()

# Parse C++ code with specific standard
header = backend.parse(
    code,
    "myheader.hpp",
    extra_args=["-std=c++17", "-DDEBUG=1"]
)

Source code in headerkit/backends/libclang.py

class LibclangBackend:
    """Parser backend using libclang.

    Uses LLVM's libclang to parse C and C++ code. This backend supports
    the full C++ language including templates, classes, and namespaces.

    Properties
    ----------
    name : str
        Returns ``"libclang"``.
    supports_macros : bool
        Returns ``False`` - macro extraction is limited in Python bindings.
    supports_cpp : bool
        Returns ``True`` - full C++ support.

    Example
    -------
    ::

        from headerkit.backends.libclang import LibclangBackend

        backend = LibclangBackend()

        # Parse C++ code with specific standard
        header = backend.parse(
            code,
            "myheader.hpp",
            extra_args=["-std=c++17", "-DDEBUG=1"]
        )
    """

    supported_languages: frozenset[str] = frozenset({"c", "cpp"})
    supported_classifications: frozenset[str] = frozenset({"header", "source"})

    def __init__(self) -> None:
        self._index: Any = None
        # Cache for parsed headers (path -> Header) to avoid re-parsing
        self._parse_cache: dict[str, Header] = {}
        # Visited set to prevent circular includes
        self._visited: set[str] = set()
        # Normalized paths clang itself classified as system headers, accumulated
        # across every translation unit this backend has parsed.
        self._clang_system_headers: set[str] = set()
        _LIVE_BACKENDS.add(self)

    def _is_system_include(self, header_path: str, project_prefixes: tuple[str, ...] | None) -> bool:
        """Decide whether an included header is a system header.

        clang's answer is authoritative: it knows which search paths it treated
        as system (``-isystem``, its resource directory, the platform defaults),
        which no path heuristic can track across platforms. The heuristic in
        :func:`_is_system_header` remains as the fallback for paths clang has
        not classified in this process.

        :param header_path: Path to the included header.
        :param project_prefixes: Path prefixes that override every system classification.
        :returns: True if the header must not be descended into.
        """
        if _matches_project_prefix(header_path, project_prefixes):
            return False
        if normalize_path(str(header_path)) in self._clang_system_headers:
            return True
        return _is_system_header(header_path, project_prefixes)

    @property
    def name(self) -> str:
        return "libclang"

    @property
    def supports_macros(self) -> bool:
        # Supports simple numeric macros (#define NAME 123)
        # Complex macros (expressions, function-like) are not supported
        return True

    @property
    def supports_cpp(self) -> bool:
        return True

    def is_available(self) -> bool:
        return _configure_libclang()

    def _get_index(self) -> Any:
        """Get or create the clang index."""
        if self._index is None:
            self._index = _cindex.Index.create()
        return self._index

    def _resolve_include_path(
        self,
        include_path: str,
        base_dir: str,
        include_dirs: list[str],
    ) -> str | None:
        """Resolve an include path to an absolute path.

        :param include_path: The include path as it appears in the header
        :param base_dir: Directory of the including file
        :param include_dirs: List of include search directories
        :returns: Absolute path to the header file, or None if not found
        """
        # If already absolute, return as-is
        if os.path.isabs(include_path):
            if os.path.exists(include_path):
                return os.path.abspath(include_path)
            return None

        # Try relative to base directory first
        candidate = os.path.join(base_dir, include_path)
        if os.path.exists(candidate):
            return os.path.abspath(candidate)

        # Try each include directory
        for inc_dir in include_dirs:
            candidate = os.path.join(inc_dir, include_path)
            if os.path.exists(candidate):
                return os.path.abspath(candidate)

        return None

    def _parse_header_file(
        self,
        header_path: str,
        include_dirs: list[str],
        extra_args: list[str],
        use_default_includes: bool,
    ) -> Header:
        """Parse a single header file.

        :param header_path: Absolute path to header file
        :param include_dirs: Include directories
        :param extra_args: Extra compiler arguments
        :param use_default_includes: Whether to use system includes
        :returns: Parsed Header IR
        """
        # Check cache
        if header_path in self._parse_cache:
            return self._parse_cache[header_path]

        # Read the file
        with open(header_path, encoding="utf-8", errors="replace") as f:
            code = f.read()

        # Parse using the main parse method
        # Use the basename for the filename to match expected behavior
        filename = os.path.basename(header_path)
        header = self.parse(
            code,
            filename,
            include_dirs=include_dirs,
            extra_args=extra_args,
            use_default_includes=use_default_includes,
            recursive_includes=False,  # Prevent infinite recursion
        )

        # Cache the result
        self._parse_cache[header_path] = header
        return header

    def _parse_recursively(
        self,
        main_header: Header,
        main_path: str,
        include_dirs: list[str],
        extra_args: list[str],
        use_default_includes: bool,
        max_depth: int,
        current_depth: int = 0,
        project_prefixes: tuple[str, ...] | None = None,
        allowlist_paths: _PathSet | None = None,
        denylist_paths: _PathSet | None = None,
    ) -> Header:
        """Recursively parse included headers and combine declarations.

        :param main_header: The main header that was parsed
        :param main_path: Path to the main header file
        :param include_dirs: Include directories
        :param extra_args: Extra compiler arguments
        :param use_default_includes: Whether to use system includes
        :param max_depth: Maximum recursion depth
        :param current_depth: Current recursion depth
        :param project_prefixes: Optional tuple of path prefixes to treat as project (not system)
        :param allowlist_paths: Resolved allowlist entries, or None when the caller supplied
            none. Included headers not named here are not descended into.
        :param denylist_paths: Resolved denylist entries, or None. Named headers are not
            descended into even when the allowlist names them -- deny wins over allow.
        :returns: Combined Header with declarations from all includes
        """
        if current_depth >= max_depth:
            return main_header

        all_declarations: list[Declaration] = list(main_header.declarations)
        main_dir = os.path.dirname(os.path.abspath(main_path))

        for include_path in sorted(main_header.included_headers):
            # Skip system headers (unless allowlisted via project_prefixes)
            if self._is_system_include(include_path, project_prefixes):
                continue

            # Get absolute path
            abs_path = self._resolve_include_path(
                include_path,
                main_dir,
                include_dirs,
            )

            if abs_path is None:
                # Could not resolve - skip
                continue

            # Check if already visited (circular include)
            if abs_path in self._visited:
                continue

            # The allowlist and denylist narrow the *merged* result, not only the
            # main translation unit: an included header the caller did not allow,
            # or explicitly denied, is never descended into and so contributes
            # nothing to the merge.  Deny wins over allow.
            resolved_include = _resolve_path(abs_path)
            if denylist_paths is not None and denylist_paths.matches(resolved_include):
                continue
            if allowlist_paths is not None and not allowlist_paths.matches(resolved_include):
                continue

            self._visited.add(abs_path)

            try:
                # Parse the included header
                sub_header = self._parse_header_file(
                    abs_path,
                    include_dirs,
                    extra_args,
                    use_default_includes,
                )

                # Recursively process its includes
                sub_header = self._parse_recursively(
                    sub_header,
                    abs_path,
                    include_dirs,
                    extra_args,
                    use_default_includes,
                    max_depth,
                    current_depth + 1,
                    project_prefixes,
                    allowlist_paths,
                    denylist_paths,
                )

                # Add declarations from sub-header
                all_declarations.extend(sub_header.declarations)

            except (OSError, RuntimeError, ValueError):
                import logging

                logging.getLogger(__name__).debug("Failed to parse included header: %s", abs_path, exc_info=True)
                continue

        # Deduplicate declarations
        unique_declarations = _deduplicate_declarations(all_declarations)

        # Return combined header
        return Header(
            path=main_header.path,
            declarations=unique_declarations,
            included_headers=main_header.included_headers,
        )

    def parse(
        self,
        code: str,
        filename: str,
        include_dirs: list[str] | None = None,
        extra_args: list[str] | None = None,
        *,
        use_default_includes: bool = True,
        recursive_includes: bool = True,
        max_depth: int = 10,
        project_prefixes: tuple[str, ...] | None = None,
        allowlist: list[str] | None = None,
        denylist: list[str] | None = None,
    ) -> Header:
        """Parse C/C++ code using libclang.

        Handles raw (unpreprocessed) code and performs preprocessing internally.

        Traversal and filtering
        -----------------------
        ``recursive_includes`` (True by default) governs *traversal*: it descends
        into each non-system header named by an ``#include``, parses it as its own
        translation unit, and merges what it declares.  ``project_prefixes``
        (through :func:`_is_system_header`) decides which paths count as project
        rather than system headers, ``max_depth`` bounds the descent, and an
        internal visited set stops mutually including headers from recursing
        forever.

        ``allowlist`` and ``denylist`` govern *filtering*, and they narrow the
        merged result in **both** modes.  A header the allowlist does not name is
        neither descended into nor admitted out of ``filename``'s own translation
        unit; a header the denylist names is excluded by both routes as well.

        * ``recursive_includes=True`` with no ``allowlist`` emits the declarations
          of every non-system included header.
        * ``recursive_includes=True`` with ``allowlist=["other.h"]`` emits
          ``filename``'s own declarations and ``other.h``'s, and nothing else.
        * An allowlist naming nothing that is actually included yields
          ``filename``'s declarations alone.  This is not an error.
        * ``recursive_includes=False`` with no ``allowlist`` yields ``filename``'s
          declarations alone.
        * **Deny wins over allow.**  A file named by both lists is excluded.
        * The main file is never denied.  ``filename`` always contributes its own
          declarations, because returning an empty result for the file the caller
          asked to parse is a silent failure with nothing to explain it.

        :param code: C/C++ source code to parse (raw, not preprocessed).
        :param filename: Source filename for error messages and location tracking.
        :param include_dirs: Additional include directories (converted to ``-I`` flags).
        :param extra_args: Additional compiler arguments (e.g., ``["-std=c++17"]``).
        :param use_default_includes: If True (default), automatically detect and add
            system include directories by querying the system clang compiler.
            Set to False to disable this behavior.
        :param recursive_includes: If True (default), descend into every non-system
            included header and merge its declarations into the result. System
            headers are always skipped. Set to False to parse only the main file,
            which is the only way to keep included declarations out entirely.
        :param max_depth: Maximum recursion depth for include processing (default 10).
            Prevents infinite recursion from circular includes.
        :param project_prefixes: Optional tuple of path prefixes to treat as project
            headers (not system). Use this for umbrella headers of libraries installed
            in system locations (e.g., ``("/opt/homebrew/include/sodium",)``).
        :param allowlist: Files whose declarations are kept, alongside ``filename``'s
            own.  ``None`` keeps every non-system file reached by traversal; a list
            narrows the result to exactly ``filename`` plus the files it names, in
            both ``recursive_includes`` modes.  An absolute entry is used as-is; a
            relative entry (including a bare basename) is resolved against the
            directory of ``filename`` first, then each of ``include_dirs``, then the
            current working directory.  An entry containing ``*``, ``?`` or ``[`` is
            an :mod:`fnmatch` pattern whose directory part is resolved the same way.
            Matching is on whole absolute, symlink-resolved paths -- never on
            substrings, so ``er.h`` does not match ``other.h``.
        :param denylist: Files whose declarations are dropped, using the same
            resolution and glob rules as ``allowlist``.  Deny wins over allow: a
            file named by both lists is excluded.  A denylist with no allowlist
            means "everything except these".  An entry matching nothing is not an
            error.  ``filename`` itself cannot be denied.
        :returns: :class:`~headerkit.ir.Header` containing parsed declarations.
        :raises RuntimeError: If parsing fails with errors.

        Example
        -------
        ::

            # Basic usage
            header = backend.parse(
                code,
                "myheader.hpp",
                include_dirs=["/usr/local/include"],
                extra_args=["-std=c++17", "-DNDEBUG"]
            )

            # Umbrella header (all-includes) pattern: descend into the includes
            # and keep everything they declare.  This is the default.
            header = backend.parse(
                code,
                "LibraryAll.h",
                include_dirs=["./include"],
                recursive_includes=True
            )

            # Keep only one included header's declarations: no descent, and
            # exactly one file admitted from the main translation unit.
            header = backend.parse(
                code,
                "main.h",
                recursive_includes=False,
                allowlist=["other.h"]
            )

            # Main file only -- nothing reached through #include survives
            header = backend.parse(
                code,
                "main.h",
                recursive_includes=False
            )

            # Umbrella header in system location
            header = backend.parse(
                code,
                "sodium.h",
                include_dirs=["/opt/homebrew/include"],
                project_prefixes=("/opt/homebrew/include/sodium",)  # Allowlist sodium/*
            )
        """
        # Ensure libclang is configured before parsing.  This is a no-op
        # after the first successful load (short-circuits on Config.loaded).
        if not _configure_libclang():
            from headerkit.backends import LibclangUnavailableError

            raise LibclangUnavailableError(
                "libclang shared library not found. Install LLVM/clang or run: pip install libclang"
            )

        args: list[str] = []

        is_cplus = _detect_cplus(filename, extra_args)

        # Add user-specified include directories FIRST
        # This is important for C++ where user headers may need to come before system libc++
        if include_dirs:
            for inc_dir in include_dirs:
                args.append(f"-I{inc_dir}")

        # Add system include directories if enabled
        # Always add them when use_default_includes=True, regardless of other -I flags
        if use_default_includes:
            args.extend(get_system_include_dirs(cplus=is_cplus))
            has_stddef = any(
                os.path.isfile(os.path.join(p.removeprefix("-isystem").removeprefix("-I"), "stddef.h")) for p in args
            )
            if not has_stddef:
                for builtin_inc in _find_libclang_builtin_include_dirs():
                    args.append(f"-isystem{builtin_inc}")

        # Add extra arguments
        if extra_args:
            args.extend(extra_args)

        # Auto-resolve Nim's standard library directory when parsing a header that includes "nimbase.h"
        uses_nimbase = "nimbase.h" in code
        nim_libpath_added = False
        if uses_nimbase:
            if "NIM_INTBITS" not in code and not any("NIM_INTBITS" in a for a in args):
                args.append(f"-DNIM_INTBITS={struct.calcsize('P') * 8}")
            if not _has_nimbase_in_dirs(include_dirs, extra_args, filename):
                nim_libpath = _find_nim_libpath()
                if nim_libpath is not None:
                    args.append(f"-isystem{nim_libpath}")
                    nim_libpath_added = True

        # Parse the code with detailed preprocessing record for macro extraction
        index = self._get_index()
        tu = index.parse(
            filename,
            args=args,
            unsaved_files=[(filename, code)],
            options=_cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD,
        )

        # If an included header (e.g. via umbrella #include) needed nimbase.h, retry with Nim libpath
        if not nim_libpath_added and any(
            diag.severity >= _cindex.Diagnostic.Error and "nimbase.h" in str(diag.spelling) for diag in tu.diagnostics
        ):
            uses_nimbase = True
            if "NIM_INTBITS" not in code and not any("NIM_INTBITS" in a for a in args):
                args.append(f"-DNIM_INTBITS={struct.calcsize('P') * 8}")
            if not _has_nimbase_in_dirs(include_dirs, extra_args, filename):
                nim_libpath = _find_nim_libpath()
                if nim_libpath is not None:
                    args.append(f"-isystem{nim_libpath}")
                    nim_libpath_added = True
                    tu = index.parse(
                        filename,
                        args=args,
                        unsaved_files=[(filename, code)],
                        options=_cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD,
                    )

        # Debug Nim --header output emits inline nimFrame() bodies referencing undeclared
        # framePtr__system_u*; retry skipping function bodies if a Nim header fails body check
        if uses_nimbase and any(diag.severity >= _cindex.Diagnostic.Error for diag in tu.diagnostics):
            tu = index.parse(
                filename,
                args=args,
                unsaved_files=[(filename, code)],
                options=(
                    _cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD
                    | _cindex.TranslationUnit.PARSE_SKIP_FUNCTION_BODIES
                ),
            )

        # Check for fatal errors
        for diag in tu.diagnostics:
            if diag.severity >= _cindex.Diagnostic.Error:
                raise RuntimeError(f"Parse error: {diag.spelling}")

        # Collect included headers
        included_headers: set[str] = set()
        for inclusion in tu.get_includes():
            # inclusion.include is a File with name attribute
            header_path = str(inclusion.include.name)
            # Store full path - caller can extract basename if needed
            included_headers.add(header_path)
            # Ask clang whether it treated this file as a system header. This is the
            # authoritative answer -- clang knows its own resource directory and
            # -isystem search paths, which vary per platform and per toolchain
            # version, so the path heuristic alone leaks libc internals on Linux.
            if _location_is_in_system_header(tu, inclusion.include):
                self._clang_system_headers.add(normalize_path(header_path))

        # Convert to IR
        search_dirs = _filter_search_dirs(filename, include_dirs)
        allowlist_paths = None if allowlist is None else _PathSet(allowlist, search_dirs)
        denylist_paths = None if denylist is None else _PathSet(denylist, search_dirs)
        converter = ClangASTConverter(
            filename,
            project_prefixes=project_prefixes,
            is_cplus=is_cplus,
            allowlist_paths=allowlist_paths,
            denylist_paths=denylist_paths,
            parse_args=tuple(args),
        )
        header = converter.convert(tu)

        # Attach included headers to the IR
        header.included_headers = included_headers

        # Check if we should do recursive include processing
        # ``recursive_includes`` is the caller's answer, so nothing here second-guesses
        # it from the header's shape. Scope is bounded by _is_system_header /
        # project_prefixes, allowlist, max_depth, and self._visited.
        has_project_include = any(not self._is_system_include(h, project_prefixes) for h in header.included_headers)
        if recursive_includes and has_project_include:
            # Reset visited set for each top-level parse
            self._visited = set()
            # Add current file to visited
            if os.path.exists(filename):
                abs_filename = os.path.abspath(filename)
            else:
                # For in-memory code, use filename as-is
                abs_filename = filename
            self._visited.add(abs_filename)

            # Recursively parse included headers
            header = self._parse_recursively(
                header,
                abs_filename,
                include_dirs or [],
                extra_args or [],
                use_default_includes,
                max_depth,
                project_prefixes=project_prefixes,
                allowlist_paths=allowlist_paths,
                denylist_paths=denylist_paths,
            )

        # Included headers are parsed in isolation, so a macro that a *later*
        # header redefines is otherwise emitted with the stale value it had in
        # the header that first defined it. The main translation unit's
        # preprocessing record is the only place the real, ordered macro history
        # is visible, so it decides the final state.
        self._apply_final_macro_state(header, tu, converter)
        self._drop_undefined_macros(header, filename, code, args)

        # Record the language clang actually parsed this unit as. Writers need it:
        # a `std::string` field and a C `typedef struct {...} string;` reach the IR
        # as the identical `CType(name="string")`, and nothing else in the IR tells
        # them apart. `_detect_cplus` is already the authority that chose the parse
        # mode, so this records that decision rather than re-deriving one.
        header.language = "cpp" if is_cplus else "c"

        return header

    def _drop_undefined_macros(self, header: Header, filename: str, code: str, args: list[str]) -> None:
        """Remove macro Constants for names that are no longer defined at end of translation.

        libclang's preprocessing record carries no entry for ``#undef``, so a
        macro a header undefines is still reported as defined and would be
        emitted as a declaration of a symbol the compiler cannot see. Re-parsing
        the same source with an ``#ifdef`` probe appended per candidate asks the
        preprocessor itself which names survive, so the answer cannot disagree
        with the compiler.
        """
        candidates = [
            d.name
            for d in header.declarations
            if (isinstance(d, Constant) and d.is_macro) or isinstance(d, MacroFunction)
        ]
        if not candidates:
            return

        probe = [code, "\n"]
        probe.extend(
            f"#ifdef {name}\nint {_MACRO_PROBE_PREFIX}{index};\n#endif\n" for index, name in enumerate(candidates)
        )

        try:
            probe_tu = self._get_index().parse(
                filename,
                args=args,
                unsaved_files=[(filename, "".join(probe))],
                options=_cindex.TranslationUnit.PARSE_SKIP_FUNCTION_BODIES,
            )
        except _cindex.TranslationUnitLoadError:
            return

        if any(diag.severity >= _cindex.Diagnostic.Error for diag in probe_tu.diagnostics):
            # The probe did not compile; its silence is not evidence of an #undef.
            return

        defined: set[str] = set()
        for cur in probe_tu.cursor.get_children():
            if cur.kind != CursorKind.VAR_DECL:
                continue
            spelling = cur.spelling
            if spelling.startswith(_MACRO_PROBE_PREFIX):
                index = spelling[len(_MACRO_PROBE_PREFIX) :]
                if index.isdigit():
                    defined.add(candidates[int(index)])

        header.declarations = [
            d
            for d in header.declarations
            if not (
                ((isinstance(d, Constant) and d.is_macro) or isinstance(d, MacroFunction)) and d.name not in defined
            )
        ]

    @staticmethod
    def _final_macro_state(tu: Any, converter: ClangASTConverter) -> dict[str, Constant | MacroFunction | None]:
        """Map each macro name to the declaration its LAST ``#define`` yields, or None.

        The preprocessing record lists ``#define`` directives in the order the
        preprocessor met them and omits directives in branches it did not take,
        so iterating it and letting later entries win reproduces the macro table
        as it stands at the end of the translation unit.
        """
        state: dict[str, Constant | MacroFunction | None] = {}
        for cur in tu.cursor.get_children():
            if cur.kind != CursorKind.MACRO_DEFINITION:
                continue
            if cur.location.file is None:
                # Compiler builtin, not written in any source file.
                continue
            name = cur.spelling
            if name:
                state[name] = converter._build_macro_decl(cur, name)
        return state

    @classmethod
    def _apply_final_macro_state(cls, header: Header, tu: Any, converter: ClangASTConverter) -> None:
        """Drop or correct macro declarations that the final macro table contradicts."""
        state = cls._final_macro_state(tu, converter)
        if not state:
            return

        kept: list[Declaration] = []
        for decl in header.declarations:
            if (
                (isinstance(decl, Constant) and decl.is_macro) or isinstance(decl, MacroFunction)
            ) and decl.name in state:
                final = state[decl.name]
                if final is None:
                    # Last definition has no value - emitting it would not compile.
                    continue
                if not _same_macro_decl(decl, final):
                    kept.append(final)
                    continue
            kept.append(decl)
        header.declarations = kept

parse(code, filename, include_dirs=None, extra_args=None, *, use_default_includes=True, recursive_includes=True, max_depth=10, project_prefixes=None, allowlist=None, denylist=None)

Parse C/C++ code using libclang.

Handles raw (unpreprocessed) code and performs preprocessing internally.

Traversal and filtering

recursive_includes (True by default) governs traversal: it descends into each non-system header named by an #include, parses it as its own translation unit, and merges what it declares. project_prefixes (through :func:_is_system_header) decides which paths count as project rather than system headers, max_depth bounds the descent, and an internal visited set stops mutually including headers from recursing forever.

allowlist and denylist govern filtering, and they narrow the merged result in both modes. A header the allowlist does not name is neither descended into nor admitted out of filename's own translation unit; a header the denylist names is excluded by both routes as well.

• recursive_includes=True with no allowlist emits the declarations of every non-system included header.
• recursive_includes=True with allowlist=["other.h"] emits filename's own declarations and other.h's, and nothing else.
• An allowlist naming nothing that is actually included yields filename's declarations alone. This is not an error.
• recursive_includes=False with no allowlist yields filename's declarations alone.
• Deny wins over allow. A file named by both lists is excluded.
• The main file is never denied. filename always contributes its own declarations, because returning an empty result for the file the caller asked to parse is a silent failure with nothing to explain it.

::

# Basic usage
header = backend.parse(
    code,
    "myheader.hpp",
    include_dirs=["/usr/local/include"],
    extra_args=["-std=c++17", "-DNDEBUG"]
)

# Umbrella header (all-includes) pattern: descend into the includes
# and keep everything they declare.  This is the default.
header = backend.parse(
    code,
    "LibraryAll.h",
    include_dirs=["./include"],
    recursive_includes=True
)

# Keep only one included header's declarations: no descent, and
# exactly one file admitted from the main translation unit.
header = backend.parse(
    code,
    "main.h",
    recursive_includes=False,
    allowlist=["other.h"]
)

# Main file only -- nothing reached through #include survives
header = backend.parse(
    code,
    "main.h",
    recursive_includes=False
)

# Umbrella header in system location
header = backend.parse(
    code,
    "sodium.h",
    include_dirs=["/opt/homebrew/include"],
    project_prefixes=("/opt/homebrew/include/sodium",)  # Allowlist sodium/*
)

Parameters:

Name	Type	Description	Default
code	str	C/C++ source code to parse (raw, not preprocessed).	required
filename	str	Source filename for error messages and location tracking.	required
include_dirs	list[str] | None	Additional include directories (converted to -I flags).	None
extra_args	list[str] | None	Additional compiler arguments (e.g., ["-std=c++17"]).	None
use_default_includes	bool	If True (default), automatically detect and add system include directories by querying the system clang compiler. Set to False to disable this behavior.	True
recursive_includes	bool	If True (default), descend into every non-system included header and merge its declarations into the result. System headers are always skipped. Set to False to parse only the main file, which is the only way to keep included declarations out entirely.	True
max_depth	int	Maximum recursion depth for include processing (default 10). Prevents infinite recursion from circular includes.	10
project_prefixes	tuple[str, ...] | None	Optional tuple of path prefixes to treat as project headers (not system). Use this for umbrella headers of libraries installed in system locations (e.g., ("/opt/homebrew/include/sodium",)).	None
allowlist	list[str] | None	Files whose declarations are kept, alongside filename's own. None keeps every non-system file reached by traversal; a list narrows the result to exactly filename plus the files it names, in both recursive_includes modes. An absolute entry is used as-is; a relative entry (including a bare basename) is resolved against the directory of filename first, then each of include_dirs, then the current working directory. An entry containing *, ? or [ is an :mod:fnmatch pattern whose directory part is resolved the same way. Matching is on whole absolute, symlink-resolved paths -- never on substrings, so er.h does not match other.h.	None
denylist	list[str] | None	Files whose declarations are dropped, using the same resolution and glob rules as allowlist. Deny wins over allow: a file named by both lists is excluded. A denylist with no allowlist means "everything except these". An entry matching nothing is not an error. filename itself cannot be denied.	None

Returns:

Type	Description
Header	:class:~headerkit.ir.Header containing parsed declarations.

Raises:

Type	Description
RuntimeError	If parsing fails with errors. Example

Source code in headerkit/backends/libclang.py

def parse(
    self,
    code: str,
    filename: str,
    include_dirs: list[str] | None = None,
    extra_args: list[str] | None = None,
    *,
    use_default_includes: bool = True,
    recursive_includes: bool = True,
    max_depth: int = 10,
    project_prefixes: tuple[str, ...] | None = None,
    allowlist: list[str] | None = None,
    denylist: list[str] | None = None,
) -> Header:
    """Parse C/C++ code using libclang.

    Handles raw (unpreprocessed) code and performs preprocessing internally.

    Traversal and filtering
    -----------------------
    ``recursive_includes`` (True by default) governs *traversal*: it descends
    into each non-system header named by an ``#include``, parses it as its own
    translation unit, and merges what it declares.  ``project_prefixes``
    (through :func:`_is_system_header`) decides which paths count as project
    rather than system headers, ``max_depth`` bounds the descent, and an
    internal visited set stops mutually including headers from recursing
    forever.

    ``allowlist`` and ``denylist`` govern *filtering*, and they narrow the
    merged result in **both** modes.  A header the allowlist does not name is
    neither descended into nor admitted out of ``filename``'s own translation
    unit; a header the denylist names is excluded by both routes as well.

    * ``recursive_includes=True`` with no ``allowlist`` emits the declarations
      of every non-system included header.
    * ``recursive_includes=True`` with ``allowlist=["other.h"]`` emits
      ``filename``'s own declarations and ``other.h``'s, and nothing else.
    * An allowlist naming nothing that is actually included yields
      ``filename``'s declarations alone.  This is not an error.
    * ``recursive_includes=False`` with no ``allowlist`` yields ``filename``'s
      declarations alone.
    * **Deny wins over allow.**  A file named by both lists is excluded.
    * The main file is never denied.  ``filename`` always contributes its own
      declarations, because returning an empty result for the file the caller
      asked to parse is a silent failure with nothing to explain it.

    :param code: C/C++ source code to parse (raw, not preprocessed).
    :param filename: Source filename for error messages and location tracking.
    :param include_dirs: Additional include directories (converted to ``-I`` flags).
    :param extra_args: Additional compiler arguments (e.g., ``["-std=c++17"]``).
    :param use_default_includes: If True (default), automatically detect and add
        system include directories by querying the system clang compiler.
        Set to False to disable this behavior.
    :param recursive_includes: If True (default), descend into every non-system
        included header and merge its declarations into the result. System
        headers are always skipped. Set to False to parse only the main file,
        which is the only way to keep included declarations out entirely.
    :param max_depth: Maximum recursion depth for include processing (default 10).
        Prevents infinite recursion from circular includes.
    :param project_prefixes: Optional tuple of path prefixes to treat as project
        headers (not system). Use this for umbrella headers of libraries installed
        in system locations (e.g., ``("/opt/homebrew/include/sodium",)``).
    :param allowlist: Files whose declarations are kept, alongside ``filename``'s
        own.  ``None`` keeps every non-system file reached by traversal; a list
        narrows the result to exactly ``filename`` plus the files it names, in
        both ``recursive_includes`` modes.  An absolute entry is used as-is; a
        relative entry (including a bare basename) is resolved against the
        directory of ``filename`` first, then each of ``include_dirs``, then the
        current working directory.  An entry containing ``*``, ``?`` or ``[`` is
        an :mod:`fnmatch` pattern whose directory part is resolved the same way.
        Matching is on whole absolute, symlink-resolved paths -- never on
        substrings, so ``er.h`` does not match ``other.h``.
    :param denylist: Files whose declarations are dropped, using the same
        resolution and glob rules as ``allowlist``.  Deny wins over allow: a
        file named by both lists is excluded.  A denylist with no allowlist
        means "everything except these".  An entry matching nothing is not an
        error.  ``filename`` itself cannot be denied.
    :returns: :class:`~headerkit.ir.Header` containing parsed declarations.
    :raises RuntimeError: If parsing fails with errors.

    Example
    -------
    ::

        # Basic usage
        header = backend.parse(
            code,
            "myheader.hpp",
            include_dirs=["/usr/local/include"],
            extra_args=["-std=c++17", "-DNDEBUG"]
        )

        # Umbrella header (all-includes) pattern: descend into the includes
        # and keep everything they declare.  This is the default.
        header = backend.parse(
            code,
            "LibraryAll.h",
            include_dirs=["./include"],
            recursive_includes=True
        )

        # Keep only one included header's declarations: no descent, and
        # exactly one file admitted from the main translation unit.
        header = backend.parse(
            code,
            "main.h",
            recursive_includes=False,
            allowlist=["other.h"]
        )

        # Main file only -- nothing reached through #include survives
        header = backend.parse(
            code,
            "main.h",
            recursive_includes=False
        )

        # Umbrella header in system location
        header = backend.parse(
            code,
            "sodium.h",
            include_dirs=["/opt/homebrew/include"],
            project_prefixes=("/opt/homebrew/include/sodium",)  # Allowlist sodium/*
        )
    """
    # Ensure libclang is configured before parsing.  This is a no-op
    # after the first successful load (short-circuits on Config.loaded).
    if not _configure_libclang():
        from headerkit.backends import LibclangUnavailableError

        raise LibclangUnavailableError(
            "libclang shared library not found. Install LLVM/clang or run: pip install libclang"
        )

    args: list[str] = []

    is_cplus = _detect_cplus(filename, extra_args)

    # Add user-specified include directories FIRST
    # This is important for C++ where user headers may need to come before system libc++
    if include_dirs:
        for inc_dir in include_dirs:
            args.append(f"-I{inc_dir}")

    # Add system include directories if enabled
    # Always add them when use_default_includes=True, regardless of other -I flags
    if use_default_includes:
        args.extend(get_system_include_dirs(cplus=is_cplus))
        has_stddef = any(
            os.path.isfile(os.path.join(p.removeprefix("-isystem").removeprefix("-I"), "stddef.h")) for p in args
        )
        if not has_stddef:
            for builtin_inc in _find_libclang_builtin_include_dirs():
                args.append(f"-isystem{builtin_inc}")

    # Add extra arguments
    if extra_args:
        args.extend(extra_args)

    # Auto-resolve Nim's standard library directory when parsing a header that includes "nimbase.h"
    uses_nimbase = "nimbase.h" in code
    nim_libpath_added = False
    if uses_nimbase:
        if "NIM_INTBITS" not in code and not any("NIM_INTBITS" in a for a in args):
            args.append(f"-DNIM_INTBITS={struct.calcsize('P') * 8}")
        if not _has_nimbase_in_dirs(include_dirs, extra_args, filename):
            nim_libpath = _find_nim_libpath()
            if nim_libpath is not None:
                args.append(f"-isystem{nim_libpath}")
                nim_libpath_added = True

    # Parse the code with detailed preprocessing record for macro extraction
    index = self._get_index()
    tu = index.parse(
        filename,
        args=args,
        unsaved_files=[(filename, code)],
        options=_cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD,
    )

    # If an included header (e.g. via umbrella #include) needed nimbase.h, retry with Nim libpath
    if not nim_libpath_added and any(
        diag.severity >= _cindex.Diagnostic.Error and "nimbase.h" in str(diag.spelling) for diag in tu.diagnostics
    ):
        uses_nimbase = True
        if "NIM_INTBITS" not in code and not any("NIM_INTBITS" in a for a in args):
            args.append(f"-DNIM_INTBITS={struct.calcsize('P') * 8}")
        if not _has_nimbase_in_dirs(include_dirs, extra_args, filename):
            nim_libpath = _find_nim_libpath()
            if nim_libpath is not None:
                args.append(f"-isystem{nim_libpath}")
                nim_libpath_added = True
                tu = index.parse(
                    filename,
                    args=args,
                    unsaved_files=[(filename, code)],
                    options=_cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD,
                )

    # Debug Nim --header output emits inline nimFrame() bodies referencing undeclared
    # framePtr__system_u*; retry skipping function bodies if a Nim header fails body check
    if uses_nimbase and any(diag.severity >= _cindex.Diagnostic.Error for diag in tu.diagnostics):
        tu = index.parse(
            filename,
            args=args,
            unsaved_files=[(filename, code)],
            options=(
                _cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD
                | _cindex.TranslationUnit.PARSE_SKIP_FUNCTION_BODIES
            ),
        )

    # Check for fatal errors
    for diag in tu.diagnostics:
        if diag.severity >= _cindex.Diagnostic.Error:
            raise RuntimeError(f"Parse error: {diag.spelling}")

    # Collect included headers
    included_headers: set[str] = set()
    for inclusion in tu.get_includes():
        # inclusion.include is a File with name attribute
        header_path = str(inclusion.include.name)
        # Store full path - caller can extract basename if needed
        included_headers.add(header_path)
        # Ask clang whether it treated this file as a system header. This is the
        # authoritative answer -- clang knows its own resource directory and
        # -isystem search paths, which vary per platform and per toolchain
        # version, so the path heuristic alone leaks libc internals on Linux.
        if _location_is_in_system_header(tu, inclusion.include):
            self._clang_system_headers.add(normalize_path(header_path))

    # Convert to IR
    search_dirs = _filter_search_dirs(filename, include_dirs)
    allowlist_paths = None if allowlist is None else _PathSet(allowlist, search_dirs)
    denylist_paths = None if denylist is None else _PathSet(denylist, search_dirs)
    converter = ClangASTConverter(
        filename,
        project_prefixes=project_prefixes,
        is_cplus=is_cplus,
        allowlist_paths=allowlist_paths,
        denylist_paths=denylist_paths,
        parse_args=tuple(args),
    )
    header = converter.convert(tu)

    # Attach included headers to the IR
    header.included_headers = included_headers

    # Check if we should do recursive include processing
    # ``recursive_includes`` is the caller's answer, so nothing here second-guesses
    # it from the header's shape. Scope is bounded by _is_system_header /
    # project_prefixes, allowlist, max_depth, and self._visited.
    has_project_include = any(not self._is_system_include(h, project_prefixes) for h in header.included_headers)
    if recursive_includes and has_project_include:
        # Reset visited set for each top-level parse
        self._visited = set()
        # Add current file to visited
        if os.path.exists(filename):
            abs_filename = os.path.abspath(filename)
        else:
            # For in-memory code, use filename as-is
            abs_filename = filename
        self._visited.add(abs_filename)

        # Recursively parse included headers
        header = self._parse_recursively(
            header,
            abs_filename,
            include_dirs or [],
            extra_args or [],
            use_default_includes,
            max_depth,
            project_prefixes=project_prefixes,
            allowlist_paths=allowlist_paths,
            denylist_paths=denylist_paths,
        )

    # Included headers are parsed in isolation, so a macro that a *later*
    # header redefines is otherwise emitted with the stale value it had in
    # the header that first defined it. The main translation unit's
    # preprocessing record is the only place the real, ordered macro history
    # is visible, so it decides the final state.
    self._apply_final_macro_state(header, tu, converter)
    self._drop_undefined_macros(header, filename, code, args)

    # Record the language clang actually parsed this unit as. Writers need it:
    # a `std::string` field and a C `typedef struct {...} string;` reach the IR
    # as the identical `CType(name="string")`, and nothing else in the IR tells
    # them apart. `_detect_cplus` is already the authority that chose the parse
    # mode, so this records that decision rather than re-deriving one.
    header.language = "cpp" if is_cplus else "c"

    return header

tree-sitter Backend

Registered by the headerkit[treesitter] extra (installed via pip install "autopxd2[treesitter]"). Select it with get_backend("tree-sitter") or autopxd --backend tree-sitter. See Parser Backends for its limitations.