analiza aiqbench
This research introduces PHAGE, a method that integrates claim-level dependency hierarchies into transformer attention mechanisms to improve patent classification and retrieval. By distinguishing between reliable legal citations and noisy technical relations, the model enhances representation learning without requiring graph structures at inference time. The approach demonstrates significant performance gains over existing baselines in handling complex patent documents where structural relationships are critical.
Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning
arXiv:2605.10073v3 Announce Type: replace Abstract: Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.
Abstract:Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.
Computation and Language (cs.CL)
arXiv:2605.10073 [cs.CL]
(or arXiv:2605.10073v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.10073 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 275 cuvinte
Subiecte conexe
HiGFRL: Hierarchical Graph Fusion-Driven Reinforcement Learning for Dependency-Aware Task Scheduling in Heterogeneous Cloud
arXiv · 15 sept. 2026
STHMoE: Hypergraph-Enhanced Heterogeneous Dependency Coordination for LLM-Based Urban Traffic Data Forecasting
arXiv · 16 sept. 2026
MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion
arXiv · 20 iul. 2026
Rethinking Floorplan Generation with Space Syntax: From Design Theory to Model Improvement
arXiv · 14 oct. 2026