Neuro-Symbolic Entropy Regularization

Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck, in UAI, 2022.

Download the full text

Abstract

In structured prediction, the goal is to jointly predict many output variables that together encode a structured object – a path in a graph, an entity-relation triple, or an ordering of objects. Such a large output space makes learning hard and requires vast amounts of labeled data. Different approaches leverage alternate sources of supervision. One approach – entropy regularization – posits that decision boundaries should lie in low-probability regions. It extracts supervision from unlabeled examples, but remains agnostic to the structure of the output space. Conversely, neuro-symbolic approaches exploit the knowledge that not every prediction corresponds to a valid structure in the output space. Yet, they does not further restrict the learned output distribution. This paper introduces a framework that unifies both approaches. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object. It is obtained by restricting entropy regularization to the distribution over only valid structures. This loss is efficiently computed when the output constraint is expressed as a tractable logic circuit. Moreover, it seamlessly integrates with other neuro-symbolic losses that eliminate invalid predictions. We demonstrate the efficacy of our approach on a series of semi-supervised and fully-supervised structured-prediction experiments, where we find that it leads to models whose predictions are more accurate and more likely to be valid.

Bib Entry

@inproceedings{ahmadneuro2022,
  title = {Neuro-Symbolic Entropy Regularization},
  author = {Ahmed, Kareem and Wang, Eric and Chang, Kai-Wei and den Broeck, Guy Van},
  booktitle = {UAI},
  year = {2022}
}

Related Publications

AVIS: Autonomous Visual Information Seeking with Large Language Models, NeurIPS, 2023
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models, NeurIPS, 2023
A Survey of Deep Learning for Mathematical Reasoning, ACL, 2023
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step, ACL, 2023
On the Paradox of Learning to Reason from Data, IJCAI, 2023
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning, ICLR, 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, NeurIPS, 2022
Semantic Probabilistic Layers for Neuro-Symbolic Learning, NeurIPS, 2022