Xiaoyan Wang 王笑颜

We present GenJAX, a new language and compiler for vectorized programmable probabilistic inference. GenJAX integrates the vectorizing map (vmap) operation from array programming frameworks such as JAX into the programmable inference paradigm, enabling compositional vectorization of features such as probabilistic program traces, stochastic branching (for expressing mixture models), and programmable inference interfaces for writing custom probabilistic inference algorithms. We formalize vectorization as a source-to-source program transformation on a core calculus for probabilistic programming (λGEN), and prove that it correctly vectorizes both modeling and inference operations. We have implemented our approach in the GenJAX language and compiler, and have empirically evaluated this implementation on several benchmarks and case studies. Our results show that our implementation supports a wide and expressive set of programmable inference patterns and delivers performance comparable to hand-optimized JAX code.

@article{becker2026vectorized,
  title = {{Probabilistic Programming with Vectorized Programmable Inference}},
  author = {Becker, McCoy R. and Huot, Mathieu and Matheos, George and Wang, Xiaoyan and Chung, Karen and Smith, Colin and Ritchie, Sam and Saurous, Rif A. and Lew, Alexander K. and Rinard, Martin C. and Mansinghka, Vikash K.},
  journal = {Proceedings of the ACM on Programming Languages},
  volume = {10},
  number = {POPL},
  pages = {2523--2554},
  year = {2026},
  doi = {10.1145/3776729},
}

In this paper we introduce Fairy a minimalist yet robust adaptation of image-editing diffusion models enhancing them for video editing applications. Our approach centers on the concept of anchor-based cross-frame attention a mechanism that implicitly propagates diffusion features across frames ensuring superior temporal coherence and high-fidelity synthesis. Fairy not only addresses limitations of previous models including memory and processing speed. It also improves temporal consistency through a unique data augmentation strategy. This strategy renders the model equivariant to affine transformations in both source and target images. Remarkably efficient Fairy generates 120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds outpacing prior works by at least 44x. A comprehensive user study involving 1000 generated samples confirms that our approach delivers superior quality decisively outperforming established methods.

@inproceedings{Wu_2024_CVPR,
  author = {Wu, Bichen and Chuang, Ching-Yao and Wang, Xiaoyan and Jia, Yichen and Krishnakumar, Kapil and Xiao, Tong and Liang, Feng and Yu, Licheng and Vajda, Peter},
  title = {{Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis}},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2024},
  pages = {8261--8270},
  url = {https://openaccess.thecvf.com/content/CVPR2024/html/Wu_Fairy_Fast_Parallelized_Instruction-Guided_Video-to-Video_Synthesis_CVPR_2024_paper.html},
}

Recent advances in diffusion models have successfully enabled text-guided image inpainting. While it seems straightforward to extend such editing capability into the video domain there have been fewer works regarding text-guided video inpainting. Given a video a masked region at its initial frame and an editing prompt it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact. There are three main challenges in text-guided video inpainting: (i) temporal consistency of the edited video (ii) supporting different inpainting types at different structural fidelity levels and (iii) dealing with variable video length. To address these challenges we introduce Any-Length Video Inpainting with Diffusion Model dubbed as AVID. At its core our model is equipped with effective motion modules and adjustable structure guidance for fixed-length video inpainting. Building on top of that we propose a novel Temporal MultiDiffusion sampling pipeline with a middle-frame attention guidance mechanism facilitating the generation of videos with any desired duration. Our comprehensive experiments show our model can robustly deal with various inpainting types at different video duration ranges with high quality.

@inproceedings{Zhang_2024_CVPR,
  author = {Zhang, Zhixing and Wu, Bichen and Wang, Xiaoyan and Luo, Yaqiao and Zhang, Luxin and Zhao, Yinan and Vajda, Peter and Metaxas, Dimitris and Yu, Licheng},
  title = {{AVID: Any-Length Video Inpainting with Diffusion Model}},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2024},
  pages = {7162--7172},
  url = {https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_AVID_Any-Length_Video_Inpainting_with_Diffusion_Model_CVPR_2024_paper.html},
}

Compared to the wide array of advanced Monte Carlo methods supported by modern probabilistic programming languages (PPLs), PPL support for variational inference (VI) is less developed: users are typically limited to a predefined selection of variational objectives and gradient estimators, which are implemented monolithically (and without formal correctness arguments) in PPL backends. In this paper, we propose a more modular approach to supporting variational inference in PPLs, based on compositional program transformation. In our approach, variational objectives are expressed as programs, that may employ first-class constructs for computing densities of and expected values under user-defined models and variational families. We then transform these programs systematically into unbiased gradient estimators for optimizing the objectives they define. Our design enables modular reasoning about many interacting concerns, including automatic differentiation, density accumulation, tracing, and the application of unbiased gradient estimation strategies. Additionally, relative to existing support for VI in PPLs, our design increases expressiveness along three axes: (1) it supports an open-ended set of user-defined variational objectives, rather than a fixed menu of options; (2) it supports a combinatorial space of gradient estimation strategies, many not automated by today’s PPLs; and (3) it supports a broader class of models and variational families, because it supports constructs for approximate marginalization and normalization (previously introduced only for Monte Carlo inference). We implement our approach in an extension to the Gen probabilistic programming system (genjax.vi, implemented in JAX), and evaluate our automation on several deep generative modeling tasks, showing minimal performance overhead vs. hand-coded implementations and performance competitive with well-established open-source PPLs.

@article{becker2024variational,
  title = {{Probabilistic Programming with Programmable Variational Inference}},
  author = {Becker, McCoy R. and Lew, Alexander K. and Wang, Xiaoyan and Ghavami, Matin and Huot, Mathieu and Rinard, Martin C. and Mansinghka, Vikash K.},
  journal = {Proceedings of the ACM on Programming Languages},
  volume = {8},
  number = {PLDI},
  pages = {2123--2147},
  year = {2024},
  doi = {10.1145/3656463},
}

We introduce PPL Bench, a new benchmark for evaluating Probabilistic Programming Languages (PPLs) on a variety of statistical models. The benchmark includes data generation and evaluation code for a number of models as well as implementations in some common PPLs. All of the benchmark code and PPL implementations are available on Github. We welcome contributions of new models and PPLs and as well as improvements in existing PPL implementations. The purpose of the benchmark is two-fold. First, we want researchers as well as conference reviewers to be able to evaluate improvements in PPLs in a standardized setting. Second, we want end users to be able to pick the PPL that is most suited for their modeling application. In particular, we are interested in evaluating the accuracy and speed of convergence of the inferred posterior. Each PPL only needs to provide posterior samples given a model and observation data. The framework automatically computes and plots growth in predictive log-likelihood on held out data in addition to reporting other common metrics such as effective sample size and r_hat

@inproceedings{kulkarni2020ppl,
  title = {{PPL Bench: Evaluation Framework for Probabilistic Programming Languages}},
  author = {Kulkarni, Sourabh and Shah, Kinjal Divesh and Arora, Nimar and Wang, Xiaoyan and Li, Yucen Lily and Tehrani, Nazanin Khosravani and Tingley, Michael and Noursi, David and Torabi, Narjes and Masouleh, Sepehr Akhavan and Lippert, Eric and Meijer, Erik},
  booktitle = {International Conference on Probabilistic Programming (PROBPROG)},
  year = {2020},
  url = {https://arxiv.org/abs/2010.08886},
  eprint = {2010.08886},
  archiveprefix = {arXiv},
}

There has been a shift of attention in the AI research where people gradually abandon traditional statistical models in favor of deep neural architectures. While effective in learning input-output mappings from two arbitrary distributions, the complex nature of neural models makes them hard to interpret. In this thesis, we introduce a more interpretable hierarchical bigram (HiBi) model, which is extended based on the simple bigram language model. It contains a few components inspired by theories of human cognition, and has been shown through experiments to be effective in learning meaningful representation from sequential inputs without any labeling. We hope that HiBi could be a starting point to develop more complex cognitive models that are both interpretable and effective for representation learning.

@mastersthesis{wang2020hibi,
  title = {{HiBi: A Hierarchical Bigram Model for Associative Learning}},
  author = {Wang, Xiaoyan},
  school = {University of Illinois at Urbana-Champaign},
  year = {2020},
  url = {https://www.ideals.illinois.edu/items/115631},
}

Natural Language Inference (NLI) is fundamental to many Natural Language Processing (NLP) applications including semantic search and question answering. The NLI problem has gained significant attention due to the release of large scale, challenging datasets. Present approaches to the problem largely focus on learning-based methods that use only textual information in order to classify whether a given premise entails, contradicts, or is neutral with respect to a given hypothesis. Surprisingly, the use of methods based on structured knowledge – a central topic in artificial intelligence – has not received much attention vis-a-vis the NLI problem. While there are many open knowledge bases that contain various types of reasoning information, their use for NLI has not been well explored. To address this, we present a combination of techniques that harness external knowledge to improve performance on the NLI problem in the science questions domain. We present the results of applying our techniques on text, graph, and text-and-graph based models; and discuss the implications of using external knowledge to solve the NLI problem. Our model achieves close to state-of-the-art performance for NLI on the SciTail science questions dataset.

@inproceedings{wang2019improving,
  title = {{Improving Natural Language Inference Using External Knowledge in the Science Questions Domain}},
  author = {Wang, Xiaoyan and Kapanipathi, Pavan and Musa, Ryan and Yu, Mo and Talamadupula, Kartik and Abdelaziz, Ibrahim and Chang, Maria and Fokoue, Achille and Makni, Bassem and Mattei, Nicholas and Witbrock, Michael},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume = {33},
  number = {01},
  pages = {7208--7215},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33017208},
}

Open-domain question answering (QA) is an important problem in AI and NLP that is emerging as a bellwether for progress on the generalizability of AI methods and techniques. Much of the progress in open-domain QA systems has been realized through advances in information retrieval (IR) methods and corpus construction. In this paper, we focus on the recently introduced ARC Challenge dataset, which contains 2,590 multiple choice questions authored for grade-school science exams. These questions are selected to be the most challenging for current QA systems, and current state of the art performance is only slightly better than random chance. We present a system that reformulates a given question into queries that are used to retrieve supporting text from a large corpus of science-related text. Our rewriter is able to incorporate background knowledge from ConceptNet. In tandem with a generic textual entailment system trained on SciTail that identifies support in the retrieved results, our system outperforms several strong baselines on the end-to-end QA task despite only being trained to identify essential terms in the original source question. We use a generalizable decision methodology – over the retrieved evidence and answer candidates – to select the best answer. By combining query reformulation, background knowledge, and textual entailment, our system is able to outperform several strong baselines on the ARC dataset.

@inproceedings{musa2019answering,
  title = {{Answering Science Exam Questions Using Query Reformulation with Background Knowledge}},
  author = {Musa, Ryan and Wang, Xiaoyan and Fokoue, Achille and Mattei, Nicholas and Chang, Maria and Kapanipathi, Pavan and Makni, Bassem and Talamadupula, Kartik and Witbrock, Michael},
  booktitle = {Automated Knowledge Base Construction (AKBC)},
  year = {2019},
  url = {https://openreview.net/forum?id=HJxYZ-5paX},
}

We propose a framework for constructing and analyzing multiclass and multioutput classification metrics, i.e., involving multiple, possibly correlated multiclass labels. Our analysis reveals novel insights on the geometry of feasible confusion tensors – including necessary and sufficient conditions for the equivalence between optimizing an arbitrary non-decomposable metric and learning a weighted classifier. Further, we analyze averaging methodologies commonly used to compute multioutput metrics and characterize the corresponding Bayes optimal classifiers. We show that the plug-in estimator based on this characterization is consistent and is easily implemented as a post-processing rule. Empirical results on synthetic and benchmark datasets support the theoretical findings.

@article{wang2019consistent,
  title = {{Consistent Classification with Generalized Metrics}},
  author = {Wang, Xiaoyan and Li, Ran and Yan, Bowei and Koyejo, Oluwasanmi},
  journal = {arXiv preprint},
  year = {2019},
  url = {https://arxiv.org/abs/1908.09057},
  eprint = {1908.09057},
  archiveprefix = {arXiv},
}

Answering complex questions involving multiple entities and relations is a challenging task. Logically, the answer to a complex question should be derived by decomposing the complex question into multiple simple sub-questions and then answering those sub-questions. Existing work has followed this strategy but has not attempted to optimize the order of how those sub-questions are answered. As a result, the sub-questions are answered in an arbitrary order, leading to larger search space and a higher risk of missing an answer. In this paper, we propose a novel reinforcement learning (RL) approach to answering complex questions that can learn a policy to dynamically decide which sub-question should be answered at each stage of reasoning. We leverage the expected value-variance criterion to enable the learned policy to balance between the risk and utility of answering a sub-question. Experiment results show that the RL approach can substantially improve the optimality of ordering the sub-questions, leading to improved accuracy of question answering. The proposed method for learning to order sub-questions is general and can thus be potentially combined with many existing ideas for answering complex questions to enhance their performance.

@article{zhang2019learning,
  title = {{Learning to Order Sub-questions for Complex Question Answering}},
  author = {Zhang, Yunan and Cheng, Xiang and Zhang, Yufeng and Wang, Zihan and Fang, Zhengqi and Wang, Xiaoyan and Huang, Zhenya and Zhai, Chengxiang},
  journal = {arXiv preprint},
  year = {2019},
  url = {https://arxiv.org/abs/1911.04065},
  eprint = {1911.04065},
  archiveprefix = {arXiv},
}