Publications

Publications are listed below. For more details, please visit my Google Scholar page.

2025

1. SIGIR 2025 LiveRAG

   

   RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge - GRAG: Generation-Retrieval-Augmented Generation

   Kun Ran, Shuoqi Sun, Khoi Nguyen Dinh Anh, Damiano Spina, and Oleg Zendel

   2025

   Abs arXiv Bib PDF Code

   This paper presents the RMIT–ADM+S participation in the SIGIR 2025 LiveRAG Challenge. Our Generation-Retrieval-Augmented Generation (GRAG) approach relies on generating a hypothetical answer that is used in the retrieval phase, alongside the original question. GRAG also incorporates a pointwise large language model (LLM)-based re-ranking step prior to final answer generation. We describe the system architecture and the rationale behind our design choices. In particular, a systematic evaluation using the Grid of Points (GoP) framework and N-way ANOVA enabled comparison across multiple configurations, including query variant generation, question decomposition, rank fusion strategies, and prompting techniques for answer generation. Our system achieved a Relevance score of 1.199 and a Faithfulness score of 0.477 on the private leaderboard, placing among the top four finalists in the LiveRAG 2025 Challenge.

   @misc{ran2025rmitadmssigir2025liverag,
     title = {RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge - GRAG: Generation-Retrieval-Augmented Generation},
     author = {Ran, Kun and Sun, Shuoqi and Anh, Khoi Nguyen Dinh and Spina, Damiano and Zendel, Oleg},
     year = {2025},
     eprint = {2506.14516},
     archiveprefix = {arXiv},
     primaryclass = {cs.IR},
     url = {https://arxiv.org/abs/2506.14516},
   }

2. ECIR 2025

   

   An Investigation of Prompt Variations for Zero-shot LLM-based Rankers

   Shuoqi Sun, Shengyao Zhuang, Shuai Wang, and Guido Zuccon

   In Advances in Information Retrieval, 2025

   Abs DOI arXiv Bib PDF Code

   We provide a systematic understanding of the impact of specific components and wordings used in prompts on the effectiveness of rankers based on zero-shot Large Language Models (LLMs). Several zero-shot ranking methods based on LLMs have recently been proposed. Among many aspects, methods differ across (1) the ranking algorithm they implement, e.g., pointwise vs. listwise, (2) the backbone LLMs used, e.g., GPT3.5 vs. FLAN-T5, (3) the components and wording used in prompts, e.g., the use or not of role-definition (role-playing) and the actual words used to express this. It is currently unclear whether performance differences are due to the underlying ranking algorithm, or because of spurious factors such as better choice of words used in prompts. This confusion risks to undermine future research. Through our large-scale experimentation and analysis, we find that ranking algorithms do contribute to differences between methods for zero-shot LLM ranking. However, so do the LLM backbones – but even more importantly, the choice of prompt components and wordings affect the ranking. In fact, in our experiments, we find that, at times, these latter elements have more impact on the ranker’s effectiveness than the actual ranking algorithms, and that differences among ranking methods become more blurred when prompt variations are considered.

   @inproceedings{sun2025investigation,
     title = {An Investigation of Prompt Variations for Zero-shot LLM-based Rankers},
     author = {Sun, Shuoqi and Zhuang, Shengyao and Wang, Shuai and Zuccon, Guido},
     booktitle = {Advances in Information Retrieval},
     year = {2025},
     publisher = {Springer Nature Switzerland},
     address = {Cham},
     pages = {185--201},
     isbn = {978-3-031-88711-6},
     doi = {10.1007/978-3-031-88711-6_12},
     url = {https://doi.org/10.1007/978-3-031-88711-6_12},
   }