Nguyễn Văn Lộc

Research

Twelve peer-reviewed papers on generative editing, multimodal retrieval and media authenticity, and the challenges they won.

Papers

Peer-reviewed research

12 papers · newest first

2026

2 papers
ICMEW ’26● Accepted

Zero-Shot Mass-Similar and Multi-Object Removal in Single Pass

Dinh-Khoi Vo, Van-Loc Nguyen, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

Clutter Erase, an interactive system that removes dense or mass-similar objects in a single pass, powered by PANDORA.

Abstract

Removing objects from natural images remains a significant challenge due to strict visual consistency requirements and limited user interaction flexibility. Existing systems typically rely on single-modality interactions, struggling to adapt to complex, multi-object scenarios. In this demonstration, we present Clutter Erase, an interactive system for seamless multi-object removal. The system is powered by PANDORA, our zero-shot, diffusion-based object removal model. Clutter Erase supports diverse interaction modalities, including manual masking, prompt-based selection, click-based segmentation, and similar object search, enabling highly flexible and intuitive editing. By eliminating the need for iterative passes, the system empowers users to efficiently navigate and edit complex scenarios, such as removing dense or mass-similar objects, while iteratively exploring multiple optimal results.

Fig. 1 · the Clutter Erase interface, from “Zero-Shot Mass-Similar and Multi-Object Removal in Single Pass”Fig. 1 · the Clutter Erase interface
ICME ’26● Accepted

PANDORA: Pixel-wise Attention Dissolution and Latent Guidance for Zero-Shot Object Removal

Dinh-Khoi Vo, Van-Loc Nguyen, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

Zero-shot object removal by dissolving attention and guiding latents inside a diffusion model.

Abstract

Removing objects from natural images is challenging due to difficulty of synthesizing semantically coherent content while preserving background integrity. Existing methods often rely on fine-tuning, prompt engineering, or inference-time optimization, yet still suffer from texture inconsistency, rigid artifacts, weak foreground–background disentanglement, and poor scalability for multi-object removal. We propose a novel zero-shot object removal framework, namely PANDORA, that operates directly on pre-trained text-to-image diffusion models, requiring no fine-tuning, prompts, or optimization. We propose Pixel-wise Attention Dissolution to remove object by nullifying the most correlated attention keys for masked pixels, effectively eliminating the object from self-attention flow and allowing background context to dominate reconstruction. We further introduce Localized Attentional Disentanglement Guidance to steer denoising toward latent manifolds favorable to clean object removal. Together, these components enable precise, non-rigid, prompt-free, and scalable multi-object erasure in a single pass. Experiments demonstrate superior visual fidelity and semantic plausibility compared to state-of-the-art methods.

Fig. 1 · single, multiple and mass-similar removal, from “PANDORA: Pixel-wise Attention Dissolution and Latent Guidance for Zero-Shot Object Removal”Fig. 1 · single, multiple and mass-similar removal

2025

4 papers
SOICT ’25● Accepted

Visionary: Optimized Temporal Video Retrieval via Large Language Model-Enhanced Query Processing

Van-Loc Nguyen*, Gia-Huy Vuong*, Ngoc-Do Tran, Tien-Thanh Nguyen-Dang, Van-Son Ho, Van-Tu Ninh, Minh-Triet Tran* equal contribution

LLM-enhanced queries and adaptive keyframes for large-scale video retrieval.

Abstract

The rapid growth of video content necessitates efficient, real-time event retrieval systems. Addressing the Ho Chi Minh City AI Challenge 2025, we present Visionary, a new generation of the NewsInsight systems. Our system introduces four key contributions: (1) a novel adaptive keyframe extraction algorithm; (2) an enhanced pre-processing pipeline using the Qwen3-VL model for metadata generation and integrated optical character recognition; (3) a flexible architecture supporting multiple embedding models; and (4) the use of Reciprocal Rank Fusion to synthesize retrieval results. These enhancements aim to substantially improve retrieval accuracy and overall performance for complex, large-scale video retrieval tasks.

Fig. 1 · system overview, from “Visionary: Optimized Temporal Video Retrieval via Large Language Model-Enhanced Query Processing”Fig. 1 · system overview
SOICT ’25● Accepted

EnAug: ENT Endoscopy Images Classification Using Ensemble and Augmentation Methods

Thanh-Son Nguyen*, Van-Loc Nguyen*, Cong-Luan Le*, Viet-Tham Huynh, Minh-Triet Tran* equal contribution

An ensemble with targeted augmentation for ENT endoscopy classification: 95.82% accuracy.

Top-1 · ENTRep 2025
Abstract

Ear, nose, and throat (ENT) endoscopy is a key diagnostic tool for detecting a variety of head and neck conditions. However, automated analysis of endoscopic images remains challenging due to inconsistent image quality, limited labeled data, and the inherent variability in human interpretation. In this work, we present a robust classification framework based on an ensemble of deep learning models, designed specifically for ENT endoscopy images. To address class imbalance and improve generalization, we employ a novel data augmentation strategy that combines symmetry-based label flipping with Mixup, Mosaic, and other augmentation techniques. Our approach is evaluated on a curated ENT dataset covering seven anatomical categories, achieving an accuracy of 95.82%, which surpasses several competitive baselines. In addition to strong overall performance, our method demonstrates improved robustness on underrepresented classes, showing its potential for real-world deployment in clinical settings. This work highlights the effectiveness of deep learning ensembles and thoughtful augmentation in building scalable AI tools for medical imaging.

Fig. 1 · ensemble classification pipeline, from “EnAug: ENT Endoscopy Images Classification Using Ensemble and Augmentation Methods”Fig. 1 · ensemble classification pipeline
ACM MM ’25● Published

EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions

Dinh-Khoi Vo, Van-Loc Nguyen, Minh-Triet Tran, Trung-Nghia Le

Event-aware multimodal retrieval that matches news images to realistic captions.

Top-1 · EVENTA 2025
Abstract

Event-based image retrieval from free-form captions presents a significant challenge: models must understand not only visual features but also latent event semantics, context, and real-world knowledge. Conventional vision-language retrieval approaches often fall short when captions describe abstract events, implicit causality, temporal context, or contain long, complex narratives. To tackle these issues, we introduce a multi-stage retrieval framework combining dense article retrieval, event-aware language model reranking, and efficient image collection, followed by caption-guided semantic matching and rank-aware selection. We leverage Qwen3 for article search, Qwen3-Reranker for contextual alignment, and Qwen2-VL for precise image scoring. To further enhance performance and robustness, we fuse outputs from multiple configurations using Reciprocal Rank Fusion (RRF). Our system achieves the top-1 score on the private test set of Track 2 in the EVENTA 2025 Grand Challenge, demonstrating the effectiveness of combining language-based reasoning and multimodal retrieval for complex, real-world image understanding.

MAPR ’25● Published

SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification

Dinh-Khoi Vo*, Van-Loc Nguyen*, Minh-Triet Tran, Trung-Nghia Le* equal contribution

CLIP matching re-ranked by silhouette shape to identify 3D objects.

Abstract

Retrieving 3D objects in complex indoor environments using only a masked 2D image and a natural language description presents significant challenges. The ROOMELSA challenge limits access to full 3D scene context, complicating reasoning about object appearance, geometry, and semantics. These challenges are intensified by distorted viewpoints, textureless masked regions, ambiguous language prompts, and noisy segmentation masks. To address this, we propose SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification. SAMURAI integrates CLIP-based semantic matching with shape-guided re-ranking derived from binary silhouettes of masked regions, alongside a robust majority voting strategy. A dedicated preprocessing pipeline enhances mask quality by extracting the largest connected component and removing background noise. Our hybrid retrieval framework leverages both language and shape cues, achieving competitive performance on the ROOMELSA private test set. These results highlight the importance of combining shape priors with language understanding for robust open-world 3D object retrieval.

Fig. 3 · five retrieval strategies compared, from “SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification”Fig. 3 · five retrieval strategies compared

2024

6 papers
SOICT ’24● Published

NewsInsight2.0: An Enhanced Version Integrating Large Language Model-based Query Optimization with Advanced Temporal Mechanisms

Gia-Huy Vuong, Van-Loc Nguyen, Van-Son Ho, Tien-Thanh Nguyen-Dang, Ngoc-Do Tran, Van-Tu Ninh, Minh-Triet Tran

CLIP on DFN-5B with LLM-generated queries for temporal news-video search.

3rd prize · AI Challenge 2024
Abstract

We introduce NewsInsight2.0, a cutting-edge evolution of the NewsInsight system, specifically designed for the Ho Chi Minh AI Challenge 2024. Building on the strengths of ViewsInsight, NewsInsight2.0 addresses previous performance limitations with significant enhancements. Our optimized architecture is tailored to deliver exceptional search capabilities for AIC 2024. At its core, NewsInsight2.0 leverages the CLIP (Contrastive Language-Image Pre-training) model, trained on a vast dataset of 5 billion parameters (DFN-5B). Additionally, we have refined our temporal query mechanism with a more efficient algorithm and an intuitive user interface. Furthermore, NewsInsight2.0 features an automatic query generator powered by open-source large language models, streamlining the process of optimizing user input queries.

Fig. 1 · system overview, from “NewsInsight2.0: An Enhanced Version Integrating Large Language Model-based Query Optimization with Advanced Temporal Mechanisms”Fig. 1 · system overview
ACCV-W ’24● Published

An Approach to Complex Visual Data Interpretation with Vision-Language Models

Thanh-Son Nguyen*, Viet-Tham Huynh*, Van-Loc Nguyen*, Minh-Triet Tran* equal contribution

Prompting and voting ensembles over vision-language models; top score 0.85 at LAVA 2024.

1st prize · LAVA 2024
Abstract

The LAVA Workshop 2024 challenge aimed to assess the capability of Large Vision-Language Models (VLMs) to interpret and understand complex visual data accurately. This includes intricate visual formats such as data flow diagrams, class diagrams, Gantt charts, and architectural blueprints. In response to this challenge, our research focuses on adapting the MMMU (Massive Multi-discipline Multimodal Understanding) benchmarks to better align with the requirements of visual data interpretation. We propose a comprehensive approach that leverages advanced prompt engineering techniques and incorporates a voting-based ensemble method for aggregating model predictions. This method improves the model’s ability to generalize across different types of visual inputs. Our approach was rigorously evaluated within the context of the challenge, resulting in a total score of 0.85, ultimately securing the top position in the competition. This result demonstrates the effectiveness of combining prompt engineering with simple yet powerful ensemble strategies for enhancing the performance of VLMs on complex multimodal tasks.

Fig. 2 · prompting and voting ensemble, from “An Approach to Complex Visual Data Interpretation with Vision-Language Models”Fig. 2 · prompting and voting ensemble
MAPR ’24● Published

AI-Enhanced Photo Authenticity: A User-Focused Approach to Detecting and Analyzing Manipulated Images

Minh-Triet Tran, Bao-Tin Nguyen, Van-Loc Nguyen, Thanh-Son Nguyen, Mai-Khiem Tran, Trong-Le Do

A user-facing system that helps people check whether a photo was manipulated.

Abstract

In the digital age, the authenticity of photographs is increasingly questioned due to the rise of sophisticated AI and human manipulation techniques. This paper presents the development of an AI-assisted system designed to aid users in identifying and verifying the genuineness of photos. Unlike fully automated solutions, our approach emphasizes user empowerment, providing tools to enhance human expertise in photo authentication. The system integrates advanced algorithms to detect potential manipulations and localize regions of interest, directing users’ attention to specific areas that warrant closer examination. Users can conduct more efficient and effective authenticity checks without scrutinizing the entire image by focusing on these targeted regions. Our goal is to augment, not replace, the critical role of human judgment in verifying photo authenticity. The system’s design and implementation are discussed, along with case studies demonstrating its practical application in various scenarios. This research aims to contribute to the growing field of digital forensics by providing a robust, user-friendly tool that bridges the gap between AI capabilities and human expertise in photo verification.

Fig. 1 · system overview, from “AI-Enhanced Photo Authenticity: A User-Focused Approach to Detecting and Analyzing Manipulated Images”Fig. 1 · system overview
AsiaCCS-W ’24● Published

A Hybrid Approach for Cheapfake Detection Using Reputation Checking and End-To-End Network

Bao-Tin Nguyen*, Van-Loc Nguyen*, Thanh-Son Nguyen*, Duc-Tien Dang-Nguyen, Trong-Le Do, Minh-Triet Tran* equal contribution

Source-reputation checks plus an end-to-end network to catch cheapfakes.

Abstract

In today’s era dominated by digital manipulations, the identification of Cheapfakes, especially those involving out-of-context visuals, is imperative for upholding the integrity of information dissemination and fostering trust in multimedia content. This research introduces an innovative approach to detect cheapfakes, utilizing a hybrid methodology that merges real-time online verification with extensive training on a unified end-to-end network, augmented by generative synthetic data. This combined strategy offers a promising avenue for enhancing the accuracy and efficiency of cheapfake detection, thus addressing the pressing need to combat misinformation in the digital landscape. We focus our solutions on two problems of Contextual Integrity Prediction and Image-Caption Authenticity Analysis. Our experiments reveal that this innovative network achieves an impressive 85.70% accuracy on a public test dataset through training solely on synthetic data for Contextual Integrity Prediction, also proving its effectiveness in Image-Caption Authenticity Analysis. This research highlights the immense promise of using our proposed hybrid method in the fight against Cheapfakes, making a significant contribution to preserving the integrity of multimedia content.

Fig. 6 · end-to-end network with reputation checking, from “A Hybrid Approach for Cheapfake Detection Using Reputation Checking and End-To-End Network”Fig. 6 · end-to-end network with reputation checking
ICMR ’24● Published

A Unified Network for Detecting Out-Of-Context Information Using Generative Synthetic Data

Van-Loc Nguyen*, Bao-Tin Nguyen*, Thanh-Son Nguyen*, Duc-Tien Dang-Nguyen, Minh-Triet Tran* equal contribution

Spotting out-of-context image–caption pairs with a network trained on synthetic data.

Abstract

In our modern world, the manipulation of digital content, especially the usage of out-of-context images known as Cheapfakes, has become a significant challenge for the endorsement of integrity and trustworthiness for information on the Internet. This highlights an urgent need for effective detection of the misuse of images and their accompanied captions. Motivated by this issue, our research presents an innovative approach to the solution of this problem. Participating in the ACM ICMR 2024 Grand Challenge on Detecting Cheapfakes, we leverage a unified end-to-end network, integrated with generative synthetic data for training. After complete evaluation, our proposed network demonstrated a remarkable accuracy of 95.60% on the public test dataset for Task 1, as well as efficiency in Task 2. This paper highlights the notable potential of employing an end-to-end network for Cheapfakes detection, which composes a significant contribution to the advancement of multimedia content integrity.

Fig. 1 · synthetic data and training pipeline, from “A Unified Network for Detecting Out-Of-Context Information Using Generative Synthetic Data”Fig. 1 · synthetic data and training pipeline
MMM ’24● Published

Transparent Tracking of Spermatozoa with YOLOv8

Bao-Tin Nguyen*, Van-Loc Nguyen*, Minh-Triet Tran* equal contribution

Transparent, reproducible detection and tracking of sperm cells with YOLOv8.

Abstract

Accurate spermatozoa tracking is crucial for understanding fertilization and reproductive health and developing novel sperm-based diagnostics and therapies. This paper explores the application of YOLOv8, a state-of-the-art object detection model, for automated and robust spermatozoa tracking in microscopic videos. Our approach employs transfer learning, fine-tuning the pre-trained YOLOv8 model on the VISEM-Tracking dataset of labeled spermatozoa images. We evaluate the performance of the proposed method on a specific part of the dataset, which does not participate in the training step. By providing a reliable and efficient tool for automated spermatozoa tracking, this work paves the way for further research, advancements, and applications in the field of reproductive medicine.

Awards

Challenges won

5 awards
Won top 1 at the ACM Multimedia 2025 Grand Challenge on Event-Enriched Image Analysis.
ACM MM 2025
Won top 1 in Track 1 of the ENTRep Challenge: Advancing Vision-Language AI for ENT Endoscopy Analysis at ACM Multimedia 2025.
ACM MM 2025
Won third prize at the 26th Student Scientific Research Award — Euréka, 2024.
Euréka · VN
Won third prize at the AI Challenge, Ho Chi Minh City 2024.
AI Chall. · VN
Won first prize at the LAVA Workshop Challenge, ACCV 2024.
ACCV 2024