Commonsense-Guided Open-World Object Detection Using LLMs and Visual-Semantic Matching

Traditional object detectors such as YOLO and Faster R-CNN are limited to fixed category labels and struggle with novel or unseen objects. Open-vocabulary models like CLIP offer greater flexibility but often misinterpret user intent due to a lack of commonsense reasoning. To address these limitations, we propose a commonsense-guided open-world object detection framework that integrates YOLOv8 for fast region proposals, CLIP for visual-text alignment, LLaVA for scene understanding, and GPT-4 for trait-based reasoning. By pre-generating over 100 object descriptions with GPT-4, our system embeds functional and contextual knowledge that enables intent-aware detection beyond static labels. Experiments on COCO, Open Images, and a custom dataset of unseen objects demonstrate that our approach significantly improves recall on novel queries while maintaining high precision. These results highlight the importance of combining vision-language models with commonsense reasoning for open-world detection. For implementation details, https://github.com/ibrohimgets/CommonsenseVision.git