Melika Ayoughi

Training machines that understand the rich semantic structure of the world requires not only learning from limited labelled data and structured prior knowledge, but also leveraging the vast underlying knowledge and relationships present in raw data itself. This thesis investigated how externally defined (chapters 2-4) and internally available (chapters 5-6) semantic structure can be leveraged to improve visual representation learning. Chapter 2 focused on external semantics, specifically hierarchies. It examined how the structural design of hierarchies affects the quality of hyperbolic embeddings. Through a systematic empirical study, it showed that hierarchy properties such as branching factor, depth, and inheritance patterns have a substantial impact on embedding distortion and reconstruction, establishing hierarchy structure as a key design variable rather than a fixed input. Chapter 3 built on these findings by introducing a prompt-based approach using large language models to automatically restructure hierarchies for improved hyperbolic embedding quality. The proposed method consistently improved embedding performance across diverse real-world hierarchies while providing explainable justifications for the structural changes. Chapter 4 extended hierarchical learning from static datasets to continual settings, where data arrives incrementally over time. It proposed a hyperbolic continual-learning framework that preserves both instance-level and class-level semantic structure, enabling hierarchical classification at multiple granularities. Chapter 5 shifted focus to internal semantics derived from visual data alone, proposing a self-supervised framework that captures visual composition through relative spatial relationships between image patches. The results demonstrated that enforcing relational structure enables models to learn more semantically meaningful and compositional representations without external supervision. Chapter 6 further extended internal semantics to multi-modal settings by leveraging video-caption pairs to discover visual entities. By aligning textual and visual cues, the proposed framework showed that multi-modal relational signals are sufficient to infer visual named entities without relying on curated datasets or task-specific labels. Together, this thesis shows that semantic structure—whether externally provided or internally discovered—can be systematically exploited to advance visual representation learning across different settings.