Preliminary Exploration of the Use of Geographical Information for Content-based Geo-tagging of Social Video

08-04-2023

Challenge the future

DelftUniversity ofTechnology

Preliminary Exploration of the Use of Geographical Information for Content-based Geo-tagging of Social VideoXinchao Li, Claudia Hauff, Martha Larson, Alan Hanjalic

2Visual similarity measures for semantic video retrieval

System Overview

• Goal

derive location information from the visual content of videos

• Challenge

• no tags: 35.7%, only one tag: 13.1%

• improve metadata-based system

System Overview

3Visual similarity measures for semantic video retrieval

Great Victoria Desert

South Pole

System Overview

• Assumption

divide the world map into regions that have a high within-region visual stability and a high between-region variability

4Visual similarity measures for semantic video retrieval

Different Division Methods

• Baseline

Different Division Methods

5Visual similarity measures for semantic video retrieval

• Temperature Data based

Different Division Methods

6Visual similarity measures for semantic video retrieval

• Temperature Data based

Different Division Methods

6 temperature regions: from -20◦C to 40◦C with 10◦C intervals.

7Visual similarity measures for semantic video retrieval

• Biomes Data based

Different Division Methods

8Visual similarity measures for semantic video retrieval

Run Results

Run Results

9Visual similarity measures for semantic video retrieval

Run Results

Run Results

22 Biomes classification: 12.17% (random, 4.55%)

10Visual similarity measures for semantic video retrieval

Discussion

• Visual Content of Test Videos

• Indoor (42%)

• Outdoor Event (32%)

• Normal Outdoor (26%)

• Visual Content of Training Photos

458 photos from the 3M training set

• Indoor (27.5%)

Discussion

500 videos from the 4182 videos (12%)

11Visual similarity measures for semantic video retrieval

Discussion

Indoor (42%)

12Visual similarity measures for semantic video retrieval

Discussion

Outdoor Event (32%)

13Visual similarity measures for semantic video retrieval

Discussion

Normal (26%)

14Visual similarity measures for semantic video retrieval

• Recall our assumption

“we can divide the world map into regions

that have a high within-region visual stability and a

high between-region variability.”

• indoor images are noisy information

• Only use outdoor videos to train and test

Discussion

Conclusion and Future work

15Visual similarity measures for semantic video retrieval

Thank you!

X.Li-3@tudelft.nl