Currently, there is a large number of online photographic content that is publicly available in photo sharing and social networking applications. Despite the fact that such content is generated by simple users, it is often of substantial quality and covers aspects of the real world (e.g. interesting spots and events) that would otherwise remain unknown to the public.
However, due to the massive amount of available user generated content and due to the lack of structure in this content, it is impractical to explore such collections. To this end, our R&D result, called ClustTour, facilitates the exploration of large photo collections by clustering the photos of a city into groups that depict the same real-world object/event, by classifying the derived clusters into landmarks or events, and by projecting them on a map by use of the photos geotagging information.
ClustTour makes use of a novel graph-based clustering scheme in order to cluster a large set of photos into meaningful groups corresponding to objects and events. The used clustering scheme addresses two profound challenges of Social media content:
- Massive amounts of content: the employed clustering scheme is very efficient and scales almost linearly to the number of photos to be clustered
- Noise: due to the unrestricted nature of content contributed by users, a large number of photos are irrelevant to any points of interest and events of a place. The graph-based clustering scheme is only extracting clusters of photos that are similar to each other and leaves out outliers that are similar to just one or two members of a cluster.
In addition, ClustTour makes use of aggregated features of the extracted photo clusters in order to classify them into landmarks and events. These features pertain to the temporal characteristics of the cluster, the number of users who contributed photos to the cluster and the tags used to describe the cluster.
Finally, the framework contains a module for automatic labeling of the landmark clusters by processing the titles and tags of the individual photos contained in the clusters.
Applications (existing / potential)
The results of the described analysis are currently exploited in a web application that enables users to interactively explore the main Points of Interest and Events of a city. The web application is called ClustTour (available online: http://www.clusttour.gr) and currently supports more than 20 European cities (e.g. London, Paris, Berlin, Barcelona, Prague, Athens, etc.). The extracted image clusters are presented as markers on the map of a city and, upon click, the user gets a summarized view of them. It is possible to zoom in a particular cluster of photos and then get recommendations on the nearby clusters. In addition, an Augmented Reality layer has been developed (based on the AR platform Layar, http://www.layar.com) that permits users of smart phones (iphone, Android) to explore Points of Interest and Events in their vicinity (by use of the GPS location of the mobile phone).