With the rapid growth of digital photography, sharing of photos with friends and family has become very popular. When people share their photos, they usually organize them into albums according to events or places. To tell the story of some important events in one’s life, it is desirable to have an efficient summarization tool which can help people to receive a quick overview of an album containing large number of photos. In this paper, we present and analyze an approach for photo album summarization through a novel social game “Epitome” as a Facebook application. This social game can collect research data, and, at the same time, it provides a collage or a cover photo of the user’s photo album, while the user enjoys playing the game. The proof of concept of the proposed method is demonstrated through a set of experiments on several photo albums. As a benchmark comparison to this game, we perform automatic visual analysis considering several state-of-the-art features. We also evaluate the usability of the game by making use of a questionnaire on several subjects who played the “Epitome” game. Furthermore, we address privacy issues concerning shared photos in Facebook applications.
Rapid growth of digital photography in recent years has increased the size of personal photo collections. People use their digital cameras or mobile phones equipped with cameras to take photos. Besides storing them on computer hard drives, they often share their digital photos with friends, family, and colleagues through social networks. Facebook (
Photos are often organized into albums (collections) based on places, events or dates, and people. Consumers tend to take several photos from one scene, hoping that one of them will be outstanding, and this leads to large number of similar photos. Therefore, it can be very time-consuming to go through all photos in one of these albums. Summarization is an effective way to provide a quick overview of a set of photos. In this paper, album summarization is defined as selecting a set of photos from a larger collection which best represents the visual information of the entire collection. Selected photos can be used to create a collage of a given album or a cover for an album or to be included in a photo book. However, as already mentioned, manual photo album summarization can be very time-consuming.
Which photos are the most suitable to summarize a photo album? Creation of a photo summary is a very subjective task. There are different criteria upon which a human user would rate digital photos. The color, composition, content, lighting, and sharpness of a photo, all contribute to viewer’s response to that photo (
Besides spending a lot of time sharing and consuming content in online social networks, people also use online applications, especially social games. Players pour huge amounts of time and efforts into games. For example, a recent survey [
In this paper, we present and evaluate an approach for photo album summarization through a novel social game “Epitome,” which was previously introduced in [
The paper is organized as follows. We introduce related work in Section
The proposed game is related to different research fields including visual analysis, automatic photo album summarization, and gaming. Therefore, the goal of this section is to review the most relevant work in these fields.
State-of-the-art techniques for automatic photo album summarization are based on time-separated events, spatial information using GPS coordinates, and content-based image similarities. Harada et al. [
Regarding content-based image similarity, various visual features have been used in automatic photo album summarization. Bag of Words (BoW) model is based on the histogram of local features [
Ames and Naaman [
Following the presented state-of-the-art techniques, a game-based approach for photo album summarization called “Epitome” was developed first in [
In this section two algorithms for photo album summarization are described. First, the proposed “Epitome” game is described which takes advantage of many casual gamers to solve the complex problem of album summarization. Then, an automatic visual algorithm is presented as a comparison benchmark to the task.
A social game, “Epitome,” provides an intuitive and enjoyable user interface as a Facebook application, as shown in Figure
Screenshot from the Epitome game.
The scenario of the game is as follows. A Facebook user, denoted as a player in this paper, installs the game in his/her Facebook applications page and allows access to his/her photo gallery, as shown in Figure
In order to perform summarization using players’ inputs, the application calculates three different values:
Finally, vectors
An example of selecting the three most representative photos within one Facebook album through the “Epitome” game.
Automatic photo album summarization is performed considering different visual and temporal features. After extracting these features, the album is segmented into five parts by calculating the four highest Euclidean distances of the consecutive photos features. For each image in a particular segment, we calculate the sum of the Euclidean distances between that feature of the photo and the rest of the image features in the segment. The image with the lowest sum is then selected as the most representative photo in that segment.
Different features can be used for segmentation and to select the most representative photo in the segments. We considered the following features: Bag of Words (BoW) method based on Speeded Up Robust Features (SURF), Histogram of Oriented Gradient (HOG), HSV (Hue, Saturation, Value) color histogram, “tiny” features, and time stamp, as described below.
Evaluation of the “Epitome” game can be performed in two ways: performance and usability of the game.
The performance of summarizing albums with the “Epitome” game is evaluated with respect to the ground truth given by humans.
The dataset of photos used for performance evaluation is the official dataset from “HP Challenge 2010: High Impact Visual Communication” at the “Multimedia Grand Challenge 2010” [
Some example photos for each of six albums. Photos in each row belong to the same album. The albums cover a large variety of objects and scenes usually taken during a vacation.
We first constructed a ground truth by asking different people to subjectively perform summarization and then tested our algorithm against the ground truth data. We recruited 63 participants, among whom
For simplicity of the explanation on how the designed photo selection tool (social game) was evaluated, let us consider only one album with
Then, the participants are asked to play our game with the selected dataset. The vectors
Furthermore, the performance of this game is compared to the performance of an automatic image selection which considers different visual and time features described in Section
Comparison between different visual and time features. The best performance is achieved with “color histogram” feature for both “Split it!” and “Select the best!” tasks. Dark red color indicates the best (
Figure
The distribution of the participants’ performance. The results of the “Epitome” game are shown with square markers and the results of automatic visual analysis with circle marker. Different colors of the markers correspond to different albums.
The comparison of normalized performance in summarizing photo albums performed by the “Epitome” game, automatic photo selection using color histogram, and users who participated in creating the ground truth data.
Photos from album 3. The most representative photos selected by the proposed method are marked with green bounding box, while the red bounding box denotes photos selected by making use of color histogram.
The usability of the “Epitome” game is evaluated through a user study. We asked participants (users) to play the game with different Facebook photo albums and to provide us with their feedback on the game in the form of a questionnaire.
We recruited 40 participants, aged 23–46 (average age was 28), with different cultural backgrounds. First, all participants were introduced to the “Epitome’’ game by showing them basic rules on how to play the game. Then, all participants spent sufficient time to play the game. After a participant played with the “Epitome’’ game with different Facebook photo albums, a questionnaire was used to obtain the feedback from the participant. The questionnaire consists of three groups of questions: general questions about motivation to play the game and enjoyment; questions to assess different platforms for playing the game (mobile, Facebook, or simple web page), for example, satisfaction with visual presentation for each of them; questions about privacy issues regarding showing one’s photos to his/her friends, friends of friends, and everybody or nobody.
In this study, we used discrete rating scales with adjective description of each level. Depending on the question, participants had to choose one of the answers or to rank answers according to their preferences. For each of the questions we calculated mean of the participants responses.
In this paper, we do not describe the whole questionnaire and results, but we rather discuss some of the interesting outcomes from our study. All questions are listed in the appendix. The questionnaire with choices is publicly available (
Questions (1)–(6), (9)–(14), and (21) listed in the appendix belong to questions about motivation to play the game. Results showed that
Average rank of different motivations to play the “Epitome” game. Lower average ranks are better.
Different permission pages used in our study: (a) default Facebook permission page, (b)
One of the questions was about preferred patterns of playing the game. Like other casual games, players would like to play our game several times a month, and around five minutes every time.
An important question we discuss here is about
We also addressed
In our experiments, users were asked whether they allow access to their data using the default Facebook permission page either on the mobile phone or in Facebook, and were separately asked if they would allow access to any of the three new permission pages. We measured how many players allow the “Epitome” game to access photos they have uploaded, photos they have been tagged in, and photos their friends have uploaded. Results are depicted in Figure
Acceptance rate for the default Facebook and permission pages used in our study.
The users do not have sufficient control over details about permission in Facebook applications. From the questions related to permission settings of shared photos (questions (15)–(20) listed in the appendix), we conclude that players would not like to give more permissions to the application compared to the permission they already set for their photos in Facebook. For example,
The “Epitome’’ game was published on Facebook in June 2011, and during two months, 49 users played it 5870 times on a dataset of 21780 photos. Distribution of players’ score is shown in Figure
The distribution of players’ score in the “Epitome” game. Scores are sorted in descending order.
The distribution of photos’ scores in the “Epitome” game, that is, the number of votes per appearance of each photo. Scores are sorted in descending order. The rest of the photos did not yet appeared or nobody voted for them.
The number of photos changed in collages over time since the “Epitome” game was launched.
In summary, the “Epitome” game has the following advantages. Performance of the game-based album summarization is better than using only computer vision approaches, which was shown in [ People like to watch their friends’ photos through this game, which also encourages social interaction between them. The game itself is interesting and people can have fun through the game.
However, a disadvantage is the processing time for generating fine album summarization, as shown in [
In this paper, we described and analyzed a social game, “Epitome,” for photo album summarization on Facebook. The game is a social application to enjoy photos of one’s Facebook friends, while contributing to summarization of their photo albums and collecting research data. The proof of concept of the game was demonstrated and validated through a set of experiments on several photo albums. The results of the experiments showed that the summarization game achieves
As a future study, we will make the game more attractive for users and also consider to include in this approach more sophisticated visual analysis. We also plan to improve the game by reducing the bandwidth which is necessary to load all images.
The usability of the “Epitome” game is evaluated by making use of a questionnaire (a user study) on several subjects who played the game. The questionnaire consists of three groups of questions: ( Are you satisfied with the Epitome game? If not completely satisfied, what is the main reason for that? Please rank the motivations to play the Epitome game according to your preferences in order to make it more enjoyable? Please rank the improvements of the Epitome game according to your preferences in order to make it more enjoyable? How often would you play the Epitome game? How long would you play the Epitome game at once? Would you prefer to play only one integrated game? Please rank three platforms for playing the Epitome game according to your preferences? How do you like the Mobile interface? How do you like the Facebook interface? Would you enjoy the Epitome game more if you play with less than nine images? If yes, how many images should be displayed? How much do you prefer to watch your friends’ photos compared to the photos of unknown people? Is it good to show your rank and compare it with your friends’ ranks for the enjoyment of Epitome? Is it good to have your summarization sequence as a result of the Epitome game? How many images in album summarization sequence of photos would you prefer? There are two statements: 1st statement—to have perfectly summarized Facebook album, but waiting for it long time period; 2nd statement—to have preliminarily summarized Facebook album after a short time. Which of these statements is more important for you? To whom would you allow Epitome to show your private photos which are not shared even with your friends in order to receive a good summarization of your Facebook albums? To whom would you allow Epitome to show your private photos which are shared just with your friends in order to receive a good summarization of your Facebook albums? To whom would you allow Epitome to show your private photos which were shared with friends of friends in order to receive a good summarization of your Facebook albums? To whom would you allow Epitome to show photos in which you were tagged in order to receive a good summarization of your Facebook albums? To whom would you allow Epitome to show photos of your friends in order to receive a good summarization of your Facebook albums? Do you want to play with photos of your friends even if they do not play Epitome? Any suggestions to improve the game?
This work was supported by the Swiss National Foundation for Scientific Research in the framework of NCCR Interactive Multimodal Information Management (IM2) and the Swiss National Science Foundation Grant “Multimedia Security” (no. 200020-113709), partially supported by the European Network of Excellence PetaMedia (FP7/2007-2011).