Classifying events is challenging in Twitter because tweets texts have a large amount of temporal data with a lot of noise and various kinds of topics. In this paper, we propose a method to classify events from Twitter. We firstly find the distinguishing terms between tweets in events and measure their similarities with learning language models such as ConceptNet and a latent Dirichlet allocation method for selectional preferences (LDA-SP), which have been widely studied based on large text corpora within computational linguistic relations. The relationship of term words in tweets will be discovered by checking them under each model. We then proposed a method to compute the similarity between tweets based on tweets’ features including common term words and relationships among their distinguishing term words. It will be explicit and convenient for applying to k-nearest neighbor techniques for classification. We carefully applied experiments on the Edinburgh Twitter Corpus to show that our method achieves competitive results for classifying events.
Twitter (
Classifying events in Twitter is a difficult task that focuses on the automatic identification and classification of various types of events in tweet texts. In Twitter, events are topics that often draw public attention, for example, football matches or natural disasters. Several approaches have been proposed to classify events for detection such as wave analysis [
Some samples of discussed tweets in two events.
Category | Tweets | Relatedness with event |
---|---|---|
Event 1 |
|
Yes |
|
Yes | |
|
No | |
|
||
Event 2 |
|
Yes |
|
Yes | |
|
No |
In this paper, we investigate the use of generative and discriminate models for identifying the relationship of objects in tweets that describe one or more instance of a specified event type. We adapt language modeling approaches that capture how descriptions of event instances in text are likely to be generated. Our method will find the distinguishing term words between tweets and examining them with a series of relationships, extracted by language models such as ConceptNet [
The rest of this paper is structured as follows. Section
Several applications have detected events in Web to apply to weblogs [
Some research has focused on summarizing Twitter posts for detecting events [
Recently, several approaches have been proposed to detect events from tweets using topic model approach [
In this paper, we investigate the use of generative and discriminate models for identifying the relationship among objects in tweets that describe one or more instances of a specified event type. We adapt language modeling approaches that capture how descriptions of event instances in text are likely to be generated. We use language models to select plausible relationships between term words in tweets such as the relationship of “Object-Object” or “Object-relation-Object,” which aim to detect the relatedness of an event in tweets. We assume that the data collection of language models contains suitable knowledge on the relationships among term words to discover the elemental relationship among tweets with a statistical analysis to classify events. We explore two types of language models that have obtained high correlation with human judgment such as ConceptNet and LDA-SP. These models are used for calculating the similarity of a pairwise of tweets for detecting events. The relationship between the discriminate term words of the tweets will be discovered by checking their relatedness under pairs of relations. In addition, the similarity between tweets is computed based on their common term words and the relationship between their discriminate term words. It is intuitive and convenient to apply it in classifier algorithms to classify events in Twitter. The general proposed method consists of four stages as
Proposed method.
To model the “Object-Object” relationships in tweets, we consider the ConceptNet [
ConceptNet model. (a) List of relations. (b) Samples of extracted relations.
|
MadeOf | AtLocation | MotivatedByGoal | ReceivesAction | ||||
---|---|---|---|---|---|---|---|
Atomic bomb | Uranium | Nasa | United states | Fight war | Freedom | Bacteria | Kill |
Computer | Silicon | Alcoa | Pittsburgh | Get drunk | Forget life | Army tank | Warfare |
Gas | Oil | Tv channel | Russia | Pen | Write letter | Bread | Cook |
Song | Music | Aozora bank | Japan | Join army | Defend country | Candle | Burn for light |
Person | Live cell | Apartheid | Mall | Kill | Hate someone | Tomato | Squash |
Light | Energy | Golden gate | Bridge | Live life | Pleasure | Tobacco | Chew |
Carton | Wax paper | Art | Gallery | Sing | Performance | Supply | Store |
Chocolate | Cocoa bean | Audience | Theatre | Socialize | Be popular | Ruby | Polish |
Telephone | Electronics | Crab | Coastal area | Study | Concentrate | Money | Loan |
Window | Glass | Handgun | Army | Visit museum | See history | Life | Save |
⋯ | ⋯ | ⋯ | ⋯ | ⋯ | ⋯ | ⋯ | ⋯ |
To model the “Object-relation-Object” relationships in tweets, we adapt the LDA-SP model [
LDA-SP model. (a) Samples of list topics; (b) sample of list relations.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Relations | Relationship of topics ( |
---|---|
Be cite | 525–561; 251–286; 286–251; 251-251; 542–251; 371–286; 542–371; 542–286; 251–162; 134–286; 162–286; 371–251; 286–162; 286–171; 542–454; 286–538; 454–286; 286–10; 134–24; 538–286; 285-286; 575–454; 572–286; 328–286; 19–454; … |
|
|
Blame on | 116–428; 329–531; 116–531; 329-329; 329–116; 116–584; 329–584; 584–531; 314–531; 116–329; 480–531; 171–116; 116–160; 239–584; 458–531; 404–531; 584–116; 196–116; 531–458; 584-584; 531–116; 196–531; 176–531; 545–147; 171–2; … |
|
|
Crash into | 428–287; 428–571 390–106; 428–139; 428–390; 428-428; 390–139; 390-390; 390–287; 390–428; 428–570; 390–570; 139–106; 139–428; 139-139; 428–328; 287–106; 139–390; 390–328; 139–287; 428–374; 390–374; 287–139; 570–287; 106–428; … |
|
|
Spot in | 114–433; 433-433; 116–525; 114–287; 287–433; 114–570; 405–433; 433–405; 251–433; 114-114; 223–433; 570–433; 433–570; 114–132; 287–405; 114–251; 543–433; 230–433; 223–570; 114–424; 433–287; 433–114; 570-570; 433–132; 223–279; … |
Graphical model of LDA model (a) versus LDA-SP (b).
Classifying events in tweets from Twitter is a very challenging task because a very few words cooccur in tweets. Intuitively, the problem can be solved by exploring the relationships between tweets well; the intrinsic relationship among words may be discovered with a thesaurus. Hence, we present a method to discover the intrinsic relationships between objects based on statistical analysis of language models and then gain the similarity between tweets accordingly. We consider two types of relationships in tweets such as “Object-Object” and “Object-relation-Object.”
“
Relationship “
“
Relationship “
Our method extracts relation tuples from language models such as ConceptNet and LDA-SP. We treat all tweets from Twitter that are contained in the collection equally and then perform to match models of tuples generated from ConceptNet and LDA-SP with them. Hence, if we can discover relation tuples as “third-party” for both tweets and calculate the similarity between the two tweets by comparing the distinguishing term words with these tuples, we may find the real relationship underlying the two tweets. We assume that the data collection language models contain sufficient knowledge about the relationships among term words, from which we can find the elemental relationship among tweets.
For computing similarity between tweets, we derive a set of relations,
With the relationship between the two distinguishing term words on a diversity of assigned model tuples, we can calculate the similarity of vectors
For classifying events from tweets, many classifiers first need to calculate the similarity between tweets.
We have conducted experiments on the Edinburgh Twitter Corpus [
Experimental datasets.
Category | Description | Number of tweets | Checked |
---|---|---|---|
Event 1 | Death of Amy Winehouse | 774 | ✓ |
Event 2 | Space shuttle Atlantis lands safely, ending NASA’s space shuttle program | 45 | |
Event 3 | Betty Ford dies | 8 | |
Event 4 | Richard Bowes, victim of London riots, dies in hospital | 27 | |
Event 5 | Flight Noar Linhas Aereas 4896 crashes, all 16 passengers dead | 9 | |
Event 6 | S&P downgrades US credit rating | 275 | ✓ |
Event 7 | US increases debt ceiling | 73 | ✓ |
Event 8 | Terrorist attack in Delhi | 40 | |
Event 9 | Earthquake in Virginia | 271 | ✓ |
Event 10 | Trevor Ellis (first victim of London riots) dies | 63 | |
Event 11 | Goran Hadzic, Yugoslavian war criminal, arrested | 2 | |
Event 12 | India and Bangladesh sign a peace pact | 3 | |
Event 13 | Plane carrying Russian hockey team Lokomotiv crashes, 44 dead | 225 | ✓ |
Event 14 | Explosion in French nuclear power plant Marcoule | 137 | ✓ |
Event 15 | NASA announces discovery of water on Mars | 110 | ✓ |
Event 16 | Google announces plans to buy Motorola Mobility | 130 | ✓ |
Event 17 | Car bomb explodes in Oslo, Norway | 21 | |
Event 18 | Gunman opens fire in children’s camp on Utoya island, Norway | 28 | |
Event 19 | First artificial organ transplant | 16 | |
Event 20 | Petrol pipeline explosion in Kenya | 27 | |
Event 21 | Famine declared in Somalia | 71 | ✓ |
Event 22 | South Sudan declares independence | 26 | |
Event 23 | South Sudan becomes a UN member state | 7 | |
Event 24 | Three men die in riots in Birmingham | 12 | |
Event 25 | Riots break out in Tottenham | 19 | |
Event 26 | Rebels capture Tripoli international airport, Libya | 4 | |
Event 27 | Ferry sinks in Zanzibar, around 200 dead | 21 |
In this study, experiments are evaluated based on the precision, recall, and
Checking similarity between tweets before experiments, we select some samples of tweets from experimental datasets as shown in Table
Sample of similarities calculated by the proposed methods and the tf-
Tweets | tf- |
tf- |
tf- |
---|---|---|---|
|
0.16 | 0.365 | 0.4 |
|
|||
|
0.123 | 0.078 | 0.084 |
|
|||
|
0 | 0 | 0 |
|
|||
|
0.433 | 0.452 | 0.468 |
|
|||
|
0.272 | 0.146 | 0.104 |
To classify events, 70% of the tweets for each category are randomly selected for training, and the rest is for testing. In our experiments, we compare the performance of four classifiers implemented as follows:
Experimental results.
Category |
|
SVM |
|
|
||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
| |
Event 1 | 76.3 | 71.6 | 73.8 | 75.2 | 75.5 | 75.3 | 86.1 | 77.5 | 81.6 | 88.2 | 82.6 |
|
Event 6 | 84.6 | 85.4 | 84.9 | 86.9 | 87.2 | 87.1 | 91.1 | 89.4 |
|
89.1 | 86.4 | 87.7 |
Event 7 | 78.9 | 72.3 | 75.5 | 80.4 | 76.2 | 78.2 | 87.5 | 82.3 |
|
82.4 | 78.9 | 80.6 |
Event 9 | 83.9 | 78.8 | 81.3 | 85.5 | 80.2 | 82.3 | 93.8 | 92.9 |
|
87.2 | 83.3 | 85.2 |
Event 13 | 83.6 | 72.4 | 77.5 | 82.8 | 75.6 | 79.1 | 86.2 | 80.5 | 83.3 | 87.3 | 82.6 |
|
Event 14 | 70.1 | 67.8 | 68.9 | 71.6 | 70.0 | 70.8 | 85.2 | 78.7 |
|
83.8 | 74.3 | 78.8 |
Event 15 | 79.3 | 71.5 | 75.2 | 81.0 | 70.8 | 75.6 | 90.1 | 87.9 |
|
88.8 | 85.8 | 87.3 |
Event 16 | 80.5 | 72.4 | 76.2 | 82.5 | 73.1 | 77.5 | 85.7 | 80.0 |
|
85.5 | 79.6 | 82.5 |
Event 21 | 81.6 | 74.1 | 77.7 | 82.4 | 76.8 | 79.5 | 83.9 | 77.8 | 80.7 | 85.4 | 77.1 |
|
Overall | 79.5 | 74.4 | 76.8 | 79.9 | 77.0 | 78.4 | 87.9 | 82.4 |
|
87.4 | 82.3 | 84.7 |
The overall performance comparison is presented in Figure
Overall performance comparisons.
We believe that effective performance of our proposed methods is result of the following reasons.
First, noise and exclamative and repeated texts usually occur in the tweets of each event. The following are examples of such tweets.
The second reason we believe our method had effective performance is that quality universal datasets are used to build language models. In this study, more than five billion relation records extracted from Concept are used to build the models. In addition, models from LDA-SP are built by extracting 2.4 million tuples of relations and 601 topics. Furthermore, ConceptNet is a graphical relationship model which uses predefined rules. However, LDA-SP still has some errors [
The third reason believed to be behind our method’s effective performance is that the models extracted from LDA-SP are intensely analyzed compared to ConceptNet for relationship. However ConceptNet obtained better performance results. Texts from tweets are incomplete sentences that result in failures in grammar parsing for analyzing relation. We did not include grammar parsing for analyzing tweets based on LDA-SP model. Therefore, ConceptNet exhibits a better performance for classifying events from Twitter than LDA-SP.
We have presented methods to classify events from Twitter. We first find the distinguishing terms between tweets in events and calculate their similarity with learning language models: LDA-SP and ConceptNet. Next, we discover the relationship between the distinguishing terms of the tweets by examining them under each model. Then, we calculate the similarity between two tweets based on their common terms and the relationship between their distinguishing terms. The outcomes make it convenient to apply
Regarding future work, the research has been suggested with attractive aspects to improve as follows. First, this approach can be considered for future work, including it with a larger corpus and experimenting with other event types. Second, we will continue to investigate how to apply grammar parsing in tweets so that we can analyze deeply relationships to serve for classifying events. Finally, the research can be applied unsupervised learning with semantic similarity models as pointwise mutual information (PMI) [
The authors declare that there is no conflict of interests regarding the publication of this paper.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2012R1A1A2006906). The authors would like to thank the anonymous reviewers for their constructive comments and suggestions on the paper.