sentiwordnet3

S ENTI W ORD N ET3.0:An Enhanced Lexical Resource

for Sentiment Analysis and Opinion Mining

Stefano Baccianella,Andrea Esuli,and Fabrizio Sebastiani

Istituto di Scienza e Tecnologie dell’Informazione

Consiglio Nazionale delle Ricerche

Via Giuseppe Moruzzi1,56124Pisa,Italy

E-mail: https://www.wendangku.net/doc/fa11256531.html,stname @https://www.wendangku.net/doc/fa11256531.html,r.it

Abstract

In this work we present S ENTI W ORD N ET3.0,a lexical resource explicitly devised for supporting sentiment classi?cation and opinion mining applications.S ENTI W ORD N ET3.0is an improved version of S ENTI W ORD N ET1.0,a lexical resource publicly available for research purposes,now currently licensed to more than300research groups and used in a variety of research projects worldwide.Both S ENTI W ORD N ET1.0and3.0are the result of automatically annotating all W ORD N ET synsets according to their degrees of positivity, negativity,and neutrality.S ENTI W ORD N ET1.0and3.0differ(a)in the versions of W ORD N ET which they annotate(W ORD N ET 2.0and3.0,respectively),(b)in the algorithm used for automatically annotating W ORD N ET,which now includes(additionally to the previous semi-supervised learning step)a random-walk step for re?ning the scores.We here discuss S ENTI W ORD N ET3.0,especially focussing on the improvements concerning aspect(b)that it embodies with respect to version1.0.We also report the results of evaluating S ENTI W ORD N ET3.0against a fragment of W ORD N ET3.0manually annotated for positivity,negativity,and neutrality;these results indicate accuracy improvements of about20%with respect to S ENTI W ORD N ET1.0.

1.Introduction

In this work we present S ENTI W ORD N ET3.0,an enhanced lexical resource explicitly devised for supporting sentiment classi?cation and opinion mining applications(Pang and Lee,2008).S ENTI W ORD N ET3.0is an improved version of S ENTI W ORD N ET1.0(Esuli and Sebastiani,2006),a lexical resource publicly available for research purposes, now currently licensed to more than300research groups and used in a variety of research projects worldwide.

S ENTI W ORD N ET is the result of the automatic annota-tion of all the synsets of W ORD N ET according to the no-tions of“positivity”,“negativity”,and“neutrality”.Each synset s is associated to three numerical scores P os(s), Neg(s),and Obj(s)which indicate how positive,nega-tive,and“objective”(i.e.,neutral)the terms contained in the synset are.Different senses of the same term may thus have different opinion-related properties.For example,in S ENTI W ORD N ET1.0the synset[estimable(J,3)]1, corresponding to the sense“may be computed or esti-mated”of the adjective estimable,has an Obj score of1.0(and P os and Neg scores of0.0),while the synset [estimable(J,1)]corresponding to the sense“de-serving of respect or high regard”has a P os score of0.75, a Neg score of0.0,and an Obj score of0.25.

Each of the three scores ranges in the interval[0.0,1.0], and their sum is1.0for each synset.This means that a synset may have nonzero scores for all the three categories, which would indicate that the corresponding terms have,in the sense indicated by the synset,each of the three opinion-related properties to a certain degree.

This paper is organized as follows.Section2.brie?y

1We here adopt the standard convention according to which a term enclosed in square brackets denotes a synset;thus [poor(J,7)]refers not just to the term poor but to the synset consisting of adjectives{inadequate(J,2),poor(J,7), short(J,4)}.charts the history of S ENTI W ORD N ET,from its very ear-liest release to the current version,thus providing context for the following sections.Section3.examines in detail the algorithm that we have used for generating S ENTI W ORD-N ET3.0,while Section4.discusses accuracy issues.

S ENTI W ORD N ET3.0is freely available for non-pro?t research purposes from http://sentiwordnet. https://www.wendangku.net/doc/fa11256531.html,r.it/

2.A brief history of S ENTI W ORD N ET Four different versions of S ENTI W ORD N ET have been dis-cussed in publications:

1.S ENTI W ORD N ET1.0,presented in(Esuli and Sebas-

tiani,2006)and publicly made available for research purposes;

2.S ENTI W ORD N ET1.1,only discussed in a technical

report(Esuli and Sebastiani,2007b)that never reached the publication stage;

3.S ENTI W ORD N ET2.0,only discussed in the second

author’s PhD thesis(Esuli,2008);

4.S ENTI W ORD N ET3.0,which is being presented here

for the?rst time.

Since versions1.1and2.0have not been discussed in widely known formal publications,we here focus on dis-cussing the differences between versions1.0and3.0.The main differences are the following:

1.Version1.0(similarly to1.1and

2.0)consists of an

annotation of the older W ORD N ET2.0,while version

3.0is an annotation of the newer W ORD N ET3.0. 2.For S ENTI W ORD N ET1.0(and1.1),automatic anno-

tation was carried out via a weak-supervision,semi-supervised learning algorithm.Conversely,for S EN-

TI W ORD N ET(2.0and)3.0the results of this semi-supervised learning algorithm are only an intermedi-ate step of the annotation process,since they are fed to an iterative random-walk process that is run to conver-gence.S ENTI W ORD N ET(2.0and)3.0is the output of the random-walk process after convergence has been reached.

3.Version1.0(and1.1)uses the glosses of W ORD-

N ET synsets as semantic representations of the synsets themselves when a semi-supervised text classi?cation process is invoked that classi?es the(glosses of the) synsets into categories P os,Neg and Obj.In ver-sion2.0this is the?rst step of the process;in the sec-ond step the random-walk process mentioned above uses not the raw glosses,but their automatically sense-disambiguated versions from E X TENDED W ORD N ET (Harabagiu et al.,1999).In S ENTI W ORD N ET3.0 both the semi-supervised learning process(?rst step) and the random-walk process(second step)use instead the manually disambiguated glosses from the Prince-ton WordNet Gloss Corpus2,which we assume to be more accurate than the ones from E X TENDED W ORD-N ET.

3.Generating S ENTI W ORD N ET3.0

We here summarize in more detail the automatic anno-tation process according to which S ENTI W ORD N ET3.0 is generated.This process consists of two steps,(1)a weak-supervision,semi-supervised learning step,and(2)

a random-walk step.

3.1.The semi-supervised learning step

The semi-supervised learning step is identical to the pro-cess used for generating S ENTI W ORD N ET1.0;(Esuli and Sebastiani,2006)can then be consulted for more details on this process.The step consists in turn of four substeps:(1) seed set expansion,(2)classi?er training,(3)synset classi-?cation,and(4)classi?er combination.

1.In Step(1),two small“seed”sets(one consisting of

all the synsets containing7“paradigmatically posi-tive”terms,and the other consisting of all the synsets containing7“paradigmatically negative”terms(Tur-ney and Littman,2003))are automatically expanded by traversing a number of W ORD N ET binary relations than can be taken to either preserve or invert the P os and Neg properties(i.e.,connect synsets of a given polarity with other synsets either of the same polarity –e.g.,the“also-see”relation–or of the opposite po-larity–e.g.,the“direct antonymy”relation),and by adding the synsets thus reached to the same seed set (for polarity-preserving relations)or to the other seed set(for polarity-inverting ones).This expansion can be performed with a certain“radius”;i.e.,using radius k means adding to the seed sets all the synsets that are within distance k from the members of the origi-nal seed sets in the graph collectively resulting from the binary relationships considered.

2https://www.wendangku.net/doc/fa11256531.html,/glosstag. shtml

2.In Step(2),the two sets of synsets generated in the pre-

vious step are used,along with another set of synsets assumed to have the Obj property,as training sets for training a ternary classi?er(i.e.one that needs to clas-sify a synset as P os,Neg,or Obj).The glosses of the synsets are used by the training module instead of the synsets themselves,which means that the resulting classi?er is indeed a gloss(rather than a synset)clas-si?er.S ENTI W ORD N ET1.0uses a“bag of words”

model,according to which the gloss is represented by the(frequency-weighted)set of words occurring in it.In S ENTI W ORD N ET3.0we instead leverage on the manually disambiguated glosses available from the Princeton WordNet Gloss Corpus,according to which

a gloss is actually a sequence of W ORD N ET synsets.

Our gloss classi?ers are thus based on what might be called a“bag of synsets”model.

3.In Step(3)all W ORD N ET synsets(including those

added to the seed sets in Step(2))are classi?ed as be-longing to either P os,Neg,or Obj via the classi?er generated in Step(2).

4.Step(2)can be performed using different values of

the radius parameter,and different supervised learn-ing technologies.For reasons explained in detail in (Esuli and Sebastiani,2006),annotation turns out to be more accurate if,rather that a single ternary classi?er,

a committee of ternary classi?ers is generated,each of

whose members results from a different combination of choices for these two parameters(radius and learn-ing technology).We have set up our classi?er com-mittee as consisting of8members,resulting from four different choices of radius(k∈{0,2,4,6})and two different choices of learning technology(Rocchio and SVMs).In Step(4)the?nal P os(resp.,Neg,Obj) value of a given synset is generated as its average P os (resp.,Neg,Obj)value across the eight classi?ers in the committee.

3.2.The random-walk step

The random-walk step consists of viewing W ORD N ET3.0 as a graph,and running an iterative,“random-walk”pro-cess in which the P os(s)and Neg(s)(and,consequently, Obj(s))values,starting from those determined in the pre-vious step,possibly change at each iteration.The random-walk step terminates when the iterative process has con-verged.

The graph used by the random-walk step is the one implicitly determined on W ORD N ET by the de?niens-de?niendum binary relationship;in other words,we assume the existence of a directed link from synset s1to synset s2if and only if s1(the de?niens)occurs in the gloss of synset s2 (the de?niendum).The basic intuition here is that,if most of the terms that are being used to de?ne a given term are pos-itive(resp.,negative),then there is a high probability that the term being de?ned is positive(resp.,negative)too.In other words,positivity and negativity are seen as“?owing through the graph”,from the terms used in the de?nitions to the terms being de?ned.

However,it should be observed that,in“regular”W ORD N ET,the de?niendum is a synset while the de?niens is a non-disambiguated term,since glosses are sequences of non-disambiguated terms.In order to carry out the random-walk step,we need the glosses to be disambiguated against W ORD N ET itself,i.e.,we need them to be sequences of W ORD N ET synsets.While for carrying out the random-walk step for S ENTI W ORD N ET2.0we had used the auto-matically disambiguated glosses provided by E X TENDED-W ORD N ET(Harabagiu et al.,1999),for S ENTI W ORD N ET 3.0we use the manually disambiguated glosses available from the above-mentioned Princeton WordNet Gloss Cor-pus.

The mathematics behind the random-walk step is fully described in(Esuli and Sebastiani,2007a),to which the in-terested reader is then referred for details.In that paper, the random-walk model we use here is referred to as“the inverse?ow model”.

Two different random-walk processes are executed for the positivity and negativity dimensions,respectively,of S ENTI W ORD N ET,producing two different rankings of the W ORD N ET synsets.However,the actual numerical values returned by the random-walk process are un?t to be used as the?nal P os and Neg scores,since they are all too small(the synset top-ranked for positivity would obtain a P os score of8.3?10?6);as a result,even the top-ranked positive synsets would turn out to be overwhelmingly neu-tral and only feebly positive.Since,as we have observed, both the positivity and negativity scores resulting from the semi-supervised learning step follow a power law distribu-tion(i.e.,very few synsets have a very high P os(resp., Neg)score,while very many synsets are mostly neutral), we have thus?t these scores with a function of the form F P os(x)=a1x b1(resp.,F Neg(x)=a2x b2),thus deter-mining the a1and b1(resp.,a2and b2)values that best?t the actual distribution of values.The?nal S ENTI W ORD-N ET3.0P os(s)(resp.,Neg)values are then determined by applying the resulting function F P os(x)=a1x b1(resp., F P os(x)=a2x b2)to the ranking by positivity(resp.,by negativity)produced by the random-walk process.

Obj(S)values are then assigned so as to make the three values sum up to one.In the case in which P os(s)+ Neg(s)>1we have normalized the two values to sum up to one and we have set Obj(s)=03.

As an example,Table1reports the10top-ranked posi-tive synsets and the10top-ranked negative synsets in S EN-TI W ORD N ET3.0.

4.Evaluating S ENTI W ORD N ET3.0

For evaluating the accuracy of S ENTI W ORD N ET3.0we follow the methodology discussed in(Esuli,2008).This consists in comparing a small,manually annotated subset of W ORD N ET against the automatic annotations of the same synsets as from S ENTI W ORD N ET.

4.1.Micro-WN(Op)-3.0

In(Esuli,2008),S ENTI W ORD N ET1.0,1.1and2.0were evaluated on Micro-WN(Op)(Cerini et al.,2007),a care-fully balanced set of1,105W ORD N ET synsets manually 3This happened only for16synsets.annotated according to their degrees of positivity,negativ-ity,and neutrality.

Micro-WN(Op)consists of1,105synsets manually an-notated by a group of?ve human annotators(hereafter called J1,...,J5);each synset s is assigned three scores P os(s),Neg(s),and Obj(s),with the three scores sum-ming up to1.Synsets1-110(here collectively called Micro-WN(Op)(1))were tagged by all the annotators work-ing together,so as to develop a common understanding of the semantics of the three categories;then,J1,J2and J3independently tagged all of synsets111–606(Micro-WN(Op)(2)),while J4and J5independently tagged all of synsets607–1105(Micro-WN(Op)(3)).Our evaluation is performed on the union of synsets composing Micro-WN(Op)(2)and Micro-WN(Op)(3).It is noteworthy that Micro-WN(Op)as a whole,and each of its subsets,are rep-resentative of the distribution of parts of speech in W ORD-N ET:this means that,e.g.,if x%of W ORD N ET synsets are nouns,also x%of Micro-WN(Op)synsets are nouns. Moreover,this property also holds for each single part Micro-WN(Op)(x)of Micro-WN(Op).

As for the evaluation of S ENTI W ORD N ET3.0,it should be noted that Micro-WN(Op)is the annotation of a sub-set of W ORD N ET2.0,and cannot be directly used for evaluating S ENTI W ORD N ET 3.0,which consists of an annotation of W ORD N ET3.0.Deciding which W ORD-N ET3.0synset corresponds to a given synset in Micro-WN(Op)cannot be determined with certainty,and may even be considered an ill-posed question.In fact,several of the synsets in Micro-WN(Op)do not exist any longer in W ORD N ET3.0,at least in the same form.For example, the synset[good(A,22)]does no longer exist,while the synset{gloomy(A,2),drab(A,3),dreary(A,1), dingy(A,3),sorry(A,6),dismal(A,1)}now contains not only all of these words(although with differ-ent sense numbers)but also blue(A,3),dark(A,9), disconsolate(A,2),and grim(A,6).

As a result,we decided to develop an automatic map-ping method that,given a synset s in W ORD N ET2.0,iden-ti?es its analogue in W ORD N ET3.0.We then took all of the W ORD N ET2.0synsets in Micro-WN(Op),identi?ed their W ORD N ET3.0analogues,assigned them the same P os(s),Neg(s),and Obj(s)as in the original Micro-WN(Op)synset,and used the resulting1,105annotated W ORD N ET3.0synsets as the gold standard against which to evaluate S ENTI W ORD N ET3.0.

Our synset mapping method is based on the combina-tion of three mapping strategies,which we apply in this order:

1.W ORD N ET sense mappings:We?rst use the sense

mappings between W ORD N ET 2.0and 3.0avail-able at http://wordnetcode.princeton.

edu/3.0/WNsnsmap-3.0.tar.gz.These map-pings were derived automatically using a number of heuristics,and are unfortunately limited to nouns and verbs only.Each mapping has a con?dence value asso-ciated to it,ranging from0(lowest con?dence)to100 (highest con?dence).The majority of mappings have a100con?dence score associated to them.As recom-

Table1:The10top-ranked positive synsets and the10top-ranked negative synsets in S ENTI W ORD N ET3.0.

Rank Positive Negative

1good#n#2goodness#n#2abject#a#2

2better off#a#1deplorable#a#1distressing#a#2 lamentable#a#1pitiful#a#2sad#a#3 sorry#a#2

3divine#a#6elysian#a#2inspired#a#1bad#a#10un?t#a#3unsound#a#5

4good enough#a#1scrimy#a#1

5solid#a#1cheapjack#a#1shoddy#a#1tawdry#a#2 6superb#a#2unfortunate#a#3

7good#a#3inauspicious#a#1unfortunate#a#2

8goody-goody#a#1unfortunate#a#1

9amiable#a#1good-humored#a#1good-humoured#a#1dispossessed#a#1homeless#a#2roof-less#a#2

10gainly#a#1hapless#a#1miserable#a#2misfortu-nate#a#1pathetic#a#1piteous#a#1 pitiable#a#2pitiful#a#3poor#a#1 wretched#a#5

mended in the documentation associated to the map-pings,we have used only the highest-valued mappings (those with a100score),ignoring the others.Heuris-tics used for the determination of mappings include the comparison of sense keys,similarity of synset terms, and relative tree location(comparison of hypernyms).

By using these mappings we have mapped269Micro-WN(Op)synsets to W ORD N ET3.0.

2.Synset term matching:If a Micro-WN(Op)synset

s i(that has not already been mapped in the previous step)contains exactly the same terms of a W ORD N ET

3.0synset s j,and such set of terms appears only in one

synset in both W ORD N ET2.0and3.0,we consider s i and s j to represent the same concept.

3.Gloss similarity:For each Micro-WN(Op)synset s i

that has not been mapped by the previous two meth-ods,we compute the similarity between its gloss and the glosses of all W ORD N ET3.0synsets,where a gloss g i is represented by the set of all character tri-grams contained in it.Similarity is computed via the Dice coef?cient4

Dice(g1,g2)=2|g1∩g2|

|g1|+|g2|

(1)

In Equation1a higher Dice(g1,g2)value means a stronger similarity.Given a Micro-WN(Op)synset s i, its most similar W ORD N ET3.0gloss is determined, and the corresponding synset is chosen as the one matching s i.

The Princeton research group had originally not used gloss similarity to produce the sense mappings used in Step1be-cause,as reported in the documentation distributed with the mappings,“Glosses(...)are often signi?cantly modi?ed”. We have found,by manually inspecting a sample of the re-sults,that gloss similarity mapping was rather effective in our case.

4See also https://www.wendangku.net/doc/fa11256531.html,/wiki/Dice_ coefficient

The?nal result of this mapping process,that we call Micro-WN(Op)-3.0,is publicly available at http:// https://www.wendangku.net/doc/fa11256531.html,r.it/.It should be noted that the results of the automatic mapping process have not been completely checked for correctness,since checking if there is a better map for Micro-WN(Op)synset s than the current map requires in theory to search among all the W ORD N ET3.0synsets with the same POS.Therefore,the results of evaluations obtained on Micro-WN(Op)-3.0are not directly comparable with those obtained on the original Micro-WN(Op).

4.2.Evaluation measure

In order to evaluate the quality of S ENTI W ORD N ET we test how well it ranks by positivity(resp.,negativity)the synsets in Micro-WN(Op)-3.0.As our gold standard we thus use a ranking by positivity(resp.,negativity)of Micro-WN(Op)-3.0,obtained by sorting the Micro-WN(Op)-3.0synsets ac-cording to their P os(s)(resp.,Neg(s))values).Similarly, we generate a ranking by positivity(resp.,negativity)of the same synsets from the P os(s)and Neg(s)values as-signed by S ENTI W ORD N ET3.0,and compare them against the gold standard above.

We compare rankings by using the p-normalized Kendallτdistance(notedτp–see e.g.,(Fagin et al.,2004)) between the gold standard rankings and the predicted rank-ings.Theτp distance,a standard function for the evaluation of rankings that possibly admit ties,is de?ned as:

τp=

n d+p·n u

Z

(2)

where n d is the number of discordant pairs,i.e.,pairs of objects ordered one way in the gold standard and the other way in the tested ranking;n u is the number of pairs which are ordered(i.e.,not tied)in the gold standard and are tied in the tested ranking;p is a penalization to be attributed to each such pair,set to p=1

2

(i.e.,equal to the probability that a ranking algorithm correctly orders the pair by ran-dom guessing);and Z is a normalization factor(equal to the number of pairs that are ordered in the gold standard)

Table2:τp values for the positivity and negativity rankings derived from S ENTI W ORD N ET1.0and3.0,as measured on Micro-WN(Op)and Micro-WN(Op)-3.0.

Rankings

Positivity Negativity

S ENTI W ORD N ET1.0.349.296

S ENTI W ORD N ET3.0.281(-19.48%).231(-21.96%) whose aim is to make the range ofτp coincide with the [0,1]interval.Note that pairs tied in the gold standard are not considered in the evaluation.The lower theτp value the better;for a prediction which perfectly coincides with the gold standard,τp equals0;for a prediction which is exactly the inverse of the gold standard,τp is equal to1.

4.3.Results

Table2reports theτp values for the positivity and negativ-ity rankings derived from S ENTI W ORD N ET1.0and3.0,as measured on Micro-WN(Op)and Micro-WN(Op)-3.0,re-spectively.The values for S ENTI W ORD N ET1.0are ex-tracted from(Esuli and Sebastiani,2007b).As already pointed out in Section4.1.,we warn the reader that the com-parison between the S ENTI W ORD N ET1.0and3.0results is only partially reliable,since Micro-WN(Op)-3.0(on which the S ENTI W ORD N ET3.0results are based)may contain annotation errors introduced by the automatic mapping pro-cess.

Taking into account the above warning,we can ob-serve that S ENTI W ORD N ET3.0is substantially more ac-curate than S ENTI W ORD N ET1.0,with a19.48%relative improvement for the ranking by positivity and a21.96% improvement for the ranking by negativity.

We have also measured(see Table3)the difference in accuracy between the rankings produced by S ENTI-W ORD N ET3.0-semi and S ENTI W ORD N ET3.0,where by “S ENTI W ORD N ET3.0-semi”we refer to the outcome of the semi-supervised learning step(described in Section 3.1.)that led to the generation of S ENTI W ORD N ET3.0. The reason we have measured this difference is to check whether the random-walk step of Section3.2.is indeed ben-e?cial.The relative improvement of S ENTI W ORD N ET3.0 with respect to S ENTI W ORD N ET3.0-semi is17.11%for the ranking by positivity,and19.23%for the ranking by negativity;this unequivocally shows that the random-walk process is indeed bene?cial.

It would certainly have been interesting to also mea-sure the impact that the manually disambiguated glosses available from the Princeton WordNet Gloss Corpus have had in generating S ENTI W ORD N ET,by comparing the performance obtained by using them(either in the semi-supervised learning step,or in the random-walk step,or in both)with the performance obtained by using the au-tomatically disambiguated ones from E X TENDED W ORD-N ET.Unfortunately,this is not possible,since the former glosses are available for W ORD N ET-3.0only,while E X-TENDED W ORD N ET is available for W ORD N ET-2.0only.Table3:τp values for the positivity and negativity rankings derived from(a)the results of the semi-supervised learning step of S ENTI W ORD N ET3.0,and(b)S ENTI W ORD N ET 3.0,as measured on Micro-WN(Op)-3.0.

Rankings

Positivity Negativity

S ENTI W ORD N ET3.0-semi.339.286 S ENTI W ORD N ET3.0.281(-17.11%).231(-19.23%)

5.References

S.Cerini,https://www.wendangku.net/doc/fa11256531.html,pagnoni,A.Demontis,M.Formentelli, and G.Gandini.2007.Micro-WNOp:A gold stan-dard for the evaluation of automatically compiled lexi-cal resources for opinion mining.In Andrea Sans`o,edi-tor,Language resources and linguistic theory:Typology, second language acquisition,English linguistics,pages 200–210.Franco Angeli Editore,Milano,IT.

Andrea Esuli and Fabrizio Sebastiani.2006.S ENTI-W ORD N ET:A publicly available lexical resource for opinion mining.In Proceedings of the5th Conference on Language Resources and Evaluation(LREC’06),pages 417–422,Genova,IT.

Andrea Esuli and Fabrizio Sebastiani.2007a.Random-walk models of term semantics:An application to opinion-related properties.In Proceedings of the3rd Language Technology Conference(LTC’07),pages221–225,Pozna′n,PL.

Andrea Esuli and Fabrizio Sebastiani.2007b.S ENTI-W ORD N ET:A high-coverage lexical resource for opin-ion mining.Technical Report2007-TR-02,Istituto di Scienza e Tecnologie dell’Informazione,Consiglio Nazionale delle Ricerche,Pisa,IT.

Andrea Esuli.2008.Automatic Generation of Lexical Re-sources for Opinion Mining:Models,Algorithms,and Applications.Ph.D.thesis,Scuola di Dottorato in Ingeg-neria”Leonardo da Vinci”,University of Pisa,Pisa,IT. Ronald Fagin,Ravi Kumar,Mohammad Mahdiany, D.Sivakumar,and Erik https://www.wendangku.net/doc/fa11256531.html,paring and ag-gregating rankings with ties.In Proceedings of ACM In-ternational Conference on Principles of Database Sys-tems(PODS’04),pages47–58,Paris,FR.

Sanda M.Harabagiu,George https://www.wendangku.net/doc/fa11256531.html,ler,and Dan I. Moldovan.1999.WordNet2:A morphologically and semantically enhanced resource.In Proceedings of the ACL Workshop on Standardizing Lexical Resources (SIGLEX’99),pages1–8,College Park,US.

Bo Pang and Lillian Lee.2008.Opinion mining and sen-timent analysis.Foundations and Trends in Information Retrieval,2(1/2):1–135.

Peter D.Turney and Michael L.Littman.2003.Measur-ing praise and criticism:Inference of semantic orienta-tion from association.ACM Transactions on Information Systems,21(4):315–346.