arXiv:2004.02202v1 [cs.CL] 5 Apr 2020

Stylistic Dialogue Generation viaInformation-Guided Reinforcement Learning Strategy

Yixuan Su1, Deng Cai2, Yan Wang3, Simon Baker1, Anna Korhonen1, Nigel Collier1, and Xiaojiang Liu3

1University of Cambridge2The Chinese University of Hong Kong

3Tencent AI Lab{ys484,sb895,alk23,nhc30}@cam.ac.uk, [email protected]

{brandenwang,kieranliu}@tencent.com

Abstract

Stylistic response generation is crucial forbuilding an engaging dialogue system for in-dustrial use. While it has attracted much re-search interest, existing methods often gener-ate stylistic responses at the cost of the con-tent quality (relevance and fluency). To enablebetter balance between the content quality andthe style, we introduce a new training strategy,know as Information-Guided ReinforcementLearning (IG-RL). In IG-RL, a training modelis encouraged to explore stylistic expressionswhile being constrained to maintain its con-tent quality. This is achieved by adoptingreinforcement learning strategy with statisti-cal style information guidance for quality-preserving explorations. Experiments on twodatasets show that the proposed approach out-performs several strong baselines in terms ofthe overall response performance.

1 Introduction

Most early research on dialogue response genera-tion focused on generating grammatically correctand contextually relevant responses (Ritter et al.,2011; Chen et al., 2017; Martinovsky and Traum,2003). While good performance has been achieved(Wen et al., 2016; Wang et al., 2016), syntacticallycoherent responses alone do not guarantee an en-gaging and attractive chatbot. In practice, from anindustrial point of view, we found that if a chatbotcould possess certain language style that is con-sistent with his/her basic character (male, female,optimistic, humorous), the users’ satisfaction andaverage rounds of interaction can be notably im-proved (Song et al., 2019a).

While the definition of language style can bespecified in different contexts (Roberts, 2003;Bell, 1984; Bell and Johnson, 1997; Niederhofferand Pennebaker, 2002; Traugott, 1975), our workrefers to language style as any characteristic style

of expression, from a purely computational stand-point. For example, gender preference can be re-garded as one kind of language style. Consideringa conversation context “Let’s go out of town to re-lax this weekend!”, it is good for a chatbot withmale preference to respond like “That’s great bro.I will go with my buddies together!” and with fe-male preference to respond like “That’s so sweetof you. I will bring my besties!”. Besides genderpreference, our work is also in line with previouswork on dialogue generation with emotion (Zhouand Wang, 2018; Zhong et al., 2019); response at-titude (Niu and Bansal, 2018), and speaker person-ality (Li et al., 2016b).

The majority of the existing approaches forstylistic response generation (Huang et al., 2018;Zhou et al., 2018; Li et al., 2016b; Zhou and Wang,2018; Zhong et al., 2019; Song et al., 2019b)take the style information as an additional inputto the generation model and maximize the prob-ability of generating a response given the inputquery. However, these methods require large par-allel corpora (consisting of conversation pairs withspecified styles) and often tend to output dull andgeneric responses (Li et al., 2016a). As an alter-native, reinforcement learning (RL) can provide amore efficient way to optimize the style expres-sions contained in the generated responses (Niuand Bansal, 2018). Typically, a style-classifier isadopted as a reward agent to evaluate the stylescore of generated responses and the generationmodel is then optimized to generate responseswith higher scores.

However, the RL framework could overempha-size the expression of style at the cost of responsequality: because during the RL process, the gen-eration model could learn to fool the style-classierusing simple stylistic patterns. We show some ex-amples from our preliminary experiments in Ta-ble 1. As observed, the RL-based approach first

arX

iv:2

004.

0220

2v1

[cs

.CL

] 5

Apr

202

0

Input Query: What did you do in the morning?Gender Style: FemaleVanilla Seq2seq: In my old ways.Memory Networks: My husband is very handsome.RL: I went to the school. I like him. I like him.Desired Response : I had my breakfast with my boyfriend.

Table 1: Examples of response with female gender style

generates generic-style text and then appends asimple phrase “I like him” to express a femalestyle, as this phrase receives a high score fromthe female style classifier. Such tricks bring seem-ingly high style score but significantly harm thecontent quality (relevance and fluency). A satis-factory stylistic response should express the de-sired style on the premise of maintaining a highresponse quality, as with the last row of Table 1.

To address this, we propose a new information-guided reinforcement learning (IG-RL) strategy tobetter balance the trade-off between the stylisticexpression and the content quality. Our key idea isto restrict the exploration space of the generationmodel during training, preventing it from collaps-ing to some trivial solutions. Specifically, we sep-arate the vocabulary into two sets, stylistic wordsand neutral words, according to the point-wisemutual information (PMI) (Church and Hanks,1990) between words and styles. At the trainingstage, given the reference response, the model isconstrained to maintain the tokens at the positionsof neutral words. On the other hand, at the po-sitions of stylistic words, the model is allowedto freely explore the entire vocabulary space tosearch for words that maximize the reward of thestyle-classifier and are coherent to the surroundingcontext. In this way, the generation model learns togenerate possible stylistic expressions while main-taining a high response quality. 1

To facilitate future research in this area, we in-troduce a new large-scale gender-specific dialoguedataset. Experimental results on this new datasetand another public benchmark dataset demonstratethat the proposed approach fosters dialogue re-sponses that are both stylistic and high in quality. Itoutperforms standard RL models and other strongbaselines in terms of the overall response quality.

In summary, the contributions of this work are:(i) A novel training strategy to train a model togenerate stylistic responses under the reinforce-

1Please note that the PMI is required during training only.During inference, it generates stylistic responses without anyexternal signals.

ment learning paradigm. This strategy can prop-erly balance the trade-off between the style expres-sion and the content quality via an information-guided learning process. Human evaluation showsthat the proposed approach can generate responseswith both high content quality and desired styles.It significantly outperforms existing methods. (ii)A new gender-specific dialogue dataset whichcontains over 4.5 million query-response pairs. Toour best knowledge, this dataset is the first one fo-cusing on gender-specific dialogue generation andcan greatly facilitate further work in this area.

2 Related Work

Stylistic dialogue response generation has beenan active research area in recent years. Li et al.(2016b) proposed a model that represents person-ality as embeddings and incorporate it into a de-coder of a seq2seq model. Huang et al. (2018)appended the emotion embeddings into decoderstates to generate responses with desired emotions.Zhong et al. (2019) proposed to embed VAD in-formation into words to control the generation ofemotional responses. Zhou et al. (2018) used emo-tion category embedding, internal emotion stateand external emotion memory for emotional dia-logue generation. However, explicitly incorporat-ing style information into the model configurationmay significantly bias the generation process andcause a drastic drop in the response quality.

For RL-based methods, Niu and Bansal (2018)train an attitude classifier as the reward agent toguide the learning process. However, due to thenature of unconstrained sampling, the dialogueagent could learn a simple behaviour that ex-presses the desired style. As a result, little knowl-edge is actually learned by the model during thetraining stage which further undermines the qual-ity of the generated responses.

It should be noted that the stylistic dialoguegeneration is different from the task of text styletransfer. Text style transfer aims to rewrite the in-put sentences such that they possess certain styles,while rigorously preserving their semantic content(Jin et al., 2019). On the other hand, stylistic dia-logue generation does not aim to preserve the se-mantic meaning of the input sentences. Instead, itaims to generate responses that are adequate to theinput query, while expressing pre-specified styles.

Figure 1: Unconstrained Sampling: by generating amale-style-phrase at the end of response, the generatorcan easily acquire high scores from the reward agent.

3 Background

RL-based systems (Niu and Bansal, 2018) firsttrain a style-classifier on an annotated dataset asa reward agent. In the training stage, the dialoguegeneration model generates a response and ob-serves a style score from the reward agent. Theparameters of the generation model are optimizedto maximize the expected style score.

The learning objective is typically defined as:

L(θ) = −EY∼pθ [rs′(Y − b)],

where pθ is the policy (probability distribution) de-fined by the model parameters θ and s′ is the de-sired style, rs′ is the score of the desired style pro-vided by the style-classifier. Y = (y1, ..., yN ) isthe sampled response and yt is the token sampledat time step t. The baseline b is used to reduce thevariance in the training process.

Typically, techniques like Monte-Carlo or top-k sampling (Paulus et al., 2018) are used to gen-erate response Y in training. We refer to theseapproaches as unconstrained sampling, since thegenerated response is solely drawn from the distri-bution pθ that is defined by the model parameters.Therefore, the model is allowed to freely explorethe entire vocabulary to learn a policy (probabilitydistribution) that optimizes the predefined reward.

However, conducting efficient exploration isvery hard for the unconstrained sampling since thesearch space is exponentially large, so only fre-quent patterns which match the reward functionare reinforced as shown in Table 1. Another ex-ample is provided in Figure 1. When learning togenerate male responses, the model learns a sim-ple mechanism that generates a typical male-style-phrase “I am a man” at the end of the response to

“cheat” the reward agent and thus acquire a highmale score. Obviously, the learned policy is notideal since little knowledge other than the simplebehaviour is actually learned by the model, and thegenerated responses can hardly satisfy the users.

4 Information-Guided ReinforcementLearning

In stylistic dialogue generation task, the trainingdata can be formulated as (X ,Y,S), where Xis the set of input queries, Y is the set of re-sponses and S is the set of all possible styles. Eachdata instance follows the format of (X,Y, s),where X = (x1, ..., xT ) is the input query, Y =(y1, ..., yN ) is the reference response and s ∈ S isthe style of the reference response Y.

To address the problem of unconstrained sam-pling introduced in §3, we propose a new train-ing strategy which uses PMI information to guidethe training of the generation model under the re-inforcement learning framework. As illustrated inFigure 2, during training, stylistic and stylelesswords are first identified according to the PMI in-formation. Then the model is learned to generatewords same with the reference response (“My”,“and”, “friends” in Figure 2) at the positions ofstyleless words, and set free to explore stylisticexpressions (“wife”, “her” in Figure 2) to maxi-mize the expected style score at the positions ofstylistic words. Finally, the model parameters areupdated via the REINFORCE algorithm (Suttonet al., 1999). During inference, the model directlygenerates stylistic responses without any externalsignals. We denote the proposed training strategyas Information-Guided Reinforcement Learning(IG-RL) since its training policy is guided by someexternal information other than sampling in the en-tire action space (the entire vocabulary set) in anunconstrained manner.

4.1 Stylistic Words Indication

To indicate whether a token x is stylistic or notgiven the style s, we use the pointwise mutual in-formation (PMI) (Church and Hanks, 1990) whichis defined as

PMI(x; s) = logp(x, s)

p(x)p(s),

where p(x, s) is the frequency that the word x ap-pears in a response with style s in the training cor-pus. We define a word x is stylistic given the style

Figure 2: Framework Overview of Information-Guided Reinforcement Learning

s if PMI(x, s) ≥ ts. In the experiments, we empir-ically set ts = 3

4 ×maxv∈V PMI(v; s), where V isthe whole vocabulary set.

4.2 Constrained SamplingDuring the RL training stage, we impose dy-namic constraints on the sampled response whichis then propagated to the pre-trained classifier toacquire a reward signal. Given a reference re-sponse Y = (y1, ..., yN ), PMI between the tokensand the styles are adopted to determine which to-kens are stylistic. At sampling step t, if yt is neu-tral (styleless), then the model is constrained andonly permitted to sample yt. Otherwise, the modelwill be permitted to freely sample a new tokenfrom the entire vocabulary set.

The neutral words in the sampled response con-struct a neutral training skeleton that is closely re-lated to the query. Based on this training skele-ton, the model learns to express the desired styleby sampling at the positions selected by PMI. Anillustration can be found in the right part of Fig-ure 2, where the model learns to generate maleresponses. In this example, in reference response“My husband and his friends”, “husband” and“his” are denoted as stylistic words. By maskingthese stylistic words, a neutral training skeleton“My and friends” is constructed. The modelis only permitted to sampling new words at themasked positions, and the desired response is “Mywife and her friends” which has high content qual-ity and expresses the desired style (male).

The detailed description of the proposed ap-proach is presented in Algorithm 1, where ts is thestyle-specific threshold as described in §4.1.

Algorithm 1 Constrained Sampling

Input: Input query X = (x1, ..., xT ); Referenceresponse Y = (y1, ..., yN ); Reference re-sponse style s;

Output: Constrained sampling trajectory Y1: Y ← (〈Start of Sentence〉)2: for i = 1 to N do3: if PMI(yi; s) ≥ ts then4: Sample yi ∼ pθ(·|Y; X)5: else6: yi ← yi7: end if8: Y ← Y ∪ yi9: end for

4.3 OptimizationGiven a sampling trajectory Y, based on the RE-INFORCE algorithm, the learning objective is de-scribed as

LRL(θ) ≈ −(rs′(Y)− b) log pθ(Y)

= −(rs′(Y)− b)∑yi∈Y

log pθ(yi|y1, ..., yi−1; X),

where rs′ is the score of the desired style s′, Thebaseline b is used to reduce the variance during thetraining process and X is the input query.

To optimize this objective, Y should satisfyboth the reward agent and the conditional lan-guage model. Since the sampling process is depen-dent on a neutral skeleton, the model has to learnto sample words that not only express the desiredstyle but are also compatible with its context (thesurrounding neutral skeleton).

To stabilize the training process, we incorpo-rate a standard Maximum Likelihood Estimation

Table 2: Gender-specific Dialogue Dataset summary

(MLE) objective. Given the input query X and thereference response Y, the objective is defined as

LMLE(θ) = −N∑i=1

log pθ(yi|y1, ..., yi−1; X).

In addition, the MLE objective tends to train amodel that is overfit to the training set (Pereyraet al., 2017), therefore the model is less willinglyto explore other possibilities during the RL train-ing process. To mitigate this side effect, we uselabel smoothing (Szegedy et al., 2016) as an auxil-iary regularization. Instead of using a uniform dis-tribution over all words in the vocabulary as target,we introduce a new form of target distribution. Inwhich case, we use the bigram frequency distri-bution that acquired from the training corpus astarget and the detailed computation is shown as

Lsmo(θ) =

−N∑i=2

∑v∈V

f(yi−1, v) log pθ(v|y1, ..., yi−1; X),

f(yi−1, v) =#(yi−1, v)∑

v?∈V #(yi−1, v?),

where #(yi−1, v) is the bigram count of tokenyi−1 and v in the training corpus.

The final learning objective is defined as

Lhybrid(θ) = LMLE(θ) + αLsmo(θ) + βLRL(θ),

where α and β are weights of different parts.

5 Experiments

5.1 DatasetsTo facilitate future research in this area, we con-structed a gender-specific dialogue dataset. At thefirst step, based on the gender information ofusers, we collected 100,000 query-response pairswhose responses generated by female users fromDouban2. Similarly, we also collected 100,000query-response pairs from male users.

Then we hired 6 professional annotators (3 ofthem are female and others are male) to further2https://www.douban.com

verify the collected results. Because there is norigorous guideline on how to quantify the genderpreference expressed in daily life conversation, welet the annotators to judge the results based ontheir own understanding. The annotators are askedto assign a female(male) label to the response ifit is very unlikely uttered by a male(female) user.Otherwise, a neutral label will be assigned. To en-sure a reasonable annotation, each response is an-notated by all six annotators, and we only keep thepairs whose response label is agreed by at least fiveannotators.

From the 200,000 collected pairs, 5,184 re-sponses are annotated as male instances and10,710 responses are annotated as female in-stances. To keep data statistic balanced, we ran-domly select 15,000 neutral instances to build thehigh-quality gender classification dataset. Thenwe fine-tuned a Chinese BERT (Devlin et al.,2019) on the constructed dataset to build a gender-classifier. And the classification accuracy is about91.7%. We further use this classifier to automat-ically annotate the STC-Sefun dataset (Bi et al.,2019) to obtain a big gender-specific dialoguedataset3 whose data statistic is shown in Table 2.

For a comprehensive evaluation, in addition tothe proposed gender-specific dialogue dataset, wealso conduct experiments on a public emotionaldialogue dataset (Zhou et al., 2018).

5.2 Implementation Details

The proposed model is implemented using Py-torch (Paszke et al., 2017). We use two-layerLSTMs with 500 hidden units to construct the en-coder and decoder of the generation model. Theword embedding size is set to 300 and it is ran-domly initialized. The vocabulary size is limitedto 15,000.

We use Adam (Kingma and Ba, 2015) to opti-mize our model with a batch size of 64 and learn-ing rate of 1e-3. For all experiments, we first pre-train a seq2seq model with the MLE objective for3 epoches on the training set. Then the learned pa-rameters are used to initialize the policy networks.We set the reference reward b and α, β in the learn-ing objective as 0.3, 0.2, and 0.25 respectively.Similar to recent works (Fan et al., 2018; Qin et al.,2019), we use top-k sampling during the inferencestage with k set to 20.

3The released version of the proposed dataset can be foundhere: https://ai.tencent.com/ailab/nlp/dialogue/#datasets

Style MetricBaselines Ours

Seq2seq Speaker ECM Polite-RL w/o G IG-RL

FemaleQuality↑ 3.24† 1.45 2.22 2.33 2.96† 2.99

Style Expression↑ 3.03 3.26 3.19 3.63† 3.60† 3.64Ranking↓ 2.77 3.56 3.20 2.81 2.01 1.75

MaleQuality↑ 3.13† 1.41 1.97 2.31 2.93 3.02

Style Expression↑ 2.99 3.56 3.49 4.58† 3.25 4.03Ranking↓ 2.94 3.75 3.42 2.13 2.71 1.72

Overall

Quality↑ 3.19† 1.43 2.10 2.32 2.94 3.01Style Expression↑ 3.01 3.41 3.34 4.11† 3.43 3.84

Ranking↓ 2.86 3.66 3.31 2.47 2.36 1.73Distinct-1(%)↑ 19.73 15.95 13.77 13.22 20.23 24.92Distinct-2(%)↑ 67.81 61.05 59.72 55.02 70.44 74.83

Table 3: Evaluation Results on Gender-Specific Dialogue Generation: ↑ means the higher the better and ↓ meansthe lower the better, bold font denotes the best scores for each metric. Sign tests on evaluation scores show that theproposed model significantly outperforms other models with p-value < 0.05 with the only exception marked by †.

Style MetricBaselines Ours

Seq2seq Speaker ECM Polite-RL w/o G IG-RL

LikeQuality↑ 3.13† 2.41 2.38 2.48 3.01† 2.96

Style Expression↑ 2.90 3.72† 3.78† 4.42† 3.27 3.66Ranking↓ 3.35 3.14 3.24 2.51 2.59 2.14

DisgustQuality↑ 3.03† 2.24 2.15 2.29 2.95† 2.65

Style Expression↑ 2.82 3.58 3.81 4.59 2.94 3.72Ranking↓ 3.26 3.49 3.39 2.33 2.68 2.24

HappinessQuality↑ 3.08 2.51 2.39 2.69 2.93 3.27


AngerQuality↑ 3.24† 2.43 2.15 2.06 2.99† 2.84


SadnessQuality↑ 3.00† 2.10 2.04 2.24 2.77† 2.71


OverallQuality↑ 3.09† 2.34 2.22 2.35 2.93† 2.89


Distinct-1(%)↑ 17.41 15.69 13.65 11.50 19.59† 20.46Distinct-2(%)↑ 65.07 58.39 52.19 51.12 67.34 73.37

Table 4: Evaluation Results on Emotion-Specific Dialogue Generation

5.3 Compared Models

We compare the proposed approach with severalrepresentative and competitive baselines.

5.3.1 BaselinesSeq2seq: Standard sequence-to-sequence modelwith attention mechanism (Luong et al., 2015).

Speaker: Model proposed by Li et al. (2016b)which incorporates distributed style embeddingsinto the structure of decoding cell to control thegeneration process.

ECM: Model proposed by Zhou et al. (2018)which adopts internal and external memory mod-ules to control the generation of stylistic expres-sions in the generated responses.

Polite-RL: Approach proposed by Niu andBansal (2018) which leverages RL to teach themodel to generate stylistic responses. In our ex-periments, we also use BERT as the reward agent.

5.3.2 Ablation StudyIG-RL: The full model proposed in this work.For a fair comparison, we construct our genera-tor using the same structure as the one in Niu andBansal (2018).

w/o G: In the ablated model, we examine howthe guidance provided by PMI knowledge effectsthe model’s performance. To this end, in the RLtraining stage, instead of using PMI to guide thesampling process, we let the model sample a tokenwith a equal probability of 0.2 or just simply copythe corresponding token in the reference response.

5.4 Evaluation Metrics

The quality of the responses generated by a dia-logue system is well known to be difficult to mea-sure automatically (Deriu et al., 2019); thereforewe rely on human evaluation. In our experiments,each response is judged by five independent anno-tators who are hired from a commercial company.

To prevent possible bias from the annotators, allresults are randomly shuffled before being evalu-ated and are evaluated following metrics below.

Quality: This metric evaluates the content qual-ity of the generated responses. The annotators areasked to give a score within 5-point scale where5 means perfectly human-like response (relevant,fluent and informative), and 1 means unreadable.

Style Expression: This metric measures howwell the generated responses express the desiredstyle. The annotators give a score within 5-pointscale, where 5 means very strong style, 3 meansneutral or no obvious style and 1 means very con-flicted style. The style conflict means the generatedstyle is conflicted to the desired one (e.g. female tomale, positive to negative emotion).

Ranking: The annotators are further asked tojointly evaluate the content quality and the styleexpression of the generated responses from differ-ent approaches. Then the annotators give a rankingto each result where top 1 means the best.

We measure the agreement of our annotatorsusing Fleiss′ kappa (Fleiss et al., 1971): for thegender-specific dialogue generation, the resultsof Quality and Style Expression are 0.442 and0.654, which indicate “moderate agreement” and“substantial agreement” respectively. As for emo-tional dialogue generation, the results are 0.432and 0.628, which indicate “moderate agreement”and “substantial agreement” respectively.

5.5 Main Results

The evaluation results are shown in Tables 3 and 4in which we also present the averaged evaluationscores among different styles.

From the results, we can see that the IG-RLmethod achieves top two performances on boththe quality metric and the style metric for bothdatasets. Compared to other methods, it ensuresboth high quality and desired stylistic expres-sion. For the ranking metric which jointly evalu-ates both the content quality and the style expres-sion, the proposed approach outperforms all otherbaselines by a substantial margin. In addition, wealso measure the diversity of the generated re-sponses with two automatic metrics: Distinct-1and Distinct-2 (Li et al., 2016b), and the resultsshow that the IG-RL method generates the mostdiverse responses among all methods.

Figure 3: Style Acceptance Evaluation

It can be observed that Polite-RL generally ob-tains the highest style expression score but getsmuch lower performance on the quality and theranking metric comparing to the proposed ap-proach. This confirms our early analysis that thevanilla RL methods may achieve high style inten-sity at the cost of the content quality.

The performance on individual styles also pro-vides some insights. In the happiness style, theproposed approach achieves the highest scores onall three metrics. The reason is that the happinessstyle is similar to the neutral style and we have rel-atively sufficient data. A similar phenomenon canalso be found in the female style response. There-fore, we can conclude that for those styles withsufficient data, the proposed IG-RL can achievehigh performance on both of the quality and styleaspects. On the other hand, when stylistic data islimited, it also maintains a well balance betweenthe response quality and the style expression.

5.6 Further Analysis

Here, we present further discussion and empiricalanalysis of the proposed method.

5.6.1 Style AcceptanceA fundamental requirement of a stylistic dialoguesystem is to generate responses that are not con-flicted with respect to the desired style. For in-stance, generating male style responses is notacceptable for a female chatbot; likewise, for apositive emotional chatbot (e.g. like, happiness),generating negative responses (e.g. disgust, sad-ness and anger) is not acceptable. To quantita-tively evaluate how acceptable a stylistic dialoguesystem is, we propose two novel metrics: hu-man style acceptance rate (H-SAR) and, automaticstyle acceptance rate (A-SAR). We compute H-SAR based on the style expression scores in hu-man evaluation. It is defined as the ratio of gener-ated results whose style expression score is greateror equal to 3. As for A-SAR, we use the pre-trained style-classifier to compute the ratio of the

Table 5: Sample responses generated by different approaches, the input query does not appear in the training set.

Figure 4: Balance between Quality and Style: The≥ 3-ratio means the ratio of responses whose both scoresare greater or equal to 3; ≥ 4-ratio means the ratio ofresponses whose both scores are greater or equal to 4.

generated responses that display a style which isnot conflicted to the desired one.

The results are shown in Figure 3 and we cansee that H-SAR and A-SAR are highly correlated.Considering results in Table 3 and 4, although theproposed approach does not generate responseswith the highest style expression score, it is theonly system which achieves the best H-SAR andA-SAR performances, suggesting that our systemis more robust than others since it makes fewerstyle conflict mistakes.

5.6.2 Balance between Quality and StyleA satisfactory stylistic dialogue system should ex-press the desired style while maintaining the con-tent quality. Based on the human evaluation met-ric, 3 is the marginal score of acceptance. So wedeem a response as marginally acceptable by ac-tual users when both the quality and style expres-sion scores are greater or equal to 3. On the otherhand, 4 is the score that well satisfies the users, soresponses with both scores greater or equal to 4are deemed as satisfying to actual users.

The ratios of both scores≥ 3 and≥ 4 are shownin Figure 4, from which we see that our systemoutperforms all other systems on ≥ 3-ratio and≥ 4-ratio. Obviously, the proposed IG-RL bestbalances the trade-off between the response qual-ity and the style expression and therefore generat-ing the most acceptable and satisfying responses.

5.6.3 Ablation StudyWe analyze the effect of removing guidance pro-vided by the PMI signal. Comparing the ablatedmodel (w/o G) with our full model (IG-RL), fromresults in Tables 3 and 4, we can observe thatthe quality score is slightly influenced, but thestyle expression score drops significantly. Thisdemonstrates that although utilizing reference re-sponse helps in maintaining the response quality,the guidance provided by PMI information is in-dispensable for generating stylistic responses.

5.6.4 Case StudyWe use an input query that is unseen in bothdatasets to generate responses with different stylesusing different systems (example responses pre-sented in Table 5). Due to limited space, we onlycompare different approaches with respect to gen-ders. As for emotions, we present the results ofIG-RL only.

We can see that although the result fromSeq2seq approach is relevant but it is conflictto female style. As for other compared methods,the memory networks-based approaches (Speaker,ECM) can generate responses with the desiredstyle but not very relevant to the input query. For

Polite-RL approach, it first generates part of theresponse that relates to the input query and thensimply generates a phrase which expresses in-tense desired style (e.g. “I like him” in female re-sponses). Given all genders and emotions, only theresponses generated by IG-RL generally maintainhigh content quality and properly express the de-sired style.

6 Conclusion

We have proposed a new training strategy thatleverages stylistic information as guidance to con-duct a quality-preserving learning process. To fa-cilitate future research, we have also constructedand annotated a new dataset for gender-specificdialogue generation. Our experimental resultsdemonstrate that the proposed IG-RL approachoutperforms existing baselines in terms of theoverall response performance.

ReferencesAllan Bell. 1984. Language style as audience design.

Language in Society, 13(2):145204.

Allan Bell and Gary Johnson. 1997. Towards a so-ciolinguistics of style. University of PennsylvaniaWorking Papers in Linguistics, 4(1):2.

Wei Bi, Jun Gao, Xiaojiang Liu, and Shuming Shi.2019. Fine-grained sentence functions for short-textconversation. In ACL 2019, Florence, Italy, July28- August 2, 2019, Volume 1: Long Papers, pages3984–3993.

Hongshen Chen, Xiaorui Liu, Dawei Yin, and JiliangTang. 2017. A survey on dialogue systems: Recentadvances and new frontiers. SIGKDD Explorations,19(2):25–35.

Kenneth Ward Church and Patrick Hanks. 1990. Wordassociation norms, mutual information, and lexicog-raphy. Computational linguistics, 16(1):22–29.

Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, GuillermoEchegoyen, Sophie Rosset, Eneko Agirre, and MarkCieliebak. 2019. Survey on evaluation methods fordialogue systems. CoRR, abs/1905.04071.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: pre-training ofdeep bidirectional transformers for language under-standing. In NAACL-HLT 2019, Minneapolis, MN,USA, June 2-7, 2019, Volume 1 (Long and Short Pa-pers), pages 4171–4186.

Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018.Hierarchical neural story generation. In ACL 2018,Melbourne, Australia, July 15-20, 2018, Volume 1:Long Papers, pages 889–898.

J.L. Fleiss et al. 1971. Measuring nominal scale agree-ment among many raters. Psychological Bulletin,76(5):378–382.

Chenyang Huang, Osmar R. Zaıane, Amine Trabelsi,and Nouha Dziri. 2018. Automatic dialogue genera-tion with expressed emotions. In NAACL-HLT 2018,New Orleans, Louisiana, USA, June 1-6, 2018, Vol-ume 2 (Short Papers), pages 49–54.

Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews,and Enrico Santus. 2019. Unsupervised textstyle transfer via iterative matching and translation.CoRR, abs/1901.11333.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In ICLR 2015,San Diego, CA, USA, May 7-9, 2015, ConferenceTrack Proceedings.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,and Bill Dolan. 2016a. A diversity-promotingobjective function for neural conversation models.In NAACL-HLT 2016, San Diego California, USA,June 12-17, 2016, pages 110–119.

Jiwei Li, Michel Galley, Chris Brockett, Georgios P.Spithourakis, Jianfeng Gao, and William B. Dolan.2016b. A persona-based neural conversation model.In ACL 2016, August 7-12, 2016, Berlin, Germany,Volume 1: Long Papers.

Thang Luong, Hieu Pham, and Christopher D. Man-ning. 2015. Effective approaches to attention-basedneural machine translation. In EMNLP 2015, Lis-bon, Portugal, September 17-21, 2015, pages 1412–1421.

Bilyana Martinovsky and David Traum. 2003.2003.the error is the clue: Breakdown in human-machine interaction. In In Proceedings of the ISCAWorkshop on Error Handling in Dialogue Systems.

Kate Niederhoffer and James Pennebaker. 2002. Lin-guistic style matching in social interaction. Journalof Language and Social Psychology, 21:337–360.

Tong Niu and Mohit Bansal. 2018. Polite dialogue gen-eration without parallel data. TACL, 6:373–389.

Adam Paszke, Sam Gross, Soumith Chintala, Gre-gory Chanan, Edward Yang, Zachary DeVito, Zem-ing Lin, Alban Desmaison, Luca Antiga, and AdamLerer. 2017. Automatic differentiation in pytorch.In NIPS-W.

Romain Paulus, Caiming Xiong, and Richard Socher.2018. A deep reinforced model for abstractive sum-marization. In ICLR 2018, Vancouver, BC, Canada,April 30 - May 3, 2018, Conference Track Proceed-ings.

Gabriel Pereyra, George Tucker, Jan Chorowski,Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Reg-ularizing neural networks by penalizing confidentoutput distributions.

Lianhui Qin, Michel Galley, Chris Brockett, XiaodongLiu, Xiang Gao, Bill Dolan, Yejin Choi, and Jian-feng Gao. 2019. Conversing by reading: Contentfulneural conversation with on-demand machine read-ing. In ACL 2019, Florence, Italy, July 28- August2, 2019, Volume 1: Long Papers, pages 5427–5436.

Alan Ritter, Colin Cherry, and William B. Dolan. 2011.Data-driven response generation in social media. InEMNLP 2011, 27-31 July 2011, John McIntyre Con-ference Centre, Edinburgh, UK, pages 583–593.

Julie Roberts. 2003. Style and sociolinguistic varia-tion. American Anthropologist, 105.

Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, andXuanjing Huang. 2019a. Generating responses witha specific emotion in dialog. In Proceedings of the57th Conference of the Association for Computa-tional Linguistics, ACL 2019, Florence, Italy, July28- August 2, 2019, Volume 1: Long Papers, pages3685–3695.

Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, andXuanjing Huang. 2019b. Generating responses witha specific emotion in dialog. In Proceedings of the57th Annual Meeting of the Association for Compu-tational Linguistics.

Richard S. Sutton, David A. McAllester, Satinder P.Singh, and Yishay Mansour. 1999. Policy gradi-ent methods for reinforcement learning with func-tion approximation. In Advances in Neural Infor-mation Processing Systems 12, [NIPS Conference,Denver, Colorado, USA, November 29 - December4, 1999], pages 1057–1063.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,Jonathon Shlens, and Zbigniew Wojna. 2016. Re-thinking the inception architecture for computer vi-sion. In CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818–2826.

Elizabeth Closs Traugott. 1975. William labov, soci-olinguistic patterns. (conduct and communication,4.) philadelphia: University of pennsylvania press,1972. Language in Society, 4(1):89107.

Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, HangLi, and Qun Liu. 2016. Dropped pronoun generationfor dialogue machine translation. In ICASSP 2016,Shanghai, China, March 20-25, 2016, pages 6110–6114.

Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic,Lina Maria Rojas-Barahona, Pei-Hao Su, StefanUltes, David Vandyke, and Steve J. Young. 2016.Conditional generation and snapshot learning inneural dialogue systems. In EMNLP 2016, Austin,Texas, USA, November 1-4, 2016, pages 2153–2162.

Peixiang Zhong, Di Wang, and Chunyan Miao. 2019.An affect-rich neural conversational model with bi-ased attention and weighted cross-entropy loss. InAAAI 2019, Honolulu, Hawaii, USA, January 27 -February 1, 2019., pages 7492–7500.

Hao Zhou, Minlie Huang, Tianyang Zhang, XiaoyanZhu, and Bing Liu. 2018. Emotional chatting ma-chine: Emotional conversation generation with in-ternal and external memory. In AAAI-18, New Or-leans, Louisiana, USA, February 2-7, 2018, pages730–739.

Xianda Zhou and William Yang Wang. 2018. Mojitalk:Generating emotional responses at scale. In ACL2018, Melbourne, Australia, July 15-20, 2018, Vol-ume 1: Long Papers, pages 1128–1137.

arXiv:2004.02202v1 [cs.CL] 5 Apr 2020

Documents