BART with lexical & syntactic constraints for slogan generation šŸ“Œ

Published May 8, 2021 by Yeoun Yi
#advertising #nlp #generation #languagemodel


I tried to make a model that generates fancy advertising slogans in this post , and it turns out the loss is not decreasing any more!! I gave up with that model and used pretrained BART instead. BART was already pretrained with a task named Mask Infilling, which was perfect for my task. šŸ˜ Full code at github.

1. Original BART with lexical constraints

BART can be used for Masked Language Model, but <mask> token doesnā€™t need to be corresponding only to a single token. It can be corresponding to multiple tokens or zero token. So, if I give BART a input sequence of 2 keywords that I want BART to make a slogan with, <mask> keyword1 <mask> keyword2 <mask>, it can generate a sentence containing these 2 keywords.

from transformers import BartTokenizer, BartForConditionalGeneration

tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
model =  BartForConditionalGeneration.from_pretrained("facebook/bart-base")

inputs = tokenizer('<mask> bake <mask> cake <mask>', return_tensors='pt')
outputs = model.generate(**inputs)[0]

tokenizer.decode(outputs, skip_special_tokens=True)
# >> 'How to bake a cake?'

I fine-tuned BART with 30,759 slogans I crawled and it gives me 'i bake a chocolate cake from scratch every week.' Not bad. But I want the slogan to have a repeated syntactic structure as well, to highlight the rhyme between 'bake' and 'cake'.

2. Custom BART with syntactic constraints

I added Syntactic Encoder to original BART structure to fuse the syntactic information that desired slogans need to have.


The Syntactic Encoder has the same architecure with BART Encoder, but didnā€™t share the embedding with Semantic Encoder and Decoder. But the Semantic Encoder and Decoder shared their embeddings.

I used Spacy POS Tagger to get the golden syntactic structure of certain slogan, extracted keywords, and used these information as inputs. The model was trained to generate the original slogan, given these syntactic and lexical information.

The outputs of Syntactic Encoder and that of Semantic Encoder have different lengths, so I repeated the Syntactic Encoderā€™s last hidden state of <s> for the length of Semantic Encoderā€™s output, then added them. This sum was given to the Decoder.

The result was GREAT!

from transformers import BartTokenizer
from SynSemBartForConditionalGeneration import SynSemBartForConditionalGeneration

tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
tokenizer.add_tokens('<name>')
tokenizer.add_tokens('<loc>')
tokenizer.add_tokens('<year>')

pos_tokenizer = BartTokenizer(vocab_file='/home/yeoun/BART/POSvocab.json', merges_file='/home/yeoun/BART/merges.txt')

model =  SynSemBartForConditionalGeneration.from_pretrained("/home/yeoun/BART/SynSemBart")
model.resize_token_embeddings(len(tokenizer))

mask_seq = '<mask> bake <mask> cake <mask>'
pos = ['VERB', 'DET', 'NOUN', 'PUNCT', 'VERB', 'DET', 'NOUN']

inputs = tokenizer(mask_seq, return_tensors="pt")
pos_inputs = pos_tokenizer.encode_plus(pos, is_pretokenized=True, return_tensors='pt')
outputs = model.generate(input_ids = inputs.input_ids, attention_mask = inputs.attention_mask,
                         pos_input_ids = pos_inputs.input_ids, pos_attention_mask = pos_inputs.attention_mask)[0]

tokenizer.decode(outputs, skip_special_tokens=True)
# >> 'Bake the cake, Get the break.'

'Bake the cake, Get the break.'
Its generation contained both of the keywords I gave to the Lexical Encoder, which are bake and cake, and have the exact syntactic structure that I gave to the Syntactic Encoder, which is ['VERB', 'DET', 'NOUN', 'PUNCT', 'VERB', 'DET', 'NOUN'].

Find more about the data or model in this paper.



*****

© Yeoun Yi