[email protected] [email protected] Derik ...

Graph Relation Transformer: Incorporating pairwise object features into theTransformer architecture

Michael [email protected]

Aditya [email protected]

Zachary [email protected]

Derik Clive [email protected]

Carnegie Mellon UniversityPittsburgh, USA

Abstract

Previous studies such as VizWiz find that Visual QuestionAnswering (VQA) systems that can read and reason abouttext in images are useful in application areas such as as-sisting visually-impaired people. TextVQA is a VQA datasetgeared towards this problem, where the questions requireanswering systems to read and reason about visual objectsand text objects in images. One key challenge in TextVQAis the design of a system that effectively reasons not onlyabout visual and text objects individually, but also aboutthe spatial relationships between these objects. This moti-vates the use of ‘edge features’, that is, information aboutthe relationship between each pair of objects. Some currentTextVQA models address this problem but either only usecategories of relations (rather than edge feature vectors) ordo not use edge features within the Transformer architec-tures. In order to overcome these shortcomings, we proposea Graph Relation Transformer (GRT), which uses edge in-formation in addition to node information for graph atten-tion computation in the Transformer. We find that, withoutusing any other optimizations, the proposed GRT methodoutperforms the accuracy of the M4C baseline model by0.65% on the val set and 0.57% on the test set. Qualita-tively, we observe that the GRT has superior spatial rea-soning ability to M4C.1

1. Introduction

Visual Question Answering (VQA) is the task of answer-ing questions by reasoning over the question and the image

1The code used to obtain our results can be found at https://github.com/michaelzyang/graph-relation-m4c andhttps://github.com/derikclive/transformers

Figure 1: An example from the TextVQA dataset whichshows the importance of reasoning about spatial relation-ships between objects.Q What is the last number to theright? Ans 13.

corresponding to the question. Although VQA models haveshown a lot of improvement in recent years, these mod-els still struggle to answer questions which require readingand reasoning about the text in the image. This is an im-portant problem to solve because studies have shown thatvisually-impaired people frequently ask questions which in-volve reading and reasoning about the text in images. The‘TextVQA’ task [20] is a VQA task where the questions arefocused on the text in images as shown in Figure 1. In or-der to answer this question, the model needs to understandthat the visual object ‘13’ is the last object among all ob-jects towards the right of the image. This leads to the intro-duction of an additional modality involving the text in theimages which is often recognized through Optical Charac-ter Recognition (OCR). The incorporation of this additionalmodality enhances the difficulty of this task as compared toa standard VQA task.

1

arX

iv:2

111.

0607

5v1

[cs

.CV

] 1

1 N

ov 2

021

The current state-of-the-art models [5, 12] in this domainuse graph attention paired with multimodal Transformerbased approaches to model relationships between image ob-jects, OCR objects and question tokens. Both approachestry to capture the relationship between these objects withinthe image using rich edge features. However, we see thatthey either use edge features for representation learning be-fore the Transformer layers [5] or do not incorporate richedge features when graph attention computation is done inTransformer layers [12]. These rich edge features can beleveraged within the Transformer layers to allow for betterrepresentations between image and OCR objects.

In this work, we propose a novel Graph Relation Trans-former (GRT) which uses rich, vector-based edge featuresin addition to node information for graph attention compu-tation in the Transformer. The proposed GRT outperformsthe M4C baseline model [10] while also improving the spa-tial reasoning ability of the model. We also provide qual-itative examples of cases where our proposed approachesperforms better than the M4C baseline model [10].

2. Related Work

2.1. VQA

For VQA tasks, the main multimodal challenges are howto represent the visual and language modalities and how tofuse them in order to perform the Question Answering (QA)task. In terms of representing the questions, word embed-dings such as GloVe [17] are commonly used in conjunctionwith recurrent neural networks (RNNs) such as Long Short-Term Memory (LSTM) networks [6], for example by Fukuiet al. [4]. For representing the visual modality, grid-basedConvolutional Neural Networks (CNNs) such as Resnet [9]are often used as visual feature extractors.

For representation, Bottom-up And Top-down Attention(BUTD) [1] is a canonical VQA method. Previous meth-ods used a fixed-size feature map representation of the im-age, extracted by grid-based CNN, whereas BUTD uses aRegion-CNN such as Faster R-CNN [7] to propose a num-ber of variably-sized objects, bottom-up, over which thequestion representation will attend, top-down.

Applying several tweaks to the BUTD model, the Pythiamodel [11] improved its performance further. Pythia re-ported performance gains from ensembling with diversemodel setups and different object detectors, architecture,learning rate schedule and data augmentation and fine-tuning on other datasets.

Bilinear Attention Networks (BANs) [13] aim to im-prove multimodal fusion by allowing for attention betweenevery pairwise combination of words in the question andobjects in the image. This setup is motivated by the com-mon occurrence that different words in the question refer todifferent objects in the image and thus allowing for bilin-

ear interaction improves grounding between the modalities.Low-rank matrix factorization is used to keep the compu-tation cost tractable and the Multimodal Residual Network(MRN) [13] method is used to combine all the representa-tions from the bilinear attention maps.

2.2. TextVQA

Fusion Methods Most of the methods proposed forTextVQA involve the fusion of features from the differentmodalities - Text, OCR and Images. LoRRA [20] and M4C[10] are two such model architectures that primarily use at-tention mechanisms to capture the interactions between theinputs from different modalities. In addition, both of thesemodels employ a pointer network [22] module to directlycopy tokens from the OCR inputs. This enables the modelsto reduce the number of out-of-vocabulary words predictedby supplementing the fixed answer vocabulary with the in-put OCR context.

LoRRA [20] was proposed as the baseline for the 2019TextVQA challenge and is composed of three major com-ponents - one to combine image and question features, an-other to combine OCR and question features and the thirdone to generate the answer. M4C improves the fusion of theinput modalities through the use of Multimodal Transform-ers which allows both inter-modal and intra-modal interac-tions. The multimodal Transformer uses features from allthree modalities and uses a pointer-augmented multi-stepdecoder to generate the answer one word at a time unlikethe LoRRA model that uses a fixed answer vocabulary.

Graph Attention Methods Graph attention networks areanother approach to try to capture the relationships betweenobjects detected within an image. This approach was firstused by [16] to solve the VQA task by detecting objectswithin an image and then treating each of these objects asnodes within a graph. This approach uses several itera-tions of modifying each node’s representation with that ofits neighbors’. This approach produced better results thanexisting methods at the time. Extending this approach tothe TextVQA task, detected OCR tokens were added as ad-ditional nodes within these graphs.

The Stuctured Multimodal Attention (SMA) approach[5] won the 2020 CVPR TextVQA challenge using a varia-tion of the standard graph attention described above. Theyused various attention mechanisms to create one attendedvisual object representation and one attended OCR objectrepresentation, which were then fed into the MultimodalTransformer layers instead of the individual objects. Impor-tantly, these attention mechanisms involve the use of edgefeatures between objects, an idea which we implement dif-ferently in our GRT. A limitation of this work is that usingedge features for representation learning before the Trans-former layers does not leverage the graph computation in-herent to Transformers. It also does not create contextual-

2

ized representations for each object unlike those that can becreated using the Transformer layers.

The Spatially Aware Multimodal Transformers (SAMT)approach was the 2020 CVPR TextVQA runner-up [12].They created a spatial relation label for every pair of ob-jects and restricted each Transformer attention head to at-tend over paired objects with certain labels only. A limi-tation of this work is that there is no natural way to makeuse of rich edge features since masking is based on discreteedge categories, not continuous / vector edge features.

We propose to overcome the limitations of both SMA[5] and SAMT [12] in this work using a novel Graph Re-lation Transformer which uses rich edge information forgraph attention computation in the Transformer. Integratingrich edge information within the Transformer self-attentionlayer helps us overcome the limitation of SMA [5]. Contin-uous space, rich edge features helps us overcome the limi-tation of SAMT [12].

2.3. Graph Networks outside VQA

Graph networks have also been proposed for tasks out-side the VQA domain. Cai et al. [3] propose a graph trans-former for the graph to sequence learning task. They use alearned vector representation to model the relation betweennodes. Yun et al. [24] propose graph transformer networksbased on graph convolution networks for node classifica-tion tasks. Unlike our proposed approach, they do not userich edge features between objects based on the appearanceand position of objects in the image which provide valuableinformation in image-based tasks like the TextVQA task.There have also been efforts to generalize the Transformerarchitecture for arbitrary graphs related tasks. [24] pro-pose modifications to the original transformer architectureby leveraging the inductive bias present in graph topologieswhile also injecting edge information into the architecture.

3. Proposed ApproachIn this section, we motivate and explain GRT, our novel

extension to the M4C model.

3.1. Graph Relation Transformer (GRT)

A common theme among the top performing models[5, 12] is extracting and using graph relations between de-tected objects in images. However, no prior work in theVQA domain has used edge features within the Transformerlayers for representation learning. Here, we will describefour types of edge features with which we will experimentand how they will be used in the Transformer. A key ob-servation of the Transformer Encoder architecture is thatthe self-attention inherently performs graph computationacross each input as if the inputs were structured as a fully-connected graph, as each input attends over all inputs. How-ever, there is no natural place to inject edge relations in the

Transformer Encoder layer. To achieve this, we proposeGraph Relation Self-Attention in Transformers.

Edge features We hypothesize that four types of edgefeatures will be useful for the model to better understandthe relationships between objects. Each of these features aredefined for a pair of objects (which may be from the visualmodality or the OCR modality). These are (1) Appearancesimilarity (2) Spatial translation feature (3) Spatial interac-tion labels (4) Modality pair labels. Figure 2 illustrates howthese are extracted for every pair of visual objects in an im-age.

Appearance similarity may be especially useful for theOCR modality, as OCR tokens in an image that belongtogether in a sentence or paragraph usually have similarfont and overall appearance. We use the cosine similaritybetween the 2,048-dimensional Faster R-CNN embeddingsas the (1-dimensional) appearance similarity feature. Thisedge feature is illustrated in Figure 2a.

The spatial translation feature from object i to object j issimply a 2-dimensional feature that is the translation fromthe center of object i’s bounding box to object j’s in the xand y directions, where these translations are normalized tothe length and width of the object such that this feature’srange is [-1, 1] in both directions. Clearly, this feature pro-vides the model information about the relation between ob-jects in terms of spatial direction. This edge feature is illus-trated in Figure 2b.

Similar to the labels proposed by Kant et al. [12], we use5 mutually exclusive class labels indicating different typesof spatial interaction [is self, is contains, is in, is overlap,not overlap], where is self indicates a self-edge betweenobject i to itself, is contains indicates if object i’s boundingbox completely contains object j’s, is in indicates if objectj’s bounding box completely contains object i’s, is overlapindicates if there is any overlap between the bounding boxesof objects i and j and they do not fall under any of the previ-ous classes and not overlap indicates if there is no overlapbetween the two objects’ bounding boxes.2 These labelsprovide the model information about how objects interactwith each other in space. This edge feature is illustrated inFigure 2c.

Following Gao et al. [5], we also include modal-ity pair labels [is obj to obj, is obj to ocr, is ocr to obj,is ocr to ocr] that indicate the modalities of each pair ofobjects. For example, the is obj to ocr label is positive ifthe ‘self’ object is a visual object recognized by the FasterR-CNN and the ‘other’ object is an OCR token. These la-bels will help the model learn different interactions betweenmodalities. This edge feature is illustrated in Figure 2d.

2We do not include the 8 directional labels used by Kant et al. [12] asthis information should be provided by the previously mentioned spatialtranslation feature.

3

(a) Appearance Similarity edge fea-ture

(b) Spatial Translation edge feature (c) Spatial Interaction Label edgefeature

(d) Modality Pair Label edge fea-ture

Figure 2: Illustration of the four edge features we propose to use for the TextVQA task. For each pair of visual objects i and jin an image, we extract a vector of edge features eij by concatenating these four edge features: appearance similarity(i, j),translation(i, j), interaction(i, j) and modality pair(i, j).

Finally, these features are only meaningful for pairs ofTransformer inputs that originate from the image (the visualand OCR modalities). Therefore, when a question modalityTransformer input is involved, we set the aforementionedfeatures to be zero.

Vanilla Transformer Self-Attention For some object ortoken i, the Transformer [21] self-attention module com-putes the attended representation of each of nobj objects as

Attention(qi,K, V ) = softmax(

qiKT

√dk

)V (1)

where qi, a row vector of dimension dk, is the queryfor object i. K is a matrix composed of nobj stacked rowvectors of dimension dk, representing the keys of the nobjobjects. V is a matrix composed of nobj stacked row vectorsof dimension dv , representing the values of the nobj objects.

The node features matrix X comprises nobj row vectorsof node features of dimension din (visualized in Figure 3).3

We obtain K and V using projection matrices Wk and Wv

to project the nobj rows of X to vector spaces of dimensiondk and dv respectively.

K = XWk (2)

V = XWv (3)

3In the first Transformer layer, X is simply the input features, whereasin subsequent layers, X is the output of the previous layer.

Notice how there is no place to incorporate edge featuresand that the same K and V matrices are used to computeAttention(qi,K, V ) ∀i.

Graph Relation Transformer Self-Attention We wouldlike to fuse the n2obj pairwise edge features between each ofthe nobj objects into the self-attention module. Suppose wehave our edge feature tensor E of dimension nobj × nobj ×de, where the vector Eij , which we will henceforth refer toas eij , represents the de-dimensional edge feature betweenobjects i and j with i as the ‘self’ object and j as the ‘other’object (the edges are directional). Figure 3 visualizes thisE tensor. Essentially, E is a stack of nobj matrices, whereeach of these matrices, Ei, represents the nobj sets of edgefeatures where object i is the ‘self’ object. These featuresmay be any fixed number of dimensions of discrete and/orcontinuous variables.

We propose three different places to fuse these featuresin the Transformer architecture: (1) in the keys, (2) the val-ues and (3) both the keys and values. We illustrate this fu-sion in the table present in Figure 3. Two central ideas mo-tivate fusion in the keys and the values. On the one hand,fusing in the keys makes object i’s attention weight for ob-ject j depend on eij . This means that the relevance of objectj on the representation of object i should depend on howthe two objects are related. On the other hand, because self-attention ultimately computes a weighted average of the val-ues, fusing in the values makes the influence of object j onthe attended representation of object i depend on eij

4

Figure 3: Illustration of how the Graph Relation Transformer (GRT) extends the ‘Vanilla’ Transformer Architecture [21],noting where this module can be inserted into the M4C model for the TextVQA task. We propose that edge features eij be-tween objects i and j can be incorporated using a fusion function φ into the keys and/or values of the Transformer Multi-HeadAttention module. φconcat (concatenation) or φadd (addition) are possible fusion functions that can be used, for example.Elements of this illustration are adapted from figures in [10] and [21].

We will now define this formally. When we fuse the edgefeatures in the keys, we now compute a different set of keys,Ki, for each object i in order to allow the relations betweeneach object i with all other objects to vary for different i.This is defined in the Equation 4. Similarly, fusing the edgefeatures in the values via Vi is defined in Equation 5. Fi-nally, Equation 6 defines fusing in both keys and values.

GraphAttnk(qi,K, V ) = softmax(

qiKTi√dk

)V (4)

GraphAttnv(qi,K, V ) = softmax(

qiKT

√dk

)Vi (5)

GraphAttnkv(qi,K, V ) = softmax(

qiKTi√dk

)Vi (6)

Now, in order to obtain the fused keys Ki, we fuse the ob-ject features X with Ei using a fusion function φ beforeprojecting the representation to the ‘keys’ vector space.

Ki = φ(X,Ei)Wk (7)

In the same way, we can fuse object features with edge fea-tures in the values by

Vi = φ(X,Ei)Wv (8)

Notice the difference between Equations 2-3 and 7-8. Inthis work, we propose two fusion functions: φconcat andφadd.

φconcat(X,Ei) = [X;EiWc] (9)

is the concatenation fusion function where [; ] is the concate-nation operation and the de × de′ matrix Wc projects Ei toan intermediate vector space of dimension de′ . We includethis projection to give each Transformer layer the flexibilityto use the edge features differently. However, one could setWc to the identity matrix to effectively disable this projec-tion.4 And

φadd(X,Ei) = X + EiWa (10)

is the addition fusion function where the de × din matrixWa projects Ei to the same space as X so that addition canbe performed.

3.2. Parameter Learning

The GRT is an extension to the baseline M4C model andthe extra parameters we introduce are trained end-to-end

4Clearly, if Wc or Wa are not the identity matrix, then the fusion func-tion itself will also contain parameters / weights to be learned along withthe rest of the model.

5

along with the existing parameters using the same multi-token sigmoid loss as M4C. We define this loss formally inEquation 11, where N is the number of training examples,Mi is the number of tokens in the ground truth answer forexample i, yij is the vocabulary index of the jth answer to-ken of the ith example and yij is the the model’s sigmoidactivation value for that token.

L = − 1

N

N∑i=1

1

Mi

Mi∑j=1

(yij log (yij)+(1−yij) log (1− yij))

(11)The extra parameters introduced by GRT are those used

by φ (i.e. Wc or Wa) inside the Transformer layers.

4. Experimental Setup4.1. Dataset

We use the TextVQA dataset [20] for this task since theTextVQA dataset is the official dataset for the TextVQAtask. The TextVQA dataset contains 45,336 questions on28,408 images where all questions require reading and rea-soning about the text in images. The dataset contains im-ages gathered from the Open Images dataset [14] and con-tains images which contain text like traffic signs, billboardsetc. We use the standard TextVQA dataset split for di-viding the dataset into training, validation and testing sets.The training set consists of 34,602 questions from 21,953images, the validation set consists of 5000 questions from3166 images and the test set consists of 5734 questions from3289 images. The TextVQA dataset contains visual (im-ages) and text (questions) modalities. In addition to this, thedataset also provides Rosetta [2] OCR tokens which formsthe additional OCR modality in the task. Each question-image pair has 10 ground-truth answers annotated by hu-mans.

4.2. Evaluation Metric

We use the evaluation metric given by the VQA v2.0challenge [8]. This metric does some preprocessing beforelooking for exact matches between the human annotated an-swers and the model’s output. This metric averages all ofthe 10 choose 9 sets of the human annotated answers asthey determined this approach is more robust to inter-humanvariability in phrasing answers. The evaluation metric equa-tion is shown below where ‘a’ is the model’s output:

Acc(a) = min

{# humans that said a

3, 1

}(12)

4.3. Baseline model: M4C

We build upon the M4C baseline model for all our ex-periments. The M4C model first projects the feature repre-sentations of the different entities (question words, objects,

OCR tokens) into a common embedding space. The ques-tion words are embedded using a pre-trained BERT model.The visual objects are represented using Faster R-CNN fea-tures (appearance (xfri ) and location features (xbi ) for someobject i) which are then projected onto a common embed-ding space, followed by layer normalization (LN) and addi-tion as shown in Equation 13:

xobji = LN(W1xfri

)+ LN

(W2xbi

)(13)

M4C uses the Rosetta OCR system to recognize OCR to-kens in the image along with their location. For some recog-nized OCR token j, a 300-dim FastText vector (xftj ), appear-ance feature from Faster-RCNN (xfrj ), a 604-dim PyramidalHistogram of Characters (PHOC) vector (xpj ) and a 4-dimlocation vector (xbj ) are extracted. Similar to the visual ob-ject features, these are projected to the common embeddingspace to get the final OCR feature for each token as shownin Equation 14:

xocrj = LN

(W3xft

j +W4xfrj +W5xpj)+LN

(W6xb

j

)(14)

After embedding all the modalities, they are all passedthrough a multi-layer Transformer with multi-headed self-attention. Each entity here can attend to all entities in thesame modality and in other modalities. The output from theTransformer is then passed through a pointer network to it-eratively decode the answer. At each time step, the pointernetwork chooses a token from either the fixed training vo-cabulary or the OCR tokens by taking in the embedding ofthe previously predicted word. The model is trained using amulti-label sigmoid loss as shown in Equation 11.

4.4. Experimental Methodology for the GRT

Since the GRT builds upon the existing M4C baseline,we used the same initialization method as M4C for each ofthe parameters within our model. Due to the limited ac-cess of the TextVQA test set evaluation server, we used theTextVQA evaluation metric to compare each of the methodsaccuracy on the validation set. To evaluate and compare thebest graph attention fusion methods and fusion locations,we trained a model for each scenario for 5,000 update stepsto determine which of the graph attention setups was thebest before training each model fully to convergence. Oncethe best graph attention setup was chosen, this became ourGraph Relation Tranformer shown in Table 1. We trainedthe GRT method for 24,000 updates for a fair comparisonas this was the number of updates originally used by theM4C authors [10]. Additionally, we conducted an ablationstudy in which we removed one of the edge feature typesand trained the model until convergence to provide clarityto the information content gained by each of the edge fea-ture types.

6

All our models use 4 Transformer layers with 12 atten-tion heads.5 Answer decoding is done for a maximum of12 tokens. On four V100 GPUs, training the baseline M4Cmodel for 24,000 updates took 5 hours. Adding the compu-tation necessary to include all our edge features increasedthis training time to 8 hours when fusing them in either thekeys or values and 12 hours in both. However, the appear-ance embedding cosine similarity was a computationallyexpensive feature and if it is removed, the time is reducedto 6 hours when fusing edge features in values only.

5. Results and Discussion

Architecture Opts Val (%) ∆Val (%) Test (%)

LoRRA∗ 27.17 -M4C∗ 38.93 0 N/AGRT∗ 39.58 0.65 39.58

M4C 39.40 0 39.01SMA 39.58 0.18 40.29

M4C 3 42.7 0 N/ASAMT 3 43.90 1.20 N/A

Table 1: Performance comparison on the TextVQA val-idation(’Val’) and Test(’Test’) sets. ∆Val refers to theimprovement of a model’s accuracy over the M4C modeltrained with no optimizations other than the architecturalchange proposed by the model. Rows marked with * areresults we obtained ourselves by using the MMF [19] frame-work with default settings. The raw SAMT accuracy val-ues are due to both their novel architecture as well asother optimizations (Opts), namely adding 2 extra Trans-former layers, using Google OCR as the OCR module anda ResNext-152 [23] Faster R-CNN model [18] trained onVisual Genome [15]. The GRT model shown here uses the‘spatial translation’, ‘spatial interaction labels’ and ‘modal-ity pair labels’ edge features and it fuses these edge featuresin the values of the Multiheaded Attention Module by addi-tion (the φadd function).

As seen in Table 1, the Graph Relation Transformer ar-chitecture surpasses the LoRRA and M4C baseline models.We also show results for the SMA [5] and SAMT [12] mod-els as originally reported by their authors.

Compared to the experimental settings originally usedby the M4C authors Hu et al. [10], the authors of SMAand SAMT report results on models that include not onlythe novel architectural changes they propose, but also other

5Note that in previous work [5, 12], simply increasing the number ofTransformer layers has been found to improve performance. We have notreproduced this tweak as it is not relevant to the ideas we explore in thispaper.

optimizations. Namely, SMA uses a custom Rosetta-enOCR module and SAMT uses 2 extra Transformer layers, aGoogle OCR module and a ResNext R-CNN. Therefore, weshow the improvement in accuracy over the correspondingM4C model trained with the same optimizations. SMA andSAMT achieve 0.18% and 1.2% accuracy increases overtheir corresponding M4C models respectively.

From our experiments, modifying the baseline M4Cmodel with the GRT architecture alone yields a 0.65% per-formance increase. This result is from the best model setupdetermined from the Transformer fusion exploration and theedge feature ablation study. This best model fused the edgefeatures into the values location, used the φadd fusion func-tion and used all edge feature types except for the visualsimilarity edge feature type.

The same model (coincidentally) achieved 39.58% accu-racy on both the test set (evaluated by the official test server)and validation set, submitted as team cmu mmml.

5.1. Fusion Function for Edge Features

Fusion Location Fusion function φ Accuracy (%)

Keys φconcat 35.51Keys φadd 35.17Values φconcat 36.90Values φadd 37.30Keys & Values φconcat 31.89Keys & Values φadd 35.41

Table 2: Performance on the TextVQA validation set ofGraph Relation Transformer with different methods of fus-ing the edge features. As the purpose of these runs wasmodel selection only, training for each model was stoppedafter 5,000 steps (before convergence). The fusion func-tions are defined in Equations 9-10.

To determine the best fusion location and fusion func-tion, we compared all of the possible combinations shownabove in Table 2. The intuition behind the keys only fu-sion location is that the importance given to another objectwould be affected by the spatial relationships between theobjects. The intuition behind the values only fusion loca-tion is that the representation of an object would changedepending on the spatial relationship between the objects.As shown above, the values only fusion location and theφadd fusion function performed the best. This shows thatthe model was better able to reason about the image whenthe model was able to change the representation of anotherobject based on the context and spatial relationship betweenthe current object.

7

(a) What word is printed under interior design onthe book in the middle?M4C: paraGraph Relational Transformer: inspirationsBest answer: inspirations

(b) What kind of establishment is in the backgroundnext to the red and white truck?M4C: plusGraph Relational Transformer: barBest answer: bar

(c) What company is on the left side of thescreen?M4C: comericaGraph Relational Transformer: meijerBest answer: meijer

Figure 4: Qualitative Analysis: The Figure shows qualitative examples of illustrative cases comparing the M4C model andthe Graph Relational Transformer. The red boxes have been applied for this figure specifically and are not present in theoriginal image. ‘Best answer’ refers to the consensus human answer in the dataset.

Feature set Accuracy (%)

All features 39.40- Appearance similarity 39.58- Spatial translation feature 39.06- Spatial interaction labels 38.78- Modality pair labels 38.96

Table 3: Ablation Study on Edge Features: Performance onthe TextVQA validation set of Graph Relation Transformerwith various groups of features ablated. Each model fusesedge features at self-attention ‘values’ only, using ‘add’ asthe fusion function φ. Each model was trained 24,000 steps(to convergence).

5.2. Ablation Study on Edge features

To evaluate the impact of each of the edge feature types,an ablation study was done for each edge feature type asshown in Table 3. We observe that while performance suf-fered when most of our edge features were dropped (as ex-pected), the one exception was that when the appearancesimilarity edge feature was dropped, performance actuallyimproved. Recall that our appearance similarity feature iscosine similarity, which is nothing more than a (scaled) dotproduct between the visual embeddings from the R-CNN.Our explanation for why this cosine similarity does not im-prove performance is that in the baseline Transformer archi-tecture, the attention module already does a scaled dot prod-uct between (projections of) the visual embeddings. There-fore, the cosine similarity does not add anything meaningfulto the model apart from introducing noise.

5.3. Qualitative Analysis

We further performed a qualitative comparison betweenthe predictions of the M4C and the Graph Relational Trans-former to identify instances that benefited from injectingobject relational features into the transformer. Figure 4shows some typical examples which demonstrate the effec-tiveness of the Graph Relational Transformer. Figure 4a andFigure 4b provides instances where the Graph RelationalTransformer was better able to reason about the relation be-tween the OCR and objects in the image to generate thecorrect answer. Figure 4c provides one instance where theGraph Relational Transformer was able to answer questionsinvolving positions better that the M4C baseline. Here, thespatial translation feature and the modality pair labels giveuseful information to the model to answer these kinds ofquestions as they allow the model to build a superior rep-resentation of the tokens by selectively interacting with theother tokens based on their relationships.

6. Conclusion and Future DirectionsIn this work we propose a Graph Relation Transformer

for TextVQA. Our best performing model uses the φadd fu-sion function at the attention ‘values’, and incorporates allof the edge features except for the visual similarity edgefeature. It outperforms the M4C model due to its improvedspatial reasoning ability. Our quantitative and qualitativeresults support our hypothesis that incorporating spatial re-lationships between objects in the image lead to better per-formance. Additionally, most of the errors in the baselineand our models were due to the OCR system incorrectlydetecting words within the image. Hence, we believe thatthere is significant performance headroom from improvingthe OCR system, as accurate OCR tokens are the first cru-

8

cial step in reasoning about the image. Beyond TextVQA,we have noted that the GRT can be applied to any taskwhere relations between objects can be represented by vec-tors. Exploring the TextVQA dataset further, possible fu-ture work includes exploring additional edge feature typesand using an improved OCR system with the GRT architec-ture. Beyond these ideas, applying the GRT architecture todifferent datasets and tasks present additional potential re-search directions. We also note that the GRT architectureis generalizable to any application or task with one or moremodalities where relations between objects are informative.We leave the validation of this hypothesis to future work.

References[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien

Teney, Mark Johnson, Stephen Gould, and Lei Zhang.Bottom-up and top-down attention for image captioning andvisual question answering. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition(CVPR), June 2018.

[2] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar.Rosetta: Large scale system for text detection and recogni-tion in images. In Proceedings of the 24th ACM SIGKDDInternational Conference on Knowledge Discovery & DataMining, pages 71–79, 2018.

[3] Deng Cai and Wai Lam. Graph transformer for graph-to-sequence learning. In Proceedings of the AAAI Confer-ence on Artificial Intelligence, volume 34, pages 7464–7471,2020.

[4] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach,Trevor Darrell, and Marcus Rohrbach. Multimodal com-pact bilinear pooling for visual question answering and vi-sual grounding. arXiv:1606.01847, 2016.

[5] Chenyu Gao, Qi Zhu, Peng Wang, Hui Li, Yuliang Liu, An-ton van den Hengel, and Qi Wu. Structured multimodal at-tentions for textvqa. arXiv preprint arXiv:2006.00753, 2020.

[6] Felix Gers. Long short-term memory in recurrent neural net-works, 2001.

[7] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter-national conference on computer vision, pages 1440–1448,2015.

[8] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba-tra, and Devi Parikh. Making the v in vqa matter: Elevatingthe role of image understanding in visual question answer-ing. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 6904–6913, 2017.

[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition, 2015.

[10] Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Mar-cus Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In Proceed-ings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 9992–10002, 2020.

[11] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach,Dhruv Batra, and Devi Parikh. Pythia v0.1: the winningentry to the vqa challenge 2018, 2018.

[12] Yash Kant, Dhruv Batra, Peter Anderson, Alex Schwing,Devi Parikh, Jiasen Lu, and Harsh Agrawal. Spatially awaremultimodal transformers for textvqa, 2020.

[13] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bi-linear attention networks. In S. Bengio, H. Wallach, H.Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett,editors, Advances in Neural Information Processing Systems31, pages 1564–1574. Curran Associates, Inc., 2018.

[14] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, SamiAbu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Ui-jlings, Stefan Popov, Andreas Veit, et al. Openimages: Apublic dataset for large-scale multi-label and multi-class im-age classification. Dataset available from https://github.com/openimages, 2(3):2–3, 2017.

[15] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan-tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, andLi Fei-Fei. Visual genome: Connecting language and visionusing crowdsourced dense image annotations. InternationalJournal of Computer Vision, 123(1):32–73, 2017.

[16] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering,2019.

[17] Jeffrey Pennington, Richard Socher, and Christopher D Man-ning. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods innatural language processing (EMNLP), pages 1532–1543,2014.

[18] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.Faster r-cnn: Towards real-time object detection with regionproposal networks. In C. Cortes, N. Lawrence, D. Lee, M.Sugiyama, and R. Garnett, editors, Advances in Neural Infor-mation Processing Systems, volume 28. Curran Associates,Inc., 2015.

[19] Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, YuJiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, DhruvBatra, and Devi Parikh. Mmf: A multimodal framework forvision and language research. https://github.com/facebookresearch/mmf, 2020.

[20] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang,Xinlei Chen, Dhruv Batra, Devi Parikh, and MarcusRohrbach. Towards vqa models that can read. In Proceed-ings of the IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), June 2019.

[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and IlliaPolosukhin. Attention is all you need. In Proceedings of the31st International Conference on Neural Information Pro-cessing Systems, NIPS’17, page 6000–6010, Red Hook, NY,USA, 2017. Curran Associates Inc.

[22] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointernetworks. In Advances in neural information processing sys-tems, pages 2692–2700, 2015.

[23] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, andKaiming He. Aggregated residual transformations for deepneural networks. In 2017 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 5987–5995,2017.

9

[24] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang,and Hyunwoo J Kim. Graph transformer networks. Advancesin Neural Information Processing Systems, 32:11983–11993,2019.

10