Technical Report: Leveraging Video Descriptions to Learn Video Question Answering Kuo-Hao Zeng *† , Tseng-Hung Chen * , Ching-Yao Chuang * Yuan-Hong Liao * , Juan Carlos Niebles † , Min Sun * * Department of Electrical Engineering, National Tsing Hua University † Department of Computer Science, Stanford University {khzeng, jniebles}@cs.stanford.edu {s104061544@m104, s102061145@m102, s102061137@m102, sunmin@ee}.nthu.edu.tw Auto-generated Questions vs. Human-generated Questions We compare auto-generated questions (Auto-QG) with human-generated questions (Human-QG) by distribution and question length. Distribution Both distributions of Auto-QG and Human-QG are plotted in Fig. 1. The distributions are similar with a few noticeable differences. For examples, the most frequent question type is “What” and the second one is “Yes/No” type questions, including “Does”, “Do”, “Did”, “Are”, “Were”, “Is” and “Was” for both types of QG. However, Auto-QG has about 13% more questions starting with “Who”. Although Auto- QG does not have exactly the same distribution as Human- QG, it is still a very cost effective way to obtain QA pairs from descriptions. (a) Auto-Question Distribution (b) Human-Question Distribution What 25.33% Where 0.26% Does 23.67% How 0.28% Was 2.47% When 0.62% Did 14.95% Do 3.50% Who 24.44% Is 2.04% other 2.43% What 34.71% Did 16.73% Who 10.99% Does 8.99% Where 8.91% Was 7.78% Is 4.20% How 3.20% Are 0.87% Were 0.87% other 2.75% Figure 1: The distributions of auto-generated questions (Left- panel) and human-generated questions (Right-panel). Length The statistic analysis of questions length are listed in Ta- ble. 1. It reveals that questions generated by Human-QG are shorter than those generated by Auto-QG. The reason is that human typically generates questions after understanding (but not systematically parsing) descriptions of the video. Hence, the questions tend to be more compact. Finally, we show more typically examples of Auto-QG (Left-panel) and Human-QG (Right-panel) in Fig. 2. Copyright c 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Length Max. Min. Mean Std. Median Auto-QG 36 2 10.8 5.3 9 Human-QG 37 2 7.3 3.2 7 Table 1: Length analysis of questions for Auto-QG and Human- QG. Max., Min., Mean, Std. and Median denote the maximum, the minimum, the average, the standard deviation and the median length of questions, respectively. Q: Who gets closer to the ground? A: Base jumper Q: Does base jumper hit side of cliff? A: Yes Q: What does base jumper hit side of? A: Cliff Q: What crashes into BMX motorcyclists and overturns? A: ATV Q: Does two BMX motorcyclists overturn this ATV rider's quad? A: Yes Q: What happened to the bmx on the back stair? A: Crash Q: Was there a crash involving a bmx? A: Yes Q: What is being ridden? A: Bike Q: Did the skateboarder wear a blue hat? A: No Q: What color was the skateboarders hat? A: Red Q: Who took a nasty slide across the pavement? A: Skateboarder Auto-QG Human-QG Figure 2: Typical examples for Auto-QG and Human-QG. Video-QA examples We show more typical examples in Fig. 3.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Technical Report: Leveraging Video Descriptions to Learn Video QuestionAnswering
Kuo-Hao Zeng∗†, Tseng-Hung Chen∗, Ching-Yao Chuang∗
Yuan-Hong Liao∗, Juan Carlos Niebles†, Min Sun∗
∗Department of Electrical Engineering, National Tsing Hua University†Department of Computer Science, Stanford University
{khzeng, jniebles}@cs.stanford.edu{s104061544@m104, s102061145@m102, s102061137@m102, sunmin@ee}.nthu.edu.tw
Auto-generated Questions vs.Human-generated Questions
We compare auto-generated questions (Auto-QG) withhuman-generated questions (Human-QG) by distributionand question length.
DistributionBoth distributions of Auto-QG and Human-QG are plottedin Fig. 1. The distributions are similar with a few noticeabledifferences. For examples, the most frequent question typeis “What” and the second one is “Yes/No” type questions,including “Does”, “Do”, “Did”, “Are”, “Were”, “Is” and“Was” for both types of QG. However, Auto-QG has about13% more questions starting with “Who”. Although Auto-QG does not have exactly the same distribution as Human-QG, it is still a very cost effective way to obtain QA pairsfrom descriptions.
(a) Auto-Question Distribution (b) Human-Question Distribution
What25.33%
Where0.26%Does23.67%How0.28%
Was2.47%
When0.62%
Did14.95%
Do3.50%
Who24.44%
Is2.04%
other2.43%
What34.71%
Did16.73%
Who10.99%
Does8.99%
Where8.91%
Was7.78%
Is4.20%
How3.20%
Are0.87%
Were0.87% other
2.75%
Figure 1: The distributions of auto-generated questions (Left-panel) and human-generated questions (Right-panel).
LengthThe statistic analysis of questions length are listed in Ta-ble. 1. It reveals that questions generated by Human-QGare shorter than those generated by Auto-QG. The reason isthat human typically generates questions after understanding(but not systematically parsing) descriptions of the video.Hence, the questions tend to be more compact.
Finally, we show more typically examples of Auto-QG(Left-panel) and Human-QG (Right-panel) in Fig. 2.
Copyright c© 2017, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.
Length Max. Min. Mean Std. MedianAuto-QG 36 2 10.8 5.3 9
Human-QG 37 2 7.3 3.2 7Table 1: Length analysis of questions for Auto-QG and Human-QG. Max., Min., Mean, Std. and Median denote the maximum,the minimum, the average, the standard deviation and the medianlength of questions, respectively.
Q: Who gets closer to the ground?A: Base jumperQ: Does base jumper hit side of cliff?A: YesQ: What does base jumper hit side of?A: Cliff
Q: What crashes into BMX motorcyclists and overturns?A: ATVQ: Does two BMX motorcyclists overturn this ATV rider's quad?A: Yes
Q: What happened to the bmx on the back stair?A: CrashQ: Was there a crash involving a bmx?A: YesQ: What is being ridden?A: Bike
Q: Did the skateboarder wear a blue hat?A: NoQ: What color was the skateboarders hat?A: RedQ: Who took a nasty slide across the pavement?A: Skateboarder
Auto-QG Human-QG
Figure 2: Typical examples for Auto-QG and Human-QG.
Video-QA examplesWe show more typical examples in Fig. 3.
Description: This snowboarder skims the surface of a pond until unintentionally cartwheeling andface planting into the water.Question: Who skims the surface of the pond? Anspred: Snowboarder
time
Ansgt: Snowboarder
Description: These two very brave souls took an incredible jump off a very high bridgeoverlooking a gorgeous beach.Question: What did the two guys jump off of? Anspred: Bridge
time
Ansgt: Bridge
Description: Maddie the Labrador Retriever knows that if you want something done right, youhave to do it yourself. So when she wanted to take a dip on a hot summer day, she took it uponherself to grab the hose and fill up the kiddie pool in the backyard.Question: What does the yellow labrador fill up the kiddie pool with? Anspred: Water
time
Ansgt: Water
Description: A boy at a skatepark rides his scooter up a ramp to attempt a backflip, but bailshalfway up and falls butt-first to the ground.Question: Did the kid fall on his butt? Anspred: Yes
time
Ansgt: Yes
Description: This basketball player was successful in dunking a ball into the hoop, but hewasn't as graceful on his landing as he faceplanted hard on the gym floor.Question: Was the basketball player successful in dunking a ball into the hoop? Anspred: Yes
time
Ansgt: Yes
Description: This athlete runs and leaps off a concrete ledge on a roof, but ends up mistimingthe stunt and crashes shoulder first into another wall nearby.Question: Will athlete do parkour again? Anspred: No
time
Ansgt: No
Figure 3: More qualitative Video QA results. In each row, we show a typical example with corresponding description, question, predictedanswer and ground truth answer. In total, we show 3 “Others” type examples and 3 “Yes/No” type examples.
Related Documents
