TEST SUITE DEVELOPMENT TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS OF EMAIL PROTOCOLS Anastasia Tugaenko Anastasia Tugaenko Scientific Adviser Scientific Adviser: Nikolay Pakulin, PhD Nikolay Pakulin, PhD Institute for System Programming RAS, Moscow Institute for System Programming RAS, Moscow SYRCoSE 2010. 1-2 June, 2010. Nizhny Novgorod, Russia
25
Embed
TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS
TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS. Anastasia Tugaenko Scientific Adviser : Nikolay Pakulin, PhD Institute for System Programming RAS, Moscow. SYRCoSE 2010. 1-2 June, 2010 . Nizhny Novgorod, Russia. Conformance Testing. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
TEST SUITE DEVELOPMENT TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING FOR CONFORMANCE TESTING
Institute for System Programming RAS, MoscowInstitute for System Programming RAS, Moscow
SYRCoSE 2010. 1-2 June, 2010. Nizhny Novgorod, Russia
2/25
Conformance Testing
• Many implementations of protocols from different developers are functioning in the contemporary Internet
• The reliability of data transfer substantially depends on implementations compatibility
• The basic method of attesting implementations compatibility is conformance testing (testing to conform to the standard)
3/25
Simple Mail Transfer Protocol
User agent (UA)
Local mail transfer agent (Local MTA)
Relay mail transfer agent (Relay MTA)
Relay mail transfer agent (Relay MTA)
Local mail transfer agent (Local MTA)
User agent (UA)
Internet
4/25
Simple Mail Transfer ProtocolS: 220 foo.com Simple Mail Transfer Service Ready C: EHLO bar.com S: 250-foo.com greets bar.com S: 250-8BITMIME S: 250 HELP C: MAIL FROM:<[email protected]> S: 250 OK C: RCPT TO:<[email protected]> S: 250 OK C: RCPT TO:<[email protected]> S: 550 No such user here C: RCPT TO:<[email protected]> S: 250 OK C: DATA S: 354 Start mail input; end with <CRLF>.<CRLF> C: Blah blah blah... C: ...etc. etc. etc. C: . S: 250 OK C: QUIT S: 221 foo.com Service closing transmission channel
5/25
Post Office Protocol (version 3)
UserUser
Agent MailboxPOP
server
commands
replies and data
6/25
Post Office Protocol (version 3)S: +OK POP3 server readyC: USER mrose S: +OK mrose is a real hoopy frood C: PASS secret S: +OK mrose's maildrop has 2 messages (320 octets) C: LIST S: +OK 2 messages (320 octets) S: 1 120 S: 2 200 S: . C: RETR 1 S: +OK 120 octets S: <the POP3 server sends message 1> S: . C: DELE 3 S: -ERR no such message C: QUIT S: +OK dewey POP3 server signing off (maildrop empty)
7/25
Mail Protocols Features
• Mail protocols are underspecified
• Mail protocols are nondeterministic
• Mail protocols requirements differ in the level of obligations (MUST, SHOULD, MAY, …)
• Protocol architecture is extensible
8/25
Requirements for Test Suite
• Requirements traceability
• Availability of option defining the set of requirements supported by IUT
• Completeness in terms of requirements coverage
9/25
Traditional Methodology of Conformance Testing
• Test suite consists of formal given tests which are not connected to the implementation
• Connections between requirements and test purposes and between test purposes and their tests are informal
• Test purposes describe situations to be tested. Test purpose is realized in one or several tests
• Implementation is considered to conform to the standard if all test purposes have been passed
Connecting to localhost:25<-220 petr SMTP Server (JAMES SMTP Server 2.3.1) ready + <date>-> HELO example.org<-250 petr Hello example.org (localhost [127.0.0.1])-> QUIT<-221 2.0.0 petr Service closing transmission channelclosing
Script:
Result:
11/25
UniTESK Technology
• Specifications formalize requirements as pre- and postconditions
From specifications test oracles are generated
• Test sequence is generated from test state machine traversal
Observable behavior is automatically verified by test oracles
12/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
13/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
14/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
15/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
16/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
17/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
Developing of elementary tests
18/25
Method for Mail Protocols Conformance Testing
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
• The absence of quick test suites updating ability (but after all preparations have done one got not a single test but a set of tests responsible for certain requirements class)
• Formal specification description is very laborious process
23/25
Method Analysis: the Advantages
• High order of reusing test suite components Test oracles Test data iterators
• Automated generation of test sequences
• Automated calculation of reached test coverage
• Automated verdict pronouncement
24/25
Conclusion• The method for test sequence generation for
email protocols implementation testing is presented
• Test suite for conformance testing of email protocols is constructed
• The following disagreements between email implementations and standards were detected:
Absence of required commands supporting Protocol rules violation Wrong reply codes to the protocol commands Cycling while redirecting mail