NREN NOC Best Practices...NREN NOC Best Practices Yasuichi Kitamura ([email protected]) Jin Tanaka ([email protected]) Yuichi Kurokawa ([email protected]) Takatoshi Ikeda ([email protected])

NREN NOC Best Practices

Yasuichi Kitamura ([email protected])Jin Tanaka ([email protected])

Yuichi Kurokawa ([email protected])Takatoshi Ikeda ([email protected])

2010年8月16日月曜日

mailto:[email protected]






Who are we?What is APAN?



What is APAN?

Asia-Pacific Advanced Network

None-profit organization

Exchange point operation



Fiber cut in 2006


What happened under the sea?

These pictures were provided by KDDI.

CN-US Cable Outside CN-US Cable Inside


APAN-

JP

US

APAN

-TW

KR

TEIN

2

CST

NET

THSG

ID

MY

VN

HK

EU

AU

CER

NET


KOREN

Australia

Seattle

Honolulu

TEIN2

CN

TH

MY

ID

VN SGAPAN Tokyo XP

155Mbps 10Gbps 10Gbps

10Gbps

10Gbps

155Mbps

1Gbps


KOREN

Australia

Seattle

Honolulu

TEIN2

CN

TH

MY

ID

VN SGAPAN Tokyo XP


10Gbps

10Gbps

155Mbps

1Gbps


KOREN

Australia

Seattle

Honolulu

TEIN2

CN

TH

MY

ID

VN SGAPAN Tokyo XP


10Gbps

10Gbps

155Mbps

1Gbps

155Mbps


Routing Path in Asia Pacific(Estimated Path except APAN-JP)

APAN-JP CERNET APAN-TW KOREN ASTI CSTNET TEIN2 SingAREN UniNet ThaiSARN KREONET2 AARNET

APAN-JP TP2-Abilene

JP-TWDirect APII None KDDI-GW None

SingARENvia JGN2

SG

TP2-Abilene None TP2-

Abilene

TP2-AbileneHawaii

CERNETTP2-

AbileneUS?

APAN-APAN-JP

US?

CN-KRDirect None ? TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

APAN-TW JP-TWDirect

APAN-JPUS? APAN-JP None None None ? ? ? APAN-JP

APII APAN-JP

KOREN APII CN-KRDirect APAN-JP None None TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs Direct TEIN2

PoPs

ASTI None None None None None None None None None None None

CSTNET KDDI-GW ? None None None None ? ? ? None ?

TEIN2 None TEIN2-PoPs None TEIN2

PoPs None None TEIN2PoPs

TEIN2PoPs TEIN2 KOREN TEIN2

PoPs

Src

Normal

Multiple Path

Always been anomaly Path

Connectivity Lost or Commodity

Dest

Src

Poor Path

Dec. 27. 2006



APAN-JPAPII

SingARENHawaii

JP-TWDirect APII None

TP2 Abilene - RBnet

KDDI-GW

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

TP2-Abilene

TP2-AbileneHawaii

CERNETAPII

SingARENHawaii

APAN-JP CN-KRDirect None ? TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs ? TEIN2

PoPs

APAN-TW JP-TWDirect APAN-JP APAN-JP None None None Direct APAN-JP APAN-JP ? APAN-JP


PoPsTEIN2PoPs

TEIN2PoPs


PoPs


CSTNET

TP2 Abilene - RBnet

KDDI-GW

? ? None None ? ? ? ? None ?

TEIN2SingAREN via JGN2-

SG

TEIN2-PoPs

APAN-KRAPAN-JP

TEIN2PoPs None None TEIN2

PoPsTEIN2PoPs TEIN2 KOREN TEIN2

PoPs

Dest

Src


Normal

Multiple Path Connectivity Lost or Commodity

Single Path Always been anomaly Path

Jan. 4. 2007


東京

シンガポール

香港

APCN2ケーブル

中国内陸線

This figure is composed by KDDI.2010年8月16日月曜日


APAN-JPAPII

SingARENHawaii

JP-TWDirect APII None

TP2 Abilene - RBnet

KDDI-GW

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

TP2-Abilene

TP2-AbileneHawaii

CERNETAPII

SingARENHawaii

APAN-JP CN-KRDirect None ? TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs ? TEIN2

PoPs

APAN-TW JP-TWDirect APAN-JP APAN-JP None None None Direct APAN-JP APAN-JP ? APAN-JP


PoPsTEIN2PoPs

TEIN2PoPs


PoPs


CSTNET

TP2 Abilene - RBnet

KDDI-GW

? ? None None ? ? ? ? None ?


SG

TEIN2-PoPs

APAN-KRAPAN-JP

TEIN2PoPs None None TEIN2


PoPs

Dest

Src


Normal



Jan. 4. 2007



APAN-JPAPII

SingARENHawaii

JP-TWDirect APII MAFF-PH

Direct

TP2 Abilene - RBnet

KDDI-GW

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

SingAREN via JGN2-

SG

TP2-Abilene

TP2-AbileneHawaii

CERNETAPII

SingARENHawaii

APAN-JP CN-KRDirect APAN-JP TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

APAN-TW JP-TWDirect APAN-JP APAN-JP APAN-JP TEIN2APA

PAN-JP None Direct APAN-JP APAN-JP APAN-JPAPII APAN-JP

KOREN APII CN-KRDirect APAN-JP APAN-JP TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs


PoPs

ASTI MAFF-PHDirect APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP

CSTNET

TP2 Abilene - RBnet

KDDI-GW

TEIN2PoPs

TEIN2APAPAN-JP

TEIN2PoPs

TEIN2APAPAN-JP

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs


SG

TEIN2-PoPs

APAN-KRAPAN-JP

TEIN2PoPs APAN-JP None TEIN2


PoPs

Dest

Src

Jan. 9. 2007


Normal





APAN-JP JP-CN Direct

JP-TWDirect APII MAFF-PH

Direct JP-CN Direct

TEIN2 JP-SGJP-HK

JGN2SG

Direct

TEIN2JP-SGJP-HK

TEIN2JP-SGJP-HK

TP2-Abilene

TEIN2PoPs

Hawaii

CERNET JP-CN Direct APAN-JP CN-KR

Direct APAN-JP TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

TEIN2PoPs

APAN-TW JP-TWDirect APAN-JP APAN-JP APAN-JP TEIN2APA

PAN-JP None Direct APAN-JP APAN-JP APAN-JP APAN-JP

KOREN APII CN-KRDirect APAN-JP APAN-JP TEIN2

PoPsTEIN2PoPs

TEIN2PoPs

TEIN2PoPs


PoPs

ASTI MAFF-PHDirect APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP APAN-JP

CSTNET JP-CN Direct

TEIN2PoPs

TEIN2APAPAN-JP

TEIN2PoPs

TEIN2APAPAN-JP None APAN-JP APAN-JP APAN-JP Direct APAN-JP

TEIN2TEIN2 JP-SGJP-HK

TEIN2-PoPs

APAN-KRAPAN-JP

TEIN2PoPs APAN-JP None TEIN2


PoPs

Dest

Src

Feb. 20. 2007


Normal




• Typhoon No. 8 (Japanese name), Morakot (English name) attacked Kaohsiung in 2008.


Prof. Kasahara’s report


Temporal Rou,ng in Asia during Undersea Cable Disrup,on

APAN-‐JP NOC

13th August 2009


Normal Layer3 Topology and Traffic of Asian NRENs

Normal Traffic Flow


First impact on August 6th

• TEIN3 HK-‐SG 2.5G circuit was interrupted due to EAC cable system frailer

• The traffic between TEIN3-‐HK and SG Pops shiSed to NII’s TEIN3-‐JP back up links

Normal Traffic FlowBack-uped Traffic Flow


Second impact on August 9th

• TEIN3 SG-‐KR 2.5G circuit was interrupted due to “EAC cable system” frailer

• The traffic between KOREN and TEIN3 network shiSed to APII/JGN2plus 10G link



Third impact on August 12th (Current)

• The fault was detected on Segment 7 of APCN2

• Addi,onally, segment 1 of the APCN2 is also currently under repair

• Both NII and JGN2plus　JP-‐SG 622 circuit were outage

• The connec,vity of TEIN3 SG(and around NREN) and other Asian NREN was interrupted

• JP MAFF’s JP-‐PH 155M circuit was also affected. ASTI has been disrupted the connec,vity with R&E network



Rou,ng affected as Undersea Cable Disrup,on


Proposal for temporary backup rou,ng • According to our inves,ga,on by comPATH tool, currently there are many intercon,nental loop /inefficient routes on Asian NRENs.

• It is important to restore the connec,on from APAN, TEIN3 to Singapore area.

• We have only one choice to get the connec,vity to Singapore area by crea,ng temporary TEIN3-‐SG -‐ AARnet -‐ ASGC -‐ APAN-‐JP – TEIN3-‐HK path.

• The table on next slide shows the expected rou,ng when we implement this temporal backup rou,ng, and Intercon,nental loop routes will be disappeared.


Temporal Rou,ng in Asia during Undersea Cable Disrup,on


Summary of NOC’s job

• Remember or Data base is required for...

• Topology map

• Bandwidth

• routing policy

• friendly talking with other NOCs

• NOC should have the policy experts for...

• violating the routing policy.


NREN NOC Best Practices...NREN NOC Best Practices Yasuichi Kitamura ([email protected]) Jin Tanaka ([email protected]) Yuichi Kurokawa ([email protected]) Takatoshi Ikeda ([email protected])

Documents