AR99

New AR: View AR | AR without Proprietary

History | States | Assignment | TimetrackingAR Text Search | More | CARES Home

Assistance Request 1-5522436

Customer Ticket n/a

Short TEC support needed to check the RC of this issue, and if it’s

Description affecting the KPIs or not, the network deployed with PTP sever

for Sync.

Current Closure approval granted

Summary

State Closed : Solution/Service Provided

Outage No

Severity 3

Priority 3

Service Request Remote Technical Support

Request Type Support

Sub-Type Diagnose

Category Software

Internal Yes

Assignment Events

Product

Product 9926 DBS-WCDMA

Version LR13.1W

No Change Request is linked to this AR

Product Location

Site ETISALAT ABU DHABI

Instance 9926DBS WCDMA-Etisalat : In Service

Site Company Emirates Telecommunications Corporation

City Abu Dhabi

Country United Arab Emirates

Contact

Name Karim IBRAHIM ABD EL NABY

Company Alcatel-Lucent : Open on behalf of Customer

Phone 20 127 109 0616 Email [email protected]

Request Method Email-CR

Dates

Occurred 06-Jan-2015 10:27 Time now 29-Aug-2015 13:21 (GMT+4)

Reported 06-Jan-2015 10:27 AR Created 06-Jan-2015 10:33

Service Start 06-Jan-2015 10:27

Next Customer Contact 07-Jan-2020 03:00

Responded 06-Jan-2015 23:00 Respond Target 07-Jan-2015 10:27 SA - Calculated

Restored 23-Jan-2015 18:20 Restore Target -- --- ---- SA - None

Resolved 23-Jan-2015 18:27 Resolve Target 24-Mar-2015 03:15 SA - Calculated

Targets for Restore & Resolve were extended by 16 days

to account for Pending time

Closed 23-Jan-2015 18:29

Last Modified 24-May-2015 16:05 Modified By archive

Entitlement

Agreement 242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script This request is entitled to service.

People

Owner TSCr-MoA-WCDMA-SK : rories

Assignee CloQ-WLS-WCDMA-vGlobal : unassigned

Referred 1 NorP-WLS-WCDMA-vGlobal : rories

Resolve Group NorP-WLS-WCDMA-vGlobal

Submitter ansikors

Description

Attachments

From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<[email protected]>

Sent: 2015-01-06 7:27:12

To: ALU Support <[email protected]>

Subject: Repeated alarm "SYN NOT locked to the Network"

Hello,

Short Description : Repeated PTP alarm fail over the network,

on behalf of the alarm “SYN NOT locked to the Network”

Current Summary : TEC support needed to check the RC of this

issue, and if it’s affecting the KPIs or not, the network deployed with PTP

sever for Sync.

Contact Surname : Karim

Contact Given Name : Ibrahim

Contact Phone :+971 56 354 4568

Contact Company : Alcatel-Lucent

Company : Emirates Telecommunications Corporation

Contract Number : OXIA 627085

Region : EMEA

Country : United Arab Emirates

City : Abu Dhabi

Site : ETISALAT ABU DHABI

Product Instance : 9926 DBS-WCDMA

Logs: : HFB Attached.

Severity : 3

Priority : 3

Product Version : LR13.3W

Outage : No

Internal/External: :Internal

--

Best Regards,

Karim Ibrahim

LTE/WCDMA Integration Professional

META WLS Access Network (GNE EMEA)

M(UAE): +

<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatel-lucent.com%2fowa%2fUrlBlockedError.aspx>

971-56-354-4568

M(Egy): +

<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatel-lucent.com%2fowa%2fUrlBlockedError.aspx>

2010-668-44-810

ONNET: 2205 5534

Showing Investigation and Proprietary logs together in time order Show separately

Investigation Log and Proprietary Log

1. 06-Jan-2015 10:34 ansikors

Update to Current Summary: Assigned to Workgroup

2. 06-Jan-2015 10:45 ALCATEL-LUCENT PROPRIETARY ansikors

Outgoing E-mail to Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Workflow Status: Queued

Subject: Re: Repeated alarm "SYN NOT locked to the Network"/1-5522436

Contact: Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Agent: ansikors

From: [email protected]

To: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<[email protected]>

Cc: "BERIDY, AHMED (AHMED)" <[email protected]>, "KHEDR, MAHMOUD

(MAHMOUD)" <[email protected]>, "ABDEL-HALIM, SAYED (SAYED)"

<[email protected]>, "ZAHABI, RAMSEY (RAMSEY)"

<[email protected]>, "OKASHA, TAHER (TAHER)"

<[email protected]>, "REDA, RAMY (RAMY)"

<[email protected]>, "NABIL GEORGES, MARTIN (MARTIN)** CTR **"

<[email protected]>, "ELHAKIM, Mohamed (Mohamed)** CTR

**" <[email protected]>

Bcc:

Sent on:

Dear Customer,

Thank you for contacting the Alcatel-Lucent Welcome Center.

Your request has been processed and the ticket number for your reference is AR

1-5522436

Please reference that number when contacting Alcatel-Lucent for follow-up.

An Alcatel-Lucent representative will contact you regarding your request.

Kind Regards,

Anna Sikorska

ALCATEL-LUCENT

GLOBAL WELCOME CENTER

Visit OnLine Customer Support for Contact details

3. 07-Jan-2015 01:05 rkosorin

Update to Current Summary: TPS WIP

4. 07-Jan-2015 01:38 rkosorin

Hello Karim,

In the attached hfb there is:

- The oldest alarm is raised 27.12.2014

- It is raised to appr 30 NodeBs

Are those NodeBs Do those NodeBs belong to the same RNC? Do they belong to the

same network cluster/topology?

The first alarm appeared 27.12.2014 or is the sync related issue older issue?

Since when is it present? Can you mp any network maintenance to this date?

Please provide network topology design.

Please provide the conclusion of the team, who is doing TP5000 support for you.

Please choose NodeB where the issue is present and provide output of following

commands:

/pltf_519/pltf/syn/ info

/pltf_519/pltf/syn/ synh

/pltf/ptp> ptph -a

Login to the NodeB as root and for the vlan used for synchronization please setup

tcpdump:

xCCM-nodeb-nodeb> su root

Password:nodeb3G

Check your interface config:

xCCM-root-nodeb> route

and vlan config:

xCCM-root-nodeb> ifconfig

And setup the tcpdump for the synchronization carrying vlan:

xCCM-root-nodeb> tcpdump -i your_vlan -vv /tmp/sync_trace.pcap

At the end please run IMT ZAP on the NodeB

Please provide: IMT ZAP, hfb , network snapshot, sync_trace.pcap and the output of

the commands.

Ticket will be assigned when the initial info & logs are available to me.

BR

Rasto

Update to Current Summary: additional logs & info needed

5. 07-Jan-2015 14:14 rkosorin

Update to Current Summary: logs provided - however still waiting for complementary

info.

6. 08-Jan-2015 02:01 rories

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: 07 January 2015 10:47

To: Kosorin, Rastislav (Rastislav)

Cc: ABDEL-HALIM, SAYED (SAYED); OKASHA, TAHER (TAHER); ZAHABI, RAMSEY (RAMSEY)

Subject: RE: Repeated alarm "SYN NOT locked to the Network"/1-5522436

Hi Rasto,

? This issue appeared after any site integration

? For the HFB I sent I take the 27.12.2014 as reference , @ logs I sent you all

stability files from 1 Nov.

? I performed Audit on the HFB I send to you and the most impacted site is 2513,

so I performed the AP on it.

? This Nodes are disturbed on 2 the live RNCs (RNC231&RNC235).

? All logs are @/EtisalatUAE/1-5522436

? The network topology diagram is below:

? Please we need to find the RC of this issue, a recovery soln & if it affect

the KPIs or not.

Thanks,

--

BR,

Karim

7. 08-Jan-2015 02:01 rories

From: Kosorin, Rastislav (Rastislav)

Sent: Wednesday, January 07, 2015 2:11 PM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Cc: ABDEL-HALIM, SAYED (SAYED); OKASHA, TAHER (TAHER); ZAHABI, RAMSEY (RAMSEY);

RIES, Robert (Robert)


Hello Karim,

Thank you for the info. Ticket is now assigned to Robert, who will support you.

So the issue is permanent and appeared since integration of the NodeBs, right?

In your feedback we are still missing the issue view from the perspective of TP5000.

BR

Rasto

Rastislav KOSORIN

8. 08-Jan-2015 02:02 rories


Sent: Wednesday, January 07, 2015 11:22 AM

To: Kosorin, Rastislav (Rastislav)


RIES, Robert (Robert)


Hi Rasto, Reis,

Yes most of sites are impacted after integration, but not with the same trend as

per attached, (Audit counted from 27/12/2014).

I’m trying to find the who is contact of TP5000 support,

We are waiting your investigation,

Thanks,

--

BR,

Karim

9. 08-Jan-2015 02:02 rories

From: RIES, Robert (Robert)

Sent: Wednesday, January 07, 2015 10:59 PM


Cc: ABDEL-HALIM, SAYED (SAYED); OKASHA, TAHER (TAHER); ZAHABI, RAMSEY (RAMSEY)


Hi Karim

Overview:

Based on logs, you sent we can see history of strong synchro break downs of the

NodeB “T_WR_MARRAGE_HALL_MZD_U_2513”:

1) IMT has recorded in all flr files progressive streams of alarms like this:

LOST HIGHER PRIORITY SYN SOURCE

SYN LOCK SOURCE CHANGED

SYN NOT LOCKED TO THE NETWORK

NO GRANT FROM PTP SERVER1

- which are raised and cleared repeatedly every few hours.

All of them are exclusively connected with synchronization corruption.

2) Stability files are full of alarms like this:

For FDD cell

RNC_0021_00020 - State change to Disable Failed

RNC_0028_00020 - State change to Enable Degraded

RNC_0030_00020 - State change to Disable Dependency

RNC_3004_29755 - ANO_CELL_NOT_HSDPA_CAPABLE_INFORMATION

RNC_3015_29772 - ANO_CELL_NOT_E-DCH_CAPABLE_INFORMATION

For Control port

RNC_0085_00036 - State change to Disable Failed

General

RNC_3034_29699 - ANO_NODEB_IP_USER_PLANE_FAILURE

All of them are connected with transport quality.

However there is suspicious increase of the size of hfb files the 7th Dec. This

date (and all subsequent days) we can see very strong increase of all above

mentioned transport alarms.

3) In network snapshot I checked two important parameters connected with PTP

synchro: “ptpSyncDuration” & “ptpAnnounceDuration”. Both are set correct to

value 300.

4) I needed to check DSCP values you have set for synch messages. But in tcp dump

you sent, there are no such synch messages captured, because you captured only OAM

virtual interface, although for synchronization you have set Telekom interface:

/pltf_594/emo/ip$ VirtualInterface

========= The Virtual Interface Object Parameters =========

nbVirtualInterface : 2

VirtualInterface [0] :

nbTypeOfTraffic : 1

TypeOfTraffic : OAM

IpAddress : 10.235.158.76

VlanId : 970

VirtualInterface [1] :

nbTypeOfTraffic : 3

TypeOfTraffic : PTP

IpAddress : 10.241.188.200

VlanId : 907

/pltf_594/emo/ip$

AP:

Please ask your transport team what changes were performed around the 7th Dec

either in settings of transport devices, or in general transport topology

especially in the path between TP5000 an NodeB(s).

Perform please tcp dump of telekom interface (VlanId 907). As tcp dump does not

capture UP traffic, it`s size should not be very big.

And please inspect if you have correctly set next recommended network settings for

PTP type of synchronization:

Set PTP packets (event & general messages) with highest priority (i.e. maximum

precedence, typically DSCP CS7) in TP 5000 and in E2E between TP5000 and NodeBs.

Make sure firewall and IPSec encapsulation are disabled between the TP 5000

servers and NodeB(s).

Configure the dither parameter as “enable” on the TP 5000.

Best Regards

Robert Ries

10. 09-Jan-2015 14:05 rories


Sent: Friday, January 09, 2015 10:51 AM



EL-MIDANY, AHMED (AHMED)


Hi Karim

Thanks for pcap trace. There is set DSCP value CS7 in PTP synchro message, which

is correct:

Regarding TP 5000 settings - we are UTRAN support, we have no knowledge about its

configuration.

Regarding time shift between TP 5000 and WMS time - I thing, it is no problem from

functionality point of view. But remind that in this case you have not time

correlated alarms, WOs, KPI counters, and so on ... This in result complicates any

UTRAN investigation.

However when I performed pings from RNC to NodeB and vice versa, the transport

network showed it`s very bad quality. I set pings to packet size 1500b to better

see the quality fluctuations. Look at attached document "ping log.docx", please. I

highlighted lost messages in red in it.

While your transport network is not able to deliver packets without losses, it has

no sense to investigate any UTRAN issue connected to transport (which is almost

whichever UTRAN issue).

Best Regards

Robert Ries

11. 09-Jan-2015 14:05 ALCATEL-LUCENT PROPRIETARY rories

Time Tracking Entry Added - IR01-TSA3

12. 16-Jan-2015 16:38 rories


Sent: Friday, January 09, 2015 12:22 PM

To: RIES, Robert (Robert)




Hi Ries,

Thanks for your feedback,

This is the same result we got from your team regarding the same site and 2 other

which had TX instabillty.

I Agree with it's a clear transport issue and nothing can be done from our side

but the problem that when we eslcated the issue before to ETC TX team they said no

issue from thier side, from your point of view what should be the problem between

our UTRAN and TX( configration mismtsch or BW limitation , or what?) Do you have

something from the log can give us any indication about why we recieve all this

packet and the TX is OK?,

Thanks for your usual feedback.

--

BR

Karim Ibrahim



14. 16-Jan-2015 16:40 rories


Sent: Monday, January 12, 2015 4:28 AM





Hi Karim

I cannot investigate issues on transport network as I am responsible only for

UTRAN issues. Regarding transport I don't know network topology and used network

devices settings and configurations. BTW I suppose, that even most of them are not

Alcatel products (however it is not important).

IuB traceroute shows us, that there are more transport devices used (as we

expected):

For download in control plane:

IUBCPLANE => 10.238.14.142 (RNC CP address)

1> ping -src(10.238.14.142) -ip(10.241.188.200) -size(1500) -trace Vr/5 Ip Icmp

Vr/5 Ip Icmp

IP Trace Route for 10.241.188.200:

Path taken:

Hop 1: 10.238.14.93 (time = 2ms)

Hop 2: 0.0.0.0 (time = 0ms)

Hop 3: 10.241.188.200 (time = 4ms)

ok 2015-01-09 13:06:35.42

For download in user plane:

UPLANE => 10.238.14.158 (RNC UP address)

2> ping -src(10.238.14.158) -ip(10.241.188.200) -size(1500) -trace Vr/5 Ip Icmp

Vr/5 Ip Icmp

IP Trace Route for 10.241.188.200:

Path taken:

Hop 1: 10.238.14.93 (time = 2ms)

Hop 2: 10.241.188.194 (time = 5ms)

Hop 3: 10.241.188.200 (time = 5ms)

ok 2015-01-09 13:06:46.42

For upload in control plane:

eCCM-nodeb-nodeb> traceroute 10.238.14.142

traceroute to 10.238.14.142 (10.238.14.142), 30 hops max, 38 byte packets

1 10.241.188.194 (10.241.188.194) 0.918 ms 0.736 ms 0.631 ms

2 * 10.238.14.93 (10.238.14.93) 5.269 ms 5.256 ms

3 10.238.14.94 (10.238.14.94) 4.014 ms 3.739 ms *

For upload in user plane:

eCCM-nodeb-nodeb> traceroute 10.238.14.158

traceroute to 10.238.14.158 (10.238.14.158), 30 hops max, 38 byte packets

1 10.241.188.194 (10.241.188.194) 0.774 ms 0.671 ms 0.722 ms

2 * 10.238.14.93 (10.238.14.93) 5.317 ms 5.894 ms

3 10.238.14.94 (10.238.14.94) 3.910 ms 3.960 ms *

Each of intermediate nodes with above IP addresses has to fulfil conditions I

wrote in mail with AP in yellow. Transport network must be transparent for packets

which enter it on the one side and output it on the other side.

Please require transport team to solve any violation of this behavior => network

part which are they responsible for must deliver packets without losses ! Without

exception.

Consider, you provided them log with more or less (un)successful pings. And I

don`t know what they need more ? You (and me also) have no any other tool to

inspect transport network quality. Pings with lost packets are pretty enough.

Karim, I understand you situation. It is not easy to vindicate UTRAN disability

while transport team doesn`t want to cooperate on transport issues which caused

this disability.

However my position is only to point out the problems and if they are caused by

UTRAN part, also provide solution. I pointed out that UTRAN part has no

possibility to catch synchronization to the network, which is caused based of

disappearing packets sent as end to end style. So no UTRAN issue. In this case

provide please permission to close the ticket.

Thank you

Best Regards

Robert Ries



16. 16-Jan-2015 16:42 rories


Sent: Thursday, January 15, 2015 2:52 PM



EL-MIDANY, AHMED (AHMED); BERIDY, AHMED (AHMED)


Hello Reis,

Kindly regarding the site 2513, we already have another AR which investigating a

KPIs degradation on the site due to TX instability also, now the TX team performed

action onsite and the KPIs is stable,

For PTP alarms we still receive PTP alarms but from my mentoring without the same

rate as before,

Please provide another AP, so you can check from your side.

Waiting your feedback.

Thanks,

--

BR,

Karim

17. 16-Jan-2015 16:44 rories







Hi Karim

I performed the same ping check as last time. Finally it is clear of packet loss.

I checked PTP synch NodeB settings and all seems be OK.

I checked NodeB traffic. There are active sessions in progress.

I downloaded IMT file and no problem I found in it, except Alarms trace:

Filename: /rmem/pltf.Alarms.trace

----------------------------------------------------------

TIME REF: 18.0000791630

STAMP REF: 1.0467287408

2015-01-14 01:54:32:474:940588F1 3105.2483390705 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 01:54:29 NO GRANT FROM PTP SERVER1

2015-01-14 01:54:32:480:940B2647 3105.2483758663 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 01:54:29 LOST HIGHER PRIORITY SYN SOURCE

2015-01-14 01:54:32:480:940B306E 3105.2483761262 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 01:54:29 SYN NOT LOCKED TO THE NETWORK

2015-01-14 01:54:32:480:940B3903 3105.2483763459 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 01:54:29 NO GRANT FROM PTP SERVER1

2015-01-14 01:59:19:793:C25E04C2 3109.3260941506 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 01:59:17 SYN LOCK SOURCE CHANGED

2015-01-14 07:20:16:860:FC888710 3389.4236805904 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 07:20:14 LOST HIGHER PRIORITY SYN SOURCE

2015-01-14 07:20:16:860:FC889071 3389.4236808305 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 07:20:14 SYN LOCK SOURCE CHANGED

2015-01-14 07:20:16:860:FC889982 3389.4236810626 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 07:20:14 SYN NOT LOCKED TO THE NETWORK

2015-01-14 07:20:16:860:FC88A233 3389.4236812851 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 07:20:14 NO GRANT FROM PTP SERVER1

2015-01-14 07:20:16:880:FC9AD9F9 3389.4238006777 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 07:20:14 LOST HIGHER PRIORITY SYN SOURCE

2015-01-14 07:20:16:880:FC9AE446 3389.4238009414 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 07:20:14 SYN NOT LOCKED TO THE NETWORK

2015-01-14 07:20:16:880:FC9AECF3 3389.4238011635 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 07:20:14 NO GRANT FROM PTP SERVER1

2015-01-14 07:25:10:208:41565DA6 3394.1096179110 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 07:25:07 SYN LOCK SOURCE CHANGED

2015-01-14 10:48:38:184:E7976150 3571.3885457744 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 10:48:36 LOST HIGHER PRIORITY SYN SOURCE

2015-01-14 10:48:38:184:E7976AAE 3571.3885460142 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 10:48:36 SYN LOCK SOURCE CHANGED

2015-01-14 10:48:38:184:E79773D8 3571.3885462488 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 10:48:36 SYN NOT LOCKED TO THE NETWORK

2015-01-14 10:48:38:184:E7977C52 3571.3885464658 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-14 10:48:36 NO GRANT FROM PTP SERVER1

2015-01-14 10:48:38:193:E7A03141 3571.3886035265 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 10:48:36 LOST HIGHER PRIORITY SYN SOURCE

2015-01-14 10:48:38:193:E7A03B8F 3571.3886037903 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 10:48:36 SYN NOT LOCKED TO THE NETWORK

2015-01-14 10:48:38:193:E7A0442E 3571.3886040110 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 10:48:36 NO GRANT FROM PTP SERVER1

2015-01-14 10:53:57:015:8B553121 3576.2337616161 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-14 10:53:55 SYN LOCK SOURCE CHANGED

2015-01-15 08:40:31:649:56199CF0 4717.1444519152 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:40:30 LOST HIGHER PRIORITY SYN SOURCE

2015-01-15 08:40:31:649:5619A6C9 4717.1444521673 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:40:30 SYN LOCK SOURCE CHANGED

2015-01-15 08:40:31:649:5619B03E 4717.1444524094 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:40:30 SYN NOT LOCKED TO THE NETWORK

2015-01-15 08:40:31:649:5619B8BC 4717.1444526268 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:40:30 NO GRANT FROM PTP SERVER1

2015-01-15 08:40:31:654:561ECB0B 4717.1444858635 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:40:30 LOST HIGHER PRIORITY SYN SOURCE

2015-01-15 08:40:31:654:561ED4AF 4717.1444861103 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:40:30 SYN NOT LOCKED TO THE NETWORK

2015-01-15 08:40:31:654:561EDD08 4717.1444863240 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:40:30 NO GRANT FROM PTP SERVER1

2015-01-15 08:45:26:434:A0427B03 4721.2688711427 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:45:25 SYN LOCK SOURCE CHANGED

2015-01-15 08:45:32:204:B5C161BD 4721.3049349565 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:45:31 LOST HIGHER PRIORITY SYN SOURCE

2015-01-15 08:45:32:204:B5C16B6C 4721.3049352044 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:45:31 SYN LOCK SOURCE CHANGED

2015-01-15 08:45:32:204:B5C174EA 4721.3049354474 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:45:31 SYN NOT LOCKED TO THE NETWORK

2015-01-15 08:45:32:204:B5C17D75 4721.3049356661 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-15 08:45:31 NO GRANT FROM PTP SERVER1

2015-01-15 08:45:32:211:B5C7F0DE 4721.3049779422 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:45:31 LOST HIGHER PRIORITY SYN SOURCE

2015-01-15 08:45:32:211:B5C7FA72 4721.3049781874 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:45:31 SYN NOT LOCKED TO THE NETWORK

2015-01-15 08:45:32:211:B5C802D2 4721.3049784018 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:45:31 NO GRANT FROM PTP SERVER1

2015-01-15 08:50:49:226:52C16101 4726.1388404993 [MsgSeq/37b394b0] BTS SYNCHRONISATION/0 (BTS EQUIPMENT/0) CLEARED 2015-01-15 08:50:48 SYN LOCK SOURCE CHANGED

----------------------------------------------------------

As you can see, the 14th Jan there were three synchro interruptions at 01:54,

07:20, 10:48 (distinguished by different text colors).

The 15th Jan there is just one at 08:40.

Please look at the size of the hfb files from last days. While hfbs up to the 11th

Jan have many MB, the hfbs from the 14th an 15th Jan are considerably less than

1MB in size.

The 14th Jan there is just one alarm for NodeB "T_AR_U_2513_MarrageHallMzd" at

01:28 - "loss of supervision".

The 15th Jan there is no any alarm for this NodeB.

Above described alarms should be caused either by transport fluctuations, or by hw

failure - eCCM. So if you think, the transport fluctuations are finally solved,

replace the eCCM board please.

Best Regards

Robert Ries



19. 23-Jan-2015 18:24 rories







Hello Reis,

Many thanks for your detailed investigation,

I want to mention that the TX action had been taken also on below sites(Alarm

audit done last week) as we already have KPIs degradation and TX alarms on

them(Also as mentioned before we receiving Sync alarms on them but we took 2513 as

example as it's the most impacted site), now the KPIs of most of them stable but

we still facing Sync alarms.

BTS Name RNC Name Count of Alarm Code

M_AR_BaynounhFodJnctn_U_2764 RNC231 88

EP_AR_JebelBarakhEast_U_2525 RNC235 41

ER_AR_BayyaJunction_U_2501 RNC235 32

O_WR_GAYATHI_SOUTH_U2253 RNC235 19

ET_AR_BayaShabiya_U_2375 RNC235 14

EP_AR_AlDhafraSportsAndCultureClub_U_2551 RNC231 10

Should we proceed for CCM change for all them? Or we can find another corrective

action?, Also please we already receiving the Sync lock alarm all over the live

sites as you know(see attached the audit done when we opened this AR).

For now we will proceed for CCM change for 2513 as trial and will keep you

updated, Thanks to check other high impacted sites and let us know if you have any

other corrective action,

Thanks,

--

BR,

Karim



21. 23-Jan-2015 18:24 rories


Sent: Monday, January 19, 2015 5:38 PM





Hello Karim

At the beginning of this ticket investigation we chose one most impacted NodeB

(2513), we focused on. We found clear RC (the backhaul instability) proved by

system pings across UTRAN part of network. After some remedial actions on backhaul

part, you reported essential improvement in KPIs.

Next if you still observe few synch alarms, you can replace CCM as we talked about

already.

However if you observe similar symptoms even on another sites, these issues should

be investigated in another ARs. I keep our general rule => one ticket = one issue.

So for another NodeBs, open new tickets, please.

But look at the attached log. I performed at least basic pings also for site 2764,

even though I shouldn`t.

There are clear visible the same backhaul defects caused loosing of ping commands

as in case of site 2513.

As a result - it has no sense to change CCMs on all other NodeBs, while there is

such unreliable transport network. First manage solving of the backhaul problems,

please.

Best Regards

Robert Ries

22. 23-Jan-2015 18:24 rories


Sent: Tuesday, January 20, 2015 11:07 PM





Hi Reis,

Thanks for your feedback, we will try to change the CCM and see the if there are

any enhancement of the Sync alarms or not.

For the Rule we respect it for sure and following for all tickets, but this ticket

opened to include all live sites on the network which have the same Sync alarms..

so when we solve the issue for some sites or one sites from my point of view not

mean to close the ticket and open another one with the same description.

As you know we still investigation with the transport team and will keep you

updated.

Thanks for your usual support,

Thanks,

--

BR,

Karim

23. 23-Jan-2015 18:25 rories


Sent: Thursday, January 22, 2015 9:22 AM





Hi Reis,

Kindly note that we changed the CCM for the site 2513 @ 17.1.2015 but the site

still suffering from PTP SYNC alarms as attached, as you know the site is stable

from TX's side.

Please can you check and propose a soln for this issue.

Thanks,

--

BR,

Karim

24. 23-Jan-2015 18:25 rories


Sent: Thursday, January 22, 2015 2:34 PM





Hi Karim

Look at attached xls file, please. I added the new one sheet "Alarm duration"

where are just exported columns H (Event Time) and L (Clear Time) from the sheet

"HFB_2513". I highlighted in red those time values where there was any observable

time shift present between these two time values (in any words, where there is

some difference between Event and Clear Time). But alarms, which occurred and

disappeared in the same second are in black. So in this sheet you can see duration

of alarms from site 2513 more clearly.

In summary there is 87 alarms which occurred and disappeared just in the same

second and only 36 alarms which shows any time duration. And as you can see this

duration is in average only few minutes.

So what results from this extrapolation ? NodeB 2513 has no systematic problem

with synchronization. If it would had, the alarms would be systematically present.

But we can see only occasional synchro fluctuations which last from less than one

second up to just few minutes in fact.

Moreover column Q in the sheet "HFB_2513" shows you Probable cause of incoming

alarms. Here you can see, only two type of records "lossOfSignal" and

"communicationsProtocolError" which points to transport network quality again.

As issue was answered and if there is nothing else I can help you regarding this

issue, kindly could I close the ticket ?

Best Regards

Robert Ries

25. 23-Jan-2015 18:25 rories


Sent: Friday, January 23, 2015 10:44 AM





Hello Reis,

Thanks for this detailed investigation, I have one question then we can close this

AR, as you mentioned below regarding column Q, I can see also that the Specific

Problem is "BTS/0 NO GRANT MESSAGE FROM PTP SERVER 1"

My question is: Based on your investigation, NO issue with our PTP server? And

it's totally TX issue? Or may we have any configuration mismatch or PTP failure

affecting this issue?

Probable Cause Specific Problem

communicationsProtocolError BTS/0 NO GRANT MESSAGE FROM PTP SERVER 1

Thanks,

--

BR,

Karim

26. 23-Jan-2015 18:25 rories







Hello Karim

Look to the alarm sequences in the xls file HFB_2513 again, please. These alarm

chunks exhibits a cyclic repetition in similar sequences like this (with comments

copied from Alarm Reference Guide):

=================================================

BTS_0358_00002 - LOST HIGHER PRIORITY SYN SOURCE => This alarm indicates that the

NodeB is not able to be synchronized.

BTS_0007_00002 - SYN NOT LOCKED TO THE NETWORK => The alarm indicates that the BTS

is unable to derive reference timing from the Pulse Code Modulation (PCM) or from

the synchronization sources.

BTS_0388_00033 - NO GRANT MESSAGE FROM PTP SERVER 1 => This alarm indicates that

preferred PTP time server does not provide GRANT message to the NodeB REQUEST

UNICAST TRANSPORTATION messages.

BTS_0333_00002 - SYN LOCK SOURCE CHANGE => This alarm indicates that SYN LOCK

Source of a BTS is changed.

================================================

Important fact in this case is, that each chunk starts with alarm BTS_0358_00002

with indicates synchronization lost (probable cause "lossOfSignal"). And alarm

BTS_0388_00033 is just consequence of foregoing alarms indicated lost of

synchronization to the network.

In case you have any configuration mismatch in your PTP server, the NodeBs

synchronization problems would be permanent.

As issue was answered and if there is nothing else I can help you regarding this

issue, kindly could I close the ticket ?

Best Regards

Robert Ries

27. 23-Jan-2015 18:27 rories







Hi Reis,

Thanks again, You can close the AR,

Thanks,

--

BR,

Karim

Update to Current Summary: Issue confirmed as backhaul instability.

28. 23-Jan-2015 18:29 rories

Update to Current Summary: Closure approval granted

Resolution

Issue confirmed as backhaul instability.

End of Assistance Request 1-5522436

New AR: View AR | AR without Proprietary | OLCS

History | States | Assignment | TimetrackingAR Text Search | More | CARES Home

AR99

Documents

karim karim

karim contact

dbs wcdmaetisalat

contact company

dbswcdma logs

network current summary

network hello

customer contact