For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub. We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback. 1 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures ___________________ ___________________________ Abstract This guide helps you identify and address common issues that can cause SyncIQ to fail. July 2, 2019 EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE SYNCIQ FAILURES OneFS 7.2 - 8.1.0
24
Embed
EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures · troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.
Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.
For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the Online Support site.
Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.
Configure screen logging through SSHWe recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.
1. Open an SSH connection to the clsuter and log in by using the root account.
Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.
2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:
cd / i f s/ dat a/ I si l on_Suppor t
3. Run the following command to capture all input and output from the session:
scr een - L
This will create a file named scr eenl og. 0 that will be appended to during your session.
IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.
If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.
Capture the error for the failing policy as follows:
1. Obtain the SyncIQ job ID by running the following command on the source cluster, where <pol i cy- name> is the name of theSyncIQ policy. See Appendix C for example output.
i si sync r epor t s l i st - - pol i cy- name=<pol i cy- name> - - sor t j ob_i d
2. View the report by running the following command, where <pol i cy- name> is the name of the SyncIQ policy and <j obI D> is the job ID you obtained in step 1. The output of the command lists the error. See Appendix C for example output:
i si sync r epor t s vi ew - - poi l cy=<pol i cy- name> <j obI D> | l ess
In the report, find the error that relates to the failure. Note the start and end times for the policy. See Appendix C for example output.
Did the policy fail within the first five
minutes of starting, or after the first five minutes?
- Page 9 - Sync fails within the first five minutes (5)- Page 11 - Sync fails after the first five minutes (2)
Read the following notes about this error. Then continue on to troubleshooting the error.
Notes about this error
Error: SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker )
This error occurs:When the sync fails at any time during the sync job.
This error appears: In the sync policy report on the OneFS web administration interface or command-line interface and in the / var / l og/ i si _mi gr at e. l og file.
Cause of this error: The source pool does not include node 1. The bandwidth/throttle daemon cannot be reached because it always runs on node 1.
Example of error:2012- 12- 27T18: 10: 37- 06: 00 <3. 3> cl ust er 1- 8( i d8) i si _mi gr at e[ 11771] : coor d[ pol i cy1] : si q_cr eat e_al er t : t ype: 11 ( pol i cy name: pol i cy1 t ar get : cl ust er 1. company. com) SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker ) . Pl ease ver i f y al l SyncI Q daemons ar e r unni ng. Unabl e t o connect t o t hr ot t l e host f or l ast 1080 seconds
Upload log files and contact Isilon Technical Support, as instructed in
Appendix A.
No
Yes
Use the OneFS web administration interface to add a source pool restriction as follows:
1. Click Data Protection > SyncIQ > Policies.2. For the policy that you want to set the restriction on, click View/Edit.3. Click Edit Policy.4. In the Source Cluster section, in the Restrict Source Nodes
section, select the radio button for Run the policy only on nodes in the specified subnet and pool.
5. Select a subnet and pool from the drop-down list.6. Click Save Changes.
IMPORTANT!Make sure that the front-end network ports on all of the nodes in the source pool restriction
- Page 13 - SyncIQ error connecting to daemon (2)- Page 14 - SyncIQ error connecting to daemon (3)
Run the following command to view a list of the network pools within a groupnet or subnet:
i si net wor k pool s l i st
Extract the gr oupnet : subnet value for the desired pool name from the output, for example, gr oupnet 1. subnet 3 for pool 5, and provide it as an input for the following command to check
the nodes within that pool.
i si net wor k pool s vi ew <gr oupnet _name>. <subnet _name>. <pool _name>
See Appendix E for example output of both commands.
Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.
Upload node log files and the screen log file to Isilon Technical Support1. When troubleshooting is complete, type exi t to end your screen session.
2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your method of uploading files. If you are not sure which method to use, use FTP.
ESRS: i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0
FTP: i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0
HTTP: i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0
SupportIQ:Copy and paste the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.
i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \- - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p
3. If you receive a message that the upload was unsuccessful, refer to article 304567 for directions on how to upload files over FTP.
Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.
CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.
End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.
Optional process step
IntroductionDescribes what the section helps you to accomplish.
Example outputcl ust er - 1# i si sync r epor t s l i st - - pol i cy- name=pol i cy1 - - sor t j ob_i dPol i cy Name Job I D St ar t Ti me End Ti me Act i on St at e - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -pol i cy1 1 2016- 02- 02T11: 06: 30 2016- 02- 02T11: 06: 38 r un f i ni shedpol i cy1 2 2016- 02- 02T11: 06: 44 2016- 02- 02T11: 06: 53 r un f i ni shedpol i cy1 3 2016- 02- 02T11: 18: 37 2016- 02- 02T11: 18: 41 r un f ai l ed
Example outputcl ust er - 1# i si sync r epor t s vi ew - - pol i cy=dot t est 1 | l ess Pol i cy Name: dot t est Job I D: 1 St ar t Ti me: 2016- 02- 02T17: 27: 55 End Ti me: 2016- 02- 02T17: 33: 10 Act i on: r un St at e: f ai l ed I D: 1- dot t est Pol i cy I D: a12345678b901c23456abc78912d34dc Sync Type: i nval i d Dur at i on: 5m12s Er r or s: No node on sour ce cl ust er was abl e t o connect t o t ar get cl ust er . , Sour ce node coul d not connect t o t ar get cl ust er .<truncated>
A source pool restriction can be found by running the following commands. The bold lines in the example output identify the
restrictions. In these examples, the restrictions are subnet 1: pool 0.
Example outputcl ust er - 1# i si sync pol i es vi ew pol i cy1 I D: 1234567891234a5a67890f 1234ba8cab Name: pol i cy1 Pat h: / i f s/ dat a/ backup Act i on: sync Enabl ed: No Tar get : cl ust er . company. com Descr i pt i on: Check I nt egr i t y: YesSour ce I ncl ude Di r ect or i es: -Sour ce Excl ude Di r ect or i es: - Sour ce Subnet : subnet 1 Sour ce Pool : pool 0<truncated>
Example outputi si net wor k pool s l i st gr oupnet 1. subnet 3
I D SC Zone Al l ocat i on Met hod- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -gr oupnet 1. subnet 3. pool 5 dat a. company. com st at i cgr oupnet 1. subnet 3. pool 7 dat a. company. com dynami c- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Example outputi si net wor k pool s vi ew gr oupnet 1. subnet 3. pool 5 to display the nodes within pool5
I D: gr oupnet 0. subnet 3. pool 5Gr oupnet : gr oupnet 1subnet : subnet 3Name: pool 5Rul es: -Access Zone: zone3Al l ocat i on Met hod: st at i cAggr egat i on Mode: l acpSC Suspended Nodes: -Descr i pt i on: -I f aces: 1: ext - 2, 2: ext - 2, 3: ext - 2I P Ranges: 203. 0. 223. 12- 203. 0. 223. 22- - - - - - - - - - -<truncated>
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com