The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science.

The PEACE Database:

Maintaining your SDDAS database access, data export and data visualisation tools

R.J. Wilson and A.N. Fazakerley

Mullard Space Science LaboratorySpring 2006Version 1.1

Content

• Introduction • Overview of the activities• Recommended approach• Graphic “how to” guide• Explanation of each activity

Note:This document tells you how to update your own local SDDAS installation, on your own machine.If you use msslAB to run SDDAS then all this is done for you.

Introduction (1)

The PEACE Database uses IDFS format files, and a suite of software known as SDDAS is freely available which enables you to work with the database, export data and visualise the data

The Database changes with time as new data is added and as calibrations change

Processed data in the Database (SPINPAD, moments, spacecraft potential estimates etc.) may be reprocessed from time to time

Tools discussed here allow you to check whether your local copy of the database is up to date, and to make updates when necessary

Other tools allow you to update the SDDAS software

We also provide suggestions on maintenance of your local copy of the database

Introduction (2)

The Master Server for all PEACE data is at MSSL (UK)Its IP number is currently 128.40.70.117

A mirror copy of the PEACE database is at SwRI (USA)Its IP number is currently 129.162.154.69 It is intended to be regularly updated to track changes at MSSL

Overview of the activities

• Sync With Server Update your local copy of the Hierarchy from root server i.e. Tells your computer what products are available

• MegaMetaPromote Using your local copy of the Hierarchy, update your local copy of the data catalogue, from which you may promote data.

• Download Beta Apps Get the latest versions of SDDAS applications to plot/export data.

• Clearing out your promoted data area Stop your local Hard Disk filling up, and stops you using older data.

Recommended approach to maintenance

Proposed Schedule and proposed order or execution: 1. Clearing out your promoted data area:

At least monthly for Cluster, weekly for Double Star(Weekly)

2. Download Beta Apps:At least monthly

(Monthly)

3. Sync With Server:Monthly, or if you hear a new virtual has been added.

(Monthly)

4. MegaMetaPromote (always after Sync with Server):Weekly at least.If you’re working with very recent data consider daily.

(Daily)

For reference, the main MSSL group computer is updated at the frequency in blue.

Caring for SDDAS…in pictures…

Select Admin Tools

• Sync with Server

• MegaMetaPromote

• Download Beta Apps

Select Admin Tools• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get:

• You can select just parts the you want – but easier to just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Select Admin Tools• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get:

• You can select just parts the you want – but easier to just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get:

• You can select just parts the you want – but easier to just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get:

• You can select just parts the you want – but easier to just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117

• Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get:

• You can select just parts the you want – but easier to just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117

• Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get a GUI like the one shown here:

• You can highlight just parts the you want to update, and press “OK” – or just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Material that is not up to date on your computer

Material that is no longer on the master server,so can be deleted from your local system.

Sync with Server• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Config Files Editor”

• Select Pull Down Menu:“Action > Sync With Server…”

• From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117

• Then select “Ok”.

• If files are already in sync it will tell you, otherwise you get a GUI like the one shown here:

• You can highlight just parts the you want to update, and press “OK” – or just select “Select All” and then “OK”

• You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”

DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

When saving, it is normal to see a pop-up box asking you to confirm that you want to overwrite the current .cfg files

MegaMetaPromote• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Promote Meta Data”

• Select “Source”

• You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII…

• At his point just click “Ok” as you want all the Cluster spacecraft.

• This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update.

[NB Do not Promote VIDFs as there are 100,000s of them.]

Always do this after, not before, Sync with Server!

MegaMetaPromote• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Promote Meta Data”

• Select “Source”

• You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII…

• At his point just click “Ok” as you want all the Cluster spacecraft.

• This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update.

[NB Do not Promote VIDFs as there are 100,000s of them.]

Always do this after, not before, Sync with Server!

MegaMetaPromote• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Promote Meta Data”

• Select “Source”

• You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…)

• At this point just click “Ok” as you want all the Cluster spacecraft.

• This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update.

[NB Do not Promote VIDFs as there are 100,000s of them.]

Always do this after, not before, Sync with Server!

MegaMetaPromote• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Promote Meta Data”

• Select “Source”

• You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…)

• At this point just click “Ok” as you want all the Cluster II spacecraft.

• This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update.

[NB Do not Promote VIDFs as there are 100,000s of them.]

Always do this after, not before, Sync with Server!

MegaMetaPromote• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Promote Meta Data”

• Select “Source”

• You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…)

• At this point just click “Ok” as you want all the Cluster II spacecraft.

• This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update.

[NB Do not Promote VIDFs as there are 100,000s of them.]

Always do this after, not before, Sync with Server!

Download Beta Apps• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Download BETA Apps”

• Select “Go”

• Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Download BETA Apps”

• Either select “Go”, or…

• Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Download BETA Apps”

• Either select “Go”, or…

• Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want.

Download Beta Apps• At console prompt, type:

sdcontrol &

• Select “Administration Tools”

• Select “Download BETA Apps”

• Either select “Go”, or…

• Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want.

• To finish now select “Quit”.

Clearing out your promoted data area• When you promote data (or VIDFs, PIDFs) they go in to the hierarchy subdirectories such as

those beneath: /home/sddas/data/CLUSTERII (for Project CLUSTERII)

/home/sddas/data/DOUBLESTAR (for Project DOUBLESTAR)

• These files stay there and take up room on your hard disk.– i.e. a single sensor Cluster PEACE data file (*D) from PAD may be 10 Mb in size…

If want both HEEA & LEEA for a short period (hours), over all 4 spacecraft that’s 80 Mb.

– A single day Double Star PEACE 3DF data file (*D) can easily be 500 Mb,If you want both TC-1 and TC-2 then that’s 1000Mb, or 1 Gb.

• You local Hard Disk will quickly become full if you do not delete these files after use.This can then stop your entire computer working properly due to lack of disk space.

• Delete files you no longer need!!!

Files to delete are…

Clearing out your promoted data area• PIDFS: Only small files (~80k), relatively few of them.

• VIDFS: Only small files (~80k) but many.

• Header files: Small-ish files (up to a few Mb) but many.

• Data files: LARGE files (10s to100s of Mb) and many of them.

• Why delete the small (or large) files at all? Maybe they’ve been updated on the master server with new calibrations or re-processed. Deleting them ensures you promoted the latest version next time you use them.

• A brute force method of deleting all promoted files, for project CLUSTERII, assuming your local data directory is in /home/sddas/data/ :rm /home/sddas/data/CLUSTERII/*/*/*/*.pidf.v2 - delete PIDFsrm /home/sddas/data/CLUSTERII/*/*/*/*V.v3 - delete version 3 VIDFsrm /home/sddas/data/CLUSTERII/*/*/*/*[IV] - delete version 2 VIDFs, both file typesrm /home/sddas/data/CLUSTERII/*/*/*/*[HD] - delete Header and Data filesSimilar for projects DOUBLESTAR or MARS.

Do not delete anything from the Database directories, i.e. leave /home/sddas/data/CLUSTERII/*/*/Database alone.The use of wildcards illustrated above is safe as no files in the Database directory match the ending letters above.(…but if you do after all delete from the Database directories you can recover using MegaMetaPromote)

Explanation of the Activities

Anatomy of IDFS 1• In IDFS you have a hierarchy

from mission down to instrument:Project > Mission > Experiment > Instrument

i.e.CLUSTERII > CLUSTER-1 > PEACE > 3DF

• At the Instrument level you have “virtual instruments”, which are subsetsof the instruments data.

i.e. CP3DFH and CP3DFL

• “virtuals” have a set naming convention: (Project, experiment, description)CP3DFH = Cluster PEACE 3DF HEEADP3DF = DoubleStar PEACE 3DF (single sensor)CFUNIT = Cluster FGM UNIT vectorDSPRIME = DoubleStar Staff PRIME parameters Name must be unique and no more than 8 characters.DHIAPRIM = DoubleStar HIA PRIME parametersDHIDPRIM = DoubleStar HID PRIME parameters

(Exception to convention: SPINAD with virtuals such as HSPAD & LSPAD)

Anatomy of IDFS 2There are 4 types of file, VIDF, PIDF, Header and Data

• PIDF: Plot Interface Description File.– Just one per virtual, contains labels and unit choices.

Filenames of: {virtual}.pidf.v2 - text file that you can readi.e. CPPADH.pidf.v2

• VIDF: Virtual Instrument Description File.– Contains calibrations and conversions for units.– Can be many VIDFs per virtual, each file contains a start and stop time inside it.– Filename contains start time (to nearest minute).– VIDFS must not overlap in period.

{virtual}{start-time}V - version2 VIDF text file (for human use){virtual}{start-time}I - version2 VIDF binary file (for SDDAS use){virtual}{start-time}V.v3 – version3 VIDF text file (for human & SDDAS use)

{Start-time} in format: (year)(doy)hhmm, ie. 20050381400 for 2005-Feb-07 14:00i.e. CPPADH20050381400V.v3

• HEADER and DATA file: they come as a pair, contain the actual data.– Header file is tiny, data file is BIG.– Can be up to 3 different file-pairs per day for Cluster, 1 file-pair per day for Double Star.

{virtual}{start-time}H - binary file (for SDDAS use){virtual}{start-time}D - binary file (for SDDAS use)i.e. CPPADH20050381400H & CPPADH20050381400D

Sync With Server• Updates your machine with the root server that contains the latest version of

the Hierarchy. i.e. including any new virtuals recently added.

Modifies:• /usr/local/sddas/config/DB.cfg

Contains list of what servers you can promote from for each Project. (IP addresses listed rather than server names.)

• /usr/local/sddas/config/PROJECTS.cfg

Contains list of full hierarchy for each Project.

• Doing a Sync with Server updates these files for you.It shows differences only between your current copy and the master server.Makes sure you have permissions to write/save these files, if not use the chmod command or run as superuser.

• You need to know the IP address of the server you wish to Sync to:

CLUSTERII/DOUBLESTAR IP at MSSL128.40.70.117

MegaMetaPromote• In your local SDDAS Data directory (e.g. here we assume it is/home/sddas/data/ ?) you have

your Hierarchy. Under each experiment you will have a Database directory, i.e. /home/sddas/data/CLUSTERII/CLUSTER-1/PEACE/Database

• Within the Database directory, each virtual has 4 files in two pairs:{virtual}.HD.DBF {virtual}.HD.NDX - Header & Data list{virtual}.I.DBF {virtual}.I.NDX - VIDF list

• These contain what your computer thinks is available for promotion,as opposed to what is really available for promotion from the master server.

• Do a MegaMetaPromote to update this list from the master server.

Makes sure you have permissions to write/save these files in *ALL* the Database directories, if not use the chmod command or run as superuser.

• This is essential as you can only promote the data that your computer thinks is available...

Download Beta Apps

• SDDAS is made up from a number of separate programs ( ~80!)

• To ensure you have the latest version of each you Download Beta Apps

• Download Beta Apps updates older versions of programs with newer ones from Southwest Research Institute (SwRI).

• Older versions are copied to /usr/local/sddas/backups rather than being over-written. Should the new version be worse (i.e. new bugs) than the last you can then restore the old one from the backups directory.

Makes sure you have permissions to write/save these files in the /usr/local/sddas/backups directory, if not use the chmod command or run as superuser.

• Do not use “Download Applications” – this is rarely updated by SWRI,Always use the Beta Apps.