Python lab 5: The Internet and 3D visualisation › ... › media › lab5_talk.pdf · Python libraries One of the great strengths of Python is the huge number of libraries which

Python lab 5: The Internet and 3D visualisation

Dr Ben Dudson

Department of Physics, University of York

25th February 2011

http://www-users.york.ac.uk/∼bd512/teaching.shtml

Dr Ben Dudson Introduction to Programming - Lab 5 (1 of 20)

Python libraries

One of the great strengths of Python is the huge number oflibraries which are available for it

It’s used all over the place in many different projects

This means that many individuals and companies have writtenand released free libraries to make creating programs inPython quicker and easier

So far we’ve seen some of the things Python can do forscientific problems

This lab we’re going to look at just some of the other thingsPython can be made to do


Networking

In addition to reading from files, Python makes it easy to readdata across a network, or from sites on the Internet

For example, the electronics department has a weather stationand makes the data available online

http://weather.elec.york.ac.uk/archive.html

Opening web addresses can be done in a similar way to files:

import u r l l i b 2

addr=” h t t p : / / weather . e l e c . y o r k . ac . uk/ data / v a i s a l a /a r c h i v e s /1110 . t x t ”

f = u r l l i b 2 . u r l o p e n ( addr )l i n e = f . r e a d l i n e ( )f . c l o s e ( )p r i n t l i n e


Networking









Networking









Reading network addresses

Internet address File

import urllib2f = urllib2 .urlopen(”http ://... ”) f = open(”...”, ”r”)line = f. readline () line = f. readline ()f . close () f . close ()

To open an address, we can use the urllib2 module, then justuse urllib2 .urlopen() instead of open()

To get the data, the same methods can be used

When we’re done, both should be closed


Reading network addresses

Internet address File

import urllib2f = urllib2 .urlopen(”http ://... ”) f = open(”...”, ”r”)line = f. readline () line = f. readline ()f . close () f . close ()

To open an address, we can use the urllib2 module, then justuse urllib2 .urlopen() instead of open()

To get the data, the same methods can be used

When we’re done, both should be closed


Reading weather data

Reading from

http://weather.elec.york.ac.uk/data/

vaisala/archives/1110.txt

gives the York weather data from November 2010:

Temp Temp Temp Wind Wind Dir Press.

Ave Min Max Ave Max Ave Min

Date (C) (C) (C) (MPH) (MPH) (Deg) (hPa)

01_11_10 10.3 8.9 12.4 9.2 30.6 227 999.1

02_11_10 12.1 10.7 14.3 14.7 38.1 220 992.1

To turn this data into something useful, we can use the samemethods we did last time for files

This lab has tasks on reading and plotting this data


Reading weather data

Reading from

http://weather.elec.york.ac.uk/data/

vaisala/archives/1110.txt

gives the York weather data from November 2010:

Temp Temp Temp Wind Wind Dir Press.

Ave Min Max Ave Max Ave Min

Date (C) (C) (C) (MPH) (MPH) (Deg) (hPa)

01_11_10 10.3 8.9 12.4 9.2 30.6 227 999.1

02_11_10 12.1 10.7 14.3 14.7 38.1 220 992.1

To turn this data into something useful, we can use the samemethods we did last time for files

This lab has tasks on reading and plotting this data


Websites

You can of course also read data from other websites, so

import u r l l i b 2f = u r l l i b 2 . u r l o p e n ( ” h t t p : / /www. g o o g l e . com” )l i n e = f . r e a d l i n e ( )f . c l o s e ( )p r i n t l i n e

<!doctype html><html><head><meta http-equiv=

"content-type" content="text/html; charset=

ISO-8859-1"><title>Google</title><script>

window.google={kEI:"WLfyTOHWJs-G4gaZ0uGDCw",

...

This is HTML, which tells browsers how to draw web pages. It’s alittle more work to get data out of this, but still possibleFortunately there is a lot more to the internet than just websites...


Websites







...

This is HTML, which tells browsers how to draw web pages. It’s alittle more work to get data out of this, but still possible

Fortunately there is a lot more to the internet than just websites...


Websites







...

This is HTML, which tells browsers how to draw web pages. It’s alittle more work to get data out of this, but still possibleFortunately there is a lot more to the internet than just websites...


Twitter feeds

Twitter makes its data available publicly not only through theweb, but also in more programmer-friendly ways to allow otherprograms to access the data

In particular, it allows us to download tweets in JSON formatwhich has a module for Python. The public timeline data isavailable from

http:// twitter .com/statuses/ public timeline . json

If you have an account then you can also access it throughPython, but here we’ll just look at the publicly available data


Twitter feeds






Twitter feeds






Reading the data

Reading data from is made easy with urllib2

import u r l l i b 2addr=” h t t p : / / t w i t t e r . com/ s t a t u s e s / p u b l i c t i m e l i n e . j s o n ”f = u r l l i b 2 . u r l o p e n ( addr )data = f . r e a d ( ) # Read a l l the dataf . c l o s e ( )

data now contains the tweet information as JSON text:

[{"in_reply_to_status_id":null,"truncated":false,"created_at":

"Sun Nov 28 16:32:21 +0000 2010","geo":null,"favorited":false,

"source":"web","in_reply_to_status_id_str":null,"id_str":

"8921147244544000","contributors":null,"coordinates":null,

"in_reply_to_screen_name":null,"in_reply_to_user_id_str":

To convert this into something more useful, we need to decode(parse) it. We could write some code to do this, but luckelysomeone’s already done all this work - the json module


Reading the data

Reading data from is made easy with urllib2

import u r l l i b 2addr=” h t t p : / / t w i t t e r . com/ s t a t u s e s / p u b l i c t i m e l i n e . j s o n ”f = u r l l i b 2 . u r l o p e n ( addr )data = f . r e a d ( ) # Read a l l the dataf . c l o s e ( )

data now contains the tweet information as JSON text:

[{"in_reply_to_status_id":null,"truncated":false,"created_at":

"Sun Nov 28 16:32:21 +0000 2010","geo":null,"favorited":false,

"source":"web","in_reply_to_status_id_str":null,"id_str":

"8921147244544000","contributors":null,"coordinates":null,

"in_reply_to_screen_name":null,"in_reply_to_user_id_str":

To convert this into something more useful, we need to decode(parse) it. We could write some code to do this, but luckelysomeone’s already done all this work - the json module


Processing Twitter data

Using the json module to process the data:

import j s o na = j s o n . l o a d s ( data )

’a’ now contains the data in a more useful form. We can find outwhat it is using

p r i n t t y p e ( a )<t y p e ’ l i s t ’>

so ’a’ is list of tweets (len(a) = 20): a [0] contains the first, a [1]the second.

To find out how each tweet is stored, try getting the type again

p r i n t t y p e ( a [ 0 ] )<t y p e ’ d i c t ’>

This says that a [0] is a ’dict’ or dictionary type.


Processing Twitter data

Using the json module to process the data:

import j s o na = j s o n . l o a d s ( data )

’a’ now contains the data in a more useful form. We can find outwhat it is using

p r i n t t y p e ( a )<t y p e ’ l i s t ’>

so ’a’ is list of tweets (len(a) = 20): a [0] contains the first, a [1]the second.To find out how each tweet is stored, try getting the type again

p r i n t t y p e ( a [ 0 ] )<t y p e ’ d i c t ’>

This says that a [0] is a ’dict’ or dictionary type.


Dictionaries

Dictionaries are like arrays, except rather than using numbers(0,1,2,...) to refer to the individual values they can use text:

d = d i c t ( [ ] )d [ ’ monday ’ ] = 1d [ ’ t u e s d a y ’ ] = 2p r i n t d [ ’ monday ’ ] −> 1

The names are called keys. To see a list of keys, we can use

p r i n t a [ 0 ] . k e y s ( )[ u ’ f a v o r i t e d ’ , u ’ i n r e p l y t o u s e r i d ’ ,u ’ c o n t r i b u t o r s ’ , u ’ t r u n c a t e d ’ , u ’ t e x t ’ ,u ’ c r e a t e d a t ’ , u ’ r e t w e e t e d ’ , . . . ]

One of these is called “text”, so is probably the actual tweet text:

p r i n t a [ 0 ] [ ’ t e x t ’ ]”AJAX 2−0 VVV, e i n d e l i j k weer 3 punten ”

And we get the result - a Dutch football match score


Dictionaries







And we get the result - a Dutch football match score


Dictionaries







And we get the result - a Dutch football match scoreDr Ben Dudson Introduction to Programming - Lab 5 (10 of 20)

Small Twitter client

We can put this together to make a program which prints out thelast 20 public tweets:

import u r l l i b 2 , j s o naddr=” h t t p : / / t w i t t e r . com/ s t a t u s e s / p u b l i c t i m e l i n e . j s o n ”f = u r l l i b 2 . u r l o p e n ( addr )data = f . r e a d ( )f . c l o s e ( )a = j s o n . l o a d s ( data )f o r tweet i n a :

p r i n t ”At ”+tweet [ ’ c r e a t e d a t ’ ]+ \” u s e r ”+tweet [ ’ u s e r ’ ] [ ’ s c reen name ’ ]

p r i n t ” ” + tweet [ ’ t e x t ’ ]

At Sun Nov 28 16:32:21 +0000 2010 user Niek_vl

AJAX 2-0 VVV, eindelijk weer 3 punten

At Sun Nov 28 16:32:14 +0000 2010 user KachingDeals

*Please RT* How To Make Money Online Fast...


Small Twitter client

We can put this together to make a program which prints out thelast 20 public tweets:

import u r l l i b 2 , j s o naddr=” h t t p : / / t w i t t e r . com/ s t a t u s e s / p u b l i c t i m e l i n e . j s o n ”f = u r l l i b 2 . u r l o p e n ( addr )data = f . r e a d ( )f . c l o s e ( )a = j s o n . l o a d s ( data )f o r tweet i n a :

p r i n t ”At ”+tweet [ ’ c r e a t e d a t ’ ]+ \” u s e r ”+tweet [ ’ u s e r ’ ] [ ’ s c reen name ’ ]

p r i n t ” ” + tweet [ ’ t e x t ’ ]

At Sun Nov 28 16:32:21 +0000 2010 user Niek_vl

AJAX 2-0 VVV, eindelijk weer 3 punten

At Sun Nov 28 16:32:14 +0000 2010 user KachingDeals

*Please RT* How To Make Money Online Fast...


Other sources of data

There are many sources of data online which can be used in yourPython programs. How easily you can extract the data will vary

Infochimps is a big catalogue of online data sources on allsorts of data

http://infochimps.com/

Research Pipeline, variety of datasets

http://www.researchpipeline .com/datasets.html

Many sites can be automated, for example Google translatecan be automated to detect language and translate text

http://www.halotis.com/2009/09/15/google-translate-api-python-script/

Government organisations often make their data available, forexample the World Bank has lots of economic data:

http://data.worldbank.org/
























Visualisation

A very nice module is Visual Python (VPython), which makescreating 3D visualisations much easier

from v i s u a l import ∗b a l l = s p h e r e ( ) # Draw a sphe r e


Visualisation

A very nice module is Visual Python (VPython), which makescreating 3D visualisations much easier

from v i s u a l import ∗b a l l = s p h e r e ( ) # Draw a sphe r e


Visualisation

We can change the size and color of the sphere

from v i s u a l import ∗b a l l = s p h e r e ( r a d i u s =1, c o l o r=c o l o r . r e d )


Visualisation

We can add many different objects, such as boxes

from v i s u a l import ∗b a l l = s p h e r e ( pos =(0 ,4 ,0) , r a d i u s =1, c o l o r=c o l o r . r e d )f l o o r = box ( pos =(0 ,0 ,0) , s i z e = ( 4 , 0 . 5 , 4 ) ,

c o l o r=c o l o r . b l u e )


3D animations

The change in position in a small time δt is given by δx = vδt fora velocity v . We can use VPython vector to represent the velocity:

b a l l . v e l o c i t y = v e c t o r (0 ,−1 ,0)dt = 0 . 0 1

To change the position of the ball, we can just modify ball .pos:

b a l l . pos = b a l l . pos + b a l l . v e l o c i t y ∗ dt

In the same way, the change in velocity in a small time δt is givenby δv = aδt where a is the acceleration. In this case we want toaccelerate downwards (in y) due to gravity:

b a l l . v e l o c i t y . y = b a l l . v e l o c i t y . y − 9 . 8∗ dt

This will just change the position and velocity once. To create ananimation, we can move the ball repeatedly in a loop.


3D animations









3D animations









3D animations









3D animations



whi le True : # Loop f o r e v e rr a t e ( 1 0 0 ) # Lim i t to 100 t imes pe r secondb a l l . pos = b a l l . pos + b a l l . v e l o c i t y ∗ dtb a l l . v e l o c i t y . y = b a l l . v e l o c i t y . y − 9 . 8∗ dt

Each time this loops it updates the ball position and velocity.Unfortunately the ball will just go straight through the floor andkeep going downwards. We need to detect when the ball hits thefloor


3D animations



whi le True : # Loop f o r e v e rr a t e ( 1 0 0 ) # Lim i t to 100 t imes pe r secondb a l l . pos = b a l l . pos + b a l l . v e l o c i t y ∗ dti f b a l l . y < b a l l . r a d i u s :

b a l l . v e l o c i t y . y = −b a l l . v e l o c i t y . ye l s e :


When the ball reaches the floor, this reverses the direction of thevelocity so the ball bounces back up again.


Visual Python

Using VPython, it is possible to produce visualisations andanimations of physical situations. You can add arrows and plotcurves


Summary

Python can do many more things than just calculations

It is ideal for tasks involving quickly putting togetherprograms to manipulate and visualise data

Reading from internet sources is not much more complicatedthan from files. Built-in modules such as urllib2 and jsonsimplify the process

Visual Python makes it straightforward to visualise andanimate physical situations

Many drawing functions including arrow, box, cone, curve,cylinder, ellipsoid, helix, points, pyramid, ring, and sphere.

http://www-users.york.ac.uk/∼bd512/teaching.shtml


Python lab 5: The Internet and 3D visualisation › ... › media › lab5_talk.pdf · Python libraries One of the great strengths of Python is the huge number of libraries which

Documents