Windows System Programming

Windows System Programming using

PythonMark Hammond

[email protected]

OReilly Open Source Python ConferenceAugust 1999, Monterey, CA

O’Reilly Python ConferenceAug 24, 1999

Windows System Programming using Python

Slide 2

About this PresentationMost content taken directly from upcoming book for O’Reilly

Python Programming on Win32By Mark Hammond and Andy Robinson

http://www.ora.com/catalog/pythonwin32/



Slide 3

Who this talk is for?Existing Python programmers

– Even those without any Windows experience should follow this without problem.

Although there is not enough time to explain the relevant Windows APIsExisting Windows Programmers

– Even without Python experience, you should immediately see the similarities between your existing language.



Slide 4

What is PythonInterpreted, dynamic high-level languageObviously open source

– Hence we are here.Often used in a similar problem domain to Perl/TclProponents consider readability and maintainability a big strong-

point http://www.python.org



Slide 5

Python and WindowsEach Python version comes with an installer package for

Windows.Standard Python port contains all cross-platform Python

features, but very few Windows specific features.Python for Windows extensions contains many useful Windows

extensions for Python.http://www.python.org/windows



Slide 6

Python Windows ExtensionsIncludes:

– Pythonwin: MFC based GUI environment and IDE/Debugger– win32com: Interfaces Python and COM– win32 Extensions: Interfaces to native Win32 API.

Official releases can be found at http://www.python.org/windowsExtensions home is at http://starship.python.net/crew/mhammond



Slide 7

System Level Programming?For this talk, we define system level programming as

working with low-level features of Windows Files, Pipes, Processes, Threads, Services, Event Log and so forth.

Python and similar languages really not suitable for device-driver type development, and other more system-like Systems Programming!



Slide 8

Why not just use Python?Python has excellent native support for files, processes,

threads etc.These features are typically limited to those defined by ANSI C.

– Many advanced Windows features are not exposed using these interfaces.

– Standard implementation of some of the standard library functions leaves a little to be desired in a Windows environment.



Slide 9

Portable Process Control (1 of 3)

Standard Python functions all work– Just often not quite how we would like!

os.system()import osos.system(“notepad C:\\autoexec.bat”)

Problems– Creates a new console window when run from a GUI.– Waits for process to terminate.



Slide 10


os.execv family– Doesn’t search system path, and doesn’t parse command lines

os.execv("c:\\Winnt\\notepad.exe", \ ("c:\\autoexec.bat",) )

– Does clobber your existing process - the call to os.execv() never returns!



Slide 11


os.popen()>>> file = os.popen("echo Hello")>>> file.read()'Hello\012'– Works fine from Windows NT console programs, but fails miserably from a

GUI!– win32pipe module in the Win32 extensions provides a working replacement.



Slide 12

Better Process Control (1 of )

win32api module provides some high-level, Windows specific functions.

win32api.WinExec()– Very similar to os.system(), but overcomes limitations.– >>> import win32api– >>> win32api.WinExec("notepad")– Optional parameter allows you to specify the Window’s initial state (eg,

minimized)



Slide 13

Better Process Control (2 of 2)

win32api.ShellExecute()– Typically opens “documents” - eg, execute “foo.doc”, and (typically) Word will

open.– Finer control over the new process.– Can also execute arbitrary executables - not limited to documents.– For example, to print a specific document:win32api.ShellExecute(0, "print", \ "MyDocument.doc", None, "", 1)



Slide 14

Ultimate Process Control (1 of 2)

win32process module exposes the low level Win32 API.Full support for CreateProcess, CreateProcessAsUser,

CreateThread etc.Full support for Windows Handles

– Files can be passed as stdin/out/err– Process and thread handles are waitable using the win32event module.



Slide 15

Ultimate Process Control (2 of 2)

Able to set thread and process priority and affinity levels– Very handy for bugs that only appear in multi-processor

machines.Able to do all this for both existing and new processes.Process usage samples included in distribution.



Slide 16

Introduction to our SampleFull sample Windows NT Service

– Provides event log and performance monitor information. Clients connect using Named Pipes

– Less than 75 lines of code for all this functionality.– Still too big to present in one hit

Selected excerpts included in slides Full code on CD, and at

http://starship.python.net/crew/mhammond/conferences/ora99



Slide 17

Portable Files and PipesPython has excellent built-in file support

– Inherits platform stdio support - wont bother discussing them here.Native Windows files only useful for highly-advanced features, such as:

– Overlapped IO– IO Completion Ports– Named Pipes– NT Security requirements



Slide 18

Native Files (1 of 4)

win32file.CreateFile() used for most file operations– Create and open regular files, memory mapped files, etc.– Takes seven parameters - c.f. open()’s two!– Returns a PyHANDLE object

Can be passed to any Python API wrapper expecting a handle.Auto-closed when last reference removed.



Slide 19

Native Files (2 of 4)

Overlapped IO for asynchronous operations– File opened for overlapped IO requires a Windows event object.– All IO operations return immediately.– Event object signalled when IO complete

Great for high-performance servers– Simple to support multiple concurrent file operations per thread.



Slide 20

Native Files (3 of 4) NT Completion ports for even better asynchronous control

– Windows manages associating the completed IO operation with a connection

– More complex to use.– Often requires a state-machine implementation.– Offers excellent performance - Microsoft’s recommended architecture

for scalable, high-performance servers.



Slide 21

Native Files (4 of 4) Full support for NT Security

– Default security can be used by passing None– Many real-world applications require explicit security configuration

Full support for Windows Impersonation– Processes can automatically impersonate the remote pipe client.– Impersonate any user given their password.



Slide 22

File Sample (1 of 2)

Overlapped IO is used self.overlapped = \ pywintypes.OVERLAPPED()# create the event to be used.self.overlapped.hEvent = \ win32event.CreateEvent(None,0,0,None)

Special security for pipes is needed for services sa = win32security.SECURITY_ATTRIBUTES()# Allow full access!sa.SetSecurityDescriptorDacl ( 1, None, 0 )



Slide 23

File Sample (2 of 2)

Create pipe and connect to client pipeHandle = \ win32pipe.CreateNamedPipe(pipeName, openMode, pipeMode, win32pipe.PIPE_UNLIMITED_INSTANCES, 0, 0, 6000, # 6 second timeout. sa)...hr = win32pipe.ConnectNamedPipe(pipeHandle,\ self.overlapped)



Slide 24

Windows NT ServicesSimilar concept to a Unix daemonA few special requirements

– Must respond to asynchronous commands from NT to (e.g.) Shutdown

– Must be capable of reporting status to NTNT has built-in UI for configuring and controlling services.



Slide 25

Python controlling ServicesFull exposure of the Windows NT Service Control Manager API

– Start, Stop, Pause Services, Install or Remove services, etcwin32serviceutil module makes it simple >>> win32serviceutil.StopService("Messenger")(32, 3, 0, 0, 0, 6, 20000)>>> win32serviceutil.StartService(...)>>>



Slide 26

Implementing ServicesSimple to implement Services in Python

– Smallest service is around 16 lines of code!Includes debug support, self-installation, and fully controllable from Windows NTFew more lines needed to make it something useful :-)

Simply sub-class Service base-class, and implement control features– Minimum required is StopService– Trivial to implement most controls



Slide 27

Windows NT Event LogCentral place for programs to log information.

– Ideal for Services - can not present effective GUIsBenefits to programmer

– Built in transaction and thread safety, maximum size ability, etc.Benefits to Administrator

– Central log of messages, and third party tools to help analysis.



Slide 28

Python reading the Event LogComplex native API from win32evtlogSimpler interface from win32evtlogutil

– Define Feeder functiondef DumpRecord(record): print ”Got event ID”, record.EventID

– And feed it!win32evtlogutil.FeedEventLogRecords( \ DumpRecord)Got Event ID -2147483645Got Event ID -2147483645



Slide 29

Python writing the Event LogMore complex than reading - event sources

must be registered.win32evtlog and win32evtlogutil used here tooPython Service framework also supports

simple event log writing



Slide 30

Writing the Event Log SampleOur sample uses the Service Framework functions which

makes it trivialEvents are logged by ID, rather than explicit text.

– Our sample uses a built-in ID to log a “service starting” message import servicemanagerservicemanager.LogMsg( servicemanager.EVENTLOG_INFORMATION_TYPE, servicemanager.PYS_SERVICE_STARTED, (self._svc_name_, ''))



Slide 31

NT Performance MonitorBuilt-in NT tool for monitoring an applications performance.Application must be written to provide this data.API designed for smallest impact on program supplying

data– Moves the burden to the collecting application.– Not trivial to work with



Slide 32

Python reading from PerfMonUseful for overcoming limitations in built-in tool

– For example, sample the data hourly and log to a longer-term database.– Useful for reading standard information about another process.

Process ID, Memory usage, etc.win32pdhutil module for reading data

– Good sample code in the source file



Slide 33

Python supplying PerfMonComplex installation and setup procedure.

– C .h file and custom .ini file necessary at application installation– Supported by the Python Service framework

Quite trivial to use once running– Simply increment a counter - Performance Monitor handles

conversions to required units.



Slide 34

PerfMon SampleInstallation more than we can cover here

– See InitPerfMon() methodWorking with counters trivial

– Increment our connections counter each connection self.counterConnections.Increment()

Perfmon manages converting to connections-per-second automatically.



Slide 35

Our Sample in Detail (1 of 8)

PipeService2.py– The service implementation.

PipeService2_install.hPipeService2_install.ini– Required for performance monitor installation– Most services don’t need this!

PipeServiceClient.py– Sample client to connect to our service.



Slide 36


Service functionality in PipeService class– Base class handles most of the grunt– We simply supply service name and other optional attributes

class PipeService(\ win32serviceutil.ServiceFramework): _svc_name_ = "PythonPipeService" _svc_display_name_ = "A sample Python ... "



Slide 37


We respond to an NT ServiceStop request by telling NT we are stopping, and setting a Windows Event

def SvcStop(self): # Before we do anything, tell the # SCM we are starting the stop process. self.ReportServiceStatus( \ win32service.SERVICE_STOP_PENDING) # And set my event. win32event.SetEvent(self.hWaitStop)



Slide 38


Overlapped IO means we can wait for either a connection, or our Stop request

win32pipe.ConnectNamedPipe(pipeHandle,\ self.overlapped)

... # Wait for either a connection, or # a service stop request. waitHandles = self.hWaitStop, \ self.overlapped.hEvent

rc = win32event.WaitForMultipleObjects(\ waitHandles, 0, timeout)



Slide 39


Install our service simply by executing the service script– If not for Performance Monitor, the command-line would be simple:

C:\Scripts> PipeService2.py install– But PerfMon needs an extra arg

C:\Scripts> PipeService2.py \--perfmonini=PipeService2_install.ini installInstalling service PythonPipeService to ...Service installed



Slide 40


Once installed, we can start the service– Start from Control Panel, or using the script itself

C:\Scripts> python.exe PipeService2.py startAnd start a client test sessionC:\Scripts> python.exe PipeServiceClient.py \ Hi thereThe service sent back:You sent me:Hi there



Slide 41


We will have Event Log records...



Slide 42

Our Sample in Detail (8 of 8) And Performance

Monitor Data.



Slide 43

SummaryPython library flexible, rich and portable enough for many tasksLow-level Windows programming achieved by using extensions

that expose the raw Windows APIThis talk should have shown:

– An overview of how this programming is done in Python.– How simple it is to do complex Windows tasks.



Slide 44

More InformationPython and Python Documentation

– http://www.python.org– http://www.python.org/doc

Python for Windows Extensions– http://starship.python.net/crew/mhammond– http://www.python.org/windows– Reference Manuals and Samples included with the distributions



Slide 45

Thanks for comingMark Hammond

– [email protected]– http://starship.python.net/crew/mhammond

Windows System Programming

Documents