FINITE PERMUTATION GROUPS AND FINITE SIMPLE ...

FINITE PERMUTATION GROUPS AND FINITESIMPLE GROUPS

PETER J. CAMERON

1. Introduction

In the past two decades, there have been far-reaching developments in theproblem of determining all finite non-abelian simple groups—so much so, that manypeople now believe that the solution to the problem is imminent. And now, as Icorrect these proofs in October 1980, the solution has just been announced. Ofcourse, the solution will have a considerable effect on many related areas, bothwithin group theory and outside. The purpose of this article is to consider the theoryof finite permutation groups with the assumption that the finite simple groups areknown, and to examine questions such as: which problems are solved or solvableunder this assumption, and what important problems remain?

Let us begin with an example. The best-known problem in finite permutationgroup theory is that of deciding whether there are any 6-transitive groups other thanthe symmetric and alternating groups. It was conjectured by Schreier that the outerautomorphism group of a finite simple group is soluble. Such a conjecture is easilychecked if the list of simple groups is known. (Indeed, at present it seems very likelythat Schreier's conjecture will be proved in this way rather than directly.) Wielandt[67] reduced the first of these problems to the second: his result, refined by Nagao[46] and O'Nan [48], asserts that a 6-transitive group must be symmetric oralternating, provided that the composition factors of its proper subgroups havesoluble outer automorphism groups. However, unless a direct proof of Schreier'sconjecture can be found, this is the wrong way to settle the question. As we shall see,the determination of the finite simple groups enables us, with more effort, todetermine all the 2-transitive groups. By inspection, none, except the symmetric andalternating groups, is 6-transitive.

In order to discuss the consequences of knowing the finite simple groups, wemust say something about the sense in which they are known.

The multiplication tables of the finite simple groups are "recursive", in the sensethat if they are encoded by Godel numbers in some way, the resulting set M ofnumbers is recursive. This means that we could in principle construct a machinewhich would decide, given a natural number n, whether or not n e M. By the resultsof Davis, Matijasevic and Robinson on Hilbert's tenth problem (see [18]), we knowthat M can even be expressed as the set of positive values of a polynomial. However,even if such a polynomial were explicitly known, it would not constitute asatisfactory solution to the classification problem. Each group must be knownsufficiently well that questions about its automorphisms, permutationrepresentations, local subgroups, and so on, can be answered. In this article, we willoften invoke the following hypothesis (S). Its use could be compared to that of thecontinuum hypothesis or the Riemann hypothesis in other parts of mathematics(though the analogy does not run very deep).

Received 24 April, 1980.

[BULL. LONDON MATH. S O C , 13 (1981), 1-22]

2 PETER J. CAMERON

(S) A non-abelian finite simple group is an alternating group, a group of Lie type, orone of finitely many sporadic groups.

The alternating groups require little comment: there is one such simple group foreach degree n ^ 5. Groups of Lie type are the Chevalley groups [15] and twistedanalogues discovered by Steinberg [60], Suzuki [61], Ree [53, 54] and Tits [63]. Itwill be convenient to divide them into two types: the classical linear, symplectic,unitary and orthogonal groups (which fall into six families, each parametrised by adimension n and a field order q), and the exceptional groups (comprising ten familieseach parametrised by a field order). Sometimes, we assume a more refined version of(S) in which the only sporadic groups are those whose existence is known orsuspected at present (see [37] for the list), and these are "known" in the sensepreviously described; it will be clear how to modify the arguments should finitelymany more groups be discovered.

Obviouly our knowledge of these groups is a function of time. Examples in thepaper illustrate how the present knowledge is used. It should be stressed that themost important problem to be faced now is that of obtaining more informationabout these groups, especially those of Lie type.

To make the paper self-contained, Sections 2 and 3 outline the elementary theoryof permutation groups, explaining the central role played by the primitive groups.The next section discusses a theorem stated by Michael O'Nan and Leonard Scott atthe symposium on Finite Simple Groups at Santa Cruz in the summer of 1979,concerning the socle of a primitive group. (Although the proof of this theorem iselementary, and all or part of it may have been known previously to some people, Iknow of no explicit reference in the literature. The proceedings of the Santa Cruzsymposium are to appear in the Proceedings of Symposia in Pure Mathematics,published by the American Mathematical Society; but the theorem quoted here istaken from the preliminary version circulated to participants. I would like to thankthose people who made their copies available to me.) The O'Nan-Scott theoremenables many questions about primitive groups G to be reduced to the case whereT ^ G ^ Aut (T) for some non-abelian simple group T; this is where hypothesis (S)comes into play.

The next four sections discuss the implications of (S) for some classical problems:the determination of primitive groups of small rank and, in particular, all doublytransitive groups; the orders and degrees of primitive groups not containing thealternating group; and remarks on the problem of finding all primitive groups.Section 9 is more in the spirit of traditional permutation group theory, dealing withproblems which are not solved or trivialised by (S). Section 10 discusses briefly someissues raised by the problem of computer-aided recognition of permutation groups; itis based on discussions with John Cannon. The final section considers related areas,especially automorphism groups of combinatorial structures and infinitepermutation groups.

The proof of (S), when it is completely checked, will be an achievement of greatmagnitude, involving thousands of pages of group theory written by an army ofresearchers. However, it raises philosophical problems of a very different kindfrom those posed by a recent well-publicised breakthrough, the proof of the four-colour conjecture by Appel and Haken [1]: human beings are less reliable thanproperly-programmed machines. It is certain that the first published proof of (S),spread over many journals'and theses, will contain mistakes. Thus, it is important

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 3

that the work of revision and the search for better proofs should continue. The sameapplies to the material of Sections 5-7 of this paper, where it is also desirable foraesthetic reasons that proofs should be found which do not depend on (S). Nowhereis this more so than in the classification of doubly transitive groups.

The material in this paper grew out of a discussion of the aftermath of the SantaCruz symposium; those who attended the meeting will recognise my debt to MichaelO'Nan and Leonard Scott. I would also like to record my gratitude to the Universityof Sydney, for giving me the opportunity to deliver lectures in which the ideas herepresented were worked out; to the audience at those lectures, for their help in thediscussions; to John Cannon, for introducing me to the material in Section 10; and toPeter Neumann, who suggested a large number of improvements to the paper.

2. Transitivity and primitivity

A permutation group on a set ft is simply a subgroup of the symmetric group onQ. It is convenient, however, to have a more general concept, an action of G on ft,which may be regarded as a homomorphism from G into the symmetric group. (Ofcourse the image, which we write as Gn, is a permutation group, and can be identifiedwith G if the action is faithful.) The image of a e ft under the permutationcorresponding to g e G will be written ag.

We say that G acts transitively on ft if, for any a, /? e ft, there exists g e G withag = p. In general, the relation ~ on ft defined by the rule that a ~ P if ocg = P forsome g e G is an equivalence relation, whose equivalence classes are called orbits; Gacts transitively on each orbit. We call the group G* of permutations induced on anorbit O by G a transitive constituent of G. Thus, the study of arbitrary permutationgroups "reduces" to that of transitive groups. (However, it should be noted that G isnot determined by its transitive constituents. It is contained in their cartesianproduct, and is in fact a subcartesian product: that is, the projection map associatedwith each factor maps G onto that factor; but more cannot be said in general.Subsequent "reductions" will involve a similar loss of information.)

If H is a subgroup of G, there is a transitive action of G on the set (G : H) of rightcosets of H, by right multiplication: (Hx)g = H(xg). Let ft be a set on which G actstransitively, and Ga = {g e G | <xg = a} the stabiliser of the point a e Q . Then Ga is asubgroup of G, and there is a bijection $ : Q -> (G : GJ satisfying (/?</>)# = (Pg)<t> forall P e ft, g € G. (It is defined by P<f> = {g e G | ag = /?}.) Thus the study of transitivegroups is equivalent to that of coset spaces.

A permutation group G is called semiregular if Ga = 1 for all a e ft, and regular ifit is transitive and semiregular. For regular groups G, the bijection <f> identifies ftwith G, giving the right regular representation of G. It can be shown that apermutation group is transitive if and only if its centraliser in the symmetric group issemiregular, and vice versa (Wielandt [68], page 9). In particular, the centraliser of aregular group is regular. Specifically, the centraliser of the right regularrepresentation is the left regular representation, in which a group element g acts asleft multiplication by g~x.

Suppose G has a normal subgroup N which is regular on ft. Then, as above, wemay identify ft with N. If a eft is mapped to the identity element of N, then theactions of Ga on ft and on N by conjugation agree under this identification: that is,for fi e ft, g e Ga, (Pg)4> = g~x(p(f))g (Wielandt [68], page 27).

4 PETER J. CAMERON

Let G act on Q. Then G is transitive if the only G-invariant subsets of Q aretrivial (Q and the empty set). Consider now the related assertion that the onlyG-invariant partitions of Q are trivial (the partition into singletons and the partitionwith a single part). Obviously this condition implies that G is transitive provided that|Q| > 2. A transitive group with no nontrivial invariant partitions is called primitive.Also, a nontrivial G-invariant partition is called a system of imprimitivity, and itsparts are blocks (of imprimitivity). A non-identity normal subgroup of a primitivegroup is transitive (Wielandt [68], page 17)—indeed, if N is normal in G, then thepartition of Q, into orbits of N is invariant under G.

Primitivity of the action on a coset space can be recognised group-theoretically:the action of G on (G : H) is primitive if and only if if is a maximal subgroup of G(Wielandt [68], page 15).

For a natural number fc < |Q|, a permutation group G on Q is k-transitive if itsinduced action on the set of ordered /c-tuples of distinct elements of Q is transitive.The symmetric group of degree n is n-transitive, and the alternating group is (n — 2)-transitive. Apart from these examples, no known finite group is 6-transitive, and onlytwo (the Mathieu groups M1 2 and M24) are 5-transitive. Note that a 2-transitivegroup is primitive (Wielandt [68], page 20).

3. Wreath products

Let C be a group, and D a permutation group on a set A. The wreath productC Wr D is the split extension of a base group CA (the cartesian product of |A| copiesof C) by D, where D acts on CA by permuting the factors as it does the elements of A.Identifying CA with the set of functions / : A -> C, we have d~ifd(d) = f(Sd~i) fordeD,de&.

Suppose C acts on a set F. Then there are two natural actions of C WrD.

1. The imprimitive action on F x A. The base group acts on the first coordinate,by the rule (y, d)f = (y/(<5), S), and D acts on the second coordinate in the usualway. As the name suggests, this action is imprimitive if |F|, |A| > 1: the partitionF x A = [J F x {3} is a system of imprimitivity. It is universal in the following sense.

Se D

PROPOSITION 3.1. Let G act imprimitively on fi. Let F be a block of imprimitivity,A an index set for the parts of the corresponding system of imprimitivity, C = Gp(where Gr denotes the setwise stabiliser of F), and D = GA. Then Q can be identifiedwith F x A in such a way that Gn is a subgroup of C Wr D (in its imprimitiverepresentation).

Thus, if G is imprimitive, it is built from smaller permutation groups. If Q is finite,we may continue refining the decomposition to obtain primitive components for G.However, there is no Jordan-Holder theorem here, since different refinements mayyield different primitive components. (Consider, for example, the right regularrepresentation of the symmetric group S3. If we choose for F the subgroup of order3, then we have C = C3, D = C2, where Cn is the cyclic group of order n in itsregular representation; whereas if F is a subgroup of order 2, then C = C2 butD = S3 in its natural action on 3 letters.) It is also true that the primitivecomponents do not determine the group uniquely; the comment concerning thereduction of arbitrary groups to transitive ones applies here too.

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 5

2. The product action on FA, where the base group acts coordinatewise (that is, if4> e F A , / e CA, then (</>/)(<5) = (f){S)f(d)), and D acts by permuting the coordinates

= (\>{dd~x) for $ 6 TA, d e D). This action is usually primitive:

PROPOSITION 3.2. Suppose D is transitive on A, and C is primitive but not regularon F. Then C WrZ), in its product action on FA, is primitive.

Note that a primitive regular group is necessarily cyclic of prime order (Wielandt[68], page 17), since the trivial subgroup of such a group is maximal. If |A| > 1, thehypotheses are also necessary for primitivity.

4. A theorem of O'Nan and Scott

Let G be a primitive permutation group on a finite set Q. Suppose that N is aminimal normal subgroup of G. Then N is transitive. Now the centraliser CG(N) isalso a normal subgroup. If CC{N) ^ 1, then CC{N) is transitive, whence N and CG{N)are both regular; they are equal or distinct according as N is abelian or not. In thiscase N and CC(N) are minimal normal subgroups, and G has no further minimalnormal subgroups (since such subgroups centralise each other). Moreover, N andCG{N) are isomorphic, since they are right and left regular representations of thesame group. If, on the other hand, CG(N) = 1, then N is the unique minimal normalsubgroup of G, and G is isomorphic to a subgroup of Aut(iV), the automorphismgroup of N. (For, in any group G with normal subgroup N, the action of G on N byconjugation provides a homomorphism from G into Aut (NJ with kernel CG(N).) Ineither case, we see that the socle of G (the product of its minimal normal subgroups)is a direct product of isomorphic simple groups.

Let S,S!,..., Sh be groups, and for 1 ^ i ^ h let 0 , : S -• S, be an isomorphism.The diagonal subgroup of Sx x...xSh (relative to the given isomorphisms) is thesubgroup D = {(scf) t , . . . , s(f)h) \ s e S}.

THEOREM 4.1 (O'Nan-Scott). Let G be a primitive permutation group on Q, withdegree n and socle N. Then one of the following occurs.

(i) N is elementary abelian of order pd and regular, n = pd, where p is prime andd ^ 1.

(ii) N = Tx x ...xTm, where Tl,...,Tm are all isomorphic to a fixed simplegroup T. Moreover, either

(a) T is the socle of a primitive group Go of degree n0, and G ^ G0Wr5m

(with the product action), where n = n%; or

(b) N n Ga = D, x... x Dh where m = kl for some k, Dt is the diagonalsubgroup of T(i_X)k+{ x... x Tik, and n = \T\{k~1)l.

In view of the importance of this theorem for the next four sections, a proof willbe outlined. If N is abelian then its simple factors are cyclic of prime order, and N iselementary abelian; so (i) holds. Otherwise, (ii) follows from the reasoning precedingthe theorem. Since Ga is a maximal subgroup of G, it follows that N n Ga is amaximal Ga-invariant subgroup of N. If N n Ga projects onto one factor of the directproduct, then it projects onto all, and it is a product of diagonal subgroups; it can be

6 PETER J. CAMERON

verified that (ii)(b) holds. Otherwise, the projection St of N n Ga into Tx is amaximal (NG{TX) n Ga)-invariant subgroup of Tl; its projection S, into 7] is theimage of Sj under an isomorphism from Tx to 7> and N n Ga is the direct product ofthe projections Sx,..., Sm. (For, if R{ is a ( ^ ( T J n Ga)-invariant subgroup of Tu

and Ri the corresponding subgroup of 7], then R{x ...x Rm is a Ga-invariantsubgroup of N.) Thus (ii)(a) holds.

Remarks. 1. In case (i), N is the additive group of a vector space V of dimensiond over the field GF(p). Identifying Q with V, we see that Ga is a group ofautomorphisms of N, that is, a subgroup of the general linear groupGL (V) = GL (d, p), Moreover, Ga is irreducible, since any invariant subspace wouldbe a block of imprimitivity. Conversely, if H is an irreducible linear group on a finitevector space V, then the split extension G = VH is a primitive group falling undercase (i) of the theorem.

2. In case (ii)(b), the permutation group induced by G on {r i 5 . . . , 7̂ ,,} has{7l 5 . . . , 7̂ } as a block of imprimitivity. The group induced on the set of blocks istransitive. Also, the stabiliser of a block leaves invariant no nontrivial partition ofthat block. If it is the trivial group (with k = 2) then G has two minimal normalsubgroups; otherwise, it is primitive, and N is the unique minimal normal subgroupof G.

3. A group of type (ii)(b) with / > 1 is a subgroup of Go Wr S, (with the productaction), where Go is primitive of type (ii)(b) with / = 1.

Thus the study of primitive groups is "reduced" to the following four parts:

(i) irreducible linear groups over finite prime fields;

(ii) the product action of wreath products;

(iii) groups of type (ii)(b) with / = 1;

(iv) primitive groups with nonabelian simple socle.

(Of course, the earlier remarks about "reductions" apply here.)A complete classification of irreducible linear groups appears somewhat remote,

and may not be very helpful if it were known. So we often ignore this case, merelyremarking that for some problems (examples of which are given below) it presents nodifficulties. In a similar way, the second and third of the above topics can often behandled easily in a particular case. The crucial point at which our hypothesis (S)needs to be used is in the fourth problem, the determination of primitive groups withsimple socle. This is equivalent to determining all maximal subgroups of groups Gsatisfying T ^ G ^ Aut(T), where T is a nonabelian simple group. (This questionasks for more than a determination of the maximal subgroups of T, since there canbe maximal subgroups H of G for which H n T is not maximal in T.)

5. Rank and multiple transitivity

The rank of a transitive permutation group G on fi is the number of orbits of Gin its action on Q x Q . (This is equal to the number of orbits of Ga on Q, for a e Q,

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 7

(see Section 9).) Thus, for |Q| > 1, G has rank at least 2, with equality if and only if itis 2-transitive. (The diagonal {(a, a) | a e 0} is one G-orbit.)

Is there any relation between the rank of a primitive group and the structure ofits socle? Clearly no such relation is possible in case (i) of Theorem 4.1, since thesplit extension of V by GL (V) is 2-transitive for any finite vector space V. In case (ii),however, we have the following result.

PROPOSITION 5.1. Let G be a primitive group of rank r on Q, whose socle is adirect product ofm nonabelian simple groups. Then m ^ r - 1 . Moreover, assuming (S),in case (ii)(b) of Theorem 4.1, the order of each simple group (and hence the degree andorder of G) is bounded by a function ofr.

Proof. It is not difficult to show that, if Go is a transitive group of rank r0, then

the rank of G0WrSm (in its product action) is r' = I ° I; and of course, if

G ^ Go Wr Sm, then r ^ r'. Thus, in case (ii)(a), given r, both m and r0 are bounded,and m ^ r — 1.

Let h denote the number of orbits of the group K of permutations of T generatedby automorphisms and inversion (the map t\—• t"1). Then a group of type (ii)(b)with / = 1 has rank r satisfying r ^ l + (/i —l)Qm] (see below). So, given r,both h and m are bounded. Since T contains elements of at least four different orders,we have h ^ 4 and m < r — l.

The largest subgroup of Sn in which Tx x . . .x Tm is normal, in case (ii)(b) with/ = 1, is an extension of Ti x . . .x Tm by Out[T)xSm, where Out(T) is the outerautomorphism group of T; the automorphisms act in the same way on each factor,and the symmetric group permutes the factors. The subgroup T 1 x . . .xT m _ 1 isregular, though not normal, so Q can be identified with Tm~x. Now the stabiliser ofthe identity is Aut {T)xSm, and is generated by elements of three types:

(i) automorphisms of T, acting in the same way on each coordinate;

(ii) permutations of the coordinates (elements of Sm_x); and

(iii) the map (tl,...,tm-l)\-+(t;l,tilt2,...,tiltm_l)

(induced by the transposition (lm) e Sm). Thus, as t runs through a set ofrepresentatives of the nontrivial K-orbits, the elements (t,t, ...,t,l,...,i) (with ientries t, for 1 ^ i ^ [^m]), lie in different suborbits. This gives the boundr ^ l + (h — \)[jm~\. It can be attained if m = 2, but it is very far from bestpossible for m > 2. For example, when T = A5, we have h = 4, but the group(A5)3(C2 x S3) of degree 3600 has rank 17.

To prove the last assertion in the proposition, it is necessary to show that theorder of the simple group T is bounded by a function of h. This is true for the knownsimple groups. I do not know a proof independent of (S). Clearly we may replace hby the number hx of orbits of Aut (T) on T. The question is related to the well-knownfact that the order of any finite group is bounded by a function of the number of itsconjugacy classes (the orbits of the group of inner automorphisms). See Burnside [9],page 461.

As a corollary of Proposition 5.1, we obtain the following result of Burnside ([9],page 202); more properly, Proposition 5.1 should be seen as a generalisation ofBurnside's theorem.

8 PETER J. CAMERON

PROPOSITION 5.2. A 2-transitive group has a unique minimal normal subgroup,which is elementary abelian or simple.

From this and the work of Huppert, Hering, Maillet, Howlett, Curtis, Kantor,Seitz, and others, comes a major consequence of (S).

THEOREM 5.3 (S). All finite 2-transitive groups are known.

Proof. Suppose first that the minimal normal subgroup N of such a group G iselementary abelian. Then Ga is a subgroup of GL (d, p), transitive on nonzero vectorsof the vector space. The soluble 2-transitive groups were determined by Huppert[36], so we may assume that Ga is insoluble. Hering [26] showed that Ga has aunique nonabelian composition factor, and subsequently [28] he examined theknown simple groups and determined all such situations in which each could occuras such a composition factor.

In the other case (when N is simple), we have G ^ Aut (N). If N is an alternatinggroup Ak, then G = Ak or Sk (for k ^ 5, k ± 6); all the 2-transitive representationsof these groups were determined by Maillet [43]. When N is a group of Lie type, thecorresponding problem was solved by Curtis, Kantor and Seitz [17] (see alsoHowlett [35]). It is implicit in (S) that the sporadic groups can be handled by ad hocarguments.

We illustrate one method of proving theorems of this type, by dealing with thesymmetric groups. Suppose G = Sk, and G acts 2-transitively on Q. There is aconjugacy class C of /̂c(/c — 1) elements of order 2 in G (the transpositions). Choosea e Q. For each further point P of Q, there is a member of C which interchanges aand p. If H = Ga, then \Sk: H\ = |Q| < 1 + |C| = |(/c2 -k + 2). Comparing this withknown lower bounds for the index of subgroups of Sk (see the next section), weconclude that H = Sk.1 with a few known exceptions.

We list here the simple groups T which can occur as minimal normal subgroupsof 2-transitive groups of degree n, according to Theorem 5.3. The number k is themaximum degree of transitivity of a group G with socle T.

T n k Remarks

An,n^5 n n Two representations if n = 6PSL(d, q), d 2 2 ( ^ _ ! ) / ( , _ i) 3 if rf = 2 (d, 9 ) ^ (2, 2), (2, 3)

2 if d > 2 Two representations if d > 2PSU(3,g) <73 + l 2 q>22B2(q) (Suzuki) q2 + l 2 q = 22a+1>22G2{q)(Ree) q3 + l 2 q = 32 a + 1 > 3PSp(2</,2) 2

2d-1+2d-1 2 d>2PSp(2</,2) 1u-\_1i-\ 2 d>2PSL(2, 11) 11 2 Two representationsPSL(2,8) 28 2An 15 2 Two representationsM, , (Mathieu) 11 4M M (Mathieu) 12 3M1 2 (Mathieu) 12 5 Two representationsM2 2 (Mathieu) 22 3M 2 3 (Mathieu) 23 4M2 4 (Mathieu) 24 5HS (Higman-Sims) 176 2 Two representationsCo3 (Conway) 276 2

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 9

Notes. 1. There is one duplication in the table, explained by the isomorphismbetween A5 and PSL(2,4). The additional isomorphisms PSL(2,5) = A5,PSL(2,7)^PSL(3,2), PSL(2,9) s A6, and PSL(4,2)^X8 give examples ofabstract groups with more than one 2-transitive permutation representation, inaddition to the examples noted in the remarks and the cases PSL(2, 8), PSL(2,11),PSp{2d,2)(d > 2),A1 and M n .

2. The socle T is /c-transitive in all cases except T = An ((n — 2)-transitive),T = PSL(2, q), q odd (2-transitive), and T = PSL(2, 8), degree 28 (1-transitive).

3. The groups PSL(2,2)^S3, PSL(2, 3) s X4, PSU(3,2) and 2B2{2) aresoluble, with degrees 3, 4, 9 and 5 respectively; 2G2(3) has socle PSL(2, 8), of degree28, as a subgroup of index 3; and PSp(4, 2) has socle of index 2 which acts as A6 ofdegree 6 and PSL(2,9) of degree 10 (so that PSp(4,2) s S6).

4. Apart from the fact that any simple group of degree n is contained in An, theinclusions among groups T in the table are as follows:

PSL(2,11) (degree 11) ^ M n (degree 11);

PSL(2, 11) (degree 12) ^ M12;

M u (degree 12) ^ M12;

A, (degree 15) < PSL(4, 2);

PSL(2,23)^M24;

PSL(2, 8) (degree 28) ^ PSp(6, 2) (degree 28);

PSU(3, 3) (degree 28) ^ PSp(6, 2) (degree 28).

Note also that PSL(2, 7) is contained in the holomorph of C2, a 3-transitive group ofdegree 8 with elementary abelian socle.

5. Some of these representations correspond to familiar geometrical objects.For example, PSL(d, q) acts on the points and hyperplanes of the projective spacePG(d — l,q); PSU(3,<j) acts on the points of a Hermitian variety in PG(2,q2);PSp(6, 2) (degree 28) is the "group of the bitangents to a nonsingular quartic curve";and PSp(2d,2) acts with orbits of size 22 d"1+2d"1 and 2 2 d " 1 -2 d " 1 on thequadratic forms which polarise to a given symplectic form in 2d variables overGF(2).

COROLLARY 5.4 (S). A 6-transitive finite permutation group is symmetric oralternating.

Remark. For this and many other consequences of Theorem 5.3, it is notnecessary to know in detail the 2-transitive groups having elementary abelian socle.It is very easy to prove directly (see Wielandt [68], page 28) that S4 is the only4-transitive group with a regular normal subgroup. In determining the 3-transitivegroups, we may use either the theorem of Hering, Kantor and Seitz [29] (seeTheorem 10.2) or the determination by Cameron and Kantor [13] of 3-transitivegroups with elementary abelian socle. Both of these theorems are independent of (S);

10 PETER J.CAMERON

the group theory used in the proof of the second is elementary, in the sense that ituses nothing more than Sylow's theorem.

We note two further results of the same type as Theorem 5.3.

THEOREM 5.5 (Kantor-Liebler [41]). All primitive permutation groups of rank 3whose socle is a classical simple group are known.

A similar result for groups of rank at most 5 whose socle is an alternating groupwas given by Bannai [4].

THEOREM 5.6 (S). Given a positive integer r, all but finitely many primitivepermutation groups of rank r with simple socle are known.

This was proved by Bannai [4] when the socle N is an alternating group, and byKantor [40] (depending on results of Seitz [55]) in case N is of Lie type; the sporadicgroups are irrelevant here. The idea of the proof is that, as above, if G is primitiveand has a conjugacy class C, then Ga has an orbit of size at most \C\, whence G hasdegree at most (\C\r- 1)/(|C| - 1 ) (Wielandt [68], page 47; see Section 9). This upperbound is confronted with a suitable lower bound for the index of a subgroup of aknown simple group (see, for example, Kantor [40]).

6. Order

A very old question in permutation group theory is: how large can a primitivegroup G of degree n and not containing the alternating group be? This question wasfirst examined by Jordan and his contemporaries. Perhaps the most memorableresult from this period is Bochert's theorem, asserting that \Sn: G\ ^ [2(1 +1)]! Thisis sufficient for the illustrative argument in the proof of Theorem 5.3; but it does notreduce the dominant term (nlogn) in the asymptotic expansion of log |G| below thatderived from the trivial bound \G\ ^ n\.

Using methods similar to Jordan's (that is, bounding the orders of Sylowp-subgroups for all primes p less than n), Wielandt [69] was able to give asubstantially better bound: \G\ < c", provided that G is not 2-transitive. (Wielandtgave c = 24.) Praeger and Saxl [50] extended the result to all primitive groups, atthe same time reducing Wielandt's constant to 4.

Recently, Babai [3] has obtained a substantial improvement: if G is not2-transitive, then \G\ ^ ,i4"l/2|og". This result is close to best possible, since both thesymmetric group Sm acting on unordered pairs (with n = jm(m — 1)) and the groupSmWrS2 (with n = m2) have order roughly n"1'2. (More accurately,log |G| ~ n1/2logtt in each case.)

Babai's method was quite different from Wielandt's. A base for a permutationgroup is an ordered sequence of points which is fixed pointwise only by the identity.Any group element is uniquely determined by its action on a base. For this reason,bases are used extensively in machine computations involving permutation groups(especially in the hands of Sims [59]), and in theoretical investigations ofcomputational complexity. It was in the latter context that Babai showed that aprimitive but not 2-transitive group G of degree n has a base of cardinality at most4n1/2logn; from this, his bound for \G\ follows.

It is possible to do much better by using the "classification theorem".

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 11

THEOREM 6.1 (S). There is a {computable) constant c with the property that, if Gis a primitive permutation group of degree n, then one of the following holds:

(i) G has an elementary abelian regular normal subgroup;

(ii) G is a subgroup of Aut(T)WrS, containing Tl, where T is either analternating group acting on k-element subsets, or a classical simple groupacting on an orbit of subspaces or (in the case T = PSL(d, q)) pairs ofsubspaces of complementary dimension, and the wreath product has theproduct action;

(iii) \G\ ^ nclos{osn.

Remark. Groups under (i), and those under (ii) for which T is a classical group,satisfy \G\ ^ ncl°8" for some constant c'. The order of SmWrS, (where Sm acts on/c-element subsets), for fixed k and / and large m, is bounded above and belowby expressions of the form nc*"1/u, where the number c* in each bound depends on kand /.

We outline the proof of Theorem 6.1, which is by induction on n. Assume thetheorem true for groups of degree smaller than n. If G ^ G0WrS, where Go isprimitive of degree n0 and / > 1, the inductive hypothesis shows that either Go fallsunder (ii) with / = 1 (in which case G itself falls under (ii)) or \G0\ ^ n£logl08"0. Now

<- cl log log HO//

_ ^ c log log no + log//log no

< nclO8log"0+elog' provided that c ^ I/log 5

_ j|ClOglog/l

If G is of type (ii)(b) in Theorem 4.1 with / = 1, then n = \T\k~l and

\G\ ^ |7f|Out(T)|ifc!.

From (S) we deduce that |Out(T)| ^ \T\; since k + l ^ 3(/c — 1) we have

\G\ ^ n2kk ^ n3+lo8log".

Thus we may assume that the socle T of G is simple. We distinguish four cases,according as T is an alternating group, a classical group, an exceptional group of Lietype, or a sporadic group. We may transfer any finite number of groups from the firstthree categories to the fourth, and any finite number of subfamilies from the secondto the third (where a subfamily here consists of the classical groups of given type anddimension over arbitrary finite fields).

Suppose that T = Am. If m ± 6, then G ^ Aut(T) = Sm; so regarding A6 assporadic, G = Sm or Am. Let H be the stabiliser of a point. If H is intransitive on{l, . . . ,m}, then its maximality ensures that it is the stabiliser of a /c-subset of{1, . . . , m) for some k, so that (ii) holds. If H is transitive but imprimitive then it is thestabiliser of a partition of {1, . . . , m] into k parts of size / for some k, I. There is aconstant b with the property that the number n of such partitions is at least bm (withb > 1); since \G\ < mm, conclusion (iii) of the theorem follows. If H is primitive,

12 PETER J. CAMERON

Bochert's bound n ^ |Q(m + l)]! is more than good enough to show (iii).

If T is a classical group other than PSL(d, q), PSp(4, q) (for q even) or PQ+(8, q),then Aut(T) (and hence G) consists of semilinear transformations of the underlyingvector space. (In the exceptional cases, there are additional automorphisms, inducedby dualities of the projective space for PSL(d, q) and triality maps of the quadric forPQ+(8,g); see Todd [64] for a geometric description of the additionalautomorphisms of PSp(4, q) for q even.) If H is reducible, it is the stabiliser of asubspace; if it is irreducible, bounds due to Kantor [40] show that \G\ ^ n5 if thedimension of T is sufficiently large. In the case T = PSL(d, q), a similar argumentapplies. If H is not contained in the group PFL(d, q) of semilinear transformationsand H n PTL(d, q) fixes a subspace, then H fixes a pair of subspaces. Groups ofbounded dimension can be regarded as exceptional.

If T is an exceptional group of Lie type of rank greater than 1, then T contains asubgroup PSL(2, q) or SL(2, q). For q ± 9, a non-trivial permutation representationof this subgroup has degree at least q (a result of Galois [23] for q prime andDickson [19] in general); so n ^ q. Also \G\ ^ qk for a fixed constant k; so \G\ < nk.The case q = 9 can be treated similarly, or the finitely many exceptional groups overGF(9) regarded as sporadic. Groups of rank 1 can be dealt with directly.

Finally, the sporadic groups satisfy (iii) if c is large enough.

Remark. We have pulled ourselves up by the bootstraps, using Bochert's boundto derive a much stronger result. The power of Kantor's bounds for the orders ofirreducible subgroups of classical groups suggests that results even stronger thatTheorem 6.1 may be possible.

7. Degree

A problem which may be easier than that of determining all primitive groups is todecide for which integers n it is true that a primitive group of degree n must besymmetric or alternating.

Let D be the set of integers with this property. The first few members of D are 1,2, 3, 4, 34, 39, 46, 51, .... Neumann and Saxl [47] showed that integers havingcertain expressions involving prime numbers belong to D. Specifically, they exhibitedthe following members n of D (where p, q, r denote primes):

n = 2p = 4q + 2 = r + 3 > 22 (examples: 46, 166, 214,...);

n = 2p = 4<? + 2 = r + 5 > 22 (examples: 46, 94, 118, ...);

n = 2p = 4q + 2 = 5r + 3 > 22 (examples: 118, 358, 958, ...);

n = 3p = 6q + 3 — r + 4 > 33 (examples: 141, 177, 321, ...);

n = 3p = 6q + 3 = 4r + 5 > 33 (examples: 177, 249, 321, ...);

n = 3p = 6<? + 3 = 5r + 4 (examples: 69, 789, 1149, ...);

n = 3p = 6q + 3 = r + 8 (examples: 69, 249, 5681, . . .) .

There are probably infinitely many integers of each form; but this has not yet beendemonstrated in any case, and seems comparable in difficulty to the "twin primes"problem. Thus, only finitely many integers are at present known to lie in D.

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 13

However, if hypothesis (S) is invoked, then considerably more can be proved.One way in which this can be done involves the following theorem of Wielandt [66].

THEOREM 7.1. Let G be a primitive permutation group of degree n = 2p (p prime).Then G has rank at most 3; and ifn — l is not a square, then G is 2-transitive.

According to Theorem 5.3, the integers n which are degrees of 2-transitive groupsother than An and Sn are (with finitely many exceptions) prime powers, or of the form(qd — \)/{q — 1) for q a prime power, or of the form 22d~l ± 2d~l. If n is twice a primeand n > 10, then n cannot be of the first or third of these forms. For p = 2 or8 (mod 9) and p > 2, 2p — l is neither a prime power nor a square; and the set{{qd — 1 )/(<? — 1) | d ^ 3,q a prime power} has smaller density than the set{n = 2p | p prime, p = 2 or 8 (mod 9)}. We deduce:

PROPOSITION 7.2 (S). The set D is infinite.

However, it seems likely that very much more is true. According to Theorem 4.1,the complement of D is the union of four sets: the proper powers of integers, theprimes, the orders of simple groups, and the degrees n of primitive groups withsimple socle (other than An). The first three have limiting density zero. (We mayinvoke (S) for the best possible estimate for the density of the third set, though muchless is needed here.) There seems to be some hope that bounds for the index ofmaximal subgroups, of the kind discussed in the last section, can be used to estimatethe contribution of each type of simple group to the density of the fourth set. For

example, the alternating groups contribute the binomial coefficients I I and certain\kj

divisors of n! greater than n!//iclogn (using a weak form of Theorem 6.1). Evidencesuggests that the fourth set is dominated by numbers of the form p + 1 , for p prime.Thus it may be possible to prove, using (S), that the set D contains almost allintegers, in the sense that lim \D n {1, . . . , n}\/n = 1.

8. All primitive groups'?

Dynkin [20] determined the maximal subgroups of Lie groups, using knownresults about the representation theory of these groups. At the AMS Symposium atSanta Cruz, Scott outlined a procedure for determining the maximal subgroups ofgroups G satisfying T ^ G ^ Aut(T), where T is a finite simple group of Lie type,based on recent advances in our knowledge of the representation theory of thegroups of Lie type. Suppose, for example, that T is a classical group, and that if is amaximal subgroup of G. We may assume that H n T acts irreducibly on the naturalmodule for T; Clifford's theorem then gives information about the action of aminimal normal subgroup N of H n T on this module. Assuming that AT is a knowngroup with known representations over finite fields, we can determine N and henceH. There remains the problem of deciding when such known groups H are maximalsubgroups of G = <H, T>.

A complete solution to this problem would enable us in principle to determine allprimitive permutation groups with non-abelian socle. For, as explained earlier, wemay assume that the socle T is simple; we are assuming the question settled when T

14 PETER J. CAMERON

is of Lie type; and we may assume that the sporadic groups present no difficulty ofprinciple. Now the case when T is an alternating group is dealt with by the"bootstraps" argument used earlier. For, if T = Am, then G = Am or Sm, with m < n.If H is intransitive, then G acts on /c-sets for some k; if H is transitive but imprimitive,then G acts on partitions into k sets of size / for some k, /; and if H is primitive, thenit has occurred earlier in the list as a primitive group of degree m.

9. Orbitals, graphs, matrices

In this section we return to something more like a traditional account ofpermutation groups. We will see that there are still some problems, which havereceived much attention in the past, and which are relatively unaffected by (S).

Let G be a transitive permutation group on fi. There is a natural componentwiseaction of G on Qxf i ; an orbital is a G-orbit in fixfi. (For example, the diagonal{(a, a) | a e fi} is an orbital.) The number of orbitals is the rank of G, as defined inSection 5. There is a pairing on the set of orbitals, under which F is paired with itsconverse F = {(/?, a) | (a, /?) e F}. There is also a bijection between orbitals andsuborbits, the Ga-orbits in fi, defined by

This enables us to label the suborbits in a consistent way so that, for example,T{ag) = T{a)g for all geG. Now the pairing of orbitals induces a pairing ofsuborbits.

It was suggested by C. C. Sims [58] that we should consider an orbital F as theset of edges of a directed graph with vertex set Cl. Clearly G is a group ofautomorphisms of this graph, acting transitively on vertices and on edges. If F is notthe diagonal, then the graph has no loops. If F is self-paired (F = F'), then we mayconsider the graph as being undirected.

Sims and D. G. Higman observed that the group G is primitive if and only if allthe orbital graphs corresponding to non-diagonal orbitals are connected. It isunnecessary to distinguish between the two definitions of connectivity possible fordirected graphs (respecting or ignoring the directions), since they are equivalent forfinite vertex-transitive graphs. This observation allows simple proofs to be given fora number of classical theorems about primitive groups. For example, if G has rank rand a non-diagonal subdegree k, then |fi| ^ 1 +k + k2 + ... + kr~1: the right-hand sideis an obvious upper bound for the number of vertices in a connected graph ofdiameter r— 1 and valency k. (This bound was quoted in the discussion of Theorem5.6.)

Suppose that G is a primitive group with a non-diagonal subdegree k. It wasconjectured by Sims that the order of Ga is bounded by a function of k. Certainly thetransitive constituent of Ga of degree k has order at most k\\ the conjecture assertsthat the kernel K(<x) of the action of Ga on this suborbit F(a) has bounded order.

Thompson [62] took an important step towards proving the conjecture whenhe showed that, under these hypotheses, Ga has a normal subgroup P of prime-powerorder whose index is bounded by a function g(k) of the subdegree k. (We may ofcourse have P — 1.) Wielandt [70] showed that we may take g(k) = k\({k — l)!)fc;substantially better bounds are known under additional hypotheses.

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 15

How could (S) contribute to a proof of Sims' conjecture? Observe first that if Ghas a regular normal subgroup, then Ga is an irreducible linear group, and so eachGa-orbit other than {0} contains a basis; thus Ga acts faithfully on each orbit. So wemay ignore this case. It follows from Thompson's theorem and the maximality of Ga

in G that it would suffice to prove the conjecture in the case where Ga is a localsubgroup (that is, the normaliser of a non-trivial p-subgroup of G). Put another way,we may assume that G acts by conjugation on a conjugacy class of p-subgroups. So,in a sense, the conjecture is reduced to a specific question about p-subgroups of"known" groups; but obviously it still requires some effort.

Goldschmidt and Scott [25] have pointed out a class of examples which showthat the kernels of the actions of Ga on paired suborbits need not be equal.

It is conjectured that, if G is primitive and Ga acts regularly on the suborbitT(a) f {a}, then Ga acts faithfully on T(a) (that is, |r(a)| = |GJ) (Wielandt [68],page 50). The conjecture is true under additional hypotheses, such as the solubility ofGa [42].

The subrank of a transitive group is defined to be the maximum rank of thetransitive constituents of the stabiliser of a point. The author has conjectured that, ifG is a primitive group with subrank m, then either the rank of G is bounded by afunction of m, or \Ga\ ̂ m. (The condition \Ga\ ̂ m implies that Ga acts regularly onsome orbit, and is equivalent to it if the previous conjecture is true.) The assertion istrue for m ^ 3 (Cameron [11]). It is not at all clear whether (S) can help in the studyof this problem. Even if all primitive groups were known, the testing of the conjecturewould require a great deal of work.

The minimal degree of a permutation group is the smallest number of lettersmoved by a non-identity group element. (See Wielandt [68], page 42.) There are onlyfinitely many primitive groups with given minimal degree, and none with minimaldegree 9, 25 or 49. (These results were proved by Jordan in 1871 and 1874: see [38],pp. 408, 453.) Presumably these results can be strengthened very substantially byinvoking (S).

Given a transitive permutation group G on a set Q, we can represent elements ofG by permutation matrices. The set of all complex matrices commuting with G is analgebra, the centraliser algebra of G (Wielandt [68], Chapter 5). It is semisimple andcompletely reducible. Its dimension is equal to the rank of G, and it is spanned bythe adjacency matrices A(T) of the orbital graphs F. Let r0, . . . , Tr_l be the orbitals,and set A{ = A(rj). Then

r - l

ifc = 0

where the numbers aiJk are non-negative integers and have a combinatorialinterpretation: aijk is the number of points y with (a, y) e Th (y, /?) e F}, whenever(a,/?)eFfc. From knowledge of these intersection numbers aijk it is possible inprinciple to compute the degrees of the irreducible representations of the centraliseralgebra and their multiplicities in the regular representation. These degrees andmultiplicities are algebraic functions of the intersection numbers which must be non-negative integers. This so-called integrality condition has been important in the studyof primitive permutation groups (see Higman [31], for example). Some informationabout the intersection numbers can be obtained from knowledge of the degrees andmultiplicities: see the paper of Wielandt [66] on groups of degree 2p cited earlier.

16 • PETER J. CAMERON

The following question is due to Scott (personal communication). If G isprimitive, is every basis matrix Ax necessarily diagonalisable? (There are examples ofnon-normal basis matrices.) Again it is not clear how (S) would bear on thisquestion. Perhaps it assumes too much: we could merely assume G transitive. (This isequivalent to considering adjacency matrices of edge-transitive directed graphs. Anexample due to Godsil and McKay shows that the vertex-transitivity is notsufficient.)

We have seen that there is a pairing defined on the orbitals of a transitive group.The basis matrices of paired orbitals are transposes. We may speak about the pairedsubconstituents Gl(a) and G^{a\ These need not be isomorphic, even as abstractgroups; but there are some relations between them. For example, Sims (see [51])showed that they have a common non-trivial homomorphic image, and Cameron[10] that if one is 2-transitive then so is the other. Stronger relationships hold inprimitive groups; exactly how strong, is not known.

We conclude this section with a problem to which (S) is relevant. Manning ([44],see also [10], [51]) showed that, if G is primitive but not 2-transitive, and if Ga acts2-transitively on a suborbit F(a) with subdegree /c, then there is a suborbit A(a)with subdegree / = k{k-l)/c, where c < k-1 if k > 2 (so that I > k). (The orbitalA is defined as {(a,/?)|3yeQ with (y,a),(y,j5)Gf;a =/= /?}.) This seemingly technicalresult has been useful in studying primitive groups of small or special degree andfixed points of elements of prime order. However, in all known examples, c ^ 6. Aproof of this empirical fact would enable substantially better theorems to be provedin these areas. By Theorem 5.3, we may assume that the 2-transitive group G[(ot) is aknown group. The assertion that c ^ 6 has been proved in many cases under thisassumption (see [10], [14]).

10. Recognising primitive groups

Suppose that a permutation group on a moderate number of points (severalhundred or thousand) is given, say by a set of generating permutations, and we wantto put a name to the group. (Such a group may arise as an automorphism group of acombinatorial object, a homomorphic image of a polyhedral group, etc.) The twoconstraints on the problem are the limits of our knowledge of the existing groups,and the computational techniques available. While some things are easy to compute(such as stabilisers and centralisers), others require a substantial calculation (the lastterm in the derived series, or the largest normal p-subgroup), while yet others arecurrently thought to be intractable (the socle, or one of its composition factors).

Let us suppose that the group G being considered is primitive. (This can bechecked easily; and using the theory in Sections 2 and 3, knowledge of the primitivecomponents gives substantial information about an arbitrary group.) In the spirit ofthe O'Nan-Scott theorem, we would like to compute the socle of G. In our particularcase this is less intractable than the corresponding problem for an arbitrary group.The socle of G is elementary abelian if and only if the degree n is a power of a primep, and Op{G) (the largest normal p-subgroup of G) is non-trivial. Assume that thisdoes not occur, so that the socle is non-abelian. Then G ^ Aut (T) Wr Sm, where T issimple, and the socle of G is isomorphic to Tm. Suppose that G has rank at most 5.Then m ^ 4 (by Proposition 5.1), so Sm is soluble; and Aut {T)/T is soluble, since thetruth of Schreier's conjecture follows from (S); so the socle of G is computable as the

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 17

last term in the derived series. The same conclusion holds in other situations too. Forexample, there are only two degrees below 104 for which it can fail, namely55 = 3125 (G ^ S5 WrS5) and 65 = 7776 (G ^ S6WrS5).

Once the socle of G is known, we wish to find a composition factor. The knownsimple groups are characterised (within the class of all known simple groups) bytheir orders, with known exceptions:

|PSp(2m,g)| = |PQ(2m + \,q)\ for m ^ 3 and q odd, and |PSL(3,4)| = \A8\

(Artin [2]). Does a similar statement hold for characteristically simple groups (directproducts of isomorphic simple groups)? Artin's technique should enable thisproblem to be settled. For a classical group T, he showed that with knownexceptions, the largest prime power dividing the order of T is a power of thecharacteristic. If q is the field order and n the dimension, inspection of the orderformulae shows that any prime other than the characteristic which divides T mustdivide ql — \ for some i ^ n. Now ql — i has a primitive divisor (a prime divisorwhich does not divide pk — 1 for any smaller power pk of the characteristic p), by atheorem of Zsigmondy [71]. So the other prime divisors of T can be used todetermine factors q' — l , and hence to recognise the group. A similar procedure willwork for any direct power of T. However, it is not known whether it is unambiguous:are there simple groups Tx and T2 for which \Tx\

m = |T2|n with m ± n? Note that

|PSp(4, q)\ is a square whenever %(q2 + l) is a square. For example,|PSp(4, 7)| = 117602, but there is no simple group of order 11760. (This example wascommunicated to me by Don Taylor.)

In view of the computational difficulty of finding the socle, and possibleobjections to the use of (S), it is worth looking for theorems which recogniseparticular classes of groups. The value of a recognition theorem depends on theexpected frequency of the groups to which it applies and the time taken to performthe test; it is clearly a good idea to perform simple recognition tests for commongroups before embarking on a more comprehensive test. Such tests can be derivedfrom known characterisation theorems. However, these theorems sometimes requiremodification. The most satisfying hypotheses to a mathematician are not always theeasiest for a computer to check; and sometimes better results follow from hypotheseswhich are stronger but impose no great computational burden. These points will beillustrated by examples.

The simplest tests to apply to rank 3 groups are characterisations by subdegrees.An example is the following theorem of Higman [32].

THEOREM 10.1. Let G be a permutation group of degree n = \m(m — 1), in whichGa has orbit lengths 1, 2(m —2), and j(m — 2)(m — 3). If m is not one of a known finitelist of exceptional values, then G < Sm (acting on unordered pairs).

For m = 9, an exception is known to occur: the group G2(2). The chief difficultyin the proof of the theorem (and the reason for the exceptions) is the necessity toshow that \i = 4 in order to apply a graph-theoretic characterisation due to Chang[16] and Hoffman [34]. (Here \i is the intersection number |F(a) n T(j5)|, where F(a)is the Ga-orbit of size 2(n-2), and /? £ T(a) u {a}.) But, if the group G is given, it is atrivial computational matter to check whether \i = 4; if it is, then the theorem holdswithout exception.

18 PETER J.CAMERON

Thus, an important open problem is to obtain characterisations of all the knownrank 3 groups under hypotheses which are easily checked by computer. Suchhypotheses may be graph-theoretic, involving subdegrees and intersection numbers(as above) or configuration theorems (see Shult [57]), or group-theoretic, as in thefollowing example. Fischer [22] has determined the finite groups generated by aconjugacy class of 3-transpositions (involutions with the property that the product ofany two of them has order 1, 2 or 3). Under suitable hypotheses on its solublenormal subgroups, such a group G acts by conjugation on the class Q of3-transpositions as a primitive rank 3 permutation group. For each a e fi, we canrecognise the corresponding involution ta by computing the centraliser of Ga. Tocheck that {ta | a e fi} consists of 3-transpositions, it suffices to compute tatp and taty,where /? and y are representatives of the non-diagonal suborbits.

Analogous results for groups of higher rank are rare. However, for groups ofgiven degree, those of larger rank tend to have smaller subdegrees and so (by Sims'conjecture) smaller order, and direct calculation of the socle is easier.

For doubly transitive groups, there are many known characterisation theorems,but still some difficulties in applying them. Consider the following result of Hering,Kantor and Seitz [29].

THEOREM 10.2. Let G be a 2-transitive permutation group on fi. Suppose that, fora e fi, Ga has a normal subgroup which acts regularly on fi- {a}. Then either G has anabelian regular normal subgroup, or the socle of G is isomorphic to PSL(2,q),PSU(3, q), Sz(<j), or a group of Ree type.

(In fact, the "groups of Ree type" have been recently shown to be Ree groups byBombieri, building on work of Thompson.)

Suppose that G is 2-transitive and satisfies the hypotheses of the proposition, andwe have verified that G has no regular normal subgroup. Then the socle of G isdetermined by \G\; there is no need to compute terms in the derived series. However,G itself is not determined by its order, even when its socle is known. Note, too, thatthere is no point in applying this test unless the degree of G is a prime power plusone.

Groups with regular normal subgroup occur so frequently in characterisationtheorems that it is desirable to have techniques to deal with them. If such a group Ghas degree p", then Ga is an irreducible subgroup of GL(n,p). However, it may besemilinear (or even linear) over a larger field; detecting this gives more scope forusing recognition theorems on the action of Ga on points of the projective space. If Ga

is semilinear over GF(q), and G' denotes the derived group of G, then G'a is linearover GF(q), so G'aP fixes pointwise the affine line containing a and /?. Conversely, itcan be shown, under additional hypotheses, that if G'ap fixes q points, then Ga issemilinear over GF(q).

11. Related topics

Knowledge about permutation groups can be used in the study of automorphismgroups of combinatorial structures, especially when the groups are assumed to havesome transitivity property. One instance of this concerns 2-designs (balanced

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 19

incomplete block designs) whose automorphism groups are 2-transitive on points. Ofcourse, in view of Theorem 5.3, virtually all the problems in Kantor's extensivesurvey on this topic [39] are trivialised by (S). (However, if Theorem 5.3 is to beproved without using (S), the proof will almost certainly require more work on someof these problems!)

In other areas, the effect of (S) is less dramatic. Consider the case of connectedgraphs (possibly directed) whose automorphism groups are transitive on orderededges. The considerations of Section 9 show that the study of such graphs generalisesthe study of primitive permutation groups; and indeed the two theories have much incommon. Thus, many open problems remain here. We might expect (S) to be moreuseful if additional transitivity is required. Three ways in which this has been doneare s-arc transitivity, distance-transitivity, and n-tuple transitivity. (See Gardiner[24] for a survey.)

An s-arc in a graph is a sequence (a0,..., as) of vertices in which a, and <xi+l areadjacent and a, =/= ai + 2 for all relevant i. In an s-arc transitive undirected graph (onewhose automorphism group is transitive on s-arcs) with s ^ 2, the stabiliser of avertex acts 2-transitively on its neighbours; using (S), we may assume that thissubconstituent is known. Substantial progress has been made under this assumption(see [65]). The "main problem" in the area is to show that s ^ 7 for all graphs otherthan cycles, and to determine graphs with relatively large s.

On the other hand, recent progress towards proving (S) has so far been less usefulin studying distance-transitive graphs, those whose automorphism group actstransitively on pairs of vertices at any given distance. It is conjectured that there areonly finitely many such graphs with any given valency. This was proved for valency 3by Biggs and Smith [6], and the truth of Sims' conjecture was crucial for the proof.Another example is the use of the Feit-Thompson theorem in the determination ofdistance-transitive directed graphs of odd girth [5].

A graph F is n-tuple transitive if any isomorphism between induced subgraphs onat most n vertices extends to an automorphism of F. (This is a specialisation tographs of diameter 2 of a more general definition due to Meredith [45].) Usingpurely combinatorial arguments, Cameron [12] determined all 5-tuple transitivegraphs. Recently, Buczak [8] has shown that only two further graphs are 4-tupletransitive. His proof uses (S) in a strong sense: detailed knowledge of the simplegroups and their properties is required.

If transitivity is not assumed, it becomes more difficult to apply (S), but this canbe done in some instances. For example, Hering [27] showed that, if a group G actsirreducibly on a finite projective plane (fixing no point, line, triangle, or propersubplane), and G contains perspectivities, then the socle of G is either elementaryabelian of order 9 or non-abelian simple. So it is reasonable to test the known simplegroups for possible irreducible actions—see [30] for some results.

Another such instance is the study of computational complexity of graphisomorphism testing (see [52]). Fast algorithms for checking the isomorphism of twographs often fail if one of the graphs admits non-trivial automorphisms, since such agraph cannot be labelled in a canonical way. So we need to know how large theautomorphism group of a graph can be, and to have recognition theorems for graphswith large groups. The material of Section 6 is obviously relevant here: see Babai [3].

We conclude by turning to another area, infinite permutation groups. The studyof such groups is still largely at the "collecting specimens" stage. We mention a few ofthese to show how the subject differs in spirit from the finite.

20 PETER J. CAMERON

1. There is no analogue of the positive results of Section 9 about pairedsubconstituents: any two transitive groups whatever occur as paired subconstituentsin a transitive group. For we may assume that the proposed subconstituents arerepresentations of the same group (not necessarily faithful) and that the pointstabilisers K and L are isomorphic. Form the HNN extensionG = (H,t:t~lKt = L> (see [33]). Britton's lemma [7] guarantees thatH r\t~xHt = L, H n tHt'1 = K, so that G acting on the cosets of H has thedesired subconstituents.

However, something can be salvaged. If paired constituents in a transitive groupG have no non-trivial homomorphic image in common, then Sims' argument showsthat G is transitive on directed paths of any finite length in the orbital graph F, sothat in particular F contains no directed cycles. If it could be shown that F is a tree,then the Bass-Serre theory of groups acting on trees [56] would show that G is anHNN extension, as in the example.

2. For every positive integer k there exists a group which is /c-transitive but not(k + Intransitive. This shows that the situation for infinite permutation groups isvery different from that for finite groups.

3. There exist groups in which all non-identity elements are conjugate (see[33]). If T is such a group, consider the action of TxT on T by the rule(g, h): x \—>g~xxh. Then G = TxT is 2-transitive, and each direct factor is a non-abelian simple regular normal subgroup of G.

Added in proof, October 1980: Some results which have recently come to myattention include the following.

(i) L. Babai has extended his sub-exponential bound for the orders ofuniprimitive permutation groups to multiply transitive groups (see Section 6).

(ii) D. N. Teague has shown, assuming (S), that if Tx and T2 are simple groupswith |T,|m = \T2\

n for positive integers m and ri, then |TJ = \T2\. He has also broughtArtin's resu!f on coincidences of orders of simple groups up to date (see Section 10).

(iii) R. M. Weiss has shown, assuming (S), that if a finite graph with valency atleast 3 is s-arc transitive, then s ^ 7 (see Section 11).

(iv) In connection with Hering's result on irreducible collineation groupscontaining perspectivities (see Section 11), A. Reifart and G. Stroth have shown that,of the 26 sporadic simple groups, only the second Janko group can occur in thissituation.

References

1. K. Appel and W. Haken, "Every planar map is four colorable", Bull. Amer. Math. Soc. 82 (1976),711-712.

2. E. Artin, "The orders of the classical simple groups", Comm. Pure Appl. Math. 8 (1955), 455-472.3. L. Babai, "On the order of uniprimitive permutation groups", to appear.4. E. Bannai, "Maximal subgroups of low rank of finite symmetric and alternating groups", J. Fac. Sri.

Univ. Tokyo 18 (1972), 475-486.5. E. Bannai, P. J. Cameron and J. Kahn, "Nonexistence of certain distance-transitive digraphs",

submitted.6. N. L. Biggs and D. H. Smith, "On trivalent graphs", Bull. London Math. Soc. 3 (1971), 155-158.7. J. L. Britton, "The word problem", Ann. Math. 77 (1963), 16-32.

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS 21

8. J. M. J. Buczak, Finite group theory, D.Phil. Thesis, Oxford University, 1980.9. W. Burnside, Theory of groups of finite order (Cambridge University Press, Cambridge, 1911).

10. P. J. Cameron, "Permutation groups with multiply transitive suborbits", Proc. London Math. Soc. (3)25 (1972), 427-440.

11. P. J. Cameron, "Bounding the rank of certain permutation groups", Math. Z. 124 (1972), 343-352.12. P. J. Cameron, "6-transitive graphs", to appear in J. Combinatorial Theory.13. P. J. Cameron and W. M. Kantor, "2-transitive and antiflag transitive collineation groups of finite

projective spaces", J. Algebra 60 (1979), 384-422.14. P. J. Cameron and C. E. Praeger, "Graphs, permutation groups, and orthogonal geometry", in

preparation.15. C. Chevalley, "Sur certains groupes simples", Tohoku J. Math. 7 (1955), 14-66.16. L. C. Chang, "The uniqueness and non-uniqueness of the triangular association schemes", Sci. Record

Peking Math. (New Ser.) 3 (1959), 604-613.17. C. W. Curtis, W. M. Kantor and G. Seitz, "The 2-transitive permutation representations of the finite

Chevalley groups", Trans. Amer. Math. Soc. 218 (1976), 1-57.18. M. Davis, Y. Matijasevic and J. Robinson, "Hilbert's tenth problem: Diophantine equations: positive

aspects of a negative solution", Mathematical developments arising from Hilberts problems.Proceedings Symposia in Pure Mathematics 28 (American Mathematical Society, Providence,R.I., 1976), pp. 323-378.

19. L. E. Dickson, Linear Groups, Dover, New York, 1958.20. E. B. Dynkin, "Maximal subgroups of semisimple Lie groups and the classification of primitive groups

of transformations", Doklady Akad. Nauk SSSR (New Ser.) 75 (1950), 333-336.21. W. Feit and J. G. Thompson, "Solvability of groups of odd order", Pacific J. Math. 13 (1963),

775-1029.22. B. Fischer, "Finite groups generated by 3-transpositions", Invent. Math. 13 (1971), 232-246.23. E. Galois, "Oeuvres mathematiques: lettre a M. Auguste Chevalier", J. Math. Pures Appl. (Liouville)

(1846), 408-415.24. A. D. Gardiner, "Symmetry conditions in graphs", Surveys in Combinatorics, London Math. Soc.

Lecture Note Series 38, (ed. B. Bollobas, Cambridge University Press, Cambridge, 1979),pp. 22-43.

25. D. M. Goldschmidt and L. L. Scott, "A problem of W. A. Manning on primitive permutation groups",Math. Z. 161 (1978), 97-100.

26. C. Hering, "On linear groups which contain an irreducible subgroup of prime order", Proc. Internat.Conf. Projective Planes (Washington State University Press, Pullman, 1973), pp. 99-105.

27. C. Hering, "On the structure of finite collineation groups of projective planes", Abh. Math. Sem. Univ.Hamburg 49 (1979), 155-182.

28. C. Hering, to appear.29. C. Hering, W. M. Kantor and G. Seitz, "Finite groups with a split BN-pair of rank 1", J. Algebra 20

(1972), 435-475.30. C. Hering and M. Walker, "Perspectivities in irreducible collineation groups of projective planes",

Math. Z. 155 (1977), 95-101.31. D. G. Higman, "Finite permutation groups of rank 3", Math. Z. 86 (1964), 145-156.32. D. G. Higman, "Characterization of families of rank 3 permutation groups by the subdegrees", Arch.

Math. (Basel) 21 (1970), 151-156.33. G. Higman, B. H. Neumann and H. Neumann, "Embedding theorems for groups", J. London Math.

Soc. (1) 26 (1949), 247-254.34. A. J. Hoffman, "On the uniqueness of the triangular association scheme", Ann. Math. Statist. 31

(1960), 492-^97.35. R. B. Howlett, "On the degrees of Steinberg characters of Chevalley groups", Math. Z. 135 (1974),

125-135.36. B. Huppert, "Zweifach transitive, auflosbare Permutationsgruppen", Math. Z. 68 (1957), 126-150.37. J. F. Hurley and A. Rudvalis, "Finite simple groups", Amer. Math. Monthly 84 (1978), 693-714.38. C. Jordan, Oeuvres, Vol. I (ed. J. Dieudonne, Gauthier-Villars, Paris, 1961).39. W. M. Kantor, "2-transitive designs", Combinatorics (ed. M. Hall Jr. and J. H. van Lint, Reidel,

Dordrecht, 1975), pp. 365-418.40. W. M. Kantor, "Permutation representations of the finite classical groups of small degree or rank",

J. Algebra 60 (1979), 158-168.41. W. M. Kantor and R. A. Liebler, "The rank 3 permutation representations of the finite classical

groups", to appear.42. W. Knapp, personal communication.43. E. Maillet, "Sur les isomorphes holoedriques et transitifs des groupes symetriques ou alternes",

J. Math. Pures Appl. (5) 1 (1895), 5-34.44. W. A. Manning, "A theorem concerning simply transitive primitive groups", Bull. Amer. Math. Soc. 35

(1929), 330-332.45. G. H. J. Meredith, "Triple transitive graphs", J. London Math. Soc. (2) 13 (1976), 249-257.

22 FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS

46. H. Nagao, Multiply Transitive Groups, (California Institute of Technology, Pasadena, 1967).47. P. M. Neumann and J. Saxl, "The primitive permutation groups of some special degrees, II: small

multiples of certain large primes", Math. Z. 169 (1979), 205-222.48. M. E. O'Nan, "Normal structure of the one-point stabiliser of a doubly-transitive permutation group",

Trans. Amer. Math. Soc. 214 (1975), 1-74.49. C. E. Praeger, "On primitive permutation groups with a doubly transitive suborbit", J. London Math.

Soc. (2) 17(1978), 67-73.50. C. E. Praeger and J. Saxl, "On the orders of primitive permutation groups", Bull. London Math. Soc.

37(1980), 303-307.51. W. L. Quirin, "Primitive permutation groups with small orbitals", Math. Z. 122 (1971), 267-274.52. R. C. Read and D. G. Corneil, "The graph isomorphism disease", J. Graph Theory 1 (1977), 339-363.53. R. Ree, "A family of simple groups associated with the simple Lie algebra of type (F4)", Amer. J. Math.

83 (1961), 401-420.54. R. Ree, "A family of simple groups associated with the simple Lie algebra of type (G2)", Amer. J. Math.

83 (1961), 432-462.55. G. M. Seitz, "Small rank permutation representations of finite Chevalley groups", J. Algebra 28

(1974), 508-517.56. J.-P. Serre, "Arbres, amalgames, SL2", Asterisque 46 (1977).57. E. E. Shult, "Characterizations of certain classes of graphs", J. Combinatorial Theory (B), 13 (1972),

1-26.58. C. C. Sims, "Graphs and finite permutation groups", Math. Z. 95 (1967), 76-86.59. C. C. Sims, "Computational methods in the study of permutation groups", Computational Problems in

Abstract Algebra (ed. J. Leech, Pergamon, London, 1970), pp. 169-183.60. R. Steinberg, "Variations on a theme of Chevalley", Pacific J. Math. 9 (1959), 875-891.61. M. Suzuki, "A new type of simple group of finite order", Proc. Nat. Acad. Sci. 46 (1960), 868-870.62. J. G. Thompson, "Bounds for orders of maximal subgroups", J. Algebra 14 (1970), 135-138.63. J. Tits, "Les groupes simples de Suzuki et de Ree", Sem. Bourbaki, Expose 210, Paris, 1960.64. J. A. Todd, "As it might have been", Bull. London Math. Soc. 2 (1970), 1-4.65. R. Weiss, "Groups with a {B, N) pair and locally transitive graphs", to appear.66. H. Wielandt, "Primitive Permutationsgruppen von Grad 2p", Math. Z. 63 (1956), 478-485.67. H. Wielandt, "Uber den Transitivitatsgrad von Permutationsgruppen", Math. Z. 74 (1960), 297-298.68. H. Wielandt, Finite permutation groups, (Acad. Press, New York-London 1964).69. H. Wielandt, Permutation groups through invariant relations and invariant functions, (Ohio State

University, Columbus, 1969).70. H. Wielandt, Subnormal subgroups and permutation groups, (Ohio State University, Columbus, 1971).71. K. Zsigmondy, "Zur Theorie der Potenzreste", Monats.fur Math, und Phys. 3 (1892), 265-284.

Merton College,Oxford OX1 4JD.