Modular symmetry and non-Abelian discrete flavor symmetries in string compactification

We study the modular symmetry in magnetized D-brane models on $T^2$. Non-Abelian flavor symmetry $D_4$ in the model with magnetic flux $M=2$ (in a certain unit) is a subgroup of the modular symmetry. We also study the modular symmetry in heterotic orbifold models. The $T^2/Z_4$ orbifold model has the same modular symmetry as the magnetized brane model with $M=2$, and its flavor symmetry $D_4$ is a subgroup of the modular symmetry.


I. INTRODUCTION
Non-Abelian discrete flavor symmetries play an important role in particle physics. In particular, many models with various finite groups have been studied in order to explain quark and lepton masses and their mixing angles. (See for review [1][2][3].) Those symmetries may be useful for dark matter physics and multi-Higgs models.
Superstring theory is a promising candidate for unified theory including gravity. It has been shown that some non-Abelian discrete flavor symmetries appear in superstring theory with certain compactifications. Heterotic string theory on toroidal Z N orbifolds can realize non-Abelian flavor symmetries, e.g. D 4 , and ∆(54) [4]. (See also [5,6].) 1 Furthermore, magnetized D-brane models within the framework of type II superstring theory can lead to similar flavor symmetries [8][9][10][11][12]. Intersecting D-brane models are T-dual to magnetized D-brane models. Then, one can realize the same aspects in intersecting D-brane models as in the magnetized ones. 2 On the other hand, superstring theories on tori and orbifolds have the modular symmetry.
Recently, behavior of zero-modes under modular transformation was studied in magnetized D-brane models in Ref. [14].(See also [15].) Also, behavior of twisted sectors under modular transformation was alreadly studied in Ref. [16][17][18]. These modular transformations also act non-trivially on flavors and transform mutually flavors each other. The remarkable difference is that modular transformation also acts Yukawa couplings as well as higher order couplings, while those couplings are trivial singlets under the usual non-Abelian symmetries.
The purpose of this paper is to study more how modular transformation is represented by zero-modes in magnetized D-brane models, and to discuss relations between modular transformation and non-Abelian flavor symmetries in magnetized D-brane models. Intersecting D-brane models have the same aspects as magnetized D-brane models, because they are T-dual to each other. Furthermore, intersecting D-brane models in type II superstring theory and heterotic string theory have similarities, e.g. in two-dimensional conformal field theory. Thus, here we study modular symmetry and non-Abelian discrete flavor symmetries in heterotic orbifold models, too. 1 In Ref. [7], a relation between gauge symmetries and non-Abelian flavor symmetries is discussed at the enhancement point. 2 See also [13]. This paper is organized as follows. In section II, we study the modular symmetries in magnetized D-brane models and the relation to the D 4 flavor symmetry. In section III, we study the modular symmetries in heterotic orbifold models. Section IV is conclusion and discussions. We give brief reviews on non-Abelian discrete flavor symmetries in magnetized D-brane models and heterotic orbifold models in Appendix A and B, respectively.

II. MODULAR TRANSFORMATION IN MAGNETIZED D-BRANE MODELS
In this section, we study modular transformation of zero-mode wavefunctions in magnetized D-brane models.

A. Zero-mode wavefunction
Here, we give a brief review on zero-mode wavefunctions on torus with magnetic flux [15].
For simplicity, we concentrate on T 2 with U(1) magnetic flux. The complex coordinate on T 2 is denoted by z = x 1 + τ x 2 , where τ is the complex modular parameter, and x 1 and x 2 are real coordinates. The metric on T 2 is given by We identify z ∼ z + 1 and z ∼ z + τ on T 2 .
On T 2 , we introduce the U(1) magnetic flux F , which corresponds to the vector potential, Here we concentrate on vanishing Wilson lines.
On the above background, we consider the zero-mode equation for the spinor field with the U(1) charge q = 1, The spinor field on T 2 has two components, The magnetic flux should be quantized such that M is integer. Either ψ + or ψ − has zeromodes exclusively for M = 0. For example, we set M to be positive. Then, ψ + has M zero-modes, while ψ − has no zero-mode. Hence, we can realize a chiral theory. Their zeromode profiles are given by with j = 0, 1, · · · , (M − 1), where ϑ denotes the Jacobi theta function, Here, N denotes the normalization factor given by with A = 4π 2 R 2 Im τ .
The ground states of scalar fields also have the same profiles as ψ j,M . Thus, the Yukawa coupling including one scalar and two spinor fields can be computed by using these zero-mode waverfunctions. Zero mode wavefunctions satisfy the following relation, By use of this relation, their Yukawa couplings are given by the wavefunction overlap integral, Hence, we can definite Z g charges in these couplings [8].

B. Modular transformation
Here, we study modular transformation. First we give a brief review on results of modular transformation [14]. (See also [15].) Then, we will study more in details.
The T 2 is constructed by R 2 /Λ, and the lattice Λ is spanned by the vectors (α 1 , α 2 ), where α 1 = 2πR and α 2 = 2πRτ . However, the same lattice can be described by another where a, b, c, d are integer with satisfying ad − bc = 1. That is SL(2, Z) transformation.
They satisfy On top of that, if we impose the algebraic relation, that corresponds to the congruence subgroup of modular group, Γ(N). For example, it is found that Γ(2) ≃ S 3 , Γ(3) ≃ A 4 , Γ(4) ≃ S 4 , and Γ(5) ≃ A 5 . Since the group A 4 is the symmetry of tetrahedron, it is often called the tetrahedral group T = A 4 . Also, it may be useful to mention about ∆( (12), and S 4 ≃ ∆(24).
Following [14], we restrict ourselves to even magnetic fluxes M (M > 0). Under S, the zero-mode wavefunctions transform as [14,15] On the other hand, the zero-mode wavefunctions transform as [14] ψ j, under T . Generically, the T -transformation satisfies on the zero-modes, ψ j,M . Furthermore, in Ref. [14] it is shown that on the zero-modes, ψ j,M .
In what follows, we study more concretely.
They construct a closed algebra with the order 192, which we denote here by G (2) . By such an algebra, modular transformation is represented by two zero-modes, ψ 0,2 , ψ 1,2 . We find that (S (2) T (2) ) 3 is a center. Indeed, there are eight center elements and their group is Z 8 .
Other diagonal elements correspond to Z 4 , which is generated by T (2) . Here, we denote The diagonal elements are represented by a m a ′n , i.e. Z 8 × Z 4 .
Here, we examine the right coset Hg for g ∈ G (2) , where H is the above H = {a m a ′n }. There would be 6(= 192/(8 × 4)) cosets. Indeed, we obtain the following six cosets: H, HS (2) , HS (2) with k = 1, 2, 3. By simple computations, we find HS (2) (2) . Furthermore, we would make a (non-Abelian) subgroup with the order 6 by choosing properly six elements such that we pick one element up from each coset and their algebra is closed. The non-Abelian group with the order 6 is unique, i.e. S 3 . For example, we may be able to obtain the Z 3 generator from HS (2) T (2) because (S (2) T (2) ) 3 ∈ H. That is, we define b = a m a ′n S (2) T (2) .
Similarly, we can obtain the Z 2 generator e.g. form HS (2) T 2 (2) S (2) because (S (2) T 2 (2) S (2) ) 2 ∈ H. Then, we define We find c 2 = I when n ′ = −m ′ mod 4. On top of that, we require (bc) 2 = I, and that leads The six elements of the subgroup are written explicitly, where ρ = e 2πi/8 . They correspond to S 3 ≃ Γ(2) ≃ ∆(6) because they satisfy the following algebraic relations, Moreover, they satisfy the following algebraic relation with Thus, the algebra of We have started by choosing HS (2) T 2 (2) S (2) for a candidate of the Z 2 generator. We can obtain the same results by starting with HS (2) for a candidate of the Z 2 generator.
This is a reducible representation. In order to obtain irreducible representations, we use the flowing basis, This is nothing but zero-modes on the T 2 /Z 2 orbifold [20]. The former corresponds to Z 2 even states, while the latter corresponds to the Z 2 odd state. Note that (ST ) 3 transforms the lattice basis (α 1 , α 2 ) → (−α 1 , −α 2 ). Thus, it is reasonable that the zero-modes on the T 2 /Z 2 orbifold correspond to the irreducible representations.
The S and T -representations by the Z 2 odd zero-mode are quite simple, and these are represented by Their closed algebra is Z 8 .
On the other hand, the S and T -transformations are represented by the Z 2 even zeromodes, They satisfy the following algebraic relation, We denote the closed algebra of S (4)+ and T (4)+ by G (4)+ . Its order is equal to 768, and it includes the center element (S (4)+ T (4)+ ) 3 , i.e. Z 8 . Other diagonal elements correspond to Z 8 , which is generated by T (4)+ . Again, we denote a = (S (4)+ T (4)+ ) 3 and a ′ = T (4)+ , and the diagonal elements are written by a m a ′n , i.e. Z 8 × Z 8 .
Similar to the case with M = 2, we examine the coset structure, Hg. Indeed, there are the following 12 cosets: where k = 1, · · · , 7 and ℓ = 2, 4, 6. By simple computation, we find that for k = odd and ℓ = even.
We make a subgroup with the order 12 by choosing properly 12 elements such that we pick one element up from each coset and their algebra is closed. The non-Abelian group with the order 12 are D 6 , Q 6 and A 4 . Among them, A 4 would be a good candidate. Indeed, we can obtain the Z 3 generator from HS (4)+ T (4)+ , gain. That is, we define The solutions for t 3 = I are obtained by (m, n) = (1, 4), (3,6), (5,0), and (7,2). We also define s = a m ′ a ′n ′ S (4)+ T 4 (4)+ S (4)+ .
The representations of T (M ) are simply obtained by and Both correspond to Z 2M .
On the other hand, the S (M )± transforms This representation is also written by Thus, the S-transformation is represented on the T 2 /Z 2 orbifold basis by These are written by For example, for M = 6, S and T are represented by Z 2 even zero-modes, while S and T are represented by Z 2 odd zero-mode, C. Non-Abelian discrete flavor symmetries In Ref. [8], it is shown that the models with M = 2 as well as even magnetic fluxes have the D 4 flavor symmetry. See Appendix A. One of the Z 2 elements in D 4 corresponds to (T (2) ) 2 on the zero-modes, ψ 0,2 and ψ 1,2 , i.e.
In addition, the permutation Thus, the D 4 group, which includes the eight elements (A8), is subgroup of However, there is the difference between the modular symmetry and the D 4 flavor symmetry, which studied in Ref. [8]. The modular symmetry transforms the Yukawa couplings, while the Yukawa couplings are invariant under the D 4 flavor symmetry. In order to study this point, here we examine the Yukawa couplings among ψ i,2 , ψ ′j,2 and ψ k,4 . Both ψ i,2 and ψ ′j,2 are D 4 doublets, and their tensor product 2 × 2 is expanded by Thus, the products ψ i,2 ψ ′j,2 correspond to four singlets, On the other hand, by use of Eq.(9), the products ψ i,2 ψ ′j,2 are expanded by ψ k,4 . For example, we can expand as up to constant factors, where Thus, the zero-modes ψ 0,4 ± ψ 2,4 are indeed D 4 singlets, 1 +± when we identify (T (4) ) 2 and (S (4) T (4) T (4) S (4) ) as Z 2 and Z C 2 of D 4 . In this sense, the D 4 flavor symmetry is a subgroup of the modular symmetry. Also, it is found that the above Yukawa couplings, Y (m/4) (16τ ), with m = 0, 1, 2, 3 are invariant under T 2 and ST T S transformation.
Then, the mode (ψ 1,4 + ψ 3,4 ) exactly corresponds to the D 4 singlet, 1 −+ . We find that Here, we give a comment on the T 2 /Z 2 orbifold. The T 2 /Z 2 orbifold basis gives the irreducible representations of the modular symmetry. The D 4 flavor symmetry is defined through the modular symmetry, as above. That is the reason why the D 4 flavor symmetry remains on the T 2 /Z 2 orbifold [11,12].

III. HETEROTIC ORBIFOLD MODELS
Intersecting D-brane models in type II superstring theory is T-dual to magnetized D-brane models. Thus, intersecting D-brane models also have the same behavior under modular transformation as magnetized D-brane models. Furthermore, intersecting D-brane models in type II superstring theory and heterotic string theory on orbifolds have similarities, e.g.
in two-dimensional conformal field theory. For example, computations of 3-point couplings as well as n-point couplings are similar to each other. Here, we study modular symmetry in heterotic orbifold models. Using results in Ref. [16][17][18], we compare the modular symmetries in heterotic orbifold models with non-Abelian flavor symmetries and also the modular symmetries in the magnetized D-brane models, which have been derived in the previous section.

A. Twisted sector
Here, we give a brief review on heterotic string theory on orbifolds. The orbifold is the division of the torus T n by the Z N twist θ, i.e. T n /Z N . Since the T n is constructed by R n /Λ, the Z N twist θ should be an automorphism of the lattice Λ. Here, we focus on two-dimensional orbifolds, T 2 /Z N . The six-dimensional orbifolds can be constructed by products of two-dimensional ones. All of the possible orbifolds are classified as T 2 /Z N with N = 2, 3, 4, 6.
On orbifolds, there are fixed points, which satisfy the following condition, where x i are real coordinates, α i k are two lattice vectors, and m k are integer for i, k = 1, 2. Thus, the fixed points can be represented by corresponding space group elements (θ n , k m k α i k ), or in short (θ n , (m 1 , m 2 )).
The heterotic string theory on orbifolds has localized modes at fixed points, and these are the so-called twisted strings. These twisted states can be labeled by use of fixed points, σ θ,(m 1 ,m 2 ) . All of the twisted states σ θ,(m 1 ,m 2 ) have the same spectrum, if discrete Wilson lines vanish. Thus, the massless modes are degenerate by the number of fixed points.

B. Modular symmetry
In Ref. [16], modular symmetry in heterotic string theory on orbifolds was studied in detail. Here we use those results.

T 2 /Z 4 orbifold
The S and T transformations are represented by the first twisted sectors of T 2 /Z 4 orbifold as [16], These are exactly the same as representations of S (2) and T (2) on two-zero modes, ψ 0,2 and for m = 0, 1 and the second twisted sector, for m, n = 0, 1. The above Z 4 transformation (72) is nothing but (S Z 4 T Z 4 ) 6 as clearly seen from Eq. (24). Thus, the whole flavor symmetry originates from the modular symmetry.
Using results in Ref. [16], it is found that This is the same as behavior of the Yukawa couplings under T 2 studied in section II C.

T 2 /Z 2 orbifold
Here, let us study the T 2 /Z 2 orbifold in a way to similar to the previous section on the The S transformation is represented by the four twisted states on the T 2 /Z 2 orbifold [16], Also the T transformation is represented as The representation S Z 2 is similar to S Z 4 and S (2) . Indeed, we find that S Z 2 = S (2) ⊗ S (2) .
However, the representation T Z 2 is different from T Z 4 and T (2) .
The matrices S Z 2 and T Z 2 satisfy the following relations, These correspond to the D 6 . Indeed, the order of closed algebra including S Z 2 and T Z 2 is equal to 12. At any rate, these matrices are reducible. We change the basis in order to obtain irreducible representations, Then, σ 1 and σ 2 correspond to the D 6 doublet, while σ 3 and σ 4 correspond to the D 6 singlets.
The twisted sector on the T 2 /Z 2 orbifold has the flavor symmetry (D 4 ×D 4 )/Z 2 . However, this flavor symmetry seems independent of the above D 6 , because they do not include any common elements. The twisted sector on the S 1 /Z 2 orbifold has the flavor symmetry D 4 .
The flavor symmetry of T 2 /Z 2 orbifold is obtained as a kind of product, D 4 × D 4 , although two D 4 groups have a common Z 2 element. Thus, the flavor symmetry of T 2 /Z 2 originates from the product of symmetries of the one-dimensional orbifold. On the other hand, the modular symmetry appears in two or more dimensions, but not in one dimension. Hence, these symmetries would be independent. When we include the above D 6 as low-energy effective field theory in addition to the flavor symmetry (D 4 × D 4 )/Z 2 , low-energy effective field theory would have larger symmetry including D 6 and (D 4 × D 4 )/Z 2 , although Yukawa couplings as well as higher order couplings transform non-trivially under D 6 .

T 2 /Z 3 orbifold
The S and T transformations are represented by the first twisted sectors of T 2 /Z 3 orbifold as [16], These forms look similar to S and T transformations in magnetized models (18) and (19).
Indeed, they correspond to submatrices of S (6) and T (6) in the magnetized models with the magnetic flux M = 6. Alternatively, in Ref. [17] the following S and T representations were At any rate, the above representations are reducible representations. Thus, we use the flowing basis,  where σ ± = (σ 1 ± σ − )/ √ 2. The (σ + , σ 0 ) is a doublet, while σ − is a singlet. The former corresponds to the Z 6 invariant states among the θ 2 twisted sector on the T 2 /Z 6 orbifold.
Alternatively, we can say that the doublet (σ + , σ 0 ) corresponds to Z 2 even states and the singlet σ − is the Z 2 odd states, where the Z 2 means the π rotation of the lattice vectors, . This point is similar to the aspect in magnetized D-brane models, where irreducible representations correspond to the T 2 /Z 2 orbifold basis. Also, note that the first twisted states of the T 2 /Z 4 orbifold correspond already to the Z 2 -invariant basis.
For example, we represent S ′ Z 3 and T ′ Z 3 on the above basis [17] , on the doublet (σ + , σ 0 ) T , while σ − is the trivial singlet. Here, we define Then, they satisfy the following algebraic relations [17,18], This group is the so-called T ′ , which is the binary extension of A 4 = T .
The non-Abelian discrete flavor symmetry on the T 2 /Z 3 orbifold is ∆(54), and the three twsisted states correspond to the triplet of ∆(54). Thus, this modular symmetry seems independent of the ∆(54) flavor symmetry.
Two representations are related as flavor symmetry seems independent of the modular symmetry in the T 2 /Z 3 orbifold models.
Note that the first twisted states on the T 2 /Z 4 are Z 2 -invariant states. In this sense, we find that the modular symmetry is the symmetry on the Z 2 orbifold in both heterotic orbifold models and magnetized D-brane models. The symmetries, which remain under the Z 2 twist, can be realized as the modular symmetry.
We have set vanishing Wilson lines. It would be interesting to extend our analysis to magnetized D-brane models with discrete Wilson lines on orbifolds [22]. It would be also interesting to extend our analysis on zero-modes to higher Kaluza-Klein modes [23].
The non-Abelian flavor symmetries such as D 4 can be anomalous. (See for anomalies of non-Abelian discrete symmetries, e.g. [2,32,33]. ) In certain models, the modular symmetries are related with the non-Abelian flavor symmetry D 4 . It would be interesting to study their anomaly relations.
We also give a comment on phenomenological application. Recently, the mixing angles in the lepton sector were studied in the models, whose flavor symmetries are congruence subgroups, Γ(N) [34,35]. In those models, the couplings are non-trivial representations of Γ(N) and modular functions. Our models show massless modes represent larger finite groups. It would be interesting to apply our results to derive realistic lepton mass matrices as well as quark mass matrices. In this Appendix, we give a brief review on non-Abelian discrete flavor symmetries in magnetized D-brane models [8].
As mentioned in section II A, the Yukawa coulings as well as higher order couplings have the coupling selection rule (11). That is, we can define Z g charges for zero-modes. Such Z g transformation is represented on ψ i,M =g by where ρ = e 2πi/g . Furthermore, their effective field theory has the following permutation symmetry, and such permutation can be represented by 0 0 1 0 · · · 0 . . .
This is another Z C g symmetry. However, these two generators do not commute each other, Thus, the flavor symmetry corresponds to the closed algebra including Z and C. Its diagonal elements are given by Z m Z ′n , i.e. Z g × Z ′ g where and the full group corresponds to (Z g × Z ′ g ) ⋊ Z C g . Furthermore, the zero-modes ψ i,M =gn with the magnetic flux M = gn also represent (Z g ×Z ′ g )⋊Z C g . The zero-modes, ψ i,M =gn have Z g charges (mod g). Under C, they transform as For example, the model with g = 2 has the D 4 flavor symmetry. The zero-modes, correspond to the D 4 doublet 2, where eight D 4 elements are represented by In addition, when the model has the zero-modes ψ i,4 (i = 0, 1, 2, 3), the zero-modes, ψ 0,4 and ψ 2,4 ( ψ 1,4 and ψ 3,4 ) transform each other under C, and they have Z 2 charge even (odd).
The twisted string x i on the orbifold satisfy the following boundary condition: similar to Eq. (66). Thus, the twisted string can be characterized by the space group element g = (θ n , k m k α i k ). The product of the two space group elements (θ n 1 , v 1 ) and (θ n 2 , v 2 ) is computed as (θ n 1 , v 1 )(θ n 2 , v 2 ) = (θ n 1 θ n 2 , v 1 + θ n 1 v 2 ). (B2) The space group element g belongs to the same conjugacy class as hgh −1 , where h is any space group element on the same orbifold.
Now, let us consider the couplings among twisted strings corresponding to space group elements (θ n k , v k ). Their couplings are allowed by the space group invariance if the following condition: is satisfied up to the conjugacy class. That includes the point group selection rule, k θ n k = 1, which is the Z N invariance on the Z N orbifold. We can define discrete Abelian symmetries from the space group invariance as well as the point group invariance. These symmetries together with geometrical symmetries of orbifolds become non-Abelian discrete flavor symmetries in heterotic orbifold models. We show them explicitly on concrete orbifolds.

S 1 /Z 2 orbifold
The S 1 /Z 2 orbifold has two fixed points, which are denoted by the space group elements, (θ, mα) with m = 0, 1, where α is the lattice vector. In short, we denote them by (θ, m) and the corresponding twisted states are denoted by σ (θ,m) . These states transform under the Z 2 twist. In addition, the space group invariance requires k m k = 0 (mod 2) for the couplings corresponding to the product of the space group elements k (θ, m k ) with m k = 0, 1. Hence, we can define another Z 2 symmetry, under which σ (θ,0) is even, while σ (θ,1) is odd. That is, another Z 2 transformation can be written by There is also the permutation symmetry of the three fixed points, that is, S 3 . Thus, the flavor symmetry is ∆(54) ≃ (Z 3 × Z 3 ) ⋊ S 3 .

T 2 /Z 4 orbifold
As shown in Section III, the T 2 /Z 4 orbifold has two θ fixed points denoted by (θ, m) with There is also the permutation symmetry of the two fixed points. Thus, the flavor symmetry is almost the same as one on the S 1 /Z 2 orbifold. The difference is the Z 4 twist, although its squire is nothing but the Z 2 twist. Hence, the flavor symmetry can be written as (D 4 × Z 4 )/Z 2 .

T 2 /Z 2 orbifold
As shown in Section III, the T 2 /Z 4 orbifold has two θ fixed points denoted by (θ, (m, n)) with m, n = 0, 1, and the corresponding twisted states are denote by σ θ,(m,n) . The space group invariance requires k m k = j n j = 0 (mod 2) for the couplings corresponding to the product of the space group elements k (θ, (m k , n j )) with m k , n j = 0, 1. There are two