Reflection-Symmetry Protected Antiferromagnetic Topological Insulator in Three-Dimensional Heavy-Fermion Systems

Nowadays, the notion of band topology [1, 2] has become much more ubiquitous and the topological materials are of great interest in condensed matter physics. The characteristics of topological insulators (TIs) is insulating bulk and their metallic edge (surface) states which are protected by local symmetry. Furthermore, the notion of topology has been generalized to metallic (gapless) phases [3–5] such as topological semimetals or generalized to phases with crystalline (non-local) symmetry such as topological crystalline insulators [6, 7]. The topological properties of materials have been of great interest from the viewpoint of theoretical and experimental investigations. Recently, the effects of Coulomb interaction on topological phases have attracted much attention because of nontrivial features that do not emerge in weakly correlated systems [8–11]. For example, topological Kondo insulator [12–14], magnetic orders in topological insulating phases, and the change of topological classification [15,16], etc. In the presence of the strong correlation effects, topological states become much more enriched and interesting because of an interplay between topology and correlation. In previous studies, focusing on the topological properties of antiferromagnetic (AF) phases [17, 18], an AF topological insulator (AFTI) has been proposed for three-dimensional (3D) systems, which is referred to as a Z2-AFTIs characterized by a Z2 topological number. In such AFTIs, the AF order breaks time-reversal (Θ) and primitive-lattice translation (T1/2) symmetries but preserves their combined symmetry S = T1/2Θ. Under this symmetry, the system has a Z2 topological number, which is related to a strong index of 3D topological insulators with time-reversal symmetry. Note that this Z2 number is allowed only for 3D systems. Recently, it is found that the van der Waals layered compound MnBi2Te4 [19] could be a possible candidate for Z2-AFTI. In f -electron materials, SmB6 and YbB12 are good candidates of topological Kondo insulator, providing a good platform to study an interplay of Coulomb interaction and band topology. Although SmB6 [20, 21] has been known as a Kondo insulator for about 40 years, it has been a long-standing problem about the saturation of its electrical resistivity below 4 K. It has been recently proposed that


Introduction
Nowadays, the notion of band topology [1,2] has become much more ubiquitous and the topological materials are of great interest in condensed matter physics. The characteristics of topological insulators (TIs) is insulating bulk and their metallic edge (surface) states which are protected by local symmetry. Furthermore, the notion of topology has been generalized to metallic (gapless) phases [3][4][5] such as topological semimetals or generalized to phases with crystalline (non-local) symmetry such as topological crystalline insulators [6,7]. The topological properties of materials have been of great interest from the viewpoint of theoretical and experimental investigations.
Recently, the effects of Coulomb interaction on topological phases have attracted much attention because of nontrivial features that do not emerge in weakly correlated systems [8][9][10][11]. For example, topological Kondo insulator [12][13][14], magnetic orders in topological insulating phases, and the change of topological classification [15,16], etc. In the presence of the strong correlation effects, topological states become much more enriched and interesting because of an interplay between topology and correlation.
In previous studies, focusing on the topological properties of antiferromagnetic (AF) phases [17,18], an AF topological insulator (AFTI) has been proposed for three-dimensional (3D) systems, which is referred to as a Z 2 -AFTIs characterized by a Z 2 topological number. In such AFTIs, the AF order breaks time-reversal (Θ) and primitive-lattice translation (T 1/2 ) symmetries but preserves their combined symmetry S = T 1/2 Θ. Under this symmetry, the system has a Z 2 topological number, which is related to a strong index of 3D topological insulators with time-reversal symmetry. Note that this Z 2 number is allowed only for 3D systems. Recently, it is found that the van der Waals layered compound MnBi 2 Te 4 [19] could be a possible candidate for Z 2 -AFTI.
In f -electron materials, SmB 6 and YbB 12 are good candidates of topological Kondo insulator, providing a good platform to study an interplay of Coulomb interaction and band topology. Although SmB 6 [20,21] has been known as a Kondo insulator for about 40 years, it has been a long-standing problem about the saturation of its electrical resistivity below 4 K. It has been recently proposed that the saturation originates from the topological surface state and this has been confirmed by angleresolved photoemission spectroscopy measurements [22,23]. More recently, another aspect of SmB 6 has been addressed experimentally. A non-magnetic topological Kondo insulator, SmB 6 , at ambient pressure turns into a metallic AF phase with increasing the pressure above 6GPa [24][25][26][27][28], but its magnetic configuration is not yet clarified experimentally. The topological properties of the AF phase have attracted much attention from the viewpoint of theoretical and experimental investigation. From the first-principle calculation, it is pointed out that the pressurized SmB 6 would serve as a better candidate for a Z 2 -AFTI and the A-type AF (A-AF) configuration is proposed for its ground state [29].
In this paper, we focus on the topological properties in AF phases from the viewpoint of crystalline symmetry in addition to the above Z 2 -AFTI. To be specific, we here consider it in the 3D heavy-fermion systems and demonstrate how the paramagnetic (PM) phase changes into the AF phase by using the Hartree-Fock (HF) approximation. We focus on the A-AF phase of SmB 6 which is proposed by the first-principle study [29]. In the previous study, we proposed a reflection-symmetric AFTI phase in 2D [30], which can have a topologically nontrivial structure specified by a mirror Chern number. We here demonstrate a reflection-symmetric AFTI phase in 3D and point out the possibility of material realization in pressurized SmB 6 .
The rest of paper is organized as follows. In Sec. 2, we describe our setup and give a brief explanation of our approach. In Sec. 3, we study the topological properties of the AF phase. The last section 4 is devoted to a brief summary.

Model and Methods
We consider the following periodic Anderson model with nonlocal d-f hybridization in the orthorhombic crystal in Fig. 1(a), with iσ is the number operator, k is a wave number, and σ i and τ i (i = 0, x, y, z) are the Pauli matrices for spins and sublattices. The annihilation operators are defined as and t ′ f are hopping parameters, V denotes the d-f hybridization, and ǫ f is the difference between the d-and f -electron energies. The topological properties stem from nonlocal d-f hybridization [12,13] which reflects spin-orbit coupling.
In order to study the topological properties of the AF phase in Eq. (1), we employ the following HF approximation for the Hubbard interaction term: Here, · · · denotes the expectation value at zero temperature. Thus, the mean-field Hamiltonian is given by

Topological Properties of Antiferromagnetic Phases
The non-magnetic topological Kondo insulator SmB 6 at ambient pressure turns into a metallic AF phase by high pressure above 6GPa, although its magnetic ordering pattern is not clarified experimentally. In a previous theoretical study [29], it is pointed out that the ground state is the A-AF phase with m ẑ and the Z 2 -AFTI phase emerges in this AF phase. In this paper, motivated by recent experimental studies in pressurized SmB 6 , we consider the topological properties of the AF phase from the viewpoint of reflection symmetry. In our study, we consider the A-AF phase as shown in Fig. 1(a) with m ẑ and m x. The model we employ here is a topological mirror Kondo insulator which was previously used to address a nonmagnetic cubic Kondo insulating phase [31] with band inversion at X-point [20,21] which is a two-orbital effective model of SmB 6 in its PM phase. We elucidate below that the system can change form a PM phase to an AF phase without breaking its reflection symmetry, thus leading to an AFTI protected by the reflection symmetry.

Definition of topological invariants
In order to consider the magnetically ordered topological insulating states with a reflection symmetry, we introduce the corresponding topological number. In the reflection-symmetric plane in 3D Brillouin zone (BZ), all the eigenstates are characterized by their reflection parities and divided into two subspaces as where M means the reflection operator with M 2 = −1 and its eigenvalues are M = ±i. The Chern numbers of each reflection subspace C M=±i can be defined on the 2D subspace of 3D BZ, like k z = 0 plane and it is given as where Next, we discuss the Z 2 invariant of the AF phases [17] with inversion symmetry. In the collinear AF phase, the AF order breaks time-reversal (Θ) and primitive-lattice translational (T 1/2 ) symmetries but preserves their combined symmetry S = T 1/2 Θ. The topological invariant of this AF phase with inversion symmetry is given by a Z 2 number ν(= 0, 1) as follows, (−1) ν = k m δ m , where k m are four Kramers degenerate momenta and δ m = i ξ i (k m ) are products of the parity ξ i (k m ) of the occupied band i. At the Kramers degenerate momenta, the parity of d-( f -) orbital is given by ξ i (k m ) = +1(−1). We note, in A-AF phase with the doubled z-axis, that the Z 2 number can be defined by the product of each parity at X, S, Y, and Γ in Fig. 1(b).

Numerical results
First, we here discuss the obtained results for the A-AF phase with m ẑ at half-filling as shown in Fig. 1(a). In this case, the independent reflection plane is only k z = 0 where the reflection operator M inverts the z-axis, thus we consider the mirror Chern number C m0 on k z = 0 plane. The values of the parameters we employ in the following are t d = 1 (energy unit), In this parameter set, the non-interacting Hamiltonian realizes the strong topological insulator with band inversion at X-point [31,32]. The magnetic properties are studied by the HF method and the Chern number is calculated by the method proposed in Ref. [33], which is efficient for numerical calculations. We obtain some AF topological (AFT) phases for V = 0.8 as shown in Figs.1(c)-(e). Note that the results are not sensitive to the value of V and we discuss it later. With increasing the interaction U f , the PM phase changes into the AF phase at U f = u c with a large hysteresis loop in Figs.1(c) and a magnetic configuration in Fig. 1(a). Figure 1(d) indicates that in the weak interacting and PM region U f < u 1 , the topological insulating state is characterized by non-trivial topological numbers (ν, C m0 ). In the strongly interacting and AF region u c < U f < u 5 , the AFT phase which is characterized by the Z 2 number and the mirror Chern C m0 numbers emerges. In the hysteresis region, the mirror Chern has a hysteresis loop which is induced by the change of the band structure in the AF phase. The above topological phase transition points are consistent with the gap closing point of direct gap in Fig. 1(e). In the AF phase, the system becomes metallic where the indirect gap is closed in U f > u 4 (or u 3 ) in Fig. 1(e). Note, however, that the topological properties still remain intact in this region because the direct gap is not closed. The topological numbers are still well-defined in the region where the direct gap is open. We confirm an aspect of the reflection symmetry in an AFT semetallic phase characterized by a mirror Chern number and a Z 2 number.
Summarizing all these results with m ẑ, we obtain the phase diagram shown in Fig. 2(a). We also study the case of m x in Fig. 2(b). The horizontal axis denotes the strength of the interaction U f and the vertical axis the strength of hybridization V. The above analysis is done for V = 0.8 on the blue line in Fig. 2(a). In the AF phase for these parameters, a semimetallic AFT phase is realized, which we refer to as an AFT semimetal. The spin configuration in the AF phase is shown in Fig.  2(c), where the d-and f -magnetization are pointed in the same direction on the same site. The mirror Chern number and the Z 2 number have various values in the phase diagram, which are changed by the shift of the f -band.
In addition, we discuss another case with m x in Fig. 2(b). In this case, the independent reflection planes are k x = 0 and k x = π where the reflection operator M flips the x-axis, thus we consider the mirror Chern numbers C m0 and C mπ on k x = 0 and k x = π planes. We obtain some AFT semimetallic phases. A prominent feature in this case is that a reflection-symmetric AFT phase emerges, which is characterized by only non-zero mirror Chern number (0, 0, −2) and has never been discussed in the previous study. The spin configuration in the AF phase is shown in Fig. 2(d), where the d-and f -magnetization are aligned in the opposite direction on the same site.

Summary and Discussion
In this paper, we have analyzed the AFT phase in 3D by taking into account reflection symmetry. Specifically, motivated by recent experimental studies of the magnetic metallic phase in pressurized SmB 6 , we have elucidated the novel aspect of the reflection-symmetry protected AFT in two types A-AF phases. Our numerical results have revealed the emergence of the 3D topological crystalline insulating states in AF phase for interacting systems such as A-AF phase in SmB 6 . In particular, we have shown the AFT states characterized by mirror Chern numbers in addition to Z 2 number. In the case of m x, a reflection-symmetric AFT semimetallic phase appears.
We finish this paper with a comment on a related work. In Ref. [29] a first principle calculation for SmB 6 has elucidated the surface excitation spectrum for the magnetic phase. In that article, it has been reported that at a reflection plane, there exist the gapless surface states which cannot be understand only with the magnetic translation symmetry (i.e., the product of the translation and the time-reversal operation).
We consider that our results elucidating the additional topological properties with reflection symmetry may explain the origin of these gapless states whose origin remains unclear. In order to understand it, detailed numerical calculations are required. We leave this issue as a future work.
This work was partly supported by JSPS KAKENHI Grant No. JP15H05855, JP18H01140, JP18H05842 and JP19H01838. The numerical calculations were performed on the supercomputer at the Institute for Solid State Physics in the University of Tokyo, and SR16000 at Yukawa Institute for Theoretical Physics in Kyoto University.