We report the growth of hexagonal boron gallium nitride alloys, h-GaxB1-xN, on hexagonal boron nitride (h-BN) templates by metalorganic chemical vapor deposition and the observation of the critical thickness (LC) phenomenon in the h-GaBN/BN heterostructure system. It was observed that GaxB1-xN alloys in a pure hexagonal phase can be obtained when the film thickness is below LC. X-ray diffraction (XRD) θ-2θ measurement results revealed the formation of separate wurtzite (w)-GaN domains within the h-GaBN matrix when the film thickness is beyond LC. XRD results were supported by photoluminescence spectroscopy which revealed the absence of the band edge emission of w-GaN near 3.4 eV in very thin layers, but an increase in the w-GaN band edge emission with an increase in the h-GaxB1-xN layer thickness beyond LC. Despite the fact that layered structured materials generally possess a weak interlayer interaction, our results revealed that phase separation still occurs in layered h-GaxB1-xN alloys and the critical thickness depends on the Ga composition. The present study also provided insights into possible ways to synthesize layered GaBN/BN heterostructures and quantum wells in the pure hexagonal phase with tunable bandgaps and optical properties, which would open up many new applications.