Ultrafast electrochemical supercapacitors (EC) that can work at or above kilohertz (kHz) frequency, 3-4 orders higher than traditional EC, call for a structure with extremely low equivalent serial resistance (ESR) and a reasonably large surface area. Three-dimensional perpendicularly-oriented graphene (POG) network, grown inside of Ni foam (NF) by microwave plasma chemical vapor deposition, is reported as electrode to fabricate such ultrafast EC. The folded POG inside NF provides a large surface area, while the straight-forward and wide-open porous structure of POG ensures fast ion migration. In conjunction with the intrinsic high electronic conductivity of graphene and Ni, POG/NF electrode based ultrafast EC was demonstrated with a specific cell capacitance of 0.32 mF/cm2 at 1 kHz, a relaxation time constant of 0.248 ms, and an ESR of 70 mΩ. A charge-discharge rate as high as 500 V/s was also measured, at which the cyclic voltammogram maintained a rectangular shape, corresponding to a single electrode capacitance of 0.83 mF/cm2.