Discussing the economic and financial costs and benefits of UK's EU membership with the Treasury Select Committee, Carney said that "certain firms would take a view in terms of relocation".
He added that it would be vital to negotiate with the EU to keep banks in the Capital and give recognition to the financial services sector in London.
However he raised concerns that negotiations to protect the current landscape would take a long time, and that Britain would almost certainly lose influence over regulations.
Discussing the Brexit more generally, Carney said that "the situation could bring some challenges to financial stability".
Earlier in the hearing, Carney said that leaving the EU could lead to instability which is associated with “poor economic outcomes”. However he said it was not in the Bank of England's remit to speculate on how a Brexit would impact jobs or prices.
Conservative MP Jacob Rees-Mogg criticised Mark Carney for making “speculative” pro-EU claims.
However Carney stressed that "we will not be making, and nothing we say should be interpreted as making, any recommendation with respect to that decision".
He did however say that EU membership had "likely increased the dynamism of the UK economy and correspondingly its ability to grow without generating risks to the Bank's primary objectives of monetary and financial stability".
