Denise Cote, a senior federal judge, ruled that RBS and Japanese bank Nomura made false statements when they sold the securities prior to the 2008 financial crisis.
Cote said that the “offering documents did not correctly describe the mortgage loans”, and that "the magnitude of falsity, conservatively measured, is enormous".
The US regulator could recover around $450m (£289m).
The lawsuit is the first to reach trial out of the 18 banks that sold $200 billion of mortgage-backed securities to Fannie Mae and Freddie Mac.
