Over the years I've heard a lot of arguments about complexity, but I've never heard any any refutation of the related ideas that a) most humans learn their native language adequately in about a ten-year period, and b) really good language grammars are of roughly equal size.
As Dixon puts it, it's certainly possible that some languages are twice as complex as others, and no one really maintains that we know that languages are
less differentiated than this in complexity. Do the critics seriously maintain that some languages are, say, ten times as complex as others? E.g., that they take ten times as long for a native child to learn?
I really think a lot of people, including unfortunately a few linguists, approach the problem primarily thinking about teaching the written language in a college course. When the average non-linguist thinks about complexity, that's what they mean too: 2L acquisition with a focus on the difficulties of the first year, or even the first month. (I expect most English speakers would think "Russian is hard" because you have to learn an alphabet, not because you have to learn all the cases.) In that sense, there's certainly a lot more to do in the first year of Latin than, say, Swedish. If you look at things like "when can you read an ordinary newspaper article", the answer might be "four months" for French and "six years" for Mandarin.
But again, is it really maintained that, say, a Swedish child will learn the language ten times faster than a Roman child? Things like Romance verbal morphology are hard for the college student. And they're hard for the toddler, too! But they have the time and energy for it, and can't complain effectively about it, and morphology is really dwarfed by all the lexicon they have to learn anyway.
Subjectively, some languages do really seem intimidating. I have Valentine's Nishnaabemwin grammar, and it scares me. But that's not a linguistic judgment, and it's arguably just provincialism. The complexity is not what I know from other areas. Plus, honestly, Valentine is not the best writer.
There are a bunch of caveats, of course, which I'll briefly go over to avoid straw men. There's an obvious increase in complexity due to writing systems, which are
not all equal. There's a huge increase when to be educated you have to learn a second language-- most obvious for (say) Akkadians who had to learn Sumerian, but also for medieval Europeans who had to learn Latin, or pre-1905 Chinese who had to learn Old Chinese. There's also arguably a lot of complexity in technical terms, and in languages with a long, preserved history. Most of it is arguably not linguistic, but some is. That's one reason a 12-year-old, though fluent, may still have an awful lot left to learn. Plus of course pidgins and conlangs may be less complex. Possibly creoles, though I seldom hear what actual creole speakers think about this.
Finally, my point (b) above is of course highly informal. Brad already quoted my standard invocation of Li & Thompson. But I think people still routinely underestimate the complexity of English syntax. I've written a 300-page book on it, and believe me, it's a primer. McCawley's textbook is 800 pages and many of his sections are bare overviews of the phenomena they discuss. Compare how many pages it takes to list, say, all the irregularities of Latin morphology.
(Once when I brought this up, I think someone said that we don't know that Latin syntax wasn't equally complex. That may be so! But the advocates of particular-language complexity can hardly just make the assumption that, oh, all syntax is equally complex. It's reasonable, though hardly proven, that the work not done by a language's morphology has to be done by its syntax.)
Ah, and I should add that I haven't read Trudgill's book and I'm not trying to address it. I have nothing against the idea of sociolinguistic barriers to 2L acquisition; in fact I think it's a big part of why the major standard languages retain features that are hard to learn.