Oxford High Street, 1890s photochrome.
One of the most interesting land use riddles is the question of how traditional towns and cities so often achieved such good results in terms of their layouts, massing, and architecture, given the much smaller toolbox of legal devices that existed prior to the early 20th century. Did people simply have better eyes in the past? Were they more inclined to cooperate with their neighbors? It’s hard to believe that human nature has really changed that much.
Here are some things I’ve been able to come up with. I’m sure there are others:
(1) Tradition. Some writers have suggested that the advent of more thorough land use laws actually tied the hands of builders who had once worked in their own rich tradition of aesthetic solutions. As early as 1909, Sir Raymond Unwin observed that English building bye-laws were twisting the architecture of their communities in unintended ways. Intuitively, this observation makes a good deal of sense, but it doesn’t answer the entire question. That is to say, if we accept that certain aesthetic traditions of folk architecture suffered a fatal blow when they ran up against the advent of technical legal requirements, we still don’t know what motivated their evolution in the first place; or what has prevented them from adapting, over the subsequent decades, to the fairly standard legal and transportation paradigm shifts that are now nearly a century old. In other words, we still have the basic question: why do people build so much crap these days? This question may be begging another, more fundamental one about how much crap really was built in the past. That is because low-quality buildings are more likely to have fallen apart or been torn down since they ceased to be new, making the new ones much easier to find. And even at that, a trip through certain parts of Jersey City would dispel any romantic notions about US urbanism during the Victorian period. Still, there is something uniquely awful about certain elements of the post-World War II American landscape.
A fire insurance map of Brooklyn from 1868 shows building footprints and owners' uses. Source: NYPL.
I think solving the riddle about what’s gone missing requires figuring out what fostered the greater communication between parcels in towns of the past. This is a separate question from what caused the decline in quality of the architecture of individual structures, and it goes much more to the essence of what effective land-use planning ought to address. That is, a dozen boxes could be arranged along a table in a logical way, or they could be scattered around haphazardly. When assessing whether you have order or chaos, the colors and shapes of the individual boxes are less important than their qualities and placement with regard to each other. And while there is a basic order to the technical arrangement of Euclidean suburbs, even a cursory comparison of a cascade of strip malls and a 19th century town center would reveal that Euclidean zoning doesn’t approach the intricacy and specialization that can be found in traditional towns and cities. So, aside from tradition, what factors drove the organization of urbanism– when it worked– before land use laws?
Leicester Square, London. Preserved by Tulk v. Moxhay.
(2) Private law devices. Between private parties covenants were available at common law. Covenants could be useful for ensuring a certain consistency across development after a parcel was subdivided, but they were somewhat limited in their permanent application because, in order for covenants to bind subsequent title holders, the law required something called horizontal privity between their original parties. Essentially, this meant that the affected parcels had to have originated from a single property, or at least from owners who shared some legal interests when they made their agreement. Because of this, neighboring owners couldn’t simply covenant to have their properties bound by a set of rules in perpetuity. Eventually, more flexible equitable servitudes were recognized by chancery courts in America (early 19th century) and England (following the 1848 case of Tulk v. Moxhay), but I have no idea whether these played a significant role in town planning.
(3) Building codes, bye-laws, etc. In Town Planning in Practice (1909), Unwin described the UK’s building bye-laws, which were forerunners to modern building codes and schedules of zoning regulations. Their scope included a lot of what’s regulated today– but not uses:
Under the modern urban bye-laws adopted in most English towns the number of houses to the acre is … limited by the regulations which fix, first, the minimum width of streets; secondly, the minimum space allowed to be left at the back of buildings. 
Building regulations cover such a multitude of matters, and the combination of circumstances under which difficulties may arise are so numerous, that it is quite impossible to frame absolute regulations on all these points without a considerable amount of needless harassment and restriction of really good buildings. 
So, much of the regulation that now falls under the zoning umbrella was established well before the advent of land use zoning, in the forms of building codes, fire codes, and health and safety regulations. The history of how these ordinances came about, and became increasingly elaborate, would likely shed more light on some of the dimension standards that guided the layout and massing form of traditional towns.
(4) Physical constraints. Before the onset of industry, building materials and engineering capabilities simply did not permit the height or sprawl that are now possible. Also, the distances between buildings and the canals, highways, and railroads had to be walkable, or at least sufficiently short to travel in reasonable times with the help of animals.
(5) The market. The realities of physical constraints put a premium on maximizing land use efficiency on parcels close to transportation arteries; with the hard physical work of transportation as its alternative, crowding was better than distance. The tension between the desire to maximize the use of space and the physical and economic constraints of pre-industrial building techniques presumably resulted in the relative consistency of the traditional urban scale.
From all of this, we can start to see the landscape that influenced traditional urban development in the common law world. Before land use planning, a town developed in the matrix of a delicate balance of tradition, private covenants, ad hoc bye-laws, physical constraints, and market forces. Presumably, community social pressures also played a role. I suspect that this balance more accurately reflected the interests of communities and their individuals than what we have today, and that this balance is what began to fail when confronted in the mid-19th century by the super-capital of large industry. In the wake of that moment, the traditional cultural-legal-market constellation was simply lost.
The story of what’s happened since that time is the familiar one.