Perhaps it happens at the dead of night when you are fast asleep, or waiting for a bus. Either way it takes time and depth to create a world for fantasy writing. Tiny scraps of paper soon add up. Then the note books start piling up, and at some stage you find yourself surrounded with lots of little snippets. A world can take a great deal of planning this can be rather frustrating when the end is far from sight. It is like giving yourself permission to linger in the moment and wonder how time works, how plants grow, how a city thrives and falls. These are all great questions that can remain unanswered but this is the time to think things through. It is not about finishing anything, simply using the time to wonder about connections.
Not all things come together at the same time, especially when world building. Some of it can be quite slow, and like civilizations ideas will rise and fall. Over time some will linger just long enough to wedge their way into your memory and develop further in the space between thinking. Worlds can be old or young, have different dangers, civilizations will rise and fall. Somewhere between all this a story will emerge.
The real world is full of unknowns, dangers and the unseen. This does not have to be bleak yet what it does mean that there are many layers. World building requires depth, a perception of history rather than an explanation. There is no need to describe every detail as long as the main threads are there. Starting a story is like beginning in the middle where no end has been reached. Where there is the illusion that the created world will continue on long after the story is gone. The world is a robust place, full of life and connections that may not be clear at first glance. Spend time in your world before you begin, get to know how it feels then let it take hold, and to a certain extent let go. World building is about creating something that can thrive and expand.