How important is setting?
As a reader and not a writer, the first thing I really take note of is how the author has created his setting. Some masters of world-building include Tokien, David Eddings, and unquestionably George R.R. Martin (even though it's a bit strong on all the hetero-sexual annointing of said areas "down there", women performing needlework, and little girls being the object of desire for grown men). But hey, whatever George...amirite?
Now, in film... one Director really stands out and that's James Cameron in the blockbuster hit Avatar. Think of how real Pandora looked what with its floating mountains, flying monsters, and glowing moss. It was an astounding vehicle for the delivery of that story (which was essentially just a re-telling of Pocahontas).
In my opinion, aspiring writers disregard the idea of world-building and just slog on with characters that are nothing more than humans with a little quirk. Example: So-and-so is an angel/ demon/ vampire/ werewolf and the story is going to be about how unusual this is and how they come to terms with the issues and prejudices associated with these roles.
Your setting will help you develop your plot. A plot that takes place in outer space is going to be a lot better developed for your story of industrializing the moon, than is say...a tropical forest. But I don't think it has to be hard. If you keep in mind what kind of story you're telling, you will automatically have ideas about your setting.
Readers like to understand why characters do the things they do. In Million Dollar Baby, despite the fact that Hillary Swank decides that it's best for her to just die at the end of the show, we know that this is a choice that her character would have made. It may not be the same choice that everyone else makes, but the setting (her being a fighter, her family life, and the fact that she was a reflection of all the elements of the setting around her) made it irrefuteable that death is what that character would have chosen.
Anyway...I'm linking the above story on the Kowloon walled city here so that you can watch it. I think that it is an amazing example of a real-life setting that stepped right out of the pages of some science-fiction book. It's no longer around, but wow... I could think of ten or more stories that could have taken place in there.