UESPWiki talk:Oblivion Map Design/Problems with the SI Map

This is an archive of past UESPWiki talk:Oblivion Map Design discussions. Do not edit the contents of this page, except for maintenance such as updating links.

Problems with the SI Map

In a nutshell, the SI map isn't showing a full picture of the Shivering Isles. Compare this from our map to this taken directly from the CS. Our map is missing most of the trees and bushes shown on the CS version.

This is the same problem that has prevented me regenerating the Oblivion map and running my artifact removal program over it to get rid of the nasty lines and blobs. Compare this from our map to this that I get when I use the "Create Local Maps" option.

It seems therefore that we have two options:

1. Ignore the problem. Leave SI as it is and regenerate OB using the treeless tiles.
2. Get whoever did the original OB maps to do them again, and also do the SI ones.

In favour of option 1 is that nobody has complained about the lack of detail on the SI map, but I did once get a little confused while trying to match up my in-game surroundings to the map give the lack of trees.

In favour of option 2 is that we really ought to be trying to fix things if we know they're wrong, and I'd rather not mess up the OB map either.

So what does everybody else think? –Rpeh^{•T•C•E•} 08:46, 10 April 2009 (EDT)

Does nobody else have any thoughts on this at all? –Rpeh^{•T•C•E•} 04:34, 14 May 2009 (EDT)

Yup I do. It's just that I missed this one, but we have discussed this before on IRC. I'll get back to this later, have to hurry this reply. Just to let you know I haven't missed it this time ;) --Timenn < talk > 10:47, 14 May 2009 (EDT)

I tried creating the [-1,1]-[1,-1] tiles on my computer and at first, same thing: no trees. But then I explored the CS options. Under File->Preferences->LOD, there's a check box "Force Full Tree LOD", which was not originally checked. Checking that box, then regenerating the maps, I get the trees. I tried the [-1,-1] cell a few times, with/without checkbox, and it seems to pretty reliably control whether or not trees appear. However, the check box seems to behave a bit weirdly: it always seems to reset itself to off after you create a set of maps (is it even possible it resets itself during a set of maps, thus causing inconsistency?) I also experimented minimally with some of the other settings under Preferences (LOD for Items, Objects; clipping distances under Render Window). Notably, something I did appeared to turn off some of the red outline artifacts, but instead added some black blob artifacts, but I'm not immediately sure what ;)

Any chance those settings have an effect for you, too? If not, I could see whether my luck with [-1,-1] extends to other cells. It also raises questions about what the other settings do and whether there's some combination that minimizes (or even eliminates?) the various artifacts. --Nephele^Talk 18:37, 3 June 2009 (EDT)

I've just spent much too long fixating over what I did in my earlier tests that changed the artifacts on the map (turned off red outlines; turned on black blobs), and then just as mysteriously reversed them again. As a result, I've continued to experiment, finding out for example that the options under the View menu are also important. Some of the things I've found out:

• Under the View menu, both Leaves and Trees need to be selected.
• Turning Markers off gets rid of some unnecessary artifacts, but not many.
• Definitely do not turn on Wireframe or Collision Geometry -- all the extra overlays will appear on the map tiles, too!
• The control that seems to influence the artifacts the most is fade mult for objects (under Preferences->LOD). As you decrease the value of fade mult, more and more objects disappear from the map. At low enough values, artifacts such as red outlines disappear (but so have a lot of useful objects). And at various intermediate values (not at max or min), black blobs appear in place of certain objects, as the items transition from visible to missing. I've come to the conclusion that it's best to keep fade mult set to the maximum, because that provides the most detail. Plus, rpeh already has a better way to remove the red outlines.

The effects of all these options are clearly visible in the render window (even the tree LOD, in particular if you scroll out in the render window and get a broader view: the trees don't disappear, but they do become flattened/distorted if tree LOD is off). What seems to be the most unpredictable is whether or not the options take effect on any map tiles I generate. Sometimes, the map tiles match my currently selected options; other times, they seem stuck on old set of options, and won't update. Whether updates occur seem somewhat related to what cell is displayed in the render window, and whether or not I refresh that window, but there's something random going on too (or something I don't understand). Whether that has implications when you're generating thousands of cells at a time, instead of just 1-10 cells, I don't know.

Finally, under World->World Testing, there's an "Update Distant LOD Data" menu, where it states "You must export LOD for all worldspaces in order to be able to see objects such as trees and buildings in the distance". Clicking the export option causes some type of lengthy computation, but I have no idea what it actually did; map tiles did not appear any different afterwards FYI, I did this after nearly all my other tests. --Nephele^Talk 21:09, 3 June 2009 (EDT)

Looks like you've got the same problem I have - able to get to the point where some trees appear but not all.

The "Markers" option turns on and off the door / north markers on the map, but most artifacts come from collision boxes, activators and creature markers that are actual objects on that map. The only way to remove them is to manually hack the map to remove them all or to do the kind of post-processing I did for the SI map.

I must have spent hours playing around with the LOD settings. Everything appears perfect on the render window of the CS but generating local maps doesn't show all the trees. I've tried it on big sections, small sections, vanilla and SI, and everything else I can think of. Looks like we'll need Daveh to re-export the tiles. –Rpeh^{•T•C•E•} 00:41, 4 June 2009 (EDT)

Or another idea: have you tried using the original (1.0.x) version of the construction set? Which in turn also means digging up an unpatched version of Oblivion.esm (I already had one that I use to check what data has changed). I just started the full map set using the original construction set, and so far I'm getting trees (I last checked [-52,1], after starting from the default -64,x). If that's the solution, then there'll be some extra complications trying to figure out what to do for the SI map, but it's at least some progress. --Nephele^Talk 01:19, 4 June 2009 (EDT)

I've played around with those settings in the past too, but I noticed the trees appear quite random. In other words, it's very hard to make a complete tree coverage. But I might have another idea.

Instead of trying to make the CS export all trees correctly, we can pick up what we have, and add the trees ourselves. I believe it's possible to extract all tree locations from the SEWorld and their scale. Then we use a script to add the various tree images to the various map images, based on the location and scale data we retrieved. Take a look at Image:Tree_PasteIn_Example.png to see what I mean with tree images.
Considering a tree type always looks the same (apart from difference in size) in the CS, we need not worry about losing diversity in the map images. --Timenn < talk > 06:00, 4 June 2009 (EDT)

Using CS 1.0.303, I seem to be getting complete tree coverage (now at column 17, i.e., everything from Bravil west done). Every tile I've looked at looks good: either it has trees, or it matches the tiles that Daveh originally created. Actually, in many cases better than Daveh's tiles: I just checked the area around Wellspring Cave, and I don't have any black blobs, but instead I've got the correct rocks. So worst case, even if I'm the only person who can get it to work in the old CS, I think we'll have one set of good tiles. --Nephele^Talk 12:37, 4 June 2009 (EDT)

Yes! I get the same result. I never thought to try the old version of the CS... (curses).

Unfortunately, hacking an SI version of Oblivion.esm so it's loadable in the old CS doesn't work - the local maps come out in an incredible lurid purple. In the worst case, this at least means we can have a clean Oblivion map with full detail and the current clean SI map without some trees. Best case is that we can hack things somehow so that the SI version has trees added - perhaps using Timenn's method.

I'm going to spend some time tweaking my artifact remover to make sure I'm getting as much as I can without impacting the map (the reason I found this problem in the first place was that I thought I was smoothing out the red trees...) Once I've got a good version we can work out whether it's going to be easier for me to run and upload, or give you the code so you can do it. –Rpeh^{•T•C•E•} 13:59, 4 June 2009 (EDT)

Take a look at this ZIP file. It contains 45 JPGs showing tiles before and after artifact removal. I think it's worked pretty well - especially around the Temple of the Ancestor Moths. There's a bit of blurring in a few placed, however, so I wanted to see what people thought before going forward.

A couple of other points: first, these are the tiles before resizing so several of them have weird black borders: just ignore those. Second, there are a few pinkish dots to be found. I spent quite some time trying to remove those without destroying everything else before nipping into the CS and discovering that they're mushrooms so should probably still be there.

I'm going to try to do higher-res versions of the cities next, but please tell me if you think these are okay. –Rpeh^{•T•C•E•} 04:42, 6 June 2009 (EDT)

You already convinved me earlier with the cleanup program. ;)

Now that we are busy with these tiles, should we consider adding water? I realise this will require much work, but it would make the map look better. We might be able to use the typical greyish brown color to define the water areas. Only Blackwood has a different ground texture. --Timenn < talk > 07:17, 6 June 2009 (EDT)

I'd love to fix the water, but I've spent a long, long time trying to do so and it's horrible. As far as I can see there are three basic approaches:

1. Post-processing based on colour. Adjust the shade of every pixel withing a certain range of colours (roughly what I do for artifact removal). This doesn't work because there are just too many colours that represent water. "Greyish brown" somewhat oversimplifies the range of colours used in water - even just the water in the River Niben. The main problem is that the colours are very similar to roads, farmland and even buildings. Take a look at the area near Barren Cave, for instance. Without knowing, can you tell which areas are road, which are path, which are water and which are shore? A variant of this approach would be to "flood fill" areas of water - ie, start with a pixel known to represent water and scan the surrounding pixels to see if they are similar. This is slightly less vulnerable to the problem described above, but introduces a new one: the number of "seed" points required. Look around Fort Doublecross and try to work out how many would be needed.
2. The other method is to delve into the game data, calculate the height of each pixel, compare it to the height of the water in the cell and colour accordingly. The trouble is that this requires not just looking at the terrain (in which case it would be fairly easy) but the objects in the cell. In that Fort Doublecross example, for instance, all the trees would end up shaded blue if objects aren't considered.
3. The other approach would be to do it all manually. That's possible... but it's way beyond one person. We'd need to package up groups of cells for download and get a project going.

Unless someone can come up with a better idea, I imagine we'll end up doing #3. That's going to take a long time, though. For now I'm intending to get the cleaned-up map with hi-res cities in place and we can look at further improvements later. Now we can generate the tiles at least the whole thing isn't on hold any more. –Rpeh^{•T•C•E•} 16:06, 6 June 2009 (EDT)

I agree completely: I'd love to fix the water, but I can't see an easy way to do it. So I just curse Bethesda for not making it possible to simply add the water to the maps, given that the render window is already doing such a nice job of calculating where the water exists ;)

I think it is completely impossible to do it based on colour, because the colours simply have no true relationship to where water exists. Look closely at shorelines and compare the map tiles to the CS waterline: there are cases where the map tile is essentially monochromatic, but the water line cuts diagonally through the tile. One example I noticed this week is CamoranParadise [-5,2] -- it took me 5 minutes just to convince myself the map tile and CS were the same cell. I also remember much too clearly that all of the little islands in the northwest and northeast sections of Random Oblivion World 2 are impossible to map out by colour (the strange bright contours around the lava islands on Image:OB-Map-OblivionRD002.jpg were one attempt to map out the islands, but it was all done manually). Therefore, even if you overcome the problems pointed out by rpeh to create a colour-based algorithm, the result will be misshapen islands that bear no resemblance to the actual in-game islands.

One semi-alternative to rpeh's #2 is to use the CS Heightmap editing window to create a mask with all of the water in blue and the rest of the map in white. (To get a crisp line, it helps to set three contours right at the water line, e.g., at 4095, 4096, and 4097). However, I don't know how to export the resulting mask (other than screenshots), the mask is "blocky" (no diagonal lines when you zoom in, just vertical/horizontal), and it still doesn't fix the trees-above-the-water problem pointed out by rpeh. Colours might help to identify trees/rocks above water, but probably not terribly well. It's more likely that any automatically-generated overlays would best be used as a starting point for subsequent manual review/correction.

In the meantime, rpeh's new tiles are unquestionably a huge leap forward from the existing tiles, especially if hi-res cities can be superimposed -- scanning through the examples, they all look amazing (especially the yellow invisible walls on top of castle walls... and temple of the ancestor moths is so much better!) So I'm all in favour of proceeding with these tiles.

However, I do get the same hideous purple problem with back-versioned SI: there's nothing useable in those tiles right now. For some reason, the old CS isn't loading any of the SI textures. Possibly it's something as ridiculously trivial as the fact that the SI textures are in texture subdirectories? It's possible that if we could figure out how to get SI textures to load into the old CS, then we could then unleash a script on the .esm and do any necessary revamping. Otherwise, I think we're looking at either adding trees as suggested by Timenn, or exporting tiles a handful at a time and checking each tile to see whether the trees are there. --Nephele^Talk 17:05, 6 June 2009 (EDT)

(outdent) Right. Here are zoom-15 level hi-detail images of the cities. I'd be grateful if somebody else could look them over to see what I've messed up. I promise this isn't a fishing-for-compliments "Yes Rob those are great" exercise - it's just that I've been staring at these things for the best part of a day and I really can't see them any more. For instance, having already gone over all the images to make sure there wasn't anything too glaringly wrong, it's only now that I notice the huge error on the Cheydinhal map (top center). Please, if there are any areas of the maps that have big discontinuities like that one, or if there are low-res areas or anything else that might need fixing, let me know. Right now I'm very tired and want to finish my beer and go to bed.

Incidentally, is there a way in which we can test a new set of tiles? Nephele has mentioned the testmap area for map code, but does it reference the live map data or a separate set of tiles? It would be useful if there was a way of seeing the map "as live".

Also incidentally, we did a "zoom17" zoomout for the Morrowind map. I don't think it's as necessary for this one, but could easily be done if people want it. Thoughts on that? –Rpeh^{•T•C•E•} 19:39, 6 June 2009 (EDT)

A quick note about the water. When I originally output the Oblivion map tiles I tried playing with getting blue water but ran into the same issues you've found. I started playing with actually modifying the water texture/shaders to make them blue in game and seem to recall having some success but never followed up on it. I don't remember exactly what I was playing with but another possible option if someone has some time to play around with things. -- Daveh 20:19, 6 June 2009 (EDT)

From a quick scan of the maps: are the Skingrad Castle Courtyard and Anvil Castle Courtyard worlds being used? Or perhaps those areas are just blurry anyway? Other than that, I didn't notice anything -- but I'm sure our helpful readers will be very quick to point out anything we overlook :P

Yep, it's possible to create a set of testmap directories -- and one little side benefit of creating those directories is that you (rpeh) should be able to put files directly into those directories. This link takes you to the test version of the oblivion map, which is coming up completely gray because the /home/uesp/maps/testmap/zoom* directories are all empty, waiting to be filled with test tiles.

At the moment, I don't think a zoom17 is needed -- IIRC, with MW part of the issue was that MW cells are 4 times larger than OB cells. --Nephele^Talk 20:36, 6 June 2009 (EDT)

I thought I was using the Anvil Courtyard but I'd forgotten to include it. Skingrad's castle courtyard is actually an interior cell and I wasn't using it, but I am now - after a little tweaking. It doesn't fit brilliantly but it's good enough.

I've uploaded everything into the zoom directories on content1, but something's not working. Even before I did the upload I wasn't getting a grey map: I got a copy of the existing map. After the upload, I get the same thing. I'm also getting old versions if I go direct for the tiles ([1] and [2] for instance).

Anything I do at this point is more likely to break everything than fix it so I'll leave it to the experts :)

Doing the water by changing the shaders sounds interesting. The concept is completely new to me, and the Shader Editor I just downloaded crashes rather than doing anything helpful but I might play around if I have time. –Rpeh^{•T•C•E•} 08:33, 7 June 2009 (EDT)

Have you tried ctrl+F5 on the testmap to make sure your browser is using the current versions of all the javascript files? Because it's definitely working for me.

As for the individual tiles, it seems there's something strange with the zoom16 tiles you uploaded: they're the non-artifact but low-res city versions. zoom15/14/13/12 have high-res cities, but not zoom16. You can see it on the Anvil tile you pointed to: there's no cyan outline over the bridge on that tile, but the courtyard is blurry; similarly near the Imperial City (no green dots outside Chestnut Handy Stables [3], but blurry inside the walls). --Nephele^Talk 10:05, 7 June 2009 (EDT)

Okay, I win a prize for idiocy. I'd managed to upload the hi-res cities into a zoom16/zoom16 directory due to various zip-related problems. Now works fine - I hope. I still can't persuade FireFox to clear its cache, but given that it works fine in IE, Opera and Chrome (inasmuch as Chrome's buggy engine works with our map) it's obviously a problem at my end. –Rpeh^{•T•C•E•} 11:33, 7 June 2009 (EDT)

Yeah, now Firefox has gone screwy on me, too. Earlier I had the testmap and obmap up side-by-side in two separate windows, and was able to simultaneously view the two different versions of the map. Now FF is showing the old map tiles on the testmap :| One way to probably force FF to realize that the two are different is to rename all of the testmap tiles (e.g., rename tamriel-18-69-16.jpg to tamtest-18-69-16.jpg). I can quickly do a batch rename of all the tiles -- but it would make updating the tiles much more complicated. Is it worth it, or are you OK with testing in IE (I'm FF-only at the moment, but I'm not the one doing the work!) --Nephele^Talk 12:21, 7 June 2009 (EDT)

I'm happy with what we've got. Since I did the stitching together of Tamriel and City namespaces I know exactly where to spot the dodgy joins and I can see them if I look. Ditto, I know where to look for slight tree blurring. Other than that, I think this set of tiles is okay and I'm having fun smashing the server my scrolling around an unblemished map :) –Rpeh^{•T•C•E•} 12:31, 7 June 2009 (EDT)

Actually, I just found another way to force Firefox to figure out the correct filenames. I renamed the GetTileUrl function and hard-wired the correct directory name into the function. Some combination of those changes forced Firefox to recognize the correct directory names after another ctrl-F5. --Nephele^Talk 12:38, 7 June 2009 (EDT)

\=> How about VMWare, Nephele? ;)

The city worldspaces and the Tamriel world space don't always match, which makes sense since we are only given the idea by the context that they should fit together. As for the method of adding water. Method #1 may be used to save time, but not necessarily mean we have to do everything automatically. Blackwood is going to be a serious problem, and since the map won't look right with most of the major water elements added we have to agree to finish it before we start. --Timenn < talk > 12:46, 7 June 2009 (EDT)

The city/Tamriel non-matching is what took me an extra day. I have already contacted the Dark Brotherhood with regards to the designers of Anvil and Cheydinhal (the other cities had minor problems, but those two were painful). In cases where I had to make a choice, I went for the city-space. Hence, in Cheydinhal you'll see some dark spaces north of the city where the Tamriel map had rocks and bushes. There was a huge difference between the two sets of images and I had to go for one. I decided on the city-space since there was nothing external that would be of any interest.

For real geeks, you should also look at the joins between districts in the IC: the towers sometimes have two outlines. The adjacent districts (eg Elven Gardens/Market) don't properly match up, and things get worse when the Tamriel image comes into play. In all those cases, I fudged, blurred and tweaked and given that nobody has spotted it (or at least, has said anything!) so I think it'll do.

Having spent several minutes checking out the new map, I can't spot any problems caused by the new tiles. Having said that, I've spotted several problems with the existing map. Nothing serious - just a matter of tweaking. For instance, Cloud Ruler Temple's only marker is its map marker rather than the doors or the courtyard. Hermaeus Mora's Shrine is another example. I know these are map markers but this is one case where I think strict, CS-based accuracy could be sacrificed for more comprehensible information. I think we should look at taking off the map markers (already suggested on the list of tasks) or shunting them around where no sensible door marker exists. –Rpeh^{•T•C•E•} 13:41, 7 June 2009 (EDT)

OK, just to be really obnoxious :P Is it worth doing Fathis Aren's tower (aka TGTempleOfTheEmperorZero world)? I think that's the only other "world" that superimposes on the Tamriel map. --Nephele^Talk 16:15, 7 June 2009 (EDT)

I just realized one major problem with the testmap: it only works when www.uesp.net forwards your request to content1. I think that's probably the root cause of our earlier frustrations with the testmap. However, you can't directly link to content1, because the license only works with www.uesp.net. But I also can't fix the testmap on content2 right now because of some technical issues. So whether or not the testmap will work is probably hit-or-miss right now. --Nephele^Talk 21:16, 7 June 2009 (EDT)

Daveh to the rescue! The testmap should now work no matter which server ends up doing the behind-the-scenes work. --Nephele^Talk 22:34, 7 June 2009 (EDT)

Argh! I'd forgotten about Fathis Aren! Okay, I've fixed the tile and my code is running on the tiles now. I'll do the upload this evening. I'd discovered the content1 problem earlier. Good to know that's fixed too. –Rpeh^{•T•C•E•} 00:57, 8 June 2009 (EDT)

Okay, That is now done. I'm going to spend a while scrolling around to make sure there's nothing else to be fixed, but I think we're pretty much done. –Rpeh^{•T•C•E•} 12:17, 8 June 2009 (EDT)