Shivering:Reference Bug

Overview

This bug, first recognized around the beginning of April 2007 is now fixed by an official patch (1.2.0416) released on April 30 2007. The patch is included in copies of Shivering Isles for PS3, so the Reference Bug does not occur.

The bug (a.k.a. FormID Bug) results from the combination of two factors:

1. Certain internal resources (FormIDs) used up in normal gameplay were not being recycled as they should be
2. An error in the scripting for some NPCs in Shivering Isles caused those resources to be used up much more rapidly than expected

Problem number one is repaired by the official patch. The patch renders fixing problem number two unnecessary; however, a user-produced patch is available for the PC that fixes this problem as well.

Notes:

• The primary symptom of the bug is the disappearance of dropped items when reloading a save. If this has occurred, those items are lost forever.
• Speed with which the bug hits is directly proportional to the average frame rate. Higher FPS means the bug hits sooner. As a result, the bug hits much sooner for high performance PCs (as early as 50 hours) than it does for Xbox users (typically 150 hours or more).
• The bug can also affect PC users with mods suffering from bugs similar to the SI NPCs. In fact the complaint about the bug (recognized in retrospect) appeared several months before Shivering Isles was released.

Advice

• Get and install the patch immediately.
• Fix is available on Xbox Live.

Details

Symptoms

The most frequently reported symptom of this bug is that items disappear from the game. Newly created items (newly enchanted items, quest rewards, new spells, etc) and items dropped on the ground will at first appear to behave as usual. But after restarting Oblivion and loading your save game, the items will have vanished. Once the bug starts, all new items created from that point on in the game will disappear once the game is reloaded.

Much more severe problems are also possible as the bug progresses. The bug is overwriting memory used to store a range of game information. What exactly gets overwritten and what happens as a result will vary from player to player, and could potentially not become apparent until later. In the worst cases, save games have been corrupted to the point that they could no longer be loaded. Also, some players have had all of their character's information be completely erased, so that the character reverts to being a level 1 imperial male named "Bendu Olo" with almost no possessions or skills.

Cause

The basis of the bug is that an internal "space" of identification numbers (also called FormIDs) for objects created in the game (e.g., a dropped arrow, a new creature, etc.) becomes exhausted. Master and Plugin files (*.esm and *.esp file extensions) each get their own hexadecimal number assigned during loading (00 to FE), and all components of those masters and plugins have that number as a prefix to longer hex numbers (such as the FormID codes used for items with the Additem console cheats and such; the FormID series 00 is assigned to the Oblivion.esm and all objects within start with the formID "00"). The last series, FF, is used for objects created from in-game additions (creature spawn points, dropped objects, etc). Once these numbers are exhausted, newly created objects are stored in memory reserved for other game information. The visible result is that those new objects do not get copied into your save file the next time you save (and therefore vanish when you load the save game). The less visible result is that other game information is corrupted when the object information is stored in the wrong memory area, leading to any number of unexpected consequences.

The fundamental problem, namely only having a limited number of FormIDs for new objects, is a bug that has been present in Oblivion since it was first released. However, before the release of Shivering Isles it would have taken more than 1000 hours of playing with a single character to run out of FormIDs, and therefore was unlikely to ever affect most players.

With Shivering Isles, six of the new NPCs have scripts that use up FormIDs at an extremely fast rate. Several FormID are used up per frame of gameplay (hence higher FPS systems get hit sooner). The unofficial FormID Fixer Patch works by altering the scripts for these NPCs so that they no longer use up ID numbers so quickly.

In retrospect, it is recognized that some user mods have introduced similar scripts, that use up ID numbers at a fast rate (although not as rapid as the Shivering Isles scripts). Therefore, problems reported in the past for certain user mods have the same cause.

Although these scripts have aggravated the situation, the real problem is with how the game stores identification numbers. The game is supposed to actually recycle ID numbers, in which case the available storage is more than sufficient for any length of gameplay, no matter how quickly various scripts choose to use ID numbers. Bethesda's Official Patch fixes the problem by making the recycling system work as originally intended.

Xbox

The patch to resolve this issue has been released on Xbox Live. The Xbox One and Xbox Series X/S releases of the game, as well as Xbox Cloud Gaming, all have the newest patch preinstalled which fixes this issue.

PC

An official engine patch is available and should be installed immediately. An unofficial Shivering Isles patch is also available. The unofficial SI patch is not required if the official engine patch installed, but may have some performance benefits.

Advice (PC)

If you have Shivering Isles installed on your PC, you should get and install the Official Patch. NOTE: The official patch from Bethesda is not compatible with the version of SI purchased from Direct2Drive.com

There is also an unofficial patch mod directed at the fixing the problematic NPCs in Shivering Isles. Since this will greatly delay the onset of the formid bug, it means that the unofficial patch can to some degree be used in place of the official patch. However, since the official patch solves the problem permanently, and solves it for non-SI mods as well as for SI mods, it is again strongly recommended that players get and install the official patch. Note: It is possible to use both official engine patch and the unofficial patch mod, and there may be a performance benefit to doing so. (This has not been verified however.)

Note that if you have already suffered consequences of the bug, neither official nor unofficial patches will repair those problems. Lost items will still be lost, and it's possible that other odd effects may appear. Safest course would be to roll back to save game from before when the bug hit.

Beta Patch

Bethesda previously released two beta patches, available for download. The beta patches were only available for PC (beta patches are not released on Xbox live). As explained on the Patch page, installation of this patch could require first uninstalling and reinstalling all of Oblivion.

Tests of the patch indicated that it worked in much the same way as the current Official Patch, that is to say it fixed the root cause of the bug by modifying the game's executable. The identifiers used to keep track of temporary objects are now continuously recycled. Instead of running out of space if 17,000,000 (roughly rounded from 16,777,216) objects have ever been created, the game will now only run out of space if the game has that number of temporary objects all currently in existence, which should never happen. This means that:

• There is no longer an artificial limit on the total length of time you can spend playing the game (at least not because of this issue).
• Even if you have played 150+ hours of Shivering Isles, once you install the patch you can continue to play without worrying that you might run out of space.
• If your game had already hit the bug, this patch will essentially fix all of your save games without needing to use any repair tools. You will no longer have problems with items disappearing. However, anything that was previously lost is gone, and cannot be recovered.
• Other, similar problems reported to occur when using some unofficial mods are also fixed.

The script that was aggravating this root problem has not been modified by the patch. However, with the patch that script will no longer cause any problems.

Update: If you had previously installed the first beta patch it is strongly recommended that you update to the second beta patch. The Version 1.2.0416 patch has remedied several issues reported with the first beta patch, such as crashes, a problem with enchanting items, and (as of yet) the character reset problem.

With the first beta patch (but not the second), there were several reports of entire save games' characters being reset after the beta patch was installed. All quests, map locations, and items were wiped, all stats reset, and even the name changed to the default "Bendu Olo". There have also been numerous reports of crashing and other problems occurring only after this patch was installed. If you wish to download this beta patch now, it is recommended that you create a backup of your saves. This is a beta patch only, which means it has not been fully tested and thus overlooked errors may occur with your game with this patch installed. This is obviously now irrelevant, as an Official Patch has now been released.

FormID Fixer Patch

An unofficial patch mod by Dateranoth is available for the PC that fixes the formid usage problems of the problematic NPCs in Shivering Isles. While the official patch renders this mod unnecessary, players concerned about formid usage may wish to use it anyway (it should not hurt the game, and possibly might help performance). It can be safely removed at any time.

The patch mod can be found at:

• The Elder Scrolls.info
• World of ElderScrolls.de

To use the patch: download it, and copy it to your Oblivion\Data directory. Then start the Oblivion Launcher, click on Data Files and enabled the new mod (i.e., put a check next to its name).

The patch may not immediately correct the problem if you are in the general area of New Sheoth. Leave the Shivering Isles as soon as possible to reset the bugged NPCs if the RefIDs of dropped objects are still increasing at a rate of more than 10 per minute.

The changes in this standalone fix are also present in the Unofficial Shivering Isles Patch mod, which also fixes numerous other bugs noted on the UESP. If you are using the USIP you don't need the standalone fix.

Repair

For users who are already suffering from the bug, or for whom the bug impact is imminent (formid starts with FFD or FFE), installation of the official patch fixes the problem (but does not recover lost items).

Previously, two user tools (Wrye Bash and zBobG's ID Bug Fix Tool) provided unofficial repair of the problem. These should no longer be used since the official patch now does a better job.

Third Party Mods

Third party mods can also cause the reference bug. Modders who create complex mods should begin testing to ensure that their mods don't create more references than is necessary. With the beta patch, this is no longer an issue, because the root problem that had been limiting the available references has been fixed.

Diagnosis

Terminal Diagnosis

If you drop an object from inventory on the ground, save, quit, reload and it's not there, that indicates that the rollover has already occurred. This is the way the problem first showed up on the PC, and will also occur on Xbox 360.

Early Diagnosis

If you have SI, your available playing time, until the bug appears and starts to corrupt your game, is limited. On the PC, you can determine how close you are to having the bug appear.

You can see how far along your are by dropping an arrow or ingredient from your inventory, then entering the console and clicking on the object. The object's FormID will display at the top of the screen, like so: "Iron Arrow" FF6cb394. Be sure that the name appears! If it doesn't you've probably just clicked on the ground or some other object -- try clicking in a different place, and you'll get worried for the wrong reason.

The FormID number should begin with FF -- that's the modindex, it means the object belongs to the savegame. The following 6 (hex) digits are the objectindex. Ordinarily, this number increases whenever Oblivion needs to create a new "in world" object -- e.g., because you shoot an arrow, drop some item, or a creature is spawned. However, due to scripts causing an NPC pathing loop, 6 of these are used per frame. When the number hits FFFFFF, it tries to roll over, but instead the whole number (modindex+objectindex) rolls over, resulting in newly created objects with modindex "00" (instead of FF).

I.e., if you use the console to check to see if you have the bug and the modindex is already "00" your game has already been affected and continued play runs the risk of lost items or worse. If the number is very high, e.g. with object index > C00000, then you are yet to be affected, but your savegame has limited playlife left in it. How much life depends on how much higher the number is, performance of your system and whether you have any other mods with similar problems. Repair of affected and soon to be affected saves is possible using the instructions listed above.

With the beta patch, FormIDs no longer work in this way. The game now searches starting at object index 000000 and uses the first available index. Whenever a formid is no longer in use, it is returned to the pool of available indices. Therefore, a high object index no longer indicates that you limited playtime left, as long as you install the beta patch as soon as possible.

Wrye Bash (v 0.70 or higher) users can also check the formid by right clicking on the save game in the Save tab and selecting the "Statistics" command. The next formid will display near the top of the report under "FBomb Counter".

Wrye Bash can also be used to check Xbox 360 savegames if they are first transferred to the PC.

Remaining Playtime

The following table will let you estimate how many hours of playtime you have before the bug will impact your savegame. These are the number of hours you can spend playing with a single character, after Shivering Isles has been installed and without the beta patch.

These estimates are made assuming that the bug generates 6 references per frame, and that you spend only about one third of your playing time in normal play mode. Tests have shown that whenever the game is in normal play mode (i.e., not in a menu or otherwise paused), the game always generates 6 references per frame. However, most players do not spend the majority of their time in normal play mode, and also frame rates will decrease when a lot of action (i.e., fighting) is occurring. From the playtimes reported by multiple players, it seems that most players spend less than half of their time (probably only one-quarter to one-third of their time) in normal play mode. If you spend a lot of time in menus (reading books, creating spells, checking the map, etc) it will take longer than estimated, because references are not generated while you are in menu mode.

Formid from Drop Test	Playing at 30 FPS (Xbox 360)	Playing at 40 FPS	Playing at 50 FPS	Playing at 60 FPS
FF0?????	145.6	109.2	87.4	72.8
FF1?????	135.9	101.9	81.6	68.0
FF2?????	126.2	94.7	75.7	63.1
FF3?????	116.5	87.4	69.9	58.3
FF4?????	106.8	80.1	64.1	53.4
FF5?????	97.1	72.8	58.3	48.5
FF6?????	87.4	65.5	52.4	43.7
FF7?????	77.7	58.3	46.6	38.8
FF8?????	68.0	51.0	40.8	34.0
FF9?????	58.3	43.7	35.0	29.1
FFA?????	48.5	36.4	29.1	24.3
FFB?????	38.8	29.1	23.3	19.4
FFC?????	29.1	21.8	17.5	14.6
FFD?????	19.4	14.6	11.7	9.7
FFE?????	9.7	7.3	5.8	4.9

Again, current evidence is that the problem can be repaired (on the PC) by using Wrye Bash. However, it is better to apply the patch early and avoid the problem altogether. If Bash is used, it should be used while the savegame is in the FFC to FFE range. (Not earlier, because it's not yet known for sure that Bash's approach is safe, and not later because it's better to make the fix before the counter rolls over and the bug takes effect so that there is no risk of important NPCs or Items disappearing.)