For most of the last week, Linden Lab's public data/statistical feeds have been showing signs of being a bit ill. Starting on Wednesday evening, concurrency information started to fall ten or fifteen minutes behind actual data - an unusual occurrence. It's only in the last 24 hours that the system has started to catch up again.

At the same time, the figure for active users ceased updating entirely, with the most recent update to that figure being Wednesday night (24 October), at midnight SLT (US Pacific). Perhaps these subsystems are considered relatively minor, but Linden Lab have said themselves that they consider the accurate and ongoing publication of this data critical to their program of transparency.

The public JIRA has been spitting unusual (and some incomprehensible) Java error messages at people since this-morning. Sometimes it works and sometimes it doesn't. Rob Lanphier, Linden Lab's open source liaison says that Linden Lab have been working on the problem, but do not know what the cause is.

It seems to be happening less, at present, but there's been no further word that the issue has been isolated and repaired.

Linden Lab says that they believe some users have been billed twice for tier (land use fees). They are encouraging anyone affected by the double-billing (it appears you will have to go and check - or maybe check with your card company) to file a ticket through the Support Portal.

If the stories we hear have any grain of truth to them, then double-billing land-use fees to some users is an ongoing problem with Linden Lab's billing system - as well as users who are unable to process payments to Linden Lab at all.

At 2:50AM SLT (US Pacific) Second Life login servers apparently failed. It was brief, and thankfully just after minimum concurrency. At the time the problem commenced 28,788 users were online.

Linden Lab responded with unusual rapidity, and the problem was resolved by 3:10AM, a scant 20 minutes.

[Update: 3:35AM - Linden Lab posted a notice about the problem after the fact, and apologizing for the problems]

• Teleports are optional,
• Search won't,
• Friends lists are optional,
• Money is Loading....,
• Changing clothes is a luxury,
• Europeans are reporting packet loss at high levels too

There's a new release candidate available on the optional downloads page. It's one with known problems and it isn't actually a candidate for release. The previous release candidate turned out to have some pretty severe problems, and it seems some people just won't quit using it anyway. So this release is a roundup of fixes for the folks out there who are eating bees.

So, we get the impression that, esssentially, bee-eaters this release is for you. Not eating bees? Give it a miss, and wait for the next release candidate, which might actually be a release candidate.

At 8:30AM SLT (US Pacific) all Second Life login services ceased responding.

At the time of the fault, 41,337 users were online. With users unable to log on, concurrency numbers have been falling rapidly since that time.

[UPDATE: 8:45 AM - Linden Lab acknowledges the problem, and promises further updates]

[Update: 9:05 AM - Logins are back up]

[Update: 9:15 AM - Logins remain up. Search has failed]

[Update: 10:00 AM - Search started responding to queries again about 5 minutes ago, and is hanging in there.]

We're getting some signs that the Second Life grid may be under some unusual strain. Concurrency, while fairly flat (rather than the sharp decline that would be normal at this time of day) is reporting late, with updates coming approximately ten to fifteen minutes later than they should.

Additionally, Teeple Linden reports that the Friends Online page is being taken off the air for a few hours. No cause given.

An extremely terse note from Linden Lab implies but does not state that there is a problem with stipend payments this week. The exact wording is "We are presently working on a solution to ensure all residents receive their stipend as soon as possible."

The phrasing suggests that it's broken rather than slow, being that it is the sort of phrasing that's used in corporate statements when something important is on fire, stolen, or unexpectedly vanished into a parallel universe. Hopefully, it's just an accident of sentence assembly and all we are looking at this week is general slowness.

[Update: 10PM SLT (US Pacific) - Linden Lab reports that stipend payments are in progress and expected to finish sometime before dawn]

Many of you who have used the latest release candidate, 1.18.4(0) have probably noticed that the tooltip popup on the map that shows you the number of people in a sim/region.. well, doesn't actually show you the number. Instead it relentlessly reports zero for every sim except the one you are in, where it reports one (that's you).

How did this happen? Well, when you modify or refactor code, there's two important requirements. First, you have to know how to code. Second, you have to understand what is going on. Lacking the first is less dangerous. You can go safely stick pencils up your nose. Lacking the second, however, means you risk sticking pencils up everyone else's nose. This bug was relatively harmless, if irritating.

Jacek Antonelli, who heroically dived in and fixed this bug up for me said that one of the causes was, "editing the code without understanding everything it does."

"It looks like they refactored the rendering code into its own function (good for them), but missed this crazy side-effect that was going on."

That side effect was, basically, that as it looped through each agent to draw them, it also incremented a counter, amassing a running total of agents in the sim. At the end of processing that sim, the resulting was stored.

In 1.18.4(0), the dot-rendering process in the loop was pulled into it's own function, and the running total was simply left at zero. Worse, when that zero is stored - it actually overwrites an existing correct number that is calculated elsewhere, presumably after this piece of code was originally written. So doing the counting and storing the result is just plain unnecessary.

Someone may have half figured that out that that counting everything up again was a waste of time and precious CPU cycles - the counting was removed - but apparently forgot to also remove the code that stored the running total, allowing it to essentially overwrite good data with bad.

All of this seems to result from a lack of knowledge of what each section of the code is responsible for, and how it (and it's intentional side-effects) mesh with other sections.

Some number of errors are, of course, inevitable. However without a map, how can you not get lost?

Today's release candidate doesn't seem to be winning a lot of hearts and minds out there today. The public JIRA issue tracker is accumulating problem reports, many of which seem - just darn peculiar.

Aside from crashes, virtually all of them seem to be related to the user-interface, and new ways in which it behaves (or misbehaves). On the Second Life development mailing list of late, there has increasing discussion of scrapping the existing user-interface code (which is tangled up in, well, everything) and changing to something more modular and discrete.

At present, as the complaints go, the user-interface code and some of the back-end code have become such an organic tangle that it is difficult to change one without affecting the other, and that some theoretically simple user-facing changes have become monstrously involved.

That appears to be the sort of thing we are seeing here. Changes that should be quite straightforward causing an extensive tangle of dependencies and concomitant unexpected issues.

Linden Lab's third-party enabled Support Portal failed, by all reports at approximately 8:35PM SLT (US Pacific time). It is not allowing users to authenticate to lodge or submit support tickets. Only the Guest Login option seems to be working and that has very limited access.

No response from Linden Lab acknowledging the problem at this time.

[Update: 9:05PM - Linden Lab acknowledges the problem]

[Update: 9:45PM - The Support Portal appears to be working again]

Linden Lab's public statistical data feeds all went down at 3:40PM SLT (US Pacific) yesterday. The sudden cessation of data updates led to some unreliable figures in our reporting, so we marked them as such at the time.

The problem was today traced to the faulty deployment of a backend script, which was fixed and redeployed at approximately 11:40AM SLT today. As such, today's statistics are also going to be a little out of whack, but the numbers should all be on the mark again for tomorrow.

Every MMO/NGO/VW has bugs. Every patch, every update for every one of them brings a slew of fresh bugs, glitches, imbalances and lost property. Dealing with those bugs, glitches and whatnot can be a pain in the ass quite frankly, but in Second Life, we're not just dealing with peoples' fun. We're also dealing with their revenue.

While Ginsu Linden made quite an effort to make everyone understand that Linden Lab understands the position that it is in, relative to the users, it's obvious enough from reading the comments that many of the users themselves either do not understand Linden Lab's position, or do not think Linden Lab understands it. Pick one.

Part of that position involves a bug. Technically, you'd call it an exploit. That being, a bug which (if you know how to use it) allows you to profit in a way that is unintentional or accomplish some task that is not normally permitted. We're not talking about cheating your cash balance, or purchasing land that isn't for sale, or making prims that are larger than the system nominally permits.

We're talking about the issue with llSetLinkPrimitiveParams(). A sitting avatar becomes a part of a set of linked prims. Until this function came along the fact that a sitting avatar was just another (temporary) member of the linkset wasn't really all that relevant, as there wasn't really anything you could do to the sitting avatar as a member of the linkset that wasn't already able to be done.

With llSetLinkPrimitiveParams() however, the avatar appeared (to this function) to be an additional prim, and that meant it could be manipulated. Those manipulations were pretty much limited to two things: Repositioning the avatar relative to the object, and certain other manipulations that would crash the simulator.

The repositioning tasks were all able to be accomplished by other means, though you needed extra prims to do the job. Being able to handle it directly was a time-saver, although apparently an unintentional one. Yes, it was a bug - or an exploit if you prefer. It was also a superior solution to a common scripting task. It made all sorts of furniture and multi-pose systems more efficient - at the risk of intentionally or unintentionally causing simulators to crash.

The community took to it like wildfire, as it was discovered, and almost all the new creations used the new function to perform tasks that had previously been performed in less efficient ways. Many creators retrofit their old gadgets to work the new way too.

And then, at the beginning of the month, Linden Lab found the simulator crash problem and fixed it. They fixed it by making sure the function did what it was supposed to do - that is, not allow the manipulation of avatars.

Then the screaming began. The use of the function in this way was nearly universal by this time, and the people at Linden Lab who worked on this code were - essentially - completely unaware of that fact. To them, it was a bug - an exploit, even. The code was permitting something that they had not intended to permit, and worse, people were using it to crash simulators.

So they fixed it in the simplest possible way. Hide the avatars from the function. Done.

Did I mention the screaming?

Intentional or not, the use of this exploit was widespread. It was everywhere. It was a part of people's play. A part of their businesses. The fix went in, and everything, everywhere in Second Life that used it stopped working, all at once.

Hastily (it took several days), Linden Lab responded to the huge quantity of broken product and refixed the function. This isn't actually the first time something like this has happened, where an unintentional glitch that has been fixed has caused widespread problems, breakage, or breakdown in user experience. It's probably not going to be the last, either. Sometimes those fixes mess up resident work, and sometimes they mess up the work of other staff at Linden Lab itself, even.

Whether it is intentional, or unintentional, anything that is perceived as a feature quickly becomes widespread. Look at megaprims. They're everywhere these days. Take a tour of the Linden inworld offices and you'll see quite a few megaprims in the construction of some of them.

The problem is one of communication, certainly. In each case, where these fixes went in, the people doing the fixing were simply unaware of the impact of the changes they were making. While I know it is not their policy to discuss exploits, perhaps Linden Lab really needs to bring some of their own bugs and bugfixes to their bug triage sessions for other eyes to look at.

According to a friend in Twitter, SL is experiencing problems with search and tp again. I don't have time to completely check this out, I have to go and do some things in RL, but a very quick check for Eloise Pasteur stalled at "Searching..." so be careful, it may be a bumpy ride.