Not meant to be snarky- why is the other board so slow?

[gurujulp]

I really want a factual answer, but am totally willing to be sent to TD or whatever.

Why is the SDMB the only forum I have ever found that responds so slowly?

What is it with the server/software? Or whatever it is I just don't get. The search I get. Lots of data. But what is up with the overall 'nothing happens when I click this button' -ness?

[Dragon]

I go to a board that has more members, more posts per hour and has avatars, animated smileys and you can post large images and there are many, many of those images and they get C&P'ed a lot too and it is fast as blazes..... Individual owned and operated....... I don't think he is rich either......

They are not all created equal.......

[Harmonious Discord]

The only people that could answer that are the operators of that board. They are however incapable of figuring that out, and are just turning on and off software features hoping for a resolution to just happen by accident.

[What Exit?]

It is a large board with a poorly tuned database/database access is the most likely answer. There are other factors but that seems the likeliest.

As a comparison I once had a process that took 6 hours to run when I started the job. We got much faster hardware and it cut the job to 3 hours. I had a chance to completely rewrite the application after that and studied where the bottlenecks were and made a few calls to IBM and the new code ran the job in 20 minutes.

In another case on an earlier job, I had a canned part description search that was useless as it took 3-5 minutes to run. My job was to build a new tool that could do it in seconds. I designed a nightly work file that broke out and keyed the description file and part number and a few other pieces of data and ended up with a routine that took less than a second but had the slight drawback of not reflecting changes made that day. The nightly routine ran in about 20 minutes and caused no problems. Everyone was quite happy with it.

So while I have no clue what is wrong with the database and access to it, something is wrong. vB can handle the load that exists over there.

[Cluricaun]

Poorly tuned, or poorly maintained. I'll bet that they could hot rod the Dope pretty quickly if they could offload a ton of the older stuff to another server, which will then be plagued with the same issues but free up the main board to work like, yanno, this one.

[silenus]

You thought we were kidding about the hampsters.

[What Exit?]

Poorly tuned, or poorly maintained. I'll bet that they could hot rod the Dope pretty quickly if they could offload a ton of the older stuff to another server, which will then be plagued with the same issues but free up the main board to work like, yanno, this one.

I don't know MySQL at all, but I am guessing someone skilled with it could probably get the entire database working. My database skills are not in the SQL world but I know several MS Sequel and Oracle people that tell me the Dope's load is nothing for their software. I have seen posters say the MySQL should be up to the job.

There are better search solutions out there for vB but it does apparently require a separate server and the parent company is effectively out of money and I believe Jerry is seriously overloaded for time and also I do not think he is a MySQL guy.

[McNutty]

I don't know MySQL at all, but I am guessing someone skilled with it could probably get the entire database working.

I agree. It's very easy to have a mis-tuned db server hit a performance wall when a dataset gets large. At my job, we recently moved some tables to a new db machine that was way more powerful than the old one and due to a buffer size setting that was too small, a full db import which should have taken 18 minutes took over 6 hours. We scratched our heads over that one for a while.

[gotpasswords]

It's very easy to have a mis-tuned db server hit a performance wall when a dataset gets large.

Maybe they need to ditch MySQL and change over to Teradata. :ugeek:

[McNutty]

Maybe they need to ditch MySQL and change over to Teradata. :ugeek:

Or Oracle, or probably any number of heavy-hitters. I'm not a DB nerd, so I don't really know. But I have heard of much larger boards than the SDMB running on MySQL, so that's probably not necessary.

[gurujulp]

Thanks to all of you! Many good (uh, I think) technical answers, 'cept for the hamsters. The obvious answer there is MORE hamsters. Duh.

Anybody know how to mention any or all of these suggestions to TPTB at sdmb without an issue? I have no rancor towards them (while I am a bit annoyed that things have scared me enough that I have to ask this question) and would like to bring it to their attention.

[featherlou]

No snark here either, but why would you bother? People have been having that discussion for years and getting nowhere.

[gurujulp]

No snark here either, but why would you bother? People have been having that discussion for years and getting nowhere.

I guess I just haven't been disillusioned yet.

Not stating that I won't be...

I am here because just about everyone I really dig came here, but I plan to be there for a bit. I tend to prefer posting here, however.

[McNutty]

Anybody know how to mention any or all of these suggestions to TPTB at sdmb without an issue?

What are you going to tell them, though? "Hire some bad-asses"? It seems pretty obvious that they need three things: 1) a staff that really knows what they're doing as far as DB/vBulletin scaling 2) enough of that staff's time dedicated to it, and 3) sufficient resources (hardware, a good database) for them to do a good job.

At least one of these things is missing. Notice I'm absolutely not saying they don't know what they're doing. They could be the biggest ass-kickers in the world and if they don't have numbers 2 and 3, they're screwed. Or they could have the fire-breathinest server that is run by monkeys. The key is that all three of these are things the people actually running the board (e.g. Jerry) can't just come up with if they don't have them. It has to come from up higher. I have a lot of sympathy for Jerry.*

ETA: * and I'm not just saying that because I could be the next Jerry if this board's performance goes in the crapper.

[Batman]

Why do they keep the older stuff? Would anybody miss the posts from 2000?

[Harmonious Discord]

Why do they keep the older stuff? Would anybody miss the posts from 2000?

Yes. There is some really good material in there. Occasionally it's fun to read the old stuff. With the search how it is currently, it's almost impossible to find.

[CRSP]

Is mySQL really all that puny? Isn't it widely used all over the Internet? I can't see how the SDMB's performance can be blamed on mySQL.

[hatesfreedom]

mysql is fine but apparantly this thread seems to suggest that they're keeping old posts in the same database and they allow search and they need a better storage array thats just balls fast. none of it hard to do but none of it cheap at all.

[Feirefiz]

Why do they keep the older stuff? Would anybody miss the posts from 2000?

Yes. There is some really good material in there. Occasionally it's fun to read the old stuff. With the search how it is currently, it's almost impossible to find.

There is a gentler alternative to deleting the old posts. They could just move everything older than a few years into a special archive. One possibility would be a second basically normal board -- just read-only and with a separate database. Alternatively you could extract the data and store it in some other format, perhaps something superficially resembling the vbulletin archive function (but unlike that one not generated from the same database.) Probably there is some existing software for such a task, but even written from scratch it wouldn't be that hard as long as you have access to the original database.

[Tuckerfan]

Last I can recall reading, the server they were using was seriously underpowered. Like Pentium III with 256 megs of RAM, underpowered.

[gurujulp]

OOOOooooo

I am waiting five to THIRTY seconds to get updates here on domebo tonight.

Don't make me go all ****SPRrinGS1!1!!! on your, Oh, *** AsS!

ahem.

Cough, and such.

meant to be snarky

[McNutty]

Really? I'm browsing right now and haven't noticed a problem. Then again, I've been drinking, so take that for what it's worth. I do want to hear about long delays in page loads.

[gurujulp]

Anybody know how to mention any or all of these suggestions to TPTB at sdmb without an issue?

What are you going to tell them, though? "Hire some bad-asses"? It seems pretty obvious that they need three things: 1) a staff that really knows what they're doing as far as DB/vBulletin scaling 2) enough of that staff's time dedicated to it, and 3) sufficient resources (hardware, a good database) for them to do a good job.

At least one of these things is missing. Notice I'm absolutely not saying they don't know what they're doing. They could be the biggest ass-kickers in the world and if they don't have numbers 2 and 3, they're screwed. Or they could have the fire-breathinest server that is run by monkeys. The key is that all three of these are things the people actually running the board (e.g. Jerry) can't just come up with if they don't have them. It has to come from up higher. I have a lot of sympathy for Jerry.*

ETA: * and I'm not just saying that because I could be the next Jerry if this board's performance goes in the crapper.

I wanna tell them exactly that, (if it is accurate), as I have no pony in the race. I just don't want it discounted because A- I am just an asshole or B- I have a pony in the race...

If they will read it and take it for sooth, I will post, and damn the mods or bans...

[Tuckerfan]

They won't read it, and they won't pay attention. We've been there, done that, got the T-shirt, and washed it till its faded.

[Clayton_e]

I recall a thread about the search of the other boards stated that "oops" sometimes things get lost and "oops" things get mislabeled.

I never understood this in the age of computers.. But I'm no expert.

[TheFlame]

[modhat:2nl76ha4]Moved from QA to the Thunderdome.[/modhat:2nl76ha4]

There wasn't anything wrong with starting the thread here, but IMO the original question has been satisfactorialy answered. Ongoing discussion of the failings of the SDMB's admins would be better suited to the TD.

[leander]

Who the hell are we kidding? Snark or no snark, Jerry is just incredibly incompetent. As bad as Ed is at running a business (and he's really, really bad), Jerry is equally bad at running a message board. There are so many more qualified engineers hanging around (including our own QED) to run the boards, and yet TPTB refuse, year in and year out, to allow any input from the teemings.

I run a small company of 25 which has 5 engineers. Any one of them could run the Dope properly, and at least 2 can barely speak English.

Poor management at the top, and poor engineering in the middle. That's why the board is slow.

[cheesesteak]

I go to a board that has more members, more posts per hour and has avatars,

fast as blazes.....

Individual owned and operated.......

That last line is the key. Owned and operated by someone who wants to own and operate a message board. SDMB is owned and operated by folks who want to run a newspaper / newspaper column.

The message board is the unwanted dog that showed up at their door. They give the poor thing just enough food and water for it to keep going, but they don't play with it, don't bring it to the vet, and it better damn well not crap on the floor, or it's history.

[PonderThis]

I think it is unlikely that Jerry is incompetent as such. It's just that he has made it abundantly clear over the years that he has absolutely no interest in the SDMB. None, zero, zilch. He only shows up when he absolutely has to, does the absolute minimum to get the job done, and disappears again. I suspect this is because he already has a full time job at the Reader, probably as their in-house network and computer repair tech. That's what he was hired to do, that's what he gets evaluated on for raises. Then sometime in 1999 Ed comes up to him and says, "oh btw can you also look after this web site and message board? Thanks, bye." So Jerry does what he has to do, but you can tell from how little he interacts with the SDMB and its membership that he really wishes he didn't have anything to do with it.

As far as Jerry is concerned, that's OK. I have no problem with Jerry, it's that Ed and company seem to have this harebrained idea that Jerry can spend a few minutes here and a few minutes there and have everything work. And as far as they're concerned everything is working, but that's only because they really don't understand what they have and why they should put any more effort into it.

It's like Ed has this 1950's model sedan with no seat belts and monstrously terrible gas mileage and leaks oil like nobody's business and on top of that he's not the one using it, so he doesn't think about it much and so he never takes it in for maintenance and as a result it frequently breaks down so Ed takes it in to the cheapest mechanic in town (who incidentally knows nothing about Ed's particular make and model) and tells him to just fix whatever is has stopped the car from moving and don't worry about anything else.

The problem is not the mechanic, it's the owner.

[beebs]

Jerry is equally bad at running a message board.

What PonderThis said.

Jerry's job isn't to run a message board.

[Laughing Lagomorph]

No snark here either, but why would you bother? People have been having that discussion for years and getting nowhere.

I guess I just haven't been disillusioned yet.

Not stating that I won't be...

....

It's really better just to give in to the disillusionment with the way The Other Place is run. Much less painful in the long run.

[featherlou]

<snip>
I run a small company of 25 which has 5 engineers. Any one of them could run the Dope properly, and at least 2 can barely speak English.

Poor management at the top, and poor engineering in the middle. That's why the board is slow.

I don't know nuthin' bout no message boards and servers, but I've worked as an accounting clerk at many large companies who have all of their employees working on programs that are run on servers, often in Canada and the US and sometimes worldwide, and I don't really see a reason why a simple little message board in one place can't run better than a huge multinational company's Oracle program that's being used around the world. I mean, other than what you've said. It can be done; it just isn't.

[ShelliBean]

I usually have to wait maybe 2 seconds for a page to load when I am over there. Am I just lucky? It's not like I have anything better than a standard wireless connection on my laptop.

I've never had the problems you guys are talking about (unless it was something where everyone said "we're working on something/fixing the database/whatever") in the 8 or so years that I've been there.

[Laughing Lagomorph]

I usually have to wait maybe 2 seconds for a page to load when I am over there. Am I just lucky? It's not like I have anything better than a standard wireless connection on my laptop.

I've never had the problems you guys are talking about (unless it was something where everyone said "we're working on something/fixing the database/whatever") in the 8 or so years that I've been there.

I also never/very rarely have the page loading or timeout problems over there that everyone is constantly complaining about. My dissatisfaction though is from their apparent indifference to what seems to be a very widespread problem, and also the search issues.

[Dragon]

11 to noon central time most days, it gets slow for me. Lunch time check-in?

I don't use the search too much I guess. I just go to the forum and look for thread I might be interested in. I don't hunt up old posts nor use the "new Posts" feature, as I have no interest in in any forum but 4... So I don't really have a problem with the speed.

The problem is I have been there so long that the I want to remove from the gene pool the constant wanting to know why the world is round from the new youth of America who have not learned anything in school, from their parents and never seem to have ever gone out of their house without a cocoon around them... ::: sheesh ::::

[McNutty]

I'm really only referring to the search function sucking. It's gotten faster than it used to be, but only because they've restricted its usage so much, and it still gives the old "database error" sometimes. I've caught myself looking at the results of a search here and thinking "damn it, I should have used a different search term ... hey wait, I'm allowed to search again!" It's like suddenly realizing things don't have to suck.

There were times long ago where pages took 40, 50, 60 seconds to load where my usual browsing method was to go to the forum view, middle-click all the threads I was interested in (opening each in a new tab), then go do something else for a few minutes while those attempted to load. When I came back, I'd hit reload on all the ones that failed and read the ones that succeeded (repeat until all tabs are closed). There was a similarly complicated routine for attempting to post that involved copying the text to a notepad in case the submit failed, and if it did time out, open the thread verify that your post had not actually gone through (this, too, requiring the patience+reload routine) before reattempting the post. But those performance problems seem to have largely been addressed. I see that type of slowdown maybe once every few months now, and generally only for a minute before everything's back to normal.

[ShelliBean]

Ah. I almost never use search so that explains some of it.

[bump]

Not that I know that much about the other board's architecture, but there are only 10 million or so posts. Even with full text searching, it shouldn't be as sluggish as it is. Searching on metadata like author, dates, etc... should be all sorts of zippy - 10 million isn't very much really, unless you're running it on some absurdly slow computer, like that P3 w/256 megs of RAM someone mentioned.

I think it's a combination of incompetence and disinterest, in that I think that the TPTB over there don't care, and the people who do over there are out of their depth.

[sublight]

I've caught myself looking at the results of a search here and thinking "damn it, I should have used a different search term ... hey wait, I'm allowed to search again!" It's like suddenly realizing things don't have to suck.

Same thing happens to me: "Oh, crap, I meant to show threads, not posts. Great, now I have to... uh... just hit 'submit' again. Oh yeah."

I tried searching the SDMB a couple of weeks ago. I'd logged in a little while earlier so I could use the search function, entered my search terms, and got the message "you must be logged in to search". Fine, log in, start the search, get the message "you must wait 300 seconds before performing another search." Crap. Wait five minutes, start search again, get message "you must be logged in to search." Log in, search again, "you must wait 300 seconds before performing another search." Give up.

[Clayton_e]

Sometimes things run just fine when I'm over there, but sometimes I'm waiting ten, twenty seconds for a page to load..

Though that would be lightning fast compared to when I saw the boards on 56k...

Reminds me of how absurd I thought it was when the NBC news was loading as a podcast and I was annoyed because it was only downloading at twice the normal watching speed because I wanted to leave RIGHT THEN.

But still, as a message board its sometimes impossible to navigate with that lag.

[RobuSensei]

Random thoughts: Is there any other board on the net that requires a five-minute wait between searches?

Is there actually a group of posters over there that thinks this kind of thing is normal?

[What Exit?]

Random thoughts: Is there any other board on the net that requires a five-minute wait between searches?

Is there actually a group of posters over there that thinks this kind of thing is normal?

Not normal but there is a group that just doesn't care about the searches. The 5 minute wait has been driving me nuts since it was introduced.

[Borborygmi]

Good thing that it's only a temporary measure while they look into the issue!

[/meant to be snarky]

[Exy]

Not normal but there is a group that just doesn't care about the searches.

There were, and probably still are, a huge number of people who will defend anything about the SDMB. Five minute wait between searches, crappy moderation, not being allowed to edit one's posts, TubaDiva, literally anything.

[PonderThis]

Not normal but there is a group that just doesn't care about the searches.

There were, and probably still are, a huge number of people who will defend anything about the SDMB. Five minute wait between searches, crappy moderation, not being allowed to edit one's posts, TubaDiva, literally anything.

Fucking word. There are people over there who I ordinarily got along with just fine. If we were discussing some recipe or a movie or a book or whatever we got along great. But you say something just a little bit (OK, sometimes a whole lot) critical of some moderation decision or a technical problem and they are on you like stink on shit. I have literally jumped back in my chair at the vehemence of a post from someone I thought to be laid back and easygoing.

[Exy]

I have literally jumped back in my chair at the vehemence of a post from someone I thought to be laid back and easygoing.

Yeah. There was a strong culture there of not criticizing anything about the SDMB ever. I don't understand it. For some reason posters took even pretty gentle complaints about the place really, really personally and would get openly hostile, or passive aggressive and insulting (by, for instance, implying that anyone who complained at all was part of some imagined troop of malcontents who did nothing but bitch about the SDMB.)

It's hard to even blame TubaDiva. If I was in charge of a place and the people acted like that, I'd probably grow contemptuous of them and assume I could get away with treating them like crap too. There were a lot of things that were good about the place but that wasn't one of them; at times it really made for an oppressive atmosphere. Like some kind of horror film. "Smile and think nice thoughts or THEY'LL KILL YOU."

[elmwood]

Maybe a bit too late to add something, but from what I understand, there's almost no IP blocking on the SDMB. Most message board operators will blacklist IP blocks from several high-risk countries, sources of malicious and poorly behaved bots and spiders, and so on. I believe the SDMB has no such blocking, so on top of the hundreds of users logged in, it also has to deal with hundreds of Russian and Chinese bots, Korean search spiders, and other useless non-human traffic at the same time.

In the past, the SDMB used to block email registration from hundreds of free email providers. Today, I don't think that's the case. Check out the most recent users: Google their names, or search back through the list of recently registered users by manually decreasing the user number in the member profile URL: for instance,

http://boards.straightdope.com/sdmb/member.php?u=82780

Note how many are banned.

I don't think the database is optimized and repaired on a regular basis; doing so will speed up the site immensely. The server settings (/usr/local/apache/conf/httpd.conf) may also not be optimized.

That being said, it's a lot faster than it used to be.

[zebra]

I really never had that much of a problem with the speed.

[RickJay]

Of course, it's completely down now.

[Baldwin]

Seems to be down again. Now that I'm no longer banned, I may just drop SDMB and stick to domebo. And giraffeboards, although the vast bulk of my posting is here. But three message boards is too much.