MODERATOR CLEAN UP

The easier league, for smaller, less time consuming competitions.

Moderator: Moderators

MODERATOR CLEAN UP

Postby bythelee » Fri May 24, 2013 3:18 pm

Hi Guys,

Thoughts and advice, please.
Anyone can post ideas here, but the AR forum has a problem, and we need to spring clean. A lot.

New accounts are being signed up by spam bots, maybe spam humans, that then add posts and new topics to all threads and forums.
These are held as "awaiting approval" by a moderator, but they ALL appear to moderators. So we don't get the clean view that regular users see.
For example, I had to scroll to page 2 to find the 251 AroundtheBox thread, last posted to (legitimately) less than 12 hours ago.

So, action is needed, and the danger is that new legitimate members will be swept aside with the rest of the spam.


Moderator choices are powerful, but inconvenient.
For example, we can very quickly "disapprove" new topics and posts, by using the moderator queue of unapproved things, using the "mark all" option you may have noticed in your PM inbox, and letting rip.
But it isn't clear what will happen to this stuff. For example, in the AR2 A forum alone, there are 28,526 topics awaiting approval with 507 approved ones. Makes sense - 250 challenges plus results threads = 500, plus a few general announcements.
In addition, there are a further 31921 unapproved posts in legitimate threads.

So you can see the scale of the problem. All of the other forums are being equally hit. In all, there are 753000+ topics waiting for approval, and a further 803000+ posts in legitimate threads. And this is since March 2012 for posts, and August 2012 for topics (I guess someone cleared out March to August 2012 already)


I tracked one post from the last competition. It's clearly spamware, computer generated non-english that pastes together random words and phrases to seem like it's a person. Invariably there are links to click. The user joined on April 6 2013, has seven posts in the queue (Apr7, 10, 29, May17,22,23,23) so it's a repeat offender, and potentially escalating. The posts have been in a few different places, but mostly the "recent" ones, like competitions, AR1 Dark Masters, and Custom Levels All Terrain Vehicle. It's adding posts to existing threads, not creating their own new threads.

If "disapproval" meant deletion, then I'd go for it. But it looks like history is kept (well, logs at least), so this stuff possibly still hangs around somewhere, making the forums big and slow.
Moderators do have the power to delete topics entirely, but this has to be done one at a time, by looking at each thread. This would be unworkable.
Trouble is, once stuff is "disapproved", it disappears. I can't find where it goes to. So it might be deleted.

Thing is, the numbers are so big that getting rid of this baggage would seriously reduce the size of the forum, so making sure it's gone and not just stored under another list would be best.

I'm wondering if anyone has any ideas about how these forums work, to give us a bit of confidence before we take the plunge.

PS: It's just Ioncorpse and me at present, but if anyone else wants moderator status, I think it'll take a PM or email to Peter Stock to get it - do it now while he's here.... (There are other moderators listed, but none active).
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bythelee » Fri May 24, 2013 6:41 pm

One offer of "irregular" moderator from decsystem10 so far. Thanks.

But consider this: rather than boosting your present account to moderator status, and having your world forever changed, I suggest would-be moderators create a new account.
Ioncorpse and I can approve it, then we email Peter asking for the new account to be boosted to moderator status.
That keeps your present account as it is, and gives you a new one to wield the power of [insert deity name here] with.

The more moderators, the more redundancy! With numbers as they are, I think anyone who has competed in a challenge should be eligible. What say others?

We could create a general purpose moderator account and given everyone access to it as they want it. But this feels a little too loose and lacking in accountability to me. I'd prefer people to have their own, unique accounts at this time. Again, what say others?

PS, the forum designer and "owner" of the server this forum runs on, is PeterT, who hasn't been active since 2009.
So our options for site admin are limited to what Peter Stock can do (creating moderators)
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bytheMod » Sat May 25, 2013 3:15 pm

Posting to get approval for my new moderator account (I hope)
bytheMod
 
Posts: 28
Joined: Sat May 25, 2013 3:12 pm

Re: MODERATOR CLEAN UP

Postby bythelee » Sat May 25, 2013 3:19 pm

OK, I registered a new account, made a post here, and approved it with my moderator status.
Now to ask Peter Stock to add moderator status for it (and knock bythelee back down to a normal user).
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bythelee » Sat May 25, 2013 4:23 pm

CleaningUp methods:

I've made a start chipping the spam out of the AR2 A league forum.
It's heart-stopping stuff, deleting new topics.
I once thought I'd blown away the latest competition, but then realised another 8 new topics had arrived in the last minute, shifting it down the list. PHEW!
At the moment, I am only a moderator for AR2 A League, so I can't do forum-wide stuff easily. I've emailed Peter about changing that.
And AR2 A league needs special care, since our challenges and results are dispersed within the clutter.

"Moderation Queue" only permits disapproving or approving of posts and topics.
Deletion can be done by operating from the MCP (Moderator Control Panel) "Main" tab, via the View Forum and View Topic sub-tabs on the left.
(To get "View Forum", you might have to switch from Main to the Moderation Queue, and select AR2 A League rather than all forums, then switch back to Main. This is because I don't have permission to moderate all forums (the default), only the AR2 A league)

I have been deleting one page at a time. But it's risky, as accidentally marking a challenge or results topic means losing it instantly, but spotting the rare valid topic (one per page, maximum) is difficult, and needs vast concentration.
After the scare with losing thePits, I've switched to a much safer, if slightly more long-winded approach: Merging.

Having Marked the spam (leaving one spam topic unselected on the page), I choose the option to Merge, instead of Delete.
The view then shifts to the same list, with some topics having [select to merge] added in front of the topic title. Only topics that were not previously marked will be available.
So choose the one spam topic that was deliberately not marked (by clicking [select to merge]), and all of the selected spam topics will be merged into it, forming one big spam topic of many posts.
If a challenge or results topic is selected accidentally, it can be retrieved by splitting the merged topic.

The downside is that after merging, I get to see a Topic view of the merged post. So I have to go back to the MCP, and back to View Forum before I can do more merging. But at least this way, the valuable topics are 99.9% safe from accidental deletion.

When done, it is quick and easy to verify that the challenges and results topics are safe, because the challenge and results topics are now easily visible amongst the few spam topics. Once certain all is safe, the merged can be safely deleted.
I'm leaving the merged topics in place for now, so moderators can see what I'm doing. I'm tending to leave one merged topic in between each challenge and results entry. Because these merged topics are typically 50 posts long, and if we merge them into one huge topic, I think we could break the forum.....


PS: "Mark all" isn't presently working for me. :( Not sure if it's my browser, or a forum / server problem. So I'm having to Mark each topic manually.
But I've done about 2% of the AR2 A league crap in about ten minutes, so it'll get done eventually. If I can get Mark All to work, then that should go about 5x faster.

Plus, the New Topics are slow because of having to pick out the good stuff. If we assume there are no legitimate members with lost thread posts, then cleaning the posts awaiting approval will be much quicker again. And likewise the other forums have had so few, if any, legitimate posts and new topics over the past year, that they too will be much quicker to mark and delete.
So, maybe 2 hours work to clean up AR2 A league, and an hour per other forum. Not the insurmountable task it looked a few days ago!

I've also asked Peter about powers to ban members, because I've noticed that 50% of the spam is coming from maybe 5% of the members. There is a lot of repeat offending, and banning those folk will cut down the work. For a while, anyway.
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bythelee » Sat May 25, 2013 6:18 pm

OK, my estimates were a little off. :oops:

Merging about 8% of the AR2 A forum has taken about an hour. And I'm only about two competitions back. Which means, the problem has been escalating recently.

If "Mark All" was working, that would speed up by at least a factor of 10.

Unfortunately, some googling shows that "Mark All" not working is most likely a problem with the website set up.
It needs admin access to solve.
So, assuming that won't be available, we're in for the long haul.

I've asked Peter Stock about upping powers, especially banning, 'cos it looks like my list of 6 names is posting at least 20% of the recent crap.

Meantime, I have a tweaked tactic:
Use another browser to view the forum without the crap.
Check the date and time of the last "legitimate" entry.
In View Forum, go forward page by page until that entry is found. That tells me how many pages I can safely merge top to bottom without having to check (and this is where Mark All would come into it's own).
Starting from the page where the next entry is, and working back towards page one, select all posts on the page, and choose to merge.
When it asks where to merge to, change to page one, and pick the spam under the last legitimate entry on that page.
Rinse and repeat, this time switching to the next page number down from the last one you did (because one page will have now been removed by the merge)

If new posts arrive while you're working, the legitimate topic will be shifted down the page it is on, never forwards onto the page you're working on, so it's a little quicker and faster, while maintaining safety.
And Oh, how I wish Mark All was working.......
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby DecMod » Sat May 25, 2013 6:30 pm

Blow lamp and vacuum cleaner at the high port...
DecMod
 
Posts: 1
Joined: Fri May 24, 2013 7:20 pm

Re: MODERATOR CLEAN UP

Postby bythelee » Sun May 26, 2013 1:32 pm

I've emailed Peter Stock about raising DecMod and bytheMod to Moderator status.

Ioncorpse, if you want to register IonMod (or IoncorpseMod, or something similar) we can ask Peter to "downgrade" Ioncorpse back to a normal user, while the new account gives you the same Moderator status you have now. He said he could do that for me (bythelee) hence the bytheMod replacement.


And.... if only we had Administrator access....
http://www.youtube.com/watch?v=2NDJBusMSZE
Pruning some users would make a vast and instant difference.

It actually looks like 90% of the present incoming spam is from about 10 usernames. So a banhammer will help stem the tide, while Pruning would clean out a lot of the crap awaiting approval. I've emailed Peter with that list (and pruning idea), and also asked for banning powers for moderators.

But, truth is, if "Mark all" cannot be made to work, it's going to be an impossible task to clean the whole forum.
I'm still wading through AR2 A league (about 14% of topics cleaned) but it takes 50 clicks in the tiny mark boxes to mark one page, instead of scroll to the bottom, and click mark all. That's where the time and RSI injury can be saved.

From my google hunts, it looks like mark all not working is because of a setup problem in the website itself. Fixing that needs admin access, but if Peter has sufficient rights and is prepared to share them with us it might be possible......

Meantime, I'm going to PM PeterT (the sole Admin) to see if he's interested in fixing Mark All or sharing powers.
I'm just hoping that doesn't trigger him to realise he's been paying for an AR server all this time...... but we can hopefully negotiate a move somewhere else if that happens. There's too much history and great work on the AR forum to see it lost now. (Speaking of which.... backups....)
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bythelee » Sun May 26, 2013 2:15 pm

OK, so I've found a current email address for Peter Thoman (the Austrian Admin and owner of the metaclssofnil website, and presumably supplier/hirer of the server this forum runs on) and want to know whether it's OK to contact him.

I'm presuming he's aware that the AR forum is still clocking up bandwidth at his expense, but if we are presently "under his radar" then contacting him might cause the plug to get pulled on us. Is it OK to risk that?

I would hope that if he doesn't want to help (like, share admin powers, fix Mark All, provide spammer banning powers etc) he will allow us to "copy" the forum to a new host and server, where we can fix and run things as we want. I'm a novice at that, so if we have any "expert" members with advice (or wanting control) then please shout. I'm really just trying to have an emergency recovery plan in place in case disturbing this sleeping dog produces a fatal backlash.

Mostly, I'm a little worried that attempting to fix "mark all" on our own might break things totally beyond repair, although admin status to use the pruning feature would solve a lot of headaches.

I believe that at present rates of spam accumulation the forum will break sooner than expected, maybe within the next year. I've already noticed times when response is slow, to the point of timing out. And that's not my ISP's fault.
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby mark_man » Sun May 26, 2013 2:53 pm

All this stuff in here is way over my head I don't understand any of it .I will say for me the site is working fine as it is so ,for me if it is not broken no need to fix .Thats about the only input I can give sorry :(
I'm Just A Simple Man.
Please for everyone don't believe ANY scores I post .... :D :twisted:
mark_man
Contest Legend
Contest Legend
 
Posts: 1686
Joined: Sun Sep 10, 2006 1:31 pm
Location: England

Re: MODERATOR CLEAN UP

Postby bythelee » Sun May 26, 2013 4:03 pm

mark_man wrote:All this stuff in here is way over my head I don't understand any of it .I will say for me the site is working fine as it is so ,for me if it is not broken no need to fix .Thats about the only input I can give sorry :(


Thanks. There is an element of "if it ain't broke, don't fix it". Perhaps we do nothing now, and deal with it later.

The post immediately before yours is the really impotant one (contacting Peter Thoman), so just focus on that.

My main fear is that 90% of site traffic is spam; bandwidth and server space costs money, such that at some point in the future we might find the site just goes dark anyway. While I now know who to email if that happens, it might already be too late if the site was deleted.

If we had a little admin power to deal with the crap, things could probably run indefinitely. But trying to fix that might trigger an adverse effect.
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby mark_man » Sun May 26, 2013 4:21 pm

Someone gave you admin powers so I would assume that the site is not a ghost site .I don't know what spam you are talking about the A league is the only active thread and has been for a long time and I cannot see any spam in it ?.
all pages load in an instant for me no problems at all with time outs etc all load split second .IMO no levels should be deleted or the entry's made to them so that only leaves text and I would assume that uses very little memory
I'm Just A Simple Man.
Please for everyone don't believe ANY scores I post .... :D :twisted:
mark_man
Contest Legend
Contest Legend
 
Posts: 1686
Joined: Sun Sep 10, 2006 1:31 pm
Location: England

Re: MODERATOR CLEAN UP

Postby mark_man » Sun May 26, 2013 8:14 pm

armadillo was down for at least an hour lol after my last post anything to do with you :D
I'm Just A Simple Man.
Please for everyone don't believe ANY scores I post .... :D :twisted:
mark_man
Contest Legend
Contest Legend
 
Posts: 1686
Joined: Sun Sep 10, 2006 1:31 pm
Location: England

Re: MODERATOR CLEAN UP

Postby bythelee » Sun May 26, 2013 10:01 pm

There are a few levels of powers:
Admin is the highest, you can edit the website with that.
Then there are Global Moderators, who can moderate the whole forum at once.
Group Leaders can add users to their group and give those users whatever powers the group is allowed.
I got given Moderator powers in the moderator group, which means I can approve posts and threads, modify posts, delete them, etc.
But I can't ban users, and the "mark all" function not working means it proves impossibly arduous to clean all of the spam of the site, because I can only do it one by one, instead of 50 at a time.

Peter Stock got prompted by my email to enquiries@armadillorun.com, about SketchyGalore. He is not an Administrator - if you check the "members" page you'll see he's in a group all of his own (Armadillo Run Developer) but I think he must have powers close to Administrator, if not admin itself, because it was he that boosted me to moderator.

From the samples of Peter Thoman's code visible on his website (he hosts the forum, and is the sole Admin), I wouldn't be surprised at anything.
Go to http://metaclassofnil.com/ if you want to be REALLY scared. The google explainer under a google search says "When you have learned to snatch the error code from the trap frame, it will be time for you to leave." Which is a clue about what to do with what you see on that webpage, to get into the website - but I havn't the programming skills to work it out.

It turns out we had one active moderator, Ioncorpse, who approved Sketchy's posts which gives permission for further posts to not need approval. Now we've got me too, and decMod for a third when Peter Stock reads his emails again (don't hold your breath).


As for "not being a ghost site", Peter Thoman (the admin) has not logged in since 2009.
Peter Stock had not collected the two PMs I sent him in April 2012 until last week, when my email prompt brought him back to resolve Sketchy and bump me to moderator. So, actually, the site has been completely unattended for over a year, with the sole exception of Ioncorpse.


Ioncorpse told me (before I was a moderator) that only moderators get to see all of the unapproved posts and topics. And we can't turn it off, either.
Seeing all of it makes it pretty impossible to find anything, which is why I've been suggesting seperate moderator accounts for those wanting it.

Since 252 thePits started, we have had a further 250+ New Topics started. Which means, the 251 Results thread is now on page 6 for me. And those are just new Topics, and does not included the spam posts that are being made to existing valid topics. Anything unapproved does not show for normal users, so everyone else sees only the good stuff. We couldn't see Sketchy's posts until Ioncorpse had approved them, but they were sitting there waiting for approval even though we were blind to them.

The AR2 A league alone has about 28000 topics, instead of the 509 or so "valid" ones. That means there's almost 600 pages of topics, instead of the 5.1 pages of valid topics.
All of this spam has arrived over the past year, since mid-march 2012, about the time I joined, and Peter Stock changed the joining rules. Previously new users had to be approved before they could do anything. But Peter found it too tedious trying to filter the spam applications from the genuine ones (I got very lucky!). Now, new users get automatically registered upon activating their email, but can't post anything without moderator approval. So the moderators get flooded with the spam topic and post approval requests instead, and the website database balloons with the information it has to hold.
We're only one year into this "new routine" and I have no idea how long it can continue before it gets so big and so cluttered that it breaks the server.

Most posts are text, but are typically long (like this one). Yes, it isn't a lot of data, but it's the sheer volume of entries that makes the database big. It's a library with close to a million books, even if each book is only page long, and that takes a lot of filing, searching, and record keeping, which is what makes the database big.

This level of spam is occuring across the entire forum, in all of the categories. So, it has expanded tenfold in the past year, and is accelerating. My fear is that doing nothing will see it break under the flood (in a DDS type of problem) or the plug simply gets pulled when Peter Thoman sees the webspace and bandwidth being used by the armadillo part of metaclassofnil. He has other domains too, so metaclassofnil won't go, but the armadillo bit might.

If "mark all" was working, the moderators could clean all that crap in a few hours of work. That it isn't, means it will take me about 10 hours to clean just the AR2 A forum Topics, and at least the same again, for the spam posts within valid topics. Repeat that for every category on the website...... It simply isn't doable.

If you discovered that a neighbour was sucking ten times your average data from your wireless broadband connection, you'd probably do something about that, wouldn't you? Especially if you never used wireless connections. You'd possibly just turn off the wireless part - that's when the AR forum will just "go dark".
I'm hoping that if we approach Peter Thoman nicely, co-operatively, etc, offering to take care of it if he lets us have the tools for it, we can stem the tide and keep things happy with a little housekeeping. He's been sponsoring us without being involved for the past four years, so I hope we're not a big part of his empire, and he won't mind it continuing that way, if we take care of it.
That it's grow to be ten times the size in one year, after seven years of gentle use, means we probably will show up on his radar soon anyway.

If we do nothing, the site will eventually get slower and slower, as the server it is hosted on struggles to keep up with the ballooning database. And moderators don't currently have the working tools to fight it.

I've had a few instances where the site crawls to a halt, or fails to respond completely. Like the "site down" Mark_man experienced. I don't think the database is big enough yet to kill it, but it all depends on how much horsepower the server has. For such a lite forum, set up so many years ago, do not expect it to be much. EA failed to cope with the server loads when SimCity got launched recently - it's that sort of danger territory we may be approaching. But I can't tell how near or far we are to tipping over the edge.

It won't take much to get things healthy again, but we do need stronger powers to deal with things than what the moderators presently have.

PS
I emailed Peter Stock with the list of 13 users, and notes about "pruning" them. This removes their accounts, and deletes all of the topics and posts at once. It could cut the database in half for maybe ten minutes work, and kill maybe 50% of the presently incoming crap, so that seems like a bargain, and could win us a lot of time. If he has the admin power to do it, that is.....
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Re: MODERATOR CLEAN UP

Postby bythelee » Sun May 26, 2013 10:10 pm

And, as I tried to post the previous topic, I got an SQL database error....
Attachments
SQLdatabaseerror.jpg
bythelee
 
Posts: 875
Joined: Sat Mar 31, 2012 1:42 am
Location: United Kingdom

Next

Return to AR2 "A" League

Who is online

Users browsing this forum: No registered users and 12 guests

cron