Bug #7516

ACE Latest Update (198) seems to crash dedi server more than often.

Added by Benny over 5 years ago. Updated over 5 years ago.

Status:Closed Start date:12/29/2009
Priority:Normal Due date:
Assignee:Rommel % Done:

100%

Category:Complex
Target version:1.0.1
Component: Affected Version:
Close Reason:

Description

Our dedicated server seems to crash more than 10 times in an hour.

195 ran just fine, we could play the same mission for like 8 hours w/o crashing, now it's crashing after 3 minutes.

RPT's attached, note that the "Error in expression" never occurred with version < 198.

ArmA2Server.RPT (2 MB) Benny, 12/29/2009 12:04

arma2.RPT (2 MB) G_Rider, 01/01/2010 04:54

ArmA2Server.RPT (2 MB) Benny, 01/02/2010 10:40

20100103-ArmA2Server.7z (90 kB) cross, 01/03/2010 11:09

arma2.rpt.7z (20 kB) Paul-Hewson, 01/03/2010 14:16

arma2.RPT (68.9 kB) Paul-Hewson, 01/04/2010 19:35


Related issues

related to A.C.E. for OA - Bug #7526: ACE 2 198 Duplicate 12/29/2009

Associated revisions

Revision 32fa48f5
Added by Sickboy over 5 years ago

~ Changed: Temporary (during testing) disabled ACE Throw and Put weapons in Multiplayer. To enable them, add server-side addon; ace_c_weapon_enabled = true; publicVariable "ace_c_weapon_enabled". Refs #7516

History

Updated by Sickboy over 5 years ago

  • Category set to Complex
  • Status changed from New to Assigned
  • Assignee set to Sickboy
  • Target version set to 1.0.0

Thanks a lot for the report and rpt.

I might have a workaround for the problem, but it would need testing with a changed addon (required for both server and client), is it possible for you to do so ?
If the tests are positive, we can consider a hotfix.

Will at least create new signatures at every update for now.

Updated by Benny over 5 years ago

Tell me if you need to do some tests, can be done on clients & server, i got a full access on the server.

Updated by Sickboy over 5 years ago

  • Status changed from Assigned to Feedback
  • % Done changed from 0 to 30

Thanks man!

We just released a test-hotfix on the updater system, it would be really great if you could test it.
ACE build 199, ACEX build 162, ACEX_SM build 14, ACEX_PLA build 7, CBA build 38.

Each modfolder is now signed with a key tagged with the build number.
Please test with only CBA/ACE/ACEX on the server, And only bi + these new bikeys.

Updated by Benny over 5 years ago

Going to test, i'll post the feedback asap.

Updated by G_Rider over 5 years ago

Benny wrote:

Going to test, i'll post the feedback asap.

I'll test on the XR server also.

Updated by Sickboy over 5 years ago

Thanks guys! We have found another bug that we want to hotfix (unrelated to server crashes though), awaiting your results before moving on though :)

Updated by Sickboy over 5 years ago

  • Priority changed from Normal to High
  • % Done changed from 30 to 40

Updated by Benny over 5 years ago

So far so good, 2 hours without a single crash and going.

Updated by Sickboy over 5 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 40 to 80

Awesome. Final hotfix at 18:00 GMT. Thank you very much for your time and efforts!

The hotfix version was 199, later on 200 arrives. Have you checked which version the server has ?

Updated by Benny over 5 years ago

Nvm, some of our clients had a weird error, the message said that the server was 198, then a few moment later, the server was 199, they've reconnected and everything's fine now.

Server is 199.

Updated by Sickboy over 5 years ago

New hotfix release is up!
http://dev-heaven.net/news/show/268

Updated by Sickboy over 5 years ago

Any more issues, or is it addressed with our hotfixes? :)
At BIF seen some, but caused by old CBA versions, or no sig-check and thus having "old" ACE versions etc break the server ;)

Updated by rocko over 5 years ago

Not a single crash on our server since I updated to 202 yesterday. Shut down server right now to check rpt and logfiles.

Missions running:
mission="co@60 ACE Operation Moonwalk 1.12"; Duration ~3 hrs
mission="Warfare-051[ICED]@ACE2"; Duration ~ 30 minutes

Various users connected with ACE, two users without ACE (Vanilla, no mods).
No occasional server crash on client disconnect.

I should stress, that rpt shows NO errors (except the usual Arma errors) regarding mission scripts etc.
Also there was no signature check enabled this time. (So there might have been users with an older version of ACE)

Players used following addons mostly:
ACE, ACEX, CBA, ACEX_PLA, QUESH_KIRBUl, VOPSOUND, GDTMOD, ACEX_SM, FPS, LANDTEX

Updated by Sickboy over 5 years ago

Very nice Rocko, tnx!

Updated by kju over 5 years ago

Excellent job. Thanks to everyone helping!

Updated by zarckyp over 5 years ago

Since build 203 seems to server crashes appears again in warfare BE... :(

Updated by Benny over 5 years ago

Running fine by our side, our server's up for more than 7 hours without a crash.

Updated by Sickboy over 5 years ago

Benny wrote:

Running fine by our side, our server's up for more than 7 hours without a crash.

Thanks Benny! The responses are generally good about the last 2 versions.

zarckyp wrote:

Since build 203 seems to server crashes appears again in warfare BE... :(

Please make sure you run with signature checks on so that old versions of CBA/ACE are filtered out, especially until more people are on the latest version.
As we have especially applied changes to make ACE more compatible with non ACE servers, this can cause issues if older versions are able to get into the join process.

In any case, please as always provide arma2.rpt of at least a client and the server (arma2server.rpt). Bugz

Updated by G_Rider over 5 years ago

zarckyp wrote:

Since build 203 seems to server crashes appears again in warfare BE... :(

I agree. Playing BE Warfare single player or even online at the XR or Suicide Squad with ACE2 v203, the server crashes constantly. 201 and 202 fixes work well (server ran for hours without crashing) but now the two main warfare servers running v203 are crashing within an hour.

Hope Benny has some logs for ya.

Updated by Sickboy over 5 years ago

Do you guys run with signature checking enabled?
Are the crashes mostly occuring while others join?

The RPT you've sent is from a client, are both now crashing, and at the same time ?
How is it for Benny? Any mission updates?

Thanks!

Updated by Benny over 5 years ago

Yes we run signature, it's also updated at each updates.
Well everytime we crashed, someone was joining, when we lock the server, we don't have that issue.
My mission got updated recently, but nothing outstanding, it did the same on the version before.

Well, we played 203 yesterday with guerilla as resistance, server didn't even crash a single time during 10 hours (then i stopped it to update it to 204), might be unrelated, but ACEX_PLA might cause some crash (since most of the time, when we crash it's when we use it and the crash for client & server is an ACCESS_VIOLATION Type for both client and server).

I think that i'll remove acex_pla sometime to try with and without it, since the normal bis server crash don't give an errors in the rpt.

Uploading a fresh rpt.

Updated by Benny over 5 years ago

Bringing a small update, after restarting the arma2server process, we crashed 3 minutes after the game begining with an access violation, ACEX_PLA Stuff weren't even loaded / used in the game.

Updated by Sickboy over 5 years ago

Thanks a lot for the details Benny.
Our server has been stable as a rock, but we do not run the same mission.
Is it possible for you to boil down to what is exactly causing the crash?
Like spawning of units, killing them, firing, players joining, etc ?

From our side, the changes we've done were mostly related to the ACE variants of the Throw and Put weapons.
These changes were actually made to make the Mod more compatible with non-ACE mods.
However, it might be causing other issues now.
Or it might simply be other changes, to things we don't use at 6thsense, but your mission does, so more specific info would be welcome.

Otherwise I guess I should try your mission on our server.

Any other servers still reporting crashes?

Updated by Benny over 5 years ago

I'm trying to find what exactly crash it but it's really hardcore to find it, the last crash occured while noone joined.

Well if i proceed by logic, while we were on v203, server ran for +10 hours, all player have the ACE_Throw weapon added in my latest version, i didn't add the Put one.
Only ACE, ACEX, ACEX_PLA, ACEX_SM are allowed on our server. When we have the ACCESS VIOLATION error, some of our clients crash to desktop along with the server (a minority).

Well, it's also random, can be 3,30 or 60 minutes.

Spawning & Killing units doesn't really seems to be a problem since all kinds of units were spawn previously and most of them fired at least once.
Player joining seems to trigger it sometime, but again it need to be checked.

Mission's here if you want to try it:
http://dev-heaven.net/attachments/download/4204/WarfareV2_053Lite-ACE.Chernarus.zip

Note that when we crashed after 3 minutes, the only things that occured were: AI Spawning/Being killed, Gunfights, Players Joining. Nothing else.

Updated by Sickboy over 5 years ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Sickboy to Rommel
  • % Done changed from 80 to 50

Thanks a lot, hope to be spending time on this in the next days, and hopefully get some support from Rommel.

Updated by ICE over 5 years ago

I know i've mentioned this to you before Benny, but it sounds to me like the RPT file was at maximum size. I made a .bat file that deletes it at every new arma2server.exe start. It helped alot for my server at least.

here is my .bat command "DEL /Q C:\ArmA2-1\Server1\arma2server.RPT"

Updated by tcp over 5 years ago

Well, I've seen RPT at over 2GB. That's a lot of errors and very hard to accumulate except over a long period of constant errors. In my case, a particular mission had an error in a fast loop but the mission was still played for a long time. I don't think the RPT is related to crashing, but it is nice to have a fresh one.

Updated by cross over 5 years ago

Here is the rpt after crash..
we started to have crashes again with b204 in various missions like domination, longest day, untern..

Updated by Sickboy over 5 years ago

Thanks a lot. I believe it is related to my throw/put changes and will try to bandaid that for tomorrows update.

Updated by Paul-Hewson over 5 years ago

I don't know if it could be useful but here is my rpt as a client.

Updated by Sickboy over 5 years ago

Thanks.
As promised, I implemented a temporary bandaid.
ACE Throw and Put weapons are disabled by default in multiplayer.
See changelog notice on how to enable them server-sided, if you need to.

I hope the 205 will show no more crashes, if so, we will devise a plan to attack and resolve the situation :)

Updated by Sickboy over 5 years ago

  • % Done changed from 50 to 60

Update out with specific notes for this ticket:
http://dev-heaven.net/news/show/277

Updated by Paul-Hewson over 5 years ago

Got a server crashed at Kelly's Heroes (update to version 208 just before we launch the mission). I don't have their rpt file but here is mine.
This crashed show up approx. 30 mins after we start the mission Regroup & Retaliate.

Maybe KH members can provide the server RPT.

Updated by Sickboy over 5 years ago

Hm, Thanks;

Ref to nonnetwork object R Partisans:1 (Paulochon) (p25)
Warning: Delete out of order
Warning: Delete out of order
Warning: Delete out of order

And a lot of them.

Hopefully mission/situation related :o

Not sure if their CBA is up? http://forums.bistudio.com/showpost.php?p=1534823&postcount=909

Updated by Sickboy over 5 years ago

How's it turning out for everyone?
Any more crashes?

If the problems are gone for a day or so, if you would be so kind to add a server-side addon that enables the add of the ACE_Put and Throw weapons, like in previous version?
I will supply details by then. Thanks!

Updated by Benny over 5 years ago

Hey,

Well, for now i'd say that it's crashing less than it use to do, a special event must trigger it since we haven't touched anything since the last update.
Adding a server side addon is not a problem for us, just hope it won't conflict with the mission (since i add ACE_Throw to each players).

Updated by Sickboy over 5 years ago

Hehe, well, we disabled ACE_Throw and ACE_Put adding to units, because we believe it's the cause of the crashes.
So if you are adding them to all the players, it counteracts :D
Would be great if you could test without the ACE_Throw and Put weapons,
especially since you say it crashes less now, might relate to only players having those weaps now :P

Updated by Benny over 5 years ago

I've made an hotfix to remove the ACE_Throw, so now it's not used at all, well during our last rounds, i recall that noone used the "special" weapons.
Ah well, if you release an hotfix let me know, i'll update both the server and the mission with it (if i upload it too early our clients are going to wonder why they can't use the special weapons :P).

Updated by Sickboy over 5 years ago

Hehe, not sure if we or I misunderstand now :)
In the update we released earlier today, we disabled the ACE_Throw and ACE_Put weapons adding to units (in Multiplayer), to see if it stops the crashes.
So if you are adding them now with your mission, it would be great if you could test without adding them. If the crashes stop completely, it must be related to it :)

We can then use these test results in hopefully a proper fix, upcoming Wednesday :)

Updated by Benny over 5 years ago

Aaaaaah nvm it's late here :D

Going to update it & give some feedback asap.

Updated by Sickboy over 5 years ago

Benny wrote:

Aaaaaah nvm it's late here :D

Going to update it & give some feedback asap.

Haha np, thanks a lot!

Btw, since which version of ACE did you start adding the ACE_Throw and Put weapons in your mission?
As in 202, when you guys barely had crashes, the ACE_Throw and Put weapons were not added properly to most players.
Then we fixed that for 203, and you guys started to report the crashes again.

But if you in the meantime started adding the Throw/Put weapons manually.. maybe the problems occurred due to twice having them? :D

Updated by Benny over 5 years ago

Well, i was only using ACE_Throw, i've added it whenever player gear was changed (Gear Menu, Default Gear).

Which mean player had ACE_Throw at the begining (he had only normal grenade to throw anyway :P) & After changing it's gear.

Going to monitor our server, they're playing the hotfix.

Updated by Sickboy over 5 years ago

Thanks a lot Benny! One way I am hoping this is the cause....
But on the other hand, I am actually not, because I think it will be quite hard to resolve this from our side.

We could revert back to a situation from before the changes, but the issue with this is that if anyone joins without the mod, or with alternatively setup muzzles for the put/throw weapons,
it crashes your server, and if an ACE user joins a non-ACE server, it crashed, most of the time.

Updated by Benny over 5 years ago

Server's up for more than 6 hours by now and still going.

Updated by Sickboy over 5 years ago

Benny wrote:

Server's up for more than 6 hours by now and still going.

Nice, thanks! Before you removed the Throw code, it was still crashing now and then right?
Think we've found the cause then :)

Updated by Benny over 5 years ago

Yes, it was still crashing before i removed the ACE_Throw, Server's up for more than 10 hours with an average 30 players by now.

Updated by Sickboy over 5 years ago

Benny wrote:

Yes, it was still crashing before i removed the ACE_Throw, Server's up for more than 10 hours with an average 30 players by now.

Thanks a lot Benny. Could you share me the code you used before to add ACE_Throw ?
Im especially interested in if you applied any delays etc :)

Updated by Sickboy over 5 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 60 to 80

In anycase, for upcoming update I have restored the original Throw and Put overrides, from pre 198.
ACE_Throw and Put weapons will be inexistent again.

Updated by Sickboy over 5 years ago

And in the update; Looking forward to your results!
http://dev-heaven.net/news/show/278

Updated by Rommel over 5 years ago

  • Status changed from Resolved to In progress
  • % Done changed from 80 to 90

Any more crash reports? The servers I have been testing on have been running nice and smooth.

Updated by Rommel over 5 years ago

  • Status changed from In progress to Feedback

Updated by Xeno over 5 years ago

Any news ?

Still happening ?

Otherwise I think we can close it.

Updated by Rommel over 5 years ago

  • % Done changed from 90 to 100

I've been running an ACE server for a while now, had a four hour mission last night, no issues, and the servers been running for over 24 hours.
I'd say the major issues are fixed.

Updated by Xeno over 5 years ago

  • Status changed from Feedback to Closed
  • Priority changed from High to Normal

Yep.

Please create a new ticket if there are problems again.

Updated by Sickboy over 5 years ago

  • Target version changed from 1.0.0 to 1.0.1

Updated by Sickboy over 5 years ago

  • Target version changed from 1.0.1 to 1.0.2

Updated by Sickboy over 5 years ago

  • Target version changed from 1.0.2 to 1.0.1

Also available in: Atom PDF