Support #10306

Dedicated Server Crash

Added by FunkDooBiesT about 5 years ago. Updated almost 2 years ago.

Status:Closed Start date:04/25/2010
Priority:Normal Due date:05/28/2010
Assignee:Sickboy % Done:

100%

Category:Complex
Target version:-
Component: Affected Version:
Close Reason:

Description

Hi,

Over the past 2 weeks I've had allot of Server crashes From the error reports I can't tell you allot but it's always the same errors prior to the crash:

Server running Vanilla 1.05.

Mod list:
"@ACE" V342
"@ACEX" V227
"@ACEX_PLA" V17
"@CBA" V94
"@FDF_Podagorsk" V1
"@FFAA_Civiles_Arabes" V1
"@FFAA_Terroristas" V2
"@FRAN_quesh_kibrul" V2
"@FRAN_sbrodj" V1
"@GL4" V12
"@isla_duala" V9
"@japahto" V2
"@JTD" V1
"@MBG_Celle" V1
"@MBG_kellu" V4
"@namalsk" V8
"@panthera" V5
"@sap_clutter" V4
"@sap_everon" V1
"@SIX" V164
"@six_aiaktalik" V8
"@SIX_SHL" V4
"@SIX_WTK" V3
"@Kulima" V2
"@Thirsk" V1
"@WinterCamos" V1
"@Seals" TF86 Navy SEALs V1

RPT file always has this at time of crash:
2010/04/25, 18:42:34 Uknown entry type
2010/04/25, 18:42:34 EOF in string
2010/04/25, 18:42:34 Cannot find base Y for /value/Item8/
2010/04/25, 18:42:34 Cannot find base AY for /value/Item9/
2010/04/25, 18:42:34 .: Value type not matching: != Array =======================================================
-------------------------------------------------------
Exception code: C0000005 ACCESS_VIOLATION at 0040BE7A

Notes:
The Crash happens in the Training mission for ACE.
Mostly the crash happens when there is a player A-10 that crashes.
Some Clients had crashes aswell when this happened. I don't have those RPT files but I can get them if needed.

2010-04-25-18-43-57_ARMA2Server.rpt - First crash on Training Mission (108.5 kB) FunkDooBiesT, 04/25/2010 17:39

2010-04-25-18-57-20_ARMA2Server.rpt - Second Crash on Training Mission 14min later (35.7 kB) FunkDooBiesT, 04/25/2010 17:39

ArmA2Server.RPT (15.7 MB) Lt.Chris, 04/27/2010 10:02

arma2server.rar (49.6 kB) Lt.Chris, 04/29/2010 00:30

arma2server.rar (22.1 kB) Lt.Chris, 05/02/2010 20:48

History

Updated by zGuba about 5 years ago

What mission?

Updated by Fisgas about 5 years ago

We've been experiencing crashes on our server as well, with the same type of problem. It only does happen on large scale games (25-40).

Mods running: @CBA; @ACE; @ACEX; @ACEX_SM.

Updated by Sickboy about 5 years ago

  • Category set to Complex
  • Status changed from New to Assigned
  • Priority changed from Normal to High
  • Target version set to 1.1

Updated by Lt.Chris about 5 years ago

Heres GOL server's rpt if you want to have a look at it. Ive also chatted to other clans such as 15th MEU and they have been suffering from large scale game crashes for the past month and a half which is about the same as GOL so theres something defently going on.

Updated by Sickboy about 5 years ago

Hi Lt Chris,
could you please supply a recent rpt instead? :D

Exe timestamp: 2009/06/19 18:34:44

Current time: 2009/06/20 21:39:38

Updated by rocko about 5 years ago

And zip it

Updated by Lt.Chris about 5 years ago

ouch i just realized that myself, ill get a fresh one up providing people are willing to play to crash the server to generate the log :)

Updated by Lt.Chris about 5 years ago

here you go, the server has not actally crashed in this instance (the lads did not want to go onto a game just to crash it) but its throwing up all the errors it does get before it actally crashes. Hope this helps :)

Updated by rocko about 5 years ago

We need a logfile from an actual crash. Examining a logfile of a server that just runs fine is senseless.

Updated by rocko about 5 years ago

  • Priority changed from High to Normal
  • Target version changed from 1.1 to 395

Updated by rocko about 5 years ago

  • Target version changed from 395 to 1.1

Updated by rocko about 5 years ago

  • Target version changed from 1.1 to 395

Updated by Lt.Chris about 5 years ago

Disapointing to see something as serious as this moved to future

Updated by Xeno about 5 years ago

Might be disappointing but without any hint what went wrong we can only guess...

Just as an example, a server had massive problems with a new mission.

After they removed Namalsk and Isla Duala all their problems went away and everything was back to normal.

What I want to say is, that it is not allways ACE that is causing the troubles.

Updated by ViperMaul about 5 years ago

Lt.Chris wrote:

Disapointing to see something as serious as this moved to future

This bug report is still alive and it is still important.
Also, there were crash problems using one of the recent betas.
Need more clean data to reproduce. It would help to see a server RPTs that crash without using any betas and only using ACE mods.

Updated by rocko about 5 years ago

Lt.Chris wrote:

Disapointing to see something as serious as this moved to future

We don't need such uber useless comments from you in the future.
We simply cannot do anything without a crash rpt. Simple as that.

Updated by Lt.Chris about 5 years ago

No need for the un-needed agression roko.
Anyway I got the lads to play and the server did crash and heres the rpt, this was created right before the game so the timestap should be much better :)

Updated by Sickboy about 5 years ago

Thanks for the log, and rarring it :)

The issue mainly is that some people seem to be unable to post without flavouring their post with a certain level of drama or with unrelated personal opinions.
It is a bug tracker, not a forum, and it is basically the centre of our *work*flow.
We do what we can in any case :) Sometimes we have enough though.

Updated by rocko about 5 years ago

Regard it as educational measure, Lt.Chris. Useless information = don't post them here. Hope that makes it more clear to you. Sickboy explained it more precisely.

Anyways, your rpt does not reveal anything useful.

GLT_Missilebox in glt_missilebox\

This is not listed in the addons from the modline.
Tell us what other addons you are using. We are simply too busy to read all the addons involved from the huge addon list at the end of the rpt. (Time issues/Interest/etc)

Unless you can assure crashes are not caused by them, it's still impossible to tell the cause as the rpt does not contain anything ACE related regarding the crash.
Run those addons w/o ACE until the server crashes again.

-Edit:
Same goes for FunkDooBiesT:
Try running your server with less mixed addons. Until the server crashes.

Updated by Lt.Chris about 5 years ago

Thanks for the reply The addons we only have running on the server along with ACE and CBA are the following

ArmA version - 1.05 Offical Patch

NZXShadows M1151 - http://www.armaholic.com/page.php?id=8444
MAP EU - http://armaholic.com/page.php?id=6194
Isla Duala + the content which came with it - http://armaholic.com/page.php?id=6194
GLT - http://www.armaholic.com/page.php?id=6071
Quesh-Kibrul - http://armaholic.com/page.php?id=8698
Celle - http://www.armaholic.com/page.php?id=9448
FDF Podagorsk - http://www.armaholic.com/page.php?id=8791

After the crash we ran the server with a mission I made without ACE and with only the addons above and without ACE and it ran fine though the entire operation for a good 2-3 hours

Updated by Sickboy about 5 years ago

The crash error is at least different from the string EOF not an array etc error.
Also: "Exe timestamp: 2010/04/19 12:32:40" is this because the exe file was copied sometime or are you running the arma2 beta? Especially the last versions are said to be buggy / laggy.

On a side note, there's a lot of network messages pending.
Did you guys tune the network settings of the arma server, like MaxGuaranteed etc etc (I recently discovered that tuning those values up, actually increases the lag and should rather be considered only for low bandwidth servers instead :P)

Updated by Lt.Chris about 5 years ago

It should not be copied as the server is a fresh install and not running any betas, perhaps you can have a 1st hand look at our setup and maybe you might spot the problem? :)

Updated by cja100 about 5 years ago

Would also like to add this windows event error when the server crashes.

Event ID:26
Application popup: arma2server.exe - Application Error : The instruction at "0x0040be7a" referenced memory at "0x00000074". The memory could not be "read".

RAM is fine, Hard drive is fine and recently done a reinstall which didn't make any difference. To my knowledge our network settings have not changed for a long while and remember this does not occur when not running ACE.

Updated by Sickboy about 5 years ago

Well i'd imagine the additional strain on the system from streaming the extra model data, increased script load and what not, could have an effect on the stability. I am unsure if this is then an arma issue, server issue or ace issue :P

Updated by cja100 about 5 years ago

We can try next Sunday running ACE only + certain client side performance mods that are proven, and see what happens.

Updated by rocko about 5 years ago

  • Tracker changed from Bug to Support

Updated by Sickboy about 5 years ago

  • Assignee set to Sickboy

Thanks! Over at BIF there's reports about a possible issue when ACE is used in conjunction with recently released or updated "Pedagne" mod.
Not sure if it is related.

Updated by Sickboy about 5 years ago

According to BI, the error:

the 2010/04/25, 18:42:34 Uknown entry type
2010/04/25, 18:42:34 EOF in string
2010/04/25, 18:42:34 Cannot find base Y for /value/Item8/
2010/04/25, 18:42:34 Cannot find base AY for /value/Item9/
2010/04/25, 18:42:34 .: Value type not matching: != Array ==================================================

There's somewhere a data type that doesn't match (or is corrupted), or so it seems.
Should mean an issue while loading config / binarization, however that's a little odd if that happens mid-game, as opposed to at the start when the mission is loaded etc.

Updated by Xeno about 5 years ago

This one is also nice (server rpt on an ACE server, running one of the latest beta builds):

2010/04/26, 21:52:20 File description.ext, line 8: '/Weapons/RPG7V.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 9: '/Weapons/Binocular.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 10: '/Weapons/Huntingrifle.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 14: '/Magazines/5x_22_LR_17_HMR.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 15: '/Magazines/PipeBomb.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 16: '/Magazines/Mine.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 17: '/Magazines/PG7V.count': Missing ';' prior '}'
2010/04/26, 21:52:20 File description.ext, line 8: '/Weapons/RPG7V.count': Missing ';' prior '}'

The mission running had no weapons or magazines defined in description.ext

But, that was not the problem... It showed up in the server rpt about an hour or so after mission start.

Updated by Sickboy about 5 years ago

Maybe their pre-fetch cache has gone haywire :D

Updated by Xeno about 5 years ago

They simply removed Namalsk and Isla Duala and everything was fine again :P

Like allready posted some posts above :P

Updated by Peyoteros about 5 years ago

Can the crashings be caused by BattleEye (BE)? There was a moment when we (my clanmates) uninsalled BE from our server and crashing stopped...

Updated by Sickboy about 5 years ago

Might be on to something, 6th server hasnt been crashing past months (though low player amounts), but also doesnt run BattlEye.

Updated by Peyoteros about 5 years ago

So, have we done any tests on BE yet?
Another thing which is difficult to explain is that on some days server almost doesn't crash at all, on some crashes every ~90 minutes. Mistery.

Updated by Peyoteros about 5 years ago

Tested server with "equal mode required", worked stabile without crashing, we didn't manage to do a real "stress" test because people didn't figured out how to join the server, so it was just two of us, maybe some day we will do the test with clanmates (7-8). If you guys will do the test, you could anounce in advance so we come too and test it out. One thing for sure, something needs to be done ASAP. We did test without BE (disabled*)- server crashed...

Updated by Sickboy about 5 years ago

Just wondering, are crashes known to be happening while or just after a player is joining, or is there no relation?

Updated by Xeno about 5 years ago

Looks like some other addon is the cause, client side addon, version mismatch ?!?, whatever...

I've played for some hours on a server this weekend (more than 10 people) and no server crash (verify signatures).

Friday evening a german server hosted an ACE event with allmost 50 players, no server crash. Saturday evening TG played with 50 players, no server crash, same for ST (allmost 50 players). Yesterday a PvP server played an ACE mission with 40+ players, no server crash.

Updated by rocko about 5 years ago

  • Due date set to 05/28/2010

Updated by Sickboy about 5 years ago

k, as non ace servers are said to be crashing when ACE users join, I was wondering about the reverse situation.
Signature checking won't aid in that area though, only having ace_ addons in mission.sqm addons[] and addonsAuto[] arrays will.

I was thinking kicking users without ACE as part of the ace_fnc_mismatch as opposed to throwing them to debriefing, could aid,
however this is too late. I think the only way to prevent unexpected issues is to completely block non ACE users from going past the assignment screen, which only can happen when ace is required in the mission.sqm.
I'm unsure about ability to join when ACE_ units are used, either for playable or non playable units.

Xeno, was battle-eye enabled on mentioned servers?

Updated by Xeno about 5 years ago

Sickboy wrote:

Xeno, was battle-eye enabled on mentioned servers?

At least on the server where I played, dunno about the others.

Updated by kju about 5 years ago

AToW ran fine with 40 players. BE disabled. Sig check active.

Updated by Xeno about 5 years ago

SB, how about creating an ACE specific object on the server when a mission starts ?
Something like ACE_EmptyH or whatever. But I'm in the same boat, not sure if it will block users without ACE.

Updated by Sickboy about 5 years ago

Afaik it will just give them a message a bout missing object. Only when the object is listed in mission.sqm will there be an early warning (altough I believe it only ejects when missing addons from the addon arrays).

Updated by kju about 5 years ago

A&S ProMode shows the player an info screen if he joined without the required addons.
He can only disconnect from there.

It adds a few missing xxx classes in the rpt, but that's it.

You need to avoid that this unit creates classes that he does not know/have though.
Otherwise everyone else will get on screen error messages.

Updated by Pain about 5 years ago

This is a big problem, we tryed with both BE on and off no diff, dont think we managed to do a complete mission once (peyo claims he did one or two)
feels like there is no point of keeping server running cos' it will crash 100% every time :(
(talking bout cure server)

Updated by Sickboy about 5 years ago

Pain wrote:

This is a big problem, we tryed with both BE on and off no diff, dont think we managed to do a complete mission once (peyo claims he did one or two)
feels like there is no point of keeping server running cos' it will crash 100% every time :(
(talking bout cure server)

What about signature checking. Was this enabled?
How about the RPT files?

_

kju wrote:

A&S ProMode shows the player an info screen if he joined without the required addons.
He can only disconnect from there.

It adds a few missing xxx classes in the rpt, but that's it.

You need to avoid that this unit creates classes that he does not know/have though.
Otherwise everyone else will get on screen error messages.

Thanks but useless. The damage is already done at that moment. You need to be unable to go past assignment screen. The only way I know is by mission.sqm addons and addonsAuto.

Updated by Lt.Chris about 5 years ago

We've had no crashes on our last test which included the islands so this sunday we have 1 more pack of addons we use which we are going to enable and see what happens. My personal bet however was it could have been the GLT misslebox addon weve removed recently. However this is just speculation so we will see.

Updated by Xeno about 5 years ago

We have another crash report, and those guys use the GLT missilebox too.

#10725

Check the rpt file in the ticket...

CA_CruiseMissile in ca\air2\cruisemissile\, GLT_Missilebox in glt_missilebox\

Updated by Lt.Chris about 5 years ago

seems quite convincing that it is then. A possible "ACE BlackList" might be in order for addons which are known to crash with ace?

Updated by Sickboy about 5 years ago

Very nice job guys, thanks all involved!

A possible "ACE BlackList" might be in order for addons which are known to crash with ace?

Well I think it'll need to be proven first if it is just in combination with ACE.
Once that is known, hopefully the creator, or us can resolve the issue.
If all that is not possible, I guess then it could be added to a blacklist.
We're maintaining a list here atm; http://ace.dev-heaven.net/wagn/Compatible_mod

Updated by Lt.Chris about 5 years ago

You will get the 100% confirmation you need with our test on sunday :)

Updated by Sickboy almost 6 years ago

tnx, looking forward

Updated by Lt.Chris almost 5 years ago

Ok we ran out test and it ran fine with our remianing GOL addon pack, so that confirms it was GLT this entire time. So basicly you can say 210% certinity that GLT is a killer in combination with ACE Mod

Updated by Sickboy almost 5 years ago

  • Status changed from Assigned to Feedback
  • % Done changed from 0 to 80

Lt.Chris wrote:

Ok we ran out test and it ran fine with our remianing GOL addon pack, so that confirms it was GLT this entire time. So basicly you can say 210% certinity that GLT is a killer in combination with ACE Mod

Thank you very much for these results. Were you able to find out if GLT by itself caused these issues aswell?
I'll at least download and take a look if I can find anything obvious why it would collide with ACE.

This is btw the danger of adding mods to your mix without sufficient research/confirmation if it is stable.
This is only logical of course, but is one of the main reasons the support details are required, and why it is crucial to test the mix :)

Updated by Lt.Chris almost 5 years ago

GLT was in the mix of addons for quite a while a long time ago and it was fine before ACE came out so based on that, I think its not another addon in our addons pack in which is fusing with GLT.

You can see what addons are in the mix if you wish by downloading them from our website :)

www.gol-clan.net and the big download button on the front page

Updated by Sickboy almost 5 years ago

Cheers! No DL link for GLT though. Grabbed it from: http://www.filefront.com/13933807/GLT_Missilebox.7z
There's nothing in the addon that explains the crashing behaviour you guys had, at least not in relation to ACE.
There are no classes modified, only classes of weapons/magazines/ammo added. Were you guys using the items that GLT provides, in one of your missions? What about EASA?
Will investigate more.

Updated by Lt.Chris almost 5 years ago

From what we were seeing you didnt even need to be actally using it in a mission (strange as that sounds) for it to crash. It only needed to be running with the addons for it to do its crash and the rpt did report the GLT missle box (like the ones I and others have submitted) before it crashes.

Updated by FunkDooBiesT almost 5 years ago

Right's been some time since i've been here, thanks for all the responce to this matter. For the time being the server is not running Battleye.

When I get some time with the server with a few players to test it with mods changed i'll provide some findings.

Personaly I believe this to be related to some of the posts above but i'll need to work through em to confirm.

Thanks.

Updated by Sickboy almost 5 years ago

NP, and thank you all very much for your feedback and work/time in these matters!

Updated by rocko almost 5 years ago

  • Status changed from Feedback to Closed
  • Target version deleted (395)
  • % Done changed from 80 to 100

Also available in: Atom PDF