Konichu Posted February 10, 2021 Posted February 10, 2021 Reference Since on another forum somebody said that my server has failed in less than two months due to a socket bug (that I caused trying to fix another socket bug), I'm going to release this. Version 5187 with features. Has many systems working properly and I was willing to learn async/await pattern with this but I probably failed. I'm opening the code and except for the thread lock issue the server is fine. Don't run it on a live server unless you fix the thread race issue. Handled packets :] public enum PacketType : ushort { MsgRegister = 1001, MsgTalk = 1004, MsgUserInfo = 1006, MsgItemInfo = 1008, MsgItem = 1009, MsgName = 1015, MsgWeather, MsgFriend = 1019, MsgInteract = 1022, MsgTeam = 1023, MsgAllot = 1024, MsgWeaponSkill = 1025, MsgTeamMember = 1026, MsgGemEmbed = 1027, MsgData = 1033, MsgTrade = 1056, // MsgAccount = 1051, MsgConnect = 1052, MsgConnectEx = 1055, MsgSynpOffer = 1058, MsgEncryptCode = 1059, MsgDutyMinContri = 1061, MsgAccount = 1086, MsgPCNum = 1100, MsgMapItem = 1101, MsgPackage = 1102, MsgMagicInfo = 1103, MsgFlushExp = 1104, MsgMagicEffect = 1105, MsgSyndicateAttributeInfo = 1106, MsgSyndicate = 1107, MsgItemInfoEx = 1108, MsgNpcInfoEx = 1109, MsgMapInfo = 1110, MsgMessageBoard = 1111, MsgSynMemberInfo = 1112, MsgTitle = 1130, MsgTaskStatus = 1134, MsgTaskDetailInfo = 1135, MsgFlower = 1150, MsgRank = 1151, MsgFamily = 1312, MsgFamilyOccupy = 1313, MsgNpcInfo = 2030, MsgNpc = 2031, MsgTaskDialog = 2032, MsgFriendInfo = 2033, MsgDataArray = 2036, MsgTrainingInfo = 2043, MsgTraining = 2044, MsgTradeBuddy = 2046, MsgTradeBuffyInfo = 2047, MsgEquipLock = 2048, MsgPigeon = 2050, MsgPigeonQuery = 2051, MsgPeerage = 2064, MsgGuide = 2065, MsgGuideInfo = 2066, MsgGuideContribute = 2067, MsgQuiz = 2068, MsgFactionRankInfo = 2101, MsgSynMemberList = 2102, MsgTotemPoleInfo = 2201, MsgWeaponsInfo = 2202, MsgTotemPole = 2203, MsgQualifyingInteractive = 2205, MsgQualifyingFightersList = 2206, MsgQualifyingRank = 2207, MsgQualifyingSeasonRankList = 2208, MsgQualifyingDetailInfo = 2209, MsgArenicScore = 2210, MsgWalk = 10005, MsgAction = 10010, MsgPlayer = 10014, MsgUserAttrib = 10017 } Server: https://gitlab.com/felipevendramini/comet Database: https://mega.nz/file/fVBnDYaC#B2jwHIjpRPFHqWZpmBj3A_1isV_F_oVHQRfZmYFrWUY Quote
Spirited Posted February 10, 2021 Posted February 10, 2021 Hey Konichu,It sounds like you had a rough time with the server... I got some feedback about that from others as well. For those not aware, Comet's packet processor is multithreaded, so you'll have to find a way to process those packets in a thread safe way; otherwise, you're going to hit deadlocks (server will freeze due to two threads waiting on each other). One recommendation I made was to process work in channels per map. Aka. process work single-threaded per map - treat it like a D&D campaign where all actions (game and player) are ordered and executed one at a time. Anyways, I'm sad to see you give up on this given how excited you were at the beginning. I hope you continue learning and developing here. I know I wouldn't be where I am today without multiple server failures where I had to scrap the whole project. It's a complicated space - a lot more complicated than it first appears. Keep your head up and keep at it.You're always welcome to ask questions here and get support as well.Best,Spirited Quote
darkfox Posted February 12, 2021 Posted February 12, 2021 Quote Reference Since on another forum somebody said that my server has failed in less than two months due to a socket bug (that I caused trying to fix another socket bug), I'm going to release this. Version 5187 with features. Has many systems working properly and I was willing to learn async/await pattern with this but I probably failed. I'm oppening the code and except for the thread lock issue the server is fine. Don't run it on a live server unless you fix the thread race issue. Handled packets :] public enum PacketType : ushort { MsgRegister = 1001, MsgTalk = 1004, MsgUserInfo = 1006, MsgItemInfo = 1008, MsgItem = 1009, MsgName = 1015, MsgWeather, MsgFriend = 1019, MsgInteract = 1022, MsgTeam = 1023, MsgAllot = 1024, MsgWeaponSkill = 1025, MsgTeamMember = 1026, MsgGemEmbed = 1027, MsgData = 1033, MsgTrade = 1056, // MsgAccount = 1051, MsgConnect = 1052, MsgConnectEx = 1055, MsgSynpOffer = 1058, MsgEncryptCode = 1059, MsgDutyMinContri = 1061, MsgAccount = 1086, MsgPCNum = 1100, MsgMapItem = 1101, MsgPackage = 1102, MsgMagicInfo = 1103, MsgFlushExp = 1104, MsgMagicEffect = 1105, MsgSyndicateAttributeInfo = 1106, MsgSyndicate = 1107, MsgItemInfoEx = 1108, MsgNpcInfoEx = 1109, MsgMapInfo = 1110, MsgMessageBoard = 1111, MsgSynMemberInfo = 1112, MsgTitle = 1130, MsgTaskStatus = 1134, MsgTaskDetailInfo = 1135, MsgFlower = 1150, MsgRank = 1151, MsgFamily = 1312, MsgFamilyOccupy = 1313, MsgNpcInfo = 2030, MsgNpc = 2031, MsgTaskDialog = 2032, MsgFriendInfo = 2033, MsgDataArray = 2036, MsgTrainingInfo = 2043, MsgTraining = 2044, MsgTradeBuddy = 2046, MsgTradeBuffyInfo = 2047, MsgEquipLock = 2048, MsgPigeon = 2050, MsgPigeonQuery = 2051, MsgPeerage = 2064, MsgGuide = 2065, MsgGuideInfo = 2066, MsgGuideContribute = 2067, MsgQuiz = 2068, MsgFactionRankInfo = 2101, MsgSynMemberList = 2102, MsgTotemPoleInfo = 2201, MsgWeaponsInfo = 2202, MsgTotemPole = 2203, MsgQualifyingInteractive = 2205, MsgQualifyingFightersList = 2206, MsgQualifyingRank = 2207, MsgQualifyingSeasonRankList = 2208, MsgQualifyingDetailInfo = 2209, MsgArenicScore = 2210, MsgWalk = 10005, MsgAction = 10010, MsgPlayer = 10014, MsgUserAttrib = 10017 } Server: https://gitlab.com/felipevendramini/comet Database: https://mega.nz/file/fVBnDYaC#B2jwHIjpRPFHqWZpmBj3A_1isV_F_oVHQRfZmYFrWUY I would like to help with that, can you give me more details about what the problem is and where is it in the code? Quote
Konichu Posted February 12, 2021 Author Posted February 12, 2021 Quote Quote Reference Since on another forum somebody said that my server has failed in less than two months due to a socket bug (that I caused trying to fix another socket bug), I'm going to release this. Version 5187 with features. Has many systems working properly and I was willing to learn async/await pattern with this but I probably failed. I'm oppening the code and except for the thread lock issue the server is fine. Don't run it on a live server unless you fix the thread race issue. Handled packets :] public enum PacketType : ushort { MsgRegister = 1001, MsgTalk = 1004, MsgUserInfo = 1006, MsgItemInfo = 1008, MsgItem = 1009, MsgName = 1015, MsgWeather, MsgFriend = 1019, MsgInteract = 1022, MsgTeam = 1023, MsgAllot = 1024, MsgWeaponSkill = 1025, MsgTeamMember = 1026, MsgGemEmbed = 1027, MsgData = 1033, MsgTrade = 1056, // MsgAccount = 1051, MsgConnect = 1052, MsgConnectEx = 1055, MsgSynpOffer = 1058, MsgEncryptCode = 1059, MsgDutyMinContri = 1061, MsgAccount = 1086, MsgPCNum = 1100, MsgMapItem = 1101, MsgPackage = 1102, MsgMagicInfo = 1103, MsgFlushExp = 1104, MsgMagicEffect = 1105, MsgSyndicateAttributeInfo = 1106, MsgSyndicate = 1107, MsgItemInfoEx = 1108, MsgNpcInfoEx = 1109, MsgMapInfo = 1110, MsgMessageBoard = 1111, MsgSynMemberInfo = 1112, MsgTitle = 1130, MsgTaskStatus = 1134, MsgTaskDetailInfo = 1135, MsgFlower = 1150, MsgRank = 1151, MsgFamily = 1312, MsgFamilyOccupy = 1313, MsgNpcInfo = 2030, MsgNpc = 2031, MsgTaskDialog = 2032, MsgFriendInfo = 2033, MsgDataArray = 2036, MsgTrainingInfo = 2043, MsgTraining = 2044, MsgTradeBuddy = 2046, MsgTradeBuffyInfo = 2047, MsgEquipLock = 2048, MsgPigeon = 2050, MsgPigeonQuery = 2051, MsgPeerage = 2064, MsgGuide = 2065, MsgGuideInfo = 2066, MsgGuideContribute = 2067, MsgQuiz = 2068, MsgFactionRankInfo = 2101, MsgSynMemberList = 2102, MsgTotemPoleInfo = 2201, MsgWeaponsInfo = 2202, MsgTotemPole = 2203, MsgQualifyingInteractive = 2205, MsgQualifyingFightersList = 2206, MsgQualifyingRank = 2207, MsgQualifyingSeasonRankList = 2208, MsgQualifyingDetailInfo = 2209, MsgArenicScore = 2210, MsgWalk = 10005, MsgAction = 10010, MsgPlayer = 10014, MsgUserAttrib = 10017 } Server: https://gitlab.com/felipevendramini/comet Database: https://mega.nz/file/fVBnDYaC#B2jwHIjpRPFHqWZpmBj3A_1isV_F_oVHQRfZmYFrWUY I would like to help with that, can you give me more details about what the problem is and where is it in the code? The problem is a deadlock. Where? Who know? Quote
darkfox Posted February 12, 2021 Posted February 12, 2021 Quote Quote Quote Reference Since on another forum somebody said that my server has failed in less than two months due to a socket bug (that I caused trying to fix another socket bug), I'm going to release this. Version 5187 with features. Has many systems working properly and I was willing to learn async/await pattern with this but I probably failed. I'm oppening the code and except for the thread lock issue the server is fine. Don't run it on a live server unless you fix the thread race issue. Handled packets :] public enum PacketType : ushort { MsgRegister = 1001, MsgTalk = 1004, MsgUserInfo = 1006, MsgItemInfo = 1008, MsgItem = 1009, MsgName = 1015, MsgWeather, MsgFriend = 1019, MsgInteract = 1022, MsgTeam = 1023, MsgAllot = 1024, MsgWeaponSkill = 1025, MsgTeamMember = 1026, MsgGemEmbed = 1027, MsgData = 1033, MsgTrade = 1056, // MsgAccount = 1051, MsgConnect = 1052, MsgConnectEx = 1055, MsgSynpOffer = 1058, MsgEncryptCode = 1059, MsgDutyMinContri = 1061, MsgAccount = 1086, MsgPCNum = 1100, MsgMapItem = 1101, MsgPackage = 1102, MsgMagicInfo = 1103, MsgFlushExp = 1104, MsgMagicEffect = 1105, MsgSyndicateAttributeInfo = 1106, MsgSyndicate = 1107, MsgItemInfoEx = 1108, MsgNpcInfoEx = 1109, MsgMapInfo = 1110, MsgMessageBoard = 1111, MsgSynMemberInfo = 1112, MsgTitle = 1130, MsgTaskStatus = 1134, MsgTaskDetailInfo = 1135, MsgFlower = 1150, MsgRank = 1151, MsgFamily = 1312, MsgFamilyOccupy = 1313, MsgNpcInfo = 2030, MsgNpc = 2031, MsgTaskDialog = 2032, MsgFriendInfo = 2033, MsgDataArray = 2036, MsgTrainingInfo = 2043, MsgTraining = 2044, MsgTradeBuddy = 2046, MsgTradeBuffyInfo = 2047, MsgEquipLock = 2048, MsgPigeon = 2050, MsgPigeonQuery = 2051, MsgPeerage = 2064, MsgGuide = 2065, MsgGuideInfo = 2066, MsgGuideContribute = 2067, MsgQuiz = 2068, MsgFactionRankInfo = 2101, MsgSynMemberList = 2102, MsgTotemPoleInfo = 2201, MsgWeaponsInfo = 2202, MsgTotemPole = 2203, MsgQualifyingInteractive = 2205, MsgQualifyingFightersList = 2206, MsgQualifyingRank = 2207, MsgQualifyingSeasonRankList = 2208, MsgQualifyingDetailInfo = 2209, MsgArenicScore = 2210, MsgWalk = 10005, MsgAction = 10010, MsgPlayer = 10014, MsgUserAttrib = 10017 } Server: https://gitlab.com/felipevendramini/comet Database: https://mega.nz/file/fVBnDYaC#B2jwHIjpRPFHqWZpmBj3A_1isV_F_oVHQRfZmYFrWUY I would like to help with that, can you give me more details about what the problem is and where is it in the code? The problem is a deadlock. Where? Who know? Ok I just hoped you could tell me when it happens at least, a little more information to help me find the problem. Quote
Spirited Posted February 12, 2021 Posted February 12, 2021 Ok I just hoped you could tell me when it happens at least, a little more information to help me find the problem.It looks like the server is pretty rampant with race conditions, which might be causing the server to crash or dead lock. I know some older versions of Comet hid those exceptions, but I'm not sure if that's the case here. Attacks happen in parallel with other player attacks and monster AI, which is probably the biggest culprit of instability. But also trading happens in parallel without locking, character updates and map updates happen in parallel without locking, etc. There're ways to re-design the server to be completely lockless, but the way it is now... you might be able to save it with excessive and careful locking. Though, it'd be better if you can use channels to limit the number of threads executing certain request types. Quote
darkfox Posted February 13, 2021 Posted February 13, 2021 Ok I just hoped you could tell me when it happens at least, a little more information to help me find the problem.It looks like the server is pretty rampant with race conditions, which might be causing the server to crash or dead lock. I know some older versions of Comet hid those exceptions, but I'm not sure if that's the case here. Attacks happen in parallel with other player attacks and monster AI, which is probably the biggest culprit of instability. But also trading happens in parallel without locking, character updates and map updates happen in parallel without locking, etc. There're ways to re-design the server to be completely lockless, but the way it is now... you might be able to save it with excessive and careful locking. Though, it'd be better if you can use channels to limit the number of threads executing certain request types.Thanks for the info, not that I know much about that subject but I'll see if I find something. Quote
Konichu Posted March 6, 2021 Author Posted March 6, 2021 The current checkin does have Clans and Clan war fully working even with the Family BP on team.Movement has been queued to run on map processing to avoid thread lock problems. (not tested yet tho) Quote
Spirited Posted March 6, 2021 Posted March 6, 2021 The current checkin does have Clans and Clan war fully working even with the Family BP on team.Movement has been queued to run on map processing to avoid thread lock problems. (not tested yet tho)Feel free to contribute to the wiki if you want as well. And wish you luck on those deadlocking issues! I know they're difficult to debug. Quote
freefora11 Posted March 23, 2021 Posted March 23, 2021 Hey Konichu,It sounds like you had a rough time with the server... I got some feedback about that from others as well. For those not aware, Comet's packet processor is multithreaded, so you'll have to find a way to process those packets in a thread safe way; otherwise, you're going to hit deadlocks (server will freeze due to two threads waiting on each other). One recommendation I made was to process work in channels per map. Aka. process work single-threaded per map - treat it like a D&D campaign where all actions (game and player) are ordered and executed one at a time. Anyways, I'm sad to see you give up on this given how excited you were at the beginning. I hope you continue learning and developing here. I know I wouldn't be where I am today without multiple server failures where I had to scrap the whole project. It's a complicated space - a lot more complicated than it first appears. Keep your head up and keep at it.You're always welcome to ask questions here and get support as well.Best,SpiritedI just started using your source (Spirited, not Felipes) and love how its using dot net 3.1 and async. I have the chat system working now, many more packets to go. Anyways I like your idea on this deadlock. So you are recommending that on each map(conquer map), that it has its own dedicated threads? If I were to put this into practice, I would have a thread pool for each map, when the packet comes in, it will have the map and send it to the correct thread pool? Quote
Spirited Posted March 23, 2021 Posted March 23, 2021 Hey Konichu,It sounds like you had a rough time with the server... I got some feedback about that from others as well. For those not aware, Comet's packet processor is multithreaded, so you'll have to find a way to process those packets in a thread safe way; otherwise, you're going to hit deadlocks (server will freeze due to two threads waiting on each other). One recommendation I made was to process work in channels per map. Aka. process work single-threaded per map - treat it like a D&D campaign where all actions (game and player) are ordered and executed one at a time. Anyways, I'm sad to see you give up on this given how excited you were at the beginning. I hope you continue learning and developing here. I know I wouldn't be where I am today without multiple server failures where I had to scrap the whole project. It's a complicated space - a lot more complicated than it first appears. Keep your head up and keep at it.You're always welcome to ask questions here and get support as well.Best,SpiritedI just started using your source (Spirited, not Felipes) and love how its using dot net 3.1 and async. I have the chat system working now, many more packets to go. Anyways I like your idea on this deadlock. So you are recommending that on each map(conquer map), that it has its own dedicated threads? If I were to put this into practice, I would have a thread pool for each map, when the packet comes in, it will have the map and send it to the correct thread pool?Not necessarily threads per map. Tasks per map. Tasks are like green threads (lightweight threads that can spin up more quickly with less overhead). Multiple tasks can run in a single operating system thread using a task scheduler for managing contexts and context switching. Anyways, summary being you can run more tasks more efficiently than running threads. You can also run multiple maps in a single task. Just some ideas. Quote
freefora11 Posted March 23, 2021 Posted March 23, 2021 Hey Konichu,It sounds like you had a rough time with the server... I got some feedback about that from others as well. For those not aware, Comet's packet processor is multithreaded, so you'll have to find a way to process those packets in a thread safe way; otherwise, you're going to hit deadlocks (server will freeze due to two threads waiting on each other). One recommendation I made was to process work in channels per map. Aka. process work single-threaded per map - treat it like a D&D campaign where all actions (game and player) are ordered and executed one at a time. Anyways, I'm sad to see you give up on this given how excited you were at the beginning. I hope you continue learning and developing here. I know I wouldn't be where I am today without multiple server failures where I had to scrap the whole project. It's a complicated space - a lot more complicated than it first appears. Keep your head up and keep at it.You're always welcome to ask questions here and get support as well.Best,SpiritedI just started using your source (Spirited, not Felipes) and love how its using dot net 3.1 and async. I have the chat system working now, many more packets to go. Anyways I like your idea on this deadlock. So you are recommending that on each map(conquer map), that it has its own dedicated threads? If I were to put this into practice, I would have a thread pool for each map, when the packet comes in, it will have the map and send it to the correct thread pool?Not necessarily threads per map. Tasks per map. Tasks are like green threads (lightweight threads that can spin up more quickly with less overhead). Multiple tasks can run in a single operating system thread using a task scheduler for managing contexts and context switching. Anyways, summary being you can run more tasks more efficiently than running threads. You can also run multiple maps in a single task. Just some ideas.Appreciate the clarification! I'll see what I can do and provide details of my solution once solved. Quote
Spirited Posted March 28, 2021 Posted March 28, 2021 I wrote up some of my thoughts on Comet's forks / multi-threaded design here:https://spirited.io/multi-threaded-game-server-design/ Quote
Konichu Posted May 13, 2021 Author Posted May 13, 2021 I've been reading articles about it, including yours and I'm back in business.I'm trying to recode carefully the way that the server handles its world. Quote
Spirited Posted May 13, 2021 Posted May 13, 2021 I've been reading articles about it, including yours and I'm back in business.I'm trying to recode carefully the way that the server handles its world.Awesome! I'm glad it could be of help, and I'm glad to see you're still involved with this. Quote
Flake Posted May 31, 2023 Posted May 31, 2023 Hello, currently the project continues with this deadlock bug? Quote
Spirited Posted June 5, 2023 Posted June 5, 2023 sad she is very goodYou can always fix it. The simplest way would be to add locks on character and monster interactions (assuming the deadlocks are from improper access to dictionaries and race conditions on character / monster info). You just have to make sure you don't introduce additional deadlocks caused by ordered locks (where two locks are waiting on each other to unlock, and thus never will). The more challenging (but probably more performant) way would be to separate requests into queues per map, and set up the background threads to also enqueue on those map queues. I outlined some of my thoughts on that here: https://spirited.io/multi-threaded-game-server-design/. Quote
Flake Posted June 5, 2023 Posted June 5, 2023 I don't understand English very well, and the translator sometimes translates some words that have a different meaning, and it confuses a lot, so I researched some things in pt/br and found it very interesting Quote
Baloony Posted December 21, 2023 Posted December 21, 2023 I've set it all up, acc and game servers connected, but when i try to login I get [test] has authenticated successfully on [Dark] but the client errors saying "Invalid Login", is this related to the socket problem? Quote
Baloony Posted December 24, 2023 Posted December 24, 2023 I have moved the account server logic from the oroginal comet base and i logged in successfully, i think this has lots of unused potential.@Spirited did you mean there should be no server processor and it should be preplaced with many processors for each map? Quote
Konichu Posted December 24, 2023 Author Posted December 24, 2023 I have moved the account server logic from the oroginal comet base and i logged in successfully, i think this has lots of unused potential.@Spirited did you mean there should be no server processor and it should be preplaced with many processors for each map?See Canyon, you can choose how many processors it will have and it will distribute maps over them.But it will always have one processor for:- login/logout (which should be exclusive for removal from role manager)- pvp maps (maps with pvp flags so a dead lock on mobs don't mess with events)- map processors... (may be from 1 to Environment.ProcessorCount) Quote
Baloony Posted January 28, 2024 Posted January 28, 2024 I think i've fixed the thread race, what I did is: I looked for all the places Canyon uses Kernel.Services.Processor.Queue( and I added that to the same places in Comet-5197. One thing i've not yet done is splitting ai in its own project, processor. Do you think that would be mandatory, or allocating the ai related tasks in the correct map would be enough? Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.