Adventures in SharedCache

Dec 3, 2009 at 9:39 AM

After many months of pain i think i have got the software to behave itself.  The main problem was OutOfMemoryExceptions.  This is caused because the CacheCleanup class's Purge method in the existing code base only ever removes the first object from the list so under load the cache fills up and falls over.  On a side note the way this method is called from the TCPServer class is not thread safe, another thread could access the method with an incorrect value of calculatedcachesize before the previous thread that did the purge gets chance to remove the returned list from the local cache.  This was not really an issue more of an irritation.

Now after i fixed this i got more strange behaviour the answer to which was simple in the end, The value for LocalCache.CalculatedCacheSize would grow overtime causing the Purge method (that now works) to slowly empty out the cache.  This is caused by the LocalCache.Add method incrementing the value for objects that already exist in the cache instead of updating the size by adding or removing the difference.

I have also fixed a fair few threading issue, the worst of which was that on start up if a node gets hit by a lot of requests the number of servers from the config is mis-reported because another thread has intitialised the list but not populated it yet.

I removed the customthreadpool because its not needed and blows up when it reaches its max thread limit ( do not know why).

I have implemented Multiinstance ( more than one instance per server).  The only reason i did this was because the hardware i originally had available was a meaty box, 12GB 8 processor, (32bit OS).  OS restriction of 2GB per process.

Replicated objects do not Expire

Lots of other little tweaks.  I think there is a lot more problems ( especially threading) to find but if there wasn't life would be a bore, may even write my own if i get time in between contracts.

I can let you have all my changes but they may take a little bit of effort to rework into the code base as i have been quite aggressive in certain areas.

 

Some live stats:-

Cluster One

DateTime NodeId NodeIndex Gets Sets Deletes Misses Hits Size Items Status TotalItems BytesRead BytesWritten CurrentConnections TotalConnections

03/12/2009 10:35:09 192.168.200.3:48888 0 128895 45554 22706 37999 136450 539313955 16206 OK -1 1506827746 3459900663 18 611

03/12/2009 10:35:09 192.168.200.3:48889 1 233274 47788 25137 38027 243035 522954911 16166 OK -1 1510185169 3441639707 19 262

03/12/2009 10:35:09 192.168.200.3:48890 2 194387 44608 22189 37494 201501 515908707 16145 OK -1 1457973126 2756910297 18 432

03/12/2009 10:35:09 192.168.200.3:48891 3 203214 46548 23759 37632 212130 509885041 15951 OK -1 1857442680 3143590709 16 566

03/12/2009 10:35:09 192.168.200.3:48892 4 118822 44827 22178 37334 126315 517879727 16029 OK -1 1460916320 2366952128 15 560

03/12/2009 10:35:09 192.168.200.3:48893 5 153216 45673 22987 39187 159702 533309867 16190 OK -1 1484610103 2876610061 17 287

03/12/2009 10:35:09 192.168.200.3:48894 6 121100 44269 22059 39372 125997 502299415 16027 OK -1 1431096041 2429080574 18 472

03/12/2009 10:35:09 192.168.200.3:48895 7 233070 45967 23445 37729 241308 515813010 16219 OK -1 1499108162 9370462222 17 256

03/12/2009 10:35:09 192.168.200.3:48896 8 172558 45152 22451 69039 148671 512360308 16376 OK -1 1421221238 2632461316 16 541

03/12/2009 10:35:09 192.168.200.3:48897 9 284111 46739 24367 37784 293066 520492171 16155 OK -1 1487148139 3536794061 18 289

03/12/2009 10:35:09 192.168.200.3:48898 10 152858 46194 23643 38011 161041 517773630 16190 OK -1 1485546179 2642180540 18 346

03/12/2009 10:35:09 192.168.200.3:48899 11 212330 45694 22928 38601 219423 526853467 16448 OK -1 1469439807 4836240438 17 174

Cluster Two

DateTime NodeId NodeIndex Gets Sets Deletes Misses Hits Size Items Status TotalItems BytesRead BytesWritten CurrentConnections TotalConnections

03/12/2009 10:35:09 10.224.6.4:48888 12 11936 13651 12913 9382 16205 12356917 537 OK -1 202329238 108765315 14 934

03/12/2009 10:35:09 10.224.6.5:48888 13 15578 13935 12980 9414 20099 11050279 613 OK -1 197571737 76991274 12 799

03/12/2009 10:35:09 10.224.6.6:48888 14 13209 15626 12731 9293 19542 29739901 608 OK -1 16649427640 16799967267 16 742

03/12/2009 10:35:09 10.224.6.7:48888 15 11823 13583 12767 9288 16118 10521975 558 OK -1 191831191 146681166 12 903

03/12/2009 10:35:09 10.224.6.8:48888 16 11387 13896 13102 9420 15863 11885477 530 OK -1 202298832 166235322 10 806

03/12/2009 10:35:09 10.224.6.9:48888 17 13801 14984 13185 9582 19203 20089467 549 OK -1 8488577145 8681820771 12 909

03/12/2009 10:35:09 10.224.6.10:48888 18 11434 13883 13107 9359 15958 22670973 516 OK -1 231093121 510652758 9 806

03/12/2009 10:35:09 10.224.6.11:48888 19 15008 14824 14035 9761 20071 14621212 531 OK -1 195945815 217062823 12 989

03/12/2009 10:35:09 10.224.6.12:48888 20 13214 13738 12938 9359 17593 10220106 538 OK -1 201831400 628365419 12 845

03/12/2009 10:35:09 10.224.6.13:48888 21 13246 13683 12802 9242 17687 16643230 573 OK -1 212397864 694505310 9 858

03/12/2009 10:35:09 10.224.6.14:48888 22 11471 13607 12719 9325 15753 15617072 613 OK -1 200973232 174868198 11 867

03/12/2009 10:35:09 10.224.6.20:48888 23 12303 13570 11845 9285 16588 16149182 500 OK -1 8715492312 8721919085 15 832

03/12/2009 10:35:09 10.224.6.4:48889 24 16909 19702 18916 10251 26360 10149121 510 OK -1 199303841 147091371 14 982

03/12/2009 10:35:09 10.224.6.5:48889 25 24225 13572 12665 9226 28571 18082433 647 OK -1 210900921 3902875100 10 1003

03/12/2009 10:35:09 10.224.6.6:48889 26 11799 13609 12747 9174 16234 14790008 586 OK -1 201131943 206734336 12 862

03/12/2009 10:35:09 10.224.6.7:48889 27 10602 12406 11724 9245 13763 14992721 525 OK -1 186835425 129335486 11 812

03/12/2009 10:35:09 10.224.6.8:48889 28 11557 13634 12846 9258 15933 13031173 550 OK -1 222703659 284653427 10 832

03/12/2009 10:35:09 10.224.6.9:48889 29 13189 14810 14094 9767 18232 12313082 533 OK -1 189273189 129535135 11 865

03/12/2009 10:35:09 10.224.6.10:48889 30 11136 13853 13002 9375 15614 16582302 544 OK -1 214816242 205427154 11 825

03/12/2009 10:35:09 10.224.6.11:48889 31 13057 13478 12656 9211 17324 15528644 582 OK -1 196591712 154206351 11 963

03/12/2009 10:35:09 10.224.6.12:48889 32 11916 13643 12823 9138 16421 16274111 540 OK -1 215908457 120875256 13 931

03/12/2009 10:35:09 10.224.6.13:48889 33 12402 13900 13006 9355 16947 15782872 603 OK -1 220302411 220401763 10 824

03/12/2009 10:35:09 10.224.6.14:48889 34 38647 13762 12904 9227 43182 16214663 588 OK -1 213909883 270169398 16 704

03/12/2009 10:35:09 10.224.6.20:48889 35 12329 12893 12187 9544 15678 8256113 491 OK -1 177508019 55591795 9 777

03/12/2009 10:35:09 10.224.6.4:48890 36 10959 12457 11828 9268 14148 15732514 466 OK -1 197549926 381788669 9 888

03/12/2009 10:35:09 10.224.6.5:48890 37 14169 13600 12702 9258 18511 14013018 625 OK -1 207282477 1011627516 12 971

03/12/2009 10:35:09 10.224.6.6:48890 38 10986 13578 12755 9169 15395 13685873 582 OK -1 202127112 127932159 10 794

03/12/2009 10:35:09 10.224.6.7:48890 39 17226 13609 12813 9274 21561 9697986 551 OK -1 199801830 1754682503 10 899

03/12/2009 10:35:09 10.224.6.8:48890 40 11071 13430 12667 9085 15416 14446154 527 OK -1 194695312 203690264 10 888

03/12/2009 10:35:09 10.224.6.9:48890 41 11702 13582 12790 9190 16094 18093904 554 OK -1 205755802 217519675 10 964

03/12/2009 10:35:09 10.224.6.10:48890 42 11231 13548 12785 9108 15671 13379565 508 OK -1 203757447 223167204 10 891

03/12/2009 10:35:09 10.224.6.11:48890 43 12929 13553 12765 9211 17271 10496506 542 OK -1 191121294 185888286 10 857

03/12/2009 10:35:09 10.224.6.12:48890 44 11846 13665 12895 9266 16245 11275092 547 OK -1 200206699 218848408 10 781

03/12/2009 10:35:09 10.224.6.13:48890 45 11510 13601 12665 9297 15814 12901192 633 OK -1 202932550 114292199 14 892

03/12/2009 10:35:09 10.224.6.14:48890 46 14789 13815 12930 9351 19253 14745562 602 OK -1 201952075 242911233 11 757

03/12/2009 10:35:09 10.224.6.20:48890 47 14899 17769 16822 10036 22632 13701179 569 OK -1 210045234 154026121 12 763

03/12/2009 10:35:09 10.224.6.4:48891 48 13447 15783 15005 9736 19494 17848282 557 OK -1 222939728 342317969 11 895

03/12/2009 10:35:09 10.224.6.5:48891 49 12988 13640 12788 9320 17308 16293522 594 OK -1 207990417 669661097 10 793

03/12/2009 10:35:09 10.224.6.6:48891 50 13985 13528 12716 9238 18275 17383966 585 OK -1 209967636 135577431 10 768

03/12/2009 10:35:09 10.224.6.7:48891 51 28189 14558 13747 9318 33429 11795237 555 OK -1 201814979 410640147 13 811

03/12/2009 10:35:09 10.224.6.8:48891 52 16602 18234 17291 9833 25003 17060803 581 OK -1 211138392 189173541 12 824

03/12/2009 10:35:09 10.224.6.9:48891 53 13618 13837 13118 9431 18024 11407395 541 OK -1 206944072 3362605628 9 784

03/12/2009 10:35:09 10.224.6.10:48891 54 14723 14321 13456 9599 19445 16104342 560 OK -1 225527076 128344013 12 912

03/12/2009 10:35:09 10.224.6.11:48891 55 26359 14469 13557 9390 31438 11659556 571 OK -1 194812682 1691838987 13 1086

03/12/2009 10:35:09 10.224.6.12:48891 56 17977 13531 12764 9309 22199 13064782 568 OK -1 204055689 889413325 10 961

03/12/2009 10:35:09 10.224.6.13:48891 57 14311 13676 12750 9315 18672 18135155 636 OK -1 211342450 440018097 12 820

03/12/2009 10:35:09 10.224.6.14:48891 58 11360 13631 12760 9165 15826 11425516 635 OK -1 194079364 226346638 11 876

03/12/2009 10:35:09 10.224.6.20:48891 59 11190 13421 12693 9134 15477 16566088 525 OK -1 216971907 292817331 12 903

03/12/2009 10:35:09 10.224.6.4:48892 60 11331 13809 12987 9561 15579 18660482 564 OK -1 258630895 272682912 12 826

03/12/2009 10:35:09 10.224.6.5:48892 61 10923 13483 12633 9198 15208 19551186 608 OK -1 213408199 276815308 9 834

03/12/2009 10:35:09 10.224.6.6:48892 62 11235 13391 12586 9122 15504 16445702 550 OK -1 195053155 153073830 11 732

03/12/2009 10:35:09 10.224.6.7:48892 63 12273 13840 13028 9408 16705 13321215 558 OK -1 190163705 113064361 11 891

03/12/2009 10:35:09 10.224.6.8:48892 64 12335 13638 12807 9333 16640 27672459 598 OK -1 230011210 275108091 12 1047

03/12/2009 10:35:09 10.224.6.9:48892 65 11495 13588 12790 9148 15935 14658697 537 OK -1 227663492 243586054 11 895

03/12/2009 10:35:09 10.224.6.10:48892 66 18685 13461 12676 9142 23004 10932613 529 OK -1 198430968 266720643 10 967

03/12/2009 10:35:09 10.224.6.11:48892 67 13807 13617 12790 9249 18175 19724782 563 OK -1 215835621 305809346 8 811

03/12/2009 10:35:09 10.224.6.12:48892 68 11886 13674 12807 9324 16236 16394624 571 OK -1 206581746 149699284 9 742

03/12/2009 10:35:09 10.224.6.13:48892 69 10694 13587 12741 9136 15145 12489806 599 OK -1 203221518 80777875 10 767

03/12/2009 10:35:09 10.224.6.14:48892 70 13472 13684 12786 9145 18011 14172335 621 OK -1 205646615 282388811 10 940

03/12/2009 10:35:09 10.224.6.20:48892 71 10617 13719 12865 9255 15081 20872214 540 OK -1 214272069 248611687 11 739

03/12/2009 10:35:09 10.224.6.4:48893 72 15648 13597 12834 9256 19989 18663687 510 OK -1 220697711 496640569 10 922

03/12/2009 10:35:09 10.224.6.5:48893 73 11398 13383 12589 9188 15593 13085925 564 OK -1 196975864 128573316 12 794

03/12/2009 10:35:09 10.224.6.6:48893 74 11170 13653 12761 9292 15531 12045789 613 OK -1 189119708 126931711 10 810

03/12/2009 10:35:09 10.224.6.7:48893 75 13512 13605 12763 9248 17869 14403332 582 OK -1 211275942 256008813 14 868

03/12/2009 10:35:09 10.224.6.8:48893 76 11057 13831 13002 9330 15558 24305008 559 OK -1 226343127 334453235 9 816

03/12/2009 10:35:09 10.224.6.9:48893 77 11536 13629 12862 9349 15816 18727251 565 OK -1 208852678 278444857 12 857

03/12/2009 10:35:09 10.224.6.10:48893 78 11858 13489 12743 9210 16137 13964309 520 OK -1 207642140 305280821 9 876

03/12/2009 10:35:09 10.224.6.11:48893 79 10984 13572 12772 9232 15324 9322600 577 OK -1 195574410 104600361 10 714

03/12/2009 10:35:09 10.224.6.12:48893 80 11685 13735 12904 9393 16027 11196755 545 OK -1 196261845 201683170 11 891

03/12/2009 10:35:09 10.224.6.13:48893 81 16040 18872 17967 9936 24976 12544792 597 OK -1 199335013 96665528 11 804

03/12/2009 10:35:09 10.224.6.14:48893 82 11399 13639 12701 9219 15819 17942382 571 OK -1 210991542 233260495 11 822

03/12/2009 10:35:09 10.224.6.20:48893 83 45881 15543 14660 9458 51966 16621247 558 OK -1 203097965 305139877 16 669

03/12/2009 10:35:09 10.224.6.4:48894 84 12011 14566 12818 9238 17339 21688665 520 OK -1 8722638645 8730891326 11 760

03/12/2009 10:35:09 10.224.6.5:48894 85 17549 13710 12878 9406 21853 18345970 605 OK -1 208619625 187675238 10 826

03/12/2009 10:35:09 10.224.6.6:48894 86 11374 13661 12724 9236 15799 19566055 647 OK -1 217764943 483572222 10 828

03/12/2009 10:35:09 10.224.6.7:48894 87 15619 13528 12701 9134 20013 12747153 558 OK -1 191441923 661611132 11 753

03/12/2009 10:35:09 10.224.6.8:48894 88 22418 14899 14045 9278 28039 17818302 532 OK -1 214421435 2037145636 10 973

03/12/2009 10:35:09 10.224.6.9:48894 89 10846 13514 12685 9089 15271 14096301 517 OK -1 213963088 311087495 9 857

03/12/2009 10:35:09 10.224.6.10:48894 90 11631 13917 13108 9468 16080 22193496 569 OK -1 248109192 809223227 12 871

03/12/2009 10:35:09 10.224.6.11:48894 91 12116 13727 12822 9220 16623 19535808 604 OK -1 245111540 744524615 11 920

03/12/2009 10:35:09 10.224.6.12:48894 92 11404 13615 12780 9240 15779 13014292 586 OK -1 198055513 231323573 9 847

03/12/2009 10:35:09 10.224.6.13:48894 93 11595 13502 12667 9169 15928 13040086 568 OK -1 218107316 484602517 10 963

03/12/2009 10:35:09 10.224.6.14:48894 94 10941 13455 12633 9157 15239 13734517 575 OK -1 197771028 155030837 9 797

03/12/2009 10:35:09 10.224.6.20:48894 95 48918 14864 14028 9392 54390 11330392 558 OK -1 205394839 178218699 13 670

03/12/2009 10:35:09 10.224.6.4:48895 96 17940 13624 12848 9262 22302 15517611 528 OK -1 217032947 1145237060 12 713

03/12/2009 10:35:09 10.224.6.5:48895 97 11197 13671 12843 9331 15537 9834070 614 OK -1 189561048 74018151 8 806

03/12/2009 10:35:09 10.224.6.6:48895 98 11399 13839 12990 9353 15885 17724185 645 OK -1 217184283 355308525 10 863

03/12/2009 10:35:09 10.224.6.7:48895 99 10965 13441 12624 9231 15175 10520405 583 OK -1 196455990 89548026 10 768

03/12/2009 10:35:09 10.224.6.8:48895 100 12176 13740 12953 9389 16527 12081025 530 OK -1 194719645 365609976 11 929

03/12/2009 10:35:09 10.224.6.9:48895 101 12166 13741 12940 9384 16523 10678674 515 OK -1 195602846 268818788 12 846

03/12/2009 10:35:09 10.224.6.10:48895 102 11097 13767 12970 9331 15533 12196471 552 OK -1 208637617 152112834 9 779

03/12/2009 10:35:09 10.224.6.11:48895 103 12286 14470 12634 9179 17577 26874924 572 OK -1 8520168092 8571794150 13 773

03/12/2009 10:35:09 10.224.6.12:48895 104 11166 12620 11908 9379 14407 14961274 514 OK -1 198852227 315143699 9 890

03/12/2009 10:35:09 10.224.6.13:48895 105 10805 13607 12783 9237 15175 16623523 601 OK -1 214828367 315892433 10 834

03/12/2009 10:35:09 10.224.6.14:48895 106 12486 13848 13003 9417 16917 15427586 593 OK -1 206059784 100185742 12 831

03/12/2009 10:35:09 10.224.6.20:48895 107 10947 13786 12980 9278 15455 14576291 552 OK -1 209480897 131189168 9 789

03/12/2009 10:35:09 10.224.6.36:48888 108 21431 14971 14106 9267 27135 14992256 553 OK -1 203500875 253076149 13 856

03/12/2009 10:35:09 10.224.6.37:48888 109 11326 14371 12601 8980 16717 9606378 547 OK -1 1459879047 1351321805 12 695

03/12/2009 10:35:09 10.224.6.38:48888 110 11236 12436 11764 9281 14391 16974188 504 OK -1 212018228 522910508 9 885

03/12/2009 10:35:09 10.224.6.39:48888 111 20842 13635 12745 9325 25152 21010476 594 OK -1 212281478 340566343 13 749

03/12/2009 10:35:09 10.224.6.40:48888 112 10614 13381 12597 9072 14923 11572452 559 OK -1 199436927 199293992 10 837

03/12/2009 10:35:09 10.224.6.41:48888 113 12847 13485 12706 9204 17128 14303124 551 OK -1 201262639 435504525 12 923

03/12/2009 10:35:09 10.224.6.42:48888 114 11531 13461 12699 9138 15854 12241151 551 OK -1 196101287 200998562 8 855

03/12/2009 10:35:09 10.224.6.43:48888 115 10523 13425 12669 9179 14769 9519104 518 OK -1 187435147 53238813 12 745

03/12/2009 10:35:09 10.224.6.44:48888 116 11354 13395 12462 9072 15677 20496923 630 OK -1 210989333 175029323 9 898

03/12/2009 10:35:09 10.224.6.45:48888 117 12013 12783 11922 9380 15416 21912654 606 OK -1 241870604 1027212163 10 904

03/12/2009 10:35:09 10.224.6.46:48888 118 10555 13388 12645 9225 14718 12350586 532 OK -1 202227713 79572769 11 712

03/12/2009 10:35:09 10.224.6.52:48888 119 12273 13635 12805 9205 16703 12898864 589 OK -1 191836664 150565365 11 951

03/12/2009 10:35:09 10.224.6.36:48889 120 10921 13589 12800 9112 15398 18395147 536 OK -1 215722359 196383280 9 833

03/12/2009 10:35:09 10.224.6.37:48889 121 16521 17955 17079 9805 24671 12483818 582 OK -1 213086592 217494557 15 893

03/12/2009 10:35:09 10.224.6.38:48889 122 11897 13883 13059 9355 16425 14156100 544 OK -1 204831802 306853583 9 881

03/12/2009 10:35:09 10.224.6.39:48889 123 12920 13580 12641 9155 17345 12560014 653 OK -1 199068768 185531743 10 927

03/12/2009 10:35:09 10.224.6.40:48889 124 11971 13590 12814 9199 16362 16746712 542 OK -1 209181479 206251421 10 913

03/12/2009 10:35:09 10.224.6.41:48889 125 16340 15233 14417 9753 21820 13179966 570 OK -1 198046367 70792911 10 753

03/12/2009 10:35:09 10.224.6.42:48889 126 12641 13449 12700 8996 17094 9218542 493 OK -1 186061564 343778153 10 877

03/12/2009 10:35:09 10.224.6.43:48889 127 10441 12409 11659 9079 13771 10966633 535 OK -1 172487127 50432667 10 868

03/12/2009 10:35:09 10.224.6.44:48889 128 10418 13278 12433 9051 14645 13162526 635 OK -1 195426352 67932831 11 775

03/12/2009 10:35:09 10.224.6.45:48889 129 11795 13549 12759 9237 16107 11588239 551 OK -1 205055365 611403041 10 747

03/12/2009 10:35:09 10.224.6.46:48889 130 18147 13733 12906 9211 22669 21217828 563 OK -1 224890879 777077785 10 687

03/12/2009 10:35:09 10.224.6.52:48889 131 12063 13574 12714 9149 16488 17465147 607 OK -1 204428171 285214968 11 868

03/12/2009 10:35:09 10.224.6.36:48890 132 15269 14023 13215 9462 19830 15018654 583 OK -1 211935325 338376164 10 759

03/12/2009 10:35:09 10.224.6.37:48890 133 10475 13399 12603 9092 14782 9288021 558 OK -1 199669716 106765582 8 745

03/12/2009 10:35:09 10.224.6.38:48890 134 14772 17503 16690 9856 22419 14288569 503 OK -1 197103234 590237714 10 837

03/12/2009 10:35:09 10.224.6.39:48890 135 59596 13338 12498 8963 63971 16592440 610 OK -1 218343280 6110636605 23 530

03/12/2009 10:35:09 10.224.6.40:48890 136 12560 13547 12754 9188 16919 14510531 602 OK -1 197739059 194378902 12 791

03/12/2009 10:35:09 10.224.6.41:48890 137 10570 13355 12500 9119 14806 15818199 594 OK -1 302448996 171858854 9 783

03/12/2009 10:35:09 10.224.6.42:48890 138 11616 13274 12474 8963 15927 12312275 562 OK -1 200233618 238995435 9 830

03/12/2009 10:35:09 10.224.6.43:48890 139 81610 15497 14628 9250 87857 17032997 591 OK -1 224605722 367041580 20 569

03/12/2009 10:35:09 10.224.6.44:48890 140 13034 13827 12997 9305 17556 15391000 563 OK -1 213433752 237814802 11 799

03/12/2009 10:35:09 10.224.6.45:48890 141 12044 13519 12693 9161 16402 17839572 595 OK -1 226467015 355186870 8 831

03/12/2009 10:35:09 10.224.6.46:48890 142 14765 14097 13292 9397 19465 14096318 559 OK -1 204085846 169120120 9 796

03/12/2009 10:35:09 10.224.6.52:48890 143 10971 12352 11657 9194 14129 11423972 502 OK -1 184921951 285445226 9 898

03/12/2009 10:35:09 10.224.6.36:48891 144 12390 13635 12859 9290 16735 12606485 572 OK -1 199984160 208124874 11 880

03/12/2009 10:35:09 10.224.6.37:48891 145 14218 15447 14705 9569 20096 15204271 544 OK -1 208024339 317911087 10 820

03/12/2009 10:35:09 10.224.6.38:48891 146 10639 13435 12680 9045 15029 9956782 530 OK -1 192575242 108394361 9 859

03/12/2009 10:35:09 10.224.6.39:48891 147 10591 13216 12312 8857 14950 12127964 647 OK -1 196216835 126301287 11 840

03/12/2009 10:35:09 10.224.6.40:48891 148 19196 13743 12931 9414 23525 23304956 555 OK -1 236023244 2057842664 11 975

03/12/2009 10:35:09 10.224.6.41:48891 149 27959 14812 14019 9296 33475 18396152 547 OK -1 209452993 540211100 14 882

03/12/2009 10:35:09 10.224.6.42:48891 150 11474 13434 12681 9107 15801 19984744 519 OK -1 227187341 313794046 9 804

03/12/2009 10:35:09 10.224.6.43:48891 151 11529 13550 12762 9191 15888 16126265 552 OK -1 219381548 363917411 9 850

03/12/2009 10:35:09 10.224.6.44:48891 152 20266 14376 12565 9016 25626 30036796 596 OK -1 8623675038 9523015626 10 857

03/12/2009 10:35:09 10.224.6.45:48891 153 11225 13590 12735 9134 15681 14902689 636 OK -1 229831894 453798888 13 846

03/12/2009 10:35:09 10.224.6.46:48891 154 10888 13649 12850 9175 15362 16333712 581 OK -1 202444452 137937291 8 762

03/12/2009 10:35:09 10.224.6.52:48891 155 12559 12519 11768 9251 15827 14130073 535 OK -1 188049728 229401663 10 943

03/12/2009 10:35:09 10.224.6.36:48892 156 10653 13300 12544 9021 14932 13521734 547 OK -1 199337416 566776910 9 786

03/12/2009 10:35:09 10.224.6.37:48892 157 13709 14423 12712 9088 19044 17801650 531 OK -1 8539054497 8622641750 11 760

03/12/2009 10:35:09 10.224.6.38:48892 158 38928 40991 38278 12095 67824 10162610 537 OK -1 214005986 117636003 15 723

03/12/2009 10:35:09 10.224.6.39:48892 159 11228 13508 12568 9021 15715 14910430 644 OK -1 202539051 183126139 16 831

03/12/2009 10:35:09 10.224.6.40:48892 160 14369 14201 13438 9595 18975 13256574 539 OK -1 200723247 74701637 11 686

03/12/2009 10:35:09 10.224.6.41:48892 161 13549 13708 12954 9231 18026 11834436 554 OK -1 191268801 105272631 8 843

03/12/2009 10:35:09 10.224.6.42:48892 162 10943 13475 12674 9067 15351 23708637 558 OK -1 219714366 126454951 11 833

03/12/2009 10:35:09 10.224.6.43:48892 163 14602 13927 13053 9362 19167 13913367 600 OK -1 221188005 1201049805 10 885

03/12/2009 10:35:09 10.224.6.44:48892 164 10399 13376 12562 9067 14708 17660699 601 OK -1 213021479 151277428 10 732

03/12/2009 10:35:09 10.224.6.45:48892 165 10682 13422 12562 9093 15011 13236833 623 OK -1 195167876 177621827 9 797

03/12/2009 10:35:09 10.224.6.46:48892 166 11880 13868 13101 9261 16487 13555355 550 OK -1 321686230 306700611 10 955

03/12/2009 10:35:09 10.224.6.52:48892 167 14677 17054 16215 9768 21963 16802103 550 OK -1 207313524 178362212 10 785

03/12/2009 10:35:09 10.224.6.36:48893 168 11290 13598 12773 9258 15630 15471726 602 OK -1 195586741 312940362 10 788

03/12/2009 10:35:09 10.224.6.37:48893 169 10853 13539 12735 9125 15267 18614007 587 OK -1 208791529 215370408 11 770

03/12/2009 10:35:09 10.224.6.38:48893 170 14248 13634 12859 9166 18716 11351462 528 OK -1 195463027 308746680 10 827

03/12/2009 10:35:09 10.224.6.39:48893 171 11103 13497 12561 8998 15602 16288108 624 OK -1 204425797 450747210 10 826

03/12/2009 10:35:09 10.224.6.40:48893 172 10873 13550 12761 8998 15425 13142413 547 OK -1 192199799 292288231 9 893

03/12/2009 10:35:09 10.224.6.41:48893 173 42201 16008 15166 9449 48760 12605684 555 OK -1 211181154 482124564 16 730

03/12/2009 10:35:09 10.224.6.42:48893 174 22925 23629 22698 10124 36430 12553335 567 OK -1 211440179 276073016 16 857

03/12/2009 10:35:09 10.224.6.43:48893 175 11429 13415 12629 9092 15752 21805077 570 OK -1 233464501 317478338 10 861

03/12/2009 10:35:09 10.224.6.44:48893 176 10950 13470 12651 9095 15325 12735542 579 OK -1 201810200 119631346 13 778

03/12/2009 10:35:09 10.224.6.45:48893 177 18262 13517 12636 9108 22671 11420269 598 OK -1 191910084 775421243 9 713

03/12/2009 10:35:09 10.224.6.46:48893 178 19740 16090 15335 9802 26028 14086677 528 OK -1 203549060 579378446 11 772

03/12/2009 10:35:09 10.224.6.52:48893 179 17594 13369 12546 9019 21944 11859701 566 OK -1 191602111 858580960 11 769

03/12/2009 10:35:09 10.224.6.36:48894 180 11065 13426 12674 9060 15431 15992494 555 OK -1 207503457 142593976 9 800

03/12/2009 10:35:09 10.224.6.37:48894 181 12059 13928 12550 9181 16806 19982771 573 OK -1 4416833987 4661695694 11 855

03/12/2009 10:35:09 10.224.6.38:48894 182 12373 13427 12663 9182 16618 9460144 526 OK -1 190012054 100901430 9 929

03/12/2009 10:35:09 10.224.6.39:48894 183 17491 13475 12570 8991 21975 18472420 635 OK -1 215807495 1637197465 9 780

03/12/2009 10:35:09 10.224.6.40:48894 184 26653 16194 15385 9429 33418 20016426 566 OK -1 212632496 177880644 11 888

03/12/2009 10:35:09 10.224.6.41:48894 185 11587 13425 12705 9060 15952 12899298 528 OK -1 197803520 175741700 12 803

03/12/2009 10:35:09 10.224.6.42:48894 186 10797 13504 12771 9181 15120 13885139 507 OK -1 244461568 445863676 9 788

03/12/2009 10:35:09 10.224.6.43:48894 187 24120 13525 12695 9174 28471 13014893 553 OK -1 201301501 2290745618 12 861

03/12/2009 10:35:09 10.224.6.44:48894 188 11078 13468 12509 9007 15539 25344352 625 OK -1 508592404 997215238 11 762

03/12/2009 10:35:09 10.224.6.45:48894 189 10365 13328 12539 8886 14807 15429456 564 OK -1 206536406 180446098 7 746

03/12/2009 10:35:09 10.224.6.46:48894 190 11809 13894 13023 9441 16262 15549059 591 OK -1 209302905 323008545 12 905

03/12/2009 10:35:09 10.224.6.52:48894 191 11231 13331 12532 8979 15583 14814362 604 OK -1 202262891 263009217 11 819

03/12/2009 10:35:09 10.224.6.36:48895 192 12090 13611 12761 9136 16565 14415896 554 OK -1 216060832 549061045 10 900

03/12/2009 10:35:09 10.224.6.37:48895 193 14494 13627 12834 9328 18793 11826103 572 OK -1 203476577 536078663 9 800

03/12/2009 10:35:09 10.224.6.38:48895 194 13091 13276 12545 8998 17369 12773007 530 OK -1 200775202 890324963 11 801

03/12/2009 10:35:09 10.224.6.39:48895 195 10511 12210 11389 9036 13685 16850101 617 OK -1 209665270 328209946 9 788

03/12/2009 10:35:09 10.224.6.40:48895 196 11913 13333 12514 8982 16264 9891896 551 OK -1 197113189 618894664 9 760

03/12/2009 10:35:09 10.224.6.41:48895 197 59282 13475 12678 9222 63535 17128102 541 OK -1 231750603 6363804339 23 564

03/12/2009 10:35:09 10.224.6.42:48895 198 10800 13529 12842 9191 15138 16201066 493 OK -1 207212837 249687470 11 832

03/12/2009 10:35:09 10.224.6.43:48895 199 12629 13468 12651 9173 16924 15476436 567 OK -1 205618916 478348282 9 879

03/12/2009 10:35:09 10.224.6.44:48895 200 11161 13106 12324 8790 15477 19449147 583 OK -1 209535341 248642350 10 798

03/12/2009 10:35:09 10.224.6.45:48895 201 10968 13354 12523 9141 15181 14965605 604 OK -1 213301091 169296854 8 782

03/12/2009 10:35:09 10.224.6.46:48895 202 15098 14092 13333 9440 19750 16290606 542 OK -1 214696511 310142862 8 732

03/12/2009 10:35:09 10.224.6.52:48895 203 37052 14498 13681 9151 42399 11081447 553 OK -1 208999723 139693110 14 788

 

 

Coordinator
Dec 3, 2009 at 11:20 PM

great news, can you send me please a copy or upload one?

regards, roni

Dec 4, 2009 at 10:31 AM

I sent a copy of my sandbox to your gmail account yesterday did you get it?

Coordinator
Dec 4, 2009 at 12:04 PM

note yet, even not in my junk email :-(

Dec 4, 2009 at 12:44 PM

Maybe the attachment was too large just resent it removing all binaries and PDB's

Coordinator
Dec 4, 2009 at 4:51 PM

nope nothing :-( can you upload it to some location and send me a download link?

 

Dec 7, 2009 at 9:30 AM

Try

http://www.mammothmicro.co.uk/v3.5.0.1.zip

j

From: ronischuetz [mailto:notifications@codeplex.com]
Sent: 04 December 2009 17:52
To: john@mammothmicro.co.uk
Subject: Re: Adventures in SharedCache [SharedCache:76959]

From: ronischuetz

nope nothing :-( can you upload it to some location and send me a download link?

Read the full discussion online.

To add a post to this discussion, reply to this email (SharedCache@discussions.codeplex.com)

To start a new discussion for this project, email SharedCache@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.426 / Virus Database: 270.14.96/2549 - Release Date: 12/06/09 19:37:00

Dec 7, 2009 at 11:24 PM

Can you tell me please, how do you set the app.config to be able to handle multiple instances of sharedcache ?

Thanks..

Dec 8, 2009 at 10:53 AM

If you are using the latest release of SharedCache then no.  I had to add multi-instance functionality into the software.  If you download my version above (follow link) or wait for it to be merged into the trunk (preferrable) if Roni decides its worthy.

So assuming you are using my version above.   The important things to note are the InstanceName in the config.  I apologize that it is two places but i did not have a lot of time to get this working the installutil.exe uses the value from the appsettings collection i planned for it to read from the ServerSettings section but could not get it working in the timeframe i had available.  There may be steps that you do not need

 

Instructions are as follows:

 SERVER Config:

For each instance of shared cache create a copy of the installation folder and suffix it with an apropriate value.  I used _2,_3,_4 etc. Then in each new folder open the SharedCache.WinService.Exe.config file in a text editor (notepad will do). Example file below.

The listening port number (ServiceCacheIpPort) needs updating so for _2=  48889,  _3 = 48890 etc

The log file output directories need changing from ...... logs\2\....... to .......logs\3\........

 

<?xml version="1.0" encoding="utf-8"?>

<configuration>

        <configSections>

               <section name="nlog" type="NLog.Config.ConfigSectionHandler, NLog" />

               <section name="replicatedSharedCache" type="        SharedCache.WinServiceCommon.Configuration.Server.IndexusServerProviderSection, &#xD;&#xA;                                                                         SharedCache.WinServiceCommon" />

        </configSections>

        <appSettings>

               <add key="ClientSettingsProvider.ServiceUri" value="" />

               <add key="ServiceName" value="SharedCache.Com_2" /><!-- This is the value that Instalutil.exe uses for the service name-->

        </appSettings>

        <replicatedSharedCache defaultProvider="ServerSharedCacheProvider">

               <serverSetting

                              ServiceCacheIpAddress="10.224.6.4"

                              ServiceCacheIpPort="48889"

                              SharedCacheVersionNumber="3.0.5.12"

                              LoggingEnable="1"

                              ServiceFamilyMode="0"

                              CacheAmountOfObjects="512"

                              CacheAmountFillFactorInPercentage="90"

                              ServiceCacheCleanup="LRU"

                              ServiceCacheCleanupThreadJob="60000"

                              TcpServerMaxThreadToSet="50"

                              TcpServerMinThreadToSet="12"

                              SocketPoolMinAvailableSize="5"

                              SocketPoolValidationInterval="00:02:00"

                              SocketPoolTimeout="00:02:00"

                              InstanceName="SharedCache.Com_2"

                       />

               <!--

                       SocketPoolMinAvailableSize="5"

                       SocketPoolValidationInterval="00:02:00"

                       SocketPoolTimeout="00:02:00"

               -->

               <replicatedServers>

                       <add key="GBBRF-HUAP07" ipaddress="10.224.6.42" port="48889" />

                       <!-- DO NOT DEFINE THE INSTANCE ITSELF !!! IT WILL BE AUTOMATICALLY REMOVED AT RUNTIME -->

               </replicatedServers>

               <providers>

                       <add name="ServerSharedCacheProvider" type="SharedCache.WinServiceCommon.Provider.Server.IndexusServerSharedCacheProvider, SharedCache.WinServiceCommon">

                       </add>

               </providers>

        </replicatedSharedCache>

        <nlog autoReload="true" throwExceptions="true">

               <targets async="true">

                       <target name="shared_cache_general" type="File" layout="${longdate}|${level:uppercase=true}|${aspnet-request:item=logSession}|${message}" filename="C:\temp\logs\2\server\${date:format=yyyy-MM-dd}_shared_cache_general_log.txt" />

                       <target name="shared_cache_traffic" type="File" layout="${longdate}|${level:uppercase=true}|${aspnet-request:item=logSession}|${message}" filename="C:\temp\logs\2\server\${date:format=yyyy-MM-dd}_shared_cache_traffic_log.txt" />

                       <target name="shared_cache_tracking" type="File" layout="${longdate}|${level:uppercase=true}|${aspnet-request:item=logSession}|${message}" filename="C:\temp\logs\2\server\${date:format=yyyy-MM-dd}_shared_cache_tracking_log.txt" />

                       <target name="shared_cache_sync" type="File" layout="${longdate}|${level:uppercase=true}|${aspnet-request:item=logSession}|${message}" filename="C:\temp\logs\2\server\${date:format=yyyy-MM-dd}_shared_cache_sync_log.txt" />

                       <target name="shared_cache_memory" type="File" layout="${longdate}|${level:uppercase=true}|${aspnet-request:item=logSession}|${message}" filename="C:\temp\logs\2\server\${date:format=yyyy-MM-dd}_shared_cache_memory_log.txt" />

               </targets>

               <rules>

                       <logger name="General" minlevel="Debug" writeTo="shared_cache_general" final="true" />

                       <logger name="Traffic" minlevel="Debug" writeTo="shared_cache_traffic" final="true" />

                       <logger name="Tracking" minlevel="Debug" writeTo="shared_cache_tracking" final="true" />

                       <logger name="Sync" minlevel="Debug" writeTo="shared_cache_sync" final="true" />

                       <logger name="Memory" minlevel="Debug" writeTo="shared_cache_memory" final="true" />

                       <logger name="*" minlevel="Debug" writeTo="shared_cache_general" />

                       <logger name="*" minlevel="Info" writeTo="shared_cache_general" />

               </rules>

        </nlog>

 

This is for one server repeat for all n instances.

  

Then run the Installutil.exe located in the C:\[Windows install folder]\Microsoft.NET\Framework\v2.0.50727 for each instance

 

Eg

c:\[Windows install folder]\Microsoft.NET\Framework\v2.0.50727\installutil.exe /i “C:\Program Files\SharedCache.com_5\SharedCache.Winservice.exe”

 

repeat for each instance 

 

You need to look at the out put from the app, it will tell you if the service install completed successfully.

 

You will then need to add the new nodes to the client configs, eg

 

 

<?xml version="1.0"?>

<!-- release   -->

 

<configuration>

        <configSections>

               <section name="nlog" type="NLog.Config.ConfigSectionHandler, NLog"/>

               <section name="indexusNetSharedCache" type="SharedCache.WinServiceCommon.Configuration.Client.IndexusProviderSection, SharedCache.WinServiceCommon"/>

        </configSections>

        <indexusNetSharedCache defaultProvider="IndexusSharedCacheProvider">

               <clientSetting

                              SharedCacheVersionNumber="3.0.5.12"

                              LoggingEnable="1"

                              CompressionEnabled="0"

                              CompressionMinSize="1024000"

                              HashingAlgorithm="Ketama"

                              SocketPoolMinAvailableSize="10"

                              SocketPoolValidationInterval="00:02:00"

                              SocketPoolTimeout="00:02:00"

                              />

               <!--

                       Hashing (default - is useless if your envrionment is using different versions of CLR's)

                       Ketama

                       FvnHash32

                       FvnHash64

                       -->

               <servers>

                       <add key="GBBIT-HUAP07" ipaddress="10.224.6.10" port="48888" />

                       <add key="GBBIT-HUAP08" ipaddress="10.224.6.10" port="48889" />

         

                       <!—ADD NEW SERVERS HERE à

               </servers>

 You can also uses different ip's on the same port as long as combination is unique you are good to go.

give each instance a restart/Start.

 

J

Dec 9, 2009 at 12:30 AM

Many thanks for your helpful response and code, I will be compiling and trying this version shortly, hope to see these changes integrated into the regular sharecache builds.

Dec 10, 2009 at 12:07 PM

My pleasure, my next challenge is to make the client do multigets asynchronously. looking at the code as it stands each get of a multiget happens sequentially which when you have a large cluster such as mine can hamper performance. Will keep you guys updated on my progress.

j

Jan 4, 2010 at 7:01 PM

Hi Fryedfish!

Since you've invested a lot in SharedCache, what do you advise - going for this product or select another (nCache, or ScaleOut)? I've read a post at Facebook - they use memCached, but there isn't a current (supported) win-client... So, I'm afraid to use that... Down-side using the other two mentioned is the price, but downside to using SharedCache is maturity... What is your opinion?

Thanks in advance! Lars

Coordinator
Jan 4, 2010 at 10:19 PM
Do u use 32 or 64 bit?


Sent from my iPhone

On Jan 4, 2010, at 21:01, "Dutchboy" <notifications@codeplex.com> wrote:

From: Dutchboy

Hi Fryedfish!

Since you've invested a lot in SharedCache, what do you advise - going for this product or select another (nCache, or ScaleOut)? I've read a post at Facebook - they use memCached, but there isn't a current (supported) win-client... So, I'm afraid to use that... Down-side using the other two mentioned is the price, but downside to using SharedCache is maturity... What is your opinion?

Thanks in advance! Lars

Jan 5, 2010 at 12:38 PM

Roni,

I use 32bit build, i looked into 64bit but did not have the hardware available.  Investigating a 64bit solution is on my agenda but up to yet i have found a greater number of smaller nodes works better than a lesser amount of larger nodes.  The primary reason for this is on node failures (yet to test and implement replication) losing 1/200th of my cache surface area is better then losing 1/20th.  A caveat to this which i mentioned earlier is that multigets are synchronous so a greater number of nodes increases the get time linearly.  I am looking at working a Begin***** and End**** async pattern into the client api but this is quite a big change.  Another area that would need looking into is ensuring that a 32bit client serializes the object being cached in the same way in which a 64bit server would deserialize.

Jan 5, 2010 at 3:02 PM
Dutchboy wrote:

Hi Fryedfish!

Since you've invested a lot in SharedCache, what do you advise - going for this product or select another (nCache, or ScaleOut)? I've read a post at Facebook - they use memCached, but there isn't a current (supported) win-client... So, I'm afraid to use that... Down-side using the other two mentioned is the price, but downside to using SharedCache is maturity... What is your opinion?

Thanks in advance! Lars

You can use memcache using memcached.  I have implemented a memcached solution as well as sharedcache for my existing client.  Sharedcache is still the prefferred because it supports longer key lengths and objects larger than 1mb which the memcached server does not.  I used enyim .net memcached client.  I had to implement compression ( enyim does not support compression) and md5 hash the key to get it below the 256k limit but we can switch between the two solutions easily taking into account that large objects will not be served from the cache if memcached is enabled. Large objects tend to be lists, so you can easily get around the large object issue with memcached by distributing the contents of the list accross the cache as single objects and then also storing a list of keys for each item in the list.

I still have occasional outofmemoryexceptions being thrown from sharedcache which is still down to a bug in the expire and purge areas of the code.  This will only effect your solution under heavy load and a when a node is full. The cleanup thread packs up so cache uses too much memory.  I control this by just setting the service to restart on failure.  This is far from ideal but manageble and i will find and fix the issue when given time to do so.

Without scaling out further i think i am pushing sharedcache to its limit currently, but thats the beauty of it, just add more nodes.  The same goes for memcached i read facebook had a caching farm of 400 nodes on pretty powerful hardware, i am up to 220, each node serving an average of 4 gets per second which equates to 3 million gets per hour.

If you are looking for a paid for solution  i know of a big operation that use oracle coherance, it costs a fortune but should do the trick and has a .net client.

Things i want to find answers to are:- what effect does the garbage collector have on performance, how much of the object data in memory is swapped out to virtual memory by the OS?

think i am rambling now so hope this helps.

 

j

 

Jan 9, 2010 at 11:03 PM
Edited Jan 9, 2010 at 11:25 PM

FryedFish,

Really nice work. I'm currently making my thesis on using distributed cached systems applied to a proprietary platform.

I'm investigating for solutions, so far i've seen memcached, citrusleaf, and others that are very specific.

I came across sharedCache and seems really nice, I'm just afraid there's no more activity since june 5 in the code. Mainly the new session management inclusion. What do you think about shared cache?

Regards,

Hugo

Jan 11, 2010 at 10:24 AM

Hugo,

Its doing the job, and offers far more functionality than memcached.  It needs some time and attention especially in the TCPServer class and cleanup areas.  When it is under heavy load issues still occur.

I have made many changes to the code since june 5th but i am not a contributor so my channges are sitting with Roni currently. I do not think the project is dead, i would like to see it progress.

 

j

Jan 11, 2010 at 10:14 PM
Edited Jan 12, 2010 at 3:47 PM

I'm actually quite satisfied with this project,

I have the following requirements:

- Use with windows (checked)

- To use this with sessions, my main problem is the fault tolerance (i think replicatedSharedCache does that). 

- To store any kind of objects, my main problem is the existence of locking mechanisms. I can simulate those.

- Easy to maintain.

 

So, my questions are:

- on the replicated shared cache, can i specify how many servers i want to use to replicate data? Is so, If a server goes down, how does the client know which node to ask for the replicated information?

- Is there any plan to include locking mechanisms into the project? both optimistic and pessimistic?

- Is the cluster able to handle incoming and departing nodes in runtime?

- Fryedfish, your code has been mostly bug fix or more features?

 

Thank you,

Hugo Veiga

 

EDIT:

Now i know that the replicated shared cache replicates data to all clusters. It's in the documentation, so don't mind it.

Jan 26, 2010 at 11:00 PM

Perhaps an option might be to fork your changes on someplace like github? (If ok with Roni)