| 00:31.50 | *** join/#tomcat neobuaya (n=neobuaya@203.201.170.2) |
| 00:33.41 | neobuaya | tomcat 5.5.27,JRE 1.6,Windows Server 2003 |
| 00:34.02 | neobuaya | hi all |
| 00:34.10 | neobuaya | i have some question |
| 00:53.32 | *** part/#tomcat tmccrary (n=tmccrary@d14-69-192-41.try.wideopenwest.com) |
| 01:30.14 | *** join/#tomcat dvayanu (n=another@ppp-93-104-39-25.dynamic.mnet-online.de) |
| 02:51.05 | *** join/#tomcat drichards (n=dave@c-98-232-24-20.hsd1.wa.comcast.net) |
| 03:53.26 | *** join/#tomcat bijoo_webdev (n=bijoo_we@c-98-204-54-131.hsd1.dc.comcast.net) |
| 04:10.29 | *** join/#tomcat acidjnk (i=acid@pD950A799.dip0.t-ipconnect.de) |
| 04:42.30 | *** join/#tomcat bijoo_webdev (n=bijoo_we@c-98-204-54-131.hsd1.dc.comcast.net) |
| 05:33.10 | *** join/#tomcat Squiza (n=stuartr@tux.ihbi.qut.edu.au) |
| 06:03.00 | *** join/#tomcat tom17bombadil (n=tom@T3403.t.pppool.de) |
| 06:08.47 | Squiza | Hi all, |
| 06:08.48 | Squiza | versions: TC: 5.5.27; JVM: 1.5.0_16; OS: CentOS 5 |
| 06:08.48 | Squiza | I'm trying to find a way to restrict access to particular URLs on Tomcat, my thinking is that something equivalent to apache's <Location /somepathhere/ > would be ideal. Is there a valve that does this available for Tomcat? |
| 06:08.48 | Squiza | I'm trying to get more sensible and finer grained restrictions in place for a davenport server (to the point where I can list directories and shares that are valid). |
| 07:30.55 | *** join/#tomcat trifon (n=chatzill@79-100-38-123.btc-net.bg) |
| 07:59.53 | *** join/#tomcat vikumar (n=Vimal@nat/redhat-in/x-6bfe67c00ddc7dc4) |
| 08:03.42 | *** join/#tomcat kjkoster5489 (n=kjkoster@233-250-103-145.cust.glaslokaal.nl) |
| 08:57.28 | *** join/#tomcat bizboo (n=bizboo@meu77-1-82-241-107-111.fbx.proxad.net) |
| 09:19.32 | *** join/#tomcat dvayanu (n=another@host-62-245-224-138.customer.m-online.net) |
| 09:43.36 | *** join/#tomcat jfclere (n=jfclere@195.Red-83-33-133.dynamicIP.rima-tde.net) |
| 09:53.02 | *** part/#tomcat jo4 (i=joachimn@colargol.tihlde.org) |
| 10:11.31 | dvayanu | any ideas how to analyze jmap generated dumps? |
| 10:22.58 | kjkoster5489 | dvayanu: what are you trying to find in them? |
| 10:23.16 | dvayanu | memory leaks |
| 10:26.00 | kjkoster5489 | have you looked at jhat? |
| 10:27.02 | dvayanu | not yet |
| 10:28.25 | dvayanu | but it looks like its the right thing |
| 10:28.47 | dvayanu | thx |
| 10:28.57 | dvayanu | now the question is whether it can handle the size... |
| 10:29.11 | dvayanu | or whether it can handle the 16 gb dump on my 4gb notebook |
| 10:32.07 | kjkoster5489 | No, it cannot. It needs gobs and gobs of RAM. |
| 10:32.32 | kjkoster5489 | Have you considered reducing the run-time memory of the stricken VM, just to get a smaller dump? |
| 10:34.21 | dvayanu | it probably won't even start with less than 8gb |
| 10:34.29 | dvayanu | well start yes, but run no |
| 10:35.13 | dvayanu | the problem is also, that its the old space thats filling, so it will take some time and the real traffic to fill |
| 10:35.26 | dvayanu | and no guarantee that the user behaviour is met by the last test |
| 10:35.28 | dvayanu | load test |
| 10:35.35 | kjkoster5489 | I see. |
| 10:37.28 | dvayanu | right now i have even problems to unpack it :-) |
| 11:11.07 | dvayanu | whistles daddy buy me a 32gb machine please |
| 11:19.54 | *** join/#tomcat nixblicker (n=nixblick@dslb-088-074-047-020.pools.arcor-ip.net) |
| 11:36.38 | *** join/#tomcat loddafnir (n=mike@chello080108111137.5.11.univie.teleweb.at) |
| 11:37.13 | *** join/#tomcat loddafnir (n=mike@chello080108111137.5.11.univie.teleweb.at) |
| 12:06.54 | kjkoster5489 | dvayanu: well, barring that 32GB machine, you have little option but to reduce the RAM on your machine for a while and see if you can get it to work. |
| 12:07.49 | kjkoster5489 | What you can do is find a tool that displays the number of cycles each object lives. I'm not sure what tools do that. |
| 12:17.02 | dvayanu | i would be happy by the simple count of objects |
| 12:17.46 | dvayanu | of each class |
| 12:18.02 | kjkoster5489 | jmap -histo? |
| 12:20.13 | dvayanu | in each space |
| 12:20.21 | dvayanu | or actually just in old space |
| 12:20.39 | dvayanu | i probably could provoke it earlier by reducing the size of the old space to 1 gig or less |
| 12:21.19 | dvayanu | but i have to deploy a new moskito version first to the production system, so i can monitor the sizes of the mem pools |
| 12:21.37 | dvayanu | jhat -J-Xmx15G -J-Xms15G -refs false heapdump.web03.txt |
| 12:21.39 | dvayanu | Killed... |
| 12:22.01 | *** join/#tomcat funk (n=funkman@c-98-235-21-236.hsd1.pa.comcast.net) |
| 12:22.19 | dvayanu | hi tim |
| 12:30.40 | *** join/#tomcat bindaas (n=chatzill@59.96.161.46) |
| 13:07.30 | *** join/#tomcat apelW (n=metaltux@62.209.163.98) |
| 13:12.42 | *** join/#tomcat sdanderson (n=sdanders@c-98-204-16-99.hsd1.va.comcast.net) |
| 14:03.28 | *** join/#tomcat tanto (i=tanto@bipolar.obliviated.org) |
| 14:09.34 | dvayanu | i'm starting to doubt if it was such a good idea to use the paralleloldgc |
| 14:10.52 | kjkoster5489 | what makes you say that? |
| 14:13.50 | dvayanu | the results |
| 14:14.13 | dvayanu | with smaller old space the histogram remains constant |
| 14:14.18 | dvayanu | thus, the amount of ram uses |
| 14:14.20 | dvayanu | used |
| 14:14.46 | kjkoster5489 | So the leak does not occur with a smaller old space? |
| 14:15.52 | dvayanu | or didnt occur yet |
| 14:16.11 | dvayanu | but histogram is so ... limited, i mean i know now that i got one million hashmaps |
| 14:16.18 | dvayanu | but not where they come from |
| 14:16.45 | kjkoster5489 | Well, you code will create them somewhere. |
| 14:19.41 | dvayanu | not necessary |
| 14:19.43 | dvayanu | rl |
| 14:19.44 | dvayanu | y |
| 14:19.53 | dvayanu | i've seen a lot of tomcat buffers |
| 14:19.58 | dvayanu | in the histogram |
| 14:20.06 | *** join/#tomcat sdanderson (i=62ac1c43@gateway/web/ajax/mibbit.com/x-4bc42c361eca67f1) |
| 14:20.09 | kjkoster5489 | Hmm. |
| 14:20.21 | kjkoster5489 | Is this a reasonably recent version of Tomcat? |
| 14:20.33 | dvayanu | 5.5.20 |
| 14:20.39 | dvayanu | but it runs for a year without problem |
| 14:20.44 | kjkoster5489 | I'm quite sure Tomcat does not leak. |
| 14:20.50 | dvayanu | yes |
| 14:20.53 | dvayanu | <PROTECTED> |
| 14:20.54 | dvayanu | <PROTECTED> |
| 14:20.54 | dvayanu | <PROTECTED> |
| 14:20.57 | dvayanu | top 3 right now |
| 14:21.04 | dvayanu | :-) |
| 14:21.56 | dvayanu | <PROTECTED> |
| 14:22.03 | dvayanu | 111829 concurrent hashmaps |
| 14:22.14 | dvayanu | sounds a bit too many |
| 14:23.46 | kjkoster5489 | These are around number 60/70 in my (quiet) Tomcat. |
| 14:24.09 | dvayanu | well i'm performing it on a tomcat in production |
| 14:24.10 | kjkoster5489 | Does your applicaiton not create hashmaps, or store ones that Tomcat created? |
| 14:24.10 | dvayanu | with |
| 14:25.02 | dvayanu | 1152 users on it |
| 14:25.13 | kjkoster5489 | Do you clean out sessions properly? |
| 14:26.11 | dvayanu | yes |
| 14:26.19 | dvayanu | in some cases at least |
| 14:26.26 | dvayanu | but this is something tomcat should handle |
| 14:26.36 | kjkoster5489 | Welll... |
| 14:26.39 | dvayanu | but we don't use http session a lot, we have our own |
| 14:26.43 | dvayanu | which is not reused |
| 14:26.51 | dvayanu | hence "clean! |
| 14:26.54 | dvayanu | "clean! |
| 14:26.55 | kjkoster5489 | It is all to easy to have a shared hashmap somewhere. |
| 14:26.58 | dvayanu | "clean" |
| 14:27.09 | dvayanu | and the number of our sessions is reasonable |
| 14:27.11 | kjkoster5489 | You can disable Tomcat's sessions entirely. |
| 14:27.36 | dvayanu | well we do need them |
| 14:27.38 | dvayanu | for tags |
| 14:27.39 | dvayanu | and jsps |
| 14:27.41 | kjkoster5489 | If you are sure you don't need them, that is. |
| 14:27.44 | kjkoster5489 | Ah. |
| 14:27.55 | dvayanu | and we store the id of our session in tomcats session :-) |
| 14:28.11 | dvayanu | we have 4412 existing session objects for 1152 users, thats reasonable |
| 14:28.14 | kjkoster5489 | You could have a look at the code to see where these hashmaps are created. |
| 14:28.27 | kjkoster5489 | If you say so. |
| 14:28.29 | dvayanu | there will be probably 10.000 locations |
| 14:29.53 | dvayanu | most of them as local variables in methods |
| 14:30.38 | kjkoster5489 | Hmm. |
| 14:30.55 | kjkoster5489 | What shared state exists? |
| 14:31.00 | kjkoster5489 | Caches of maps? |
| 14:31.13 | dvayanu | caches yes |
| 14:31.22 | dvayanu | not caches of maps |
| 14:31.45 | dvayanu | search for new HashMap returned 200 hits |
| 14:33.00 | kjkoster5489 | Any of them that wrap into a concurrent hash map? |
| 14:34.39 | dvayanu | search for concurrenthashmap returned another 113 |
| 14:35.26 | dvayanu | 210536 hashmaps and 27175 concurrenthashmaps in the histogram |
| 14:35.28 | dvayanu | in one of them |
| 14:36.09 | dvayanu | hmm, between two of them normal hashmaps reduced, concurrent hashmaps increased factor 4 |
| 14:37.44 | kjkoster5489 | And the number of sessions? |
| 14:38.27 | kjkoster5489 | The problem with caches is that it causes objects to die in old space. |
| 14:38.48 | dvayanu | the number of sessions remained |
| 14:38.57 | kjkoster5489 | They survive for (say) 16 gc cycles and end up in old space. |
| 14:39.12 | dvayanu | we had some .... well very talented developers |
| 14:39.15 | kjkoster5489 | Cleaning them from eden is a lot cheaper. |
| 14:39.31 | dvayanu | who thought its a good idea to replace all hashmaps with concurrenthashmaps |
| 14:39.35 | dvayanu | to make them threadsafe |
| 14:39.37 | dvayanu | .. |
| 14:39.39 | dvayanu | geniuses |
| 14:40.39 | kjkoster5489 | Hmm. Your system is probably spending quite a bit of time in thread scheduling then. :-/ |
| 14:40.51 | kjkoster5489 | But that is not your current problem, is it? |
| 14:42.00 | dvayanu | no, my current problem is why my old space is running full |
| 14:42.10 | dvayanu | but not fast enough :-) |
| 14:42.53 | kjkoster5489 | hehe |
| 14:42.57 | dvayanu | # java.lang.OutOfMemoryError: requested -100663296 bytes for GrET in /BUILD_AREA/jdk6_05/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space? |
| 14:42.58 | dvayanu | # |
| 14:43.10 | kjkoster5489 | whoa |
| 14:43.12 | dvayanu | i hate this dump |
| 14:43.13 | dvayanu | :-) |
| 14:43.14 | kjkoster5489 | is that what you see? |
| 14:43.21 | kjkoster5489 | That is not a Java issue. |
| 14:43.27 | dvayanu | i finally found my 32 gb machine :-) |
| 14:43.42 | kjkoster5489 | Ah. |
| 14:43.50 | kjkoster5489 | Once you're done, can you ship it to me? |
| 14:43.52 | kjkoster5489 | Please? |
| 14:44.06 | kjkoster5489 | I'll ... make sure its clean. |
| 14:44.31 | dvayanu | this is our cold standby |
| 14:44.35 | dvayanu | jhat -J-Xmx28G -J-Xms28G heapdump.web03.txt |
| 14:44.39 | dvayanu | and that was the command line |
| 14:47.36 | dvayanu | the number of AttributeWrapper (session attributes) remained the same |
| 14:47.41 | dvayanu | so its rather not a session issue |
| 14:50.16 | dvayanu | but the previous peak in memory corresponds with the increase in the users in the overall system |
| 14:50.41 | dvayanu | now the user count stays the same for some time and so the memory usage |
| 14:53.30 | kjkoster5489 | 28G |
| 14:54.02 | kjkoster5489 | I must be getting old. I remember soldering 2MB into my Atari ST. |
| 14:54.33 | kjkoster5489 | The question is: what keeps those maps around? |
| 14:54.57 | kjkoster5489 | How are they used once created? Are they cached? Are they shared across sessions? |
| 14:55.12 | kjkoster5489 | Are there any caches in there that do not actually help performance? |
| 14:55.50 | dvayanu | questions :-) |
| 14:56.04 | dvayanu | well, technically the answer to your last question would be "no" |
| 14:56.32 | dvayanu | but its a large team with a lot of fluctuation so it can happen that someone built something like this somewhere |
| 14:56.48 | dvayanu | thats why i'm so hot for this dump |
| 14:57.05 | dvayanu | and btw, my first programm had to fit into 64K memory of the former BK0010 computer |
| 14:57.08 | dvayanu | ;-) |
| 14:57.48 | kjkoster5489 | You win. |
| 14:57.59 | kjkoster5489 | I did build an adder from relays, though. |
| 14:58.14 | dvayanu | http://en.wikipedia.org/wiki/Elektronika_BK |
| 14:58.18 | dvayanu | ;-) |
| 14:58.40 | kjkoster5489 | That's a solid machine. |
| 14:58.52 | kjkoster5489 | I bet standing on the keyboard as no problem at all. |
| 14:59.02 | dvayanu | it just looks big |
| 14:59.06 | *** part/#tomcat rboyd (n=rmpnt@66.36.243.149) |
| 14:59.46 | kjkoster5489 | The question before my last still stands. |
| 14:59.56 | kjkoster5489 | Are you able to access jhat now or not? |
| 14:59.57 | dvayanu | are they shared? |
| 15:00.02 | kjkoster5489 | Yes. |
| 15:00.08 | dvayanu | i am starting it |
| 15:00.12 | dvayanu | waiting for crash |
| 15:00.15 | dvayanu | adjusting settings |
| 15:00.18 | dvayanu | starting again |
| 15:00.19 | kjkoster5489 | Or rather: what keeps them in memory. |
| 15:00.23 | kjkoster5489 | Yes. |
| 15:00.36 | dvayanu | either it gets not enough memory and quits with oome or gets killed by os |
| 15:00.45 | dvayanu | right now i'm trying with 20gb |
| 15:00.48 | dvayanu | back to the cashes |
| 15:00.50 | kjkoster5489 | You have to change some kernel tunables to be able to get that much memory. |
| 15:00.50 | dvayanu | caches |
| 15:01.51 | dvayanu | generally we have a 4-layer multitier architecture |
| 15:01.57 | dvayanu | each layer is caching |
| 15:02.07 | kjkoster5489 | How are these hash maps used? They are created locally to a method. What does the method do with it? Fill it, use it then leave it, or pass it to the next layer? |
| 15:02.09 | dvayanu | there are different types of caches |
| 15:02.12 | kjkoster5489 | (or previous one) |
| 15:02.32 | dvayanu | i haven't inspected all new XXmap calls yet |
| 15:02.37 | dvayanu | so i don't know |
| 15:02.46 | dvayanu | usually the caches are bound to one layer |
| 15:02.47 | dvayanu | or should be |
| 15:02.54 | dvayanu | not passed nowhere |
| 15:03.03 | kjkoster5489 | But they are shared across sessions? |
| 15:03.03 | dvayanu | used as proxies for services for example |
| 15:03.08 | dvayanu | yes they are |
| 15:03.12 | dvayanu | well no |
| 15:03.13 | dvayanu | :-) |
| 15:03.20 | kjkoster5489 | Euh. |
| 15:03.22 | dvayanu | they are not "shared" |
| 15:03.33 | dvayanu | but they are singletons |
| 15:03.37 | dvayanu | used by many users |
| 15:03.43 | dvayanu | but not bound to sessions |
| 15:03.47 | dvayanu | or userflow in generall |
| 15:03.50 | kjkoster5489 | How is that different from "shared" ? |
| 15:04.07 | dvayanu | shared is more like linked via common variables |
| 15:04.22 | dvayanu | like something in the session that points to another object |
| 15:04.38 | dvayanu | and that object is 'shared' if multiple sessions contains 'links' to it |
| 15:04.48 | dvayanu | here, we have a remote service, say user service |
| 15:05.00 | dvayanu | whenever i want to resolve a user by his id, i call this service |
| 15:05.03 | dvayanu | like getUser |
| 15:05.32 | dvayanu | now, i don't actually call the service, but a proxy, which has a local cache, and first lookups in the local cache |
| 15:05.41 | dvayanu | and calls the remote service only if the cache is empty |
| 15:05.46 | dvayanu | there are different types of caches |
| 15:05.48 | dvayanu | some are fixed |
| 15:05.52 | dvayanu | some are expiring |
| 15:06.05 | dvayanu | for example we store many data for logging it user for 5 minutes |
| 15:06.13 | dvayanu | logging in |
| 15:06.36 | dvayanu | because in first minutes he will be most visible to other users and his profile will be clicked more often than later |
| 15:06.54 | dvayanu | than we have some regular tasks perfomred in background |
| 15:07.00 | kjkoster5489 | I see. |
| 15:07.04 | dvayanu | which also use caches to store their results |
| 15:07.07 | kjkoster5489 | This all sounds quite sane. |
| 15:07.20 | dvayanu | or "cache"-like storages |
| 15:07.35 | kjkoster5489 | The expiring caches will have the effect of moving objects into old space. |
| 15:08.05 | dvayanu | yeah |
| 15:08.12 | kjkoster5489 | In 5 minutes you will have a dozen or so gc cycles. |
| 15:08.26 | dvayanu | but the gc should be able to remove them |
| 15:08.49 | dvayanu | in other words, we have the expiring caches since 4 years, but memory problems since september |
| 15:09.03 | kjkoster5489 | Does you application die with an OOE, or freeze because of gc latency? |
| 15:09.04 | dvayanu | so its more likely one of the last releases |
| 15:09.08 | dvayanu | freeze |
| 15:09.16 | dvayanu | it performs full gces |
| 15:09.18 | dvayanu | gc-es |
| 15:09.19 | kjkoster5489 | Yes. |
| 15:09.48 | kjkoster5489 | How much changed in September? You SVN can help you locate new or changed caches. |
| 15:10.08 | dvayanu | as i said, about 12 developers |
| 15:10.20 | dvayanu | one major release each month |
| 15:10.35 | kjkoster5489 | Also, you can try to increase the size of your eden spaces. That will have the effect of keeping more objects in eden. |
| 15:10.45 | dvayanu | actually no |
| 15:10.57 | dvayanu | we were running with 16 gb heap |
| 15:11.07 | dvayanu | 11 in eden |
| 15:11.12 | dvayanu | so it was 5 for old |
| 15:11.16 | dvayanu | than old run full |
| 15:11.21 | dvayanu | i changed it to 8 |
| 15:11.27 | dvayanu | now 8 run full |
| 15:11.30 | kjkoster5489 | 8 for eden? |
| 15:11.34 | dvayanu | yeah |
| 15:11.42 | dvayanu | well 16-8 = 8, so its 8 for both :-) |
| 15:11.47 | kjkoster5489 | I see. |
| 15:12.12 | dvayanu | i have picked one webserver now and set it to |
| 15:13.16 | dvayanu | -Xmn13000M |
| 15:13.25 | dvayanu | so i hope the old space will run full sooned |
| 15:13.26 | dvayanu | sooner |
| 15:14.24 | kjkoster5489 | So old space fills up and then your app server starts doing full GC's (and freezing) |
| 15:14.29 | dvayanu | yeah |
| 15:14.45 | dvayanu | tomcat, no app server |
| 15:15.12 | kjkoster5489 | Sure, it's just a name. |
| 15:15.33 | kjkoster5489 | JHat should allow you to see what is holding your maps. |
| 15:16.12 | dvayanu | just crashed again |
| 15:16.15 | dvayanu | with 20gb :-) |
| 15:16.22 | kjkoster5489 | *sigh* |
| 15:16.38 | kjkoster5489 | Well, you need something of a factor 8 more ram for jhat. |
| 15:16.48 | kjkoster5489 | Insane. |
| 15:16.52 | dvayanu | heh |
| 15:17.13 | dvayanu | well alternative would be to run yourkit |
| 15:17.25 | dvayanu | however i highly doubt they can really run on demand |
| 15:17.47 | dvayanu | last time i used jprobe the server barely handled 5 parallel users |
| 15:18.38 | kjkoster5489 | heh |
| 15:19.14 | kjkoster5489 | Well, I predict it will say that you are holding the objects in caches. :-) Hopefully it will be able to tell you which cache to purge earlier. |
| 15:20.03 | dvayanu | i did a great mistake |
| 15:20.27 | dvayanu | i was looking something up on another machine and made comparison histograms on another webserver |
| 15:20.49 | dvayanu | which makes them ... valueless :-9 |
| 15:21.08 | dvayanu | the funny part is that jmap can't connect to my tomcat on the "monitored" machine anymore |
| 15:21.09 | dvayanu | funny |
| 15:21.39 | kjkoster5489 | More like hmm-funny and not so much hihi-funny. |
| 15:22.27 | dvayanu | yeah |
| 15:22.40 | dvayanu | well noone will die |
| 15:22.42 | dvayanu | so its ok |
| 15:23.21 | dvayanu | i hope i will get a dump |
| 15:24.44 | dvayanu | with the new moskito version i would be able to at least make a heap-usage graph |
| 15:24.47 | dvayanu | http://moskito.anotheria.net/moskitodemo/mui/mskShowProducersBySubsystem?pSubsystem=builtin |
| 15:25.04 | dvayanu | so one could see whether increases are peaks or continuous |
| 15:26.43 | kjkoster5489 | You don't use a monitor yet? |
| 15:26.51 | kjkoster5489 | You could consider using Java-monitor. |
| 15:27.12 | kjkoster5489 | Here is a live demo http://java-monitor.com/forum/livedemo.php |
| 15:27.32 | dvayanu | nope |
| 15:27.36 | dvayanu | big company |
| 15:27.43 | dvayanu | security over security over security |
| 15:27.53 | kjkoster5489 | Ah. |
| 15:28.11 | kjkoster5489 | And then they wonder why you have trouble fixing memory issues. |
| 15:28.17 | kjkoster5489 | Same-old-same-old. |
| 15:30.32 | dvayanu | well we do have a lot of monitoring |
| 15:30.51 | dvayanu | but we hadn't have monitoring of memory on per pool level yet |
| 15:31.22 | kjkoster5489 | I see. |
| 15:31.29 | kjkoster5489 | What tool do you use? |
| 15:31.49 | dvayanu | for which part |
| 15:31.56 | kjkoster5489 | Monitoring. |
| 15:32.06 | dvayanu | again :-) |
| 15:32.40 | dvayanu | we have 3 different monitoring systems on application level, and nagios and big brother, big sister on os level |
| 15:32.51 | dvayanu | i think the later isn't used anymore |
| 15:32.56 | dvayanu | and we have mrtg |
| 15:33.15 | kjkoster5489 | I see. |
| 15:33.27 | dvayanu | but the most used things are moskito for developers and nagios for operations |
| 15:33.43 | kjkoster5489 | I wrote zapcat a while back, which is a bridge between JMX and Zabbix. |
| 15:34.06 | kjkoster5489 | You could do something similar for Nagios, I'm sure. |
| 15:34.08 | dvayanu | and we have a tool called frscontrol which presents most needed info on all webservers in one screen |
| 15:34.13 | dvayanu | we actually have |
| 15:34.56 | dvayanu | but again we bundle it all via moskito |
| 15:35.07 | dvayanu | nagios gets xml feed from moskito and graphs it |
| 15:35.53 | dvayanu | we also have a monitoring database where we can compare values of each method in each remote interface over 5m intervals in the last 5 years and stuff like that |
| 15:36.34 | kjkoster5489 | Sounds cool. |
| 15:36.50 | kjkoster5489 | So it's actually pretty easy for you guys to add a memory pool tracker. |
| 15:37.02 | kjkoster5489 | That should give you a lot of insight. |
| 15:37.13 | dvayanu | we have |
| 15:37.18 | dvayanu | justn ot released yet |
| 15:37.32 | dvayanu | next week or something |
| 15:38.03 | dvayanu | our monitoring is more concentrated on the application as on the vm |
| 15:38.23 | dvayanu | so we can detect pretty fast if a part of the application is getting slower |
| 15:38.29 | dvayanu | or produces more errors |
| 15:38.30 | dvayanu | or both |
| 15:38.56 | kjkoster5489 | You could perhaps add counters to the caches, so that you can keep track of cache fill levels and hit/miss ratio. |
| 15:39.06 | dvayanu | we do :-) |
| 15:39.09 | dvayanu | on 5 m base |
| 15:39.10 | kjkoster5489 | That should help identify caches that are not actually useful. |
| 15:39.30 | dvayanu | well we gather data for all 5m, 15m, 1h and 1d time intervals |
| 15:41.46 | dvayanu | we also monitor hashmaps... |
| 15:42.10 | dvayanu | well those where the developer hasn't forget to add the monitoring code to it |
| 15:49.07 | dvayanu | Unknown oop at 0x00002aabfdedd538 |
| 15:49.07 | dvayanu | Oop's klass is null |
| 15:49.20 | dvayanu | nice error message |
| 15:55.27 | *** join/#tomcat ajt (n=alex@h69-129-20-250.wyngmi.dedicated.static.tds.net) |
| 15:57.16 | dvayanu | great, not jhat refuses to read the dump |
| 15:57.28 | dvayanu | Read record type 12, length -1278693850 at position 0x621ae9 |
| 15:57.28 | dvayanu | java.io.IOException: Bad record length of -1278693850 at byte 0x621aee of file. |
| 15:58.51 | dvayanu | http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6614052 |
| 15:58.55 | dvayanu | great :-) |
| 16:01.06 | *** part/#tomcat funk (n=funkman@c-98-235-21-236.hsd1.pa.comcast.net) |
| 16:18.02 | *** join/#tomcat nixblicker (n=nixblick@dslb-088-074-045-091.pools.arcor-ip.net) |
| 16:45.46 | *** join/#tomcat bijoo_webdev (n=bijoo_we@c-98-204-54-131.hsd1.dc.comcast.net) |
| 16:58.26 | *** join/#tomcat Hellaenergy (n=Hellaene@66.187.203.82) |
| 17:06.36 | *** join/#tomcat trifon (n=chatzill@79-100-44-42.btc-net.bg) |
| 17:10.30 | *** part/#tomcat ajt (n=alex@h69-129-20-250.wyngmi.dedicated.static.tds.net) |
| 17:34.49 | *** join/#tomcat dsoto (n=Darryl@71.5.17.90.ptr.us.xo.net) |
| 17:49.53 | *** join/#tomcat jasonb (n=jasonb@75.144.23.117) |
| 17:54.48 | *** join/#tomcat dsoto (n=Darryl@71.5.21.113.ptr.us.xo.net) |
| 17:56.49 | *** join/#tomcat da_krowa (i=Miranda@chello089076201153.chello.pl) |
| 18:17.49 | *** join/#tomcat Disconnect (n=ndis@sigkill.net) |
| 18:19.52 | Disconnect | having issues with mod-jk 1.2.26 (specifically ubuntu 1.2.26-2).. mod-dirindex is turning /user/ into /user/FOO-index.htm, then rewrite turns it into /s/check/?user=user&file=FOO-index.htm.. jkMount picks that up fine as /s/ but then requests /user/ from tomcat (which has no idea and returns 404..) I found the original security issue that was to fix, but the jkoptions workaround isn't working. |
| 18:20.29 | Disconnect | (and the original bug wasn't a bug for us anyway - it involved users being able to spoof their way to a different worker, but we've only got the one) |
| 18:25.46 | Disconnect | http://tomcat.apache.org/security-jk.html lists the security issue |
| 18:31.33 | *** join/#tomcat SJr|Work (n=unknown@216.152.207.157) |
| 18:50.30 | Disconnect | random partly related note: if anyone has edit privs on that security page, its not "JkOption Forward.." its Options. the 's' is important. (and its quoted, as in "this exact string". so while it may be 'a jkoption', its quoted to be "this configuration line right here"..) |
| 18:59.31 | *** part/#tomcat bizboo (n=bizboo@meu77-1-82-241-107-111.fbx.proxad.net) |
| 19:00.59 | *** join/#tomcat dstotoWork (n=stoflet@nat/hp/x-f56d031d149cfba3) |
| 19:14.44 | *** join/#tomcat eyalco (n=nerd@77.125.165.201) |
| 19:16.34 | *** join/#tomcat gogators (n=Bill@66.83.158.30.nw.nuvox.net) |
| 19:18.38 | gogators | hi all....has anyone run into an issue with embedding tomcat (6.0.18/java 1.5.0/windows) where it will not stay running? Works perfect in tomcat 5.5.... |
| 19:24.07 | Hellaenergy | gogators, you might want to tell us what you see in the logs. |
| 19:26.01 | gogators | Thats the problem, there isnt any errors, it states the connector has started, then stops....had the same problem on tomcat 5.5, however the problem was resovled by using the default web.xml |
| 19:28.18 | Hellaenergy | Nothing in the event log? |
| 19:28.29 | Hellaenergy | How long does it take to stop? |
| 19:33.17 | gogators | nothing in the event logs....and it stops pretty much within a second of starting |
| 19:40.41 | gogators | without turning on all logging, the info messages i get are: |
| 19:40.42 | gogators | org.apache.catalina.startup.Embedded.start() - Starting tomcat server |
| 19:40.49 | gogators | org.apache.catalina.core.StandardEngine.start() - Starting Servlet Engine: Apache Tomcat/6.0.18 |
| 19:41.23 | Hellaenergy | Well you might want to turn on all logging then ;) |
| 19:42.24 | gogators | unfortunately i had already tried that as well, and from what i could see in it, everything was starting successfully..no errors, nothing missing.... |
| 19:42.50 | gogators | is there something different between 5.5 and 6.0 with regards to embedding....5.5 works perfectly... |
| 19:44.26 | Hellaenergy | Well here is the changelog: http://tomcat.apache.org/tomcat-6.0-doc/changelog.html |
| 19:46.20 | *** join/#tomcat yassine (n=yassine@unaffiliated/yassine) |
| 19:53.20 | gogators | thanks hellaenergy....there are a couple changes working with the embedded serer, but it doesnt look like anything that relates to what i am seeing |
| 20:01.40 | gogators | i guess the only thing left to do is dig through the source and see how things get started within the bootstrap.jar |
| 20:01.47 | gogators | thanks for your help! |
| 20:03.36 | Hellaenergy | gogators, also make sure there isn't already a copy running. |
| 20:03.42 | *** part/#tomcat gogators (n=Bill@66.83.158.30.nw.nuvox.net) |
| 20:39.28 | *** join/#tomcat dstotoWork (n=stoflet@nat/hp/x-a806e9dde8d6c5ff) |
| 20:51.40 | *** join/#tomcat dvayanu (n=another@ppp-93-104-39-25.dynamic.mnet-online.de) |
| 20:56.48 | dvayanu | re |
| 20:58.12 | *** join/#tomcat bindaas (n=chatzill@59.94.97.134) |
| 21:04.07 | *** join/#tomcat odin (n=dlm@host86-143-45-16.range86-143.btcentralplus.com) |
| 21:30.44 | *** join/#tomcat dstotoWork (n=stoflet@nat/hp/x-d035331ba4026e2a) |
| 22:10.19 | *** part/#tomcat Disconnect (n=ndis@sigkill.net) |
| 22:31.56 | *** join/#tomcat clajo04_ (n=clajo04_@cpe-67-243-136-111.nyc.res.rr.com) |
| 22:38.30 | *** part/#tomcat vinse (n=vinse_@208.253.223.146) |