debug output for unknown segfault

hi paul,

thank you for your reply.

running js code with no extra added globals works perfectly, even under high concurrency. but using external objects from python breaks the execution.

mike from mongodb has constructed a testcase without using mongodb, just plain python that deadlocks:

http://mongodb.pastebin.com/m2ecdfebb

i know you are involved in couchDB, i'm sorry to mention mongodb here :)

thanks in advance,

cheers marc

marc,

That test case reproduces the bug here. I just ran it quickly this morning before work and its not pointing at anything blatantly in spidermonkey code which means this is gonna be a fun one to debug.

i know you are involved in couchDB, i'm sorry to mention mongodb here :)

lol, last I checked we aren't sworn enemies or anything. :)

Do you have a link to your web framework? I'm pretty interested in giving it a try.

hi paul,
good to hear that you are not enemies :) it's not really a webframework i'm building, just playing around with cherrypy and python-spidermonkey to get comfortable with serverside JS. here is a testcase that i've constructed for the mongodb guys, 'cause i've initially thought this was a bug in the pymongo driver.

test case: http://groups.google.com/group/mongodb-user/msg/5d58d2b138297d81?

i wanted to know if this bug is my fault or spidermonkey related, so i've tried the same thing with pyjscore and it deadlocks too. damn :)

i'm feel a bit stupid in just sending around testcases without having the ability to help you in debugging this code. but i'm very happy that you have "fun" with it :)

cheers marc

Marc,

Did a quick bit of reading. The first step I'm going to have to do is go through and wrap lots of code in calls to GIL protection clauses. This won't be the most trivial thing so it'll have to wait until the weekend. Might be a good time to do some refactoring though.

Paul

hi paul,

are there any news on the deadlock topic? is there anything i can provide you to help you with this issue (yeah i know, stupid question :)? is it possible to replace spidermonkey later with tracemonkey without rewriting the complete python-spidermonkey code?

thanks and cheers,

marc

marc,

Doh! I totally meant to get on earlier this week and leave an update on this. I started going through the code last weekend to figure out what I think I'll need to change. My guess is that somewhere the GIL is being released that I'm not expecting. My answer is to go through and add safe guards to all code that could possibly be called from a thread that doesn't have the GIL. Hopefully this is the issue.

Going through the different bits to put in the GIL stuff is going to take a bit as it reaches into the code a bit. In the mean time I've been contemplating how best to implement the whole thing. Sometimes I wish C had try/catch.

Anyway, that's the current status. I think I know what to do, its just a matter of finding time. I have an experiment to try on some other code this weekend, if that goes better than expected or blows up quickly I might get to it this weekend. If not hopefully either over a couple nights next week or next weekend.

hi paul,

that's great news. thanks for the quick update :)

cheers marc

Interesting development. I just happened to glance past the docs on building C++ extensions again. It caught my eye in that having try/catch/finally in the Spidermonkey code would be extremely helpful. I just sat down and put a try/catch in spammodule.c and built it as an extension and it works fine.

Looks like I'll be upgrading python-spidermonkey to (ab)use C++ instead of straight C. This should make quite a bit of the internals far simpler as well. I'll probably start making updates in the next couple days on a branch to see how it works out.

Hopefully I don't get struck down for abusing exception handling like this.

For anyone paying attention, I've started working on rewriting python-spidermonkey with C++. The code will be up at [1] as it appears. My initial prodding with the GIL state calls didn't give me any straight answers so in regards to this ticket it may be a bit of a haul.

Paul

[1] http://github.com/davisp/python-spidermonkey/tree/cxxrewrite

hi paul,

thanks for your effort. i've tested your newly submitted code on mac os x and unfortunately, there was no magic, it still locks...but i'm looking forward to your c++ implementation :) i've tried to figure out if this is a spidermonkey related problem, but other implementations like pyjscore (http://code.google.com/p/pyjscore/) have the same issue.

cheers marc

Marc,

Oh I haven't gotten around the underlying issue yet. But at least
you've verified that the new code builds and runs for you so there's
that. Once I get the code all translated to C++ then I'll look at how
to attack that threading issue. I'm still suprised it even exists
because I never touch the GIL myself. I tend to wonder if this isn't
some horrid sort of interaction between JS and Python that is just
gonna make me go bonkers.

Paul

Hi Paul,

i was a bit curious if Ruby in combination with Spidermonkey and MongoDB results in the same locking problem. Using Johnson (http://github.com/jbarnette/johnson/) and MongoDB for Ruby together with Rack (http://rack.rubyforge.org/) and Thin (http://code.macournoyer.com/thin/) works very impressive. No locking, no freezing.

Ok, Ruby has a completely other architecture behind it but maybe the Johnson code can help to find a solution for the GIL problem?

Cheers Marc

marc,

Sorry I haven't gotten a chance to follow up on this more thoroughly. Been a bit busy with other projects.

I'm not sure about Johnson and Ruby. I'd have to learn Ruby's threading model and I doubt that it'd end up being very intuitive. I actually had a friend familiar with the GIL look at this and when he ran it on windows it didn't fail. I fear that this is going to be a very obscure bug when I figure it out eventually.

I'm still waiting on Spidermonkey 1.8.1 to come out and hopefully as I push through that update I'll get to the bottom of this error. But I still keep thinking that I haven't the slightest on how to get to even start effectively diagnosing this. Eventually I'll get a free weekend to sink in and pick apart what's going on, but I don't see this being a fun journey either way.

Paul