What would that look like with prograde orbits? (a mess, I'd imagine, since apparently those aren't stable near the hill sphere!).  
 
I do like the spiral arms effect there...

An interesting thing about the objects that are right at the edge of stability, is that they can complete several orbits before being stripped away by the sun.  When they escape the planet, they always do so through the L1 or L2 points.  Then after orbiting the sun a few times, they get re-captured through the L1 or L2 point, complete a few orbits and then get ejected again.  It's fun to watch in the rotating frame as the positions of L1 and L2 are easy to spot.  Try this yourself:
 
File > New
Objects > Create -- Create a 1 Jupiter-mass object at SMA 1 AU.  Name it Jupiter.  Leave all inputs at their default.
 
Zoom out so you can see both objects.
View > Rotating Frame Adjustment.  Set it to the period of Jupiter.
Choose Rotating Frame.
 
Choose Jupiter as your focus object.
 
Create objects around it near the edge of stability:
 
Objects > Create Objects
With Jupiter set as the reference object, choose 0.31 AU as the SMA.  Change its color.  Leave all other inputs at their default.  Press Create.
Repeat, creating objects with sma's of 0.32, 0.33, 0.34, 0.35.  Play around with other values too.
 
Under Preferences, choose a graphics update interval of 50.
 
Set the time step to 512.
 
Sit back and watch as these objects escape Jupiter through the L1 and L2 points, orbit the Sun for a while, and get recaptured through the L1 and L2 points.

Quote from frankuitaalst on 09/11/08 at 16:14:43:

Because I know everything  
 
Or just from trying different things.  Here's a screen shot of a system with a planet of 10 Jupiter masses orbiting at 1 AU.  A test particle is randomly placed in an orbit around the planet with an SMA of 0.8 AU.  It orbits a bit, escapes through L1, gets captured again through L1, over and over again.  This is very chaotic.  Any small change will ensure that you will not get the same results twice.
 

Applying the input above as you mention Tony ( but instead of 1 Jovian - 10 Jovian masses ) at 1Au and 0.31 AU for the moon I got this amazing .gsim simulation .  
My first reaction was " what the hell..." .  
Thought first a had done something wrong . Simulation starts in rotating frame .  
Till now my question is open .... .  
I'll post an animation later on .

I've tried to reproduce the simulation above , but no succes . I can't understand what happened and why opening the same sim afterwards gives a normal situation ...
Nevertheless , I performed the simulation Tony suggested , but this time with te real solarsystem . Put 100 bodies around Jupiter at 0.283 AU and let run .  
Animation hereunder .  
( actually I was anxious to see how the inner solar system would be affected ) .  
One can see the bodies escape over the L1 , other over the L2 point .  
Note : I think the 0.283 AU setting was to much as the bodies escape rather quickly .  
 
Animation:
http://www.orbitsimulator.com/yabbfiles/Attachments/100HillObjFromJupiter.gif

For the solarsystem a Hill radius of 0.155Au to Jupiter seems to be a configuration at the edge of stabilty ( in this orbital configuration starting at 2037) .  
100 bodies were given 0.155 SMA around Jupiter  
The simulation covers almost 100 years and consists of 400 individual frames.  
One can see the system "wind up" for several tens of years , then some bodies escape to the inner solar system over the L1 point  one after another .  
Watch how Jupiter seems to shoot every time it is near  aphelion . Only a few escape over L2 , leaving their orbit towards the outer solar system .  
The animation appears to be of "poor" resolution or quality . This is due to the scaling and the reduction of colors and compression in order to fit the 1MB upload limit . The original amount of data before compression was about 1GB in bmp.  
 
Animation:
http://www.orbitsimulator.com/yabbfiles/Attachments/100HillBodies0p155AU.gif

Prior to the release of the moons by Saturn ("wind up ") the system evolves from its initial configuration at 0.185 AU from saturn as shown hereunder .  
Watch how the end of the animation a couple of moons escape at the left corner above over the L1 point .  
 
Animation:
http://www.orbitsimulator.com/gravity/images/users/frankuitaalst/Saturn0p185Deta il.gif

Quote from Tony on 09/13/08 at 12:04:01:

Well done so ...you're right , that was my point some time ago .  
Can we as users also add some command to hide the annexe ?  
 
BTW ; do you have any idea what was happening in the sim were there was a straight line in the rotating frame ?  
Seems so weird , looks as if SMA and velocity are coupled in a strange ...way . Body must orbit a spiralling ellipse but I don't understand the physics , or might it be some bug ?  
 
EDIT : if I try to open one of the attachments I don't get them . I get the message :" page not found" .