Skip to content

Any Independent Benchmark of IMDG Solutions? (Coherence, Terracotta, WXS, Gigaspaces)

November 24, 2008

It’s hard to believe that Terracotta with its great simplicity is in fact 10x times faster than Oracle Coherence?! Idea’s matter but after all Oracle is a multibillion company and Terracotta Tech is much smaller!

The point is, most of these benchmarks are from their vendors and obviously in this way it’s easy to find and obtain test which outperform in the specific cases.

What I am eager to here is this domain, is a vendor independent benchmark, not those from Oracle, IBM or Terracotta themselves! Do you know any?!

Advertisements
11 Comments
  1. Randy May permalink

    I have a tool that would be useful in performing this type of comparison. It is a reasonably full-featured performance testing client that supports pluggable cache providers. I have only recently finished the work necessary to support this feature and the documentation may take another week or so. Watch my OpenSpaces project for an update: http://openspaces.org/display/PET/PerfTest

  2. Hello, there. My company is a highly specialized systems integrator with IMDG practice. We are partners with both GigaSpaces and Terracotta and we’ve been hired many times by the customer to conduct performance benchmarks between various IMDG products, usually for internal evaluation. Although we never had a head-to-head comparison between Terracotta and Coherence, I can speak about Coherence vs. GigaSpaces for example as usually showing parity on the majority of the performance benchmarks in typical enterprise scenarios. This is attributed to the fact that both are Java-based, heap-based IMDGs with similar performance characteristics and overheads. The size of Oracle corporation has very little to do with the performance of one of its products as it is the pysics and economics that tends to define the performance boundaries is this systems. Actually, to put this argument aside entirely you may remember that Cogerence was purchased by Oracle less than 2years ago when Terracotta was still around as an independent ISV.

    When it comes to performance though it cannot be studied in isolation from other factors, such as scale, load and reliability. Some IMGD shine in large clusters. Naturally, the “ilities” usually cost the architecture some overhead, often a considerable one. For example, memcache will blow all JVM-based technologies out of the water, inlcuding Terracotta, GigaSpaces and Coherence, but this hardly proves anything.

    Actually, Terracotta is a very interesting technology, but it is not an IMDG. A particular distinguishing characteristics of a moderm IMDG is data partitioning. Terracotta tends to shine for a different set of problems then Coherence, although there are definitely some application topologies where both solutions may apply and hense can be benchmarked for performance.

    The closing comment is that a fair benchmark between two IMDGs, especially those of radically different architectures, is best consucted in a context of a specific use case under a target load.

    Sincerely,

    Victoria Livschitz
    Founder and CEO, Grid Dynamics
    vlivschitz@griddynamics.com

  3. Wanted to point out that Terracotta did not generate this test. It was a customer’s real application. And this performance delta of 10X didn’t happen once but many times this year. I do agree with Victoria on many levels as well:

    1. Oracle’s size has nothing to do with it. Coherence was built by a small group of smart, self-funded people long before Oracle bought them (Tangosol)
    2. A test could easily be generated where Coherence would be faster but in all but 1 real customer use case this year, Terracotta was faster and by between 3X – 20X each time

    What’s going on at a technical level is that Terracotta is a runtime solution, like the garbage collector or JIT. We can re-order and optimize and app on the fly and Coherence is static. This means to me that, unless you use Coherence in an idealized manner, Terracotta will win and we will continue to get faster.

    –Ari

  4. Dear Randy, Victoria, Cameron, and Ari, first of all thanks for your kind comments. It’s a pleasure for me to have comments from your side.

    For me its very hard to comment about these two great technologies while the founder of both companies have commented here, but below is my limited understanding. Please correct me know if I am wrong..

    Well Ari, I myself is doing such a performance test these days and as I get more experienced with both Terracotta and Coherence I gradually understand where this difference in performances comes from.

    All you and Victoria mentioned about Terracotta is correct but the fact is Terracotta is not a real IMDG (by the way it means today) by itself while Coherence is.

    The way Terracotta integrates with ORM solutions such as Hibernate is to provide a larger heap for 2nd level cache solutions such as EhCache while Coherence fits just as a real DB and provides write-behind feature thought integration with BerkleyDB which for me this is a _very_ great feature and to sum it up I found that the reason behind Coherence’s lower performance compared to Terracotta is simply because it does more logic for my data! :)

    Yes, it’s true that Terracotta is also using BerkelyDB to provide a persistent memory solution but these two approaches are completely difference. e.g. Terracotta does not support a second main relational database and just has its embedded CacheStore based on DBD.

    Also please note that my understandings and comments of both of these tech does not mean that I prefer one to another in all cases. In situation where we only need a distributed clustered memory (such as Http session clustering) Terracotta is shining. But I guess I will go with Coherence for a write-behind when I work with DB and want to have ACID transactions running on database which is also used by other subsystems.

    For example I myself is evaluating Terracotta for an SMPP Gateway accounting system (with very high TPS), while I study the possibility of using Coherence in another financial domain.

    BTW I was not aware that these benchmarks are provides by 3rd party companies since nothing is mentioned in docs and as you see all of them have Terracotta’s header and footer :)

    Thanks again for your comments.

  5. Amin –

    Ari’s claims sound bogus because they are bogus. The truth is that Terracotta significantly complicates a Java application by replacing simple field (e.g. property) accesses with their own byte-mangled code (which can include network round-tripping), causing significant performance degradation across the board.

    While I am sure that is possible to contrive an example where Terracotta benchmarks faster than Coherence, the very idea seems rather pointless to begin with, since Terracotta does not provide any of the capabilities that Coherence provides, so to compare their performance implies that you are ignoring or disabling all of the features that customers select Coherence for.

    For a real-world use case, one of the banks using Coherence gave a presentation in which they quantified their end-of-day processing (running on commodity blades on a standard GigE network) as sustaining a data rate of over 1.5TB/s for close to an hour .. that’s over one and a half TERABYTES of data being processed every second! (And if a server fails, it has no impact on the execution or the result!) The parallel execution functionality that Coherence provides just doesn’t exists with Terracotta, so I’m not claiming that Coherence is faster (since Terracotta can’t compare); I’m just claiming that Coherence is a data grid (which Terracotta isn’t). Don’t get me wrong: Terracotta is cute and no doubt fun to play with, but it’s not even in the same market as Coherence.

    Peace,

    Cameron Purdy
    http://www.oracle.com/technology/products/coherence/index.html

  6. dude, that’s 15 THOUSAND network ports. Nice try.

  7. [This comment got deleted once before. I am hoping it won’t get deleted again.]

    Cameron,

    Speaking of bogus claims: 1.5TB/sec is 15,000 network ports at FULL speed. Interesting as I just spoke to an Oracle engineer who said your largest single cluster in deployment is 400 nodes. Besides, if you are running a cluster of 15K instances+, then they are likely sharing NOTHING and then you aren’t clustering as much as you are just serving as a networking protocol.

    Amin: Terracotta does not like to work as 2nd level cache underneath Hibernate. I suggest you check out: http://www.terracotta.org/web/display/orgsite/Web+App+Reference+Implementation

    We use what we call DETACHED mode–advocated by JBoss.

    –Ari

  8. Ari –

    Exactly. If you have to move data (i.e. if you have to “share memory”), you lose. There’s no way to move that kind of data around. To work with those volumes, you have to have the capability to partition it and process it in parallel. That’s what a data grid does. That’s why Terracotta can’t compete with Coherence.

    BTW – which “Oracle engineer” did you speak with? Oracle has tens of thousands of engineers .. ;-)

    Peace,

    Cameron Purdy
    http://www.oracle.com/technology/products/coherence/index.html

  9. Al Fall permalink

    Ari –

    Oracle’s license agreement on otn.oracle.com says that you can’t publish benchmarks without permission. You must click “I agree to these terms and conditions” before downloading any Oracle product, including Coherence. So I ask frankly, why did you publish this benchmark? Judging by Cameron’s tone in his messages, I doubt that Oracle approved this benchmark.

    I think it’s fair for them not to want publication unaudited benchmarks, as it would be easy to win a benchmark if a competing product isn’t installed or configured properly. I’m not saying that’s what happened here, but I am saying that this looks like a legal violation. I frankly think that this puts the benchmark’s credibility into serious doubt … and this calls into question Terracotta’s marketing tactics.

  10. Just in case someone still looking for recent updated GigaSpaces benchmark – take a look on this intel site:
    http://www.intelfasterfs.com/2009/02/ultra-scalable-sun-fire-x4450-intel-7460-gigaspaces-xap-platform/

    See more of these on my blog:
    http://blog.gigaspaces.com/author/shay/

    Shay

  11. You can find some updated numbers comparing GigaSpaces Cache to Terracotta ehcache here: http://blog.gigaspaces.com/2011/12/01/terracotta-ehcache-vs-gigaspaces-cache-benchmark/

    Shay

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: