As I understand your question, you are attempting to coordinate XA transactions across multiple servers (JVMs) which requires that your servers be configured with JTS, and JBoss servers are typically configured for JTA only by default. Just to make sure we're talking the same versions here, when I eyeball my console log on starting JBoss AS 5.1, it reports the following JBossTS version:
08:46:59,678 INFO [TransactionManagerService] JBossTS Transaction
Service (JTA version - tag:JBOSSTS_4_6_1_GA_CP07)
Assuming that's in the same general vicinity of the version you are running, you should review all the docs contained in the JBossTS documentation available in this download: jbossts-full-4.6.1.GA.zip. Specifically, look at the PDF titled JBossTS 4.6.0 Server Integration Guide, which amongst other things defines the different personalities of the Arjuna Transaction Manager as:
JBossTS consists of a transaction engine, ArjunaCore, with multiple
'personalities' layered around it.
JBossTS JTA provides a JTA 1.1
compliant transaction manager for Java Enterprise Edition
applications. Transaction scope is limited to a single JVM. This is
suitable for use in applications that do not require transaction
context propagation on business method calls between JVMs. For
example, deployments in which only a single application server
instance is used or where several such instances are used for load
balancing only, with no transactional communication between them.
JBossTS JTS provides CORBA based distributed transaction management
that may be driven either though JTS native API or though the JTA
interfaces. It is suitable for situations in which transaction context
must span multiple JVMs, such as a Java EE application deployed on an
application server cluster, or where interoperability is required for
transactional business method calls between heterogeneous Java
application servers, or Java and legacy applications written in
another language with CORBA bindings.
and also asserts:
For transaction use cases involving multiple JVMs, JBossTS JTS is
required.
Also take a look at this proviso regarding JBoss Messaging XA Configuration. Make sure to follow the configuration directions outlined in the JTA docs in the section titled JBoss Messaging XA Recovery Configuration.
That document provides a decent summary of what you need to do, but the details of installation and configuration of a standalone JBossTS instance are in the document titled JBoss Transactions 4.6.0 Installation Guide, while the replacement of the AS 5.1 JTA transaction services with JTS services is outlined in a document in the AS 5.1 distro itself in <jboss-home>\docs\examples\transactions\README.txt
Having said all that, if there is any way you can implement your solution using one server, I suspect you will find that the preferable way to go.
Cheers.
//Nicholas
P.S. Once you have JTS correctly installed, the console log will display something like this on startup:
09:41:03,558 INFO [TransactionManagerService] JBossTS Transaction Service (**JTS version - tag:JBOSSTS_4_6_1_GA_CP07)**
