Many of you are probably wondering how the transaction malleability problem affects Mastercoin; in short it doesn’t. However understanding why it doesn’t is worthwhile. First of all, let’s quickly go over what transaction malleability is: In Bitcoin transactions are often referred to by the cryptographic hash of the actual data in the transaction, often called the txid. These txids are used within transactions themselves to refer to the previous transaction whose coins are being spent. Since the data being hashed is the entire transaction, not just a part of it, that includes the cryptographic signatures that the Bitcoin system uses to determine if the transaction is in fact authorized to spend the Bitcoins it is trying to spend. The problem is that cryptographic signatures can’t sign themselves – it’s kinda like asking a sealed envelope to somehow contain itself. Since the signature data itself isn’t signed, if you can find a way to change the data representing a signature without making it invalid, you’ve changed the txid of the transaction, yet it’s still valid and can still be included in the blockchain. To make a long story short, there’s a lot of ways that a signature can be mutated without making it invalid and banning all those ways from happening isn’t trivial.
The problem is with unconfirmed transactions: if you have transactions A and B that are not yet confirmed in the blockchain, and B spends an output of A, then any attacker who modifies A can get the modified A to confirm in the blockchain, making B invalid. A lot of software wasn’t written with this in mind, and got confused. Of course, you might now ask why does the txid have to cover the signatures – why can’t it leave those out? Well, what would a signature sign then?
Let’s suppose we make the signature sign for a particular combination of address and amount. That is the signature is saying “I authorize a transaction output from address Alice123 spending amount 1.0 BTC, provided the funds go to address Bob456, with a 1mBTC mining fee”. The problem here is reuse: if there ever exists another transaction output with that address and amount on the blockchain, an attacker can take that signature and create a second transaction that spends the funds in the same way. Unlike txids, addresses and amounts aren’t guaranteed to be unique in Bitcoin, so there’s always a risk an attacker will be able to exploit such an accident. (although there are special cases where being able to make such signatures would be a useful option)
Mastercoin transactions on the other hand are based on balances. They’re embedded within normal Bitcoin transactions, so the signature on them is basically saying “I authorize 1 Mastercoin to be deducted from the balance of address Alice, provided that 1 Mastercoin is added to the balance associated with address Bob, oh, and BTW, this signature is only valid if present in a Bitcoin transaction spending transaction output 0x1234abcde”. Provided that Mastercoin wallet software is written correctly there’s no circumstance where transaction malleability matters because no-where in the protocol are hashes referred too.
However, there’s disadvantages with this approach too: Suppose Alice gives Bob 1.0 Mastercoin, followed by Bob giving Charlie that 1.0 Mastercoin:
| Alice | Bob | Charlie | |
| Start | 1 | – | – |
| Tx #1 | -1 | +1 | – |
| After Tx #1 | – | 1 | – |
| Tx #2 | – | -1 | +1 |
| After Tx #2 | – | – | 1 |
But what happens if a chain reorganization occurs and the transactions end up in the blockchain out of order?
| Alice | Bob | Charlie | |
| Start | 1 | – | – |
| Tx #2 – Invalid! | – | -1 | +1 |
| Unchanged | 1 | – | – |
| Tx #1 | -1 | +1 | – |
| After Tx #1 | – | 1 | – |
The second transaction doesn’t happen – there was no balance to move – so the coins only make it to Bob. Currently Mastercoin clients avoid this situation by always waiting a few confirmations to ensure that a reorganization is extremely unlikely to cause problems. In the future the Mastercoin protocol may be changed to add the same kind of txid-based ordering that the underlying Bitcoin layer has, similar to how Colored Coins enforces order; hopefully by then Bitcoin itself will have better protections against malleability. In the meantime a simple thing Mastercoin clients can do is to use the nLockTime feature to restrict transactions from being mined before a given block height, similar to how this pull-req of mine uses it to discourage miners from reorganizing to try to “snipe” high-fee transactions for themselves
Omni 
What happen if Bob already have a balance in the second scenario where tx-2 failed?
If he has a balance then the transaction is valid, assuming the balance can cover the requested move amount.