A dumb way to test a smart contract
/From June 6 to June 14 2019, I ran a user test of a decentralized forward contract on the "Ropsten" network. Ropsten is an Ethereum network where the network token "ether" can be obtained for free and therefore has no financial value.
The test was executed in the form of a trading contest with a total of 3 ETH (approximately $800 or C$1,000 at the time of payment) in prize money. Since trading took place on a test network, a separate financial reward was needed to encourage users to trade the contract. The lack of economic consequences of losing test ether also meant that unrealistic trading behaviour was bound to occur, the most obvious being the execution of trades at inflated or deflated prices between accounts controlled by the same user. Since there was no way to effectively prevent it, the use of multiple accounts to increase chances of winning was explicitly allowed.
There was a positive to this weakness. Testers attempting to win the contest with this strategy could be expected to create a greater number of accounts and trades then they normally would, putting the whole system under a decent amount of "stress" even with relatively few actual individuals involved.
However, there was an alternative scenario. I included a feature in the smart contract which had the power to completely change the way to win the contest if discovered by the testers. Very few trades would be entered but I would have confidence that testers reviewed the contract source code and found an exploit I knew about. If no other exploit would be found during the test, I could argue that the contract was safe at the $1,000 level.
Before I describe what happened, let us look at some stats.
I posted about the contest on Reddit and Gitcoin. All traffic to the contest instructions on this website came from Gitcoin.
Website traffic
| Number of unique visitors: | 109 | |
| Visits: | 139 | |
| Number of Page Views: | 317 | |
| Countries: | 34 | (from all continents except Antarctica) |
Looking at user flow, over 90% of visitors only accessed information directly related to the contest and did not explore the rest of the website.
Test stats
| Number of testers: | 9 | |
| Number of test accounts (peak): | 16 | |
| Number of trades entered: | 251 | |
| Total amount traded: | $163,185 | |
| Total test ether supplied (peak): | 165 ETH | |
| Winners: | 2 | (one tester took 1st and 3rd place) |
Communication with testers and feedback
Communication with testers happened primarily on the test's page on Github.
Two testers asked three questions primarily related to the test object.
One of the questions concerned the possibility of gaming the system by trading between own accounts and suggested that it should be "restricted." I replied that it was “perfectly legal.”
One tester reported a user interface bug and also followed up with an email. I exchanged 11 messages with this user and I had the impression that they were providing honest feedback and were willing to work through issues. They mentioned that the application provided a "low user experience" and "high technical difficulty" and offered to provide further recommendations and feedback for a fee.
Two testers completed the test survey which included questions about bugs and recommendations. One user stated that “lots” of (unspecified) improvements could be made. Another user claimed to have found other bugs and exploits in addition to trading between own accounts, but did not give details. The tester did, however, send an email offering help with testing and fixing the contract “with or without a bounty.” Both testers stated that the documentation “could have been better.”
Test History
Below OHLC chart shows the price of the underlying (Ethereum ETH in US dollars) during the test period. The frequent trend changes and volatility created variability in the relative day-to-day performance of the competitors. This was probably beneficial to the test as it may have encouraged continued engagement, i.e. trading, by the testers.
The next chart shows volumes (bottom section) and prices of the underlying traded via the derivative contract that was the focus of the test. The derivative used the inverse of the underlying price (e.g. $250 per ETH converts to 1/250 = 0.004 ETH per $).
The derivative price chart shows a relatively smooth line and a clear negative correlation with the underlying price during the first 5 days. Traded volumes were low.
On June the 9th, the question of trading between own accounts was raised on the Github issue page and the first evidence of this "strategy" being used occurred on June 10. The pronounced kinks and trend deviations in the second half of the graph are caused by implementations of the strategy. The "flash crash" on June 11 could be a failed attempt, but judging by the accounts and amounts involved, it was a data entry error.
Although it is impossible to be absolutely certain, according to my analysis only 3 test participants made use of this method to increase their chances of winning. Furthermore, only 24 out of 251 trades match this pattern. Volumes traded on these occasions were high and the strategy was the determining factor of the final ranking. 23 out of the 24 "strategy trades" can be attributed to the two winners.
Below graph illustrates a well-executed instance of the pattern. The account selected to be the winner buys at a low price from another account controlled by the same tester and then an opposing trade is entered shortly afterwards at a considerably higher price. The notional amount traded in this example was $10,000. The process is repeated a few times and the buy-low-sell-high account is left with a profit.
Assuming that all testers had some interest in winning the contest (as it was the only way to gain financially from participating) it is noteworthy that the strategy was not used very aggressively, i.e. used only by a small number of testers, on a few occasions, and with moderate profit margins.
Trading activity was uneven. Almost 60% of all trades occurred over the span of two days from June 11 to June 12.
| Period | Trades in period |
|---|---|
| June 6 - 10 | 61 |
| June 11 | 49 |
| June 12 | 96 |
| June 13 | 24 |
| June 14 | 21 |
The "special" feature of the contract mentioned in the introduction would have made a surprise win possible if exploited just before the test ended (so as not to alert other participants), but it was not used.
Personal Conclusions and Recommendations
Offering bounties can attract a significant number of motivated individuals to participate with testing or other aspects of a software development project.
It can be expected that participants will adapt and optimize their behavior according to the incentive structure offered and not exert much effort unless it is directly rewarded. Testers may not even report bugs unless properly incentivized.
Care must be taken to offer sufficiently high bounties for every meaningful contribution. As such, it is more suitable for problems that are well-defined.