Digitex Intermediate Report – SmartDec Cybersecurity Blog
In the last blog post, we promised to give more detail on what has been done since we started working with Digitex. We collected several notes from our work until now and prepared this report.
In the beginning, we received the code from the Dublin development team that had been worked on for the launch of a centralized exchange. The project contained the following components: UI, web back-end, several databases, message queue, and the engine itself.
However, while the code was extensively documented, it was far from a production-ready implementation. The code was untested and not debugged. We still had much work to do.
Now, let’s go over details.
Web-app
Web back-end is realized using Django. We started testing and found out that parts of functionality didn’t work and parts were incomplete. For example, one could login with an account that was hardcoded into the database, but could not create a new account through the web front-end. 2FA was there in the interface design images, but not functioning. Password reset was there in code but needed debugging. The ladder interface was there, but only partially functional. There was no functionality to manually place orders not using the ladder interface.
We have now added 17 user-story images for the UI, have debugged the code, added parts that were missing in the ladder interface and have moved over to extensive testing.
Back Office
The system did not have Back Office.
We have gathered the requirements on Back Office from Adam and consulted third party experts who have been developing similar systems, created tech specs, decomposed the problem and chose what can be made using third party plug-ins. In particular, the current plan is to use ZenDesk. Everything that is not a third party component is now in development: graphical designs of pageviews have been made, web designs are being produced, functional parts are close to being available for testing.
PostgreSQL database
After we made the scheme of the database we received, we made a decision that it needed to be modified heavily, since there were a lot of unused tables in it. Now the scheme is adapted and we have started optimizing the database. Our specialists are performing database setups, creating replication scenarios, and node allocations. All of this is needed for fault tolerance and for maintaining performance characteristics.
RabbitMQ message queue
After first stress tests our team was rather surprised by the performance of the system. It was not apparent how it could support Adam’s requirements of 10,000 users. We went on to investigate the reasons. It turned out that we had a system realized using HTTP RESTful API endpoint that supported no more than 30 requests of order placement/cancelation per second.
The reasons for such low performance were:
- The system was initializing connections to redis (the cache database) and to RabbitMQ for every request.
- Two-step method of data transmission from the engine to the HTTP REST API handler, which was also using polling.
The overall performance was measured for the RabbitMQ. Tests showed that with standard setup RMQ allows transmitting 20k messages per second. Another reason the system had low performance was that for transmission between the kernel and web module message serialization was implemented with custom code. Besides that, only a limited number of message types was supported, which would prohibit future expansion of the scope of the project
During the course of the work, we have removed the intermediate link for transmitting messages from the kernel to the REST API. We also implemented the support for connecting to the Kernel directly using TCP. This connection supports up to 500,000 basic messages per second (ignoring the business logic). We use Google’s Protocol Buffers for serialization now.
Futures-Engine Kernel
We received a realization of the kernel that had 420087 lines of code in C and C++. External components contained 409478 LOC. Investigation into the code showed that a lot of external components were basically unused and could be easily removed. After that, the size of the repo was 10609 LOC, out of which, 1911 lines were tests. So, the actual codebase was 8698 lines of code. Testing and analysis of the code showed that only the simplest cases had been implemented. In order to implement the logic of the engine, the team had to deconstruct all the mathematics behind the exchange operation and futures/perpetuals market and has written most of the corresponding specifications. All of that is added to the realization of the kernel, which is ready for testing.
In order to perform extensive testing of the kernel, a decision was made to develop special robots-simulations that would imitate some strategies, mimicking users.
During the analysis, we also noticed an attempt to make the code cross-platform. For the first phase of the realization we abandoned cross-platform capabilities, which allowed us to simplify the development significantly, simplify linking for various objects and simplifying building.
The current realization of the engine supports around 40,000 transactions per second. One transaction includes order placement, matching, creating a trade and communicating the event to observers in the market, so one transaction can lead to 4 messages from the engine to users.
In Conclusion
This was a collection of notes we aggregated over the summer. Since then, the development has already moved further. We are now busy with integrational testing, building all components of the exchange together, trying to run them as a whole. The reader should not be mistaken that this means that the exchange is almost ready. There is a long road of taking the first version to the first stable secure version. However, internally, we are pretty excited as we are moving forward.
This article was created by SmartDec, a security team specialized in static code analysis, decompilation and secure development.
Feel free to use SmartCheck, our smart contract security tool for Solidity and Vyper, and follow us on Medium, Telegram and Twitter. We are also available for smart contract development and auditing work.
Published at Thu, 05 Sep 2019 20:01:34 +0000
Bitcoin Pic Of The Moment
✅ This image from Marco Verch (trendingtopics) is available under Creative Commons 2.0. Please link to the original photo and the license. 📝 License for use outside of the Creative Commons is available by request.
By trendingtopics on 2019-04-11 08:08:47
