Tari Protocol Discussion 4
The Tari community examined merge-mining pool implementations, wire protocol choices, in-process communication options, threading models, and code dependency audits during today’s Tari protocol architecture discussion.
This is the TL;DR on what was covered (full transcript included below):
How will Monero mining pools get the tools they need in order to support Tari merge-mining?
- PRs are submitted or someone has to fork the OSS mining pool software
- Closed source pools would need to do it themselves or talk to someone in the Tari community
Which wire protocols should be considered?
- In-process, ZeroMQ looks interesting
- Still looking at gRPC / http / hand-rolled also as some alternatives
What are possible in-process communication options & threading models?
- ZeroMQ is the preferred choice
- actix, Rust native (mpsc), tokio, mio, hand-rolled message brokers are possible alternatives
- More testing is needed to check speed of inprocess comms for ZeroMQ
How should code dependencies be handled?
- Audit every dependency
- Evaluate whether the risks < benefits, and go for battle-hardened, proven technologies where available
- It’s hubris to reinvent libraries that have millions of CPU hours behind them, but also it’s folly to chuck in a lib with 50 dependencies just to save 2 lines of boilerplate code
Join us for our next discussion on Freenode in #tari-dev. Discussion times proposed by the Tari community:
Mondays: 8pm CAT (1pm EST) Thursdays: 11am CAT (4am EST)
To keep up with the latest Tari developments, you can follow the project on Twitter.
Transcript of today’s discussion
7:05 PM <@cjs77> Hey all
7:07 PM <el00ruobuob_[m]> hi
7:08 PM <Krakaw> evening
7:08 PM <@fluffypony> cjs77: there aren’t tons of different pool implementations that would need PRs to support merge mining
7:08 PM <@fluffypony> just the two, as moneromooo pointed out
7:08 PM <@fluffypony> and then the ones running their own stuff must play catch-up
7:09 PM <@cjs77> Cool, so those pool have open-sourced their code?
7:10 PM <@cjs77> Any links?
7:10 PM <tk___> hey
7:12 PM <moneromooo> https://github.com/zone117x/node-cryptonote-pool is the original one, not sure what fork is the most current one though.
7:12 PM <@fluffypony> sammy007 will know
7:12 PM <mikethetike> anything specific on the agenda tonight?
7:13 PM <simian_za> Evening
7:14 PM <@cjs77> There are a couple of active topics:
7:14 PM <@cjs77> * Monero mining pools — and what we’d need to provide to help them support Tari merge mining
7:14 PM <@cjs77> Rollover from From last week:
7:14 PM <nioc> Snipa wrote the other pool software and hangs out in #monero-pools
7:15 PM <@cjs77> * wire protocol choice
7:15 PM <@cjs77> * in-process communication options & threading models
7:15 PM <@cjs77> * use of dependencies in code
7:15 PM <nioc> I guess it’s somewhere on github
7:15 PM <@cjs77> Thanks nioc
7:18 PM <@cjs77> ITO wore protocol, I’ve been doing some reading and there are a coupla alternatives:
7:18 PM <@cjs77> * NIH — I believe Grin went this route
7:18 PM <@cjs77> * gRPC — great cross platform support
7:18 PM <@cjs77> * zMQ — hands down winner imho
7:20 PM <simian_za> Zmq is also good for your second point? Interprocess comms and threading?
7:21 PM <json_> gRPC well supported in rust? What was the reason grin went with http?
7:22 PM <neonknight> gRPC and zMQ have rust bindings..
7:23 PM <@cjs77> neonknight: I really think it can be. Rust has some alternatives, like mio, tokio (which is built on mio), Actix (which is built on tokio).
7:23 PM <@cjs77> So the problem is: it’s really one stack (Rust is possibly not mature enough to have a suite of options); mio and tokio aren’t that easy to use; but Actix has a HUGE number of dependencies. Whereas ZMQ is simple, does the job; does it incredibly well; and let’s you build on top of it without much fuss
7:24 PM <mikethetike> The docs for actix-web are not bad, but for actix itself, they are very bare
7:25 PM <mikethetike> 8 out of 11 pages of the user guide just say “[WIP]”
7:25 PM <simian_za> Tokio is also very dependancy heavy
7:26 PM <@fluffypony> I’m a big fan of 0MQ
7:26 PM <@fluffypony> and vtnerd has done great work integrating it in Monero
7:26 PM <@fluffypony> he can give input in to that experience
7:27 PM <@cjs77> So to step back a sec; I like the idea of using message passing (even internally) because it allows for a very decoupled architecture; the question is that if this is the approach to take, is zmQ / 0mq performant enough on its `inproc` sockets to use a sinlge library for both “internal” and external messaging?
7:28 PM <@fluffypony> so like microservices?
7:28 PM <@cjs77> yup
7:28 PM <@cjs77> maybe not per the strict definition (like the modules would still run on the same box for performance)
7:30 PM <@cjs77> but, if you _wanted_ to run the e.g. block validator module on a different machine to the mempool (I don’t know why you would, but hey), you could change `inproc` to `tcp` and off you go (more or less)
7:32 PM <mikethetike> What did monero use before 0mq
7:33 PM — @cjs77 gathers around the fireplace to listen to fluffypony regale us with tales of yore
7:34 PM <Hansie> Hi there
7:35 PM <Hansie> zMQ: Inter plaform — yes, inter process — yes, inner process — maybe
7:36 PM <@fluffypony> mikethetike: JSON over HTTP
7:36 PM <@fluffypony> it still supports that
7:36 PM <@fluffypony> and the CLI wallet still talks to the daemon over that
7:36 PM <Hansie> A couple of simple spike tests will tell if zMQ can handle all three. If so, good
7:36 PM <@fluffypony> but it’s not really IPC, it’s more RPC
7:36 PM <@cjs77> Yeah, it’s the “maybe” I’m looking to see if anyone has some definitive experience on
7:38 PM <@fluffypony> https://grokbase.com/t/zeromq/zeromq-dev/102s8rzsvs/inter-thread-zmq-inproc-performance
7:39 PM <@fluffypony> that looks positive
7:39 PM <@fluffypony> http://zeromq.org/results:perf-howto
7:39 PM <@fluffypony> To run an inter-thread latency test, use inproc_lat instead of the above. It is a single executable and doesn’t require specifying an endpoint:
7:40 PM <@fluffypony> To run an inter-thread throughput test, use inproc_thr instead of the above. It is a single executable and doesn’t require specifying an endpoint:
7:41 PM <@cjs77> That looks great, actually.
7:41 PM <mikethetike> I suppose we’ll still need to expose parts of the API via http for android wallets etc
7:41 PM <mikethetike> or can 0mq do that natively
7:42 PM <@fluffypony> mikethetike: Android can talk via 0MQ over TCP
7:42 PM <@cjs77> I did some tests on the mac using `tcp` loopback last week and got 10,000,000 messages/sec throughput, so even that is pretty quick
7:42 PM <@fluffypony> there are bindings for Java
7:42 PM <mikethetike> noice
7:42 PM <Hansie> My experience is with ACE TAO Corba (The ACE Orb) — real-time Corba. The native OS messaging infrastructure (semaphore, WaitForSingleObject, etc.) was more efficient for inner process communication. Seems to be a heavier implementation than zMQ though.
7:43 PM <@cjs77> If we have LMDB as a key-value store and 0mq as a the message broker, I think we have a little pocket rocket on ur hands
7:45 PM <@fluffypony> yuuuuup
7:45 PM <@fluffypony> hyc would be so proud
7:45 PM <@fluffypony> *wipes tear*
7:45 PM <@cjs77> btw Googling “corba rust implementation” doesn’t come up with anything
7:46 PM <Hansie> LMDB + zMQ 0MG sounds good, but the proof is in the pudding. I am a fan of a couple of small spike projects to proof.
7:46 PM <@fluffypony> lol corba
7:49 PM <@cjs77> Yeah, we can pull up some tests. fluffy’s link had some benchmark code references
7:50 PM <@cjs77> Ok, great progress on this front. It also implicitly addresses the dependency issue.
7:50 PM <@cjs77> .. implicitly to some degree
7:51 PM <@fluffypony> cjs77: so in Monero we’ve had to submodule some deps for building static bins
7:51 PM <@fluffypony> how does Rust handle deps?
7:51 PM <@cjs77> Cargo is a packagae manager ala npm for node
7:51 PM <mikethetike> but you can also cargo a submodule if needed
7:52 PM <@cjs77> compiling in rust is (99% of the time) as simple as `cargo build`
7:52 PM <@fluffypony> ok but then is there static and dynamic binding like with C?
7:52 PM <@cjs77> nah, everything is a crate
7:52 PM <@fluffypony> are crates binary packages or source code?
7:52 PM <@cjs77> but a crate can be a gtihub link to specific commit for example
7:52 PM <mikethetike> crates are source
7:52 PM <@cjs77> source code
7:52 PM <@fluffypony> ah ok so it always builds every dep
7:53 PM <mikethetike> but you can also build a dynamic linkable binary I believe
7:53 PM <@fluffypony> let’s hope we never have boost for rust :-P
7:53 PM <@fluffypony> dynamic libraries and DLLs were always a dumb idea, imho
7:53 PM <mikethetike> I think compiling from source is what we’ll be doing most of the time though
7:53 PM <@fluffypony> DLL hell is a real thing for developers
7:54 PM <@cjs77> afaik even something ike 0mq, which are rust wrappers around the C library, will build the C code as part of the build, but I may be worng
7:54 PM <@fluffypony> kk
7:55 PM <mikethetike> I do think we need to be careful about which dependencies we use
7:56 PM <@cjs77> fluffypony: I def looks like Rust expects `libssl` and `libc` to be on your system at runtime, so there are still a few dynamic linked libs. This is by default, and I suspect you could force linking those in too if you wished
7:56 PM <mikethetike> otherwise we’ll be reviewing code in those repos forever
7:57 PM <@cjs77> mikethetike: agreed. Wich is why the ability to freeze crate versions or even commit versions is a nice feature
7:57 PM <@fluffypony> yeah like submodules are at a commit hash
7:57 PM <@fluffypony> so same but different
7:57 PM <@fluffypony> cjs77: I guess libssl is pretty standard by now
7:58 PM <Hansie> I like the ability to link to certain commit versions.
7:58 PM <neonknight> I think we should limit the number of dependencies to the bare minimum. It will be crazy to reinvent the wheel every time.
7:59 PM <@cjs77> fluffypony: yeah, on any standard system (and we supply instructions for things like building a docker image on alpine linux, for example)
8:00 PM <@cjs77> neonknight: it sounds like part b is contradicting part a?
8:00 PM <@fluffypony> lol
8:00 PM <@cjs77> it’s a thing! :)
8:01 PM <neonknight> Hehhe It is late in South Africa, some dependencies are better than none.
8:01 PM <@cjs77> kk. gotcha
8:04 PM <Hansie> neonknight: Yes, we should invent tokens not wheels
8:05 PM <@cjs77> Yeah, walk the line. Audit every dependency. Really evaluate whether the risks << the benefits, and go for battle-hardened, proven technologies where available. It’s hubris to think we could improve on libraries that have millions of CPU hours behind them, but also it’s folly to chuck in a lib with 50 dependencies just to save 2 lines of boilerplate code here and there.
8:05 PM <mikethetike> agreed
8:05 PM <endogenic> there’s the argument one ends up maintaining any dependency anyway
8:05 PM <mikethetike> Also no auto-updating of dependencies
8:06 PM <Hansie> If sense is with us sanity will prevail