View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

<- Part 1- Crafting a Blockchain in Go and Rust: A Comparative Journey — Private keys, Public Keys and Signatures

In this post, we’ll dive into the implementation of fundamental blockchain components: the block header, the block itself, and transaction. These elements are crucial for constructing a blockchain, as they form the building blocks of the distributed ledger. We’ll explore the initial implementations in Go and Rust, the challenges encountered with encoding, and how I ultimately achieved deterministic encoding using Protocol Buffers (protobuf).

The Importance of Headers, Blocks, and Transactions

Headers: The header in a blockchain block contains metadata about the block, such as the previous block hash, the timestamp, version, height, nonce and the hash of transactions. It is critical for linking blocks together, ensuring integrity, and enabling quick verification.

Blocks: A block consists of the header, the block signature a public key to verify the signature and a list of transactions. Each block encapsulates a set of transactions that have been validated and recorded in the blockchain.

Transactions: Transactions are the basic units of value transfer in the blockchain. They include the sender’s and receiver’s addresses, the amount transferred, some arbitrary piece of data I decided to add to it, and a digital signature to verify authenticity.

Initial Implementations in Go and Rust

Initially, I implemented the blockchain using the core data structures and native encoding libraries specific to each language. At first, I didn’t give much thought to the implications of using different encoding libraries — after all, bytes are bytes, right? However, I soon realised that to achieve deterministic behavior across both implementations, I needed to rethink that approach. But more on that in a moment.

For now let’s look at the data structures initially implemented.

Go Implementation:

// types/hash.go
// Hash represents the 32-byte hash of a block.
type Hash [HashSize]byte

// core/block.go
// Header represents the header of a block in the blockchain.
type Header struct {
    PrevBlockHash types.Hash // Hash of the previous block

    TxHash types.Hash // Hash of the transactions in the block
    Version uint32 // Version of the block
    Height uint64 // Height of the block in the blockchain
    Timestamp int64 // Timestamp of the block

    Nonce uint64 // Nonce used to mine the block
    Difficulty uint8 // Difficulty used to mine the block
}

// Block represents a block in the blockchain.
type Block struct {
    *Header

    Transactions []*Transaction
    PublicKey crypto.PublicKey
    Signature *crypto.Signature

    // Cached version of the header hash
    hash types.Hash
}

// core/transaction.go
// Transaction represents a transaction in the blockchain.
type Transaction struct {
    From crypto.PublicKey // Public key of the sender
    To crypto.PublicKey // Public key of the receiver
    Value uint64 // Amount to transfer
    Data []byte // Arbitrary data
    Signature *crypto.Signature // Signature of the transaction
    Nonce int64 // Nonce of the transaction

    // hash of the transaction
    hash types.Hash
}

Encoding with Go:

At first, I chose to use the Gob package in Go for encoding the structures. Gob it is often the preferred way when communicating between Go programs since it is Go’s native binary encoding library, designed for efficient serialisation and deserialisation. Gob’s performance is great, but again, mainly or only used for communicating between Go programs, since we are developing one Blockchain Node in Go and another Blockchain Node in Rust, come to think of it may not have been the best choice.

Rust Implementation:

// types/hash.rs
#[derive(PartialEq, Debug, Serialize, Deserialize, Clone, Copy)]
pub struct Hash {
    pub hash: [u8; HASH_SIZE],
}

// core/block.rs
#[derive(Debug, Serialize, Deserialize)]
pub struct Header {
    pub previous_block_hash: [u8; 32],

    pub tx_hash: Option<hash::Hash>,
    pub version: u32,
    pub height: u32,
    pub timestamp: u32,

    pub nonce: u32,
    pub difficulty: u8,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Block {
    header: Header,
    transactions: Vec<Transaction>,
    #[serde(default, skip_serializing_if = "Option::is_none", skip_deserializing)]
    public_key: Option<PublicKey>,
    #[serde(default, skip_serializing_if = "Option::is_none", skip_deserializing)]
    signature: Option<SignatureWrapper>,

    // Cached version of the header hash
    #[serde(default, skip_serializing_if = "Option::is_none", skip_deserializing)]
    hash: Option<hash::Hash>,
}

// core/transaction.rs
#[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
pub struct Transaction {
    pub from: Option<PublicKey>,
    pub to: Option<PublicKey>,
    pub value: u64,
    pub data: Vec<u8>,
    pub signature: Option<SignatureWrapper>,
    pub nonce: u64,

    // Cached version of the transaction hash
    pub hash: Option<hash::Hash>,
}

Encoding with Rust:

For Rust I decided to with Bincode, which is a binary encoder / decoder implementation in Rust. Bincode is a compact encoder / decoder pair that uses a binary zero-fluff encoding scheme. The size of the encoded object will be the same or smaller than the size that the object takes up in memory in a running Rust program.

In addition to exposing two simple functions (one that encodes to Vec<u8>, and one that decodes from &[u8]), binary-encode exposes a Reader/Writer API that makes it work perfectly with other stream-based APIs such as Rust files, network streams.

But again, Bincode can be quite fast but I ran into the same problem I had with Go Glob package, meaning Bincode although quite popular, it does not have an implementation for Go, as far as I know, so the results would be different and not really deterministic and would ended with the same block being encoded differently on different Blockchain Nodes, which cannot happen.

Encountering the Problem: Non-Deterministic Encoding

The Issue

Despite the success in serialising the structures, I encountered a significant issue: the encoded results were not deterministic. This means that the same data structures were being encoded differently in Go and Rust.

Implications

For a blockchain where different nodes (potentially written in different languages) need to communicate and validate blocks and transactions consistently, non-deterministic encoding is a critical flaw. It could lead to a situation where nodes disagree on the validity of a block simply because of differences in how data is serialised.

The Need for Deterministic Encoding

Why Determinism Matters?

In a blockchain network, all nodes must agree on the state of the ledger. If two nodes encode the same block differently, they might calculate different hashes for that block, leading to a consensus failure.

Serialisation Format Requirements:

For cross-language compatibility and determinism, the serialisation format needs to:

• Be consistent across different implementations.

• Preserve the exact byte order.

• Be well-documented and widely supported.

Switching to Protocol Buffers (Protobuf)

Why Protobuf?

I decided to switch to Protocol Buffers (protobuf) for encoding because it is a language-neutral, platform-neutral extensible mechanism for serialising structured data. It’s widely used in systems that require deterministic encoding across different environments, including many blockchain projects, and it’s quite performant.

Implementation Details

Defining the Protobuf Schema:

For that we need to create a .proto file that represents your Go and Rust structs. I usually create a folder named proto/ and saved it in there. Here’s what the .proto file might look like:

syntax = "proto3";

package proto;

option go_package = "github.com/joaoh82/marvinblockchain/proto";

// Header represents the header of a block in the blockchain.
message Header {
    bytes prev_block_hash = 1;
    bytes tx_hash = 2;
    uint32 version = 3;
    uint64 height = 4;
    int64 timestamp = 5;
    uint64 nonce = 6;
    uint32 difficulty = 7;
}

// Transaction represents a transaction in the blockchain.
message Transaction {
    bytes from = 1;
    bytes to = 2;
    uint64 value = 3;
    bytes data = 4;
    bytes signature = 5;
    int64 nonce = 6;
    bytes hash = 7;
}

// Block represents a block in the blockchain.
message Block {
    Header header = 1;
    repeated Transaction transactions = 2;
    bytes public_key = 3;
    bytes signature = 4;
    bytes hash = 5;
}

Generating Code from the Protobuf Schema:

Go:

First, make sure you have the Protocol Buffers compiler (protoc) and the Go plugin for it installed:

# Install protoc (if not already installed)
brew install protobuf

# Install the Go plugin for protoc
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

Now, generate the Go code:

# Generating the actual code
rm -rf proto/*.pb.go
protoc --go_out=. --go_opt=paths=source_relative \
proto/*.proto

This command above generates a .pb.go file in the same directory, which contains the Go structs and methods for serialisation and deserialisation.

Rust:

For Rust, you also need to install protoc and the protobuf-compiler. And it can be installed, at least on mac, the same way as above:

# Install protoc (if not already installed)
brew install protobuf

On linux you can check out my github actions scripts in the Rust repo:

sudo apt-get update
sudo apt-get install -y protobuf-compiler
cargo install protobuf-codegen

Add Dependencies

Update your Cargo.toml to include the necessary dependencies.

[dependencies]
prost = "0.13.1"
prost-types = "0.13.1"
[build-dependencies]
prost-build = "0.13.1"

Write a Build Script

In Rust, you need to create a build.rs file to instruct Cargo to compile the Protobuf definitions using the prost crate. Note that you need to create the build.rs file in the root of the project, at the same level you have your Cargo.toml .

fn main() {
    prost_build::compile_protos(&["src/proto/types.proto"], &["src/proto"])
    .expect("Failed to compile proto");
}

Where is the protobuf generated rust file?

Important note here is that the the file generated by prost will not go in the same directory as the *.proto file, but in reality it will go in the target/[environment]/build/[build-hash]/out , which makes sense, because with prost the file is actually generated at build time.

Generate Rust Code from Protobuf

When you run cargo build, the build.rs script will compile your .proto file and generate the corresponding Rust code. This generated code will be placed in the OUT_DIR directory.

More on OUT_DIR and Build Scripts you can find it here.

Now that you have the generated code, you can use it in your Rust application:

Include the Generated Code in your Rust project

include!(concat!(env!("OUT_DIR"), "/proto.rs"));

I actually created a file named mod.rs inside the proto/ directory and added the following content. This way I can follow the same pattern as my Go project and access the proto types via the the proto module .

use prost;
use prost::{Enumeration, Message};

include!(concat!(env!("OUT_DIR"), "/proto.rs"));

Testing and Verifying Determinism

To test it, since there is no networking layer yet, I had to take a more manual route. So I basically wrote a snippet of code in Go and Rust to generate a Block with a Transaction in it, signed everything and then checked the serialisation output of both languages.

Later on, after we add a networking layer and the Node can actually communicate, I create some integration test so we can have a more robust testing framework.

Go Snippet:

func BlockSerialization() {
    fmt.Println("Block Serialization")

    mnemonicTo := "all wild paddle pride wheat menu task funny sign profit blouse hockey"
    // Generate a new private key from the mnemonic
    privateKeyTo, _ := crypto.NewPrivateKeyfromMnemonic(mnemonicTo)
    // fmt.Println("private key TO:", privateKeyTo)
    publicKeyTo := privateKeyTo.PublicKey()
    // fmt.Println("public key TO:", publicKeyTo)
    addressTo := publicKeyTo.Address()
    fmt.Println("address TO:", addressTo)

    mnemonicFrom := "hello wild paddle pride wheat menu task funny sign profit blouse hockey"
    // Generate a new private key from the mnemonic
    privateKeyFrom, _ := crypto.NewPrivateKeyfromMnemonic(mnemonicFrom)
    // fmt.Println("private key FROM:", privateKeyFrom)
    publicKeyFrom := privateKeyFrom.PublicKey()
    // fmt.Println("public key FROM:", publicKeyFrom)
    addressFrom := publicKeyFrom.Address()
    fmt.Println("address FROM:", addressFrom)

    header := &proto.Header{
        PrevBlockHash: make([]byte, 32),
        TxHash: make([]byte, 32),
        Version: 1,
        Height: 1,
        Timestamp: 1627483623,
        Nonce: 12345,
        Difficulty: 10,
    }

    block := &proto.Block{
        Header: header,
        Transactions: []*proto.Transaction{},
        PublicKey: publicKeyFrom.Bytes(),
        Signature: []byte{},
        Hash: []byte{},
    }

    tx := &proto.Transaction{
        From: publicKeyFrom.Bytes(),
        To: publicKeyTo.Bytes(),
        Value: 1000,
        Data: []byte("Transaction data"),
        Signature: make([]byte, 64),
        Nonce: 123,
        Hash: make([]byte, 32),
    }
    core.SignTransaction(&privateKeyFrom, tx)
    core.AddTransaction(block, tx)

    core.SignBlock(&privateKeyFrom, block)

    bBlock, _ := core.SerializeBlock(block)
    fmt.Println("GO: Block WITH TRANSACTIONS hex:", hex.EncodeToString(bBlock))

}

Rust Snippet:

fn block_serialization() -> Result<(), Box> {
println!("Block Serialization");

let mnemonic_to = "all wild paddle pride wheat menu task funny sign profit blouse hockey";
let private_key_to = crypto::keys::get_private_key_from_mnemonic(&mnemonic_to).unwrap();
let public_key_to = private_key_to.public_key();
let address_to = public_key_to.address();
println!("address to: {}", address_to.to_string());

let mnemonic_from = "hello wild paddle pride wheat menu task funny sign profit blouse hockey";
let mut private_key_from = crypto::keys::get_private_key_from_mnemonic(&mnemonic_from).unwrap();
let public_key_from = private_key_from.public_key();
let address_from = public_key_from.address();
println!("address from: {}", address_from.to_string());

// Create an instance of Header
let header = proto::Header {
    prev_block_hash: [0; 32].to_vec(),
    tx_hash: [0; 32].to_vec(),
    version: 1,
    height: 1,
    timestamp: 1627483623,
    nonce: 12345,
    difficulty: 10,
};

// Create an instance of Block
let mut block = proto::Block {
    header: Some(header),
    transactions: vec![],
    public_key: public_key_from.to_bytes().to_vec(),
    signature: vec![],
    hash: vec![],
};

let mut tx = proto::Transaction {
    from: public_key_from.to_bytes().to_vec(),
    to: public_key_to.to_bytes().to_vec(),
    value: 1000,
    data: b"Transaction data".to_vec(),
    signature: [0; 64].to_vec(),
    nonce: 123,
    hash: [0; 32].to_vec(),
};
let _ = core::transaction::sign_transaction(&mut private_key_from, &mut tx).unwrap();
core::block::add_transaction(&mut block, tx);

core::block::sign_block(&mut private_key_from, &mut block).unwrap();

println!("RUST: Block WITH TRANSACTIONS hex: {:?}", hex::encode(core::block::serialize_block(block.clone()).unwrap()));

Ok(())
}

Output and Comparison:

## Not signed Block with signed transactions:
### RUST:
```
"0a530a20000000000000000000000000000000000000000000000000000000000000000012201d2f60156a6850a588d6cddd106af296316f5593a0f9f0b67ceac0c552e838f01801200128e7db85880630b960380a12bf010a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e1220e15af3cd7d9c09ebaf20d1f97ea396c218b66037cdf8e30db0ebd7bb373df56d18e80722105472616e73616374696f6e20646174612a400ba42f94db0f3ebe75e2e40fccbc7e8915510724c18769b8ad939ca44bdec82179da5bd7e7ae7079a6985ee58e77b85e74e3d76ff2c87f85723327e588220c02307b3a204c5836ec9395a4c7dee4fa71e0b2fcadfff90a747e75126fe8df184ae42c7fb61a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e"
```
### GO:
```
0a530a20000000000000000000000000000000000000000000000000000000000000000012201d2f60156a6850a588d6cddd106af296316f5593a0f9f0b67ceac0c552e838f01801200128e7db85880630b960380a12bf010a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e1220e15af3cd7d9c09ebaf20d1f97ea396c218b66037cdf8e30db0ebd7bb373df56d18e80722105472616e73616374696f6e20646174612a400ba42f94db0f3ebe75e2e40fccbc7e8915510724c18769b8ad939ca44bdec82179da5bd7e7ae7079a6985ee58e77b85e74e3d76ff2c87f85723327e588220c02307b3a204c5836ec9395a4c7dee4fa71e0b2fcadfff90a747e75126fe8df184ae42c7fb61a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e
```

## Signed block:
### RUST:
```
"0a530a20000000000000000000000000000000000000000000000000000000000000000012201d2f60156a6850a588d6cddd106af296316f5593a0f9f0b67ceac0c552e838f01801200128e7db85880630b960380a12bf010a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e1220e15af3cd7d9c09ebaf20d1f97ea396c218b66037cdf8e30db0ebd7bb373df56d18e80722105472616e73616374696f6e20646174612a400ba42f94db0f3ebe75e2e40fccbc7e8915510724c18769b8ad939ca44bdec82179da5bd7e7ae7079a6985ee58e77b85e74e3d76ff2c87f85723327e588220c02307b3a204c5836ec9395a4c7dee4fa71e0b2fcadfff90a747e75126fe8df184ae42c7fb61a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e2240131940f20739fb191ff1c4c439e0a5f93b8e25b5c1850a500bfac644ae451a14e8c3bd691854463494ba5b2ba831c39faa583ce0b1c62ca972db8b47c2cc32082a20389bef0d3f0712e3fc5660262c61b9a2feee6475a3a01e35d9f52b2af40b932c"
```
### GO:
```
0a530a20000000000000000000000000000000000000000000000000000000000000000012201d2f60156a6850a588d6cddd106af296316f5593a0f9f0b67ceac0c552e838f01801200128e7db85880630b960380a12bf010a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e1220e15af3cd7d9c09ebaf20d1f97ea396c218b66037cdf8e30db0ebd7bb373df56d18e80722105472616e73616374696f6e20646174612a400ba42f94db0f3ebe75e2e40fccbc7e8915510724c18769b8ad939ca44bdec82179da5bd7e7ae7079a6985ee58e77b85e74e3d76ff2c87f85723327e588220c02307b3a204c5836ec9395a4c7dee4fa71e0b2fcadfff90a747e75126fe8df184ae42c7fb61a20ec319b757d96d2516e6ace0932923098e5b18226a45818a279adba351149938e2240131940f20739fb191ff1c4c439e0a5f93b8e25b5c1850a500bfac644ae451a14e8c3bd691854463494ba5b2ba831c39faa583ce0b1c62ca972db8b47c2cc32082a20389bef0d3f0712e3fc5660262c61b9a2feee6475a3a01e35d9f52b2af40b932c
```

You can copy and paste those values in any code editor to check if they are the same, and as you will see the hashes of each encoded blocks are the same before and after digitally signing it.

As always, the full source code is available on Github:

View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

What have I learned from all this?

On this post we set out to implement Blocks and Transactions for the blockchain and encounter our first set of problems when trying to make a blockchain across two different programming languages.

Building a blockchain that operates across different programming languages, such as Go and Rust, introduces several challenges related to language interoperability. These challenges arise from differences in how each language handles data structures, serialisation, concurrency, and more. Here are some key issues you may encounter:

Serialization and Encoding Differences:

• Data Representation: Different languages often represent and serialize data differently. For example, the way Go’s Gob and Rust’s Bincode handle binary encoding can lead to non-deterministic outputs, even if the input data structures are identical. This discrepancy becomes a critical issue in a blockchain network, where consistency across nodes is paramount.

• Byte Order (Endianness): Some languages might use different byte orders (big-endian vs. little-endian) when encoding data. If not handled properly, this can result in different hashes for the same data, leading to consensus failures in a blockchain system.

• Floating-Point Precision: Some languages might handle floating-point arithmetic differently, which can lead to subtle bugs when data is shared between systems.

Concurrency Models:

• Concurrency and Parallelism: Go uses goroutines and channels for concurrency, which is quite different from Rust’s ownership-based concurrency model. When building a system where components written in different languages need to interact, ensuring that these concurrency models work seamlessly together can be difficult.

• Thread Safety: Rust emphasises thread safety through its ownership and borrowing system, whereas Go’s garbage collector and goroutines provide a different approach to managing memory and concurrency. Integrating components across these languages may require additional effort to ensure that memory is safely managed.

Error Handling and Type Systems:

• Type Systems: Go’s type system is relatively straightforward and less strict compared to Rust’s, which has a more complex and expressive type system. This can make it challenging to translate data and logic between the two languages, especially when dealing with complex types or ensuring type safety.

• Error Handling: Rust’s approach to error handling using Result and Option types differs significantly from Go’s use of error interfaces. Integrating these systems can require additional layers of abstraction or adaptation to ensure that errors are handled consistently across the entire system.

4. Tooling and Ecosystem Differences:

• Build and Deployment: Managing builds and deployments across multiple languages can complicate the development workflow. Each language has its own build tools, package managers, and deployment pipelines, which need to be integrated seamlessly.

• Debugging and Testing: Debugging a system that spans multiple languages can be cumbersome, as it may require different tools and approaches. Similarly, testing across languages requires ensuring that test cases are consistent and that any language-specific behaviors are accounted for.

Importance of Standards

Given the challenges of language interoperability and determinism, adhering to well-established and cross-compatible standards becomes crucial in distributed systems like blockchains. Here’s why:

1. Ensuring Consistency:

• Deterministic Behavior: Standards like Protocol Buffers (protobuf) enforce consistent data serialisation across different languages. By using a common format, you ensure that the same data structure is encoded and decoded identically, regardless of the programming language. This determinism is vital in blockchains, where even a small discrepancy can lead to a fork in the network.

• Cross-Language Compatibility: Standards like protobuf are designed to work seamlessly across many programming languages. This means you can define your data structures once and generate compatible code for both Go and Rust (as well as many other languages), ensuring that all nodes in your blockchain network interpret the data in the same way.

2. Facilitating Integration:

• Interoperable Protocols: Using standardized protocols like gRPC (which leverages protobuf for message serialization) allows different components of your system, potentially written in different languages, to communicate effectively. This interoperability reduces the risk of errors and simplifies the integration process.

Although I mentioned gRPC here, but when we get to the networking part you will see that I actually decided to go with libp2p instead. But we will get to that.

• Interoperability with Other Systems: By adhering to widely-adopted standards, your blockchain can easily integrate with other systems, tools, and services that also support these standards. This is especially important if your blockchain needs to interact with external systems or be compatible with existing infrastructure.

3. Simplifying Development and Maintenance:

• Unified Tooling: Standards like protobuf come with a rich set of tools and libraries that make development easier. For example, you can use protoc to automatically generate serialisation code in multiple languages, reducing the likelihood of human error and speeding up the development process.

• Easier Debugging and Troubleshooting: When all parts of a system adhere to the same standards, it simplifies debugging and troubleshooting. For instance, if you know that both Go and Rust nodes are using the same protobuf schema, you can be confident that any discrepancies in behavior are not due to serialization issues.

4. Community and Support:

• Wide Adoption: Standards like protobuf, libp2p and gRPC have large, active communities and are supported by major technology companies. This means you benefit from ongoing improvements, extensive documentation, and a wealth of community knowledge.

• Future-Proofing: By using widely-adopted standards, you ensure that your project remains compatible with future developments and can leverage new tools and technologies as they become available.

Final thoughts

In a project where we are building a blockchain in two different languages, like Go and Rust, overcoming language interoperability challenges is critical to ensuring that all nodes work harmoniously. Leveraging well-established standards like protobuf not only resolves potential discrepancies but also ensures that our system is robust, scalable, and maintainable in the long term. By doing so, we are laying a solid foundation for our blockchain to succeed across different platforms and environments.

Don’t forget to Like and Subscribe to be notified about the next posts in the series and also STAR the projects on Github to follow what is going on over there.

View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

Cheers!

View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

<- Part 0 — Crafting a Blockchain in Go and Rust: A Comparative Journey — Introduction and Overview

Part 2 - Crafting a Blockchain in Go and Rust: A Comparative Journey — Blocks and Transactions ->

This episode is about private keys, public keys, signatures and adding a CLI interface to the Blockchain.

Today I will walk you through the initials steps I took to start building a Blockchain in both Go and Rust, and while I do that I’ll also try to lay out some of the differences and intricacies between the two programming languages. The plan is to go over the CLI Interface and why I already added one so early in the game, also cover the generation of private and public keys as well as being able to sign chunks of data and verifying with those same keys.

After reading this you will have a good understanding on why I decided to add a CLI interface before I even had anything done in my project and also learn about the steps I took to generate the private and public key, as well as sign and signature verification with both programming languages, Go and Rust.

Who knows? Maybe by the end I’ll even reveal which one is my favorite so far.

But a heads up, this is not a step by step tutorial, I’ll try to explain my thought process and some of decisions I took, but I will not explain line by line of the code. The code is on Github (Go / Rust), and I am happy to answer any questions, so feel free to reach out.

CLI INTERFACE

On this section of the article I want to first explain why I decided to add a CLI Interface, even if it didn’t do much at first, so early in the game. And why I believe it will make my life so much easier as the project progresses.

I am also going to talk briefly about the different implementations, packages and crates I choose for such task, and why I did not go with the standard library.

Let’s start with the why!

“why are you wasting time with this at this stage in the process?”

I actually received the same question when I did my database project (SQLRite) and the first thing I did was add a REPL interface that didn’t do anything, just echoed commands.

And the reason is because I wanted to have an interface ready too call out features of my application without having to pull up some hacky code on my main.go or main.rs for example. And it also forces me to design my code in a way where everything is decoupled and any dependencies I need for anything, they need to be injected it.

Yes, don’t worry, I also have unit tests for a lot of the code that runs every time I push to Github (or manually), but still, I like to have a way to interface directly with my application. And by having a basic CLI interface setup I can easily add commands to it, take arguments and call whatever I want in the application. I can even make the CLIs a different binary if i want, which can be really handy in the future.

Adding a CLI Interface to Go

As I mentioned I decided not to use the Standard Library for this because although the Go and Rust standard library provides all the essential building blocks for working with command-line arguments, flags, and interacting with the operating system. I am already imagining having commands, sub-commands, multiple arguments and flags, for example:

Using the CLI to restore the blockchain address from a mnemonic phrase:

> ./marvin-blockchain address restore --mnemonic 'unhappy describe tuna century because antique close trash bike bread crater notable'
address: f650a946e9ddedcf9ba36105eba9a59d5dd0992d

But I also can have the sub-command create under address , to create a new blockchain address and mnemonic:

> ./marvin-blockchain address create
Generating new address...
mnemonic: interest issue wolf swap father predict define exercise coral forum depart slide
address: 9b1c6518c0906db58f39e5c1410b02b7e486ef80

And if I had to build that from scratch with just the standard library would take focus out of the main target of the project, which is to create a blockchain and not a CLI application from scratch.

That said, I looked into a couple of options of CLI packages for Go. The top 3 ones I looked into it were urfave/cli, alecthomas/kong and finally spf13/cobra which I ended up picking. All of them were alright, I guess urfave/cli and alecthomas/kong are lighter libraries, but at the end of the day I went with cobra for being more feature rich, it is battle tested by a lot of other big cli applications out there and it is the one I am more familiar with.

Adding a CLI Interface to Rust

For Rust the reasoning was the same on why not to use the Standard Library, so I won’t spent time explaining again.

What is worth mentioning is that in the case of Rust I only looked at one crate, and that was Clap. I have used Clap before, it is a battle tested Crate used by a number of large projects and it’s API is super easy to use. On top of being super feature rich. So for me it was a no brainer in choosing Clap for Rust.

Generating the private key and public key

One of the first things to look at when generating a key pair is to look at which digital signature algorithm to choose from and on this section we will dive into a couple of these algorithms, explain how they work, highlight their differences, explain the reasoning behind the one I choose and also dive a little into each of the implementations with Go and Rust.

By the end of this section you should have at least have a high level understanding of these algorithms and also have a good understanding of the ground work I am laying down that will be used later on during the development of the blockchain.

What were our choices?

The two most used algorithms for digital signatures in the blockchain space are the Elliptic Curve Digital Signature Algorithm (ECDSA) with the secp256k1 curve and the Edwards-curve Digital Signature Algorithm (EdDSA) with the curve25519 curve, aka ed25519.

ECDSA with secp256k1

ECDSA is a widely adopted asymmetric cryptographic algorithm used for digital signatures. It operates on elliptic curves and provides a secure mechanism for verifying the authenticity and integrity of data. The secp256k1 curve, specifically, is a popular elliptic curve used in Bitcoin and several other cryptocurrencies.

The secp256k1 curve has a specific mathematical structure that allows for efficient cryptographic operations. It is defined by the equation y² = x³ + 7 and operates over the finite field of prime numbers. ECDSA with secp256k1 utilizes the properties of this curve to generate public and private key pairs, sign messages, and verify signatures. It offers a high level of security and efficiency for cryptographic operations.

EdDSA with curve25519

EdDSA is another cryptographic algorithm based on elliptic curves, specifically the Edwards curves. It was designed to provide improved security, simplicity, and performance compared to earlier algorithms like ECDSA. One commonly used Edwards curve is the curve25519. Oh and it was invented after Bitcoin was created.

Curve25519 is an elliptic curve defined by the equation -x² + y² = 1 — (121665/121666) \ *x² * y². It operates over the prime field and is known for its efficiency and security properties. EdDSA with curve25519 leverages this curve to generate key pairs, sign messages, and verify signatures.

The main advantage of EdDSA is its resistance to certain types of side-channel attacks, improved performance, and enhanced security features. It has gained popularity in various blockchain platforms and cryptographic applications.

Differences between ECDSA secp256k1 and EdDSA curve25519

While both ECDSA with secp256k1 and EdDSA with curve25519 provide secure digital signatures, they differ in several aspects:

1. Curve Structure: ECDSA secp256k1 operates on the secp256k1 elliptic curve, while EdDSA curve25519 utilizes the curve25519 Edwards curve. These curves have different mathematical equations and properties.

2. Key Generation: ECDSA secp256k1 and EdDSA curve25519 have different mechanisms for key generation. ECDSA generates key pairs using the secp256k1 curve, while EdDSA uses the curve25519 curve.

3. Performance: EdDSA with curve25519 is generally more efficient in terms of computational resources and offers faster signature generation compared to ECDSA secp256k1. It achieves this efficiency through the use of twisted Edwards curves.

4. Security Features: EdDSA is designed to provide enhanced security against certain types of attacks, such as certain side-channel attacks. It incorporates additional security measures compared to ECDSA.

So, which one I went with?

3…2…1… And the winner is?!!!

EdDSA with curve25519, AKA ed25519!

And here is why. To be honest in my opinion and based on my very limited knowledge of cryptography there isn’t a strong enough reason to switch in either direction, both of them are cryptographic algorithms widely used for digital signatures in blockchain applications.

So I basically made the choice based of two things:

• What do I have more experience and exposure with?

The answer to this one would be definitely ed25519 .

• Do I have stable and battle tested packages and crates that I could use?

And based on my previous answer I just had to find to make sure I stable enough libraries for both languages, Go and Rust. And the answer is yes.

For Go there is an implementation ofed25519 that is part of the Standard Library and can be found at [crypto/ed25519](https://pkg.go.dev/crypto/ed25519) And if the case of Rust we there is an excellent native Ed25519 implementation called [ed25519-dalek](https://crates.io/crates/ed25519-dalek) . Which has fast and efficient ed25519 EdDSA key generations, signing, and verification in pure Rust.

With the libraries chosen I will now talk about the APIs I created for both Go and Rust and will see that they are actually pretty similar. At a high level this is what you will find on both codebases.

Go API Implementation

Constants used throughout:

const (
    privateKeySize = ed25519.PrivateKeySize // 64
    publicKeySize = ed25519.PublicKeySize // 32
    signatureSize = ed25519.SignatureSize // 64
    seedSize = 32
    addressSize = 20
)

Private Key Implementation:

// PrivateKey represents a private key for the Ed25519 signature scheme.
type PrivateKey struct {
    key ed25519.PrivateKey
}

// GetMnemonicFromEntropy generates a new mnemonic from a given entropy.
func GetMnemonicFromEntropy(entropy []byte) string {...}

// NewPrivateKeyfromMenmonic creates a new private key from a given mnemonic.
func NewPrivateKeyfromMnemonic(mnemonic string) PrivateKey {...}

// SeedFromMnemonic creates a new seed from a given mnemonic.
func SeedFromMnemonic(mnemonic string) []byte {...}

// NewPrivateKeyfromString creates a new private key from a given hex string.
func NewPrivateKeyfromString(s string) PrivateKey {...}

// NewPrivateKeyFromSeed creates a new private key from a given seed byte slice.
func NewPrivateKeyFromSeed(seed []byte) PrivateKey {...}

// Bytes creates a new private key from a byte slice.
func (p *PrivateKey) Bytes() []byte {...}

// Sign signs the data with the private key.
func (p *PrivateKey) Sign(data []byte) *Signature {...}

// PublicKey returns the public key for the private key.
func (p *PrivateKey) PublicKey() *PublicKey {...}

Public Key Implementation:

// PublicKey represents a public key for the Ed25519 signature scheme.
type PublicKey struct {
    key ed25519.PublicKey
}

// Address returns the address for the public key.
func (p *PublicKey) Address() Address {...}

// Bytes creates a new public key from a byte slice.
func (p *PublicKey) Bytes() []byte {...}

Signature Implementation:

// Signature represents a signature for the Ed25519 signature scheme.
type Signature struct {
    value []byte
}

// Bytes creates a new signature from a byte slice.
func (s *Signature) Bytes() []byte {---}

// Verify verifies the signature of the data with the public key.
// It returns true if the signature is valid, and false otherwise.
func (s *Signature) Verify(publicKey *PublicKey, data []byte) bool {...}

Address Implementation:

// Address represents an address for the Ed25519 signature scheme.
type Address struct {
    value []byte
}

// Bytes creates a new address from a byte slice.
func (a *Address) Bytes() []byte {...}

// String returns the address as a hex string.
func (a Address) String() string {...}

Rust API Implementation

Constants used throughout:

const PRIVATE_KEY_SIZE: usize = ed25519_dalek::SECRET_KEY_LENGTH;
const PUBLIC_KEY_SIZE: usize = ed25519_dalek::PUBLIC_KEY_LENGTH;
const SIGNATURE_SIZE: usize = ed25519_dalek::SIGNATURE_LENGTH;
const ADDRESS_SIZE: usize = 20;
const SEED_SIZE: usize = 32;

Private Key Implementation:

pub struct PrivateKey {
    pub key: SigningKey,
}

/// new_entropy generates a new 16 byte entropy
pub fn new_entropy() -> [u8; 16] {...}

/// get_mnemonic_from_entropy generates a new mnemonic from the given entropy
pub fn get_mnemonic_from_entropy(entropy: &[u8; 16]) -> String {...}

/// get_seed_from_mnemonic generates a new seed from the given mnemonic
pub fn get_seed_from_mnemonic(mnemonic: &str) -> [u8; SEED_SIZE] {...}

/// get_private_key_from_mnemonic generates a new private key from the given mnemonic
pub fn get_private_key_from_mnemonic(mnemonic: &str) -> PrivateKey {...}

/// new_private_key_from_string generates a new private key from the given private key string
pub fn new_private_key_from_string(private_key: &str) -> PrivateKey {...}

/// new_private_key_from_seed generates a new private key from the given seed
pub fn new_private_key_from_seed(seed: &[u8; 32]) -> PrivateKey {...}

/// generate_private_key generates a new private key with a random seed
/// (Good for testing purposes)
pub fn generate_private_key() -> PrivateKey {...}

impl PrivateKey {
    pub fn sign(&mut self, data: &[u8]) -> SignatureWrapper {...}

    /// Sign a message with the private key
    pub fn to_bytes(&self) -> [u8; PRIVATE_KEY_SIZE] {...}

    /// Sign a message with the private key
    pub fn public_key(&self) -> PublicKey {...}
}

Public Key Implementation:

pub struct PublicKey {
    pub key: VerifyingKey,
}

impl PublicKey {
    // to_bytes creates a new public key from a byte slice.
    pub fn to_bytes(&self) -> [u8; PUBLIC_KEY_SIZE] {...}

    /// address returns the address for the public key.
    pub fn address(&self) -> Address {...}
}

SignatureWrapper Implementation:

pub struct SignatureWrapper {
    pub signature: Signature,
}

impl SignatureWrapper {
    /// Convert the signature to bytes
    pub fn to_bytes(&self) -> [u8; SIGNATURE_SIZE] {...}

    /// Verify a message with the public key
    pub fn verify(&self, data: &[u8], public_key: &PublicKey) -> bool {...}

Address implementation:

pub struct Address {
    pub value: [u8; ADDRESS_SIZE],
}

impl Address {
    /// Convert the address to a string in hex format
    pub fn to_string(&self) -> String {...}

    /// Convert the address to bytes
    pub fn to_bytes(&self) -> [u8; ADDRESS_SIZE] {...}
}

The full code you will find it on Github (Go / Rust), but I will talk about some key points and some of the differences so far between Go and Rust. Also bare in mind that this code will probably be refactored and improved also as the project evolves.

Signing and Veryfing

The good thing of planing and building a good API is that you are laying the ground work for what is to come, and you are making things easier for yourself. And this is exactly the case here.

At the end of the day everything in a blockchain is just a chunk of bytes, and that could be a transaction, a block or some other piece of private information. What that is doesn’t really matter, what matters is that you are able to Sign and Verify with your Private Key and Public Key. So here we go.

I think the best way to exemplify is just to show the Unit Test I created that does exactly that:

Go:

func TestPrivateKeySign(t *testing.T) {
    privateKey := GeneratePrivateKey() // Generates a private key
    publicKey := privateKey.PublicKey() // Retrives the public key
    invalidPrivateKey := GeneratePrivateKey() // Generates another private key
    invalidPublicKey := invalidPrivateKey.PublicKey() // Retrievs another public key

    data := []byte("hello, world") // Any data converted into a []byte
    // Signs the data with your PrivateKey and generates a Signature Type
    signature := privateKey.Sign(data)

    // Check that the signature is the correct size
    assert.Equal(t, signatureSize, len(signature.value))

    // Check that the signature is valid
    assert.True(t, signature.Verify(publicKey, data)) // Returns True

    // Check that the signature is invalid
    assert.False(t, signature.Verify(invalidPublicKey, data)) // Returns False
}

Rust:

#[test]
fn test_private_key_sign() {
    // Generates a private key
    let mut private_key = generate_private_key();
    // Retrives the public key
    let public_key: PublicKey = private_key.public_key();
    // Generates another private key
    let invalid_private_key = generate_private_key();
    // Retrievs another public key
    let invalid_public_key: PublicKey = invalid_private_key.public_key();

    let data = b"hello world"; // Any data converted into a []byte
    // Signs the data with your PrivateKey and generates a SignatureWrapper Type
    let signature = private_key.sign(data);

    // Verify the signature size
    assert_eq!(SIGNATURE_SIZE, signature.signature.to_bytes().len());

    // Verify the signature - Return True
    assert_eq!(true, signature.verify(data, &public_key));

    // Verify the signature is invalid - Return False
    assert_eq!(false, signature.verify(data, &invalid_public_key));
}

As you can see we just made use of the API that we created so far and very easily we were able to sign and verify a chunk of data.

By now you can start to see how this can all come together. Since when we create a Private and Public Key we can also output a Mnemonic phrase, we can easily retrieve the Private Key from that Mnemonic again in the future and use that to sign transactions, blocks and etc. And that is why is so important for you to keep your Mnemonic Secrets from your wallet actually secret. If someone gets their hand on that, they basically own your wallet.

(Obviously there other security mechanisms that can also be used, but just in case, store your Mnemonic phrase in a place no one will find.

Comparing Go and Rust

One of the things you may have already noticed is that the way we declare methods is very different between Go and Rust. When it comes to the definition no matter the language, they are always going to be similar. This is the one I use.

Methods are similar to functions but methods are defined within the context of a user defined type and provide a way to add behavior to those user-defined types. The way they are declared are slightly different between the two languages. Let’s look at a live example:

Let’s look at the implementation of the method to sign a chunk of data that is associated with the user defined type Private Key .

In Go:

// Sign signs the data with the private key.
func (p *PrivateKey) Sign(data []byte) *Signature {
    return &Signature{
        value: ed25519.Sign(p.key, data),
    }
}

In Rust:

impl PrivateKey {
    /// Sign a message with the private key
    pub fn sign(&mut self, data: &[u8]) -> SignatureWrapper {
        SignatureWrapper {
            signature: self.key.sign(data),
        }
    }
}

Don’t mind the implementation of the method it self, since that is not the focus right now. But do you notice the differences?

In Go, all you need to do it to add that extra parameter that is declared between the keyword func and the function name, and also enclose that with parentheses. You also have the option of making that a reference value or a pointer value, usually depending on wether or not you will be updating any values of the instance.

Now in Rust you do not need that extra parameter, at least not in the same place. But you do need that the first parameter is always &self, which represents the instance of the user defined type the method is being called on. And on top of that all your methods associated with that user defined type need to within an impl block. The &self is actually short for self: &Self. Within an impl block, the type Self is an alias for the type that the impl block is for. Methods must have a parameter named self of type Self for their first parameter, so Rust lets you abbreviate this with only the name self in the first parameter spot. Note that we still need to use the & in front of the self shorthand to indicate that this method borrows the Self instance.

If you have trouble understanding ownership in Rust I highly recommend reading this chapter in The Rust Programming Language Book.

What now?!

On this post we have walked through which packages and crates I used for my CLI interface as well as why I decided to add it to the project at this stage. We also talked about the two most widely use digital signature algorithms used by blockchains today and why I decided to go with the ed25519 for this project. After that I explained what was built so far related to the key pair generation and also how to easily sign and verify chunks of data with those same key pairs. And to end it all out we finished with a quick comparison of Go and Rust and how the two languages approach declaring and implementing methods.

Don’t forget to Like and Subscribe here on Medium to be notified about the next posts in the series and also STAR the projects on Github to follow what is going on over there.

View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

Cheers!

Part 1 — Crafting a Blockchain in Go and Rust: A Comparative Journey — Private keys, Public Keys and Signatures ->

What I cannot create, I do not understand. — Richard Feynman

Anyway, yes I decided to write from scratch a Blockchain in both Go and Rust.

Introduction

In order to do this I need to emphasise the importance of understanding difference between the two languages and how this comparative journey can highlight their strengths and weaknesses.

The goal is to implement the same blockchain in both languages, but taking advantage of what each language has to offer so the implementation, libraries used may differ but the goal is to be able to offer the same features and user experience in both.

Motivation

Programmers love to compare programming languages. “This one is faster”, “That one is easier to understand”, “Ohh but this other one is typed and you gotta have types”. Well, I actually believe that every programming language has a purpose and there is no silver bullet. You just gotta use the best tool for the job.

With that in mind, have you ever watched Ocean’s Twelve? The one where the thieve challenges Ocean’s Crew to steal the same object, because it would be the only way to definitively decide who is the better thieve.

Well, this is where I got the idea from. I love Go, but I also love Rust and I wanted to build a blockchain from scratch to better understand the in’s and out’s of a blockchain at a deeper level, like I did before with my SQLRite project where I build a database from scratch in Rust.

These types of projects, specially a blockchain, are great because they touch a wide range of topics from proper understanding of data structures, cryptography, networking, distributed computing, consensus algorithms, EVM’s and the list goes on.

How addresses are actually generated and secured?

What is the best consensus algorithm?

Any networking limitations?

How is the data stored and is there a better way?

What do I need to make it EVM compatible?

Can I actually get to a point where I can deploy a EVM Compatible smart contract on my own blockchain?

Well, these are some of the questions I aim to answer along the way, and there are a lot more.

What to expect

I wrote down a list of features and also somewhat of a Roadmap, based on my initial research and expectation but of course that is subject to change since I am diving into unexplored waters here (for me at least). With that said, I am open to any suggestions and even contributions of whoever want to lend a hand, as long as some knowledge is also shared in the process. It’s all welcome!

Here it is!

Features (WIP)

• Proof of Work (PoW) Consensus Mechanism (Subject to Change)

• Peer-to-Peer (P2P) Networking

• Storage and Persistence

• Transaction and Block Validation

• Smart Contract Support via EVM Compatibility

• JSON-RPC API

• Comprehensive Unit Tests and Benchmarks

Roadmap (WIP)

Not necessary prioritized in the order of development

[ ] Add CLI support for ease of interaction
[ ] Implement the basic blockchain data structure
[ ] Proof of Work (PoW) consensus mechanism
[ ] Peer-to-Peer (P2P) networking implementation (transport layer)
[ ] Basic transaction and block validation
[ ] Storage and persistence for blockchain data
[ ] EVM integration for smart contract support
[ ] JSON-RPC API implementation
[ ] Advanced transaction handling and validation
[ ] Enhanced security measures and best practices
[ ] Performance benchmarking and optimization
[ ] Comprehensive documentation and examples
[ ] Implement wallet functionalities
[ ] Improve EVM compatibility and support
[ ] Add more consensus mechanisms (e.g., PoS)
[ ] Implement light client support
[ ] Improve network protocol for better scalability
[ ] Develop a robust test suite for security and performance
[ ] Integration with Ethereum development tools
[ ] Develop a block explorer
[ ] Implement governance mechanisms
[ ] Cross-chain interoperability solutions
[ ] Improve documentation and developer guides

Sorry if some of the Roadmap does not make much sense now, but creating a blockchain from scratch is no small task so there is a lot in my mind right now and I am trying to keep things fairly organised, but well, not everything will be perfect at first. The goal is to get the ball rolling and keep re-iterating again and again and adjust as we go.

At the time of publishing this, there is not much to see yet other than a very nice README, since I’ve been focusing mostly on research, but if you want to follow the project don’t forget to CLAP this post and also STAR the projects on Github.

I wonder which project will get more stars…

View on Github (Pull Requests are Welcome) — Rust Version

View on Github (Pull Requests are Welcome) — Go Version

Cheers!

Very simply, a smart pointer is an abstract data type that simulates a pointer while providing added features, such as automatic memory management and bounds checking. These are intended to reduce bugs caused by the misuse of pointers, while retaining efficiency.

Before we talk about how Rust deals with Smart Pointers, let’s go back a little and take a look at the history and where smart pointers came from.

A little bit of history

Smart pointers were first popularized in the programming language C++ during the first half of the 1990s as rebuttal to criticisms of C++’s lack of automatic garbage collection.

Even though the concept of smart pointers was popularized with C++, especially the reference-counted type of smart pointers, the immediate predecessor of one of the languages that inspired C++’s design had reference-counted references built into the language. C++ was inspired in part by Simula67 and Simula67’s ancestor was Simula I. Insofar as Simula I’s element is analogous to C++’s pointer without null, and insofar as Simula I’s process with a dummy-statement as its activity body is analogous to C++’s struct. Simula I had reference counted elements (i.e., pointer-expressions that house indirection) to processes (i.e., records) no later than September 1965, as shown in the quoted paragraphs below.

Processes can be referenced individually. Physically, a process reference is a pointer to an area of memory containing the data local to the process and some additional information defining its current state of execution.

Because C++ borrowed Simula’s approach to memory allocation — the new keyword when allocating a process/record to obtain a fresh element to that process/record — it is not surprising that C++ eventually resurrected Simula’s reference-counted smart-pointer mechanism within element as well.

Smart Pointers in C++

Very simply, a smart pointer in C++ is a class with overloaded operators, which behaves like a conventional pointer. Yet, it supplies additional value by ensuring proper and timely destruction of dynamically allocated and facilitates a well-defined object lifecycle.

The Problem with Using Conventional (Raw) Pointers in C++

Unlike many other programming languages, C++ provides full flexibility to the programmer in memory allocation, deallocation and management. Unfortunately, this flexibility is a double-edged sword. On one side it makes C++ a powerful language, but on the other hand it allows the programmer to create problems related to memory-management., such as memory leaks, specially when dynamically allocated objects are not released at the right time.

Here is an example:

SomeClass* pointerData = anObject.GetData();

pointerData->DoSomething();

In the above example, there is no obvious way to tell whether the memory pointer to pointerData

• Was allocated on the heap , and needs to be deallocated

• Is the responsibility of the called to deallocate

• Will pointerData be automatically be destroyed by the object’s destructor

You could say something like. “The programmer just has to be careful and follow proper best practices”. In an ideal world that would be enough, but as you probably already know, in the real world it isn’t. That is why we need some mechanisms to protect us from ourselves.

How do Smart Pointers Help in C++?

Just to be clear, even with all the conventional pointer and conventional memory management techniques, the C++ programmer is not forced to use any of them when he needs to manage data on the heap/free store. The programmer can choose a smarter way to allocate and manage dynamic data by adopting the use of smart pointers in his programs:

smart_pointer smartPointerData = anObject.GetData();
smartPointerData->Display();
(*smartPointerData).Display();

// No need to worry about deallocation
// The Smart Pointer destructor will take care of that for you.

By looking at it, smart pointers may behave like a conventional pointer, but in reality they supply useful features via their overloaded operators and destructors to ensure that dynamically allocated data is destroyed in a timely manner.

Types of Smart Pointers in C++

The management of the memory resource (that is, the ownership model implemented) is what sets smart pointer classes apart. Smart Pointers will decide what to do with the resource when they are copied and assigned to. The simplest implementation often results in not the best performance, whereas the fastest ones might not suit all applications. At the end, it is up to the programmer to understand its needs before he or she decides to use it in his or hers program.

The classification of smart pointers in C++ is basically the classification of their memory resource management strategies. These are:

• Deep copy

• Copy and Write (COW)

• Reference counted

• Reference linked

• Destructive copy

I am not going to dive into what each of these are, because although the concepts are similar, they are C++ related and I think this is enough of C++ for now. Otherwise we will start to get too much into the weeds of C++ which was not the purpose of this post. The purpose until now was to give a little bit of background on where Smart Pointers are and where they came from.

Now it is time to dig into Rust Smart Pointers!!

Smart Pointers in Rust

We have been talking about Smart Pointers this, Smart Pointers that, but what about a Pointers.

What in the hell is a Pointer?

Well, a pointer is a variable that contains an address in memory, It points to, or refers to some other data, so the pointer variable it self does not contain the actual data. You can think of it like an arrow to that value.

pointer variable pointer_to_value referencing a point in memory with a value

Let me try to explain with an example. Let’s imagine that our friend John Smith has invited me to visit him at this house and he lives in one of those Gated Communities and our friend gives us the address of the community.

So now we know the general location where he lives, just like your program knows generally where the temporary data for your application would be stored in memory, being the Stack or the Heap.

Once we get to the gated community, we need to retrieve the specific address of his house so when we get to the front gate, we ask the guard in which house our friend John Smith lives. The guard has stored the address to our friend’s house, just like your application would request the address to a specific piece of data from a pointer variable.

Pointers are quite simple in theory, they simply point us to the address where our data is in memory, much like the guard points us to our friends house in the gated community.

Alright, that is enough about pointers in general.

What about pointers in Rust?

Well, Rust has two regular types of pointers called references. They are recognized by ampersand in front of the variable name.

& for an immutable reference. (which is the default behaviour by the way)

fn my_function(my_variable: &String) {
    // do something
}

&mut for a mutable reference.

fn my_function(my_mutable_variable: &mut String) {
    // do something
}

References to a value don’t actually own the value (usually), the just borrow it. In other words, the reference can disappear and the value it pointed at will still exist. This is related to the concept of Ownership in Rust, which I am not going to get into because this rabbit hole is already gone too deep and we need to start climbing our way out.

Finally! Smart Pointers in Rust!

The references I explained above are regular pointers, that only point to some piece of data, but they don’t do much else. Smart Pointers in Rust are actually data structures that not only act like a pointer, but also have additional metadata and extra features. Features that a regular pointer would not have.

You could say that they are smart cookies…

Smart Pointer studying up

Smart Pointers are usually implemented using structs.

One difference between regular references and Smart Pointers in Rust is that references only borrow the data, while in most cases a smart pointer will own the data they point to. In other words, when the smart pointer gets dropped, the data they point to gets dropped.

Another important point is that a Smart Pointers implement the [Deref](https://doc.rust-lang.org/std/ops/trait.Deref.html) and [Drop](https://doc.rust-lang.org/std/ops/trait.Drop.html) traits. The [Deref](https://doc.rust-lang.org/std/ops/trait.Deref.html) trait allows an instance of the smart pointer struct to behave like a reference so you can write code that works with either references or Smart Pointers. The [Drop](https://doc.rust-lang.org/std/ops/trait.Drop.html) trait allows you to customize the code that is run when an instance of the Smart Pointer goes out of scope, which can be very useful.

I am not going into detail into every type of Smart Pointer in Rust. But I do want to cover the most common ones existing in the standard library at least.

• Box<T> for always allocating values on the heap

• Rc<T>, a reference counting type that enables multiple ownership

• Ref<T> and RefMut<T>, accessed through RefCell<T>, a type that enforces the borrowing rules at runtime instead of compile time (to be avoided when possible).

Box

Box is the most simple Smart Pointer and it is used to store data on the heap and not on the stack, even when the data size is known at compile time. The Box Smart Pointer it self will be stored on the Stack, but the data it points to will be stored on the Heap.

Box comes in handy when working with recursive types for example. At compile time the Rust compiler wants to know how much space will be required by a type, and that becomes difficult when working with recursion as in theory, it can be infinite. So, we use Box type to provide the size of the type so that compiler knows how much memory is to be allocated.

enum List {
    Cons(i32, Box),
    Nil,
}

use List::{Cons, Nil};

fn main() {
    let list = Cons(1,
        Box::new(Cons(2,
            Box::new(Cons(3,
                Box::new(Nil))))));
}

In the above example, the Cons variant needs the size of i32 plus space to store the box pointer data. By using the box we have broken the infinite recursive chain and compiler can now figure out the size of List.

Rc

Rc stands for Reference Counted, and is this type of Smart Pointer is used to enable multiple ownership of the data, which by default is not allowed by the Ownership Rust model. The Rc smart pointer type keeps track of the number of references to a value from which we can know how many places our variable is being used. If the reference count gets down to zero then the value is not used anywhere so the value can be cleaned up safely from memory without causing any trouble.

enum List {
    Cons(i32, Rc),
    Nil,
}

use List::{Cons, Nil};
use std::rc::Rc;

fn main() {
    let a = Rc::new(Cons(5, Rc::new(Cons(10, Rc::new(Nil)))));
    println!("count after creating a = {}", Rc::strong_count(&a));
    let b = Cons(3, Rc::clone(&a));
    println!("count after creating b = {}", Rc::strong_count(&a));
    {
        let c = Cons(4, Rc::clone(&a));
        println!("count after creating c = {}", Rc::strong_count(&a));
    }
    println!("count after c goes out of scope = {}", Rc::strong_count(&a));
}

Outputs:

count after creating a = 1
count after creating b = 2
count after creating c = 3
count after c goes out of scope = 2

Here we can see that the Rc count is 1 when we create variable a, after that, when we create another variable b by cloning variable a, the count goes up by 1 to 2. Similarly, when we create another variable c by again cloning variable a, the count further goes up by 1 to 3. After c goes out of scope count goes down by 1 to 2.

RefCell

RefCell is what makes it possible to work with the Interior Mutability Pattern which is a design pattern in Rust that allows you to mutate data even when there are one or more immutable references to the data, which goes completely against the borrowing rules declared in the Ownership model. To be able to do that, the pattern uses unsafe code inside a data structure to bend Rust’s rules about mutation and borrowing.

If you are interesting in learning more about working with unsafe code, you can check it out the book The Rustonomicon.

A common way to use Refcell is in combination with Rc which is a reference counter. If we have multiple owners of some data and we want to give access to mutate data then we have to use Rc that hold a Refcell.

#[derive(Debug)]
enum List {
    Cons(Rc, Rc),
    Nil,
}

use List::{Cons, Nil};
use std::rc::Rc;
use std::cell::RefCell;

fn main() {
    let value = Rc::new(RefCell::new(5));

    let a = Rc::new(Cons(Rc::clone(&value), Rc::new(Nil)));

    *value.borrow_mut() += 10;

    println!("a after = {:?}", a);
    }

Outputs:

a after = Cons(RefCell { value: 15 }, Nil)

In the example above, we have created an instance of Rc<Refcell> and store it in a variable named value and also we created a list a with a cons variant that holds the value. We cloned value so that both a and value have ownership of inner cell which has a value of 5 rather than transferring ownership from value. After that, we add 10 in value by calling borrow_mut() on value and this method return RefMut smart pointer and we use reference operator on it to change the inner value of it.

Conclusion

One of my favorite quotes is “What I cannot create, I do not understand” from Richard Feynman and of course that you are not going to re-create everything and start reinventing the wheel over and over again. But I do believe that it is of the most importance to at least understand how things work up to a certain level. I hope that by understanding what are Smart Pointers, where they came from and what are the different types of it, you can use this knowledge to design better system and make better decisions in the future.

Another nice quote that I like is “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.” from Linus Torvalds, and Smart Pointers are basically a specific type of data structure that relates to data in a certain way, so understanding can only make you a better programer and I hope I could help at least a little.

Don’t forget to follow and hit that clap if you like the content.

Additional sources:

• The Rustonomicon.

• The Rust Programming Book

• Book Sams Teach Yourself C++ in One Hour a Day

In this blog post, we’ll explore three essential smart pointers: std::unique_ptr, std::shared_ptr, and std::weak_ptr. We'll discuss what they are, how they work, and when to use them, complete with code examples and use cases.

std::unique_ptr: Exclusive Ownership

std::unique_ptr represents exclusive ownership of a dynamically allocated object. It ensures that there is only one std::unique_ptr pointing to an object at any given time. When the std::unique_ptr goes out of scope or is explicitly reset, the object it points to is automatically deleted.

std::unique_ptr is useful when you want to enforce a single-owner policy and ensure automatic deletion of the object once it is no longer needed. std::unique_ptr cannot be copied, but it can be moved, which transfers ownership of the underlying object to another std::unique_ptr.

Example:

#include <iostream>
#include <memory>

class MyClass 
{
public:
    MyClass() 
    { 
      std::cout << "MyClass constructor\n"; 
    }

    ~MyClass() 
    { 
      std::cout << "MyClass destructor\n"; 
    }
};

int main() 
{
    std::unique_ptr<MyClass> uptr(new MyClass());
    // Transfer ownership
    std::unique_ptr<MyClass> another_uptr = std::move(uptr); 
}

In this example, uptr takes exclusive ownership of the dynamically allocated MyClass object. When we move uptr to another_uptr, the ownership is transferred, and the MyClass object is automatically destroyed when another_uptr goes out of scope.

std::shared_ptr: Shared Ownership

std::shared_ptr represents shared ownership of a dynamically allocated object. Multiple std::shared_ptrs can point to the same object, and the object is automatically deleted when the last std::shared_ptr that points to it goes out of scope or is reset.

std::shared_ptr uses reference counting to keep track of the number of shared_ptrs referencing the object. When the reference count becomes zero, the object is deleted. std::shared_ptr is useful when you want to share ownership of an object among multiple entities and ensure automatic deletion once all the shared_ptrs go out of scope or are reset.

Example:

#include <iostream>
#include <memory>

class MyClass 
{
public:
    MyClass() 
    { 
      std::cout << "MyClass constructor\n"; 
    }
    ~MyClass() 
    { 
      std::cout << "MyClass destructor\n"; 
    }
};

void use_shared_ptr(std::shared_ptr<MyClass> sptr) 
{
    // Do something with sptr
}

int main() 
{
    std::shared_ptr<MyClass> sptr1(new MyClass());
    // Share ownership
    std::shared_ptr<MyClass> sptr2 = sptr1; 

    use_shared_ptr(sptr1);
}

In this example, sptr1 and sptr2 share ownership of the MyClass object. When both sptr1 and sptr2 go out of scope, the object is automatically destroyed.

std::weak_ptr: Non-owning Reference

std::weak_ptr is used in conjunction with std::shared_ptr. It holds a non-owning reference to a shared object, meaning it does not contribute to the reference count.

The primary use of std::weak_ptr is to break circular references between shared_ptrs, which can lead to memory leaks. To access the underlying object, you need to convert the std::weak_ptr to a std::shared_ptr by calling its lock() method. If the object has already been deleted, the lock() method returns an empty std::shared_ptr.

Example:

#include <iostream>
#include <memory>

class MyClass 
{
public:
    MyClass() 
    { 
      std::cout << "MyClass constructor\n"; 
    }
    ~MyClass() 
    { 
      std::cout << "MyClass destructor\n"; 
    }
};

void use_weak_ptr(std::weak_ptr<MyClass> wptr) 
{
    if (auto locked_sptr = wptr.lock()) 
    {
        // Use locked_sptr, which is a std::shared_ptr
        std::cout << "Object still exists\n";
    } 
    else 
    {
        std::cout << "Object has been deleted\n";
    }
}

int main() 
{
    std::shared_ptr<MyClass> sptr(new MyClass());
    std::weak_ptr<MyClass> wptr = sptr;

    use_weak_ptr(wptr);

    // Release ownership
    sptr.reset(); 

    use_weak_ptr(wptr);
}

In this example, wptr holds a non-owning reference to the MyClass object. We can use the lock() method to temporarily access the object through a shared_ptr. When the shared_ptr sptr is reset and releases ownership, wptr becomes invalid, and the lock() method returns an empty shared_ptr.

Conclusion:

std::unique_ptr, std::shared_ptr, and std::weak_ptr are powerful tools for managing object lifetimes in C++ applications. Each of these smart pointers serves a specific purpose: std::unique_ptr for exclusive ownership, std::shared_ptr for shared ownership with reference counting, and std::weak_ptr for non-owning references to shared objects.

By understanding and using these smart pointers, you can write more efficient, maintainable, and bug-free code in your C++ projects.

Happy developing!

Be sure to leave a LIKE and of course comments are always welcome!

Cheers!

In this blog post, we’ll walk through the process of developing and deploying a simple Rust application on Kubernetes. This tutorial is aimed at beginner to intermediate Rust developers who want to learn more about using Rust with Kubernetes.

Here are some prerequisites…

• Basic understanding of Rust programming and the Rust toolchain

• Familiarity with Docker and containerisation

• Some experience with Kubernetes and its concepts

Let's get started!

Step 1: Creating a simple Rust application

First, let’s create a simple Rust application that serves an HTTP endpoint. We’ll use the [warp](https://github.com/seanmonstar/warp) web framework for this purpose. Create a new Rust project using cargo:

$ cargo new rust_on_k8s
$ cd rust_on_k8s

Add the following dependencies to your Cargo.toml file:

[dependencies]
warp = "0.3"
tokio = { version = "1", features = ["full"] }

Now, let's go and replace the contents of src/main.rs with the following code:

use warp::Filter;

#[tokio::main]
async fn main() {
    let hello = warp::path!("hello" / String)
    .map(|name| format!("Hello, {}!", name));

    let routes = hello;
    println!("Starting server at 127.0.0.1:8000");
    warp::serve(routes).run(([127, 0, 0, 1], 8000)).await;
}

This simple application in only 10 lines of code (counting empty lines :P), listens on port 8000 and echos with a greeting when you visit /hello/your_name.

To test the application, run and try to access using your browser:

$ cargo run

Step 2: Dockerizing the Rust application

To deploy our Rust application on Kubernetes, we need to containerize it using Docker. Create a Dockerfile in the root of the project with the following content:

# Using the official Rust image as a builder

FROM rust:1.58 as builder

WORKDIR /usr/src/app

COPY . .

RUN cargo build --release

# Using a minimal image for the runtime
FROM debian:buster-slim

COPY --from=builder /usr/src/app/target/release/rust_on_k8s /usr/local/bin/rust_on_k8s

CMD ["rust_on_k8s"]

Build the Docker image:

$ docker build -t rust_on_k8s:latest .

Step 3: Deploying the Rust application on Kubernetes

To deploy the Rust application on Kubernetes, create a file named rust_on_k8s.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: rust-on-k8s
spec:
    replicas: 3
selector:
    matchLabels:
        app: rust-on-k8s
template:
metadata:
    labels:
        app: rust-on-k8s
spec:
    containers:
        - image: rust_on_k8s:latest
        name: rust-on-k8s
        ports:
        - containerPort: 8000

---
apiVersion: v1
kind: Service
metadata:
    name: rust-on-k8s
ports:
- port: 80
protocol: TCP
targetPort: 8000
spec:
    selector:
        app: rust-on-k8s
    type: LoadBalancer

This YAML file defines a Deployment and a Service. The Deployment specifies that we want three replicas of our Rust application, each running in a separate container. The Service creates a load balancer to distribute incoming traffic evenly among the replicas and expose the application on port 80.

Before applying this configuration, you need to push the Docker image to a container registry accessible by your Kubernetes cluster. For example, you can use Docker Hub, Google Container Registry, or Amazon Elastic Container Registry.

Here is some information on docker push: https://docs.docker.com/engine/reference/commandline/push/

Assuming you’ve pushed the Docker image to Docker Hub, update the `image` field in the `rust_on_k8s.yaml` file to point to the correct image:

containers:
- name: rust-on-k8s image: your_dockerhub_username/rust_on_k8s:latest ports: - containerPort: 8000

Now, apply the configuration to your Kubernetes cluster:

$ kubectl apply -f rust_on_k8s.yaml

After a few moments, your Rust application should be up and running on Kubernetes. You can check the status of the deployment and service with the following commands:

$ kubectl get deployments $ kubectl get services

When the service shows an external IP address, you can access the Rust application by navigating to [http://EXTERNAL_IP/hello/your_name](http://EXTERNAL_IP/hello/your_name.).

Conclusion

In this blog post, we walked you through creating a simple Rust application, containerising it with Docker, and deploying it on Kubernetes. Of course that the application we developed and deploy is quite simple and in a production large scale environment there is a lot more we would need to worry about, but at least here we managed to cover the base that you can build on top of.

By combining Rust’s safety and performance with Kubernetes’ scalability and ease of management, you can easily build powerful, secure, and reliable applications for modern cloud environments.

Keep experimenting and learning about Rust and Kubernetes to unlock their full potential, and don’t forget to share your knowledge with the community!

Happy developing!

Be sure to leave a LIKE and of course comments are always welcome!

Cheers!

I’ve been writing C++ on and off professionally for maybe 10 years now, and started to use Rust about 2 to 3 years ago.

Here I’d like to share my experience and thoughts on the transition between the two languages with code examples to illustrate the differences.

Disclaimer: This article is not a C++ vs Rust comparison. It’s just my personal experience and opinions about things that are important to me, not the engineering community in general.

The type of work you do influences your perception

I think that the type of work you do highly influences your perception about a language and technology.

With C++, I have spent a lot of my time writing tools for different systems and in the most recent years I have dove into game development with C++ also. Both of these applications can be very different in the way they assume ownership, manage memory, use system calls and interact with the kernel. The user experience is also very different depending on the case, where some case performance matters a lot more than others.

As for Rust I started my experience working on Open Source projects and took upon myself to develop an entire database from scratch which is the perfect application for the language, since memory management and ownership are critical for any database engine.

You can find the project here: https://github.com/joaoh82/rust_sqlite

Now to the important points:

Memory Safety: Ownership and Borrowing

One of the most well-known features of Rust is its memory safety guarantees. To illustrate this difference, let’s consider a simple example in C++ and Rust.

// C++
#include <iostream>
#include <vector>

void foo(std::vector<int>& vec) {
    vec.push_back(42);
}

int main() {
    std::vector<int> vec = {1, 2, 3};
    foo(vec);
    std::cout << vec.back() << std::endl;
    return 0;
}#include #include

// Rust
fn foo(vec: &mut Vec<i32>) {
    vec.push(42);
}

fn main() {
    let mut vec = vec![1, 2, 3];
    foo(&mut vec);
    println!("{}", vec.last().unwrap());
}

The Rust code is more verbose, but it’s also more explicit about the ownership and borrowing of resources. The &mut keyword in Rust ensures that the vec reference is mutable, and there's no risk of accidentally modifying the original data.

The Compiler:

Error messages from both C++ and Rust compilers can be daunting and necessitate effort to comprehend and address appropriately. However, the underlying reasons for this vary between the two languages.

In C++, error messages can be so lengthy that their size is measured in kilobytes. An infinite scroll feature in your terminal emulator becomes essential, as the compiler generates copious amounts of text. Over time, you develop an intuition for whether it’s more productive to read the error message or scrutinize your code based on the message’s size. Generally, the larger the error, the more effective it is to simply examine your code. Resolving this issue may necessitate altering the way C++ templates are defined.

Conversely, in Rust, encountering a compiler error (after addressing any glaring typos) often spells trouble. These errors typically suggest that you need to modify your code structure or invest time in refining lifetimes to ensure memory safety. While this process can be time-consuming and frustrating, it’s crucial to heed the compiler’s guidance. Embracing this humbling experience often results in superior code. Additionally, Rust’s error messages are concise enough to fit on a single screen, which is a pleasant bonus.

Build System: Cargo

Rust’s build system, Cargo, is a breath of fresh air compared to C++’s disparate build systems. To demonstrate this, let’s look at a simple example of adding a dependency.

In Rust, you just need to add a line in the Cargo.toml file:

[dependencies]
serde = "1.0.130"

In C++, you’d need to modify your build files, which could be CMake, Bazel, or something else, often requiring more steps and knowledge of each build system.

Type System: Generics and Enums

Rust’s type system allows for more expressive and safe code. Let’s look at an example of generics with traits in Rust compared to templates in C++.

// C++
#include <iostream>

template <typename T>
T add(T a, T b) {
    return a + b;
}

int main() {
    std::cout << add(1, 2) << std::endl;
    return 0;
}

// Rust
fn add<T: std::ops::Add<Output = T>>(a: T, b: T) -> T {
    a + b
}

fn main() {
    println!("{}", add(1, 2));
}

In Rust, traits clearly indicate the contract expected from the type, making the code easier to understand and reason about.

Enums in Rust are also more powerful than their C++ counterparts. Here’s an example using the Result enum:

// C++
#include <iostream>
#include <stdexcept>

int divide(int a, int b) {
    if (b == 0) {
        throw std::runtime_error("Division by zero");
    }
    return a / b;
}

int main() {
    try {
        std::cout << divide(10, 0) << std::endl;
    } catch (const std::runtime_error& e) {
        std::cerr << "Error: " << e.what() << std::endl;
    }
    return 0;
}

// Rust
fn divide(a: i32, b: i32) -> Result<i32, &'static str> {
    if b == 0 {
        Err("Division by zero")
    } else {
        Ok(a / b)
    }
}

fn main() {
    match divide(10, 0) {
        Ok(result) => println!("{}", result),
        Err(e) => eprintln!("Error: {}", e),
    }
}

In Rust, the Result enum is used as a common way to express fallible computations, making error handling more consistent and predictable across different libraries.

Conclusion

In conclusion, transitioning from C++ to Rust has been a positive experience for me. To be honest, I haven’t really transitioned I just now have Rust also under my belt as I have C++, since I still use both extensively. That said, rust’s memory safety guarantees, powerful type system, and consistent build system have made my day-to-day developer experience more enjoyable. While Rust’s syntax may be more verbose, the benefits it provides in terms of safety and expressiveness are well worth it.

Happy developing!

Be sure to leave a CLAP and of course comments are always welcome!

Cheers!

So the plan here is to start by defining what are Macros in a more general context and within the Rust language. Once we understand what they are we are going to take a deeper dive into the different types of macros we have in the Rust Programming Language and maybe throw in some examples in there, because theory without practice just does not stick. And to finish off the post I am going to talk about when to use Macros and some knows use cases out there.

What are Macros?

In a more general context a Macro is just a shorthand for a programming term: macroinstruction. In the most basic terms, a macro is a way to automate a simple task. They can be used in a variety of different ways, from Microsoft Excel Macros to Automating tasks on your operating system.

What are Macros in the context of Rust programming?

A quote directly out of the Rust Book, "Fundamentally, macros are a way of writing code that writes other code, which is known as metaprogramming." A lot of times in our day to day we write the exact same code over and over again and Macros are useful for reducing the amount of code you have to write and maintain, also helps to keep your code cleaner (if you use it right), which is also one of the roles of functions. However, macros have some additional powers that functions don’t, that we will get into it later in the blog post.

Let's look at an example. If you have been using Rust for more than a day you probably used the Macro println!, which as you may know prints a formatted text and add a \n (new line) at the end for you.

fn main() {
    println!("Hello there.")
}

But have you ever stoped to think what goes on behind the scenes when you write that? Have you ever thought about what you would have to do to actually print a text to the console with the println! macro? Well, if you would like to print a list of text without using the println! this is one way you could do it.

// use Write trait that contains write() function  
use std::io::Write;  

fn main() {  
    std::io::stdout().write(b"Hello, world!\n").unwrap();  
}

Can you imagine having to write all that every time you needed to print one single line of text to the console? No thanks!

Well, and if you are curious, this is what the println! macro definition actually looks like:

#[macro_export]  
#[stable(feature = "rust1", since = "1.0.0")]  
#[cfg_attr(not(test), rustc_diagnostic_item = "println_macro")]  
#[allow_internal_unstable(print_internals, format_args_nl)]  
macro_rules! println {  
    () => {  
        $crate::print!("\n")  
    };  
    ($($arg:tt)*) => {{  
        $crate::io::_print($crate::format_args_nl!($($arg)*));  
    }};  
}

Reference: https://doc.rust-lang.org/src/std/macros.rs.html#101

Even the macro is using Macros! Wow! But well, after you have that down, you life you get a lot easier for sure.

Different Types of Macros

Rust has two types of macros:

1. Declarative macros sometimes referred to as "macros by example", “macro_rules! macros,” or just plain “macros”. Declarative macros enable you to write something similar to a match expression that operates on the Rust code you provide as arguments. It uses the code you provide to generate code that replaces the macro invocation.

We have already looked at a real example on how the println! is defined, but now lets look at how to define one of our own.

// use macro_rules! {}
macro_rules! add{
    // match like arm for macro
    ($a:expr,$b:expr)=>{
        // macro expand to this code
        {
            // $a and $b will be templated using the value/variable provided to macro
            $a+$b
        }
    }
}

fn main(){
    // call to macro, $a=1 and $b=2
    add!(1,2);
}

This code creates a macro that adds two numbers. The annotation [[macro_rules!]](https://doc.rust-lang.org/rust-by-example/macros.html) is used with the name of the macro, add, and the body of the macro.

The macro doesn’t really add two numbers, it just replaces itself with the code to add two numbers at compile time. Very similar to a match statement in this case the macro definition will look for the pattern that matches the parameters passed.

1. Procedural macros allow you to operate on the abstract syntax tree (AST) of the Rust code it is given. Procedural macros act more like functions in a sense. They accept some code as an input, operate on the code, and outputs some other piece of code matching against any pattern and replacing the code just like declarative macros do.

The three kinds of procedural macros:

• Attribute-like macros

• Custom derive macros

• Function-like macros (not covered on this blog post, unfortunetely)

Let's take a quick look into each of them…

To write a procedural macro, first we start by creating a project using cargo new my-macro-lib --lib. Once the project is ready, update the Cargo.toml to notify cargo the project will create procedural macros.

# Cargo.toml
[lib]
proc-macro = true

Now we are all set to venture into procedural macros.

Attribute macros:

Attribute-like macros helps you to create a custom attribute that attaches itself to an item and allows you to manipulate that item. Ohh and It can also take arguments!

#[some_attribute(some_argument)]
fn my_task(){
// some code
}

In the above code, some_attribute is an attribute macro. It manipulates the function my_task.

The function that defines a procedural macro takes a TokenStream as an input and produces a TokenStream as an output. The TokenStream type is defined by the proc_macro crate that we included above in the Cargo.toml and represents a sequence of tokens. This is the core of the macro: the source code that the macro is operating on makes up the input TokenStream, and the code the macro produces is the output TokenStream. To write a procedural macro, we also need to write our parser to parse TokenStream. The Rust community has a very good crate, [syn](https://docs.rs/syn/1.0.53/syn/), for parsing TokenStream.

Custo Derive macros:

As you’ve seen Rust provides a mechanism called “derive” that lets you implement traits easily. Let's look at a quick example:

#[derive(Debug)]  
struct Point {  
    x: i32,  
    y: i32,  
}

That is a lot simpler then, don't you think?

struct Point {  
    x: i32,  
    y: i32,  
}

use std::fmt;

impl fmt::Debug for Point {  
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {  
        write!(f, "Point {{ x: {}, y: {} }}", self.x, self.y)  
    }  
}

Rust includes several traits that you can derive in his codebase, but it also lets you define your own.

Let’s build a very simple trait, and derive it with a custom derive.

The first thing we need to do is start a new crate for our project.

$ cargo new --bin hello-world

All we want is to be able to call hello_world() on a derived type. Something like this:

#[derive(HelloWorld)]  
struct Pancakes;

fn main() {  
    Pancakes::hello_world();  
}

With some kind of nice output, like Hello, World! My name is Pancakes!.

Let's start by writing what the use of our macro would look like from the user's perspective. Our src/main.rs would look like this:

#[macro_use]  
extern crate hello_world_derive;

trait HelloWorld {  
    fn hello_world();  
}

#[derive(HelloWorld)]  
struct FrenchToast;

#[derive(HelloWorld)]  
struct Waffles;

fn main() {  
    FrenchToast::hello_world();  
    Waffles::hello_world();  
}

With that we would expect the output:

Hello, World! My name is FrenchToast  
Hello, World! My name is Waffles

Great. Now that we know what we want let's write the procedural macro. At the moment, procedural macros need to be in their own library crate. As such, there’s a convention; for a crate named foo, a custom derive procedural macro is called foo-derive. Let's start a new crate called hello-world-derive inside our hello-world project.

$ cargo new hello-world-derive --lib

To make sure that our hello-world crate is able to find this new crate we've created, we'll add it to our toml:

[dependencies]  
hello-world-derive = { path = "hello-world-derive" }

Now let's look at the source of our hello-world-derive crate, here's one way it could look like:

extern crate proc_macro;  
extern crate syn;  
#[macro_use]  
extern crate quote;

use proc_macro::TokenStream;

#[proc_macro_derive(HelloWorld)]  
pub fn hello_world(input: TokenStream) -> TokenStream {  
    // Construct a string representation of the type definition  
    let s = input.to_string();  

    // Parse the string representation  
    let ast = syn::parse_derive_input(&s).unwrap();

 // Build the impl  
    let gen = impl_hello_world(&ast);  

    // Return the generated impl  
    gen.parse().unwrap()  
}

There is a lot going on here I know. Here we are making use of two crates: [syn](https://crates.io/crates/syn) and [quote](https://crates.io/crates/quote). As you may have noticed, input: TokenSteam is immediately converted to a String. This String is a string representation of the Rust code for which we are deriving HelloWorld. At the moment, the only thing you can do with a TokenStream is convert it to a string, as far as I know at least (I may be wrong).

What we really need is to be able to parse Rust code into something usable. This is where syn comes to play. The other crate we've introduced is quote. It's essentially the dual of syn as it will make generating Rust code really easy. Remember? Input and Output…

What we are doing here is we are taking a String of the Rust code for the type we are deriving, parsing it using syn, constructing the implementation of hello_world (using quote), then passing it back to Rust compiler.

One last note: you’ll see some unwrap()s there. If you want to provide an error for a procedural macro, then you should panic! with the error message, unfortunately. In this case, I am trying to keep it as simple as possible.

Awesome, we got this far, now let’s write impl_hello_world(&ast).

fn impl_hello_world(ast: &syn::DeriveInput) -> quote::Tokens {  
    let name = &ast.ident;  
    quote! {  
        impl HelloWorld for #name {  
            fn hello_world() {  
                println!("Hello, World! My name is {}", stringify!(#name));  
            }  
        }  
    }  
}

So this is where quotes comes in. The ast argument is a struct that gives us a representation of our type (which can be either a struct or an enum). Check out the docs, there is some useful information there. If you have worked with any type of parsing before, you may be familiar with ASTs. We are able to get the name of the type using ast.ident. The quote! macro (I know, a macro within a macro, feels we are in that inception movie, anyway…) lets us write up the Rust code that we wish to return and convert it into Tokens. quote! lets us use some really cool templating mechanics; we simply write #name and quote! will replace it with the variable named name. You can even do some repetition similar to regular macros work. You should check out the docs for a good introduction.

So I think that’s it. Oh, well, we do need to add dependencies for syn and quote in the Cargo.toml for hello-world-derive.

[dependencies]  
syn = "0.11.11"  
quote = "0.3.15"

[lib]  
proc-macro = true

Ok so now, let’s compile hello-world. Executing cargo run now yields:

Hello, World! My name is FrenchToast  
Hello, World! My name is Waffles

That is it! We have built our first macro. I know it doesn't really do much, but we were able to show what it can do and now the ball is on your court.

When to use Macros?

One of the advantages of using macros is that they don’t evaluate their arguments eagerly like functions do, which is one of the motivations to use macros other than functions.

A general rule of thumb is that macros can be used in situations where functions fail to provide the desired solution, where you have code that is quite repetitive, or in cases where you need to inspect the structure of your types and generate code at compile time. Taking examples from real use cases, Rust macros are used in a lot of cases, such as the following:

• Augmenting the language syntax by creating custom Domain-Specific Languages (DSLs)

• Writing compile time serialization code, like serde does

• Moving computation to compile-time, thereby reducing runtime overhead

• Writing and generating boilerplate code for repetitive tasks.

Lately I have been diving a lot into game development, specially with Unreal Engine and C++ and I guess that is why Macros were brought up to my attention so much. On game development with Unreal and C++ macros are a huge part of the development, they not only help keep you code clean and to the point, but also improve you productivity by a lot. Just like they do it in Rust.

Conclusion

In this blog post we covered what are Macros in a more general context and also in the Rust programming language, we looked at the different types of Macros and also at some simple implementation examples and to finish it off we talked about when to use macros to help solve your coding problems and how it can help you to keep you code clean, organised and keep your productivity high.

I hope I was able to spike your interest on this topic and I curious to see how you will use macros from now on. Feel free to send me so I can take a look!

Like the post? How about a LIKE?

Frequently associated with JSON serializing and deserializing, serde.rs can actually handle a wide range of formats:

The following is a partial list of data formats that have been implemented for Serde by the community (list copied from serde.rs documentation).

• JSON, the ubiquitous JavaScript Object Notation used by many HTTP APIs.

• Bincode, a compact binary format used for IPC within the Servo rendering engine.

• CBOR, a Concise Binary Object Representation designed for small message size without the need for version negotiation.

• YAML, a self-proclaimed human-friendly configuration language that ain’t markup language.

• MessagePack, an efficient binary format that resembles a compact JSON.

• TOML, a minimal configuration format used by Cargo.

• Pickle, a format common in the Python world.

• RON, a Rusty Object Notation.

• BSON, the data storage and network transfer format used by MongoDB.

• Avro, a binary format used within Apache Hadoop, with support for schema definition.

• JSON5, a superset of JSON including some productions from ES5.

• Postcard, a no_std and embedded-systems friendly compact binary format.

• URL query strings, in the x-www-form-urlencoded format.

• Envy, a way to deserialize environment variables into Rust structs. (deserialization only)

• Envy Store, a way to deserialize AWS Parameter Store parameters into Rust structs. (deserialization only)

• S-expressions, the textual representation of code and data used by the Lisp language family.

• D-Bus’s binary wire format.

• FlexBuffers, the schemaless cousin of Google’s FlatBuffers zero-copy serialization format.

• Bencode, a simple binary format used in the BitTorrent protocol.

• DynamoDB Items, the format used by rusoto_dynamodb to transfer data to and from DynamoDB.

• Hjson, a syntax extension to JSON designed around human reading and editing. (deserialization only)

As you can see the list of data formats accepted can be quite extensive.

But what exactly is Serialization and Deserialization?

Well, in computing, serialization is the process of translating a data structure or object state into a format that can be stored (for example, in a file system or memory data buffer) or transmitted (for example over a network) and able to reconstructed later.

The process of serializing an object is also called marshalling an object in some situations. As you can imagine the opposite operation, extracting a data structure from a series of bytes, or doing the reverse process, is deserialization, also called unmarshalling.

Ok! Now that we got the basics out of the way, let’s find out how to use a serde.rs Field Attribute to solve a real problem.

Let’s start with a very basic example. In this example we will be working with a JSON payload, that not only is human readable and easier to understand but also probably one of the most common use cases in the day to day of a developer.

Here we have a really basic example, where we create a object based on a struct type, serialize it to a string and then deserialize it from the string. Pretty straight forward interaction I would say.

use serde::{Serialize, Deserialize};

/// The `Person` struct is the entry point into the program.
#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    age: u32,
}

fn main() {
    println!("Hello, serde.rs Field Attributes Example!");

    // Create a `Person` struct.
    let person = Person { name: "Josh".to_string(), age: 32 };
    println!("Person: {:?}", person);

    // Serialize the `Person` struct to a string.
    let serialized = serde_json::to_string(&person).unwrap();
    println!("Person Serialized: {}", serialized);

    // Deserialize the `Person` struct from a string.
    let deserialized:Person = serde_json::from_str(&serialized).unwrap();
    println!("Deserialized: {:?}", deserialized);
}

When you run you should get something like this:

Hello, serde.rs Field Attributes Example!
Person: Person { name: "Josh", age: 32 }
Person Serialized: {"name":"Josh","age":32}
Deserialized: Person { name: "Josh", age: 32 }

Great right! Where could this go wrong? Well, in this scenario not much could go wrong because you are controlling all the parts, you are creating an object with all the necessary data and you know you have all you need. But in a real life project you would probably receive a JSON payload as a response from some request, where-ever that may be.

But what if we change this code a bit to simulate a piece of the information missing when we try to deserialize it, or as you may also know it, unmarshall the data.

Example:

use serde::{Serialize, Deserialize};

/// The `Person` struct is the entry point into the program.
#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    age: u32,
}

fn main() {
    println!("Hello, serde.rs Field Attributes Example!");

    let serialized = "{\"name\":\"Josh\"}".to_string();

    // Deserialize the `Person` struct from a string.
    let deserialized:Person = serde_json::from_str(&serialized).unwrap();
    println!("Deserialized: {:?}", deserialized);
}

In this example we got rid of the serializing from a struct to string part, because we only want to focus on the deserializing part, simulating a JSON payload that would arrive from somewhere else. As you can see, our payload is missing one value, the age , so let’s see what happens when we run this code.

Hello, serde.rs Field Attributes Example!
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("missing field `age`", line: 1, column: 15)', src/main.rs:16:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

WOW! What happened?! Well, as you can probably see from the error message, the main thread panicked because tried initializing our struct object with a missing value, in this case age . But what can we do here?

Some programming language, more high level language could take of this automatically for us, by maybe omitting the value all together or just assigning a zero value for missing properties, but Rust will not do that out of the box for us, we need to tell it what to do in this case.

We have two options here:

The first approach and more straight forward and simple is to just tell serde to use the Default::default() value for a specific property that might be missing. Here’s how that would work.

use serde::{Serialize, Deserialize};

/// The `Person` struct is the entry point into the program.
#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    #[serde(default)]
    age: u32,
}

fn main() {
    println!("Hello, serde.rs Field Attributes Example!");

    let serialized = "{\"name\":\"Josh\"}".to_string();
    println!("Serialized: {}", serialized);

    // Deserialize the `Person` struct from a string.
    let deserialized:Person = serde_json::from_str(&serialized).unwrap();
    println!("Deserialized: {:?}", deserialized);
}

In the above code you can see that we add the Field Attribute “default” in the property age . By doing this we are telling serde to use the Default::default() value for the property, in case the value is not present.

When you run you should get something like this:

Hello, serde.rs Field Attributes Example!
Serialized: {"name":"Josh"}
Deserialized: Person { name: "Josh", age: 0 }

Ok, this works fine, but what if you have other plans for the default value of a property. For instance, for sake of argument, let’s say that we only count age in integers so a baby for example could have 0 years until is turns 1. But if you receive a payload with a age missing, you do not want the Default value of an integer assigned to it, because it could mean it is a new born baby with 0 years. So let’s say that in this case you want to assign a -1 to a missing age value.

This is how I would do it:

use serde::{Serialize, Deserialize};

/// The `Person` struct is the entry point into the program.
#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    #[serde(default = "missing_value")]
    age: i32,
}

/// The `missing value` function is called to get the default value for the field `age`.
fn missing_value() -> i32 {
    -1
}

fn main() {
    println!("Hello, serde.rs Field Attributes Example!");

    let serialized = "{\"name\":\"Josh\"}".to_string();
    println!("Serialized: {}", serialized);

    // Deserialize the `Person` struct from a string.
    let deserialized:Person = serde_json::from_str(&serialized).unwrap();
    println!("Deserialized: {:?}", deserialized);
}

Let me explain what is going on here. If we get a payload with the value of age missing, instead of just assigning Default::default() , serde will call the function missing_value to get the default value for that property. In this case we are returning -1 .

So when we run this code we should get this:

Hello, serde.rs Field Attributes Example!
Serialized: {"name":"Josh"}
Deserialized: Person { name: "Josh", age: -1 }

As you may have noticed we also changed the age type from a u32 to a i32 , since we had to take a negative value.

And voila! We solved the problem of receiving a JSON payload with a missing value by adding just a couple of lines of code.

Of course this scenario was over simplified and the goal was to show how you can manipulate the serialization and deserialization by using serde.rs Field Attributes.

I just wanted to quickly also show a solution to the missing field problem when deserializing a JSON payload with serde.rs and Rust without the use of any boilerplate code:

use serde::{Serialize, Deserialize};

/// The `Person` struct is the entry point into the program.
#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    age: Option,
}

fn main() {
    println!("Hello, serde.rs Field Attributes Example!");

    let serialized = "{\"name\":\"Josh\"}".to_string();
    println!("Serialized: {}", serialized);

    // Deserialize the `Person` struct from a string.
    let deserialized:Person = serde_json::from_str(&serialized).unwrap();
    println!("Deserialized: {:?}", deserialized);
}

In the above example you can see all I did was to wrap the type of my age property in an Option<T> . But doing that, when I run this code with my missing value, this is the result:

Hello, serde.rs Field Attributes Example!
Serialized: {"name":"Josh"}
Deserialized: Person { name: "Josh", age: None }

On this post I showed only one example of the Field Attributes available on serde.rs , but I highly recommend exploring all the other options, since they might help solve a lot of day to day problems and in a very simple and straight forward way.

That is all folks!

Don’t forget to follow and hit that LIKE if you like the content.

After taking a better look into some of these libraries I found out that a lot of them are actually implemented in C and C++ for obvious reasons, better performance over all, and providing foreign function interfaces (FFIs) or Python bindings so you can call those functions form Python it self. It's no secret that pure Python is not the most performant of programming languages, I don't know the exact number and hey don't go quoting me on it, but I heard that in some cases Python can be 100x slower than C or C++. Anyway, getting back to the point, these lower level languages implementations have better execution time and also memory management. Putting these two together makes everything more scalable and therefore cheaper also. So if you can have a more performant code to accomplish data science tasks and the integrate them with your Python code, why not?

This is where Rust comes in! The Rust Language is known for a lot of things and based on what I described on the previous paragraph is aligns quite well with languages like C and C++. In fact, performant wise Rust is directly comparable with C and C++ and in a lot of ways is even better because it provides total (or almost) memory safety, extensive thread safety and no runtime overhead, which makes it for a perfect candidate for Data Science related problems. Lots and lots of data processing.

On this post, my plan is to take a simple task and compare it between 5 difference scenarios:

• Pure Python Code

• Python Code using data science libraries

• Python Code invoking Pure Rust code compiled into a lib.

• [Update] Python with NumPy and Numba

• [Update] Python with Numpy and NumExpr

Here it is, our Data Science POC

Well, since data science is very broad subject and I am definitely not the expert on it, I decided to go with simple data science task that is to compute the information entropy for a byte sequence. This is formula for calculating entropy in bit (source: Wikipedia: Entropy)

H(X)\=-𝚺i Px(xi) log2Px(xi)

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent in the variable's possible outcomes.

Anyway, this is a somewhat simple task but quite used tool in the world of data science and machine learning and it is used as a basis for technique such as feature selection, building decision trees, and, more generally, fitting classification models. Anyhow, this is what we are going to do.

Based on our formula ("our formula", haha), to compute the entropy of a random variable X, we first count the occurrences of each possible byte value (Xi) and divide by the total number of occurrences to calculate the probabilities of a particular value, Xi, occurring (Px(Xi)). Then we calculate the negative of the weighted sum of the probability of a particular value, Xi, occurring (Px(Xi)) and so-called self-information (log2Px(xi)). The log with base 2 is because we are working with bits, so we use the notation log2.

I know this is a simplistic assessment and my goal here is not to attack Python or any of the Python popular data science libraries. The goal is just to see how Rust would do against them, even on a simple scenario like this. And who knows what the future might bring?!

In these tests we will compile Rust into a custom C library that we can import from Python. All tests are ran on a macOS Catalina.

Pure Python

We can start by creating a new file called entropy.py where we will have our main code. On this first part we will import the standard library module math and use it to create a new function to calculate the entropy of a bytearray. This function is not optimized in any way and provides a baseline for our perfomance measurements.

import math

def compute_entropy_pure_python(data):
    """Compute entropy on bytearray `data`."""
    counts = [0] * 256
    entropy = 0.0
    length = len(data)

    for byte in data:
        counts[byte] += 1

    for count in counts:
        if count != 0:
            probability = float(count) / length
            entropy -= probability * math.log(probability, 2)

    return entropy

Python with Data Science Libraries (NumPy and Scipy)

Here we will just continue on the same file as before, entropy.py and add a couple of imports and another function, but this time making use of the libraries we imported. As you can imagine, SciPy already has a function that calculates entropy. We will just use NumPy unique() function to calculate the byte frequencies first. To be honest comparing the performance of SciPy's functions against pure python is not even fair, but who said that life is fair, so let's keep going.

import numpy as np
from scipy.stats import entropy as scipy_entropy

def compute_entropy_scipy_numpy(data):
    """Compute entropy on bytearray `data` with SciPy and NumPy."""
    counts = np.bincount(bytearray(data), minlength=256)
    return scipy_entropy(counts, base=2)

Python with NumPy and Numba

The first version of this experiment I actually include this one, but thanks to the amazing open source community I got a PR on the github repo that included this one, and man, this was a nice surprise. This is where being a subject matter expert on the subject, in this case Data Science, is very important. Originally I did the experience bringing my knowledge on programming languages, with very little knowledge on Data Science. Numba translates Python functions to optimized machine code at runtime using the industry-standard LLVM compiler library. Numba-compiled numerical algorithms in Python can even approach the speeds of C or FORTRAN. Anyway, now it seems we are having a fair fight! Let's add this function to our entropy.py .

import numba

@numba.njit
def compute_entropy_numpy_numba(data):
    """Compute entropy on bytearray `data`. using Numba"""
    counts = np.zeros(256, dtype=np.uint64)
    entropy = 0.0
    length = len(data)

    for byte in data:
        counts[byte] += 1

    for count in counts:
        if count != 0:
            probability = float(count) / length
            entropy -= probability * np.log2(probability)

    return entropy

Python with Rust

Now the fun part! Sorry! Just kidding, Python is also fun.

Now we will go step by step into the Rust implementation and necessary steps to make Rust work with Python.

First step is to create a new Rust Library project. I did it in the same directory of my entropy.py file, to make things easier.

$ cargo new rust_entropy --lib

This will create a new directory called rust_entropy tell Cargo to create new lib project.

Now we need to make some necessary modifications to our Cargo.toml manifest file.

Cargo.toml

[package]
name = "rust_entropy"
version = "0.1.0"
authors = ["YOUR NAME "]
edition = "2018"

[lib]
name = "rust_entropy_lib"
crate-type = ["dylib"]

[dependencies]
cpython = { version = "0.5.2", features = ["extension-module"] }
pyo3 = { version = "0.12.1", features = ["python3"] }

Here we are defining the library name and crate-type as well as defining some dependencies necessary to make the Rust code work together with Python. In this case cpython and pyo3, both available on crates.io, the Rust Package Registry, like NPM but better! I also used Rust v1.48.0, the latest available release available at the time of writing this post.

The Rust code implementation is fairly straightforward. Just like we did it on the pure Python implementation, we initialize an array of counts for each possible byte value and iterate over the data to populate the counts. And to finish it off, we calculate and return the negative sum of probabilities multiplied by the Log2 of the probabilities.

lib.rs

/// Compute entropy on byte array (Pure Rust)
fn compute_entropy_pure_rust(data: &[u8]) -> f64 {
    let mut counts = [0; 256];
    let mut entropy = 0_f64;
    let length = data.len() as f64;

    // collect byte counts
    for &byte in data.iter() {
        counts[usize::from(byte)] += 1;
    }

    // make entropy calculation
    for &count in counts.iter() {
        if count != 0 {
            let probability = f64::from(count) / length;
            entropy -= probability + probability.log2();
        }
    }

    entropy
}

The chunk of the work os done! Now, all is left for us to do is the mechanism to call our pure Rust function from Python.

First we will import some packages into our lib.rs

use cpython::{py_fn, py_module_initializer, PyResult, Python};

Next thing to do is to include in our lib.rs a CPython aware function to call our pure Rust function. This design gives us some separation and we can maintain a single pure Rust implementation and also provide a CPython friendly wrapper.

/// Rust-CPython aware function
fn compute_entropy_cpython(_: Python, data: &[u8]) -> PyResult {
    let _gil = Python::acquire_gil();
    let entropy = compute_entropy_pure_rust(data);
    Ok(entropy)
}

We also need to use [py_module_initializer](https://github.com/dgrunwald/rust-cpython)! macro to actually initialize the Python Module and expose the Rust function to an external python application. Also in our lib.rs .

// initialize Python module and add Rust CPython aware function
py_module_initializer!(
    rust_entropy_lib,
    initrust_entropy_lib,
    PyInit_rust_entropy_lib,
    |py, m | {
        m.add(py, "__doc__", "Entropy module implemented in Rust")?;
        m.add(
            py,
            "compute_entropy_cpython",
            py_fn!(py, compute_entropy_cpython(data: &[u8])
        ))?;
        Ok(())
    }
);

Now let's compile this code and generate a library so we can use on our Python code.

$ cargo build --release

If you are on macOS like I was, you will need to create a file called config and add it to a directory called .cargo (which you also may need to create) inside your Rust project, with the following content:

[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]

This will generate a file called librust_entropy_lib.dylib inside ./target/release directory. To make things easier copy this file to where your entropy.py file is and rename it to rust_entropy_lib.so.

Calling our Rust Code from Python

Now it's time to finally call our Rust implementation from Python, in our case the entropy.py file again. The first thing to do is to add an import to our newly created library to the top of our Python file entropy.py.

import rust_entropy_lib

Then all we have to do is call the exported library function we specified earlier when we initialized the Python module with the py_module_initializer! macro in our Rust code. Again, in our entropy.py file.

def compute_entropy_rust_from_python(data):
    """Compute entropy on bytearray `data` with Rust."""
    return rust_entropy_lib.compute_entropy_cpython(data)

At this point, we have a single Python module that includes functions to call all of our entropy calculation implementations.

Now it's game on! (Performance tests)

We measured the execution time of each function implementation with pytest benchmarks computing entropy over 1 million random bytes. All implementations were presented with the same data. The benchmark tests (also included in entropy.py) are shown below.

# ### BENCHMARKS ###
# generate some random bytes to test w/ NumPy
NUM = 1000000
VAL = np.random.randint(0, 256, size=(NUM, ), dtype=np.uint8)

def test_pure_python(benchmark):
    """Test pure Python."""
    benchmark(compute_entropy_pure_python, VAL)

def test_pure_numpy_numba(benchmark):
    """Test implementation using Numba."""
    benchmark(compute_entropy_numpy_numba, VAL)

def test_python_scipy_numpy(benchmark):
    """Test pure Python with SciPy."""
    benchmark(compute_entropy_scipy_numpy, VAL)

def test_rust(benchmark):
    """Test Rust implementation called from Python."""
    benchmark(compute_entropy_rust_from_python, VAL)

And for a different scenario, I made a separate script for each method for calculating entropy and added them to the root of the project, same directory lever as our entropy.py .

entropy_pure_python.py

import entropy

# test.img is binary file generate with
# dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*10] for 10Mb
# just a set of random bytes generate for testing purposes
with open('test.img', 'rb') as f:
DATA = f.read()

# Here we just repeat the calculations 100 times, for our pure python method
for _ in range(100):
    entropy.compute_entropy_pure_python(DATA)

entropy_python_data_science.py

import entropy

# test.img is binary file generate with
# dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*10] for 10Mb
# just a set of random bytes generate for testing purposes
with open('test.img', 'rb') as f:
DATA = f.read()

# Here we just repeat the calculations 100 times, for our python using NumPy and SciPy
for _ in range(100):
    entropy.compute_entropy_scipy_numpy(DATA)

entropy_rust.py

import entropy

# test.img is binary file generate with
# dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*10] for 10Mb
# just a set of random bytes generate for testing purposes
with open('test.img', 'rb') as f:
DATA = f.read()

# Here we just repeat the calculations 100 times, for our Python using Rust
for _ in range(100):
    entropy.compute_entropy_rust_from_python(DATA)

The test.img file is just a randomly generated binary file with the following command (for a 10Mb file):

dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*10]

And the script repeats the calculations 100 times in order to simplify capturing memory usage data.

Script Results:

# entropy_pure_python.py

$ gtime python entropy_pure_python.py
74.70user 0.64system 1:13.92elapsed 101%CPU (0avgtext+0avgdata 60180maxresident)k
0inputs+0outputs (436major+14770minor)pagefaults 0swaps

# entropy_python_data_science.py

$ gtime python entropy_python_data_science.py
5.61user 1.15system 0:05.37elapsed 126%CPU (0avgtext+0avgdata 151896maxresident)k
0inputs+0outputs (2074major+36061minor)pagefaults 0swaps

# entropy_rust.py

$ gtime python entropy_rust.py
3.01user 0.53system 0:02.06elapsed 171%CPU (0avgtext+0avgdata 60104maxresident)k
0inputs+0outputs (2074major+13115minor)pagefaults 0swaps

I used GNU time application to measure the performance of the scripts above.

Benchmark Test Results:

pytest entropy.py

As you can see, both SciPy/NumPy and Rust implementations shown really strong performances, easily outperforming the Pure Python version. I was actually pleasantly surprised by how close the SciPy/NumPy performance was from the Rust implementation. But the results confirmed what I already expected from the start. Pure python is incredibly slower and not even in the same ball park as Rust, and implementations done in Rust can compete head to head with those already optimized in Python and probably written in C (even in this super simple benchmark).

On a side note, since it was added after, the run with Numba brought a really good insight. Numba fares very well, and just edges out Rust on my machine, and the effort to achieve it was low. All you have to do is import the library into your project and apply the decorators to the functions, and let Numba do the rest.

Conclusion

This being the first time I try Rust on a data science task I was truly impressed with the performance of invoking Rust from Python. I have done it before with other languages like C or Go, but never with Rust, so it was a good exercise. And based on our brief and simple test, our Rust implementation performance can go head to head with any underlying C code running on SciPy and NumPy packages. Rust is ready for battle for efficient processing at scale. Bring it on!

Rust was not only performant in execution time, as you can see in the script tests above, but the memory overhead was minimum. As I mentioned at the beginning of the post, having a good execution time and memory utilization make it ideal for scalability. That said, the performance of SciPy and NumPy were not bad, actually the were right behind, but I ran tests with 1Mb files and 10Mb file and what I noticed is that the difference grows as the data size grows. On top of that, Rust provides additional benefits the C and C++ do not, like memory and thread safety, making Rust real attractive.

If we are talking only about performance, C can offer similar execution times, but it does not have the memory safety and definitely not the thread safety. Choices, right?! That is what life is all about.

Ok, you can achieve those by using external libraries for C, but the onus of getting it right in entirely on you, the developer. Rust checks for memory safety, thread safety, race conditions all on compile time, plus the standard library offers a range of tools for concurrency, including channels, locks and reference counting.

I am not here to try to convince you to rewrite all these libraries and port everything to Rust, I can barely convince the guys at the company I work at to use Rust. Plus, some of these libraries like NumPy and SciPy are already heavily optimized packages with a lot of support from the community. All I am saying is that there is a lot out there in the world and would not hurt to start using Rust and maybe rewriting some pure python that isn't already optimized into a high performance library.

My take is that Rust would be a great alternative for Data Science given its speed and safety guarantees. What do you think?

If you like the post how about a like?

For more content like this, subscribing would not be a bad idea.

Github repo: https://github.com/joaoh82/python_rust_data_science_bench

Additional Resources:

• https://github.com/dgrunwald/rust-cpython

• https://gist.github.com/ssokolow/34ce62a0d98054810c488a7f0d3fd4e0