Develop a tool for creating a Bitcoin transaction. This tool should have a good command-line interface and a test suite.







- Goal
- Contents
- Brief Summary
- Summary
- Notes
- Downloadable Assets
- How To Use The Tools
- Future Work
- Project Log









I've written a tool for creating and signing a Bitcoin transaction. It has robust input validation and optional logging.









I've written a tool for creating a Bitcoin transaction, and an accompanying test suite:
- create_transaction_4.py
- test_create_transaction_4.py


I used the existing tool create_transaction_3.py as my starting point.


The new tool has various improvements:

- Load all input data from outside the code via command-line arguments.
- More levels of input validation.
- A proper logging system, rather than print statements.
- Restructure into a tree of functions. (This is cleaner and makes it possible to write tests for the individual functions.)
- Write a test suite using onchain Bitcoin transactions.
- Remove "raw byte" option for random K value. (It's best to only use hex bytes internally so that debugging is easier and the data flow is simpler.)
- Include the random K value in the debug log output.
- Generate the txid and include it in the info log output.
- Validate the output addresses and the change address.
- Require private key and K values to be exactly 32 bytes long.
- Handle an input domain bug: Too many decimal places in a Bitcoin amount (details are in comment 2 (item 1) on Edgecase Bitcoin Storage Toolset version 2).




Tool inputs:
- Change address, Bitcoin inputs, Bitcoin outputs, private keys, random values, various control settings.

Tool outputs:
- (optional) Logging output, including the calculated txid.
- A signed transaction in hex bytes.













Important:

- Bitcoin inputs are actually unspent Bitcoin outputs from previous transactions.

- Each input in a standard Bitcoin transaction requires one ECDSA signature. A new random value must be used for every public ECDSA signature. If the transaction is signed and broadcast, and you create a new variant (e.g. if the first variant was not mined), you must supply a new random value for each input in the transaction. Two signatures made with the same private key and the same random value allows a third party to easily derive the private key.

- The input value that may be spent is limited to a default maximum percentage. All remaining value (minus the fee) will be sent to the change address. If you wish to send more than this percentage to other addresses, you must explicitly provide a new maxSpendPercentage value.

- The fee will be subtracted from the amount sent to the change address. If the fee is greater than the default maximum fee, you will need to explicitly provide a new maximum fee value.




Working definition of a standard transaction:
- It has at least one input and at least one output.
- All input and output addresses are Pay-To-Public-Key-Hash (P2PKH).
- All input scriptSigs contain uncompressed public keys.




For test data, I used Bitcoin transaction test set #2 as my source.
- The addresses that correspond to the private keys in the test transactions are available in Bitcoin address test set #2.
- Some transaction creation tests use fee rates instead of fees. These fee rates are available in comment 2 on the source article.




Logging:
- Logging at "info" level provides useful information: A bird's-eye view of the process.
- Logging at "debug" level provides a large amount of detailed information, including secret values (private keys and random values).
- At log level "info", the txid is printed just before the signed transaction.
- The txid is calculated by applying the SHA256 hash algorithm twice to the signed transaction binary data, and then converting the result to little-endian.
- Due to Transaction Malleability (transaction signatures can be slightly altered in-flight by third parties without compromising their validity), for some mined transactions the final txid may differ from the calculated version.

















Assets of this article:

List:



Asset: A tool that creates and signs a standard Bitcoin transaction. Python 2.7.12.
create_transaction_4.py

Asset: A test suite for the transaction creation tool. Python 2.7.12, Pytest 4.6.11.
test_create_transaction_4.py




Assets of other articles:

List:



Asset: A library of functions for handling standard Bitcoin data types. Python 2.7.
bitcoin_functions_2.py [paywalled]

Asset: An implementation of RIPEMD-160, written by Björn Edström. Python 2.7.
bjorn_edstrom_ripemd160.py [paywalled]

Asset: A Python implementation of ECDSA cryptography, written by Peter Pearson. Python 2.7.
ecdsa-0.10.tar.gz [paywalled]

Asset: A Python implementation of SHA256. Author unknown. Python 2.7.
pypy_sha256.py [paywalled]

Asset: A library of classes for standard Bitcoin transactions. Python 2.7.
transaction_3.py [paywalled]




Dependency tree:






The ecdsa-0.10.tar.gz package must be unpacked so that it can be imported and used.

Steps:

- Unpack the zipped tape archive file ecdsa-0.10.tar.gz, by opening a terminal, changing directory to the work directory, and running the following command:


- This unpacking produces a new directory named "ecdsa-0.10" in the work directory. The directory "ecdsa-0.10" contains a directory named "ecdsa". Copy the "ecdsa" directory into the work directory, using the following command:


- (optional) Delete ecdsa-0.10 and ecdsa-0.10.tar.gz.

















Notes:
- "spend" means to send bitcoin to any address that is not the change address.
- Be very careful when setting the changeAddress file path. The protection offered by the maxSpendPercentage setting only works if the changeAddress is correct.
- The maxSpendPercentage setting is a safeguard against accidentally choosing to send too much to a destination address.
- The maximum spend amount is checked prior to the subtraction of the fee from the change amount.
- The maxFee setting is a safeguard against accidentally choosing an excessive fee.




Example of creating a transaction:




This example uses the data for Transaction 1 in Bitcoin transaction test set #2.


Here is the data in the input files:



















Example of running the tests:






The Project Log section includes examples of:
- running the creation tool with debug & info level logging.
- creating a transaction with multiple inputs and outputs.
- running a single test with logging.

















Here are some possible future approaches:




1) Split the creation and signing operations into separate tools.

- The creation tool should produce a JSON file containing the transaction information.
-- This approach would make it somewhat easier to (if desired) manually transfer the transaction information to an offline signing machine.

- The primary utility of the creation tool would be the design of the transaction (choice of inputs, fee, various checks, etc). It can be run online i.e. it can connect directly to a node or blockchain explorer, so it can get up-to-date information for use during creation.

- The function of the signing tool would be to create the input signatures.
-- It should validate the unsigned transaction before signing it.
-- It should produce JSON-formatted output that can be read and checked by a human, and later reformatted into hex for broadcast.
-- Optionally, the signatures could be stored in files and later displayed in an easier-to-read format, so that they could be manual transferred out of an offline signing machine.




2) A transaction validation tool that can verify a signed transaction in both hex form and in a JSON format.




3) Options for the user to supply inputs & outputs with satoshi values instead of bitcoin values. Convert these to bitcoin values, then proceed as normal.




4) Options for the user to supply a fee value in bitcoin or a fee rate value in bitcoin/byte. Convert these to satoshi or satoshi/byte values, then proceed as normal.


















I have an existing tool: create_transaction_3.py

This will be my starting point.

It's a component of Edgecase Bitcoin Storage Toolset version 2, and there is a recipe for its use: Recipe for creating and signing a standard Bitcoin transaction #2.

I would like to make various improvements:
- Load all input data from outside the code via command-line arguments.
- A proper logging system, rather than print statements.
- Remove "raw byte" option for random K value. Best to stick with hex bytes as the default.
- Write a test suite using onchain Bitcoin transactions.
- Include the random K value in the debug log output.
- Generate the txid and include it in the info log output.
- Validate the output addresses and the change address.
- Require private key and K values to be exactly 32 bytes long.




Work machine:
- Name: Judgement
- Windows 10
- Windows Subsystem for Linux (WSL): Ubuntu 16.04
- I'm working in the WSL Ubuntu terminal.

My Python version:


My Pytest version:




Create a project directory, called:
"creating_a_standard_bitcoin_transaction"

Browse to Edgecase Bitcoin Storage Toolset version 2 and download the following assets:



Create a new directory named "work" in the project directory. Move the downloaded assets into the work directory.

Unpack the zipped tape archive file ecdsa-0.10.tar.gz, by opening a terminal, changing directory to the work directory, and running the following command:


This unpacking produced a new directory named "ecdsa-0.10" in the work directory. The directory "ecdsa-0.10" contains a directory named "ecdsa". Copy the "ecdsa" directory into the work directory, using the following command:


Delete ecdsa-0.10 and ecdsa-0.10.tar.gz.



For test data, I'll use Bitcoin transaction test set #2 as my source.

Notes:
- The addresses that correspond to the private keys in the transactions are available in Bitcoin address test set #2.
- Some transaction creation tests use fee rates instead of fees. These fee rates are available in comment 2 on the source article.




[development occurs here.]




I've written a tool for creating a Bitcoin transaction, and an accompanying test suite:
- create_transaction_4.py
- test_create_transaction_4.py




I drew on previous work published in these articles:
- Validating a standard Bitcoin address
- Generating a standard Bitcoin address #2
- Generating entropy from dice #2




Notes:
- "spend" means to send bitcoin to any address that is not the change address.
- Be very careful when setting the changeAddress file path. The protection offered by the maxSpendPercentage setting only works if the changeAddress is correct.
- The maxSpendPercentage setting is a safeguard against accidentally choosing to send too much to a destination address.
- The maximum spend amount is checked prior to the subtraction of the fee from the change amount.
- The maxFee setting is a safeguard against accidentally choosing an excessive fee.




Tool inputs:
- change address, inputs, outputs, private keys, random values, various control settings.

(Note: inputs = as-yet-unspent outputs)

Tool outputs:
- A signed transaction in hex bytes.




Dependency tree:





Working definition of a standard transaction:
- It has at least one input and at least one output.
- All input and output addresses are Pay-To-Public-Key-Hash (P2PKH).
- All input scriptSigs contain uncompressed public keys.

Important:
- Each input in a standard Bitcoin transaction requires one ECDSA signature. A new random value must be used for every public ECDSA signature. If the transaction is signed and broadcast, and you create a new variant (e.g. if the first variant was not mined), you must supply a new random value for each input in the transaction. Two signatures made with the same private key and the same random value allows a third party to easily derive the private key.
- The input value that may be spent is limited to a default maximum percentage. All remaining value (minus the fee) will be sent to the change address. If you wish to send more than this percentage to other addresses, you must explicitly provide a new maxSpendPercentage value.
- The fee will be subtracted from the amount sent to the change address. If the fee is greater than the default maximum fee, you will need to explicitly provide a new maximum fee value.












Let's record some examples of using
create_transaction_4.py









These examples use the data for Transaction 1 in Bitcoin transaction test set #2.




Here's the data in the input files:







Here's the help output.






Notes:
- There are default values for the input file paths. These can be altered within the code if desired.
- There are default values for the maxSpendPercentage, maxFee, feeType, fee, and feeRate. In order to exceed these limits, larger values will have to be explicitly supplied. These can be altered within the code if desired.








Here's an example of relying on the default values for maxFee and fee. feeType is by default "fee_rate" so we set it manually to "fee". maxSpendPercentage is not relevant for Transaction 1 because all the value will be sent to the change address.






An example of supplying a fee rate instead of a fee:






Notes:
- This example uses the data for Transaction 8 in Bitcoin transaction test set #2.
- In this transaction, two private key filepaths are supplied.




Here's the data in the input files:







An example of the error produced when the transaction will spend more than the maximum spend percentage:






An example of the error produced when the transaction will spend more than the maximum fee:










Let's run the tests.









Example: Run a specific test.






Example: Run a specific test quietly.







Examples: Print log data when running a test.







This is my pytest.ini:










Let's look at using logging when creating a transaction.


Notes:
- Logging at "info" level provides useful information: A bird's-eye view of the process.
- Logging at "debug" level provides a large amount of detailed information, including secret values (private keys and random values).




Here's the creation of Transaction 8, with logging at "info" level.







Notes:
- At log level "info", the txid is printed just before the signed transaction.
- The txid is calculated by applying the SHA256 hash algorithm twice to the signed transaction binary data, and then converting the result to little-endian.
- Due to Transaction Malleability (transaction signatures can be slightly altered in-flight by third parties without compromising their validity), for some mined transactions the final txid may differ from the calculated version.








Here's the creation of Transaction 1, with logging at "debug" level.









Good. That's the end of this project.