Overview of pyln-client implementation - Plugin.run() - Part 1

LNROOM #11April 25, 2023

In this episode, we look a pyln-client Python package implementation focusing specifically on the method run of the class Plugin.

Transcript with corrections and improvements

Install pyln-client

Let's install pyln-client in .venv Python virtual environment:

◉ tony@tony:~/lnroom:
$ python -m venv .venv
◉ tony@tony:~/lnroom:
$ source .venv/bin/activate
(.venv) ◉ tony@tony:~/lnroom:
$ pip install pyln-client
...

Setup

Here is my setup

(.venv) ◉ tony@tony:~/lnroom:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2
pyln-client  23.2

Start 2 Lightning nodes running on regtest

Let's start two Lightning nodes running on the Bitcoin regtest chain by sourcing the script lightning/contrib/startup_regtest.sh provided in CLN repository and by running the command start_ln:

(.venv) ◉ tony@tony:~/lnroom:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
(.venv) ◉ tony@tony:~/lnroom:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 697735
[2] 697769
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes
(.venv) ◉ tony@tony:~/lnroom:

We can check that l1-cli is just an alias for lightning-cli with the base directory being /tmp/l1-regtest:

(.venv) ◉ tony@tony:~/lnroom:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'

myplugin.py

To demonstrate how Plugin.run method works, we'll use myplugin.py plugin. This plugin registers the JSON-RPC method node-id to lightningd. That method returns the node id of the node running the plugin:

#!/usr/bin/env python

from pyln.client import Plugin

plugin = Plugin()

@plugin.method("node-id")
def node_id_func(plugin):
    node_id = plugin.rpc.getinfo()["id"]
    return {"node_id":node_id}

plugin.run()

If you don't know how to write Core Lightning plugins with pyln-client you can check Start writing Core Lightning plugins with pyln-client TODAY!.

Let's jump in our terminal, start myplugin.py plugin

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}

and call node-id method:

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}

Part of Plugin.run() "call stack"

The method Plugin.run() is an IO loop waiting for incoming JSON-RPC requests from lightningd.

Those incoming requests can be "normal" requests or notifications.

When an incoming request is received, it is passed to Plugin._multi_dispatch method.

And in the case of "normal" requests (what we are studying today), the method Plugin._multi_dispatch passes the request to Plugin._dispatch_request method.

In Plugin._dispatch_request:

  1. Plugin._exec_func method constructs the payload (what goes into the result field of the JSON-RPC response) corresponding to the method of the request and

  2. Request.set_result produces the JSON-RPC response to the request and passes it to Request._write_result method.

Then, Request._write_result replies to lightningd by writing to the plugin's stdout stream.

Finally, we are back in the IO loop waiting for incoming JSON-RPC request from lightningd.

What we've just described can be represented with the following schema where the methods in the boxes are the methods we are going to modify in that video to better understand how Plugin.run method works:

┌────────────┐
│Plugin.run()│
└┬───────────┘
 └── Plugin._multi_dispatch
     │  ┌────────────────────────┐
     └──│Plugin._dispatch_request│
        └┬───────────────────────┘
         ├── Plugin._exec_func
         │  ┌──────────────────┐
         └──│Request.set_result│
            └┬─────────────────┘
             └── Request._write_result
                 └── Plugin._write_locked
                     └── (write to Plugin.stdout)

Plugin.run

The method Plugin.run is defined in lightning:contrib/pyln-client/pyln/client/plugin.py like this:

class Plugin(object):
    ...
    def run(self) -> None:
        # If we are not running inside lightningd we'll print usage
        # and some information about the plugin.
        if os.environ.get('LIGHTNINGD_PLUGIN', None) != '1':
            return self.print_usage()

        partial = b""
        for l in self.stdin.buffer:
            partial += l

            msgs = partial.split(b'\n\n')
            if len(msgs) < 2:
                continue

            partial = self._multi_dispatch(msgs)

The requests received by run method are stored in the first field of msgs array which is then passed to Plugin._multi_dispatch.

Let's adds the following Python snippet

with open("/tmp/myplugin_out", "a") as output:
    output.write(f"\n--------\n---> in 'Plugin.run'\n{repr(msgs)}\n")

to Plugin.run (in the file plugin.py in .venv virtual environment) like this

class Plugin(object):
    ...
    def run(self) -> None:
    # If we are not running inside lightningd we'll print usage
    # and some information about the plugin.
    if os.environ.get('LIGHTNINGD_PLUGIN', None) != '1':
        return self.print_usage()

    partial = b""
    for l in self.stdin.buffer:
        partial += l

        msgs = partial.split(b'\n\n')
        if len(msgs) < 2:
            continue
        with open("/tmp/myplugin_out", "a") as output:
            output.write(f"\n--------\n---> in 'Plugin.run'\n{repr(msgs)}\n")
        partial = self._multi_dispatch(msgs)

in order to see how does the requests look like by writing each incoming request to the file /tmp/myplugin_out.

Back to our terminal we restart our plugin in order to take into account the changes we made in pyln-client package:

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}

Now we can check that the file /tmp/myplugin_out contains the getmanifest and init requests sent by lightningd to myplugin.py plugin after we started it:


--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":96,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#97","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']

Then we can call node-id method like this

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}

and observe that node-id request has been written in the file /tmp/myplugin_out which is now:


--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":96,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#97","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#698621/cln:node-id#108", "params" :[ ] }', b'']

Plugin._muti_dispatch

The method Plugin._multi_dispatch is defined in lightning:contrib/pyln-client/pyln/client/plugin.py like this:

class Plugin(object):
    ...
    def _multi_dispatch(self, msgs: List[bytes]) -> bytes:
        """We received a couple of messages, now try to dispatch them all.

        Returns the last partial message that was not complete yet.
        """
        for payload in msgs[:-1]:
            # Note that we use function annotations to do Millisatoshi
            # conversions in _exec_func, so we don't use LightningJSONDecoder
            # here.
            request = self._parse_request(json.loads(payload.decode('utf8')))

            # If this has an 'id'-field, it's a request and returns a
            # result. Otherwise it's a notification and it doesn't
            # return anything.
            if request.id is not None:
                self._dispatch_request(request)
            else:
                self._dispatch_notification(request)

        return msgs[-1]

From msgs, Plugin._multi_dispatch instantiates request object of type Request, check that the request has an id (which is the case of node-id request for instance) and passes it to Plugin._dispatch_request (we don't look at notifications in this video).

As it might be useful to see how Request class is defined we reproduce below its Request.__init__ method:

class Request(dict):
    """A request object that wraps params and allows async return
    """
    def __init__(self, plugin: 'Plugin', req_id: Optional[str], method: str,
                 params: Any, background: bool = False):
        self.method = method
        self.params = params
        self.background = background
        self.plugin = plugin
        self.state = RequestState.PENDING
        self.id = req_id
        self.termination_tb: Optional[str] = None

Plugin._dispatch_request

In Plugin._dispatch_request method, we look for the name of the request in Plugin.methods dictionary. If we find it, we set method variable as the object of the class Method associated with that name in Plugin.methods. Then we try to execute the function method.func using Plugin._exec_func and set result variable to the value returned by Plugin._exec_func. If there is no error, we finally reply to lightningd passing result to Request.set_result method:

class Plugin(object):
    ...
    def _dispatch_request(self, request: Request) -> None:
        name = request.method

        if name not in self.methods:
            raise ValueError("No method {} found.".format(name))
        method = self.methods[name]
        request.background = method.background

        try:
            result = self._exec_func(method.func, request)
            if not method.background:
                # Only if this is a synchronous (background=False) call do we need to
                # return the result. Otherwise the callee (method) will eventually need
                # to call request.set_result or request.set_exception to
                # return a result or raise an exception.
                request.set_result(result)
        except Exception as e:
            if name in hook_fallbacks:
                response = hook_fallbacks[name]
                self.log((
                    "Hook handler for {name} failed with an exception. "
                    "Returning safe fallback response {response} to avoid "
                    "crashing the main daemon. Please contact the plugin "
                    "author!"
                ).format(name=name, response=response), level="error")

                request.set_result(response)
            else:
                request.set_exception(e)
            self.log(traceback.format_exc())

To have a better understanding of the composition of Plugin.methods dictionary, we can send the following expressions

from pyln.client import Plugin

plugin = Plugin()

to a Python interpreter and then looks at plugin.methods value:

>>> plugin.methods
{'init': <pyln.client.plugin.Method object at 0x7f03545d6080>,
'getmanifest': <pyln.client.plugin.Method object at 0x7f03545d5ed0>}
>>> plugin.methods["init"]
<pyln.client.plugin.Method object at 0x7f03545d6080>
>>> plugin.methods["init"].name
'init'
>>> plugin.methods["init"].func
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7f03545d7d60>>

It means that when myplugin.py receives the init request from lightningd, the function used in Plugin._dispatch_request to build the result field of the JSON-RPC reponse is Plugin._init method.

Now if we send the following Python decorator snippet

@plugin.method("node-id")
def node_id_func(plugin):
    node_id = plugin.rpc.getinfo()["id"]
    return {"node_id":node_id}

to our Python interpreter and look at plugin.methods value

>>> plugin.methods
{'init': <pyln.client.plugin.Method object at 0x7f03545d6080>,
'getmanifest': <pyln.client.plugin.Method object at 0x7f03545d5ed0>,
'node-id': <pyln.client.plugin.Method object at 0x7f0353423cd0>}

we can see that an entry for node-id method has been added. And if we look at

>>> plugin.methods["node-id"].func
<function node_id_func at 0x7f03534315a0>

we can see that when myplugin.py receives a node-id request from lightningd, the function used in Plugin._dispatch_request to build the result field of the JSON-RPC reponse is node_id_func function we defined just below the line @plugin.method("node-id").

How does this work? Well, if you are intereted you can check Overview of pyln-client implementation - @plugin.method() - Part 2.

Back to lightning:contrib/pyln-client/pyln/client/plugin.py file and Plugin._dispatch_request method.

Let's add the following Python snippet

with open("/tmp/myplugin_out", "a") as output:
    output.write(f"---> in 'Plugin._dispatch_request\n{name}\n"+
                 f"{repr(method.func)}\n"+
                 f"{json.dumps(result)}\n")

to Plugin._dispatch_request (in the file plugin.py in .venv virtual environment) like this

class Plugin(object):
    ...
    def _dispatch_request(self, request: Request) -> None:
        name = request.method

        if name not in self.methods:
            raise ValueError("No method {} found.".format(name))
        method = self.methods[name]
        request.background = method.background

        try:
            result = self._exec_func(method.func, request)
            with open("/tmp/myplugin_out", "a") as output:
                output.write(f"---> in 'Plugin._dispatch_request\n{name}\n"+
                             f"{repr(method.func)}\n"+
                             f"{json.dumps(result)}\n")
            if not method.background:
                # Only if this is a synchronous (background=False) call do we need to
                # return the result. Otherwise the callee (method) will eventually need
                # to call request.set_result or request.set_exception to
                # return a result or raise an exception.
                request.set_result(result)
        except Exception as e:
            if name in hook_fallbacks:
                response = hook_fallbacks[name]
                self.log((
                    "Hook handler for {name} failed with an exception. "
                    "Returning safe fallback response {response} to avoid "
                    "crashing the main daemon. Please contact the plugin "
                    "author!"
                ).format(name=name, response=response), level="error")

                request.set_result(response)
            else:
                request.set_exception(e)
            self.log(traceback.format_exc())

in order to see in the file /tmp/myplugin_out for each request from lightningd

  1. which function is used to produce the result field of the JSON-RPC response and

  2. what does the result field of the JSON-RPC response contains.

Back to our terminal we restart our plugin in order to take into account the changes we made in pyln-client package:

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}

Now we can check that the following

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":187,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']
---> in 'Plugin._dispatch_request
getmanifest
<bound method Plugin._getmanifest of <pyln.client.plugin.Plugin object at 0x7f8b44d3bc10>>
{"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#188","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']
---> in 'Plugin._dispatch_request
init
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7f8b44d3bc10>>
null

has been appended to /tmp/myplugin_out file.

Let's call node-id method like this

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}

and check that the following informations about node-id request

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#699655/cln:node-id#205", "params" :[ ] }', b'']
---> in 'Plugin._dispatch_request
node-id
<function node_id_func at 0x7f8b44c59fc0>
{"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}

have been appended to /tmp/myplugin_out file.

Request.set_result

In Plugin._dispatch_request we built a result that we passed to Request.set_result method which is defined in lightning:contrib/pyln-client/pyln/client/plugin.py like this:

class Request(dict):
    ...
    def set_result(self, result: Any) -> None:
        if self.state != RequestState.PENDING:
            assert(self.termination_tb is not None)
            raise ValueError(
                "Cannot set the result of a request that is not pending, "
                "current state is {state}. Request previously terminated at\n"
                "{tb}".format(state=self.state, tb=self.termination_tb))
        self.result = result
        self._write_result({
            'jsonrpc': '2.0',
            'id': self.id,
            'result': self.result
        })
        self.state = RequestState.FINISHED
        self.termination_tb = "".join(traceback.extract_stack().format()[:-1])

In that method we build the JSON-RPC response corresponding the request that Plugin.run received in the stdin stream of the plugin process. And that response is then passed to Request._write_result method

class Request(dict):
    ...
    def _write_result(self, result: dict) -> None:
        self.plugin._write_locked(result)

which delegates the work to Plugin._write_locked that finally writes to the plugin stdout stream. This leads to reply to lightningd request:

class Plugin(object):
    ...
    def _write_locked(self, obj: JSONType) -> None:
        # ensure_ascii turns UTF-8 into \uXXXX so we need to suppress that,
        # then utf8 ourselves.
        s = bytes(json.dumps(
            obj,
            cls=LightningRpc.LightningJSONEncoder,
            ensure_ascii=False
        ) + "\n\n", encoding='utf-8')
        with self.write_lock:
            self.stdout.buffer.write(s)
            self.stdout.flush()

And we are back at the beginning of Plugin.run IO loop waiting for incoming requests from lightningd.

To see exactly what is the JSON-RPC response that we send back to lightningd, we modify Request.set_result (in the file plugin.py in .venv virtual environment) like this:

class Request(dict):
    ...
    def set_result(self, result: Any) -> None:
        if self.state != RequestState.PENDING:
            assert(self.termination_tb is not None)
            raise ValueError(
                "Cannot set the result of a request that is not pending, "
                "current state is {state}. Request previously terminated at\n"
                "{tb}".format(state=self.state, tb=self.termination_tb))
        self.result = result
        response = {
            'jsonrpc': '2.0',
            'id': self.id,
            'result': self.result
        }
        with open("/tmp/myplugin_out", "a") as output:
            output.write(f"---> in 'Request.set_result'\n{json.dumps(response)}\n")
        self._write_result(response)
        self.state = RequestState.FINISHED
        self.termination_tb = "".join(traceback.extract_stack().format()[:-1])

Now each time our plugin receives a request from lightningd we'll append to /tmp/myplugin_out file the JSON-RPC response we send back to lightningd.

Back to our terminal we restart our plugin in order to take into account the changes we made in pyln-client package:

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}

Now we can check that the following

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":232,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']
---> in 'Plugin._dispatch_request
getmanifest
<bound method Plugin._getmanifest of <pyln.client.plugin.Plugin object at 0x7fbe22f23c10>>
{"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": 232, "result": {"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}}

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#233","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']
---> in 'Plugin._dispatch_request
init
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7fbe22f23c10>>
null
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": "cln:init#233", "result": null}

has been appended to /tmp/myplugin_out file.

Let's call node-id method like this

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}

and check that the following informations about node-id request

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#700027/cln:node-id#242", "params" :[ ] }', b'']
---> in 'Plugin._dispatch_request
node-id
<function node_id_func at 0x7fbe22e41fc0>
{"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": "cli:node-id#700027/cln:node-id#242", "result": {"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}}

have been appended to /tmp/myplugin_out file.

We are done!

Terminal session

We ran the following commands in this order:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install pyln-client
$ ./setup.sh
$ source lightning/contrib/startup_regtest.sh
$ start_ln
$ alias l1-cli
$ l1-cli plugin start $(pwd)/myplugin.py
$ l1-cli node-id
$ l1-cli plugin stop $(pwd)/myplugin.py
$ l1-cli plugin start $(pwd)/myplugin.py
$ l1-cli node-id
$ l1-cli plugin stop $(pwd)/myplugin.py
$ l1-cli plugin start $(pwd)/myplugin.py
$ l1-cli node-id
$ l1-cli plugin stop $(pwd)/myplugin.py
$ l1-cli plugin start $(pwd)/myplugin.py
$ l1-cli node-id

And below you can read the terminal session (command lines and outputs):

◉ tony@tony:~/lnroom:
$ python -m venv .venv
◉ tony@tony:~/lnroom:
$ source .venv/bin/activate
(.venv) ◉ tony@tony:~/lnroom:
$ pip install pyln-client
...
(.venv) ◉ tony@tony:~/lnroom:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2
pyln-client  23.2
(.venv) ◉ tony@tony:~/lnroom:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
(.venv) ◉ tony@tony:~/lnroom:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 697735
[2] 697769
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes
(.venv) ◉ tony@tony:~/lnroom:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/lnroom/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli node-id
{
   "node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"
}

pyln-client source code

During the demo, we've modified Request.set_result method of pyln-client package like this:

class Request(dict):
    ...
    def set_result(self, result: Any) -> None:
        if self.state != RequestState.PENDING:
            assert(self.termination_tb is not None)
            raise ValueError(
                "Cannot set the result of a request that is not pending, "
                "current state is {state}. Request previously terminated at\n"
                "{tb}".format(state=self.state, tb=self.termination_tb))
        self.result = result
        response = {
            'jsonrpc': '2.0',
            'id': self.id,
            'result': self.result
        }
        with open("/tmp/myplugin_out", "a") as output:
            output.write(f"---> in 'Request.set_result'\n{json.dumps(response)}\n")
        self._write_result(response)
        self.state = RequestState.FINISHED
        self.termination_tb = "".join(traceback.extract_stack().format()[:-1])

We also have modified Plugin._dispatch_request and Plugin.run methods of pyln-client package like this:

class Plugin(object):
    ...
    def _dispatch_request(self, request: Request) -> None:
        name = request.method

        if name not in self.methods:
            raise ValueError("No method {} found.".format(name))
        method = self.methods[name]
        request.background = method.background

        try:
            result = self._exec_func(method.func, request)
            with open("/tmp/myplugin_out", "a") as output:
                output.write(f"---> in 'Plugin._dispatch_request\n{name}\n"+
                             f"{repr(method.func)}\n"+
                             f"{json.dumps(result)}\n")
            if not method.background:
                # Only if this is a synchronous (background=False) call do we need to
                # return the result. Otherwise the callee (method) will eventually need
                # to call request.set_result or request.set_exception to
                # return a result or raise an exception.
                request.set_result(result)
        except Exception as e:
            if name in hook_fallbacks:
                response = hook_fallbacks[name]
                self.log((
                    "Hook handler for {name} failed with an exception. "
                    "Returning safe fallback response {response} to avoid "
                    "crashing the main daemon. Please contact the plugin "
                    "author!"
                ).format(name=name, response=response), level="error")

                request.set_result(response)
            else:
                request.set_exception(e)
            self.log(traceback.format_exc())

    ...
    def run(self) -> None:
    # If we are not running inside lightningd we'll print usage
    # and some information about the plugin.
    if os.environ.get('LIGHTNINGD_PLUGIN', None) != '1':
        return self.print_usage()

    partial = b""
    for l in self.stdin.buffer:
        partial += l

        msgs = partial.split(b'\n\n')
        if len(msgs) < 2:
            continue
        with open("/tmp/myplugin_out", "a") as output:
            output.write(f"\n--------\n---> in 'Plugin.run'\n{repr(msgs)}\n")
        partial = self._multi_dispatch(msgs)

Source code

myplugin.py

#!/usr/bin/env python

from pyln.client import Plugin

plugin = Plugin()

@plugin.method("node-id")
def node_id_func(plugin):
    node_id = plugin.rpc.getinfo()["id"]
    return {"node_id":node_id}

plugin.run()

myplugin_out


--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":96,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#97","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#698621/cln:node-id#108", "params" :[ ] }', b'']

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":187,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']
---> in 'Plugin._dispatch_request
getmanifest
<bound method Plugin._getmanifest of <pyln.client.plugin.Plugin object at 0x7f8b44d3bc10>>
{"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#188","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']
---> in 'Plugin._dispatch_request
init
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7f8b44d3bc10>>
null

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#699655/cln:node-id#205", "params" :[ ] }', b'']
---> in 'Plugin._dispatch_request
node-id
<function node_id_func at 0x7f8b44c59fc0>
{"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":232,"method":"getmanifest","params":{"allow-deprecated-apis":false}}', b'']
---> in 'Plugin._dispatch_request
getmanifest
<bound method Plugin._getmanifest of <pyln.client.plugin.Plugin object at 0x7fbe22f23c10>>
{"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": 232, "result": {"options": [], "rpcmethods": [{"name": "node-id", "category": "plugin", "usage": "", "description": "Undocumented RPC method from a plugin."}], "subscriptions": [], "hooks": [], "dynamic": true, "nonnumericids": true, "notifications": [], "featurebits": {}}}

--------
---> in 'Plugin.run'
[b'{"jsonrpc":"2.0","id":"cln:init#233","method":"init","params":{"options":{},"configuration":{"lightning-dir":"/tmp/l1-regtest/regtest","rpc-file":"lightning-rpc","startup":false,"network":"regtest","feature_set":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}}', b'']
---> in 'Plugin._dispatch_request
init
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7fbe22f23c10>>
null
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": "cln:init#233", "result": null}

--------
---> in 'Plugin.run'
[b'{ "jsonrpc" : "2.0", "method" : "node-id", "id" : "cli:node-id#700027/cln:node-id#242", "params" :[ ] }', b'']
---> in 'Plugin._dispatch_request
node-id
<function node_id_func at 0x7fbe22e41fc0>
{"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}
---> in 'Request.set_result'
{"jsonrpc": "2.0", "id": "cli:node-id#700027/cln:node-id#242", "result": {"node_id": "03fd248f93b8352d6cfc6a168f0fd10b1462f72d84b310a56f5c18076c5cf04fe4"}}

setup.sh

#!/usr/bin/env bash

ubuntu=$(lsb_release -ds)
lightningd=$(lightningd --version | xargs printf "lightningd %s\n")
python=$(python --version)
pyln_client=$(pip list | rg pyln-client)

printf "%s\n%s\n%s\n%s\n" "$ubuntu" "$python" "$lightningd" "$pyln_client"

Resources